Hi All,

I am using Amelia to impute a time series data set generated from sensors in an industrial setting. Doing that for 8 variables (I only picked continuous variables for imputation) and ~40M readings (a reading/second).

 

Here is my Amelia code:

 

Test <- amelia(Query1[1:2e6,], m=3, p2s=2, cs=NULL, ts=”TIME”, incheck = T, parallel = “snow”, ncpus = 3, collect = T,

                Idvars = c(“D78”, “D82”, “D83”), lags = “C0”, “C1”, “C5”, “C6”, “C16”, “C17”, “C18”, “C19”),
                leads = “C0”, “C1”, “C5”, “C6”, “C16”, “C17”, “C18”, “C19”))

 

The code runs fine as long as the number of readings does not exceed ~1.2M. After that I receive the following error:

 

Error in unserialize(node$con) : error reading from connection

 

Some investigation shows that this has to do with the parallel workers. I noticed that the memory/worker does not exceed ~4GB and then goes back down before generating the error.

 

I am running Windows Server 2016 with Oracle Distribution of R v 3.3.0. Amelia is version 1.7.4.

 

I tried to troubleshoot with Oracle Community support before finding out that the issue also occurs when the data is a data.frame and not an ORE.Frame.

 

Here is the link for the troubleshooting thread - https://community.oracle.com/thread/4109587

 

Appreciate your support.

 

Regards,
Akthem