Hi All,
I am using Amelia to impute a time series data set generated from sensors in an industrial
setting. Doing that for 8 variables (I only picked continuous variables for imputation)
and ~40M readings (a reading/second).
Here is my Amelia code:
Test <- amelia(Query1[1:2e6,], m=3, p2s=2, cs=NULL, ts=”TIME”, incheck = T, parallel =
“snow”, ncpus = 3, collect = T,
Idvars = c(“D78”, “D82”, “D83”), lags = “C0”, “C1”, “C5”, “C6”, “C16”,
“C17”, “C18”, “C19”),
leads = “C0”, “C1”, “C5”, “C6”, “C16”, “C17”, “C18”, “C19”))
The code runs fine as long as the number of readings does not exceed ~1.2M. After that I
receive the following error:
Error in unserialize(node$con) : error reading from connection
Some investigation shows that this has to do with the parallel workers. I noticed that the
memory/worker does not exceed ~4GB and then goes back down before generating the error.
I am running Windows Server 2016 with Oracle Distribution of R v 3.3.0. Amelia is version
1.7.4.
I tried to troubleshoot with Oracle Community support before finding out that the issue
also occurs when the data is a data.frame and not an ORE.Frame.
Here is the link for the troubleshooting thread -
https://community.oracle.com/thread/4109587
Appreciate your support.
Regards,
Akthem