Continued Problems with Subsetting/Gold-Standard Data in Overimputation - Amelia

Matt Blackwell

21 Jan 21 Jan

11:05 p.m.

New subject: Continued Problems with Subsetting/Gold-Standard Data in Overimputation

Hi Sean, Apologies for taking so long to get back to you on this. I think what you are trying to accomplish is less for multiple overimputation. What moPrep is trying to do here is use the variance of the mismeasured observations relative to the variance of the gold standard observations. But if all of your mismeasured observations have 0 variance (since they are all 0!) then this strategy won't work. Thus, you can do one of two things: 1) Provide a standard error of the measurement error for those observations (using the error.sd argument) 2) Simply set those observations to NA and impute those observations like usual in amelia() (possibly with bounds argument to make sure they will be positive) Hope that helps! Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org On Wed, Dec 30, 2015 at 2:24 PM, Sean Kates <sk5350(a)nyu.edu> wrote:

...

After updating to the newest version of Amelia (1.7.4), I tried overimputing a dataset that has incorrect values in one of its variables. All of the error observations are measured identically (as zeros, where they should be positive). The code I originally used is below, and it triggers a warning of the type: "Some observations estimated with negative measurement error variance. Set to gold standard." dat<-data.frame(A, B, C, VS) mopd<-moPrep(dat, VS~VS, subset=VS<.0001) I looked through the github code as to what causes this error (other than, of course, the negative error variance), and more importantly, how to activate the gold.standard (which for my purposes is the rest of the values for VS) and presumably fix this issue. After trying quite a few different possible codings, I can't get it to work. I either receive the same error, or a host of errors surrounding how I've included gold.standard in the code. I would think it should be easy, since I'm basically bifurcating my data (all data under some amount is the subset measured with error; all data over the amount can be considered gold-standard data), but can't figure it out. Thanks for any help you can give, Sean -- Amelia mailing list served by HUIT [Un]Subscribe/View Archive: http://lists.gking.harvard.edu/?info=amelia More info about Amelia: http://gking.harvard.edu/amelia Amelia mailing list Amelia(a)lists.gking.harvard.edu To unsubscribe from this list or get other information: https://lists.gking.harvard.edu/mailman/listinfo/amelia

Reply

Matt Blackwell

4:43 p.m.

New subject: Continued Problems with Subsetting/Gold-Standard Data in Overimputation

Hi Skip, Unfortunately, there is no canned way to get all summary statistics, but you can use the "mi.meld" function to take a list of quantities of interest (e.g., means) and their SEs and combine them using the Rubin rules. If you don't care about the uncertainty and just want to average the quantities, then you can use these two bits of code: ## print the estimated means for each variable rowMeans(sapply(imp$imputations, colMeans)) ## print the estimated SDs for each variable rowMeans(sapply(imp$imputations, function(x) apply(x,2,sd)) Hope that helps! Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org On Tue, May 31, 2016 at 12:29 PM, Skip Barbour <russellbarbour(a)gmail.com> wrote:

...

Is there a way of getting the pooled descriptive statistics from the imputed datasets? I guess this must be spelled out somewhere, but reviewing the Amelia and Zeplig documentation I have not yet found it. Sorry for asking such a basic question Skip Barbour On Thu, Jan 21, 2016 at 11:05 PM, Matt Blackwell < mblackwell(a)gov.harvard.edu> wrote:

Hi Sean, Apologies for taking so long to get back to you on this. I think what you are trying to accomplish is less for multiple overimputation. What moPrep is trying to do here is use the variance of the mismeasured observations relative to the variance of the gold standard observations. But if all of your mismeasured observations have 0 variance (since they are all 0!) then this strategy won't work. Thus, you can do one of two things: 1) Provide a standard error of the measurement error for those observations (using the error.sd <https://urldefense.proofpoint.com/v2/url?u=http-3A__error.sd&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=1PZK_dz_3XlQ8Yml5HzwzZTMpzZoAqGD3DoCZYVAmGM&s=TvjWmjFLHNVK-WvaeeQjpuIAFv4OvMg7RmOVV7vERDo&e=> argument) 2) Simply set those observations to NA and impute those observations like usual in amelia() (possibly with bounds argument to make sure they will be positive) Hope that helps! Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mattblackwell.org&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=1PZK_dz_3XlQ8Yml5HzwzZTMpzZoAqGD3DoCZYVAmGM&s=843lXZd8Nax5M8DjLdgn3O92GJ9kusLrEIxKRhz0myU&e=> On Wed, Dec 30, 2015 at 2:24 PM, Sean Kates <sk5350(a)nyu.edu> wrote:

After updating to the newest version of Amelia (1.7.4), I tried overimputing a dataset that has incorrect values in one of its variables. All of the error observations are measured identically (as zeros, where they should be positive). The code I originally used is below, and it triggers a warning of the type: "Some observations estimated with negative measurement error variance. Set to gold standard." dat<-data.frame(A, B, C, VS) mopd<-moPrep(dat, VS~VS, subset=VS<.0001) I looked through the github code as to what causes this error (other than, of course, the negative error variance), and more importantly, how to activate the gold.standard (which for my purposes is the rest of the values for VS) and presumably fix this issue. After trying quite a few different possible codings, I can't get it to work. I either receive the same error, or a host of errors surrounding how I've included gold.standard in the code. I would think it should be easy, since I'm basically bifurcating my data (all data under some amount is the subset measured with error; all data over the amount can be considered gold-standard data), but can't figure it out. Thanks for any help you can give, Sean -- Amelia mailing list served by HUIT [Un]Subscribe/View Archive: http://lists.gking.harvard.edu/?info=amelia More info about Amelia: http://gking.harvard.edu/amelia Amelia mailing list Amelia(a)lists.gking.harvard.edu To unsubscribe from this list or get other information: https://lists.gking.harvard.edu/mailman/listinfo/amelia

-- Amelia mailing list served by HUIT [Un]Subscribe/View Archive: http://lists.gking.harvard.edu/?info=amelia More info about Amelia: http://gking.harvard.edu/amelia Amelia mailing list Amelia(a)lists.gking.harvard.edu To unsubscribe from this list or get other information: https://lists.gking.harvard.edu/mailman/listinfo/amelia

-- *There is nothing so fatal to character as half finished tasks.* David Lloyd George

Reply

Matt Blackwell

10:40 a.m.

New subject: Continued Problems with Subsetting/Gold-Standard Data in Overimputation

Hi Skip, I think that averaging across imputations for quantiles (including the median) is valid. What we are getting in this case is the posterior mean of the complete variable median (averaging over the distribution of the missing data): rowMeans(sapply(imp$imputations, function(x) apply(x,2, median)) It's a bit odd to think of, but this will work. It is, of course, difference than the posterior median of the complete variable median, which you could also calculate: apply(sapply(imp$imputations, function(x) apply(x,2, median), 2 median) Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org On Wed, Jun 1, 2016 at 9:24 AM, Skip Barbour <russellbarbour(a)gmail.com> wrote:

...

Matt, Thanks.. what about for pooled median values? Using this code for median "averages " across imputations would be inappropriate, no? Skip On Tue, May 31, 2016 at 4:43 PM, Matt Blackwell < mblackwell(a)gov.harvard.edu> wrote:

Hi Skip, Unfortunately, there is no canned way to get all summary statistics, but you can use the "mi.meld" function to take a list of quantities of interest (e.g., means) and their SEs and combine them using the Rubin rules. If you don't care about the uncertainty and just want to average the quantities, then you can use these two bits of code: ## print the estimated means for each variable rowMeans(sapply(imp$imputations, colMeans)) ## print the estimated SDs for each variable rowMeans(sapply(imp$imputations, function(x) apply(x,2,sd)) Hope that helps! Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mattblackwell.org&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=XjMRIoymm8BVqt4u3ZotvsQDHbh0k-IQboHm7mDGfJI&s=WlRtifgeZNP04r6SF8anqZ3dZR2ooecjAuoL_9lTZ-Q&e=> On Tue, May 31, 2016 at 12:29 PM, Skip Barbour <russellbarbour(a)gmail.com> wrote:

Is there a way of getting the pooled descriptive statistics from the imputed datasets? I guess this must be spelled out somewhere, but reviewing the Amelia and Zeplig documentation I have not yet found it. Sorry for asking such a basic question Skip Barbour On Thu, Jan 21, 2016 at 11:05 PM, Matt Blackwell < mblackwell(a)gov.harvard.edu> wrote:

Hi Sean, Apologies for taking so long to get back to you on this. I think what you are trying to accomplish is less for multiple overimputation. What moPrep is trying to do here is use the variance of the mismeasured observations relative to the variance of the gold standard observations. But if all of your mismeasured observations have 0 variance (since they are all 0!) then this strategy won't work. Thus, you can do one of two things: 1) Provide a standard error of the measurement error for those observations (using the error.sd <https://urldefense.proofpoint.com/v2/url?u=http-3A__error.sd&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=1PZK_dz_3XlQ8Yml5HzwzZTMpzZoAqGD3DoCZYVAmGM&s=TvjWmjFLHNVK-WvaeeQjpuIAFv4OvMg7RmOVV7vERDo&e=> argument) 2) Simply set those observations to NA and impute those observations like usual in amelia() (possibly with bounds argument to make sure they will be positive) Hope that helps! Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mattblackwell.org&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=1PZK_dz_3XlQ8Yml5HzwzZTMpzZoAqGD3DoCZYVAmGM&s=843lXZd8Nax5M8DjLdgn3O92GJ9kusLrEIxKRhz0myU&e=> On Wed, Dec 30, 2015 at 2:24 PM, Sean Kates <sk5350(a)nyu.edu> wrote: > After updating to the newest version of Amelia (1.7.4), I tried > overimputing a dataset that has incorrect values in one of its variables. > All of the error observations are measured identically (as zeros, where > they should be positive). The code I originally used is below, and it > triggers a warning of the type: "Some observations estimated with > negative measurement error variance. Set to gold standard." > > dat<-data.frame(A, B, C, VS) > mopd<-moPrep(dat, VS~VS, subset=VS<.0001) > > I looked through the github code as to what causes this error (other > than, of course, the negative error variance), and more importantly, how to > activate the gold.standard (which for my purposes is the rest of the values > for VS) and presumably fix this issue. After trying quite a few different > possible codings, I can't get it to work. I either receive the same error, > or a host of errors surrounding how I've included gold.standard in the > code. I would think it should be easy, since I'm basically bifurcating my > data (all data under some amount is the subset measured with error; all > data over the amount can be considered gold-standard data), but can't > figure it out. Thanks for any help you can give, > > Sean > > -- > Amelia mailing list served by HUIT > [Un]Subscribe/View Archive: > http://lists.gking.harvard.edu/?info=amelia > More info about Amelia: http://gking.harvard.edu/amelia > Amelia mailing list > Amelia(a)lists.gking.harvard.edu > > To unsubscribe from this list or get other information: > > https://lists.gking.harvard.edu/mailman/listinfo/amelia > -- Amelia mailing list served by HUIT [Un]Subscribe/View Archive: http://lists.gking.harvard.edu/?info=amelia More info about Amelia: http://gking.harvard.edu/amelia Amelia mailing list Amelia(a)lists.gking.harvard.edu To unsubscribe from this list or get other information: https://lists.gking.harvard.edu/mailman/listinfo/amelia

-- *There is nothing so fatal to character as half finished tasks.* David Lloyd George

Reply

Matt Blackwell

11:15 a.m.

New subject: Continued Problems with Subsetting/Gold-Standard Data in Overimputation

Hi Skip, Ah, I see. Maybe it would be easier to work with the following: imp.meds <- sapply(imp$imputations, function(x) apply(x,2, median) imp.meds This will be a (K x M) matrix where K is the number of variables in each data set and M is the number of imputations. The entry in row k and column m of this matrix will be the median value of the kth variable in the mth imputation. This will give you a sense of how the median of each variable changes across each of the imputed datasets. My guess is that these will be fairly stable across imputations. You can either choose one of the medians, take the average of the medians, or take the median of the medians. I think all of these will give fairly similar results and none of them will be very affected by outliers. Hope that helps! Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org On Wed, Jun 1, 2016 at 11:06 AM, Skip Barbour <russellbarbour(a)gmail.com> wrote:

...

Matt, Thanks for your excellent response. Our issue is that we want to use the median for a cut off, those below the median could be considered out of risk . The averages, on the other hand remain very high due to some individuals in the study that did not respond to treatment, so below average could still indicate poor health.Unfortunately we can't set a range either for reasons too complicated to discuss. I used one of the medians in the imputed data set to set the cut off. I wanted to avoid averaging individual values across the imputed datasets since it would also add uncertainty, when all I need is a relative measure below which indicates safe levels... if this makes any sense... This information is privileged,so I can't give details. Skip On Wed, Jun 1, 2016 at 10:40 AM, Matt Blackwell < mblackwell(a)gov.harvard.edu> wrote:

Hi Skip, I think that averaging across imputations for quantiles (including the median) is valid. What we are getting in this case is the posterior mean of the complete variable median (averaging over the distribution of the missing data): rowMeans(sapply(imp$imputations, function(x) apply(x,2, median)) It's a bit odd to think of, but this will work. It is, of course, difference than the posterior median of the complete variable median, which you could also calculate: apply(sapply(imp$imputations, function(x) apply(x,2, median), 2 median) Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mattblackwell.org&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=UsXWSNiCIkbEP0Kd5XJqm4SkVBzdO2A6NXyYBfNp-GI&s=WoGd06rAZWmkPIyxPCIykjSQCgXlHyjMvlVkwII6jvw&e=> On Wed, Jun 1, 2016 at 9:24 AM, Skip Barbour <russellbarbour(a)gmail.com> wrote:

Matt, Thanks.. what about for pooled median values? Using this code for median "averages " across imputations would be inappropriate, no? Skip On Tue, May 31, 2016 at 4:43 PM, Matt Blackwell < mblackwell(a)gov.harvard.edu> wrote:

Hi Skip, Unfortunately, there is no canned way to get all summary statistics, but you can use the "mi.meld" function to take a list of quantities of interest (e.g., means) and their SEs and combine them using the Rubin rules. If you don't care about the uncertainty and just want to average the quantities, then you can use these two bits of code: ## print the estimated means for each variable rowMeans(sapply(imp$imputations, colMeans)) ## print the estimated SDs for each variable rowMeans(sapply(imp$imputations, function(x) apply(x,2,sd)) Hope that helps! Cheers, Matt ~~~~~~~~~~~ Matthew Blackwell Assistant Professor of Government Harvard University url: http://www.mattblackwell.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mattblackwell.org&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=XjMRIoymm8BVqt4u3ZotvsQDHbh0k-IQboHm7mDGfJI&s=WlRtifgeZNP04r6SF8anqZ3dZR2ooecjAuoL_9lTZ-Q&e=> On Tue, May 31, 2016 at 12:29 PM, Skip Barbour < russellbarbour(a)gmail.com> wrote: > Is there a way of getting the pooled descriptive statistics from > the imputed datasets? I guess this must be spelled out somewhere, but > reviewing the Amelia and Zeplig documentation I have not yet found it. > Sorry for asking such a basic question > > Skip Barbour > > On Thu, Jan 21, 2016 at 11:05 PM, Matt Blackwell < > mblackwell(a)gov.harvard.edu> wrote: > >> Hi Sean, >> >> Apologies for taking so long to get back to you on this. I think what >> you are trying to accomplish is less for multiple overimputation. What >> moPrep is trying to do here is use the variance of the mismeasured >> observations relative to the variance of the gold standard observations. >> But if all of your mismeasured observations have 0 variance (since they are >> all 0!) then this strategy won't work. Thus, you can do one of two things: >> >> 1) Provide a standard error of the measurement error for those >> observations (using the error.sd >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__error.sd&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=1PZK_dz_3XlQ8Yml5HzwzZTMpzZoAqGD3DoCZYVAmGM&s=TvjWmjFLHNVK-WvaeeQjpuIAFv4OvMg7RmOVV7vERDo&e=> >> argument) >> >> 2) Simply set those observations to NA and impute those observations >> like usual in amelia() (possibly with bounds argument to make sure they >> will be positive) >> >> Hope that helps! >> >> Cheers, >> Matt >> >> ~~~~~~~~~~~ >> Matthew Blackwell >> Assistant Professor of Government >> Harvard University >> url: http://www.mattblackwell.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mattblackwell.org&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=EwICq0J5pL8CwgEJz8qkmauGonk0XmiLpxcYOEgk2a0&m=1PZK_dz_3XlQ8Yml5HzwzZTMpzZoAqGD3DoCZYVAmGM&s=843lXZd8Nax5M8DjLdgn3O92GJ9kusLrEIxKRhz0myU&e=> >> >> On Wed, Dec 30, 2015 at 2:24 PM, Sean Kates <sk5350(a)nyu.edu> wrote: >> >>> After updating to the newest version of Amelia (1.7.4), I tried >>> overimputing a dataset that has incorrect values in one of its variables. >>> All of the error observations are measured identically (as zeros, where >>> they should be positive). The code I originally used is below, and it >>> triggers a warning of the type: "Some observations estimated with >>> negative measurement error variance. Set to gold standard." >>> >>> dat<-data.frame(A, B, C, VS) >>> mopd<-moPrep(dat, VS~VS, subset=VS<.0001) >>> >>> I looked through the github code as to what causes this error (other >>> than, of course, the negative error variance), and more importantly, how to >>> activate the gold.standard (which for my purposes is the rest of the values >>> for VS) and presumably fix this issue. After trying quite a few different >>> possible codings, I can't get it to work. I either receive the same error, >>> or a host of errors surrounding how I've included gold.standard in the >>> code. I would think it should be easy, since I'm basically bifurcating my >>> data (all data under some amount is the subset measured with error; all >>> data over the amount can be considered gold-standard data), but can't >>> figure it out. Thanks for any help you can give, >>> >>> Sean >>> >>> -- >>> Amelia mailing list served by HUIT >>> [Un]Subscribe/View Archive: >>> http://lists.gking.harvard.edu/?info=amelia >>> More info about Amelia: http://gking.harvard.edu/amelia >>> Amelia mailing list >>> Amelia(a)lists.gking.harvard.edu >>> >>> To unsubscribe from this list or get other information: >>> >>> https://lists.gking.harvard.edu/mailman/listinfo/amelia >>> >> >> >> -- >> Amelia mailing list served by HUIT >> [Un]Subscribe/View Archive: >> http://lists.gking.harvard.edu/?info=amelia >> More info about Amelia: http://gking.harvard.edu/amelia >> Amelia mailing list >> Amelia(a)lists.gking.harvard.edu >> >> To unsubscribe from this list or get other information: >> >> https://lists.gking.harvard.edu/mailman/listinfo/amelia >> > > > > -- > *There is nothing so fatal to character as half finished tasks.* > David Lloyd George >

-- *There is nothing so fatal to character as half finished tasks.* David Lloyd George

Reply