On Wed, 8 May 2002, Matthew Vile wrote:
> Dr. King,
>
> I hope this email reaches you at a convenient time. I am student
> working on my dissertation in public opinion and I have been considering
> using your program Amelia to sovle some missing data problems.
>
> My question is - generally speaking, what is the maximum "missingness"
> that Amelia can reasonably handle. I have read your APSR article (King
> et al 2001), and I see that your Monte Carlo sims relied on a dataset in
> which @ 5% of the data were missing. In your opinion, could Amelia
> impute to a variable in which 50% were missing, 66%? Under what
> conditions would you feel comfortable doing this?
>
> Specific example - the variable measuring fear of assault was only asked
> to approxiamtely 700 individuals out of 1800. presuming I have
> indicators of this variable (I do have some) would you feel comfortable
> impute to the missing 1100 cases?
>
> I appreciate your consideration and any advice you might have.
The answer depends on both the pattern and the level of missingness, but
Amelia should work without a problem on the application you describe. The
more data are missing, however, the more model-dependent your results will
be. Of course, that would be true whether you use Amelia or any other
method.
Best of luck,
Gary King
: Gary King, King(a)Harvard.Edu http://GKing.Harvard.Edu :
: Center for Basic Research Direct (617) 495-2027 :
: in the Social Sciences Assistant (617) 495-9271 :
: 34 Kirkland Street, Rm. 2 HU-MIT DC (617) 495-4734 :
: Harvard U, Cambridge, MA 02138 eFax (928) 832-7022 :
-
amelia mailing list served by Harvard-MIT Data Center
List Address: amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
On Fri, 3 May 2002, McElroy, Brendan wrote:
> Dear Gary,
>
> Thanks for your response to the above request. I think I understand what
> you mean by estimating theta: take a random sample of the dataset, run EMis
> and take the average vector of coefficients and variance-covariance matrix
> from the five imputed datasets and input them back into Amelia using the
> AMmupr and AMsigpr options. I'm not sure what to do next, since I still
> can't load the full dataset into Amelia. Can you help?
i'd take a random sample and put that into amelia to get estimates of
theta. then, at least using the Gauss version, you would be able to use
that estimated theta matrix to produce the imputations on (all sequential)
subsets of the data. We should automate this, but I don't think its been
done yet.
>
> I've also got a second request. I can't seem to find anything on the STATA
> combining commands 'mi', either on STATA's net resources or on the
> statalist. Apparently there was a zip file on your website containing the
> ado files at some stage. If you still have them, can you send them on to
> me, or can you let me know where to go to get them?
I'd have a look at Clarify, also at my web page. Clarify is Amelia-ready.
Gary
>
> Thanks again.
>
> Brendan McElroy
> HRB Research Fellow
> Departments of Economics and General Practice
> Aras na Laoi
> University College Cork
> Western Road
> Cork
> Ireland
> Tel: +353 21 490 3522
>
>
> -----Original Message-----
> From: Gary King [mailto:king@harvard.edu]
> Sent: 27 April 2002 23:02
> To: McElroy, Brendan
> Cc: 'amelia(a)latte.harvard.edu'
> Subject: Re: [amelia] altering memory allocation in Amelia
>
>
>
> I don't know its ever been tested with that many observations.
> I think a reasonable procedure would be to take a random sample just for
> estimating theta and then doing the imputation for each observation.
> that would be computationally efficient and wouldn't lose very much
> efficiency. I'd have to look to see whether its possible to do this with
> the present version of Amelia...
>
> Gary
>
> On Fri, 26 Apr 2002, McElroy, Brendan wrote:
>
> > I'm new to Amelia and I'm having problems with memory size. I have a
> STATA
> > dataset with nine variables and 400,751 observations weighing in at 7.8Mb.
> > I can load three of the variables - cost, age and sex - into Amelia (for
> > windows) but the program crashes when I try to run it. There are only 29
> > missing records on the age variable and the other two are fully coded.
> The
> > program crashes immediately when I try to load the full dataset. One of
> the
> > variables - disability - has 58,162 missing records and this is what I
> > really want Amelia to help with. I guess I've two related questions: Is
> my
> > dataset so big that multiple imputation will take too long and I should
> > revert to something like listwise deletion or least squares imputation,
> > both of which STATA can handle easily? If it's not too big, how do I
> alter
> > the memory space allocated by the program?
> >
> > > Yours sincerely,
> > >
> > > Brendan McElroy
> > > HRB Research Fellow
> > > Departments of Economics and General Practice
> > > Aras na Laoi
> > > University College Cork
> > > Western Road
> > > Cork
> > > Ireland
> > > Tel: +353 21 490 3522
> > >
> >
> > -
> > amelia mailing list served by Harvard-MIT Data Center
> > List Address: amelia(a)latte.harvard.edu
> > Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
> >
>
-
amelia mailing list served by Harvard-MIT Data Center
List Address: amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
no, you ought to be able to do both.
Gary
On Tue, 30 Apr 2002, bob wrote:
> Thanks.
>
> Once I have identified country as fully observed (AMfully), should I NOT
> then declare it as nominal (AMnoms)? Does that tell Amelia to attempt to
> impute it?
>
> ----- Original Message -----
> From: Gary King <king(a)harvard.edu>
> To: bob <bob(a)idasact.org.za>
> Cc: Bob at UCT <rmattes(a)cssr.uct.ac.za>; Amelia Listserv
> <amelia(a)latte.harvard.edu>
> Sent: Tuesday, April 30, 2002 3:37 PM
> Subject: Re: Amelia Query
>
>
> > It might be that you have identified a fully observed variable as
> > something to be imputed (you could try it without it), but I'm not
> > positive. I'm forwarding this to the new Amelia listerv. Maybe one of my
> > colleagues or someone else can help?
> >
> > Gary
> >
> >
> > On Tue, 30 Apr 2002, bob wrote:
> >
> > > Gary,
> > >
> > > Thanks for your earlier reply re: Amelia. I think we have cleared up
> the
> > > working space problem.
> > >
> > > Now another challenge confronts us.
> > >
> > > During stage 1 of 4, (EM), the program bombs after iteration 34, and
> gives
> > > the message, "sweep: elements of m cannot be zero. Exec stopped in line
> > > 79."
> > >
> > > I have run this twice to make sure I had checked the correct global
> commands
> > > for the correct variablres. The program was using the conditional
> model. 2
> > > variables are fully obserfed, and 1 is nominal. The 12 category
> "Country"
> > > variable is both fully observed and nominal.
> > >
> > > Does the error message have something to do with this?
> > >
> > > Presently, there are 37 variables, and just over 20,000 observations.
> > >
> > > This is not time series data, but should both Country, as well as
> > > Rural/Urban be identified as cross sections?
> > >
> > > Any help you can give would be greatly appreciated.
> > >
> > > Regards,
> > >
> > > Bob Mattes
> > >
> > >
> > > ----- Original Message -----
> > > From: Gary King <king(a)harvard.edu>
> > > To: mike <mike(a)idasact.org.za>
> > > Cc: <ajoseph(a)fas.harvard.edu>; <tercer(a)latte.harvard.edu>;
> > > <kscheve(a)latte.harvard.edu>; Robert Mattes <bob(a)idasact.org.za>
> > > Sent: Thursday, March 28, 2002 3:05 PM
> > > Subject: Re: your mail
> > >
> > >
> > > >
> > > > have a look at this: http://gking.harvard.edu/amelia/node55.html
> > > > I think it will answer your question.
> > > > Gary King
> > > >
> > > > : Gary King, King(a)Harvard.Edu http://GKing.Harvard.Edu :
> > > > : Center for Basic Research Direct (617) 495-2027 :
> > > > : in the Social Sciences Assistant (617) 495-9271 :
> > > > : 34 Kirkland Street, Rm. 2 HU-MIT DC (617) 495-4734 :
> > > > : Harvard U, Cambridge, MA 02138 eFax (928) 832-7022 :
> > > >
> > > > On Thu, 28 Mar 2002, mike wrote:
> > > >
> > > > > Dear Gary King,
> > > > >
> > > > > This follows up on the message I left on your voice-mail earlier
> today.
> > > > >
> > > > > I'm a political scientist from Michigan State University who, with
> > > Robert Mattes of the University of Cape Town, co-directs the
> Afrobarometer.
> > > The Afrobarometer is a large-scale, cross-national survey research
> project
> > > on public attitudes to democracy and markets in 12 African countries.
> > > > >
> > > > > We are currently analyzing a fairly large data set (150 variables x
> > > 21,000 cases) from Round 1 of the Afrobarometer. It contains quite a
> bit of
> > > missing data, both randomly distributed and country specific. On the
> advice
> > > in the AMELIA manual, we have cut down on missingness by rescaling
> "don't
> > > knows". We have also reduced the core variables for analysis to 37
> before
> > > trying to implement AMELIA.
> > > > >
> > > > > Our basic problem is that AMELIA will not run when we include all
> 21,000
> > > cases. It keeps giving us an error message that says "insufficient
> > > workspace memory". We can get AMELIA to generate 5 imputed data sets
> for a
> > > sub-sample of 2000 cases. It takes about half an hour. And we can get
> > > AMELIA to complete the iterations in Step 1 on 5,000 cases. But the
> program
> > > bombs in Step 2 when it tries to impute covariances (after about 45
> > > minutes), again yielding the same error message. Attempts at 10,000 and
> > > 21,000 will not even start running.
> > > > >
> > > > > We have here a Pentium 4 computer with 1 Ghtz hard drive and 256 mg
> free
> > > RAM and 20 Gbyte disk.
> > > > >
> > > > > Our questions are as follows:
> > > > >
> > > > > * Is there a limit to the number of cases that AMELIA can handle?
> Will
> > > it work on a data set of 37 variables by 21,000 cases?
> > > > >
> > > > > * Does the error message refer to AMELIA's workspace or the
> computer's
> > > workspace? In other words, does the problem lie in the capacity of the
> > > hardware or the software?
> > > > >
> > > > > * If we include a 12-value multinomial variable for "country", does
> > > this take us over AMELIA's limit of 40 variables? We have tried to run
> the
> > > program both with and without this variable, with the same problematic
> > > result.
> > > > >
> > > > > * For your information, we have specified one nominal variable
> > > (country, when included) and two fully observed variables (one of which
> is
> > > country, when included).
> > > > >
> > > > > We will make one more effort to call you today before Cape Town
> closes
> > > down for Easter. In case you want to call, you can reach us at 011 27
> 83
> > > 234 0333. I have copied this message to your associates at Harvard in
> case
> > > you aren't in this week. If we have not spoken today, would you be so
> kind
> > > as to reply by e-mail, with a copy to bob(a)idasact.org.za.
> > > > >
> > > > > With many thanks for your time. We look forward to using your
> valuable
> > > program.
> > > > >
> > > > > Yours,
> > > > >
> > > > > Mike Bratton.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>
-
amelia mailing list served by Harvard-MIT Data Center
List Address: amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
It might be that you have identified a fully observed variable as
something to be imputed (you could try it without it), but I'm not
positive. I'm forwarding this to the new Amelia listerv. Maybe one of my
colleagues or someone else can help?
Gary
On Tue, 30 Apr 2002, bob wrote:
> Gary,
>
> Thanks for your earlier reply re: Amelia. I think we have cleared up the
> working space problem.
>
> Now another challenge confronts us.
>
> During stage 1 of 4, (EM), the program bombs after iteration 34, and gives
> the message, "sweep: elements of m cannot be zero. Exec stopped in line
> 79."
>
> I have run this twice to make sure I had checked the correct global commands
> for the correct variablres. The program was using the conditional model. 2
> variables are fully obserfed, and 1 is nominal. The 12 category "Country"
> variable is both fully observed and nominal.
>
> Does the error message have something to do with this?
>
> Presently, there are 37 variables, and just over 20,000 observations.
>
> This is not time series data, but should both Country, as well as
> Rural/Urban be identified as cross sections?
>
> Any help you can give would be greatly appreciated.
>
> Regards,
>
> Bob Mattes
>
>
> ----- Original Message -----
> From: Gary King <king(a)harvard.edu>
> To: mike <mike(a)idasact.org.za>
> Cc: <ajoseph(a)fas.harvard.edu>; <tercer(a)latte.harvard.edu>;
> <kscheve(a)latte.harvard.edu>; Robert Mattes <bob(a)idasact.org.za>
> Sent: Thursday, March 28, 2002 3:05 PM
> Subject: Re: your mail
>
>
> >
> > have a look at this: http://gking.harvard.edu/amelia/node55.html
> > I think it will answer your question.
> > Gary King
> >
> > : Gary King, King(a)Harvard.Edu http://GKing.Harvard.Edu :
> > : Center for Basic Research Direct (617) 495-2027 :
> > : in the Social Sciences Assistant (617) 495-9271 :
> > : 34 Kirkland Street, Rm. 2 HU-MIT DC (617) 495-4734 :
> > : Harvard U, Cambridge, MA 02138 eFax (928) 832-7022 :
> >
> > On Thu, 28 Mar 2002, mike wrote:
> >
> > > Dear Gary King,
> > >
> > > This follows up on the message I left on your voice-mail earlier today.
> > >
> > > I'm a political scientist from Michigan State University who, with
> Robert Mattes of the University of Cape Town, co-directs the Afrobarometer.
> The Afrobarometer is a large-scale, cross-national survey research project
> on public attitudes to democracy and markets in 12 African countries.
> > >
> > > We are currently analyzing a fairly large data set (150 variables x
> 21,000 cases) from Round 1 of the Afrobarometer. It contains quite a bit of
> missing data, both randomly distributed and country specific. On the advice
> in the AMELIA manual, we have cut down on missingness by rescaling "don't
> knows". We have also reduced the core variables for analysis to 37 before
> trying to implement AMELIA.
> > >
> > > Our basic problem is that AMELIA will not run when we include all 21,000
> cases. It keeps giving us an error message that says "insufficient
> workspace memory". We can get AMELIA to generate 5 imputed data sets for a
> sub-sample of 2000 cases. It takes about half an hour. And we can get
> AMELIA to complete the iterations in Step 1 on 5,000 cases. But the program
> bombs in Step 2 when it tries to impute covariances (after about 45
> minutes), again yielding the same error message. Attempts at 10,000 and
> 21,000 will not even start running.
> > >
> > > We have here a Pentium 4 computer with 1 Ghtz hard drive and 256 mg free
> RAM and 20 Gbyte disk.
> > >
> > > Our questions are as follows:
> > >
> > > * Is there a limit to the number of cases that AMELIA can handle? Will
> it work on a data set of 37 variables by 21,000 cases?
> > >
> > > * Does the error message refer to AMELIA's workspace or the computer's
> workspace? In other words, does the problem lie in the capacity of the
> hardware or the software?
> > >
> > > * If we include a 12-value multinomial variable for "country", does
> this take us over AMELIA's limit of 40 variables? We have tried to run the
> program both with and without this variable, with the same problematic
> result.
> > >
> > > * For your information, we have specified one nominal variable
> (country, when included) and two fully observed variables (one of which is
> country, when included).
> > >
> > > We will make one more effort to call you today before Cape Town closes
> down for Easter. In case you want to call, you can reach us at 011 27 83
> 234 0333. I have copied this message to your associates at Harvard in case
> you aren't in this week. If we have not spoken today, would you be so kind
> as to reply by e-mail, with a copy to bob(a)idasact.org.za.
> > >
> > > With many thanks for your time. We look forward to using your valuable
> program.
> > >
> > > Yours,
> > >
> > > Mike Bratton.
> > >
> > >
> > >
> >
>
>
-
amelia mailing list served by Harvard-MIT Data Center
List Address: amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
thanks for the note. i think it will install within a folder, but none of
the 'parent' folders can have spaces in their names. Maybe that was the
problem, or some other kind of read/write issue.
Gary King
: Gary King, King(a)Harvard.Edu http://GKing.Harvard.Edu :
: Center for Basic Research Direct (617) 495-2027 :
: in the Social Sciences Assistant (617) 495-9271 :
: 34 Kirkland Street, Rm. 2 HU-MIT DC (617) 495-4734 :
: Harvard U, Cambridge, MA 02138 eFax (928) 832-7022 :
On Fri, 26 Apr 2002, Robert Petrin wrote:
> Mr. King,
>
> FYI, I was able to solve the problem I encountered trying to
> install Ameilial 2.01:
>
> First, there was a read / write restriction on one of the drives
> on my system, as you suggested.
>
> Second, Amelia doesn't seem to install correctly if the Amelia
> folder is nested within another folder (at least on my machine running
> Windows 2000 Professional).
>
> Thanks again for your help.
>
> Rob Petrin
>
-
amelia mailing list served by Harvard-MIT Data Center
List Address: amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
I'm new to Amelia and I'm having problems with memory size. I have a STATA
dataset with nine variables and 400,751 observations weighing in at 7.8Mb.
I can load three of the variables - cost, age and sex - into Amelia (for
windows) but the program crashes when I try to run it. There are only 29
missing records on the age variable and the other two are fully coded. The
program crashes immediately when I try to load the full dataset. One of the
variables - disability - has 58,162 missing records and this is what I
really want Amelia to help with. I guess I've two related questions: Is my
dataset so big that multiple imputation will take too long and I should
revert to something like listwise deletion or least squares imputation,
both of which STATA can handle easily? If it's not too big, how do I alter
the memory space allocated by the program?
> Yours sincerely,
>
> Brendan McElroy
> HRB Research Fellow
> Departments of Economics and General Practice
> Aras na Laoi
> University College Cork
> Western Road
> Cork
> Ireland
> Tel: +353 21 490 3522
>
-
amelia mailing list served by Harvard-MIT Data Center
List Address: amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
Thanks an error that I haven't seen before. The problem looks like some
kind of read protection problem created by Windows. I'm CCing the Amelia
listserv; Does anyone else have an idea?
Gary King
On Thu, 25 Apr 2002, Robert Petrin wrote:
> Mr. King,
>
> I've been using your Windows application Amelial for some time with little
> or no diffuclty. (It's a wonderful piece of software and has been of
> great use to me in my research).
>
> Recently I had to reinstall all the software on my machine due to problems
> with one of my Windows applications. In the process, the Systems folks
> here at NORC repartitioned my hard drive to make the system more stable.
>
> I've recently made several attempts to reinstall Amelial 2.01 and am
> unable to get it to run. The error messages it produces are "Could not
> save window size information" and "error G0028:
> 'D:\PROGRAM\AMELIAL\gsrun.cfg': Can't open congifuration".
>
> When you have a moment, would you please let me know if the Amelial files
> have to be downloaded into a particular drive / directory. Any guidance
> would be appreciated.
>
> Thanks very much,
>
>
> Robert Petrin
> ----------------------------------------------------
> Graduate Student in Sociology, University of Chicago
> Research Analyst, Ogburn-Stouffer Center, NORC
> email: robp(a)cicero.spc.uchicago.edu
> office ph: (773) 256-6362
>
-
amelia mailing list served by Harvard-MIT Data Center
List Address: amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
On Mon, 22 Apr 2002, Malcolm Ree wrote:
> Dear Professor King,
>
> I have downloaded "Amelia" but cannot make it run. When I try to
> execute it from the Windows based icon, it flashes on the screen and
> quickly disappears. What am I doing wrong?
>
> Malcolm Ree
>
This is in our FAQ. Have a look at this:
http://gking.harvard.edu/amelia/node50.html
You might also join the Amelia listserv.
Gary King
: Gary King, King(a)Harvard.Edu http://GKing.Harvard.Edu :
: Center for Basic Research Direct (617) 495-2027 :
: in the Social Sciences Assistant (617) 495-9271 :
: 34 Kirkland Street, Rm. 2 HU-MIT DC (617) 495-4734 :
: Harvard U, Cambridge, MA 02138 eFax (928) 832-7022 :
-
amelia mailing list served by Harvard-MIT Data Center
amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
Note the new Amelia listserv; info at my web page. Comments below; others
may have more...
On Tue, 16 Apr 2002, David Sobek wrote:
> Professor King,
>
> I am attempting to use Amelia and have a couple of questions.
> 1. We are using a pooled-time series data set. Within this data set
> we have panels with all values missing. For example, we have no values of
> var1 for Nigeria, but we do have values of other variables. Amelia implies
> that it cannot impute any values for var1 for Nigeria, is this true?
if Nigeria is an entire variable, then there isn't anything you can do.
If the observations on Nigeria constitute some fraction of observations -
where all the countries are stacked say - then you're ok as long as you
have no dummy variable for Nigeria. if you need the latter, then you have
a problem (lack of data).
> 2. Even when I deleted the panels where we did not have any values for
> var1, Amelia still gave me a message that it could not impute values
> because some cases were missing all values. Do you have any
> suggestions as to why I am getting this error?
i'd run the descriptive statistics just to check.
it is also the case, the amelia uses listwise deletion to create starting
values and so if that would leave you with no obs, then you might have to
change the method of computing starting values.
> 3. Last, I also got an error message that said we were using too many
> variable names and had to set AMvarn = . However, I could not find
> that option in the option section.
now that's a good one. maybe my colleagues will have a suggestion. but
probably if you drop some of the variables with so many missing
observations, you'll be ok. I would start with that, in any event. Just
so you can see it work and then perhaps isolate what variable is causing
the problem.
Gary King
: Gary King, King(a)Harvard.Edu http://GKing.Harvard.Edu :
: Center for Basic Research Direct (617) 495-2027 :
: in the Social Sciences Assistant (617) 495-9271 :
: 34 Kirkland Street, Rm. 2 HU-MIT DC (617) 495-4734 :
: Harvard U, Cambridge, MA 02138 eFax (928) 832-7022 :
>
> Thank you for your time.
>
> Sincerely,
> David Sobek
>
-
amelia mailing list served by Harvard-MIT Data Center
amelia(a)latte.harvard.edu
Subscribe/Unsubscribe: http://lists.hmdc.harvard.edu/?info=amelia
