I'm guessing that all the vars are dummies. If so, there are matrix algrbra tricks to 'sweep' out the effects of those from all the other vars and to then do the regression on a managable number. The result is the same. You will need to find out about this bit of plumbing unfortunately.
Gary
-----Original Message-----
From: Michael Law <mlaw(a)fas.harvard.edu>
Date: Saturday, Mar 25, 2006 1:06 pm
Subject: Re: [gov2001-l] R Memory Limits
Thanks, Ian, I'll give that a shot. I expect that's probably what he ran it in. I'll head to the lab tomorrow as my laptop only has 512 megs of ram and a really slow hard disk...
I've already reduced it to the minimum number of variables, so no progress to be made there. I'll let you know how I make out!
On the upside, I am getting similar results to him if I subsample a few times and average the result!
Thanks,
Mike
Ian Brett Yohai wrote:
STATA sometimes handles big datasets well. Can you reduce the size of the
dataset by dropping some variables that you don't need?
> Best,
Ian
> On Sat, 25 Mar 2006, Michael Law wrote:
> > Hi Gary,
>
> An OLS regression, not sure what program but I could ask. I've tried in
> SAS too, which uses the hard disk more than R, which I think stores
> everything in memory... but no luck there either.
>
> Thanks,
>
> Mike
>
> Gary King wrote:
> >> what model did the author run and in what program? A couple of thousand covariates would make most programs choke.
>> Gary
>> -----Original Message-----
>> From: Michael Law <mlaw(a)fas.harvard.edu>
>> Date: Saturday, Mar 25, 2006 12:46 pm
>> Subject: [gov2001-l] R Memory Limits
>>
>> Hi Everyone,
>>
>> So I'm running up against some limitations in R with respect to memory size. We have a huge dataset and the author we're replicating managed to fit a model with a couple thousand covariates. So, whenever I try to run the model R chokes on it, both on my computer and on the VNC server. I tried increasing the memory limit on my machine, but I ended up killing Windows in the process, can't figure out how to do it on the server, as the memory.limit() command doesn't seem to do anything.
>>
>> Any help would be most appreciated. Thanks everyone.
>>
>> cheers,
>>
>> Mike
>> _______________________________________________
>> gov2001-l mailing list
>> gov2001-l(a)lists.fas.harvard.edu
>> http://lists.fas.harvard.edu/mailman/listinfo/gov2001-l
>>
>>
>> _______________________________________________
>> gov2001-l mailing list
>> gov2001-l(a)lists.fas.harvard.edu
>> http://lists.fas.harvard.edu/mailman/listinfo/gov2001-l
>>
>> > _______________________________________________
> gov2001-l mailing list
> gov2001-l(a)lists.fas.harvard.edu
> http://lists.fas.harvard.edu/mailman/listinfo/gov2001-l
>
>
_____________________________________________
hello,
i have reprinted the first 5 rows of a matrix that combines 5 variables from the
gss. the first column is the respondent's age, and then columns 2-6 are the
respective ages of individuals they deem to be close friends. if they did not
list any close friends, 2-6 are NA because they did not report an age.
i need to create a variable that take the SD of each row of this matrix (the
standard deviation of the ages). i realize that the sd() function does not work
if you have NAs, so what can i do here? the first three observations of this new
variable should be: 0, 0, 7.74 (NAs are disregarded).
any suggestions? thanks!
best,
anant
> age_h[1:5,]
[,1] [,2] [,3] [,4] [,5] [,6]
[1,] 52 NA NA NA NA NA
[2,] 43 NA NA NA NA NA
[3,] 52 56 40 58 59 NA
[4,] 53 NA NA NA NA NA
[5,] 34 63 36 34 36 NA
--
Anant Thaker
athaker(a)fas.harvard.edu
Hi all,
Does anyone know how to load the longtable package into Latex to permit use
of the longtable command to produce tables that flow across multiple pages?
I tried going to Tools; Packages; List and Install when in Latex. It said
"Can't check signature. Public key...not found."
Thanks,
Jacqueline
--
Jacqueline Chattopadhyay
Ph.D Student, Government & Social Policy
Harvard University
jchattop(a)fas.harvard.edu
jchattop(a)gmail.com
714.313.7022
what model did the author run and in what program? A couple of thousand covariates would make most programs choke.
Gary
-----Original Message-----
From: Michael Law <mlaw(a)fas.harvard.edu>
Date: Saturday, Mar 25, 2006 12:46 pm
Subject: [gov2001-l] R Memory Limits
Hi Everyone,
So I'm running up against some limitations in R with respect to memory size. We have a huge dataset and the author we're replicating managed to fit a model with a couple thousand covariates. So, whenever I try to run the model R chokes on it, both on my computer and on the VNC server. I tried increasing the memory limit on my machine, but I ended up killing Windows in the process, can't figure out how to do it on the server, as the memory.limit() command doesn't seem to do anything.
Any help would be most appreciated. Thanks everyone.
cheers,
Mike
_______________________________________________
gov2001-l mailing list
gov2001-l(a)lists.fas.harvard.edu
http://lists.fas.harvard.edu/mailman/listinfo/gov2001-l
Hi Everyone,
So I'm running up against some limitations in R with respect to memory
size. We have a huge dataset and the author we're replicating managed
to fit a model with a couple thousand covariates. So, whenever I try to
run the model R chokes on it, both on my computer and on the VNC
server. I tried increasing the memory limit on my machine, but I ended
up killing Windows in the process, can't figure out how to do it on the
server, as the memory.limit() command doesn't seem to do anything.
Any help would be most appreciated. Thanks everyone.
cheers,
Mike
Hi all,
say, sigma2<-exp(gamma)
where gamma belongs to par
From the Hessian, we get V(gamma). How does one get V(sigma2)?
I bet V(exp(gamma)) is not equal to exp(V(gamma)).
Best,
Juan
Hi everyone,
I am hoping someone can help. For question 3c. I keep on getting this error
when I try to plot my results. Does anyone know what it means?
Error in if (from == to || length.out < 2) by <- 1 :
missing value where TRUE/FALSE needed
Best,
Melinda
On part 3d), could someone give me a hint as to what setting the location to
zero means, or setting the shape of the t distribution? If we are using,
say, rt(1000, df=1) to make random draws with df=1, then why do we need to
use the maximum likelihood estimates from part b?
When I optim( ) my student-t I get a message saying that my hessian is
singular. When I look at it, it has nothing than zeros.
Does anyone have had experience with this before and would know what is
going on here?
Juan
Hi All,
Just a quick note that my usual office hours will be delayed by a half
hour, so I'll start at 4:30 PM or just after (as opposed to 4 PM), and
will go until 6:30 PM.
Best,
Dan
----
Ph.D. Student
Department of Government
Harvard University
Tutor, Currier House
dhopkins(a)fas.harvard.edu
http://www.danhopkins.org
