[R-lang] What happens when a random effect has a st. dev. of close to zero?

[Roger Levy]

Hi Maureen,

On Aug 21, 2009, at 3:41 PM, Maureen Gillespie wrote:

> Hi everyone,
>  I have been using weighted empircal logit linear regression (Barr,  
> 2008) to analyze data from a number of agreement error production  
> experiments. (Just as a side note, I have run into lots of problems  
> trying to use logit mixed models for this data as errors are  
> extremely rare: certain conditions produce essentially no errors and  
> all other conditions rarely have higher than 15% error rates. If  
> anyone has a better solution than the emp.logit please let me know!)

Yes, I have had similar difficulties with logit models where at least  
one condition is error-free.  One thing you may be prone to running  
into in these cases is the unreliability of the Wald z-statistic.   
Search for "standard error is inflated" on this page:

  http://userwww.sfsu.edu/~efc/classes/biol710/logistic/logisticreg.htm

> That being said, I am running what is essentially a meta-analysis.   
> I have data from 5 experiments and 104 different items (some of  
> which appear in multiple experiments, some only appear in a single  
> experiment).  My model has two continuous predictors and two random  
> effects (experiment and item).
>
> lmer(emp.logit ~ IV1 + IV2 + (1|item) + (1|exp), data, weights)
>
> When I run the model, my estimates, standard errors, and t-values  
> all appear reasonable (i.e., comparable to other single random  
> effect models I have run using this technique on similar data).   
> There is no colinearity or anything else to suggest that something  
> is wrong.  But when I use pvals.fnc() to compute CIs and p values  
> for the estimates, I find that the experiment random effect has a  
> std. dev. of 0.0000 (5.0e-11 to be exact), and this seems to inflate  
> the CI of the intercept estimate (t = 17, but it's only marginal  
> significant w/ pvals from MCMC).  If I run the same model excluding  
> the experiment random effect, estimates do not change and the CIs  
> and p values for the intercept appear normal. Strangely (or maybe  
> not) the two models have the exact same log likelihoods.
>
> Is this just an extreme example of a random effect not being  
> necessary?
>
> And, more on the conceptual end of things, why would a near-zero  
> st.dev. of a random effect inflate CIs w/MCMC sampling?

I'm not sure what you mean by inflating the CI -- do you mean making  
the CI on the fixed-effect intercept larger than it is in a model  
without the random effect of experiment?

With such a small random effect of experiment, the model probably *is*  
telling you that you don't need it.  Try comparing your model's  
likelihood with the likelihood of a model that doesn't have the random  
effect of experiment -- the likelihoods should be very similar.   
(Technically it's best to use restricted maximum likelihood  
(REML=TRUE) when doing this, but that is the default so it looks like  
you're doing that already.

Best

Roger


>
>
> Thanks in advance,
>
> Maureen Gillespie
> Northeastern University
> _______________________________________________
> R-lang mailing list
> R-lang at ling.ucsd.edu
> http://pidgin.ucsd.edu/mailman/listinfo/r-lang

--

Roger Levy                      Email: rlevy at ling.ucsd.edu
Assistant Professor             Phone: 858-534-7219
Department of Linguistics       Fax:   858-534-4789
UC San Diego                    Web:   http://ling.ucsd.edu/~rlevy