[R-lang] How to compare mixed logit models with crossed random effects

[Roger Levy]

There is a minor error in my post from earlier today that I should  
correct (see below):

On May 24, 2009, at 10:52 AM, Roger Levy wrote:

> Dear Linda,
>
> On May 20, 2009, at 3:34 AM, Linda Mortensen wrote:
>
>> Dear LanguageR users,
>>
>> I'm trying to fit a mixed logit model using the lmer function in  
>> the lme4 package. My question concerns the random effects part of  
>> this model (i.e., the random effects for my subjects and items) and  
>> how I decide between models that differ in the number of random  
>> effect terms that are estimated.
>
> First of all, in my assessment the problem of which random effects  
> terms to include in your model when the primary target of inference  
> is the fixed effects is still open.
>
>> So far, I have used two procedures:
>>
>> 1. For a given model, I remove a random effect term if it  
>> correlates very strongly with either the intercept or any of the  
>> other random effect terms. Eventually, I end up with a model in  
>> which all correlations are modest.
>
> This is an interesting idea, but I would emphasize two things:
>
> 1) it's important to distinguish between positive and negative  
> correlations.  A strong negative correlation is telling you  
> something very important about your dataset.  Imagine a word  
> recognition task where the response variable is correct answer and  
> the covariate x1 is word frequency.  A strong negative correlation  
> between intercept and x1 is telling you that participants who answer  
> more correctly overall are less sensitive to word frequency, and  
> vice versa, and that this is a very reliable generalization.  You  
> can see this in model log-likelihoods too: compare the two lmer  
> model fits below.
>
> set.seed(9)
> library(mvtnorm)
> library(lme4)
> k <- 10
> n <- 1000
> cl <- gl(k,1,n)
> x1 <- runif(n)
> sigma <- matrix(c(1,-1.8,-1.8,4),2,2)
> b <- rmvnorm(k,mean=c(0,0),sigma)
> eta <- b[cl,1] + b[cl,2]*x1
> y <- rbinom(n,1,exp(eta)/ (1+exp(eta)))
> lmer(y ~ 1 + (1 | cl),family="binomial")
> lmer(y ~ 1 + (x1 | cl),family="binomial")
>
> 2) When you say "remove a term", what really would be justified is  
> if the random parameters for covariates x1 and x2 are correlated at  
> >0.99, create a third, "proxy" parameter x12=x1+x2, add x12 to the  
> random-effects structure, and drop x1 and x2.  This would save you  
> two parameters at basically no modeling cost.

This proxy parameter x12 should be equal to x1+C*x2, for some value of  
C which you could read off of the old model fit where x1 and x2 are  
separate (divide the standard deviation of the random effect for x2 by  
the standard deviation for x1).

>
>
>> 2. I compare the quasi-log likelihood (logLik) values of a model  
>> with a given random effect term (e.g. an interaction term, ... (1 +  
>> a * b | sub) and of a model without that term (... (1 + a + b |  
>> sub). If the logLik values are very similar (i.e., if the value is  
>> not, or at least not much, smaller for the model without the term  
>> than for the model with the term), I go for the former model.
>
> This is OK, and more of the recommended practice (see Baayen et al.,  
> 2008, for discussion with respect to linear mixed-effect models).   
> You can actually do a likelihood-ratio test, though with the dual  
> caveats that (a) Laplace-approximated log-likelihood is not true  
> loglikelihood; and (b) the test is conservative.
>
>> Is it acceptable to select a model on the basis of this comparison?  
>> Or, when the logLik values are similar (which they usually are for  
>> my models), should I instead look at the measures of likelihood  
>> that take into account the number of parameters in a model when  
>> evaluating its fit (i.e., AIC, BIC, deviance)? According to these  
>> other measures, a simple model seems always to be better than a  
>> more complex one, but if I want to rule out that my fixed effects  
>> can be explained, in part, by random effects for subjects and  
>> items, then a simple model (with few random effects) is not  
>> necessarily better than a complex one, I would think.
>
> Well, first of all the deviance is just -2*logLik.  The AIC and BIC  
> are still dominated by log-likelihood too. And it's not always going  
> to be the case that the logLik will not be appreciably better for  
> more complex models -- see my above example.  Finally, I'd agree  
> with you that it's better to be cautious and include the extra, more  
> complex terms if you want to be sure that you have a "real" fixed  
> effect.
>
> Hope this helps.
>
> Best
>
> Roger
>
> --
>
> Roger Levy                      Email: rlevy at ling.ucsd.edu
> Assistant Professor             Phone: 858-534-7219
> Department of Linguistics       Fax:   858-534-4789
> UC San Diego                    Web:   http://ling.ucsd.edu/~rlevy
>
>
>
>
>
>
>
>
> _______________________________________________
> R-lang mailing list
> R-lang at ling.ucsd.edu
> http://pidgin.ucsd.edu/mailman/listinfo/r-lang

--

Roger Levy                      Email: rlevy at ling.ucsd.edu
Assistant Professor             Phone: 858-534-7219
Department of Linguistics       Fax:   858-534-4789
UC San Diego                    Web:   http://ling.ucsd.edu/~rlevy