[R-lang] Re: Why did I get different results when using different versions of lmer() ?

[Nathaniel Smith]

On Sun, Jun 20, 2010 at 10:01 AM, Xiao He <mr.xiaohe@gmail.com> wrote:
> Hi Nathaniel,
> Thank you so much for the extremely clear and very helpful explanations you
> provided. And I have to apologize for having not included real examples in
> my first post.  The differences are not drastic at all, as you can see
> below, so I supposed I don't need to be concerned about them.

Yes, those look well within the range expected. (And the differences
are probably much smaller than the error introduced by your
measurements, however you made them!)

> But there is
> one difference that seems intriguing to me. If you scroll down to the
> section about model comparisons. For the latest version, the DFs for the two
> compared models are 6 and 7 respectively. For the older version, the DFs are
> 5 and 6 respectively. Do you happen to know why there is such a difference?

It looks like newer versions of lme4 add 1 degree of freedom to
account for the residual variance parameter, which older versions
forgot to account for:
  http://thread.gmane.org/gmane.comp.lang.r.lme4.devel/722

In practice this doesn't matter for your calls to anova(), because
likelihood ratio tests don't care about the absolute degrees of
freedom, only the *difference* in the number of degrees of freedom.
Adding 1 to both of them doesn't change their difference, so it
doesn't affect likelihood ratio tests. You might need to be more
careful if doing likelihood ratio tests between an lmer() model and an
lm() model, or if using AIC to choose between totally different
models, though.

BTW, if you're using anova() to do likelihood ratio tests, then you
should be fitting your models with maximum likelihood (ML), not REML.
How you request this varies by lme4 version, though :-). In the
current version, you pass REML=FALSE to lmer. In older versions, I
think there it was method="ML" -- but I could be wrong, check
help(lmer).

-- Nathaniel

> Thank you again for your help!
> Best,
> Xiao
>>>>>>Results obtained with the last version<<<<<<
>
>> p0.lmer3
> Linear mixed model fit by REML
> Formula: logRT ~ MSubj * ESubj + (1 | Subject) + (1 | Item)
>    Data: p0
>    AIC   BIC logLik deviance REMLdev
>  748.3 779.6 -367.2    716.7   734.3
> Random effects:
>  Groups   Name        Variance  Std.Dev.
>  Item     (Intercept) 0.0058138 0.076248
>  Subject  (Intercept) 0.0551764 0.234897
>  Residual             0.1627922 0.403475
> Number of obs: 640, groups: Item, 32; Subject, 20
>
> Fixed effects:
>                Estimate Std. Error t value
> (Intercept)    5.913524   0.062909   94.00
> MSubjB         0.086620   0.045110    1.92
> ESubjB         0.009642   0.045110    0.21
> MSubjB:ESubjB -0.017068   0.063795   -0.27
>
> Correlation of Fixed Effects:
>             (Intr) MSubjB ESubjB
> MSubjB      -0.359
> ESubjB      -0.359  0.500
> MSbjB:ESbjB  0.254 -0.707 -0.707
>
>>>>>>Results obtained with an earlier version<<<<<<
>> p0.lmer3
> Linear mixed-effects model fit by REML
> Formula: logRT ~ MSubj * ESubj + (1 | Subject) + (1 | Item)
>    Data: p0
>    AIC   BIC logLik MLdeviance REMLdeviance
>  746.3 773.1 -367.2      716.7        734.3
> Random effects:
>  Groups   Name        Variance Std.Dev.
>  Item     (Intercept) 0.005813 0.076243
>  Subject  (Intercept) 0.055174 0.234890
>  Residual             0.162793 0.403476
> number of obs: 640, groups: Item, 32; Subject, 20
> Fixed effects:
>                          Estimate Std. Error t value
> (Intercept)    5.913524   0.062911   94.00
> MSubjB         0.086620   0.045110    1.92
> ESubjB         0.009642   0.045110    0.21
> MSubjB:ESubjB -0.017068   0.063795   -0.27
> Correlation of Fixed Effects:
>             (Intr) MSubjB ESubjB
> MSubjB      -0.359
> ESubjB      -0.359  0.500
> MSbjB:ESbjB  0.254 -0.707 -0.707
>>>>>>>>Model comparisons in the latest version<<<<<<<<<<<<<
> anova(p0.lmer3,p0.lmer4)
> Data: p0
> Models:
> p0.lmer4: logRT ~ MSubj * ESubj + (1 | Subject)
> p0.lmer3: logRT ~ MSubj * ESubj + (1 | Subject) + (1 | Item)
>                 Df    AIC    BIC  logLik  Chisq Chi Df Pr(>Chisq)
> p0.lmer4  6 733.75 760.52 -360.87
> p0.lmer3  7 730.66 761.89 -358.33 5.0928      1    0.02403 *
>
>>>>>>>>Model comparisons in an older version<<<<<<<<<<<<<
>> anova(p0.lmer4,p0.lmer3)
> Data: p0
> Models:
> p0.lmer4: logRT ~ MSubj * ESubj + (1 | Subject)
> p0.lmer3: logRT ~ MSubj * ESubj + (1 | Subject) + (1 | Item)
>                   Df    AIC    BIC  logLik  Chisq Chi Df Pr(>Chisq)
> p0.lmer4  5 731.75 754.06 -360.87
> p0.lmer3  6 728.66 755.42 -358.33 5.0928      1    0.02402 *
> ---
> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> ______________________________
>
>