[R-lang] Re: Why did I get different results when using different versions of lmer() ?

[Nathaniel Smith]

On Sat, Jun 19, 2010 at 7:39 PM, Xiao He <praguewatermelon@gmail.com> wrote:
> Dear R-lang users,
> I recently started using lmer() and the languageR package. Because the
> latest version of lme4 package does not support MCMC sampling (error
> message: MCMC sampling is not yet implemented in lme4_0.999375-27), so I
> cannot use pvals.fnc() to obtain p-values. As a result, I decided to go back
> to an older version of lme4() that does support MCMC sampling. However, I've
> noticed when I tried to ran the exact data set on the two versions of lme4
> packages, the results, such as t-values, SE, coefficients, I got from the
> two versions of lmer() were different. I wonder if I should be concerned
> about it. Thank you guys in advance

How different are the results we're talking about? Can you give an
example? It's a fact of life when doing numerical computations on the
computer that, since you can't represent real numbers exactly with a
finite number of digits, there will be some small amount of rounding
error in every calculation. So two different pieces of code that *in
theory* should give the same result, in practice will often give
slightly different answers, just because they performed mathematically
equivalent operations in slightly different orders or whatever and
that caused different rounding errors.

As long as the differences are numerically small, I wouldn't worry
about it -- different versions of lme4 use radically different
numerical methods to solve the same problem, so some discrepancies are
to be expected. Doug Bates keeps figuring out better ways to fit mixed
effect models, and then rewriting everything to use the new, better
method. In the long run, this is good, but in the short term, this is
why you don't have MCMC sampling -- he rewrote the core fitting
routines, but hasn't gotten around to rewriting the MCMC code to
match.

This is also why you see fewer errors when using the new version of
lmer -- some problems are what's called "numerically sensitive",
meaning that the answer is well defined in theory, but if you're not
careful then in practice rounding error will accumulate until the
algorithm breaks. This often happens for problems that are "almost"
ill-defined, like when you only *just* have enough data to fit your
model. Newer versions of lmer use methods that are less susceptible to
these problems, and will be more successful at fitting models that are
"on the edge" like this.

You might want to check with the R-sig-mixed-models list too. They'll
have a better idea what the current status of lme4 is; it moves too
fast for me to keep track!

Hope that helps,
-- Nathaniel