[R-lang] comparing two mixed-effects models

[T. Florian Jaeger]

On 5/8/07, shabnam shademan <shademan at ucla.edu> wrote:
>
>  Hi all –
>
>
>
> I am unsophisticated user in need of some help.  I have a study in which I
> have two random factors: subject and stimuli item.  I also have 3 fixed
> effects:
>
> 1. y=similarity-based score (continuous values)
>
> 2. Age (has two levels "young" and "old")
>
> 3. x=grammatical-based score (continuous values).
>
> My dependent variable is z=item ratings from subjects.
>
>
>
> I am using the following formula to predict the effect of each factor on
> z:
>
>
>
> fit1 <- lmer(Z ~Age * X * Y + (1 | Subject) + (1 | Stimulus), method =
> "ML", data = d)
>
>
>
> furthermore, I have the option of using a different grammatical model in
> order to calculate values for X.  I will call this X2.  This means that I
> could also get a fit in the following way:
>
>
>
> fit2 <- lmer(Z ~Age * X2 * Y + (1 | Subject) + (1 | Stimulus), method =
> "ML", data = d)
>
Here are the questions:
>
> 1. Is there anyway to compare fit1 and fit2?
>


Dear Shabnam,

yes. you can compare the impact of the two factors, but it is not done by
doing an anova over the two above models. Also, you have to be aware that X1
and X2 may be collinear, which may make your result unreliable. see below

2. would anova (fit1, fit2) be appropriate in this case?  (if possible,
> would you be kind enough to give me references on the answer?)
>

no.  anova() compares models where one model has a subset of the factors of
the other model. the way to do what you need is to create a third model with
X1 and X2 and all other factors contained in fit1 and fit2. then you compare
this super-model (let's call it "superfit") against the two sub-models
(fit1, fit2)

anova(superfit, fit1)
anova(superfit, fit2)

if the first comparison is significant that means that X2 [sic] improves the
model's data log-likelihood significantly (because X2 is the only factor
contained in superfit, but not fit1). If the second comparison is
significant, this means that X1 contributes significantly to the model. so,
there are four possible results (where by "matter" I mean that the factor
improves the model's data log-likelihood significantly):

   1. X1, but not X2 matters
   2. X2, but not X1 matters
   3. both X1 and X2 matters
   4. neither X1 and X2 improve fit2 and fit1, respectively

in the last case, you have to make sure that X1 or X2 contribute
significantly to fit1 and fit2, respectively (by comparing fit1 to a model
without X1 and comparing fit2 to a model without X2). If so, then X1 and X2
are significant, but too collinear to be distinguished between (using your
data set).

even though model comparisons based on data log-likelihood (anova(model1,
model2)) are considered relatively robust against problems with collinear
factors, collinearity between X1 and X2 could be a problem (generally, if X1
and X2 are too collinear in your data set, then you will have a hard time
distinguishing between them).

> 3. could I use the same formula to examine only one "Age" group (removing
> "Age" as a factor in the formula, of course), even if I am going to later
> proceed and re-examine the data for a larger young/old set?
>

I am not sure that I understand what you mean. Nothing prevents you from
exploring e.g. only data from "young" people, but be aware that whatever
conclusions you draw out of the examination of that subset of your data, may
not generalize to the entire sample (and hence not to the population
represented by the entire sample).

I hope this helps. Let me know, if something is unclear.

Florian

Any help is greatly appreciated.
>
> -shabnam
>
>
>
>
>
>
>
> _______________________________________________
> R-lang mailing list
> R-lang at ling.ucsd.edu
> https://ling.ucsd.edu/mailman/listinfo.cgi/r-lang
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://ling.ucsd.edu/pipermail/r-lang/attachments/20070508/01fee6f1/attachment.htm