[R-lang] Re: Self Paced Reading experiment using residuals with lmer

[T. Florian Jaeger]

Hi Bruno,

good question. Here's what I'd say:

If I wanted to analyze *all* word-by-word RTs in my data, I would definitely
do what Baayen et al did. However, in most SPR experiments, we are only
analyzing the *items*, but not the fillers. Often folks are even only
analyzing specific regions in the items. At that point then, I prefer to
derive per-word residual RTs *for* the region of interest in the items *from
* the entire data (to decorrelate properties of the region of interest that
I am interested in from properties that can be reduced to commonalities with
the remainder with the data). That cannot be achieved by the procedure
employed in Harald's article.

This also explains why I moved the position terms (which, btw, I model as a
non-linear term in order to be maximally conservative) into the
residualization (in addition to the standard terms for word length). This
procedure is especially useful when -in your *items*- condition is
confounded with position (of the word in the sentence or the sentence in the
list, although the latter is usually easy to avoid).

I know for a fact (unpleasant experience) that this type of confound (which
is rarely controlled in experiments with positional confounds) does matter
and that this only shows in full clarity in the procedure I proposed. I had
a nicely significant result, which went away once I applied this procedure
(which is why the procedure is described in a blog article rather than in a
journal ;)) and when I checked further I became very certain that the
analysis was doing the right thing (for the type of situation I am
describing here).

So, long story short: if you're analyzing all data, then use Baayen's
method. If you are analyzing only your items, I think there are reasons to
use the method I propose. For examples of the analysis I proposed, see
Hofmeister et al, submitted and Hofmeister in press (which also shows that
the analysis doesn't *always* kill effects ;).

*As for transformations*, several researchers have looked into this for
various tasks. Obviously, there is Smith and Levy's work, suggesting that
raw RTs should be used. There is work by Kliegl et al (2010) comparing raw,
log, and reciprocal transforms of RTs (though not for SPR), arguing -if I
recall correctly- for a reciprocal link.

Here's what I usually do (I think Baayen, Kuperman, etc. would do the same):
I look at the data (qqplots, shapiro test for normality over residuals of
model or -faster- over raw data by condition if you have a factorial design)
and compare the usual suspects (raw, log, reciprocal). For some examples,
you might find Victor Kuperman and my slides prepared for WOMM 2009 useful (
http://hlplab.wordpress.com/2009-pre-cuny-workshop-on-ordinary-and-multilevel-models-womm/,
there are some updated version of these slides at:
http://hlplab.wordpress.com/2010/05/10/mini-womm-montreal-slides-now-available/
).

When you do that, it becomes pretty obvious that the decision will in part
depend on whether you prefer to exclude outliers or not. log transforms are
useful when you do not remove outliers (as they make them less extreme
values, since most outliers in SPR experiments are large rather than small
values). I have to say that, in my experience, the transform did hardly ever
change the results (and if that happens and I know it then I try to
understand why).

HTH,
Florian

On Sun, Aug 29, 2010 at 4:22 AM, Bruno Nicenboim
<bruno.nicenboim@gmail.com>wrote:

> Hi,
> I'm analyzing the results of a SPR experiment.
>
> I saw that in Jaeger's blog (HLP/Jaeger lab blog) and in Jaeger, Fedorenko
> and
> Gibson's article "Anti-locality in English: Consequences for Theories of
> Sentence Comprehension" in order to analyze the results,  they use a linear
> model that takes as dependent variables the residuals of a model that looks
> roughly like this: (I didn't include the transformations they use)
>
> l <- lmer(RT ~  Wordlenght + positionofword + positionofstimulus +  (1 |
> SUBJ)...
>
> RTresidual <- residuals(l)
>
> (http://hlplab.wordpress.com/2008/01/23/modeling-self-paced-reading-data-
> effects-of-word-length-word-position-spill-over-etc/#more-46)
>
> Then, the final linear model looks like this:
>
> l <- lmer(RTresidual ~ CONDITION +
>            SPILLOVER_1 + SPILLOVER_2 + SPILLOVER_3 +
>            (1 | SUBJ) + (1 | ITEM)
>
> On the other hand, Baayen and Milim in "Analyzing Reaction Times" use a
> model
> that takes that takes as a dependent variable the RT (instead of
> residuals), and
> includes the word lenght and the position of the word and line in the same
> model, roughly like:
>
> l <- lmer(RT ~ CONDITION + Wordlenght + positionofword + positionofstimulus
> +
>            SPILLOVER_1 + SPILLOVER_2 + SPILLOVER_3 +
>            (1 | SUBJ) + (1 | ITEM)
>
>
> My questions are:
> Is there any advantage or disadvantage that should persuade me to use one
> approach or the other?
> Shouldn't I get similar results? (Because I don't)
> And finally, I've noticed that each researcher (not only in these two
> examples)
> uses different transformations on length, positions and reading times. Is
> there
> any way to check which transformation is the most appropriate?
>
> Thanks !
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.ucsd.edu/mailman/private/ling-r-lang-l/attachments/20101015/79739a8b/attachment.html