[R-lang] Re: p-values from pvals.fnc

[Jonathan Baron]

On 08/01/11 15:11, Levy, Roger wrote:
> Just a follow-up on this: Nathaniel is exactly right in saying that this clever 
> reparameterization doesn't put us in business for MCMC sampling within lme4, because 
> averaging over the uncertainty in the random-effects covariance matrix, which is the 
> whole point of the MCMC sampling in the first place, should also take into account 
> uncertainty in the correlation parameters.

I see the point of this argument, but I still feel a little uneasy
about giving up this idea of transforming the predictor so that the
correlation of slope and intercept is about zero.  I have a couple of
questions.

In practice, the situation is that we have a "significant" result
without including any random slopes.  We want to get an idea of
whether inclusion of random slopes would matter, that is, whether we
are getting an apparent overall effect just because some
subjects/items show a positive effect and some show a negative effect
and we happen to have sampled more of the former, by chance.  The
simplest way to do this is just to use a model with random slopes,
such as using (1+X|S) as a random effect term in lmer, where X is the
predictor of interest.  Often you can just look at the output and see
that the inclusion of the random slope doesn't make any difference.
If you want to report something, you could report the t value without
any df, and then assert that the result is real if the t value is
reasonably high (e.g., > 2) given the number of subjects, or if it is
no different from the t value for (1|S) as the random-effect term.

But it might be a little better to make an assumption that allows you
to use mcmcsamp, namely, that the slope and intercept random effects
are uncorrelated.  You can just do this without any transformation of
X, e.g, the random effects are now (1|S)+(0+X|S).  Often, when you
look at the output of the basic model, the correlation is low anyway.
But if you want to be a little more careful, you can transform X by a
constant, so that your assumption is closer to being true.  The
appropriate zero point for X is completely arbitrary anyway, in the
relevant cases, such as a numerical rating response (such as percent
confidence).

So one question is, Is this method (mcmcsamp with transformation) any
better than just looking at the t value?  And is it any better than
mcmcsamp without the transformation of X?  (Clearly it is not better
than using BUGS or JAGS, but that may be overkill for some of the
results of interest.)

The second question is why uncertainty about the correlation matters.
Note that we are interested here only in the slope, and the
correlation in question is that between the slope and intercept (as
random effects), as I understand it.  I cannot get my head around
this.  Maybe someone else can.  But my intuition is that the random
effects terms will be larger if we assume no correlation when the
correlation in fact exists than if we take the correlation into
account.  In the extreme case we might have a nearly perfect
correlation between slope and intercept, and if we assume no
correlation then we will find a substantial random effect for both,
which will increase the "error" estimate provided by mcmcsamp.  Thus,
failing to take the correlation into account may be conservative.
But I don't know what the effect is of failing to take the _error in
measuring the correlation_ into account, and that is the issue.


Jon
-- 
Jonathan Baron, Professor of Psychology, University of Pennsylvania
Home page: http://www.sas.upenn.edu/~baron