[R-lang] Re: p-values from pvals.fnc

[Levy, Roger]

Hi Jakke,

On Jul 30, 2011, at 1:30 AM, Jakke Tamminen wrote:

> Many thanks to David and Roger for helpful ideas to explore. Roger: could you please explain how to check whether the Markov chain has converged? 

Unfortunately, it's not a trivial task -- in the general case this is an open problem in statistics and computer science.  The good news is that it's generally not as hard for lme4 models as for the general case.  Here are a few suggestions:

1) if you inspect the density plots for the fixed-effects parameters for typical lme4 models, they should look roughly normal-ish.  If they look dramatically non-normal, then probably your chain hasn't run long enough.

2) My general take is that the best way is to run multiple Markov chains for the same model which are initialized with dramatically different parameter settings, and run them long enough so that they all look at each other.  Gelman and Rubin (1992) is the key reference for this, but I would recommend Section 11.6 of Gelman et al., 2004, or even more friendly, Chapter 16 of Gelman and Hill 2007.  

If you really want to take this issue on,

There's also a bit of information in Section 4.5 of my book draft at

  http://idiom.ucsd.edu/~rlevy/textbook/text.html

which may be useful in familiarizing you with what's actually going on when Markov Chain Monte Carlo inference is being used to compute "p-values".  

> Another thing I noticed that might provide a clue is that the strange behaviour of the p-values disappears if I remove the random slope for x. So
> 
> model1 = lmer(RT~x*y+(1+x|Subject)+(1|Item)
> 
> shows the problem while
> 
> model2 = lmer(RT~x*y+(1|Subject)+(1|Item)
> 
> does not. I wonder if that helps?

OK, now you have really confused me.  As far as I know, pvals.fnc() doesn't work on mer models like model1 with random slopes.  What version of R are you using?  Could you perhaps provide a working example?

Best

Roger

> 
> Jakke
> 
> 
> On 30 July 2011 07:08, Levy, Roger <rlevy@ucsd.edu> wrote:
> Hi Jakke,
> 
> It's a bit hard to give an answer to this question on the basis of anecdotal reports.  Do you have a specific dataset that gives you this behavior which you could share with the list?  That might be helpful in giving more pinpointed.
> 
> In general, one thing to check for when you find this kind of divergence, though, might be whether the Markov chain from which your "pMCMC" values are computed looks like it has converged.
> 
> Best
> 
> Roger
> 
> 
> On Jul 29, 2011, at 1:58 PM, Jakke Tamminen wrote:
> 
> > Dear R-users,
> >
> > I have been wondering about something with the pvals.fnc function. As we know, the pvals function gives two p-values, one based on the posterior distribution (pMCMC) and one based on the t-distribution. In my experience most of the time the two values are very similar. However, I have recently come across situations where they are wildly different. I have been particularly surprised to see t-values above 2 that have associated pMCMC values that are not even close to significance, while at the same time the t-distribution based p-value is significant. For example, a recent model I worked with looked something like this:
> >
> > model1 = lmer(RT~x*y+(1+x|Subject)+(1|Item)
> >
> > and gave me a t-value of 2.07 for the interaction, with a pMCMC p-value of 0.4756 and a t-distribution p-value of 0.0381. Obviously I like one of these better than the other! I know that the latter p-value is anticonservative, but the magnitude of the discrepancy is nonetheless surprising to me, given the t-value. I'd be very grateful for any advice on how to proceed in cases like this. I'm using lme4 version 0.99875-6.
> >
> > Many thanks,
> >
> > Jakke
> 
> --
> 
> Roger Levy                      Email: rlevy@ucsd.edu
> Assistant Professor             Phone: 858-534-7219
> Department of Linguistics       Fax:   858-534-4789
> UC San Diego                    Web:   http://idiom.ucsd.edu/~rlevy
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

--

Roger Levy                      Email: rlevy@ucsd.edu
Assistant Professor             Phone: 858-534-7219
Department of Linguistics       Fax:   858-534-4789
UC San Diego                    Web:   http://idiom.ucsd.edu/~rlevy