[R-lang] Re: p-values from pvals.fnc

[Levy, Roger]

Well, I typically do rely on the t-statistics for models with random slopes, but of course you need to exercise the usual caution that they are not truly t-distributed (though they are generally quite close to normal if you have lots more observations than parameters).

And I do recommend updating your packages!

Best

Roger


On Jul 30, 2011, at 1:53 PM, Jakke Tamminen wrote:

> Roger: Thanks for the information, I guess I have a lot of reading to do! 
> 
> Alex and Roger: Looks like my version of lme4 is pretty old, 0.99875-6. If the more recent versions don't give you p-values for models with random slopes, should I be looking at them (the p-values) at all, or rely on the t-statistic (and probably update my packages!)?
> 
> Jakke 
> 
> On 30 July 2011 18:45, Alex Fine <afine@bcs.rochester.edu> wrote:
> Jakke,
> 
> I'm probably missing something, so I'm not replying-all.  How do you even get pvals.fnc() to work with a model that has random slopes?  I have the most up-to-date version of the languageR package and it won't take models that have anything other than random intercepts.
> 
> thanks,
> Alex
> 
> Jakke Tamminen wrote:
> Many thanks to David and Roger for helpful ideas to explore. Roger: could you please explain how to check whether the Markov chain has converged? 
> Another thing I noticed that might provide a clue is that the strange behaviour of the p-values disappears if I remove the random slope for x. So
> 
> model1 = lmer(RT~x*y+(1+x|Subject)+(1|Item)
> 
> shows the problem while
> 
> model2 = lmer(RT~x*y+(1|Subject)+(1|Item)
> 
> does not. I wonder if that helps?
> 
> Jakke
> 
> 
> On 30 July 2011 07:08, Levy, Roger <rlevy@ucsd.edu <mailto:rlevy@ucsd.edu>> wrote:
> 
>    Hi Jakke,
> 
>    It's a bit hard to give an answer to this question on the basis of
>    anecdotal reports.  Do you have a specific dataset that gives you
>    this behavior which you could share with the list?  That might be
>    helpful in giving more pinpointed.
> 
>    In general, one thing to check for when you find this kind of
>    divergence, though, might be whether the Markov chain from which
>    your "pMCMC" values are computed looks like it has converged.
> 
>    Best
> 
>    Roger
> 
> 
>    On Jul 29, 2011, at 1:58 PM, Jakke Tamminen wrote:
> 
>    > Dear R-users,
>    >
>    > I have been wondering about something with the pvals.fnc
>    function. As we know, the pvals function gives two p-values, one
>    based on the posterior distribution (pMCMC) and one based on the
>    t-distribution. In my experience most of the time the two values
>    are very similar. However, I have recently come across situations
>    where they are wildly different. I have been particularly
>    surprised to see t-values above 2 that have associated pMCMC
>    values that are not even close to significance, while at the same
>    time the t-distribution based p-value is significant. For example,
>    a recent model I worked with looked something like this:
>    >
>    > model1 = lmer(RT~x*y+(1+x|Subject)+(1|Item)
>    >
>    > and gave me a t-value of 2.07 for the interaction, with a pMCMC
>    p-value of 0.4756 and a t-distribution p-value of 0.0381.
>    Obviously I like one of these better than the other! I know that
>    the latter p-value is anticonservative, but the magnitude of the
>    discrepancy is nonetheless surprising to me, given the t-value.
>    I'd be very grateful for any advice on how to proceed in cases
>    like this. I'm using lme4 version 0.99875-6.
>    >
>    > Many thanks,
>    >
>    > Jakke
> 
>    --
> 
>    Roger Levy                      Email: rlevy@ucsd.edu
>    <mailto:rlevy@ucsd.edu>
> 
>    Assistant Professor             Phone: 858-534-7219
>    Department of Linguistics       Fax:   858-534-4789
>    UC San Diego                    Web:      http://idiom.ucsd.edu/~rlevy <http://idiom.ucsd.edu/%7Erlevy>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

--

Roger Levy                      Email: rlevy@ucsd.edu
Assistant Professor             Phone: 858-534-7219
Department of Linguistics       Fax:   858-534-4789
UC San Diego                    Web:   http://idiom.ucsd.edu/~rlevy