[R-lang] Re: Question about power

[Ariel M. Goldberg]

Hi John,

Thank you for your response.  I should have mentioned that I'm using mixed-effects models (lmer) for which I don't believe validate() has been implemented.  Your suggestion of ~200 simulations seems good. I think I will simply resample the data and run the simulations; something similar appears to have been suggested here as a way to perform validation:

https://mailman.ucsd.edu/pipermail/ling-r-lang-l/2010-April/000012.html

Cheers,
Ari

On Aug 3, 2011, at 6:25 PM, <jkingston@linguist.umass.edu> wrote:

> Dear Ariel,
> The bootstrapping procedure -- validate() -- described in Baayen's 2008 book (6.2.4, pp. 193ff) typically consists of just 200 iterations on samples drawn with replacement from the original data set. This seems to be enough to determine the robustness of the model fit. You can specify the procedure for eliminating variables as well as the number of iterations.
> Best,
> John
> 
> Quoting "Ariel M. Goldberg" <ariel.goldberg@tufts.edu>:
> 
>> Hi,
>> 
>> I have a question about making inferences when power might be an issue.  I'm examining whether a variable has a significant effect in different parts of the syllable.  To do this, I have 2 different data sets, Onset and Coda, which I'm using to determine if the variable has effects in the syllable onset and coda, respectively.  The variable is significant (very small p-value) in the onset but is marginally significant in the coda (p= .055 in the full model, and model comparison with a baseline model that does not contain this variable gives a p-value of .07).
>> 
>> While it's always difficult to know how to interpret a marginally significant effect, one issue that complicates the matter is that the Coda dataset has fewer items and trials than the Onset dataset.  One thing that I'd like to do is determine whether the marginal effect could simply be due to a lack of power.  My idea was to take a random sample of the Onset dataset so that it matches the size of the coda dataset and see if the variable of interest remains significant even in this reduced dataset.  I figure that I would need to do this sampling many times (e.g., 10,000 times) to make sure that the effect is robust.
>> 
>> Is this a sensible approach?  Am I going to run into a Type I/II error situation by doing 10,000 model comparisons?
>> 
>> Thank you,
>> Ariel
>> 
> 
> 
> 
> John Kingston
> --------------------------------
> Linguistics Department
> University of Massachusetts
> 150 Hicks Way, 226 South College
> Amherst, MA 01003-9274
> 1-413-545-6833, fax -2792
> jkingston@linguist.umass.edu
> www.people.umass.edu/jkingstn

--
Ariel M. Goldberg
Assistant Professor
Tufts University
Department of Psychology
490 Boston Avenue
Medford, MA 02155

(617) 627-3525
http://ase.tufts.edu/psychology/psycholinglab/