[R-lang] Re: Determining whether a model fits a second data set

[T. Florian Jaeger]

Hi Amanda,

a couple of things come to mind that might be useful. Sorry, if you've
considered some of them and they didn't work.

First of all, most model functions in R allow you to use your model to make
predictions (usually that function is simply called predict()).

There are also nice functions in the arm() package by Gelman, which allows
you to run simulation using your model (this might be useful in response to
some of the responses to your question). So you can derive 'confidence
intervals' over your parameters based on novel data. (see sim() in the arm()
package).

The biggest problem with applying mixed models to unseen data is that you
are likely to encounter unseen levels of your random variables in the new
data (e.g. new speakers). theoretically it should be possible to estimate
the BLUBs for the new data based on the fixed effect parameters and random
variances fit on the old data (BLUPs are not parameters of the model, but
estimated based on the data, the fixed effect parameters and the random
variances). But i am not sure whether there already is a function
implemented for that.

As for measures of quality comparable to an R2, you could consider doing a
Nagelkerke R2, which is theoretically defined for mixed models, too. See
Jaeger, 2010:37, where I mention this measure for mixed models and provide a
link to some R code that implements it. Here's an attempt to paste that
section.

===================== paste ============================
While several pseudo R2 measures have also been proposed for logit models as
measures of overall model quality, they lack the intuitive and reliable
interpretation of the R2 available for linear models. One of the most
commonly used pseudo R2 measures is the Nagelkerke R2. The Nagelkerke R2
assesses the quality of a model with regard to a baseline model (the model
with only the intercept). While this measure is usually employed for
ordinary logit models, its definition extends to multilevel logit
models. For the current model, Nagelkerke R2 =0:34 compared against an
ordinary logit model with only
the intercept as baseline.\footnote{Despite my own skepticism about
Nagelkerke R2s, I have chosen to present them since they may be familiar to
some readers.
All Nagelkerke R2s are calculated against an ordinary logit model with only
the intercept as baseline. The R code used to calculate
the Nagelkerke R2s presented here is available at
http://hlplab.wordpress.com/2009/08/29/nagelkerke-and-coxsnellpseudo-
r2-for-mixed-logit-models/.}
===================== end paste ============================

Many other comparable measures can be extended to mixed models - none as
conceptually appealing as an R-square. The blog page linked in my paper
contains links to more information about these other measures.

HTH,
Florian

On Wed, Oct 13, 2010 at 5:59 AM, Amanda J. Owen <ajowen@gmail.com> wrote:

> Hi,
>   I think this is a more general statistics question rather than a
> particular question about the use of lmer, but I would appreciate the
> advice from this group since the question is specific to the types of
> models fit under lmer/LanguageR.
>
> I have fit a (rather complex) model to a dataset based on elicited
> data. The model is a mixed model logistic regression with 2 random
> factors and 6 predictors.  I have both a 'real' model using true
> values for the predictors and a standardized model which has
> standardized all predictors so that their relative contributions to
> predicting the outcome (past tense accuracy) can be compared more
> directly.  Some factors are significant but make a relatively small
> contribution.  The data set is fairly rich with around 8000 datapoints
> (216 past tense opportunities per child).
>
>    I have a similar but smaller data set from spontaneous data from
> the same children and a third data set from spontaneous data from a
> new group of children that is even smaller.  I can quantify the
> predictor variables (lexical frequency, phonotactics, etc) in such a
> way that I should be able to fit the same model to this second
> dataset.  I'm especially interested in whether we could actually use
> the same beta values to have a reasonably good fit because I'd like to
> be able to comment on whether the relative contributions of the
> predictors was an artifact of the data set, the particular children
> involved,  or are true more generally.
>
>   I'm not sure how to assess model fit without simply comparing a
> reduced/full model on the same dataset.  I think if I was using linear
> regression I would compare R-squared values across the two datasets
> and have some comment about the fact that the models explain similar
> amounts of variance.  But I'm stumped when it comes to logistic
> regression/the introduction of random effects.
>
> Thanks so much for any advice or help you can provide.
> Amanda
>
>
> Amanda J. Owen
> ajowen@gmail.com
>
>
> amanda-owen@uiowa.edu
> Dept of Speech Pathology & Audiology
> University of Iowa
> Iowa City, IA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.ucsd.edu/mailman/private/ling-r-lang-l/attachments/20101013/5bcdf1ee/attachment.html