<br><br><div><span class="gmail_quote">On 10/10/07, <b class="gmail_sendername">David Reitter</b> <<a href="mailto:dreitter@inf.ed.ac.uk">dreitter@inf.ed.ac.uk</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
What are the options to compare fits of logistic regression models?<br><br>My models are fitted using 'glmmPQL' from the MASS and nlme<br>libraries. The models to be compared differ in their use of<br>covariates,
e.g.,<br><br>model.1 <- glmmPQL(dep ~ pred1a * pred2, random=1|subj, family=binomial)<br>model.2 <- glmmPQL(dep ~ pred1b * pred2, random=1|subj, family=binomial)<br><br>pred1a and pred1b are correlated, and I'd like to estimate their
<br>relative predictive power.</blockquote><div><br>predictive power? you can, of course, use predict() [you have to write your own version if you want to use the random effects]. and then use cross-validation to get a measure for unseen data.
<br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Both models use the same datasets, but the covariate structure is not<br>nested.
</blockquote><div><br>you should create a superset model containing all factors from the subset models and then compare the subset models against the superset model. that's the case independent of whether the model is fit maximizing ML or quasi-likelihoods (this methods makes sure you compare nested models).
<br><br>I think some people also use the DIC (deviance information criterion) to compare non-nested models. Shravan Vasishth was talking about this. Maybe you can ask him? I have yet to be convinced that this is ok.<br></div>
<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">(Adjusted) R^2 are not available, which I think has to do with the<br>PQL fitting algorithm (it is not a maximum likelihood fit - do I
<br>remember correctly?).</blockquote><div><br>it's penalized quasi-likelihood, even though now you can also use Laplace approximated quasi-likelihood. in either case, an analytic approximation of the ML function is optimized since no analytic solution to the ML function is known. (the other alternative is Monte Carlo simulations, but they become unfeasible with as little as 3 parameters according to Agresti02).
<br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">AIC doesn't seem to be useful either ([1]). AIC, BIC and Log-<br>Likelihood are only output as "NA" using the "summary" function. If
<br>there is no valid Log-Likelihood available for a PQL model, we can't<br>compute Nagelkerke's (Pseudo) R^2, can we?</blockquote><div><br>the quasi-likelihood is an approximation of the (penalized) ML. as such, it can lead to weird comparisons (a smaller model may turn out to be more "quasi-likely" than a larger model).
<br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">The only values I am getting for my model are adjusted R^2 for "lm"<br>
fits. But because I'm using repeated-measures data (sampled from<br>dialogue corpora), fits here seem to violate not only distribution<br>assumptions of the response variable, but also independence (IID)<br>of the data points.
</blockquote><div><br>You can at least use quasi-likelihood comparisons to argue your case (using the same logic as for ML comparisons). have a look at the manuscript on mixed logit models on my website. it's not 100% correct, but with Laplace approximations I get reasonable results.
<br><br>There is a correct way of doing it, but it's complicated. I think it maybe in Quene & van den Bergh_2003, but I am not sure.<br><br>hope some of this helps.<br><br>florian<br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Any comments would be appreciated - be it to clear up some<br>misconceptions on my part, or be it to solve my problem at hand.<br><br><br><br>[1] <a href="http://tolstoy.newcastle.edu.au/R/e2/help/07/06/18477.html">http://tolstoy.newcastle.edu.au/R/e2/help/07/06/18477.html
</a><br><br><br>--<br>David Reitter<br>ICCS/HCRC, Informatics, University of Edinburgh<br><a href="http://www.david-reitter.com">http://www.david-reitter.com</a><br><br><br><br><br><br>_______________________________________________
<br>R-lang mailing list<br><a href="mailto:R-lang@ling.ucsd.edu">R-lang@ling.ucsd.edu</a><br><a href="http://pidgin.ucsd.edu/mailman/listinfo/r-lang">http://pidgin.ucsd.edu/mailman/listinfo/r-lang</a><br></blockquote></div>
<br>