From Laura.deRuiter at mpi.nl Mon Dec 22 01:57:21 2008 From: Laura.deRuiter at mpi.nl (Laura de Ruiter) Date: Mon, 22 Dec 2008 10:57:21 +0100 Subject: [R-lang] Weird outcome logit mixed-effect model Message-ID: <494F6481.1070806@mpi.nl> Dear R-users and -experts, I am performing a rather simple analysis on a small data set (pasted below this email) and keep getting a to me inexplicable result. Perhaps I am missing something here - it would be great if someone could point out to me what I am doing wrong. I want to test whether the factor "Info" (which has three levels: "new", "given", "accessible") is a significant predictor for the binary variable "DeaccYN". The random factor is "Subject". The distribution of the data looks as follows: ----------------------------------------------------------------------------- xtabs (~ DeaccYN + Info, data = dat) Info DeaccYN given new accessible no 25 42 21 yes 11 0 1 ------------------------------------------------------------------------------ This is the model: ---------------------------------------------------------------------------------------------------------- deacc.lmer = lmer (DeaccYN ~ Info + (1|Subject), data = dat, family = "binomial") ----------------------------------------------------------------------------------------------------------------- However, given the distribution above, this outcome seems rather weird to me: --------------------------------------------------------------------------------------------------------- summary (deacc.lmer) Generalized linear mixed model fit using Laplace Formula: DeaccYN ~ Info + (1 | Subject) Data: dat Family: binomial(logit link) AIC BIC logLik deviance 60.4 70.82 -26.2 52.4 Random effects: Groups Name Variance Std.Dev. Subject (Intercept) 0.18797 0.43356 number of obs: 100, groups: Subject, 21 Estimated scale (compare to 1 ) 0.7316067 Fixed effects: Estimate Std. Error z value Pr(>|z|) (Intercept) -0.8635 0.3795 -2.2754 0.0229 * Infonew -18.7451 2764.2445 -0.0068 0.9946 Infoaccessible -2.2496 1.1186 -2.0110 0.0443 * --- Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1 Correlation of Fixed Effects: (Intr) Infonw Infonew 0.000 Infoaccssbl -0.315 0.000 ---------------------------------------------------------------------------------------------------- Why should the difference between 25/11 and 21/1 be significant, but the difference between 25/11 and 42/0 not? Very odd to me seems the standard error of 2764! First I was wondering whether the problem had sth. to do with the fact that I had read in an SPSS file, but I keep getting this result even when I first save the SPSS file as a tabdelim. text and read it into R then. I couldn't find anything strange going on with the Subject factor either (and I have also tried the model with "Item" as random factor, which didn't change anything). I was wondering: Is it perhaps a problem for the model that there are no cases in the DeaccYN == "yes" category for Info == "given"? And if this is the case, why? Am I overlooking something here? Any help with this would be very much appreciated! Thanks a lot in advance for your answers & happy holidays, Laura de Ruiter --------------------------------------- The data set: dat Subject Info Deacc 1 93 new no 2 93 new no 3 93 given yes 4 93 new no 5 93 given no 6 94 new no 7 94 new no 8 94 given no 9 94 new no 10 94 given no 11 94 given no 12 94 accessible no 13 95 given no 14 95 new no 15 95 accessible no 16 95 new no 17 95 accessible no 18 95 new no 19 97 new no 20 98 new no 21 98 new no 22 98 given yes 23 98 given no 24 101 new no 25 101 new no 26 101 given yes 27 101 accessible no 28 101 new no 29 101 given yes 30 101 given no 31 101 accessible no 32 102 new no 33 102 new no 34 102 given no 35 102 given no 36 102 accessible no 37 104 new no 38 104 accessible no 39 104 new no 40 104 given yes 41 104 new no 42 104 given no 43 104 accessible no 44 107 new no 45 107 given no 46 107 accessible no 47 107 new no 48 109 new no 49 109 given yes 50 109 new no 51 110 accessible no 52 110 new no 53 110 new no 54 110 given no 55 113 new no 56 113 given yes 57 113 new no 58 113 given yes 59 114 new no 60 114 accessible no 61 114 new no 62 114 given no 63 114 accessible no 64 114 new no 65 114 given yes 66 114 accessible yes 67 115 accessible no 68 115 given no 69 115 given no 70 116 accessible no 71 116 new no 72 116 given no 73 116 new no 74 116 new no 75 116 accessible no 76 117 accessible no 77 117 given no 78 117 new no 79 117 new no 80 117 given no 81 117 accessible no 82 118 new no 83 118 accessible no 84 118 given no 85 118 new no 86 118 given no 87 118 accessible no 88 119 given no 89 119 given yes 90 119 given no 91 119 accessible no 92 121 given no 93 121 given no 94 131 new no 95 131 new no 96 131 new no 97 131 new no 98 132 given yes 99 132 given no 100 132 accessible no ------------------------------------------------ -- ////////////////////////////////////////////////////// Laura de Ruiter PhD student Language Acquisition Group Max Planck Institute for Psycholinguistics P.O. Box 310 6500 AH Nijmegen Netherlands Phone: 0031-24-3521 456 Homepage: http://www.mpi.nl/Members/LauradeRuiter /////////////////////////////////////////////////// From tiflo at csli.stanford.edu Mon Dec 22 12:59:04 2008 From: tiflo at csli.stanford.edu (T. Florian Jaeger) Date: Mon, 22 Dec 2008 15:59:04 -0500 Subject: [R-lang] Weird outcome logit mixed-effect model In-Reply-To: <494F6481.1070806@mpi.nl> References: <494F6481.1070806@mpi.nl> Message-ID: <38dc9be90812221259w3fdbee1ay19a80d25e6991b2d@mail.gmail.com> Dear Laura, this model is most certainly overfit. for a binomial model the rule of thumb for the absolute upper limit for the number of paramters in the model is min(outcome A, outcome B) / 10, which in your case is 12/ 10 ~ 1. you're using 2 parameters for fixed effects (and 1 for a random intercept). Most of your subjects will have 0 deaccented syllables, making it hard to impossible for the model to estimate the effects of info. I am still a bit puzzled by the output though. did the fitting process not through a warning (I am leaving town tomorrow, so i did not have time to load your data and see for myself). Florian On Mon, Dec 22, 2008 at 4:57 AM, Laura de Ruiter wrote: > Dear R-users and -experts, > > I am performing a rather simple analysis on a small data set (pasted below > this email) and keep getting a to me inexplicable result. Perhaps I am > missing something here - it would be great if someone could point out to me > what I am doing wrong. > > I want to test whether the factor "Info" (which has three levels: "new", > "given", "accessible") is a significant predictor for the binary variable > "DeaccYN". The random factor is "Subject". The distribution of the data > looks as follows: > > > ----------------------------------------------------------------------------- > xtabs (~ DeaccYN + Info, data = dat) > Info > DeaccYN given new accessible > no 25 42 21 > yes 11 0 1 > > ------------------------------------------------------------------------------ > > This is the model: > > > ---------------------------------------------------------------------------------------------------------- > deacc.lmer = lmer (DeaccYN ~ Info + (1|Subject), data = dat, family = > "binomial") > > ----------------------------------------------------------------------------------------------------------------- > > However, given the distribution above, this outcome seems rather weird to > me: > > > --------------------------------------------------------------------------------------------------------- > summary (deacc.lmer) > Generalized linear mixed model fit using Laplace > Formula: DeaccYN ~ Info + (1 | Subject) > Data: dat > Family: binomial(logit link) > AIC BIC logLik deviance > 60.4 70.82 -26.2 52.4 > Random effects: > Groups Name Variance Std.Dev. > Subject (Intercept) 0.18797 0.43356 > number of obs: 100, groups: Subject, 21 > > Estimated scale (compare to 1 ) 0.7316067 > > Fixed effects: > Estimate Std. Error z value Pr(>|z|) > (Intercept) -0.8635 0.3795 -2.2754 0.0229 * > Infonew -18.7451 2764.2445 -0.0068 0.9946 > Infoaccessible -2.2496 1.1186 -2.0110 0.0443 * > --- > Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 > > Correlation of Fixed Effects: > (Intr) Infonw > Infonew 0.000 > Infoaccssbl -0.315 0.000 > > ---------------------------------------------------------------------------------------------------- > > Why should the difference between 25/11 and 21/1 be significant, but the > difference between 25/11 and 42/0 not? Very odd to me seems the standard > error of 2764! > First I was wondering whether the problem had sth. to do with the fact that > I had read in an SPSS file, but I keep getting this result even when I first > save the SPSS file as a tabdelim. text and read it into R then. I couldn't > find anything strange going on with the Subject factor either (and I have > also tried the model with "Item" as random factor, which didn't change > anything). > > I was wondering: Is it perhaps a problem for the model that there are no > cases in the DeaccYN == "yes" category for Info == "given"? And if this is > the case, why? > Am I overlooking something here? > > Any help with this would be very much appreciated! > > Thanks a lot in advance for your answers & happy holidays, > > Laura de Ruiter > > --------------------------------------- > The data set: > > dat > Subject Info Deacc > 1 93 new no > 2 93 new no > 3 93 given yes > 4 93 new no > 5 93 given no > 6 94 new no > 7 94 new no > 8 94 given no > 9 94 new no > 10 94 given no > 11 94 given no > 12 94 accessible no > 13 95 given no > 14 95 new no > 15 95 accessible no > 16 95 new no > 17 95 accessible no > 18 95 new no > 19 97 new no > 20 98 new no > 21 98 new no > 22 98 given yes > 23 98 given no > 24 101 new no > 25 101 new no > 26 101 given yes > 27 101 accessible no > 28 101 new no > 29 101 given yes > 30 101 given no > 31 101 accessible no > 32 102 new no > 33 102 new no > 34 102 given no > 35 102 given no > 36 102 accessible no > 37 104 new no > 38 104 accessible no > 39 104 new no > 40 104 given yes > 41 104 new no > 42 104 given no > 43 104 accessible no > 44 107 new no > 45 107 given no > 46 107 accessible no > 47 107 new no > 48 109 new no > 49 109 given yes > 50 109 new no > 51 110 accessible no > 52 110 new no > 53 110 new no > 54 110 given no > 55 113 new no > 56 113 given yes > 57 113 new no > 58 113 given yes > 59 114 new no > 60 114 accessible no > 61 114 new no > 62 114 given no > 63 114 accessible no > 64 114 new no > 65 114 given yes > 66 114 accessible yes > 67 115 accessible no > 68 115 given no > 69 115 given no > 70 116 accessible no > 71 116 new no > 72 116 given no > 73 116 new no > 74 116 new no > 75 116 accessible no > 76 117 accessible no > 77 117 given no > 78 117 new no > 79 117 new no > 80 117 given no > 81 117 accessible no > 82 118 new no > 83 118 accessible no > 84 118 given no > 85 118 new no > 86 118 given no > 87 118 accessible no > 88 119 given no > 89 119 given yes > 90 119 given no > 91 119 accessible no > 92 121 given no > 93 121 given no > 94 131 new no > 95 131 new no > 96 131 new no > 97 131 new no > 98 132 given yes > 99 132 given no > 100 132 accessible no > ------------------------------------------------ > > -- > ////////////////////////////////////////////////////// > > Laura de Ruiter > PhD student > Language Acquisition Group > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > Netherlands > > Phone: 0031-24-3521 456 > Homepage: http://www.mpi.nl/Members/LauradeRuiter > > /////////////////////////////////////////////////// > > > _______________________________________________ > R-lang mailing list > R-lang at ling.ucsd.edu > http://pidgin.ucsd.edu/mailman/listinfo/r-lang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlevy at ucsd.edu Mon Dec 22 13:30:34 2008 From: rlevy at ucsd.edu (Roger Levy) Date: Mon, 22 Dec 2008 13:30:34 -0800 Subject: [R-lang] Weird outcome logit mixed-effect model In-Reply-To: <494F6481.1070806@mpi.nl> References: <494F6481.1070806@mpi.nl> Message-ID: <495006FA.2030803@ucsd.edu> Laura de Ruiter wrote: > Dear R-users and -experts, > > I am performing a rather simple analysis on a small data set (pasted > below this email) and keep getting a to me inexplicable result. Perhaps > I am missing something here - it would be great if someone could point > out to me what I am doing wrong. > > I want to test whether the factor "Info" (which has three levels: "new", > "given", "accessible") is a significant predictor for the binary > variable "DeaccYN". The random factor is "Subject". The distribution of > the data looks as follows: > > ----------------------------------------------------------------------------- > > > Info > DeaccYN given new accessible > no 25 42 21 > yes 11 0 1 > ------------------------------------------------------------------------------ > > > This is the model: > > ---------------------------------------------------------------------------------------------------------- > > deacc.lmer = lmer (DeaccYN ~ Info + (1|Subject), data = dat, family = > "binomial") > ----------------------------------------------------------------------------------------------------------------- > > > However, given the distribution above, this outcome seems rather weird > to me: > > --------------------------------------------------------------------------------------------------------- > > summary (deacc.lmer) > Generalized linear mixed model fit using Laplace > Formula: DeaccYN ~ Info + (1 | Subject) > Data: dat > Family: binomial(logit link) > AIC BIC logLik deviance > 60.4 70.82 -26.2 52.4 > Random effects: > Groups Name Variance Std.Dev. > Subject (Intercept) 0.18797 0.43356 > number of obs: 100, groups: Subject, 21 > > Estimated scale (compare to 1 ) 0.7316067 > > Fixed effects: > Estimate Std. Error z value Pr(>|z|) > (Intercept) -0.8635 0.3795 -2.2754 0.0229 * > Infonew -18.7451 2764.2445 -0.0068 0.9946 > Infoaccessible -2.2496 1.1186 -2.0110 0.0443 * > --- > Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1 > > [...] > > ---------------------------------------------------------------------------------------------------- > > > Why should the difference between 25/11 and 21/1 be significant, but the > difference between 25/11 and 42/0 not? Very odd to me seems the standard > error of 2764! > > [...] > I was wondering: Is it perhaps a problem for the model that there are no > cases in the DeaccYN == "yes" category for Info == "given"? And if this ^^^^^ I believe you mean "new" here. > is the case, why? > Am I overlooking something here? Dear Laura, Independently of the issue that Florian is raising...you are right that the lack of is a problem for the model (to be precise, it's a problem for estimating the significance of the parameter estimate using the z value). The z value, which is the basis of the significance level reported in the lmer summary, is based on the Wald statistic, which is z = \hat{b} / StdError(\hat{b}) where \hat{b} is the parameter estimate. However, for parameter estimates of large magnitudes (as is the case for "new" here), the standard error is inflated, which leads to a small wald statistic. This problem is discussed in a number of places, including Agresti (1996, 2002); here's a decent mention of it online: http://userwww.sfsu.edu/~efc/classes/biol710/logistic/logisticreg.htm You can see this quite clearly if you add just one "yes/given" example at the end of your data frame, for example: 101 132 new yes Then the model gives (being sure to correctly specify the levels of "Info"): > summary(deacc.lmer) Generalized linear mixed model fit by the Laplace approximation Formula: Deacc ~ Info + (1 | Subject) Data: dat AIC BIC logLik deviance 69.78 80.24 -30.89 61.78 Random effects: Groups Name Variance Std.Dev. Subject (Intercept) 0.34332 0.58594 Number of obs: 101, groups: Subject, 21 Fixed effects: Estimate Std. Error z value Pr(>|z|) (Intercept) -0.8996 0.3936 -2.286 0.02226 * Iaccessible -2.2808 1.1460 -1.990 0.04656 * Inew -3.0050 1.1395 -2.637 0.00836 ** The next question, of course, is how to deal with the dataset you actually have. If you weren't using a mixed-effects model, you could use a likelihood-ratio test by comparing your full model with a simpler model; the likelihood-ratio test isn't susceptible to problems with large parameter estimates the way the Wald test is. For example: > deacc.glm = glm (Deacc ~ Info, data = dat, family = "binomial") > deacc.glm1 = glm (Deacc ~ Info1, data = dat, family = "binomial") > anova(deacc.glm,deacc.glm1,test="Chisq") Analysis of Deviance Table Model 1: Deacc ~ Info Model 2: Deacc ~ Info1 Resid. Df Resid. Dev Df Deviance P(>|Chi|) 1 97 52.452 2 98 71.600 -1 -19.148 1.210e-05 In principle, you could do this with your mixed-effects model, being sure to use ML instead of REML fitting: > deacc.lmer = lmer (Deacc ~ Info + (1 | Subject), data = dat, family = "binomial",REML=F) > deacc.lmer1 = lmer (Deacc ~ Info1 + (1 | Subject), data = dat, family = "binomial",REML=F) > anova(deacc.lmer,deacc.lmer1) Data: dat Models: deacc.lmer1: Deacc ~ Info1 + (1 | Subject) deacc.lmer: Deacc ~ Info + (1 | Subject) Df AIC BIC logLik Chisq Chi Df Pr(>Chisq) deacc.lmer1 3 77.600 85.416 -35.800 deacc.lmer 4 60.400 70.821 -26.200 19.2 1 1.177e-05 *** There is an argument that the likelihood-ratio test is anti-conservative and hence inappropriate for comparing mixed-effects models differing only in fixed-effects structure. (See Pinheiro & Bates, 2000, around page 76 or so. The argument is made only for linear mixed-effects models and I'm not sure of the status for logit mixed-effects models.) That being said, it's not clear how strong the anti-conservativity is, and in your case it seems like you have such an exceedingly powerful effect that you might be safe in using the likelihood-ratio test here and just mentioning the potential anti-conservativity as a caveat. So the summary is: believe it or not, how to assess the significance of the parameter estimate for "new" for your model & dataset is a bit of an open question, but it seems pretty clear that the estimate is significantly non-zero. Best & hope this helps. Roger -- Roger Levy Email: rlevy at ucsd.edu Assistant Professor Phone: 858-534-7219 Department of Linguistics Fax: 858-534-4789 UC San Diego Web: http://ling.ucsd.edu/~rlevy From rlevy at ucsd.edu Mon Dec 22 13:33:10 2008 From: rlevy at ucsd.edu (Roger Levy) Date: Mon, 22 Dec 2008 13:33:10 -0800 Subject: [R-lang] Weird outcome logit mixed-effect model In-Reply-To: <495006FA.2030803@ucsd.edu> References: <494F6481.1070806@mpi.nl> <495006FA.2030803@ucsd.edu> Message-ID: <49500796.5000505@ucsd.edu> Roger Levy wrote: > Laura de Ruiter wrote: >> Dear R-users and -experts, >> >> I am performing a rather simple analysis on a small data set (pasted >> below this email) and keep getting a to me inexplicable result. >> Perhaps I am missing something here - it would be great if someone >> could point out to me what I am doing wrong. >> >> I want to test whether the factor "Info" (which has three levels: >> "new", "given", "accessible") is a significant predictor for the >> binary variable "DeaccYN". The random factor is "Subject". The >> distribution of the data looks as follows: >> >> ----------------------------------------------------------------------------- >> >> >> Info >> DeaccYN given new accessible >> no 25 42 21 >> yes 11 0 1 >> ------------------------------------------------------------------------------ >> >> >> This is the model: >> >> ---------------------------------------------------------------------------------------------------------- >> >> deacc.lmer = lmer (DeaccYN ~ Info + (1|Subject), data = dat, family = >> "binomial") >> ----------------------------------------------------------------------------------------------------------------- >> >> >> However, given the distribution above, this outcome seems rather weird >> to me: >> >> --------------------------------------------------------------------------------------------------------- >> >> summary (deacc.lmer) >> Generalized linear mixed model fit using Laplace >> Formula: DeaccYN ~ Info + (1 | Subject) >> Data: dat >> Family: binomial(logit link) >> AIC BIC logLik deviance >> 60.4 70.82 -26.2 52.4 >> Random effects: >> Groups Name Variance Std.Dev. >> Subject (Intercept) 0.18797 0.43356 >> number of obs: 100, groups: Subject, 21 >> >> Estimated scale (compare to 1 ) 0.7316067 >> >> Fixed effects: >> Estimate Std. Error z value Pr(>|z|) >> (Intercept) -0.8635 0.3795 -2.2754 0.0229 * >> Infonew -18.7451 2764.2445 -0.0068 0.9946 >> Infoaccessible -2.2496 1.1186 -2.0110 0.0443 * >> --- >> Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1 >> > > [...] >> >> ---------------------------------------------------------------------------------------------------- >> >> >> Why should the difference between 25/11 and 21/1 be significant, but >> the difference between 25/11 and 42/0 not? Very odd to me seems the >> standard error of 2764! >> > > [...] > >> I was wondering: Is it perhaps a problem for the model that there are >> no cases in the DeaccYN == "yes" category for Info == "given"? And if >> this > ^^^^^ > I believe you mean "new" here. >> is the case, why? >> Am I overlooking something here? > > Dear Laura, > > Independently of the issue that Florian is raising...you are right that > the lack of is a problem for the model (to be precise, it's a problem > for estimating the significance of the parameter estimate using the z > value). Whoops, this should have read "you are right that the lack of observations in the "yes/new" category is a problem for the model..." -- Roger Levy Email: rlevy at ucsd.edu Assistant Professor Phone: 858-534-7219 Department of Linguistics Fax: 858-534-4789 UC San Diego Web: http://ling.ucsd.edu/~rlevy From tiflo at csli.stanford.edu Mon Dec 22 14:12:53 2008 From: tiflo at csli.stanford.edu (T. Florian Jaeger) Date: Mon, 22 Dec 2008 17:12:53 -0500 Subject: [R-lang] Weird outcome logit mixed-effect model In-Reply-To: <495006FA.2030803@ucsd.edu> References: <494F6481.1070806@mpi.nl> <495006FA.2030803@ucsd.edu> Message-ID: <38dc9be90812221412j1075ae42g1e3773c9f532d44c@mail.gmail.com> A couple of additional points: (1) mixed logit models like yours are not fit using REML. The default fit uses Laplace approximation, which apparently is related to, but not the same as REML. (2) the insignificance is, of course, due to the large standard error, but the large standard error is due to the distribution of your data -- the fact that you have not new-deaccented data points. (3) you can recode Info to Info == "given", which makes it easy to show that given is different from the other two levels (consistent with much other previous research). But when you try to show that new is different from accessible, there is barely any evidence for that (as also shown, by splittin the data into a subset without "given" cases). Florian On Mon, Dec 22, 2008 at 4:30 PM, Roger Levy wrote: > Laura de Ruiter wrote: > >> Dear R-users and -experts, >> >> I am performing a rather simple analysis on a small data set (pasted below >> this email) and keep getting a to me inexplicable result. Perhaps I am >> missing something here - it would be great if someone could point out to me >> what I am doing wrong. >> >> I want to test whether the factor "Info" (which has three levels: "new", >> "given", "accessible") is a significant predictor for the binary variable >> "DeaccYN". The random factor is "Subject". The distribution of the data >> looks as follows: >> >> ----------------------------------------------------------------------------- >> >> Info >> DeaccYN given new accessible >> no 25 42 21 >> yes 11 0 1 >> ------------------------------------------------------------------------------ >> >> >> This is the model: >> >> ---------------------------------------------------------------------------------------------------------- >> >> deacc.lmer = lmer (DeaccYN ~ Info + (1|Subject), data = dat, family = >> "binomial") >> ----------------------------------------------------------------------------------------------------------------- >> >> >> However, given the distribution above, this outcome seems rather weird to >> me: >> >> --------------------------------------------------------------------------------------------------------- >> >> summary (deacc.lmer) >> Generalized linear mixed model fit using Laplace >> Formula: DeaccYN ~ Info + (1 | Subject) >> Data: dat >> Family: binomial(logit link) >> AIC BIC logLik deviance >> 60.4 70.82 -26.2 52.4 >> Random effects: >> Groups Name Variance Std.Dev. >> Subject (Intercept) 0.18797 0.43356 >> number of obs: 100, groups: Subject, 21 >> >> Estimated scale (compare to 1 ) 0.7316067 >> >> Fixed effects: >> Estimate Std. Error z value Pr(>|z|) >> (Intercept) -0.8635 0.3795 -2.2754 0.0229 * >> Infonew -18.7451 2764.2445 -0.0068 0.9946 >> Infoaccessible -2.2496 1.1186 -2.0110 0.0443 * >> --- >> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 >> >> > [...] > >> >> ---------------------------------------------------------------------------------------------------- >> >> >> Why should the difference between 25/11 and 21/1 be significant, but the >> difference between 25/11 and 42/0 not? Very odd to me seems the standard >> error of 2764! >> >> > [...] > > I was wondering: Is it perhaps a problem for the model that there are no >> cases in the DeaccYN == "yes" category for Info == "given"? And if this >> > ^^^^^ > I believe you mean "new" here. > >> is the case, why? >> Am I overlooking something here? >> > > Dear Laura, > > Independently of the issue that Florian is raising...you are right that the > lack of is a problem for the model (to be precise, it's a problem for > estimating the significance of the parameter estimate using the z value). > The z value, which is the basis of the significance level reported in the > lmer summary, is based on the Wald statistic, which is > > z = \hat{b} / StdError(\hat{b}) > > where \hat{b} is the parameter estimate. However, for parameter estimates > of large magnitudes (as is the case for "new" here), the standard error is > inflated, which leads to a small wald statistic. This problem is discussed > in a number of places, including Agresti (1996, 2002); here's a decent > mention of it online: > > http://userwww.sfsu.edu/~efc/classes/biol710/logistic/logisticreg.htm > > You can see this quite clearly if you add just one "yes/given" example at > the end of your data frame, for example: > > 101 132 new yes > > Then the model gives (being sure to correctly specify the levels of > "Info"): > > > summary(deacc.lmer) > Generalized linear mixed model fit by the Laplace approximation > Formula: Deacc ~ Info + (1 | Subject) > Data: dat > AIC BIC logLik deviance > 69.78 80.24 -30.89 61.78 > Random effects: > Groups Name Variance Std.Dev. > Subject (Intercept) 0.34332 0.58594 > Number of obs: 101, groups: Subject, 21 > > Fixed effects: > Estimate Std. Error z value Pr(>|z|) > (Intercept) -0.8996 0.3936 -2.286 0.02226 * > Iaccessible -2.2808 1.1460 -1.990 0.04656 * > Inew -3.0050 1.1395 -2.637 0.00836 ** > > > The next question, of course, is how to deal with the dataset you actually > have. If you weren't using a mixed-effects model, you could use a > likelihood-ratio test by comparing your full model with a simpler model; the > likelihood-ratio test isn't susceptible to problems with large parameter > estimates the way the Wald test is. For example: > > > deacc.glm = glm (Deacc ~ Info, data = dat, family = "binomial") > > deacc.glm1 = glm (Deacc ~ Info1, data = dat, family = "binomial") > > anova(deacc.glm,deacc.glm1,test="Chisq") > Analysis of Deviance Table > > Model 1: Deacc ~ Info > Model 2: Deacc ~ Info1 > Resid. Df Resid. Dev Df Deviance P(>|Chi|) > 1 97 52.452 > 2 98 71.600 -1 -19.148 1.210e-05 > > In principle, you could do this with your mixed-effects model, being sure > to use ML instead of REML fitting: > > > deacc.lmer = lmer (Deacc ~ Info + (1 | Subject), data = dat, family = > "binomial",REML=F) > > deacc.lmer1 = lmer (Deacc ~ Info1 + (1 | Subject), data = dat, family = > "binomial",REML=F) > > anova(deacc.lmer,deacc.lmer1) > Data: dat > Models: > deacc.lmer1: Deacc ~ Info1 + (1 | Subject) > deacc.lmer: Deacc ~ Info + (1 | Subject) > Df AIC BIC logLik Chisq Chi Df Pr(>Chisq) > deacc.lmer1 3 77.600 85.416 -35.800 > deacc.lmer 4 60.400 70.821 -26.200 19.2 1 1.177e-05 *** > > There is an argument that the likelihood-ratio test is anti-conservative > and hence inappropriate for comparing mixed-effects models differing only in > fixed-effects structure. (See Pinheiro & Bates, 2000, around page 76 or so. > The argument is made only for linear mixed-effects models and I'm not sure > of the status for logit mixed-effects models.) That being said, it's not > clear how strong the anti-conservativity is, and in your case it seems like > you have such an exceedingly powerful effect that you might be safe in using > the likelihood-ratio test here and just mentioning the potential > anti-conservativity as a caveat. > > So the summary is: believe it or not, how to assess the significance of the > parameter estimate for "new" for your model & dataset is a bit of an open > question, but it seems pretty clear that the estimate is significantly > non-zero. > > Best & hope this helps. > > Roger > > > > -- > > Roger Levy Email: rlevy at ucsd.edu > Assistant Professor Phone: 858-534-7219 > Department of Linguistics Fax: 858-534-4789 > UC San Diego Web: http://ling.ucsd.edu/~rlevy > > > _______________________________________________ > R-lang mailing list > R-lang at ling.ucsd.edu > http://pidgin.ucsd.edu/mailman/listinfo/r-lang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlevy at ucsd.edu Mon Dec 22 14:23:57 2008 From: rlevy at ucsd.edu (Roger Levy) Date: Mon, 22 Dec 2008 14:23:57 -0800 Subject: [R-lang] Weird outcome logit mixed-effect model In-Reply-To: <38dc9be90812221412j1075ae42g1e3773c9f532d44c@mail.gmail.com> References: <494F6481.1070806@mpi.nl> <495006FA.2030803@ucsd.edu> <38dc9be90812221412j1075ae42g1e3773c9f532d44c@mail.gmail.com> Message-ID: <4950137D.7030403@ucsd.edu> T. Florian Jaeger wrote: > A couple of additional points: > > (1) mixed logit models like yours are not fit using REML. The default > fit uses Laplace approximation, which apparently is related to, but not > the same as REML. Yes, true, thanks for the correction. In fact, if I now recall correctly, lme4 always uses Laplace-approximated maximum-likelihood estimation for logit models (is this right?). Roger From rlevy at ucsd.edu Mon Dec 29 15:15:41 2008 From: rlevy at ucsd.edu (Roger Levy) Date: Mon, 29 Dec 2008 15:15:41 -0800 Subject: [R-lang] Weird outcome logit mixed-effect model In-Reply-To: <495006FA.2030803@ucsd.edu> References: <494F6481.1070806@mpi.nl> <495006FA.2030803@ucsd.edu> Message-ID: <49595A1D.3020403@ucsd.edu> A clarification on my response below: I failed to note that I had defined the factor Info1 as dat$Info1 <- factor(ifelse(dat$Info %in% "accessible","accessible","givenOrNew")) Best to all, Roger > The next question, of course, is how to deal with the dataset you > actually have. If you weren't using a mixed-effects model, you could > use a likelihood-ratio test by comparing your full model with a simpler > model; the likelihood-ratio test isn't susceptible to problems with > large parameter estimates the way the Wald test is. For example: > >> deacc.glm = glm (Deacc ~ Info, data = dat, family = "binomial") >> deacc.glm1 = glm (Deacc ~ Info1, data = dat, family = "binomial") >> anova(deacc.glm,deacc.glm1,test="Chisq") > Analysis of Deviance Table > > Model 1: Deacc ~ Info > Model 2: Deacc ~ Info1 > Resid. Df Resid. Dev Df Deviance P(>|Chi|) > 1 97 52.452 > 2 98 71.600 -1 -19.148 1.210e-05 > > In principle, you could do this with your mixed-effects model, being > sure to use ML instead of REML fitting: > >> deacc.lmer = lmer (Deacc ~ Info + (1 | Subject), data = dat, family = > "binomial",REML=F) >> deacc.lmer1 = lmer (Deacc ~ Info1 + (1 | Subject), data = dat, family > = "binomial",REML=F) >> anova(deacc.lmer,deacc.lmer1) > Data: dat > Models: > deacc.lmer1: Deacc ~ Info1 + (1 | Subject) > deacc.lmer: Deacc ~ Info + (1 | Subject) > Df AIC BIC logLik Chisq Chi Df Pr(>Chisq) > deacc.lmer1 3 77.600 85.416 -35.800 > deacc.lmer 4 60.400 70.821 -26.200 19.2 1 1.177e-05 *** > > There is an argument that the likelihood-ratio test is anti-conservative > and hence inappropriate for comparing mixed-effects models differing > only in fixed-effects structure. (See Pinheiro & Bates, 2000, around > page 76 or so. The argument is made only for linear mixed-effects > models and I'm not sure of the status for logit mixed-effects models.) > That being said, it's not clear how strong the anti-conservativity is, > and in your case it seems like you have such an exceedingly powerful > effect that you might be safe in using the likelihood-ratio test here > and just mentioning the potential anti-conservativity as a caveat. > > So the summary is: believe it or not, how to assess the significance of > the parameter estimate for "new" for your model & dataset is a bit of an > open question, but it seems pretty clear that the estimate is > significantly non-zero. > > Best & hope this helps. > > Roger > > > -- Roger Levy Email: rlevy at ling.ucsd.edu Assistant Professor Phone: 858-534-7219 Department of Linguistics Fax: 858-534-4789 UC San Diego Web: http://ling.ucsd.edu/~rlevy