[R-lang] Re: analysis of acceptability judgements

[Kyle Gorman]

On Oct 15, 2010, at 9:47 AM, João Veríssimo wrote:

> Dear all,
> 
> Could anyone point me in the direction of papers/chapters that discuss
> the best way to analyse acceptability judgements data (with or without
> random effects)?

There is something of a dearth of methodological discussion of acceptability judgements, especially in linguistics. Two people who have wrote extensively about it are Jon Sprouse (his recent work) and Carson Schütze (his book). I interpret this work, and older work in psychophysics (Stanley S. Stevens' "On the theory of scales of measurement", Science 1946), to indicate that we should avoid Likert scales for linguistic judgement tasks. 

> For judgements on a scale (say, 1 to 7), I have been thinking about
> ordinal logistic regression, using ordered() and lmr(). I just don't
> know whether this makes sense with 7 or more categories.

Ordinal regression might make sense if you don't want to commit yourself to assuming a linear relationship between the ratings scale and whatever the predictors are. There are still some residual problems. 

One reason these types of Likert scales are disfavored in psychophysics is we don't know what to make of a subject who only uses values in the region [3, 5], or a subject who never uses 1, etc. This may be a central tendency bias, or a meaningful observation. The same is true of a subject's responses who are skewed away from the mean; it could be meaningful or a per-subject bias; a random intercept may be appropriate or not. For this reason, people have standardized Likert responses per subject, but the normality assumption may be bad, and there may be something meaningful in the stimuli that caused the subject not to use edge values. What if the subject has an internal rating of a stimulus as 3.5, but that value is not on the scale? We simply do not know if Likert scales are suitable for the kinds of interpretation we'd like to make about them. 

> For judgements made with a glider or line where participants can choose
> any point, the variable is continuous. But then lm() model predictions
> can be outside the scale (and I suppose variance is not constant).


Gliders are probably an improvement, but still not as good as magnitude estimation because they suffer from "boundary" problems: you can't take the slider beyond a certain point, and we face the same problems in modeling the results. If you're worried about values outside the value for the glider, that's easy to address; I'll assume the glider values runs from [0, 1]. One simple solution is to model logit(glider), since the logit function maps values on a (0, 1) scale to (-infinity, infinity). But it is undefined at [0, 1], so you'll have to truncate or delete those values (R does the former by default, just FYI). There may be other functions that make sense that scale the values to a scale bounded only by infinity. We're then assuming linearity between logit(glider) and your predictors, which may be undesirable. We also have the same per-subject issues. 

Kyle Gorman