[R-lang] Re: analysis of acceptability judgements

[T. Florian Jaeger]

Hi James,

just a couple of short comment, since I thought there were several
interesting points in your post (the program you mentioned also sounds
superbly useful!):

Finally, in the above two places (Lingua paper and MiniJudge) I make an
> as-yet totally ignored statistical proposal about how to deal with a
> notorious bias in acceptability judgments: the reduction in sensitivity
> over the course of making many similar judgments.


We've seen the same thing though I would describe this more cautiously as
sensitivity to the distribution of items in the experiment (this actually
happens to be one of my favorite example in stats workshop I give). Rather
than necessarily being a reduction of sensitivity in judgments, it may
simply have the same source that reduced effects sizes over the course of
balanced experiments of any type frequently exhibit. This source may
actually reflect good old implicit learning. In agreement with what you say
below this effect is clearly *not* only observed in binary judgment
experiments, but much much more general (for example, it also shows up in
self-paced reading experiments).

There is evidence that this might be due to linguistic (and or task)
adaptation to the distributions presented in the experiment (see Fine et al,
2010, http://www.hlp.rochester.edu/publications/Fineetal10.pdf; see also
Farmer et al, submitted). In addition to the obvious interaction test to
capture such effects (trial x conditions), Alex Fine's paper looks into
possible ways to model these effects as belief update.

This is, of course, completely in line with what you are saying. I just
wanted to point out that this "decreasing sensitivity" might actually be
pointing to a rather interesting fact about linguistic representations
(namely their plasticity) rather than merely a methodological problem (in
this context, Wells et al's 2009 article; and Clayards et al 2008 might be
of interest).

Finally, for what it's worth, it's not my experience at all that
acceptability judgments reduce to binary contrast. But, of course, that
depends partly on how the task is set up and the nature of the fillers.

apologies for the off-topic post,

Florian

references:

Clayards, Tanenhaus, Aslin, and Jacobs. 2008. Perception
of speech reflects optimal use of probabilistic
cues. Cognition, 108:804–809.

Fine, A., Qian, T., Jaeger, T.F. & Jacobs, R. (2010). Is there syntactic
adaptation in language comprehension? Proceedings of the 48th Annual Meeting
of the Association for Computational Linguistics: Workshop on Cognitive
Modeling and Computational Linguistics. Uppsala, Sweden. July, 2010.

Wells, Christiansen, Race, Acheson, and MacDonald.
2009. Experience and sentence comprehension:
Statistical learning and relative clause comprehension.
Cognitive Psychology, 58:250–271.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.ucsd.edu/mailman/private/ling-r-lang-l/attachments/20101015/2bec2356/attachment.html