[R-lang] Re: analysis of acceptability judgements

[ellen gurman]

My colleague Antonella Sorace has been pointing out for years that 
grammaticality may be binary (in v out of what the grammar produces) but 
judgments of acceptability are anything but binary.

I strongly agree with Florian about the effects of implicit learning in 
experiments.  They're one good reason for inundating participants with fillers 
or doing corpus scale work or washing out the learning statistically or all of 
the above.  We have to assume that 'terse' experimental paradigms give us 
elegant examples of adjustments to very unusual distributions of stimuli - in 
effect, a picture of what people can do but not necessarily a picture of what 
they have occasion to do in the wild.

More apologies for off-topic comments,

Ellen Gurman Bard





________________________________
From: T. Florian Jaeger <tiflo@csli.stanford.edu>
To: lngmyers <lngmyers@ccu.edu.tw>
Cc: r-lang <r-lang@ling.ucsd.edu>
Sent: Sat, October 16, 2010 7:31:53 AM
Subject: [R-lang] Re: analysis of acceptability judgements


Hi James,

just a couple of short comment, since I thought there were several interesting 
points in your post (the program you mentioned also sounds superbly useful!):

Finally, in the above two places (Lingua paper and MiniJudge) I make an
>as-yet totally ignored statistical proposal about how to deal with a
>notorious bias in acceptability judgments: the reduction in sensitivity
>over the course of making many similar judgments.

We've seen the same thing though I would describe this more cautiously as 
sensitivity to the distribution of items in the experiment (this actually 
happens to be one of my favorite example in stats workshop I give). Rather than 
necessarily being a reduction of sensitivity in judgments, it may simply have 
the same source that reduced effects sizes over the course of balanced 
experiments of any type frequently exhibit. This source may actually reflect 
good old implicit learning. In agreement with what you say below this effect is 
clearly not only observed in binary judgment experiments, but much much more 
general (for example, it also shows up in self-paced reading experiments).

There is evidence that this might be due to linguistic (and or task) adaptation 
to the distributions presented in the experiment (see Fine et al, 
2010, http://www.hlp.rochester.edu/publications/Fineetal10.pdf; see also Farmer 
et al, submitted). In addition to the obvious interaction test to capture such 
effects (trial x conditions), Alex Fine's paper looks into possible ways to 
model these effects as belief update.

This is, of course, completely in line with what you are saying. I just wanted 
to point out that this "decreasing sensitivity" might actually be pointing to a 
rather interesting fact about linguistic representations (namely their 
plasticity) rather than merely a methodological problem (in this context, Wells 
et al's 2009 article; and Clayards et al 2008 might be of interest). 

Finally, for what it's worth, it's not my experience at all that acceptability 
judgments reduce to binary contrast. But, of course, that depends partly on how 
the task is set up and the nature of the fillers.

apologies for the off-topic post,

Florian

references:

Clayards, Tanenhaus, Aslin, and Jacobs. 2008. Perception
of speech reflects optimal use of probabilistic
cues. Cognition, 108:804–809.

Fine, A., Qian, T., Jaeger, T.F. & Jacobs, R. (2010). Is there syntactic 
adaptation in language comprehension? Proceedings of the 48th Annual Meeting of 
the Association for Computational Linguistics: Workshop on Cognitive Modeling 
and Computational Linguistics. Uppsala, Sweden. July, 2010. 

Wells, Christiansen, Race, Acheson, and MacDonald.
2009. Experience and sentence comprehension:
Statistical learning and relative clause comprehension.
Cognitive Psychology, 58:250–271.


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.ucsd.edu/mailman/private/ling-r-lang-l/attachments/20101016/c5111cb7/attachment.html