[R-lang] Re: analysis of acceptability judgements

[Thomas Weskott]

Apologies in advance for one more off-topic comment, but since Antti
Arppe mentioned the work by Gisbert Fanselow and myself, I feel urged
to point to a more recent paper which was just accepted by LANGUAGE,
in which we argue that ME of linguistic acceptability, despite all its
advantages, is beset with at least two problems: (1) a severe
task-related problem (there's no zero point of acceptability (though,
admittedly, there's one for UNacceptability, e.g. "John loves Mary"),
plus the findings from psychophysics we cite that cast severe doubt
onto the mathematical soundness of the mental computations
participants perform in ME studies.  And, (2), we show that the
purported higher informativity of ME data (as compared to binary and
Likert-scale data) does not hold for two-factorial and three-factorial
designs testing word order variation in German -- we looked at partial
eta-squared and didn't find a difference between the measures. In
fact, we found ME to perform somewhat worse in the three-factorial
study, possibly due to its being prone to producing spurious variance.
If anybody's interested in the paper, drop me a line, and I'll send you a copy.

And Jon Sprouse has a manuscript on his homepage reporting on an
experiment in which he examines problem (1) mentioned above
extensively.

Best,
Thomas

On 10/16/10, ellen gurman <egb444@yahoo.com> wrote:
> My colleague Antonella Sorace has been pointing out for years that
> grammaticality may be binary (in v out of what the grammar produces) but
> judgments of acceptability are anything but binary.
>
> I strongly agree with Florian about the effects of implicit learning in
> experiments.  They're one good reason for inundating participants with
> fillers
> or doing corpus scale work or washing out the learning statistically or all
> of
> the above.  We have to assume that 'terse' experimental paradigms give us
> elegant examples of adjustments to very unusual distributions of stimuli -
> in
> effect, a picture of what people can do but not necessarily a picture of
> what
> they have occasion to do in the wild.
>
> More apologies for off-topic comments,
>
> Ellen Gurman Bard
>
>
>
>
>
> ________________________________
> From: T. Florian Jaeger <tiflo@csli.stanford.edu>
> To: lngmyers <lngmyers@ccu.edu.tw>
> Cc: r-lang <r-lang@ling.ucsd.edu>
> Sent: Sat, October 16, 2010 7:31:53 AM
> Subject: [R-lang] Re: analysis of acceptability judgements
>
>
> Hi James,
>
> just a couple of short comment, since I thought there were several
> interesting
> points in your post (the program you mentioned also sounds superbly
> useful!):
>
> Finally, in the above two places (Lingua paper and MiniJudge) I make an
>>as-yet totally ignored statistical proposal about how to deal with a
>>notorious bias in acceptability judgments: the reduction in sensitivity
>>over the course of making many similar judgments.
>
> We've seen the same thing though I would describe this more cautiously as
> sensitivity to the distribution of items in the experiment (this actually
> happens to be one of my favorite example in stats workshop I give). Rather
> than
> necessarily being a reduction of sensitivity in judgments, it may simply
> have
> the same source that reduced effects sizes over the course of balanced
> experiments of any type frequently exhibit. This source may actually reflect
> good old implicit learning. In agreement with what you say below this effect
> is
> clearly not only observed in binary judgment experiments, but much much more
> general (for example, it also shows up in self-paced reading experiments).
>
> There is evidence that this might be due to linguistic (and or task)
> adaptation
> to the distributions presented in the experiment (see Fine et al,
> 2010, http://www.hlp.rochester.edu/publications/Fineetal10.pdf; see also
> Farmer
> et al, submitted). In addition to the obvious interaction test to capture
> such
> effects (trial x conditions), Alex Fine's paper looks into possible ways to
> model these effects as belief update.
>
> This is, of course, completely in line with what you are saying. I just
> wanted
> to point out that this "decreasing sensitivity" might actually be pointing
> to a
> rather interesting fact about linguistic representations (namely their
> plasticity) rather than merely a methodological problem (in this context,
> Wells
> et al's 2009 article; and Clayards et al 2008 might be of interest).
>
> Finally, for what it's worth, it's not my experience at all that
> acceptability
> judgments reduce to binary contrast. But, of course, that depends partly on
> how
> the task is set up and the nature of the fillers.
>
> apologies for the off-topic post,
>
> Florian
>
> references:
>
> Clayards, Tanenhaus, Aslin, and Jacobs. 2008. Perception
> of speech reflects optimal use of probabilistic
> cues. Cognition, 108:804–809.
>
> Fine, A., Qian, T., Jaeger, T.F. & Jacobs, R. (2010). Is there syntactic
> adaptation in language comprehension? Proceedings of the 48th Annual Meeting
> of
> the Association for Computational Linguistics: Workshop on Cognitive
> Modeling
> and Computational Linguistics. Uppsala, Sweden. July, 2010.
>
> Wells, Christiansen, Race, Acheson, and MacDonald.
> 2009. Experience and sentence comprehension:
> Statistical learning and relative clause comprehension.
> Cognitive Psychology, 58:250–271.
>
>
>