Wednesday, December 23, 2009

Vision Affects Language Processing

Does watching a leaf fall help you process the sentence the leaf is falling down? Apparently, no, it hurts. It slows you down. Cognitive Daily reviews research supporting this conclusion. Money quote:

...people take longer to process sentences that match the movement of an animation than they do to process sentences that don't match it. Kaschak's team reasons that we must be using the same region of the brain to process the motion itself as we do to process the language describing that motion.


uzza said...

Regarding leaves falling, in a signed language like ASL the sentence often amounts to an iconic mimicry of the actual leaf falling, so at first glance this seems impossible. There are constructions though that contradict the iconicity with motion opposite to the meaning. No telling if they are processed faster.

'Yet the simple concept of "downward motion" does appear to distract from our ability to process a simple sentence describing a particular sort of downward motion.'
Since a sentence describing “x” must necessarily include “x” how can 'the simple concept of “x”' distract us? This makes no sense.

Gesture studies indicate that our mental model of “downward” includes some kind of visualspatial metaphor that we subconsciously produce in attempting word retrieval, and, gesture-speech mismatches are harder to process, not easier.

Also sign language aphasias prove that linguistic and non-linguistic processing of visual input are done by different areas of the brain, contra the author's conclusion. I have no explanation for these results, but I think neither do the authors.

Chris said...

uzza, thanks for the comment!

I must admit I found myself skeptical of the author's conclusions also. It did in fact seem unintuitive. The role of visual processing in signed languages is a great area to pursue this. I hope more gets published.


Putting the Linguistics into Kaggle Competitions

In the spirit of Dr. Emily Bender’s NAACL blog post Putting the Linguistics in Computational Linguistics , I want to apply some of her thou...