the largest whorfian study EVER! (and why it matters)

Let me take the ball Mark Liberman threw on Monday and run with it a bit. Liberman posted a thorough discussion of Fausey and Broditsky's neo-Whorfian English and Spanish speakers remember causal agents differently. Specifically, he invited readers to carefully examine the methodology of the experiments themselves, and not just focus on the conclusions. It turns out that a few years ago another set of neo-Whorfians, Jürgen Bohnemeyer and company, published a paper that addressed similar methodological concerns:

Ways to go: Methodological considerations in Whorfian studies on motion events. (With S. Eisenbeiss and B. Narasimhan) Colchester: University of Essex, Department of Language and Linguistics (Essex Research Reports in Linguistics 50: 1-19). 2006.

This paper addressed experiments involving motion events like rolling and falling whereas Fausey and Broditsky's work addressed agentivity like breaking and popping, but there's enough overlap to warrant some comparison, particularly since the Bohnemeyer et al. paper specifically addresses methodology wrt Whorfian experiments.

But before I get into the details, let me state clearly why I think this is important. In other posts, I have dismissed popular lingo-topics like language evolution as outside the mainstream of linguistics because they don't bear directly on what I consider to be the center of the linguistics universe: How the brain does language. But linguistic relativity (aka, The Whorfian hypothesis) is one of the great questions of linguistics and cognitive science precisely because it bears directly on the question of how the brain does language. And we're only just now developing the proper tools and methodologies to study the question with scientific rigor. It may turn out that language does not affect other cognitive processes or the effect is minor. I don't care. I just want to know one way or the other. And it's work like Bohnemeyer's and Broditsky's that will lead us to knowing, eventually.

Now the fun stuff.

Bohnemeyer et al. start with an assumption about language types based on Talmy's cognitive semantics typology that classes languages as either satellite-framing or verb-framing, From Talmy:

...languages fall into two main types on the basis of where the Path is represented in a sentence expressing a Motion event [...]. In this two-category typology, if the Path is characteristically represented in the main verb or verb root of a sentence, the language is "verb framed", but if it is characteristically represented in the satellite and/or preposition, the language is "satellite framed".

In this typology, English is a satellite-frame language and Spanish is a verb-frame language. Bohnemeyer et al.  conclude that “verb-framed” (V) languages lexicalize path information in the verb root. Consequently, they require a separate expression for the “manner” of the motion event [...] The additional syntactic position renders manner in V–languages less “codable”. Consequently, manner is encoded more routinely in S–language discourse. Thus, the question arises whether S–language native speakers also pay more attention to manner when committing a motion event to memory and/or comparing it to other events." (emphasis added).

So satellite-frame languages like English push path info into separate phrases and verb-frame languages like Spanish push manner into separate phrases. Take the following English sentences:

• The ball rolled down the hill.
• The ball bounced down the hill.

In English, the manner of motion (rolling vs. bouncing) is encoded on the verb (i.e., different verbs), but the path is encoded as a separate phrase ("down the hill") as opposed to something like this:

• The ball moved down the hill rollingly.

Here the verb move is fairly simple and encodes no manner info by itself. As a note, Talmy's typology is based on colloquial and high frequency language use. Many languages allow both kinds of constructions, it's just that one is highly frequent in "everyday" speech.

And there's the crux of Bohnemeyer et al.'s experiments. But, they also noticed conflicting results in several other neo-Whorfian studies that they believed were a result of methodology so they set out to investigate methodology. Those other papers used a similarity-judgement task, so Bohememeyer et al. used that task as well. Particularly, they gave particpants a series of three short animated videos to watch. For example, one video was a tomato rolling down a hill. Their methodology is difficult to understand if you just read through, so I'm going to try to help with some bold-facing and re-structuring of paragraphs and a few re-wordings. What follows is a near-quote:

...we conducted a similarity-judgment task analogous in design to those reviewed above with native speakers of 17 genetically and typologically diverse languages – to our knowledge, the largest sample of languages ever used in a Whorfian study. To control for the effects of individual manner or path contrasts, we cross-classified six path types with four manner types, realizing all possible combinations in our stimulus set and counterbalancing for frequency of occurrence.

[... FYI, the 17 languages were Basque, Catalan, Dutch, French, German, Hindi, Italian, Jalonke, Japanese, Lao, Polish, Spanish, Tamil, Tidore, Tiriyó, Turkish, and Yukatek]

The materials consisted of 72 triads (i.e., 72 sets of three videos). The targets (i.e., main videos) were 24 motion-event video-animations which systematically varied:

1. four manners of motion (SPIN, ROLL, BOUNCE, SLIDE),
2. three scenarios with different “ground” objects (inclined ramp; field with tree and rock; field with hut and cave), and
3. two directed paths (motion UP/RIGHT, DOWN/LEFT)

For each of these targets (e.g. tomato-ROLLs-UP-RAMP, see Figure1), we created a same-manner (different-path) variant (e.g. tomato-ROLLs-DOWN-RAMP), and three types of same-path (different-manner) variants (here, OUNCE/SLIDE/SPIN-UP-RAMP).

This resulted in 72 triads with a target clip, a same-manner variant and one of the three same-path variants. The variants were presented side by side, 1 second after the target-clip presentation ended (see Figure 1).

(figure from page 7)

Their methodology was as follows (emphasis added to help readability):

• The 72 triads were distributed across 6 randomized presentation lists in a Latin-square design.
• Each list was given to two participants per language (in reverse presentation order).
• Each list contained 12 triads, with the target clips combining the four manners of motions with the three scenarios so that each participant saw all 12 combinations in the target clip.
• The number of UP/RIGHT and DOWN/LEFT motions in the target- and variant-clips as well as the manners of motions in the different-manner variants was counterbalanced across the lists, as was the position in which the variants were presented on the screen.
• The position of the ground objects remained the same in all clips.
• Minimal variations in the triad clips allow us to take into account the effects of different manners, paths and scenarios, but make our test triads quite similar.
• Added 38 filler triads to each list, which involved other types of events and variations (e.g. replacing either the agent or the goal in a possession-transfer event with another character) and served to prevent the participants from settling into a fixed response pattern.
• Instructions to participants were translated into their native languages.

So participants were watching a bunch of short videos in sets of three [A, B, C]. If I understand correctly, for each set, they were asked to determine if B or C is more similar to A than the other.  For example, imagine I show you three cartoons:

A) a tomato rolling up a hill (forget physics, it's cartoon world).
B) a tomato rolling down a hill.
C) a tomato bouncing up a hill.

Then I ask you to chose B or C as more similar to A (forced choice, you gotta pick one). If you say B, then the manner of rolling is more salient for you than directional path. If you say C, then directional path is more salient than manner.

Now, if your native language forces you to push path information into separate phrases like prepositional phrases (e.g., "up a hill"), it may be the case that this would cause you to say C is more similar because the path is the same. Hence your language caused your perception of similarity to behave in a particular way, hence language affects thought.

So what did they discover?

• First, they found no simple categorical distinction between verb-frame language speakers and satellite-language speakers decisions. If they had, it would have been fairly strong evidence for linguistic relativity and maybe then they'd be the ones writing WSJ articles instead of Broditsky, haha.

• They did find that manner of motion by itself seemed to influence similarity judgement significantly (in the technical sense), but this influence was independent of language (i.e., it didn't have anything clearly to do with verb-framing or satellite-framing).

• They found that the data they gathered could not be accounted for by socio-cultural factors because they had such a diverse participant pool.

• Interestingly, they did find that contrasts between certain manner pairs DID correlate significantly with language. So, Turkish speakers (a verb-frame language) chose the video with the same manner 88% of the time when the contrast was slide vs. roll, but only 50% of the time when it was roll vs. bounce. They left open the question of why this is so.

• The most striking conclusion, however, was that more is better in the sense that they discovered that "strong claims regarding the (in)validity of the Whorfian hypothesis in the encoding of motion events cannot be made on the basis of a limited number of languages or a restricted range of manner and path contrasts." (emphasis added).

I found the following passage to be the most important in the paper, especially wrt Fausey and Broditsky's work:

If only Polish (an S-language) and Yukatek (a V-language) had been selected to test for language-specific performance effects in our nonlinguistic task, the highly significant difference between speakers of these two languages in their degree of manner preference would have supported a strong version of the Whorfian claim. Conversely, had we chosen only to contrast German (S) with Spanish (V) using the identical stimuli and experimental procedure, we would have reached the opposite conclusion, since speakers of these languages do not differ significantly from one another in the frequency with which they base their similarity judgements on manner of motion.

The lesson here is that methodology counts. Fausey and Broditsky used a very small participant pool and it's hard to tell how that affected their results.


FYI: the word profound does not occur in Bohnemeyer's article.