how (not) to do linguistics

Jonah Lehrer, the neuro-blogger, has a mixed track record, as far as I'm concerned. His initial blogging was nice, but a tad lightweight, then he started to sound a bit too Malcom Gladwell-ee (in that I wasn't entirely sure he knew what he was talking about beyond having a few short phone calls with one or two scientists then babbling on about a topic).

But he's hit a home run with this long New Yorker piece about the failure of the journal review process in science: The Truth Wears Off. He draws examples from medicine, physics, and psychology.

Perhaps the most disappointing part is the realization the the standards of testing and conclusiveness in linguistics are so far from those in more established science.

Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity.

Repeating studies is virtually unheard of in linguistics. Also, Lehrer mentions the publication bias in journals. When a result is discovered, there is a bias towards positive results. After a while, once the result is accepted, then only negative results are published because only that is "interesting" anymore. But I would expand this point to say this same bias exists at every stage of the research process. We want to find things that happen, we don't care about spending 5 years and thousands of hours discovering that X does NOT cause Y! So when young grad students begin scoping out a new study, they throw away anything that doesn't seem fruitful, where fruitful is defined as yielding positive results. This bias affects the very foundation of the research process, namely answering the basic question: what should I study?

As a side note, engineers seem perfectly happy to follow through on null results. They need to know the full scope of their problem before solving it. Scientists can learn a lot from engineers (and vice versa).

[Psychology professor Jonathan] Schooler recommends the establishment of an open-source database, in which researchers are required to outline their planned investigations and document all their results. “I think this would provide a huge increase in access to scientific work and give us a much better way to judge the quality of an experiment,” Schooler says. “It would help us finally deal with all these issues that the decline effect is exposing.”

Coincidentally, I was recently tweeting with moximer and jasonpriem about this and we agreed that research wikis are worth explolring. My vision would be something akin to Wikipedia but where a researcher stores all of their data, stimuli, results, etc, finished or not. The data could be tagged as tentative, draft, failed, successful, etc. As the research goes on, the data get updated. Not only would this record failure (which, as Leherer points out in the article) is as valuable as success, it also records change. How did a study evolve over time?

True, the data would become huge over time across many disciplines, but that just means means we need better and better data mining tools (and the boys at LingPipe are working away at those tools).

HT rapella

Eureka!

Generally I'm not a fan of new journals. Too much academic fluff is getting published already, I see no reason to fluff even more. However, this new journal struck me as having a novel and valuable mission behind it: The Journal of Serendipitous and Unexpected Results (JSUR).

An important component of scientific discovery is a disciplined examination of research results that contradict or negate extant hypotheses. Indeed the history of science is rife with examples of important discoveries arising from such results. However, there is a distinct lack of a forum in which such results can be presented and discussed in any meaningful way. We believe a forum for and dialogue on serendipitous and unexpected results will provide valuable insight and inform modern research practices (emphasis added).

It's like they created a whole journal just for Dan Everett! My first reaction was to double check that this wasn't coming from The Onion, but it appears to be legit. Jonah Lehrer recently made a similar point (see here) about the value of failure in science. In fact, there are informal forums for this kind of discussion; namely, meetings with advisors and lab meetings (as Lehrer points out). But rarely does this discussion get formalized and published. To pique the imagination of researchers, the journal editors pose a serious series of question templates. Which of the following are relevant to linguistics?

Can you demonstrate that:

• Technique X fails on problem Y.
• Hypothesis X can't be proven using method Y.
• Protocol X performs poorly for task Y.
• Method X has unexpected fundamental limitations.
• While investigating X, you discovered Y.
• Model X can't capture the behavior of phenomenon Y.
• Failure X is explained by Y.
• Assumption X doesn't hold in domain Y.
• Event X shouldn't happen, but it does.

(HT Boing Boing)

a deeply frustrating pursuit

Neuroblogger Jonah Lehrer has a new article about the value of failure in science and how it can lead to discovery. A nice, if somewhat light, read:  Accept Defeat: The Neuroscience of Screwing Up. The basic point is that our brains have two somewhat competing processes, one for perceiving errors (the “Oh shit!” circuit) and one for deleting irrelevant stuff (the Delete key). If the delete key wins, important discoveries are ignored (something like that).

Money quote:

While the scientific process is typically seen as a lonely pursuit — researchers solve problems by themselves — Dunbar found that most new scientific ideas emerged from lab meetings, those weekly sessions in which people publicly present their data. Interestingly, the most important element of the lab meeting wasn’t the presentation — it was the debate that followed. Dunbar observed that the skeptical (and sometimes heated) questions asked during a group session frequently triggered breakthroughs, as the scientists were forced to reconsider data they’d previously ignored. The new theory was a product of spontaneous conversation, not solitude; a single bracing query was enough to turn scientists into temporary outsiders, able to look anew at their own work.

On Pointiness

(screen grab from Stamp and Shout)

I've seen the Coexist bumper sticker above several times in the last week. I don't know how long it's been around, but a thought struck me the last time I saw it: there's no 'x'. All of the symbols used actually contain a version of the letter they are replacing, except the Star of David. There's no actual X figure within that symbol. Rather, it's the prevalence of pointiness that allows it to make for a suitable X replacement. I wonder if there is a different cognitive process at work? While we are reading the other letters, perhaps we are not actually reading the Star of David as an X, but rather engaging in some form of visual approximation (at least at first). Similar issues arise with textings like l8ter. Perhaps neuroscientist and reading expert Stanislas Dehaene has an answer in his new book Reading in the Brain. In Jonah Leher's review of that book, he suggests a possible answer:

One of the most intriguing findings of this new science of reading is that the literate brain actually has two distinct pathways for reading. One pathway is direct and efficient, and accounts for the vast majority of reading comprehension -- we see a group of letters, convert those letters into a word, and then directly grasp the word's meaning. However, there's also a second pathway, which we use whenever we encounter a rare and obscure word that isn't in our mental dictionary. As a result, we're forced to decipher the sound of the word before we can make a guess about its definition, which requires a second or two of conscious effort.

Perhaps this second pathway is the route needed to decipher the Star of David as X and 8 as -ate-. Just wondering out loud...

Oh, and btw, after staring at it a moment, I see that my initial reaction was wrong. There are actually four six Xs in the Star of David (thanks Q. Pheevr!), two each between each set of parallel lines. It takes a bit of magic picture blurry eye technique to see them (there's a more scientific term for that, right?). However, I doubt those Xs are recognized during the initial reading of the bumper sticker.

The Myth of 'Ghoti'

(cartoon found at Caldwell Reading)

In reviewing the new book Reading in the Brain by neuroscientist Stanislas Dehaene (do check out the cool Matrix-like book page), neuro-journalist Jonah Lehrer repeats the common claim that George Bernard Shaw coined the use of the spelling of fish as ghoti to demonstrate how weird English spelling is. I myself repeated this same claim to many students in the past, and in a few business presentations. Within linguistics, it has long been a truism. Rarely did anyone think to challenge its veracity. Until April 23, 2008 at 11:59 pm that is. Over a year and a half ago, Benjamin Zimmer debunked this claim as false on Language Log (see his post here). Zimmer showed not only that there is no record of Shaw having used it, but also that the use of ghoti goes back at least to "1855, a year before Shaw was born."

It remains a fun little example, mind you, just not attributable to Shaw.

BTW, if you do a Google image search on ghoti, as I just did, you will discover an underground, almost cultish devotion to the word involving Jedis, bimbos, and indie rock bands, oh my.