Monday, October 15, 2007

Frequency effects in linguistics

For the record, there are known to be a variety of “frequency effects” in language. A brief survey:

Zipf's law: roughly speaking, the most frequent word in a corpus will be about twice as frequent as the second most frequent (i.e., twice as many tokens).

Word recognition: Dahan et al (pdf) :“frequency affects the earliest moments of lexical access”

Sentence processing: Lau et al : Frequency effects “give rise to reaction time differences in sentence processing tasks"

No comments:

A linguist asks some questions about word vectors

I have at best a passing familiarity with word vectors, strictly from a 30,000 foot view. I've never directly used them outside a handfu...