tag:blogger.com,1999:blog-5208073967144633092024-03-13T13:16:34.482-04:00The Lousy LinguistNotes on linguistics and cognitionChrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.comBlogger657125tag:blogger.com,1999:blog-520807396714463309.post-57884783466318615712020-02-17T16:09:00.001-05:002020-02-17T16:09:24.505-05:00TV Linguistics - Pronouncify.com and the fictional Princeton Linguistics department [reposted from 11/20/10]<br />
<br />
I spent Thursday night on a plane so I missed <i>30 Rock</i> and the
most linguistics oriented sit-com episode since ... since ... um ...
okay, the most linguistics oriented sitcom episode EVER! But thanks to
the innerwebz, I have caught up on my TV addiction.<br />
<br />
The set-up has Jack Donaghy being the voice of <a href="http://www.nbc.com/30-rock/exclusives/pronouncify/">Pronouncify.com</a>, (I BEG you to sign up, PLEASE!!!!) a website that demonstrates the <i>correct </i>pronunciation
of all English words. Apparently, when Jack was a poor undergrad at
Princeton, he was hired by the "Linguistics Department" to pronounce
every word in an English dictionary to preserve the <i>correct </i>pronunciation
for generations to come. But they sold his readings, and hence his
voice is now the voice of Pronouncify.com (as well as the first perfect
microwave...).<br />
<br />
Here is as faithful a transcript of the critical dialogue as I can muster:<br />
<br />
<b><span class="Apple-style-span" style="color: red;">Jack</span></b>: <i>Those bastards</i>!<br />
<b><span class="Apple-style-span" style="color: blue;">Liz</span></b>: <i>Who bastards</i>?<br />
<b><span class="Apple-style-span" style="color: red;">Jack</span></b>: <i>Part of my Princeton scholarship included work for the <b>Linguistics department</b>.
They wanted me to record every word in the dictionary to preserve the
perfect American accent in case of nuclear war. Well, the cold war
ended, and Princeton began selling the recordings</i>.<br />
<b><span class="Apple-style-span" style="color: blue;">Liz</span></b>: <i>So people can just buy your voice</i>?<br />
<b><span class="Apple-style-span" style="color: red;">Jack</span></b>: <i>Ohhhh, the things it's been dragged into. Thomas the Tank Engine; Wu-Tang songs</i>...<br />
<br />
This
must have been the glory days before the hippies took over and started
"protecting" undergrads from "exploitation." Whatever...<br />
<br />
In
any case, it's understandable that this trivial tid-bit of academic
minutia blew right by most people, but it is a fact of the world we live
in that Princeton University does not have a linguistics department per
se. They do offer an <a href="http://www.princeton.edu/linguistics/about/">Undergraduate Program in Linguistics</a>
in which students can "pursue a Certificate in Linguistics," but this
is not an official department as far as I understand it. Jack, if he is
the same age as the actor Alec Baldwin, would have been at Princeton in
late 1970s. Maybe they had a full fledged department back then, I
honestly don't know.<br />
<br />
You can watch the episode <a href="http://www.nbc.com/30-rock/video/college/1261073/">College </a>at
NBC, or wherever else you prefer. BTW, there's an awesome ode to color
perception conundrums at the end as well. It's all kinda
linguisticee/cog sciencee (I never know how to add the -ee morpheme?).<br />
<br />
Random after-point: Near the end of Thursday's episode of <i>Community</i>, Dean Pelton actually utilized the Shakespearean subjunctive construction <i>Would that X were Y</i>... He says "<i>Would that this hoodie were a time hoodie</i>" around the 19:20 mark (see Hamlet, <i>would it were not so, you are my mother</i>). Just thought that was kinda awesome.<br />
<br />
And not for nuthin', but if you haven't seen Tina Fey's Mark Twain Prize speech, it's a gem: <a href="http://www.huffingtonpost.com/2010/11/16/tina-fey-mark-twain-speech_n_784178.html">HERE</a>.Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-21007549056883981532020-02-17T16:07:00.001-05:002020-02-17T17:26:20.244-05:00TV anachronisms - The birth of a metaphor[reposted from 4/12/13]<br />
<br />
A Twitter exchange with <a href="https://twitter.com/bgzimmer">Ben Zimmer</a> over the metaphorical use of the phrase "<i>pause button</i>" in the new TV show <a href="http://en.wikipedia.org/wiki/The_Americans_(2013_TV_series)">The Americans</a>
(set in 1981) led me to think about how metaphors begin their lives. I
didn't watch the episode in question, but apparently several viewers
noticed that the show used the phrase "pause button" metaphorically to
mean something like <i>to put a romantic relationship on hold</i>.<br />
<br />
Ben
tweeted this fact as a likely anachronism, presumably because
the technology of pause buttons was too young in 1981 to have likely
jumped to metaphorical use by then. I was not the only one
who immediately took to Google Ngrams to start testing this hypothesis.
In the end, Tweeter <a href="https://twitter.com/Manganpaper">@Manganpaper</a> found a good example from 1981 from some kind of <a href="http://books.google.com/books?id=s79X7hvRd8QC&q=%22pause+button%22">self-help book</a>.<br />
<br />
But what interests me is an example I found from 1987:<br />
<blockquote class="tr_bq">
<i>Consumers have pushed the "pause" button on sales of video-cassette recorders, for years in the fast-forward mode</i>.</blockquote>
Ben reluctantly conceded the example:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-mZuf1XOwaYc/UWeKzRxUKjI/AAAAAAAAAk0/VWM6cGyr4JU/s1600/zimmer.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="241" src="https://4.bp.blogspot.com/-mZuf1XOwaYc/UWeKzRxUKjI/AAAAAAAAAk0/VWM6cGyr4JU/s320/zimmer.jpg" width="320" /></a></div>
<br />
I'd
have to review my historical linguistics books, but I don't think
words necessarily shift their meanings radically all at once.
I believe they can take on characteristics of associated meanings
slowly, thus <a href="http://thelousylinguist.blogspot.com/2008/11/semantic-narrowing.html">widening or narrowing</a>
their meaning as their linguistic environment unfolds. Eventually, a
word can come to mean something quite radically different than it
originally meant. I see no reason that the life of a metaphor could
not follow a similar trajectory. Ben objected to the fact that the 1987
use of "pause button" I linked to was semantically linked to the literal
use of actual pause buttons because it dealt with the conceptual space
of VCR sales. But my hunch is that this is how many metaphors start
their lives, making small conceptual leaps, not big ones. I could be
wrong though. The sad truth is that finding good empirical data for the
life span of metaphors is extremely difficult. The fact is that even
with the awe inspiring large natural language data
sets currently available in many languages, studying a linguistically
high level data type like metaphor remains out of reach of most NLP
techniques.<br />
<br />
But this is why our NLP blood boils. There are miles to go before we sleep...Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com2tag:blogger.com,1999:blog-520807396714463309.post-35602927024838490362017-12-26T19:20:00.000-05:002017-12-28T16:36:05.251-05:00Putting the Linguistics into Kaggle Competitions<h3 class="graf graf--h3" name="5172">
<span style="font-size: small;"><span style="font-weight: normal;">In the spirit of Dr. Emily Bender’s NAACL blog post <a class="markup--anchor markup--p-anchor" data-href="https://naacl2018.wordpress.com/2017/12/19/putting-the-linguistics-in-computational-linguistics/" href="https://naacl2018.wordpress.com/2017/12/19/putting-the-linguistics-in-computational-linguistics/" rel="noopener" target="_blank">Putting the Linguistics in Computational Linguistics</a>, I want to apply some of her thoughts to the data from the recently opened Kaggle competition <a class="markup--anchor markup--p-anchor" data-href="https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge" href="https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge" rel="noopener" target="_blank">Toxic Comment Classification Challenge</a>.</span></span></h3>
<blockquote class="tr_bq">
<h3 class="graf graf--h3" name="5172">
<span style="font-size: small;"><span style="font-weight: normal;">Dr. Bender's point: <i>not all NAACL papers need to be linguistically informative—but they should all be linguistically informed</i>. </span></span></h3>
<h3 class="graf graf--h3" name="5172">
<span style="font-size: small;"><span style="font-weight: normal;">Toxic Comment Challenge Correlate: </span></span><span style="font-size: small;"><span style="font-weight: normal;"><i>not all Kaggle submissions need to be linguistically informative—but they should all be linguistically informed</i></span></span></h3>
</blockquote>
<h3 class="graf graf--h3" name="5172">
<span style="font-size: small;"><span style="font-weight: normal;"></span></span></h3>
<h3 class="graf graf--h3" name="5172">
<span style="font-size: small;"><span style="font-weight: normal;">First let me say I'm really excited about this competition because A) it uses real language data, B) it's not obvious what techniques will work well (it's a novel/interesting problem), and C) it's a socially important task to get right. Kudos to Kaggle for supporting this. </span></span><span style="font-size: small;"><span style="font-weight: normal;"> </span></span></h3>
<h3 class="graf graf--h3" name="5172">
<span style="font-size: small;"><span style="font-weight: normal;"> </span></span></h3>
<h3 class="graf graf--h3" name="5172">
<span style="font-size: small;"><span style="font-weight: normal;">Dr. Bender identifies four steps to put linguistics into CL</span></span></h3>
<blockquote class="tr_bq">
<span style="font-size: small;">Step 1: Know Your Data<br />Step 2: Describe Your Data Honestly<br />Step 3: Focus on Linguistic Structure, At Least Some of the Time<br />Step 4: Do Error Analysis</span></blockquote>
<div class="graf graf--p" name="ad5e">
<span style="font-size: small;">Since this is preliminary, I’ll concentrate on just steps 1 & 3</span><br />
<br />
<span style="font-size: small;"><b>UPDATE 12/28/2017 #2: </b>Robert Munro has written up an excellent analysis of this data, <a href="http://www.junglelightspeed.com/a-step-in-the-right-direction-for-nlp/" target="_blank">A Step in the Right Direction for NLP</a>. He addresses many of the points I make below. He also makes some solid suggestions for how to account for bias. Well worth a read. <b><br /></b></span><br />
<br />
<span style="font-size: small;"><b>UPDATE 12/28/2017 #1</b>: My original analysis of their IAA process was mistaken. I have corrected it. Also added another comment under "Some Thoughts" at the end.</span></div>
<h4 class="graf graf--h4" name="37cf" style="text-align: center;">
<span style="font-size: large;">Step 1: Know Your Data</span></h4>
<div class="graf graf--p" name="bda7">
<span style="font-size: small;"><b>The data download </b></span></div>
<ul>
<li><span style="font-size: small;">“large number of Wikipedia comments which have been labeled by human raters for toxic behavior. The types of toxicity are: toxic, severe_toxic, obscene, threat, insult, identity_hate” (note: there are a large number of comments with no ratings; presumably these are neutral)</span></li>
<li><span style="font-size: small;">Actual download for Training set is a CSV of ID, comment, rating (1 or 0) for each label. </span></li>
<li><span style="font-size: small;">100k talk page diffs</span></li>
<li><span style="font-size: small;">Comments come from 2001–2015</span></li>
<li><span style="font-size: small;">There are 10 judgements per diff (final rating is the average)</span></li>
<li><span style="font-size: small;">Here’s the original paper discussing this work <a class="markup--anchor markup--p-anchor" data-href="https://arxiv.org/pdf/1610.08914.pdf" href="https://arxiv.org/pdf/1610.08914.pdf" rel="noopener" target="_blank">Ex Machina: Personal Attacks Seen at Scale</a> by Wulczyn, Thain, and Dixon. </span></li>
</ul>
<div class="graf graf--p" name="7a8b">
<span style="font-size: small;"><b>Who created the data?</b></span></div>
<ul>
<li><span style="font-size: small;">Wikipedia editors</span></li>
<li><span style="font-size: small;">Largely young white men (lots of <a href="https://i-d.vice.com/en_us/article/gyqbmx/the-feminist-editors-fighting-for-gender-equality-on-wikipedia?utm_campaign=sharebutton" target="_blank">research</a> on this)</span></li>
<ul>
<li><span style="font-size: small;">2011 WMF survey, 90% of Wikipedians are men, 9% are female, and 1% are transgender/transsexual</span></li>
<li><span style="font-size: small;">53% are under 30 (see <a href="https://en.wikipedia.org/wiki/Wikipedia:Wikipedians#Demographics" target="_blank">here</a> for more info) </span></li>
</ul>
</ul>
<span style="font-size: small;"><b>Who rated the data?</b></span><br />
<ul>
<li><span style="font-size: small;">3591 Crowdflower participants (you have to dig into the paper and associated repos to discover this)</span></li>
<li><span style="font-size: small;">This is a huge number of raters. </span></li>
<li><span style="font-size: small;">Looking at the demographics spreadsheet available <a href="https://figshare.com/articles/Wikipedia_Talk_Labels_Toxicity/4563973" target="_blank">here</a>, most are not native English speakers. I'm surprised this was allowed</span></li>
<li><span style="font-size: small;">Only18% were native speakers of English</span></li>
<li><span style="font-size: small;">Only three gender options were offered (male, female, other). Only one user chose other</span></li>
<li><span style="font-size: small;">54% are under 30</span></li>
<li><span style="font-size: small;">65% chose male as their gender response</span></li>
<li><span style="font-size: small;">68% chose bachelors, masters, doctorate, or professional as level education</span></li>
<li><span style="font-size: small;">I'm admittedly confused by how they obtained their final ratings because the paper discusses a slightly different task (not involving the six labels specifically), and the authors' wiki discussion of annotation and data release (<a href="https://meta.wikimedia.org/wiki/Research:Detox/Data_Release#Annotations_Corpora" target="_blank">here</a>) also discusses what look like slightly different tasks. Were these six labels aggregated post hoc over different tasks using the same data? I don't see a clear example of asking human raters to give judgments for these six specific labels. I'll keep digging.</span></li>
<li><span style="font-size: small;">In general, I find the labels problematic (Dixon admitted to some of this in a discussion <a href="https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/46131" target="_blank">here</a>). </span></li>
</ul>
<div class="graf graf--p" name="f7ea">
<span style="font-size: small;"><b>Language varieties (in comments data)?</b></span></div>
<ul>
<li><span style="font-size: small;">Lightly edited English, mostly North American (e.g., one comment said they "corrected" the spelling of “recognised” to “recognized)</span></li>
<li><span style="font-size: small;">Some commenters self-identify as non-native speakers </span></li>
<li><span style="font-size: small;">Lots of spelling typos </span></li>
<li><span style="font-size: small;">Mixed punctuation usage</span></li>
<li><span style="font-size: small;">Some meta tags in comments</span></li>
<li><span style="font-size: small;">Inconsistent use of spaces before/after commas</span></li>
<li><span style="font-size: small;"><span style="font-size: small;">Inconsistent</span> use of caps (</span><span style="font-size: small;"><span style="font-size: small;">inconsistent for </span>both all caps and camel case)</span></li>
<li><span style="font-size: small;">Some examples of ACSII art</span></li>
<li><span style="font-size: small;">Excessive apostrophe usage (one comment has several hundred apostrophes)</span></li>
<li><span style="font-size: small;">Length of comments vary considerably</span></li>
<ul>
<li><span style="font-size: small;">From five words, to hundreds of words </span></li>
<li><span style="font-size: small;">This might have a spurious effect on the human raters</span></li>
<li><span style="font-size: small;">Humans get tired. When they encounter a long comment, they may be tempted to rely solely on the first few sentences, or possibly the mere presence of some key words.</span></li>
</ul>
<li><span style="font-size: small;">Some character encoding issues: †</span></li>
</ul>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<span style="font-size: large;"><b>Step 3: </b></span></div>
<div style="text-align: center;">
<span style="font-size: large;"><b>Focus on Linguistic Structure, At Least Some of the Time</b></span></div>
<br />
<span style="font-size: small;"><b>Lexicon</b></span><br />
<ul>
<li><span style="font-size: small;">A did a quick skim of a few dozen of the neutral comments (those with 0s for all labels) and it looked like they had no swear words.</span></li>
<li><span style="font-size: small;">I fear this will lead a model to over-learn that the mere presence of a swear word means it should get a label.</span></li>
<li><span style="font-size: small;">See the excellent blog <a href="https://stronglang.wordpress.com/" target="_blank">Strong Language</a> for reasons this is not true. </span></li>
<li><span style="font-size: small;">I would throw in some extra training data that includes neutral comments with swear words. <b><br /></b></span></li>
</ul>
<span style="font-size: small;"><b>Morphology</b></span><br />
<ul>
<li><span style="font-size: small;">Perhaps due to non-native speakers or typos, the wrong morphological variants can be found for some words (e.g., "<i>I am not the only one who finds the article too ridiculous and <u><b>trivia</b></u> to be included</i>.")</span></li>
<li><span style="font-size: small;">Lemmatizing can help (but it can also hurt)</span></li>
</ul>
<span style="font-size: small;"><b>Syntax</b></span><br />
<ul>
<li><span style="font-size: small;">Some comments lack sentence ending punctuation, leading to run ons. (e.g., "<i>It was very constructive you are just very very stupid</i>.") </span></li>
<li><span style="font-size: small;">Large amount of grammatical errors</span></li>
<li><span style="font-size: small;">If any parsing is done (shallow or full), could be some issues </span></li>
<li><span style="font-size: small;">A Twitter trained parser might be useful for the short comments, but not the long ones.</span></li>
</ul>
<span style="font-size: small;"><b><span style="font-size: x-small;">Discourse</span></b></span><br />
<ul>
<li><span style="font-size: small;">All comments are decontextual</span><span style="font-size: small;"> </span></li>
<li><span style="font-size: small;">Missing the comments before reduces information (many comments are responses to something, but we don't get the something)</span><span style="font-size: small;"> </span></li>
<li><span style="font-size: small;">Also missing the page topic</span><span style="font-size: small;"> that the comment is referring to</span></li>
<li><span style="font-size: small;">Hence, we're missing discourse and pragmatic links </span></li>
</ul>
<span style="font-size: small;"><br /></span>
<span style="font-size: small;"><b>Some Thoughts</b></span><br />
<span style="font-size: small;">The data is not simply <i>toxic comments</i>, but it’s the kind of toxic comments that young men make to other young men. And the labels are what young, educated, non-native English speaking men think counts as toxic. </span><br />
<span style="font-size: small;"><br /></span>
<span style="font-size: small;">My gut feeling is that the generalizability of any model trained on this data will be limited. </span><br />
<br />
<span style="font-size: small;">My main concern though is the lack of linguistic context (under Discourse). Exactly what counts as "toxic" under these de-contextual circumstances? Would the rating be the same if the context were present? I don't know. My hunch is that at least some of these ratings would change.</span><br />
<span style="font-size: large;"><br /></span><span style="font-size: large;">
</span><span style="font-size: small;"><br /></span>Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com4tag:blogger.com,1999:blog-520807396714463309.post-82709480619948957792017-10-18T17:13:00.000-04:002017-10-18T20:16:40.930-04:00A linguist asks some questions about word vectorsI have at best a passing familiarity with word vectors, strictly from a 30,000 foot view. I've never directly used them outside a handful of toy tutorials (though they have been embedded in a variety of tools I use professionally). There is something very linguisticee about word vectors, so I'm generally happy with their use. The basic idea goes back aways, but was most famously articulated by J. R. Firth's pithy phrase "<em>You shall know a word by the company it keeps</em>" (c. 1957).<br />
<br />
This is a set of questions I had when reading Chris McCormick's tutorial <a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/" target="_blank">Word2Vec Tutorial - The Skip-Gram Model</a>. My ideal audience is NLPers who have a good understanding of the SOTA of word vectors. <br />
<div>
Quick review of the tutorial:</div>
<ul>
<li><strong>Key Take Away</strong>: Word Vectors = the output probabilities that relate how likely it is to find a vocabulary word nearby a given input word.</li>
<li><strong>Methods</strong>: ngrams for training, order doesn't matter</li>
<li><strong>Insight</strong>: If two words have similar contexts, then the network is motivated to learn similar word vectors for the two words.</li>
</ul>
<div>
<strong><u>Questions</u></strong></div>
<ul>
<li><strong>Why sentences?</strong></li>
<ul>
<li>When choosing word window size, word vectors respect sentence boundaries. Why?</li>
<ul>
<li>By doing this, vectors are modeling edited, written language.</li>
<li>Vectors are assuming the semantics of sentences are coherent in some way.</li>
<li>Vectors are, in essence, relying on sentences to be semantically coherent. As long as they are, the method works, but when they aren’t, how does this method break down?</li>
</ul>
<li>This is an assumption often broken in spoken language with dissfluencies, topic shifts, interruptions, etc</li>
<li>How else might spoken language models differ? </li>
<li>How might methods change to account for this</li>
<li>Do words from previous sentences have semantic relationships to the words in following sentences? (Yes. Because discourse.) </li>
<li>Why miss out on that information?</li>
</ul>
<li><strong>Why the middle?</strong></li>
<ul>
<li>Vectors use words “in the middle of a sentence”? Why? </li>
<li>Do words that tend to occur in the middle of sentences differ in meaningful ways than those that do not? (Possibly. E.g, verbs and prepositions rarely begin or end sentences. Information structures affects internal ordering of constituents in interesting ways).</li>
<li>Did they look at the rate of occurrences of each unique lexical entry in-the-middle and near the peripheries?</li>
</ul>
<li><strong>Synthetic Languages</strong>: </li>
<ul>
<li>How well would this word vectors work for a synthetic language like Mohawk?</li>
<li>What pre-processing steps might need to be added/modified?</li>
<li>Would those modifications be enough to get similar/useful results?</li>
</ul>
<li><strong>Antonyms</strong>: Can the same method learn the <u>difference</u> between “ant” and “ants”?</li>
<ul>
<li>Quote: “<em>the network will likely learn similar word vectors for the words “ant” and “ants”</em></li>
<li>I haven't read it yet, but these folks seem to use word vectors to learn antonyms: <a href="https://www.aclweb.org/anthology/N15-1100" target="_blank">Word Embedding-based Antonym Detection using Thesauri and Distributional Information</a>.</li>
</ul>
</ul>
This was a fun read. <br />
<br />
<br />
The original tutorial:<br />
McCormick, C. (2016, April 19). Word2Vec Tutorial - The Skip-Gram Model. Retrieved from <a href="http://www.mccormickml.com/">http://www.mccormickml.com</a><br />
<div>
</div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com3tag:blogger.com,1999:blog-520807396714463309.post-9312646126872705002017-09-14T18:47:00.004-04:002017-09-14T18:47:48.975-04:00Nuts and Bolts of Applying Deep Learning (Andrew Ng) I recently watched Andrew Ng's excellent lecture from 2016 <em>Nuts and Bolts of Applying Deep Learning</em> and took notes. I post them as a helpful resource for anyone who wants to watch the video.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/F1ka6a13S9I/0.jpg" src="https://www.youtube.com/embed/F1ka6a13S9I?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
I broke it into the following sections<br />
<ol>
<li>End-to-End DL for rich output</li>
<li>Buckets of DL</li>
<li>Bias and variance</li>
<li>Applied machine learning work flow</li>
<li>New Era of ML </li>
<li>Build a Unified Data Warehouse</li>
<li>The 70/30 Split Revisited</li>
<li>Comparing to Human Level Performance</li>
<li>How do you define human level performance?</li>
<li>How do you build a career in machine learning? </li>
<li>AI is the new electricity</li>
</ol>
<strong>Intro</strong><br />
<ol>
<li>End to end DL – work flow</li>
<li>Bias and variance has changed in era of deep learning</li>
<li>DL been around for decades, why do they work well now?</li>
<ul>
<li>Scale of data and computation</li>
<li>Two teams</li>
<ul>
<li>AI Teams </li>
<li>Systems team</li>
<li>Sit together</li>
<li>Difficult for any one human to be sufficiently expert in multiple fields</li>
</ul>
</ul>
</ol>
<strong>End-to-End DL for rich output</strong><br />
<ol><ul>
<li>From first three buckets below</li>
<li>Traditional ML models output real numbers</li>
<li>End-to-end DL can out put more complex things than numbers</li>
<ul>
<li>Sentence captions for images</li>
<li>Speech-to-text</li>
<li>Machine translation</li>
<li>Synthesize new images (13:00)</li>
</ul>
<li>End-to-End DL not the solution to everything.</li>
<ul>
<li>End-to-end = having just a DL between input and output</li>
<li>Rules for when to use (13:35)</li>
<ul>
<li>Old way: audio ------> phonemes --> transcript</li>
<li>New DL way: audio -----------------> transcript </li>
</ul>
<li>Makes for great PR, but only works some times (15:31)</li>
<li>Achilles heel – need lots of labeled data</li>
<li>Maybe phonemes are just a fantasy of linguists (15:48)</li>
<li>Advantage of old non-end-to-end architecture is it allows you to manually add more information to the processing (18:16)</li>
<li>Also, for self-driving cars, no one has enough data (right now) to make end-to-end work) (20:42)</li>
</ul>
<li>Common problem – after first round of dev, ML not working that well, what do you do next?</li>
<ul>
<li>Collect more data</li>
<li>Train longer</li>
<li>Different architecture (e.g., switch to NNs)</li>
<li>Regularization</li>
<li>Bigger model</li>
<li>More GPUs</li>
</ul>
<li>Skill in ML engineer is knowing how to make these decisions (22:33)</li>
</ul>
</ol>
<div>
<strong>Buckets of DL</strong></div>
<ol>
<li>General models</li>
<ul>
<li>Densely connected layers – FC</li>
<li>Sequence models – 1D (RNN, LSTM, GRU, attention)</li>
<li>Image models – 2D, 3D (Convo nets)</li>
<li>Other – unsupervised, reinforcement </li>
</ul>
<li>First three buckets driving market advances</li>
<li>But "Other" bucket is future of AI</li>
</ol>
<div>
</div>
<div>
<strong>Bias and variance – evolving</strong></div>
<ol>
<li>Scenario: build human level speech rec system</li>
<ul>
<li>Measure human level error – 1</li>
<li>Training set error – 5%</li>
<li>Dev set – 6%</li>
</ul>
<li>Bias = difference between human error level and your system’s</li>
<li>TIP: For bias problems try training a bigger model (25:21)</li>
<li>Variance (overfitting): if Human 1%, Training 2%, Dev 6%</li>
<li>TIP: for variance, try adding regularization, early stopping, best bet = more data</li>
<li>Both high bias and high variance: if Human 1%, Training 5%, Dev 10%</li>
<li>“<em>sucks for you</em>” (direct quote 26:30)</li>
</ol>
<div>
<strong>Applied machine learning work flow</strong></div>
<ol>
<li>Is your training error high</li>
<ul>
<li>Yes</li>
<ul>
<li>Bigger model</li>
<li>Train longer</li>
<li>New architecture</li>
<li>Repeat until doing well on training set</li>
</ul>
</ul>
<li>Is dev error high?</li>
<ul>
<li>Yes</li>
<ul>
<li>Add data</li>
<li>Regularization</li>
<li>New architecture</li>
<li>Repeat until doing well on training set</li>
</ul>
</ul>
<li>Done</li>
</ol>
<div>
<strong>New Era of ML</strong> </div>
<ol>
<li>We now know whatever problem you are facing (high bias or high variance) you have at least one action you can take to correct</li>
<li>No longer a bias/variance trade-off (29:47)</li>
<li>“Dumb” formula of bigger model/more data is easy to implement even for non-experts and is enough to do very well on a lot of problems (31:09)</li>
<li>More data has led to interesting investments</li>
<ul>
<li>Data synthesis - Growing area</li>
<li>Examples-</li>
<ul>
<li>OCR at Baidu</li>
<li>Take random image</li>
<li>Random word</li>
<li>Type random word in Microsoft Word</li>
<li>Use random font</li>
<li>You just created training data for OC</li>
<li>Still takes some human intervention, but lots of progress</li>
</ul>
<li>Speech recognition</li>
<ul>
<li>Take clean audio</li>
<li>Add random noise to background for more data</li>
<li>E.g., add car noise</li>
<li>Works remarkably well</li>
</ul>
<li>NLP</li>
<ul>
<li>Take ungrammatical sentences and auto-correct</li>
<li>Easy to create ungrammatical sentences programmatically</li>
</ul>
<li>Video games in RL</li>
</ul>
<li>Data synthesis has a lot of limits (36:24)</li>
<ul>
<li>Why not take cars from Grand Theft Auto and use that as training data for self-driving cars</li>
<li>20 cars in video game enough to give “realistic” impression to player</li>
<li>But 20 cars is very impoverished data set for self-driving cars</li>
</ul>
</ol>
<strong>Build a Unified Data Warehouse</strong><br />
<ol>
<li>Employees can be possessive of "their" data</li>
<li>Baidu- it’s not your data, it’s company data</li>
<li>Access rights can be a different issue</li>
<li>But warehouse everything together</li>
<li>Kaggle</li>
</ol>
<strong>The 70/30 Split Revisited</strong><br />
<ol>
<li>In academia, common for test/train to come from same distribution </li>
<li>But more ommon in industry for test and train to come from different distributions</li>
<ul>
<li>E.g., speech rec at Baid</li>
<ul>
<li>Speech enabled rear view mirror (in China)</li>
</ul>
<ul>
<li>50,000 hours of regular speech data</li>
</ul>
<ul>
<li>Data not from rear-view mirror interactions though</li>
</ul>
<ul>
<li>Collect another 10 hours of rear-view mirror scenario</li>
</ul>
<li>What do you do with the original 50,000 hours of not-quite right data?</li>
<ul>
<li>Old method would be to build a different model for each scenario</li>
</ul>
<ul>
<li>New era, one model for all data</li>
</ul>
<ul>
<li>Bad idea, split 50,000 into training/dev, use 10,000 as test. DON’T DO THIS.</li>
</ul>
<ul>
<li>TIP: Make sure dev and test are from same distro (boosts effectiveness)</li>
</ul>
<ul>
<li>Good Idea: make 50,000 train, split 10,000 into dev/test</li>
</ul>
<li>Dev set = problem specification</li>
<ul>
<li>Me: "dev set = problem you are trying to solve"</li>
</ul>
<li>Also, split off just 20 hours from 50,000 to create tiny “dev-train” set </li>
<ul>
<li>this has same distro as train</li>
</ul>
</ul>
<li>Mismatched train and dev set is problem that academia doesn’t work on much </li>
<ul>
<li>some work on domain adaptation, but not much (44:53)</li>
</ul>
<li>New architecture fix = “hail mary” (48:58)</li>
<li>Takes a long time to really grok bias/variance</li>
<ul>
<li>People who really understand bias/variance deeply are able to drive rapid progress in machine learning (50:33)</li>
</ul>
</ol>
<div>
<strong>Common Theme – Comparing to Human Level Performance</strong></div>
<ol>
<li>Common to achieve human level performance, then level off</li>
<li>Why?</li>
<ul>
<li>Audience: <em>Labels come from humans</em></li>
<li>Audience: <em>Researchers get satisfied with results</em> (the <em>laziness </em>hypothesis)</li>
<li>Andrew: theoretical limits (aka <strong><u>optimal error</u></strong> rate, Bayes rate)</li>
<ul>
<li>Some audio so bad, impossible to transcribe (phone call from a rock concert)</li>
<li>Some images so blurry, impossible to interpret</li>
</ul>
<li>Humans are really good at some things, so once you surpass human level accuracy, there’s not much room left to improve (54:38)</li>
</ul>
<li>While worse than humans, still ways to improve</li>
<ul>
<li>Get labels from humans</li>
<li>Error analysis</li>
<li>Estimate bias/variance effects</li>
</ul>
<li>For tasks that humans are bad at (say 30% error rate), really hard to find guidance on how to improve</li>
</ol>
<div>
<strong>How do you define human level performance?</strong></div>
<ol>
<li>Quiz: Which is the most useful definition? (101:000</li>
<ul>
<li>Example: Medical image reading</li>
<ol>
<li>Typical non-doctor error - 3%</li>
<li>Typical doctor – 1%</li>
<li>Expert doctor – 0.7%</li>
<li>Team of expert doctors – 0.5%</li>
</ol>
<li>Answer: Team of expert doctors is best because ideally you are using human performance to proxy <strong><u>optimal error</u></strong> rate.</li>
</ul>
</ol>
<div>
<strong>What can AI do?</strong> (106:30)</div>
<ol>
<li>Anything that a typical person can do in less than one second.</li>
<ul>
<li>E.g., Perception tasks</li>
<li>Audience: if a human can do it in less than a second, you can get a lot of data</li>
</ul>
</ol>
<strong>How do you build a career in machine learning</strong> (111:00)<br />
<ol>
<li>Andrew says he does not have a great answer (me: but he does have a good one)</li>
<ul>
<li>Taking a ML course</li>
<li>Attend DL school</li>
<li>Work on project yourself (Kaggle)</li>
<li>Mimic PhD student process</li>
<ul>
<li>Read a lot of papers (20+)</li>
<li>Replicate results</li>
</ul>
<li>Dirty work</li>
<ul>
<li>Downloading/cleaning data</li>
<li>Re-running someone’s code</li>
</ul>
<li>Don’t only do dirty work</li>
<li>PhD process + Dirty work = reliable</li>
<ul>
<li>Keep it up for a year</li>
<li>Competency </li>
</ul>
</ul>
</ol>
<div>
<strong>AI is the new electricity</strong> (118:00)</div>
<ol>
<li>Transforms industry after industry</li>
<li>Get in on the ground floor</li>
<li>NOTE: this is the title of his follow-up talk, which has a video link at the end of the one above.</li>
</ol>
<div>
</div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-35332114344982199922017-04-08T18:40:00.001-04:002017-04-08T18:53:29.629-04:00NLPers: How would you characterize your linguistics background? <div dir="ltr">
That was the poll question my hero Professor Emily Bender
posed on Twitter March 30th. 573 tweets later, a truly epic thread had
been created pitting some of the most influential NLPers in the world
head to head in a contest of wit and debate that is truly what the field
of NLP needed. Unfortunately, Twitter proves hapless at being able to
view the thread as a single conversation using their app. But, student
Sebastian Meilke used a Chrome extension called Treeverse to put
together the whole thread into a single, readable format, complete with
sections!</div>
<div dir="ltr">
<br />
If you are at all interested in NLP or linguistics, this is a must read: <a data-cke-saved-href="http://sjmielke.com/nlp-cl-megathread.htm" href="http://sjmielke.com/nlp-cl-megathread.htm">NLP/CL Twitter Megathrea</a>.</div>
<div dir="ltr">
<br />
I
would be remiss if I didn't note my own small role in this. Emily's
poll was sparked by my own Tweet where I said I was glad the Linguistics
Society of America is starting their own Society for Computational
Linguistics because "the existing CL and NLP communities have gotten
farther and farther off the linguistics path"</div>
<div dir="ltr">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-Ttfs5KixXk8/WOlmwcZh0vI/AAAAAAAAAyw/9iX-9Ka2v5QM_kIaTKVuSjYzVd2CewccQCLcB/s1600/emilys_list.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://2.bp.blogspot.com/-Ttfs5KixXk8/WOlmwcZh0vI/AAAAAAAAAyw/9iX-9Ka2v5QM_kIaTKVuSjYzVd2CewccQCLcB/s400/emilys_list.PNG" width="307" /></a></div>
<div dir="ltr">
</div>
<div dir="ltr">
Happy reading</div>
<div dir="ltr">
<br /></div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com6tag:blogger.com,1999:blog-520807396714463309.post-57489748189245939462017-03-25T11:01:00.002-04:002017-03-25T11:01:46.445-04:00Small Towns Do Tech Startups TooI'm participating in the Startup Weekend event in Chico CA sponsored by <a href="http://chicostart.com/" rel="nofollow noopener" target="_blank">Chicostart</a>.
Great first night. The 60 second pitches were awesome. So much variety
and interesting problems to solve. I pitched my own idea about using
Watson to identify fake news, but I only got two blue stickers, boo hoo. I am very impressed with the Build.com space that hosted us. They are
a multi-use space that includes a lot of tech startups. Kudos for
putting such a great state-of-the-art space in a small college town. <br />
<br />
<div class="prose" itemprop="articleBody">
I was particularly impressed with the range of pitchers. Millenials,
students, professionals, middle aged folks, and at least one senior. Not
bad for a small town with big dreams. <br />
<div class="separator" style="clear: both; text-align: center;">
<div style="margin-left: 1em; margin-right: 1em;">
<img height="300" src="https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAA3QAAAAJGNmYTM3YTllLTdlZDUtNDZmOS1iYmU3LTIyNGM3NDE3YmFhZg.jpg" width="400" />
</div>
<br />
<div class="slate-resizable-image-embed slate-image-embed__resize-full-width" data-imgsrc="https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAA3QAAAAJGNmYTM3YTllLTdlZDUtNDZmOS1iYmU3LTIyNGM3NDE3YmFhZg.jpg">
</div>
<div class="separator" style="clear: both; text-align: center;">
<div style="margin-left: 1em; margin-right: 1em;">
<img height="300" src="https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAmMAAAAJGE1YjIwNGZhLTRlNzgtNGU4Mi1iNjk1LTNjNWVhMmY1NjhmMg.jpg" width="400" />
</div>
<br />
<div class="slate-resizable-image-embed slate-image-embed__resize-full-width" data-imgsrc="https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAmMAAAAJGE1YjIwNGZhLTRlNzgtNGU4Mi1iNjk1LTNjNWVhMmY1NjhmMg.jpg">
</div>
<div class="separator" style="clear: both; text-align: center;">
<div style="margin-left: 1em; margin-right: 1em;">
<img height="300" src="https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAptAAAAJGU2NDYyMWJhLTI5NTktNDY1MS04Mjg5LTAwN2M0NjNmZmQ1Nw.jpg" width="400" />
</div>
<br />
<div class="slate-resizable-image-embed slate-image-embed__resize-full-width" data-imgsrc="https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAptAAAAJGU2NDYyMWJhLTI5NTktNDY1MS04Mjg5LTAwN2M0NjNmZmQ1Nw.jpg">
</div>
<div style="text-align: left;">
FYI - I'll be helping build a startup this weekend focused on smart housing devices. Will be a lot of fun. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-zk31SwJ7FqA/WNaGC2kz87I/AAAAAAAAAyY/fgY02AreGDQNFY4st1AlTFS3qYtSU1pHQCLcB/s1600/IMG_2782.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="226" src="https://2.bp.blogspot.com/-zk31SwJ7FqA/WNaGC2kz87I/AAAAAAAAAyY/fgY02AreGDQNFY4st1AlTFS3qYtSU1pHQCLcB/s320/IMG_2782.jpg" width="320" /></a></div>
</div>
</div>
</div>
</div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com1tag:blogger.com,1999:blog-520807396714463309.post-33646462973223141412017-03-24T17:02:00.001-04:002017-03-24T17:02:34.759-04:00Chico Startup WeekendI'm looking forward to participating in <a href="http://www.up.co/communities/usa/chico/startup-weekend/9954" target="_blank">Chico Startup Weekend</a>, sponsored by <a href="http://chicostart.com/" target="_blank">Chicostart</a>. I can't say what the weekend holds, but it's nice to know there's this kind of energy and drive in small town America. Startups ain't just for Austin and Mountain View. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-BFJiU3cpU-g/WNWJQPTs_gI/AAAAAAAAAxg/SjPcHMZ6ZSU0Gp_-eatxCyPQJEaxV52TwCLcB/s1600/chico_startup_weekend.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="136" src="https://4.bp.blogspot.com/-BFJiU3cpU-g/WNWJQPTs_gI/AAAAAAAAAxg/SjPcHMZ6ZSU0Gp_-eatxCyPQJEaxV52TwCLcB/s320/chico_startup_weekend.PNG" width="320" /></a></div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-89152006593230072862017-03-20T01:01:00.000-04:002017-03-22T15:11:45.972-04:00Using IBM Watson Knowledge Studio to Train Machine Learning ModelsUsing the Free Trial version of IBM's <a href="https://www.ibm.com/us-en/marketplace/supervised-machine-learning" target="_blank">Watson Knowledge Studio</a>, I just annotated a text and created a machine learning model in about 3 hours without writing a single line of code. The mantra of WKS is that you don't program Watson, you teach Watson.<br />
<br />
For demo purposes I chose to identify personal relationships in Shirley Jackson's 1948 short story <a href="https://en.wikipedia.org/wiki/The_Lottery" target="_blank">The Lottery</a>. This is a haunting story about a small village and its mindless adherence to an old, and tragic tradition. I chose it because 1) it's short and 2) it has clear person relationships like brothers, sisters, mothers, and fathers. I added a few other relations like AGENT_OF (which amounts to subjects of verbs) and X_INSIDE_Y for things like pieces of paper inside a box.<br />
<blockquote class="tr_bq">
<strong><span style="color: blue;">Caveat</span></strong>: This short story is <em><strong>really</strong></em> short: 3300 words. So I had no high hopes of getting a <em><strong>good</strong></em> model out of this. I just wanted to go through an entire machine learning work flow from gathering text data to outputting a complete model <strong><u>without</u></strong> writing a single line of code. And that's just what I did. </blockquote>
<br />
<br />
<strong><u>WORK FLOW</u></strong><br />
I spent about 30 minutes prepping the data. E.g., I broke it into 20 small snippets (to facilitate test/train split later), I also edited some quotation issues, spelling, etc). <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-1q4Oa9K_AYc/WM9ek8c2iqI/AAAAAAAAAxI/R-x-zF3EOkAlqJ8Nmdb8SL6tk09873KHgCEw/s1600/documents.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://1.bp.blogspot.com/-1q4Oa9K_AYc/WM9ek8c2iqI/AAAAAAAAAxI/R-x-zF3EOkAlqJ8Nmdb8SL6tk09873KHgCEw/s320/documents.PNG" width="129" /></a></div>
It uploaded into WKS in seconds (by simply dragging and dropping the files into the browser tool). I then created a Type System to include entity types such as these:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-TEFQi4vnDXw/WM9W4Rl-spI/AAAAAAAAAv4/q42vpSPwkMs_hxzxRcXB9MLaoNTRBMtVQCLcB/s1600/entity_types.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="https://2.bp.blogspot.com/-TEFQi4vnDXw/WM9W4Rl-spI/AAAAAAAAAv4/q42vpSPwkMs_hxzxRcXB9MLaoNTRBMtVQCLcB/s400/entity_types.PNG" width="400" /></a></div>
<br />
And relation types such as these:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-fWEqK-Z7oBo/WM9XBkM1fPI/AAAAAAAAAv8/yV9u2-503moJ33frEqrjUrSpHJ-3nBr0QCLcB/s1600/relation_types.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://4.bp.blogspot.com/-fWEqK-Z7oBo/WM9XBkM1fPI/AAAAAAAAAv8/yV9u2-503moJ33frEqrjUrSpHJ-3nBr0QCLcB/s400/relation_types.PNG" width="363" /></a></div>
I then annotated the 20 short documents in less than two hours (as is so often the case, I re-designed my type system several times along the way; luckily WKS allows me to do this fluidly without having to re-annotate). <br />
<br />
Here's a shot of my entity annotations:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-uLuqd9bE1E0/WM9X7IFsdqI/AAAAAAAAAwE/JPFMe50Gc_Eii285iDivtpQMSms5nv_WQCLcB/s1600/entity_annotations.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="157" src="https://4.bp.blogspot.com/-uLuqd9bE1E0/WM9X7IFsdqI/AAAAAAAAAwE/JPFMe50Gc_Eii285iDivtpQMSms5nv_WQCLcB/s400/entity_annotations.PNG" width="400" /></a></div>
<br />
Here's a shot of my relation annotations:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-O2n1kgCyGUY/WM9YBj3dFqI/AAAAAAAAAwI/E7ZNr0O7Zd0wGOkLSH5zFBDD6vw_vbdNwCLcB/s1600/relations_annotations.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="185" src="https://3.bp.blogspot.com/-O2n1kgCyGUY/WM9YBj3dFqI/AAAAAAAAAwI/E7ZNr0O7Zd0wGOkLSH5zFBDD6vw_vbdNwCLcB/s400/relations_annotations.PNG" width="400" /></a></div>
<br />
I then used these manually annotated documents as ground truth to teach a machine learning model to recognize the relationships automatically using a set of linguistic features (character and word ngrams, parts-of-speech, syntactic parses, etc). I accepted the WKS suggested split of documents as 70/23/2:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-GXWBbpZEiLU/WM9ZBWekSLI/AAAAAAAAAwU/gBFqqv8dwBUsUuxZHjmcyTuDv2VD_Bl3wCLcB/s1600/train_test.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="135" src="https://3.bp.blogspot.com/-GXWBbpZEiLU/WM9ZBWekSLI/AAAAAAAAAwU/gBFqqv8dwBUsUuxZHjmcyTuDv2VD_Bl3wCLcB/s400/train_test.PNG" width="400" /></a></div>
I clicked "Run" and waited:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Lb6wG1FOnUw/WM9ZZ7FSy4I/AAAAAAAAAwY/oKimx0xF8K0ubPSx0SVgSCZZjObFGu9EACLcB/s1600/model_run.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://1.bp.blogspot.com/-Lb6wG1FOnUw/WM9ZZ7FSy4I/AAAAAAAAAwY/oKimx0xF8K0ubPSx0SVgSCZZjObFGu9EACLcB/s400/model_run.PNG" width="284" /></a></div>
<br />
The model was trained and evaluated in about ten minutes. Here's how it performed on entity types:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-0xoLWyZ1FHY/WM9bpNOquAI/AAAAAAAAAwo/nvK6hCHSqh80yIj6blprAukwOr-a_mADACEw/s1600/entity_type_stats.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="273" src="https://1.bp.blogspot.com/-0xoLWyZ1FHY/WM9bpNOquAI/AAAAAAAAAwo/nvK6hCHSqh80yIj6blprAukwOr-a_mADACEw/s400/entity_type_stats.PNG" width="400" /></a></div>
<br />
And here's how it performed on relation types:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-1DcIr0BMW1I/WM9b-WepLVI/AAAAAAAAAww/SWQxxx-nq8wtIdDchU2WRM4rpJ8KK7obgCLcB/s1600/relation_types_stats.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="270" src="https://2.bp.blogspot.com/-1DcIr0BMW1I/WM9b-WepLVI/AAAAAAAAAww/SWQxxx-nq8wtIdDchU2WRM4rpJ8KK7obgCLcB/s400/relation_types_stats.PNG" width="400" /></a></div>
<br />
This is actually not bad given how sparse the data is. I mean, an F1 of .33 on X_INSIDE_Y from only 29 training examples on a first pass. I'll take it, especially since that one is not necessarily obvious from the text. Here's one example of the X_INSIDE_Y relation:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-MrvznVPTREs/WM9dm8Zo98I/AAAAAAAAAxA/czGczbk5TgYnGZAplGdpl6RRRQkZCUEEACLcB/s1600/x_inside_y_example.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="77" src="https://4.bp.blogspot.com/-MrvznVPTREs/WM9dm8Zo98I/AAAAAAAAAxA/czGczbk5TgYnGZAplGdpl6RRRQkZCUEEACLcB/s400/x_inside_y_example.PNG" width="400" /></a></div>
<br />
So I was able to train a model with 11 entity types and 81 relation types on a small corpus in less than three hours start to finish without writing a single line of code. I did not program Watson. I taught itChrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com3tag:blogger.com,1999:blog-520807396714463309.post-54859524984412401492017-03-09T10:44:00.001-05:002017-03-09T10:44:47.238-05:00Annotate texts and create learning modelsIBM Watson has released a free trial version of their online <a href="https://www.ibm.com/us-en/marketplace/supervised-machine-learning" target="_blank">Watson Knowledge Studio</a> tool. This is one of the tools I'm most excited about because it brings linguistic annotation, rule writing, dictionary creation, and machine learning together in a single user-friendly interface designed for non engineers to use. <br />
<br />
<div style="text-align: center;">
<a href="https://www.ibm.com/us-en/marketplace/supervised-machine-learning" target="_blank">Watson Knowledge Studio</a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-XcEYMRoJr24/WMF3i9H7i6I/AAAAAAAAAvM/rdLMkXjXJ6Ikpcl1pVBDZCxSEBzT3lo9gCLcB/s1600/wks.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="226" src="https://4.bp.blogspot.com/-XcEYMRoJr24/WMF3i9H7i6I/AAAAAAAAAvM/rdLMkXjXJ6Ikpcl1pVBDZCxSEBzT3lo9gCLcB/s400/wks.PNG" width="400" /></a></div>
<br />
This tool allows users to annotate documents with mentions, relations, and coreference, then learn a linguistic model of their environments. With zero computer science background. I've trained subject matter experts in several fields to use the WKS and I'm genuinely impressed. I'll put together a demo and post it sometime this weekend. Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-83272237437466045882016-10-14T14:43:00.001-04:002016-10-14T14:43:30.784-04:00one of the best preschools in chicoMy sister's preschool, <a href="https://kidsfirstpreschool.com/">Kids First Learning Center</a>, was voted one of <a href="https://kidsfirstpreschool.com/2016/10/14/voted-third-place-in-best-of-chico-chico-news-review/">the best prescools in Chico</a> in the local News and Review Best of Chico contest. Very coolChrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-9132275164824025742016-09-25T16:32:00.002-04:002016-09-25T16:32:15.180-04:00the value of play for preschool children A nice article came up in The Atlantic about the comeback of recess and play in elementary schools. My sister posted some thought about how she encourages play when she teaches her preschool kids: <a href="https://kidsfirstpreschool.com/2016/09/25/for-our-little-chico-preschool-play-and-education-are-two-sides-of-the-same-coin/">For our little Chico preschool, play and education are two sides of the same coin</a>.Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-77370579879834787172016-09-23T09:55:00.001-04:002016-09-23T10:02:02.367-04:00from preschool to scholar athleteOne of the most endearing things about being a teacher is seeing former students go on to achieve great things. My <a href="https://kidsfirstpreschool.com/2016/09/23/our-preschoolers-go-on-to-great-things/" target="_blank">sister's preschool</a> student Jack Emanuel has been named a Subway Scholar athlete. Very cool. Congratulations Jack.Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-50611731744196266132016-09-05T19:23:00.000-04:002016-09-23T09:50:53.190-04:00some thoughts on data analytics for a micro businessMy first thought as a new business owner trying to utilize data analytics is this: <b><i>how empty this all seems</i></b>. I don't need to <i>understand </i>my market right now, I need to break into it. How can I use data analytics to open doors? I don't have time for nuance right now, I need sign ups. I'm faced with one of the most basic problems in business: getting noticed. And I wanna do it on the cheap.<br />
<br />
Let me stipulate that this post relates only to a small slice of DA proper: low-cost advertising DA available from Google & Facebook.<br />
<br />
My sister just relocated her <a href="https://kidsfirstpreschool.com/" target="_blank">preschool, Kids First</a>, to a new town, Chico CA, and I'm helping her buy a domain, set up a website, and do some promotion using Google AdWords Express, Facebook ads, and some local print advertising.<br />
<br />
This is the first time I've been on <u>this</u> side of the data analytics equation: as a <u>consumer</u> of the analysis, not a producer. And my experience is telling.<br />
<br />
<i>Business model</i>: We're not a small business. We're a <b>micro business</b>. Employees = 1 (Miss Lori). We don't need to sell a million widgets or gain a million likes. The business model of our preschool remains very old fashioned: get ten kids signed up, average that head count annually, and we're successful.<br />
<br />
<i>The market</i>: Chico is a college town with 85,000 people in the city, about 200,000 in the larger area. There is a state university, a junior college, two hospitals, two high schools, two middle schools, and several elementary schools, as well as a robust farming economy (all employing professionals with kids, right?).<br />
<br />
<i>Advertising</i>: Bought a small print ad in the local weekly for four weeks ($200/wk). One-time ad in the university student paper for the first week of the semester (hoping to grab the attention of new faculty who might finger through it out of curiosity). Small Facebook ad ($50), small Google AdWords campaign ($120/month). My sister "boosted" the <a href="https://www.facebook.com/misslorisbigkidpreschool/" target="_blank">preschool's Facebook page</a> for a few days for $20. <br />
<br />
<i>Data Analytics</i>: FB ad = zero calls. FB said 6,322 people saw her boosted virtual tour video and 411 people clicked on it in just a few days (I'm suspicious of these numbers because Lori said she limited the ad to Chico. I would be shocked if that many Chicoans clicked her video in a couple days). Zero calls. I started the AdWords campaign on September 3. I limited the area to the Chico region (those 200k people) because I don't want to pay for clicks from people who are looking for preschools in San Francisco. We've gotten 4 clicks in about 48 hours. Zero calls.<br />
<br />
<a href="http://4.bp.blogspot.com/-0kBFTmAh94U/V83nI_49ztI/AAAAAAAAAuo/u31TP7k1oM0Buv0jPmaJnpp-JA64s_jsACK4B/s1600/adWordsSep5.PNG" imageanchor="1"><img border="0" height="175" src="https://4.bp.blogspot.com/-0kBFTmAh94U/V83nI_49ztI/AAAAAAAAAuo/u31TP7k1oM0Buv0jPmaJnpp-JA64s_jsACK4B/s400/adWordsSep5.PNG" width="400" /></a><br />
<br />
This is early, of course, but still, this all seems so empty because of the nature of our business. We don't need clicks or views or likes: we need parents to sign their kids up. There is a disconnect to me between the old fashioned, brick-and-mortar business reality of trying to profitably run a preschool and the virtual reality of these meager data analytics. The NLPer in me likes the fact that AdWords shows me which search words drive clicks, but the business owner in me says, where's the beef? I need a parent to pay me money. Don't care 'bout no clicks.<br />
<br />
I'm not in the business of advertising DA, but I have some appreciation for the tools and techniques. And even I'm frustrated trying to connect the dots. Or more to the point, I don't know how to use data analytics to go from clicks to phone calls (to actual sign ups).<br />
<br />
I get that using DA may prove useful in the long run, but first we need old fashioned visibility. Without that, nuance is worthless. I can only imagine how frustrated and perhaps infuriated many small business owners are at this same disconnect between their business models and the services offered. This experience has already deepened my appreciation for the customer experience in the DA ecosystem.<br />
<br />
But ultimately what I think is this: Data analytics, schmata analytics. I don't need a data scientist. I need Don Draper.Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com3tag:blogger.com,1999:blog-520807396714463309.post-37074666755274665262016-09-01T15:17:00.001-04:002016-09-01T15:17:32.754-04:00Newest and Greatest Preschool in ChicoMy sister has just re-located her preschool to Chico, CA. <a href="https://kidsfirstpreschool.com/" target="_blank">Kids First Learning Center</a>.<br />
<br />
<a href="http://1.bp.blogspot.com/-63pSwLz12dk/V8h-nWihlzI/AAAAAAAAAuU/p77qLWSjy9YIsEWgq8LNFxfLLxNyIj8lwCK4B/s1600/KF%2BSIGN%2BPIC.jpg" imageanchor="1"><img border="0" height="180" src="https://1.bp.blogspot.com/-63pSwLz12dk/V8h-nWihlzI/AAAAAAAAAuU/p77qLWSjy9YIsEWgq8LNFxfLLxNyIj8lwCK4B/s320/KF%2BSIGN%2BPIC.jpg" width="320" /></a><br />
<br />
<span id="goog_446084502"></span><span id="goog_446084503"></span><br />Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-1667274295488414022016-08-15T17:27:00.000-04:002016-08-15T17:27:00.185-04:00Yet Another Bad review of Suicide SquadThere are lots of reviews of Suicide Squad detailing how bad it is. This is another one. This is less a movie than a series of loosely related scenes. It had roughly three sections:<br />
<blockquote class="tr_bq">
A) <b>The set-up</b>: Character introductions. The movie begins with an amateurish method of introducing the characters that is literally one person listing their names and features followed by a flashback scene for each. Dumbest exposition structure ever. </blockquote>
<blockquote class="tr_bq">
B) <b>The Mission</b>: Load everybody up in a helicopter, give them weapons, send them along their way. This plays out as an extension of A where each person gets a montage playing with their toys of choice. </blockquote>
<blockquote class="tr_bq">
C) <b>Switcheroo</b>: Plot twist that changes the mission objective. Unfortunately, the movie fails to adequately set up this core plot point because the director spent so much time showing off weapons and Margot Robbie's arse, he forgot to have a character explain the mission in any memorable way. When the twist in the mission objective is "revealed" midway into the siege, it's more confusing than revelation. At this point, no sane person is looking for coherence in the film anyway, so it hardly matters.</blockquote>
The directorial style can best be summed up as '<i>just keep everyone shooting guns, no one will notice the incoherence</i>'.Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-52075303678040066932016-07-11T12:53:00.000-04:002016-07-11T12:53:05.442-04:00Fun with Stanford's online demoI've long been a fan of Stanford's online parser demo, but now they've
outdone themselves with a demo page for their CoreNLP tools. Not only
does it take your text and show the parse and entities, it also lets you
develop a regex to capture your input text, including semantic regexes!<br />
<br />
This is just plain fun: <a href="http://corenlp.run/">http://corenlp.run/ </a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-qDi7p2kZH-U/V4POxiexwDI/AAAAAAAAAt4/lSjt6sWsRTgTQoOC-AsTa2RQr9HwDj20gCK4B/s1600/stanfordNLP.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="https://1.bp.blogspot.com/-qDi7p2kZH-U/V4POxiexwDI/AAAAAAAAAt4/lSjt6sWsRTgTQoOC-AsTa2RQr9HwDj20gCK4B/s320/stanfordNLP.png" width="320" /></a></div>
<br />
<br />Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-64748766890813235582016-06-19T21:04:00.000-04:002016-06-19T21:04:25.899-04:00IBM Watson at NAACL 2016There were several Twitter NLP flare-ups recently triggered by the contrast between academic NLP and industry NLP. I'm not going to re-litigate those arguments, but I will note that one IBM Watson question answering team anticipated this very thing in their current NAACL paper for the NAACL HLT 2016 Workshop on <a href="https://sites.google.com/a/colorado.edu/2016-naacl-ws-human-computer-qa/" target="_blank">Human-Computer Question Answering</a>.<br />
<br />
The paper is titled <a href="http://aclweb.org/anthology/W16-0101" target="_blank">Watson Discovery Advisor: Question-answering in an industrial setting</a>.<br />
<br />
<b>The Abstract</b><br />
<blockquote class="tr_bq">
<i>This work discusses a mix of challenges arising from Watson Discovery Advisor (WDA), an industrial strength descendant of the Watson Jeopardy! Question Answering system currently used in production in industry settings. Typical challenges include generation of appropriate training questions, adaptation to new industry domains, and iterative improvement of the system through manual error analyses</i>.</blockquote>
The paper's topic is not surprising given that four of the authors are PhDs (Charley, Graham, Allen, and Kristen). Hence, it was largely a group of fishes out of water: they had an academic bent, but are daily wrestling with the real-word challenges of paying-customers and very messy data.<br />
<br />
Here are five take-aways:<br />
<br />
<ol>
<li>Real-world questions and answers are far more ambiguous and domain-specific than academic training sets.</li>
<li>Domain tuning involves far more than just retraining ML models.</li>
<li>Useful error analysis requires deep dives into specific QA failures (as opposed to broad statistical generalizations).</li>
<li>Defining what counts as an error is itself embedded in the context of the customer's needs and the domain data. What counts as an error to one customer may be acceptable to another.</li>
<li>Quiz-Bowl evaluations are highly constrained, special-cases of general QA, a point I made in 2014 <a href="http://thelousylinguist.blogspot.com/2014/09/neural-nets-and-question-answering.html" target="_blank">here </a>(pats self on back). Their lesson's learned are of little value to the industry QA world (for now, at least).</li>
</ol>
<br />
I do hope you will read the brief paper in full (as well as the other excellent papers in the workshop).Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-14648130113364156172016-01-25T15:09:00.000-05:002016-01-26T13:33:43.553-05:00Genetic Mutation, Thick Data, and Human IntuitionThere are two stories trending heavily in my social network sites that are
seemingly unrelated, yet they share one obvious conclusion: the value of human
intuition in finding needles in big data haystacks. Reading them highlighted to me the
special role humans must still can play in the emerging 21<sup>st</sup> century world of
big data.<br />
<br />
In the first story, <a href="http://www.theatlantic.com/health/archive/2016/01/genetic-mutation-patient-diagnosis-priscilla-lopes-schliep/424662/">The Patient Who Diagnosed Her Own Genetic Mutation—and an Olympic Athlete's</a>,
a woman with muscular dystrophy sees a photo of an Olympic sprinter’s bulging
muscles and thinks to herself, “she has the same condition I do.” What in the
world would cause her to think that? There is no pattern in the data that would
suggest this. The story is accompanied by a startling picture of two women who,
at first glance, look nothing alike. But once guided by the needle in the
haystack that this woman saw, a similarity is illuminated and eventually a
connection is made between two medically disparate facts that, once combined,
opened a new path of inquiry into muscle growth and dystrophy that is now a
productive area of research. Mind you, no new chemical compound was discovered.
No new technique or method that allowed scientists to see something that
couldn’t be seen before was built. Nope. Nothing *new* came into being, but
rather a connection was found between two things that all the world’s experts
never saw before. One epiphany by a human being looking for a needle in a
haystack. And she found it.<br />
<br />
In the second story, <a href="https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7#.vin62xd38">Why Big Data Needs Thick Data</a>,
an anthropologist working closely to understand the user stories of just 100
Motorola cases discovers a pattern that Motorola’s own big data efforts missed.
How? Because his case-study approach emphasized context. Money quote:<br />
<blockquote class="tr_bq">
<em>For Big Data to be analyzable, it must use normalizing, standardizing,
defining, clustering, all processes that strips the the data set of context,
meaning, and stories. Thick Data can rescue Big Data from the context-loss that
comes with the processes of making it usable.</em></blockquote>
Traditional machine learning techniques are designed to find large patterns
in big data, but those same techniques fail to address the needle in the
haystack problem. This is where humans and intuition truly stands
apart. Both of these articles are well worth reading in the context of
discovering the gaps in current data analysis techniques that humans must fill.<br />
<br />
UPDATE: Here's a third story making a similar point. a human being using an automatically culled dictionary noticed a misogynist tendency in the examples it provided. <a href="https://debuk.wordpress.com/2016/01/26/a-rabid-feminist-writes/" target="_blank">A rabid feminist writes</a>…<br />
<br />
And here's a fourth: <a href="https://hbr.org/2016/01/algorithms-need-managers-too?utm_campaign=harvardbiz&utm_source=twitter&utm_medium=social" target="_blank">Algorithms Need Managers, Too</a>. Money quote: "<i>Google’s hard goal of maximizing clicks on ads had led to a situation in which its algorithms, refined through feedback over time, were in effect defaming people with certain kinds of names</i>."Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com3tag:blogger.com,1999:blog-520807396714463309.post-9391398521968352682016-01-10T17:19:00.000-05:002016-01-10T17:31:49.297-05:00Advice for linguistics grad students entering industryAt the LSA mixer yesterday I had the chance to chat with a dozen or so grad students in linguistics who were interested non-academic jobs. Here I'll note some of the recurring themes and advice I gave.<br />
<br />
<b><span style="color: blue;">The First Job</span></b><br />
<span style="color: red;"><b>Advice</b></span>: Be on the look-out and know what a good opportunity looks like.<br />
<br />
Most students were very interested in the jump. How do you make that first transition from academics to industry? In general, you need to be in the market, actively looking, actively promoting yourself as a candidate. For me, it was a random posting on The Linguist List that caught my eye. In the summer of 2004 I was a bored ABD grad student. I knew I wasn't going to be competitive for academic jobs at that point, so I checked The Linguist List job board daily. One day I saw a posting from a small consulting company. They were looking for a linguist to help them create translation complexity metrics. They listed every sub-genre in linguists as their requirements. This told me they really didn't know what they wanted. I saw that as an opportunity because I could sweep in and help them understand what they needed. I applied and after several phone calls I was asked to create a proposal for their customer. I had a conference call to discuss the proposal (I was in shorts and a t-shirt in an empty lab during the call, but they didn't know that). Long story short, I got the job*, moved to DC and spent about two years working as a consultant on that and other government contracts. That first job was a big step in moving into industry. I had very impressive clients, a skill set that was rare in the market, and a well defined deliverable that I could point to as a success.<br />
<br />
<br />
<b><span style="color: blue;">Visibility</span></b><br />
<b><span style="color: red;">Advice</span></b>: Make recruiters come to you. Maintain a robust LinkedIn profile and be active on the site on a weekly basis (so that recruiters will find you).<br />
<br />
Several students wondered if LinkedIn was considered legitimate. I believe it's fair to say that within the tech and NLP world, LinkedIn is very much legit. My <a href="https://www.linkedin.com/in/christopher-phipps-13124b2" target="_blank">LinkedIn profile</a> has been crucial to being recruited for multiple jobs, two of which I accepted. Algorithms are constantly searching this site for all kinds of jobs. In fact, most of the really good jobs for linguists are not posted on job sites, but rather are filled only by recruiter. So you need strategies for waving your flag and getting them to come to you. In the DC area, there are excellent opportunities for linguists at DARPA, CASL, IARPA, NIST, MITRE and RAND, and many other FFRDCs (federally funded research and development centers), but they rarely post these to jobs boards. You need them to find you. A good LinkedIn page is a great way to increase your visibility.<br />
<br />
Another way to increase your visibility is to go public with your projects. You can always blog descriptions and analysis. For computer science students, a <a href="https://github.com/" target="_blank">GitHub </a>account is virtually a requirement. I think linguists should follow their lead. You most likely write little scripts anyway. Maybe an R script to do some analysis, or a Python script to extract some data. Put those up on GitHub with a little README document. That's an easy place for tech companies to see your work. Also, if you have created data sets that you can freely distribute, put those up on GitHub too. I also recommend competing in a <a href="https://www.kaggle.com/competitions" target="_blank">Kaggle </a>competition. Kaggle sponsors many machine learning competitions. They provide data, set the requirements, and post results. It's a great way to both practice a little NLP and data science, and also increase your visibility (and put your Kaggle competitions on your resume!). here are two linguistically intriguing Kaggle competitions ready for you right now: <a href="https://www.kaggle.com/kaggle/hillary-clinton-emails" target="_blank">Hillary Clinton's Emails</a> (think about the many things you could analyze in those!); <a href="https://www.kaggle.com/benhamner/nips-2015-papers" target="_blank">NIPS 2015 Papers</a> (how can a linguist characterize a bunch of machine learning papers?).<br />
<br />
Have you managed to automate a process that you once did manually (either through an R script, or maybe Excel formulas), write that up on a blog post. Automating manual processes is huge in industry. You know the messy nature of language data better than anyone else, so write some blog posts describing the kind of messiness you see and what you do about it. That's gold.<br />
<br />
<br />
<b><span style="color: blue;">Resume</span></b><br />
<span style="color: red;"><b>Advice</b></span>: List tools and data sets. Do you use Praat? List that. Do you use the Buckeye Corpus? List that. Make it clear that you have experience with tools and data management. Those are two areas where tech companies always have work to perform, so make it clear that you can perform that work.<br />
<br />
<br />
<br />
*FYI, here's what the deal was with that first consultant job: The FBI tests lots of people as potential translators. So, for example, they will give a native speaker of Vietnamese several passages of Vietnamese writing, one that is simple, one that is medium complex, and one that is complex); then the applicant is asked to translate the passages into English. the FBI grades each translation. The problem was that the FBI didn't have a standardized metric for what counted as a complex passage in Vietnamese (or the many many other languages that they hire translators for). They relied on experienced translators to recommend passages from work they had done in the past. Turns out, that was a lousy way to find example passages. The actual complexity of passages was wildly uneven, and there was no consistency across languages.Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com6tag:blogger.com,1999:blog-520807396714463309.post-70190908187596088742016-01-07T18:52:00.000-05:002016-01-07T18:57:04.335-05:00LSA 2016 Evening RecomendationsWith the LSA's annual convention officially underway, I've thrown together a list of a few restaurants and bars within a short walking distance of the convention center that grad students and attendees might want to enjoy. My walking estimates assume you are standing in front of the convention center.<br />
<br />
<a href="http://www.busboysandpoets.com/" target="_blank">Busboys and Poets</a> (4 blocks west at 5th & K) - A DC Institution. You will not be forgiven if you do not make at least one pilgrimage here.<br />
<br />
<a href="http://maddystaproom.com/" target="_blank">Maddy’s Taproom</a> (4 blocks east at 13th & L) - Good beer selection.<br />
<br />
<a href="http://www.lovethebeer.com/" target="_blank">RFD Washington</a> (4 blocks south at 7th & H) - Large bottled beer selection, good draft beer selection (food ain't that great).<br />
<br />
<a href="http://churchkeydc.com/" target="_blank">Churchkey </a>(6 blocks northeast at 14th & Rhode Island) - Officially, one of the best beer rooms in the US.<br />
<br />
<a href="http://www.stansrestaurant.com/" target="_blank">Stan's Restaurant</a> (7 blocks east at L & Vermont) - Downstairs, casual. very strong drinks. Supposedly good wings (I'm a vegetarian, so I hold no opinion)<br />
<br />
<a href="http://daikaya.com/" target="_blank">Daikaya - Ramen - Izakaya</a> (7 blocks Southwest at 6th & G) - Upstairs bar can be easier to get into sometimes. It's a popular place.<br />
<br />
<a href="https://www.teaism.com/restaurant-details-40.html" target="_blank">Teaism</a>, Penn Quarter (8 blocks south at 8th & G) - Great snack place mid-way to the national Mall. Large downstairs dining area. great place to have some tea, a snack, and catch up on conference planning.<br />
<br />
There are, of course, lots of other places within a short walk. I recommend 14th street in general. 9th street has some good stuff, especially as you get closer to U, but it's a little sketchy of a walk.<br />
<br />
<br />
<div>
<br /></div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-59532473781471074882015-11-29T11:38:00.001-05:002015-11-29T16:23:14.553-05:00online psycholinguistics demos 2015I was asked recently about an <a href="http://thelousylinguist.blogspot.com/2008/07/online-psycholinguistics-experiments.html" target="_blank">old post from 2008</a> that listed a variety of online psycholinguistics demos. All of the links are dead now, so I was asked if I knew of any updated ones. This is what I can find. Any suggestions would be welcomed.<br />
<br />
<ul>
<li>Potsdam Research Institute for Multilingualism - Links to <a href="http://www.uni-potsdam.de/prim/research/resources.html" target="_blank">Psycholinguistics Sites & Resources</a></li>
</ul>
<ul>
<li>Harvard <a href="https://implicit.harvard.edu/implicit/featuredtask.html" target="_blank">Implicit Associations Task</a>: <i>Project Implicit is a non-profit organization and international collaboration between researchers who are interested in implicit social cognition - thoughts and feelings outside of conscious awareness and control. The goal of the organization is to educate the public about hidden biases and to provide a “virtual laboratory” for collecting data on the Internet.</i></li>
</ul>
<ul>
<li><a href="https://code.google.com/p/webspr/" target="_blank">webspr </a><i>- Conduct psycholinguistic experiments (e.g. self-paced reading and speeded acceptability judgment tasks) remotely using a web interface</i></li>
</ul>
<ul>
<li><a href="http://gameswithwords.org/#sthash.84zDBR10.dpuf http://gameswithwords.org/" target="_blank">Games With Words</a><i>: Learn about language and about yourself while advancing cutting-edge science. How good is your language sense?</i></li>
</ul>
<ul>
<li><a href="http://www.psytoolkit.org/experiment-library/ldt.html" target="_blank">Lexical Decision Task</a> demo<i>: In a lexical decision task (LDT), a participant needs to make a decision about whether combinations of letters are words or not. For example, when you see the word "GIRL", you respond "yes, this is a real English word", but when you see the letters "XLFFE" you respond "No, this is not a real English word".</i></li>
</ul>
<ul>
<li><a href="http://psych.hanover.edu/Research/exponnet.html" target="_blank">Psychological Research on the Net</a>:<i> links seem to be updated regularly, but not a lot on linguistics.</i></li>
</ul>
<ul>
<li><a href="http://www.ling.gu.se/~anders/KatPer/Applet/index.eng.html" style="font-style: normal;" target="_blank">Categorical Perception</a><span style="font-style: normal;">: </span><i>Categorical perception means that a change in some variable along a continuum is perceived, not as gradual but as instances of discrete categories. The test presented here is a classical demonstration of categorical perception for a certain type of speech-like stimuli</i>.</li>
</ul>
Paul Warren has a variety of demos at the site for his textbook "Introducing Psycholinguistics"<i><br /></i>
<br />
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;">
<br />
<li><a href="http://www.intro2psycholing.net/resources/McGurk/McGurk1.php" target="_blank">McGurk </a>demo</li>
</blockquote>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;">
<br />
<li>Various <a href="http://www.intro2psycholing.net/experiments/expt_index.php" target="_blank">other demos</a> from Warren's textbook</li>
</blockquote>
<br />
<div>
<br /></div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-77793681824892083882015-11-14T09:46:00.000-05:002015-11-14T09:47:18.240-05:00Google's TensorFlow and "mathematical tricks"<a href="http://tensorflow.org/" target="_blank">TensorFlow </a>is a new open source software library for machine learning distributed by Google. In some ways, this could be seen as a competitor to <a href="https://console.ng.bluemix.net/" target="_blank">BlueMix </a>(though much less user friendly). Erik Mueller, who worked on the original Watson Jeopardy system (and has a vested interest in AI with his new company <a href="http://symbolicai.com/" target="_blank">Symbolic AI</a>), just wrote a brief review of TensorFLow for Wired.<br />
<br />
<a href="http://www.wired.com/2015/11/tensorflow-alone-will-not-revolutionize-ai/" target="_blank">Google’s TensorFlow Alone Will Not Revolutionize AI</a><br />
<br />
Unfortunately, it's not really a review of TensorFlow itself, but rather makes a general point against statistical approaches, which I basically agree with, but the argument requires a much more comprehensive treatment.<br />
<br />
Some good quotes from the article:<br />
<br />
<ul>
<li>"I think [TensorFlow] will focus our attention on experimenting with mathematical tricks, rather than on understanding human thought processes."</li>
<li>"I’d rather see us design AI systems that are understandable and communicative."</li>
</ul>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com0tag:blogger.com,1999:blog-520807396714463309.post-38259589841801646642015-06-10T21:32:00.000-04:002015-06-10T21:36:15.864-04:00The Language Myth - Book ReviewLinguistics professor <a href="http://www.vyvevans.net/" target="_blank">Vyvyan Evans</a> recently published a new book that has at least <a href="http://facultyoflanguage.blogspot.com/2015/05/my-hopefully-last-ever-post-on-vyvyan.html" target="_blank">one group of linguists</a> in a state of frenzy: <a href="http://www.amazon.com/Language-Myth-Why-Not-Instinct/dp/1107619750/ref=sr_1_3?s=books&ie=UTF8&qid=1397238752&sr=1-3&keywords=Vyvyan+Evans" target="_blank">The Language Myth: Why language is not an instinct</a>. The book's blurb sums up its content:<br />
<blockquote class="tr_bq">
<i>Some scientists have argued that language is innate, a type of unique human 'instinct' pre-programmed in us from birth. In this book, Vyvyan Evans argues that this received wisdom is, in fact, a myth. Debunking the notion of a language 'instinct', Evans demonstrates that language is related to other animal forms of communication; that languages exhibit staggering diversity; that we learn our mother tongue drawing on general properties and abilities of the human mind, rather than an inborn 'universal' grammar; that language is not autonomous but is closely related to other aspects of our mental lives; and that, ultimately, language and the mind reflect and draw upon the way we interact with others in the world</i>. </blockquote>
<div>
Evans' grounds his motivation in the idea that there are a variety of false claims about how language works ("myths") deeply rooted in our culture's background knowledge as well explicated in introductory text books. He goes further to claim that these false claims have been pushed by a small number of pre-eminent scholars whose fame and influence have caused these false claims to be taken more seriously than they deserve on their face.</div>
<div>
<br /></div>
<div>
By all rights, I should be a good audience for this book. I was trained as a linguist in a department that was openly hostile to the language instinct doctrine that this book argues against (see <a href="http://thelousylinguist.blogspot.com/2015/05/the-language-myth-preliminary-thoughts.html" target="_blank">my post</a> about that experience). </div>
<div>
<br /></div>
<div>
The book is organized by two principles. First, each chapter starts by stating one false claim and providing a description of why it was proposed as an explanation of how language works. Second, each chapter then deconstructs the myth into component claims and shoots holes in each one. </div>
<div>
<br /></div>
<div>
<b>The Good</b></div>
<div>
Evans does a service to the lay audience by pointing out that that deep divisions exist within the filed of linguistics. Too often non-experts assume a technical field is homogeneous and everyone agrees on the basic theories. This is simply not true of linguistics. </div>
<div>
<br /></div>
<div>
Evans also does a service to his audience by stepping through the logic of refutation. His point-counterpoint style can be detailed at times, but I appreciate a book that doesn't treat its readers like third graders (I'm looking at you Gladwell).</div>
<div>
<br /></div>
<div>
For me, the standout chapter was 5: <i>Is language a distinct module in the mind</i>. This chapter is devoted to neurolinguistics and here Evans is at his sharpest when leading the reader through his point-counterpoint about brain regions and functionality.</div>
<div>
<br /></div>
<div>
<b>The Bad</b></div>
<div>
Evans fails to do justice to the myths he debunks. He was accused of creating straw men (and addresses this somewhat in the introduction), but ultimately I have to agree. Evans does not provide a fair description of arguments like <a href="http://en.wikipedia.org/wiki/Poverty_of_the_stimulus" target="_blank">poverty of the stimulus</a>. </div>
<div>
<br /></div>
<div>
Evans quickly shows his bias and directly attacks just two people: Noam Chomsky and Steven Pinker (and to a lesser extent, Jerry Fodor). Evans wants to debunk general notions that have crept into the general public's background beliefs about language, but what he really does is rail against two guys. And worse, he often devolves into a detailed point-counterpoint with just one book, Pinkers' 1994 <a href="http://www.amazon.com/The-Language-Instinct-Mind-Creates/dp/0061336467" target="_blank">The Language Instinct</a>. Any reader unfamiliar with that book will quickly get drowned by arguments against claims they never encountered. As an exercise, I would recommend Evans re-write this book without a single reference to Chomsky, Pinker, or Fodor. I suspect the result will be a more effective piece of writing. </div>
<div>
<br /></div>
<div>
Lest some Chomskyean take this review wrongly, let me be clear: I think Chomsky is broadly wrong and Evans is broadly right. But even though I believe Pinker is wrong and Evans is right, I find Pinker a far superior writer and seller of ideas. And that is a serious problem. </div>
<div>
<br /></div>
<div>
Evans would have been better off throwing away the anti-Chomsky rants and simply write his view of how language works. A book on its own terms. Instead he comes across as your drunk uncle at Christmas who can't stop complaining about how the ref in a high school football game 20 years ago screwed him over with a bad call. This might actually be true, but get over it. </div>
<div>
<br /></div>
<div>
I feel Evans has taken on too much. Each myth is worth a small book itself to debunk properly. This is partly what leads to the straw man arguments. Efficiency. A non-straw man version of Evans' book would be 3000 pages long and only appeal to the three people in the world who know enough detail about both Chomsky and functionalist theory to properly understand all that detail. So I *get* why Evans chose this style. I just think Pinker is better at it. Ultimately Evans alienates his lay audience by ranting about people they don't know and arguments they are unfamiliar with. </div>
<div>
<br /></div>
<div>
A detail complaint: He can be disingenuous with citations. On page 110 he uses the wording "<i>the most recent version of Universal Grammar</i>", but turn to the footnote on 264 and he cites publications from 1981 and 1993. In a book published in 2015, citations from 81 and 93 hardly count as recent. See also page 116 where he cites "<i>a more recent study</i>" that was actually published in 2004 (and probably conducted in 2002).</div>
<div>
<br /></div>
<div>
I don't want to be critical of a book that argues a position I align with, but I must be honest. This book just doesn't cut it. </div>
<div>
<br /></div>
<div>
<br /></div>
Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com3tag:blogger.com,1999:blog-520807396714463309.post-29528205786753937772015-05-17T21:50:00.000-04:002015-05-20T17:47:46.635-04:00The Language Myth - Preliminary ThoughtsI started reading <a href="http://www.amazon.com/The-Language-Myth-Why-Instinct/dp/1107619750" target="_blank">The Language Myth: Why Language Is Not an Instinct </a>by Vyvyan Evans. This book argues that Noam Chomsky is wrong about the basic nature of language. The book has sparked controversy and there have probably been published more words in blogs and tweets in response than are contained in the actual book.<br />
<br />
I'm two chapters in, but before I begin posting my review, I wanted to do a post on academic sub-culture, specifically the one I was trained in. I did my (not quite completed) PhD in linguistics at SUNY Buffalo in 1998-2004. The students only half-jokingly called it Berkeley East because, at the time, about half the faculty had been trained at Berkeley (and several others were closely affiliated in research), and Berkeley is one of the great strong-holds of anti-Chomsky sentiment. Buffalo was clearly a "functionalist" school (though no one ever really knew what that meant, functionalism never really being a field, more a culture).<br />
<br />
In any case, we were clearly, undeniably, virulently, anti-Chomsky. And that's the culture I want to describe to provide some sense of how different the associations are with the name "Chomsky" for me (and I suspect Evans), than for non linguists, and for non-Chomskyan linguists.<br />
<br />
So what was it like to be a grad student in a functionalist linguistics department, with respect to Noam Chomsky?<br />
<br />
[SPOILER ALERT - inflammatory language below. Most of this post is intended to represent a <i>thought climate</i> within functionalist linguistics, not factual evidence]<br />
<br />
I never quite drank the functionalist Kool-Aid (nor the Chomskyean Kool-Aid either, to be clear); nonetheless I remain endowed with a healthy dose of Chomsky skepticism.<br />
<br />
Here is how I remember the general critique of Chomsky echoed in the halls of SUNY Buffalo linguistics (this is my memory of ten+ years ago, not intended to be a technical critique; this is meant to give the impression of what the culture of a functionalist department felt like).<br />
<br />
<b>The Presence of Chomsky</b><br />
<br />
<ul>
<li>First, we didn't talk about Chomsky much, he was peripheral. What little we said about him was typically mocking and belittling (grad students, ya know).</li>
<li>The syntax courses, however, were designed to teach Chomsky's theories for half a semester, then each instructor was given the second half to teach whatever alternative theory they wanted. For my Syntax I course, we used one of Andrew Radford's Minimalism textbooks (then RRG for the second half). For my Syntax II, we used Elizabeth Cowper's GB textbook (then what Matthew Dryer called "Basic Theory", which I always preferred above all else).</li>
<li>We had a summer reading group for years. One summer we read Chomsky’s The Minimalist Program because we felt responsible for understanding the paradigm (we wanted to try to understand the *other*). The group included two senior faculty, both with serious syntax background. </li>
</ul>
<br />
<b>The Perception of Chomsky </b><br />
(amongst my cohort, this is what my professors and fellow grad students, and I, thought about the guy. Whether we were accurate or not is another thing)<br />
<br />
<ul>
<li>Noam Chomsky is a likable man, for those who get to meet him in person.</li>
<li>Chomsky did linguistics a great service by taking linguistics in the general direction of hard science.</li>
</ul>
<div>
However,</div>
<ul>
<li>Chomsky's ideas have never been accepted by a majority of linguists, if you include semanticists, discourse, sociolinguistics, international linguists, psycholinguists, anthropological linguists, historical linguists, field linguists, philologists, etc. Outside of American syntacticians, Chomsky is a footnote, a non factor.</li>
<li>Many of his fiercest critics were former students or colleagues.</li>
<li>Chomsky radically changes his theories every ten years or so, simply ignoring his previous claims when they're proved wrong.</li>
<li>Chomsky has never made a serious attempt to understand other theories or engage in linguistic debate; he lives in a cocoon.</li>
<li>He bases major theoretical mechanisms on scant evidence, often obsessing over a single sentence in a language he himself has never studied, based only on evidence from an obscure source (like a grad student thesis).</li>
<li>He condescendingly dismisses most linguistic evidence (like spoken data) with the unfounded distinction between narrow syntax and broad syntax. This allows him to cherry pick data that suits him, and ignore data that refutes his claims.</li>
<li>When critiques are presented by serious linguists with evidence, the evidence is discarded as *irrelevant*, the linguists are derided as foolish amateurs, and the critiques are dismissed as naive. But rarely are the points taken as serious debate.</li>
<li>Chomsky only debates internal mechanisms of his own theories; anyone who argues using mechanisms outside of those Chomsky-internals is derided as ignorant. In other words, there is only one theoretical mechanism, only one set of theoretical terms and artifacts; only these will be recognized as *legitimate* linguistics. Anything else is ignored. </li>
<li>Chomsky doesn't engage with the wider linguistics community. </li>
<li>Chomsky expects to be taken seriously in a way that he himself would never allow anyone else to be taken seriously: lacking substantial evidence, lacking external coherence, and lacking anything approximating collegiality.</li>
<li>Oh, and Chomsky himself hasn't done serious linguistic analysis since the 80s. He has devoted most of the last 30 years to stabbing at political windmills. At most, he spends maybe 10% of his time on linguistics. </li>
</ul>
<br />
That’s the image of the man as I recall from the view of a functionalist department devoted to descriptive linguistics. Let the verbal assaults begin!!!<br />
<br />
<span style="color: red;">UPDATE</span> (May 5): This post prompted <a href="http://www.reddit.com/r/linguistics/comments/36fh2w/discussion_antigenerative_views_of_chomsky_and/" target="_blank">a spirited Reddit discussion</a>, well word reading.Chrishttp://www.blogger.com/profile/09558846279006287148noreply@blogger.com5