What s Next for POS Tagging. Statistical NLP Spring Feature Templates. Maxent Taggers. HMM Trellis. Decoding. Lecture 8: Word Classes

Transcription

1 Statstcal NLP Sprng 2008 Lecture 8: Word Classes Dan Klen UC Berkeley What s Next for POS Taggng Better features! RB PRP VBD IN RB IN PRP VBD. They left as soon as he arrved. We could fx ths wth a feature that looked at the next word JJ NNP NNS VBD VBN. Intrnsc flaws remaned undetected. We could fx ths by lnkng captalzed words to ther lowercase versons Soluton: maxmum entropy sequence models Realty check: Taggers are already pretty good on WSJ journal text What the world needs s taggers that work on other text! Also: same technques used for other sequence models (NER, etc) Maxent Taggers MEMMs: use local dscrmnatve models Feature Templates Important dstncton: Features: <w 0 =future, t 0 =JJ> Feature templates: <w 0, t 0 > Tran up t w,t -1,t -2,) as a normal maxent problem, then use to score sequences Referred to as a maxent tagger [Ratnaparkh 96] Beam search effectve! (Why?) What s the advantage of beam sze 1? In maxent taggers: Can now add edge feature templates: < t -1, t 0 > < t -2, t -1, t 0 > Also, mxed feature templates: < t -1, w 0, t 0 > Decodng HMM Trells Decodng maxent taggers: Just lke decodng HMMs Vterb, beam search, posteror decodng Vterb algorthm (HMMs): Vterb algorthm (Maxent): 1

2 TBL Tagger [Brll 95] presents a transformaton-based tagger Label the tranng set wth most frequent tags TBL Tagger II What gets learned? [from Brll 95] DT MD VBD VBD. The can was rusted. Add transformaton rules whch reduce tranng mstakes MD NN : DT VBD VBN : VBD. Stop when no transformatons do suffcent good Does ths remnd anyone of anythng? Probably the most wdely used tagger (esp. outsde NLP) but not the most accurate: 96.6% / 82.0 % EngCG Tagger Englsh constrant grammar tagger [Tapananen and Voutlanen 94] Somethng else you should know about Hand-wrtten and knowledge drven Don t guess f you know (general pont about modelng more structure!) Tag set doesn t make all of the hard dstnctons as the standard tag set (e.g. JJ/NN) They get stellar accuraces: 98.5% on ther tag set Lngustc representaton matters but t s easer to wn when you make up the rules CRF Taggers Newer, hgher-powered dscrmnatve sequence models CRFs (also voted perceptrons, M3Ns) Do not decompose tranng nto ndependent local regons Can be deathly slow to tran requre repeated nference on tranng set Dfferences tend not to be too mportant for POS taggng Dfferences more substantal on other sequence tasks However: one ssue worth knowng about n local models Label bas and other explanng away effects Maxent taggers local scores can be near one wthout havng both good transtons and emssons Ths means that often evdence doesn t flow properly Why sn t ths a bg deal for POS taggng? Also: n decodng, condton on predcted, not gold, hstores CRFs Make a maxent model over entre taggngs MEMM CRFs Lke any maxent model, dervatve s: CRF So all we need s to be able to compute the expectaton each feature, for example the number of tmes the label par DT-NN occurs, or the number of tmes NN-nterest occurs n a sentence How many tmes does, say, DT-NN occur at poston 10? The rato of the scores of trajectores wth that confguraton to the score of all Ths requres exactly the same forward-backward score ratos as for EM, but usng the local potentals ph nstead of the local probabltes 2

3 Doman Effects Accuraces degrade outsde of doman Up to trple error rate Usually make the most errors on the thngs you care about n the doman (e.g. proten names) Open questons How to effectvely explot unlabeled data from a new doman (what could we gan?) How to best ncorporate doman lexca n a prncpled way (e.g. UMLS specalst lexcon, ontologes) Unsupervsed Taggng? AKA part-of-speech nducton Task: Raw sentences n Tagged sentences out Obvous thng to do: Start wth a (mostly) unform HMM Run EM Inspect results EM for HMMs: Process Forward Recurrence Alternate between recomputng dstrbutons over hdden varables (the tags) and reestmatng parameters Crucal step: we want to tally up how many (fractonal) counts of each knd of transton and emsson we have under current params: But we need a dynamc program to help, because there are too many sequences to sum over Backward Recurrence Fractonal Transtons 3

4 EM for HMMs: Quanttes EM for HMMs: Process Cache total path values: From these quanttes, we can re-estmate transtons: And emssons: Can calculate n O(s 2 n) tme (why?) If you don t get these formulas mmedately, just thnk about hard EM nstead, where were re-estmate from the Vterb sequences Meraldo: Setup Meraldo: Results Some (dscouragng) experments [Meraldo 94] Setup: You know the set of allowable tags for each word Fx k tranng examples to ther true labels Learn w t) on these examples Learn t t -1,t -2 ) on these examples On n examples, re-estmate wth EM Note: we know allowed tags but not frequences Dstrbutonal Clusterng the that the downturn was over reported the of the the of the apponted sources that sources reported the a Dstrbutonal Clusterng Three man varants on the same dea: Parwse smlartes and heurstc clusterng E.g. [Fnch and Chater 92] Produces dendrograms Vector space methods E.g. [Shuetze 93] Models of ambguty Probablstc methods Varous formulatons, e.g. [Lee and Perera 99] [Fnch and Chater 92, Shuetze 93, many others] 4

5 Nearest Neghbors Dendrograms _ Dendrograms _ Vector Space Verson [Shuetze 93] clusters words as ponts n R n context counts w M Vectors too sparse, use SVD to reduce context counts w U Σ V Cluster these dm vectors nstead. A Probablstc Verson? ( S, C) = w w 1, w+ 1 P c) c 1 c 2 c 3 c 4 c 5 c 6 c 7 c 8 the that the downturn was over = w c c P ( S, C) 1) c 1 c 2 c 3 c 4 c 5 c 6 c 7 c 8 What Else? Varous newer deas: Context dstrbutonal clusterng [Clark 00] Morphology-drven models [Clark 03] Contrastve estmaton [Smth and Esner 05] Also: What about ambguous words? Usng wder context sgnatures has been used for learnng synonyms (what s wrong wth ths approach?) Can extend these deas for grammar nducton (later) the that the downturn was over 5