LING/C SC/PSYC 438/538. Lecture 25 Sandiway Fong

Size: px

Start display at page:

Download "LING/C SC/PSYC 438/538. Lecture 25 Sandiway Fong"

Meagan Nicholson
5 years ago
Views:

1 LING/C SC/PSYC 438/538 Lecture 25 Sandiway Fong 1

2 538 Last First Sec,ons Date Romero Diaz Damian Word Senses and WordNet 7th Pike Ammon Johnson 25.2 Classical MT and the Vauquois Triangle 7th Brown Rachel 15.3 Feature Structures in the Grammar 7th Sullivan Trevor Pronominal Anaphor 7th Zhu Hongyi meanings 7th Yu Shuo to Rules 7th Muriel Jorge 14.1 Context-Free Grammars 9th Farahnak Farideh Context-Free Grammars: CKY and Learning 9th Lee Puay Leng Patricia 23.1 Retrieval 9th Xu Dongfang Reference and Phenomena 9th Nie Xiaowen 17.3 First-Order Logic 9th King Adam 19.4 Lexical Event 9th Samtani Sagar 22.1 Named 9th

3 Morphology Chapter 3 of JM Morphology: Introduc@on Stemming

4 Morphology Inflec,onal Morphology: basically: no change in category φ-features (person, number, gender) Examples: movies, blonde, actress Irregular examples: appendices (from appendix), geese (from goose) case Examples: he/him, who/whom compara,ves and superla,ves Examples: happier/happiest tense Examples: drive/drives/drove (-ed)/driven

5 Morphology Deriva,onal Morphology basically: category changing nominaliza,on Examples: formalization, informant, informer, refusal, lossage deadjec,vals Examples: weaken, happiness, simplify, formalize, slowly, calm deverbals Examples: see nominaliza3on, readable, employee denominals Examples: formal, bridge, ski, cowardly, useful

6 Morphology and Morphemes: units of meaning suffixa,on Examples: x employ y employee: picks out y employer: picks out x x read y readable: picks out y prefixa,on Examples: undo, redo, un-redo, encode, defrost, asymmetric, malformed, ill-formed, pro-chomsky

7 Stemming Normaliza,on Procedure morphology: city, improves/improved improve morphology: transform criterion preserve meaning (word senses) organ primary applica,on retrieval (IR) efficacy Harman (1991) Google doesn t didn t use stemming

8 Stemming and Search up un%l recently... Word Varia,ons (Stemming) To provide the most accurate results, Google does not use "stemming" or support "wildcard" searches. In other words, Google searches for exactly the words that you enter in the search box. Searching for "book" or "book*" will not yield "books" or "bookstore". If in doubt, try both forms: "airline" and "airlines," for instance Google didn t use stemming

Stemming and Search Google is more successful than other search engines in part because it returns bejer, i.e. more relevant, informa@on its algorithm (a trade secret) is called PageRank SEO (Search

9 Stemming and Search Google is more successful than other search engines in part because it returns bejer, i.e. more relevant, its algorithm (a trade secret) is called PageRank SEO (Search Engine Op,miza,on) is a topic of considerable commercial interest Goal: How to get your webpage listed higher by PageRank Techniques: e.g. by wri@ng keyword-rich text in your page e.g. by lis@ng morphological variants of keywords Google does not use stemming everywhere and it does not reveal its algorithm to prevent people op@mizing their pages

10 Stemming and Search search on: diet requirements (results from 2004) note can t use quotes around phrase blocks stemming 6th-ranked page

Stemming and Search search on: diet requirements <html> <head> <@tle>bbc - Health - Healthy living - Dietary requirements</@tle> <meta name="descrip@on" content="the foods that can help to prevent

11 Stemming and Search search on: diet requirements <html> <head> - Health - Healthy living - Dietary requirements</@tle> <meta name="descrip@on" content="the foods that can help to prevent and treat certain condi@ons"/> <meta name="keywords" content="health, cancer, diabetes, cardiovascular disease, osteoporosis, restricted diets, vegetarian, vegan"/> notes Top-ranked page has words diet, dietary and requirements but not the phrase diet requirements

12 Stemming and Search search on: diet requirements <HTML> <HEAD> <TITLE>PATLIB Dietary requirements</title> <LINK REL="stylesheet" HREF="styles.css" TYPE="text/css"> </HEAD> notes: 5th-ranked page does not have the word diet in it an academic conference unlikely to be op@mized for page hits ranks above 6th-ranked page which does have the exact phrase diet requirements in it

13 Stemming and Search The internet is s3ll growing quickly (2008) (2011) (2013)

14 Stemming IR-centric view Applies to open-class lexical items only: stop-word list: the, below, being, does exclude: determiners, auxiliary verbs not full morphology prefixes generally excluded (not meaning preserving) Examples: asymmetric, undo, encoding

15 Stemming: Methods use a dic3onary (look-up) OK for English, not for languages with more produc@ve morphology, e.g. Japanese, Turkish, Hungarian etc. write ad-hoc rules, e.g. Porter Algorithm (Porter, 1980) Example: Ends in doubled consonant (not l, s or z), remove last character hopping hop hissing hiss

16 Stemming: Methods dic3onary approach not enough Example: (Porter, 1991) routed route/rout At Waterloo, Napoleon s forces were routed The cars were routed off the highway notes here, the (inflected) verb form is ambiguous preceding word (context) does not disambiguate, i.e. bigram not sufficient

17 Stemming: Errors Understemming: failure to merge Example: adhere/adhesion Page 68 Overstemming: incorrect merge Example: probe/probable Claim: -able irregular suffix, root: probare (Lat.) Mis-stemming: removing a nonsuffix (Porter, 1991) Example: reply rep

18 Stemming: interacts with noun compounding Example: operating systems negative polarity items for IR, compounds need to be first

19 Stemming: Porter Algorithm The Porter Stemmer (Porter, 1980) Textbook (page 68, Chapter 3 JM) 3.8 Lexicon- Free FSTs: The Porter Stemmer URL hjp:// for English most widely used system (claimed) manually wrijen rules 5 stage approach to extrac@ng roots considers suffixes only may produce non-word roots BCPL: ex@nct programming language

20 Stemming: Porter Algorithm

21 Stemming: Porter Algorithm

22 Stemming: Porter Algorithm Test it on Stuff it should work on regular morphology agreed, agrees, agreeing, feed care, cares, cared, caring cancella3on, cancela3on, canceled, cancelled wares, spyware New words obamacare, canceliza3on, phablet(s) Understemming agreement (VC) 1 VCC - step 4 (m>1) adhesion geese Overstemming relativity - step 2 Mis-stemming wander C(VC) 1 VC

23 Porter Stemmer: Perl

24 Porter Stemmer: Perl

25 Porter Stemmer: Perl Just 3 pages of code You should be familiar enough with Perl programming and regular expressions to be able to read and understand the code

Stemming: Porter Algorithm rule format: (condi3on on stem) suffix 1 suffix 2 In case of conflict, prefer longest suffix match Measure of a word is m in: (C) (VC) m (V) C =

26 Stemming: Porter Algorithm rule format: (condi3on on stem) suffix 1 suffix 2 In case of conflict, prefer longest suffix match Measure of a word is m in: (C) (VC) m (V) C = sequence of one or more consonants V = sequence of one or more vowels examples: tree C(VC) 0 V (tr)()(ee) troubles C(VC) 2 (tr)(ou)(bl)(e)(s) Ques@on: is this a regular expression?

27 Stemming: Porter Algorithm Step 1a: remove plural suffixa3on SSES SS (caresses) IES I (ponies) SS SS (caress) S (cats) Step 1b: remove verbal inflec3on (m>0) EED EE (agreed, feed) (*v*) ED (plastered, bled) (*v*) ING (motoring, sing)

28 Stemming: Porter Algorithm Step 1b: (contd. for -ed and -ing rules) AT ATE (conflated) BL BLE (troubled) IZ IZE (sized) (*doubled c & (*L v *S v *Z)) single c (hopping, hissing, falling, fizzing) (m=1 & *cvc) E (filing, failing, slowing) Step 1c: Y and I (*v*) Y I (happy, sky)

29 Stemming: Porter Algorithm Step 2: Peel one suffix off for mul3ple suffixes (m>0) ATIONAL ATE (m>0) TIONAL TION (m>0) ENCI ENCE (valenci) (m>0) ANCI ANCE (hesitanci) (m>0) IZER IZE (digitizer) (m>0) ABLI ABLE (conformabli) - able (step 4) (m>0) IZATION IZE (vietnamiza@on) (m>0) ATION ATE (predica@on) (m>0) IVITI IVE (sensitivi@)

30 Stemming: Porter Algorithm Step 3 (m>0) ICATE IC (triplicate) (m>0) ATIVE (forma@ve) (m>0) ALIZE AL (formalize) (m>0) ICITI IC (electrici@) (m>0) ICAL IC (electrical, chemical) (m>0) FUL (hopeful) (m>0) NESS (goodness)

31 Stemming: Porter Algorithm Step 4: Delete last suffix (m>1) AL (revival) - revive, see step 5 (m>1) ANCE (allowance, dance) (m>1) ENCE (inference, fence) (m>1) ER (airliner, employer) (m>1) IC (gyroscopic, electric) (m>1) ABLE (adjustable, mov(e)able) (m>1) IBLE (defensible,bible) (m>1) ANT (irritant,ant) (m>1) EMENT (replacement) (m>1) MENT (adjustment)

32 Stemming: Porter Algorithm Step 5a: remove e (m>1) E (probate, rate) (m>1 & *cvc) E (cease) Step 5b: ll reduc3on (m>1 & *LL) L (controller, roll)

33 Stemming: Porter Algorithm Misses (understemming) SWI-Prolog/PERL Unaffected: agreement (VC) 1 VCC - step 4 (m>1) adhesion adhes vs. adher(e) Irregular morphology: drove, geese Overstemming relativity - step 2 rel Mis-stemming wander C(VC) 1 VC wander

34 Stemming: Porter Algorithm Extensions? ad hoc and English-specific errors can be fixed on a case-by-case basis i.e. use a look-up table (dic@onary) also there are a finite number of irregulari@es, e.g. drive/drove, goose/geese

Text Similarity. Morphological Similarity: Stemming

Text Similarity. Morphological Similarity: Stemming NLP Text Similarity Morphological Similarity: Stemming Morphological Similarity Words with the same root: scan (base form) scans, scanned, scanning (inflected forms) scanner (derived forms, suffixes) rescan