CS460/626 : Natural Language Processing/Speech, NLP and the Web (Lecture 27, 28 Multilingual Resource Constrained WSD)

Transcription

1 CS460/626 : Natural Language Processing/Speech, NLP and the Web (Lecture 27, 28 Multilingual Resource Constrained WSD) Pushpak Bhattacharyya CSE Dept., IIT Bombay 19 th and 22 th March, 2012

2 Long line of work Mitesh Khapra, Salil Joshi and Pushpak Bhattacharyya, It takes two to Tango: A Bilingual Unsupervised Approach for Estimating Sense Distributions using Expectation Maximization, 5th International Joint Conference on Natural Language Processing (IJCNLP 2011), Chiang Mai, Thailand, November Mitesh Khapra, Salil Joshi, Arindam Chatterjee and Pushpak Bhattacharyya, Together We Can: Bilingual Bootstrapping for WSD, Annual Meeting of the Association of Computational Linguistics (ACL 2011), Oregon, USA, June Mitesh Khapra, Saurabh Sohoney, Anup Kulkarni and Pushpak Bhattacharyya, Value for Money: Balancing Annotation Effort, Lexicon Building and Accuracy for Multilingual WSD, Computational Linguistics Conference (COLING 2010), Beijing, China, August Mitesh Khapra, Anup Kulkarni, Saurabh Sohoney and Pushpak Bhattacharyya, All Words Domain Adapted WSD: Finding a Middle Ground between Supervision and Unsupervision, Conference of Association of Computational Linguistics (ACL 2010), Uppsala, Sweden, July Mitesh Khapra, Sapan Shah, Piyush Kedia and Pushpak Bhattacharyya, Domain-Specific Word Sense Disambiguation Combining Corpus Based and Wordnet Based Parameters, 5th International Conference on Global Wordnet (GWC2010), Mumbai, Jan, Mitesh Khapra, Sapan Shah, Piyush Kedia and Pushpak Bhattacharyya, Projecting Parameters for Multilingual Word Sense Disambiguation, Empirical Methods in Natural Language Prfocessing (EMNLP09), Singapore, August, Mitesh Khapra, Pushpak Bhattacharyya, Shashank Chauhan, Soumya Nair and Aditya Sharma, Domain Specific Iterative Word Sense Disambiguation in a Multilingual Setting, International Conference on NLP (ICON08), Pune, India, December, 2008.

3 INDOWORDNET Sanskrit Wordnet Urdu Wordnet Bengali Wordnet Dravidian Language Wordnet Kashmiri Wordnet Oriya Wordnet North East Language Wordnet Konkani Wordnet Hindi Wordnet English Wordnet Gujarati Wordnet Punjabi Wordnet Marathi Wordnet

4 Goals of the endeavour synsets linked with Hindi wordnet Synsets to satisfy principles of: Minimality Coverage Replaceability Correctness (no outlier) High quality gloss and examples Semantically and lexically linked

5 Categories of Synsets (1/2) Universal: Synsets which have an indigenous lexeme in all the languages (e.g. Sun,Earth). Pan Indian: Synsets which have indigenous lexeme in all the Indian languages but no English equivalent (e.g. Paapad). In-Family: Synsets which have indigenous lexeme in the particular language family (e.g. the term for Bhatija in Dravidian languages).

6 Categories of Synsets (2/2) Language specific: Synsets which are unique to a language (e.g. Bihu in Assamese language) Rare: Synsets which express technical terms (e.g. ngram). Synthesized: Synsets created in the language due to influence of another language (e.g. Pizza).

7 Need for categorization To bring systematicity in the way the wordnet synsets are linked Universal Pan Indian Language Family Language Synthesised Rare All members have finished the Universal and Pan Indian synsets

8 Categorization methodology Hindi synsets were sent to all Indo-wordnet groups in the tool, in which they had these options to categorize: Yes No Universal synsets:- The synsets which were categorized Yes and also have equivalent English words or synsets. Pan-Indian :- The synsets which were categorized Yes and did not have equivalent English words or synsets.

9 Challenges of linking by expansion

10 Expansion approach: linking is a subtle and difficult process To link or not to link While linking: face lexical and semantic chasms Syntactic divergences in the example sentences Change of POS Copula drop (Hindi Bangla)

11 Linking kinship relations and fine grained concepts Relative Uncle Chacha Mama प न direct आब प न hypernym श Case of kashmiri

12 Important decision TWO kinds of linkages Direct Hypernymy प न direct आब प न hypernym श Case of kashmiri

13 How to express a concept not present in the language?

14 Transliteration: often employed Synset ID : 39 POS : adjective Synonyms : सन थ, (sanaatha) Gloss : जसक क ई प लन-प षण य द खभ ल करन व ल ह (orphan) Example statement : "सन थ ब लक क अन थ ब लक क मदद करन च हए (children who are looked after should help the ones who are orphans)/ स धक भ क ह ज न पर अन थ नह रहत, सन थ ह ज त ह Transliterated and adopted by Bangla and Gujarati

15 Short phrase: often employed Bangla Urdu (meaning Inauspicious)

16 Linking synsets across languages: Influence on Hindi Wordnet Hindi wordnet has to add new synsets to accommodate language specific concepts, e.g., in Gujarati (bhairav jap) ID :: CAT :: NOUN CONCEPT :: म क लए जप करत ह ए पव त पर स अपन आप क गर न (Taking God s name and throwing oneself from atop a mountain to attain liberation) EXAMPLE :: गरन र क शखर पर स य क भ रवजप करत थ एस म न ज त ह (it is thought that pilgrms used to do bhairav jap atop Girnar mountain) SYNSET-HINDI :: भ रवजप

17 Resource Constrained Multilingual WSD

18 WSD is costly! 1 WordNets Princeton Wordnet: ~80000 synsets: 30 man years Eurowordnet: 12 man years on the average for 12 languages Hindi wordnet: 24 man years Indowordnet: getting created; 15 languages; 4 people on the average; in 1 year close to synsets done Scale of effort really huge Tricky too: when it comes to expanding from one wordnet to another

19 Expansion from WN: tricky! Synset of uncle in English does not have direct correspondence with the concept of uncle in Hindi: there are different kinds of uncles E: uncle H: mama (maternal uncle) H: chacha (paternal uncle) H: phupha (father s sister s husband)

20 WSD is costly! 2 Sense Annotated corpora SemCor: ~ sense marked words SemEval/Senseval competition: to generate sense marked corpora Sense marked corpora created at IIT Bombay English: Tourism (~170000), Health (~150000) Hindi: Tourism (~170000), Health (~80000) Marathi: Tourism (~120000), Health (~50000) 12 man years for each <L,D> combination

21 Cost-accuracy trade off

22 Can we REUSE resource?

23 This is the dream! spread from one <L,D> combination to others

24 Related Work (Not mentioning references, because they are too many) Knowledge Based Approaches Supervised Approaches Unsupervised Approaches Semi-supervised Approaches Hybrid Approaches

25 No single existing solution to WSD completely meets our requirements of multilinguality, high domain accuracy and good performance in the face of limited annotation

26 Scenarios of language adaptation Scenario-1: Supervised Large amount of annotated data available in a resource rich language (say, L 1 ) No annoated data available in the target language (say, L 2 ) Scenario-2: Supervised Large amount of annotated data available in L 1 Small amount of annotated data available in L 2 Scenario-3: Unsupervised Only unannotated data available in both L 1 and L 2 Scenario-4: Semi-supervised Small amount of seed annotated data and a large amount of unannotated data are available in both L 1 and L 2

27 Understanding parameters that control WSD

28 Example to motivate the parameters The river flows through this region to meet the sea. S1: (n) sea (a division of an ocean or a large body of salt water partially enclosed by land) S2: (n) ocean, sea (anything apparently limitless in quantity or volume) S3: (n) sea (turbulent water with swells of considerable size) "heavy seas What are the parameters that influence the choice of the correct sense for the word sea?

29 Parameters for WSD: corpus dependent Domain specific distributions - water-body sense more common in tourism domain - Domain-specific sense distribution should be used Corpus Co-occurrence statistics - Co-occurring monosemous and/or already disambiguated words in the context help in disambiguation. - River dismabiguates Sea

30 Parameters for WSD: WN dependent Dominance of senses in a domain - {place, country, city, area}, {flora, fauna}, {mode of transport}, {fine arts} are dominant senses in the Tourism domain A synset node in the wordnet hypernymy hierarchy is called Dominant if the synsets in the sub-tree below the synset are frequently occurring in the domain corpora. - A sense which belongs to the sub-tree of a dominant sense should be given a higher score than the other senses Semantic distance Shortest path length between two synsets in the wordnet graph Conceptual distance between noun synsets =semantic distance between synsets / depth of the lowest common ancestor of the synsets

31 WordNet Sub-Graph Hyponymy Dwelling,abode Hypernymy Hyponymy Meronymy kitchen bckyard veranda M e r o n y m y house,home Hyponymy Gloss bedroom A place that serves as the living quarters of one or mor efamilies study guestroom hermitage cottage

32 Why Project Parameters? for resource reuse Wordnet-dependent parameters depend on the graphbased structure of wordnet Corpus-dependent parameters depend on various statistics learnt from a sense marked corpora Both the tasks, Constructing a wordnet from scratch Collecting sense marked corpora for multiple languages are tedious and expensive Can the effort required in constructing semantic graphs for multiple wordnets and collecting sense marked corpora in multiple languages be avoided/lessened?

33 How to facilitate parameter projection

34 Synset based Multilingual Dictionary Rajat Mohanty, Pushpak Bhattacharyya, Prabhakar Pande, Shraddha Kalele, Mitesh Khapra and Aditya Sharma Synset Based Multilingual Dictionary: Insights, Applications and Challenges. Global Wordnet Conference, Szeged, Hungary, January Link synsets and then link the synset word Concepts L1 (English) L2 (Hindi) L3 (Marathi) 04321: a youthful male person {malechild, boy} {लड़क ladkaa, ब लक baalak,ब च bachchaa} {म लग mulgaa, प रग porgaa, प र por} CFILT - IITB Hindi is used as the central language the synsets of all languages link to the corresponding Hindi synset. Advantage: The synsets in a particular column automatically inherit the various semantic relations of the Hindi wordnet the wordnet based parameters thus get projected 34

35 Solving the lexical choice by word level linkage CFILT - IITB म लग /MW1 mulagaa, प रग /MW2 poragaa, प र /MW3 pora लड़क /HW1 ladakaa, ब लक /HW2 baalak, ब च /HW3 bachcha, छ र /HW4 choraa male-child /HW1, boy /HW2 Marathi Synset Hindi Synset English Synset 35

36 Algorithm for WSD

37 Scoring function Motivated by the Energy expression in Hopfield network Neuron Synset Selfactivation Weight of connection between two neurons Corpus Sense Distribution Weight as a function of corpus co-occurrence and Wordnet distance measures between synsets

38 Algorithm 1: performiterativewsd(sentence) Iterative WSD 1. Tag all monosemous words in the sentence. 2. Iteratively disambiguate the remaining words in the sentence in increasing order of their degree of polysemy. 3. At each stage select that sense for a word which maximizes the score given by the Equation below

39 IWSD

40 Data

41 #Polysemous words (tokens) #monosemous words

42 #Polysemous unique words (types) Token to Type ratio

43 H S H H Average degree of WN polysemy H S H H Average degree of and corpus polysemy

44 A look at monolingual WSD performance Is IWSD good?

45 Algorithms IWSD: Our algorithm EGS: the exhaustive graph search algorithm described PPR: a state of the art knowledge based approach SVM based McCarthy et al. 2007: a state of the art unsupervised approach RB: randomly selects one of the senses of the word from the Wordnet WFS: assigns the first sense of the word from the Wordnet MFS: assigns the most frequent sense of the word as obtained from an annotated corpus.

46 Performance of different algorithms: monolingual WSD

47 Language Adaptation

48 Language Adaptation scenarios

49 Projecting the sense: example (1/2)

50 Projecting the sense: example (2/2)

51 Projecting with probabilistic cross linking: example (1/3)

52 Projecting with probabilistic cross linking: example (3/3)

53 Validating sense projection Sr. No Marathi Word Synset P(S word) as learnt from sense tagged Marathi corpus P(S word) as projected from sense tagged Hindi corpus 1 क मत { worth } (kimat) { price } र त (rasta) { roadway } ठक ण (thikan) {road, route} { land site, place} { home } For Hindi Marathi Average KL Divergence=0.29 Spearman s Correlation Coefficient=0.77 For Hindi Bengali Average KL Divergence=0.05 Spearman s Correlation Coefficient=0.82 There is a high degree of similarity between the distributions learnt using projection and those learnt from the self corpus.

54 Co-occurrence parameter Projection Sr. No Synset Cooccurring Synset P(cooccurrence) as learnt from sense tagged Marathi corpus P(cooccurrence) as learnt from sense tagged Hindi corpus Within a domain, the statistics of co-occurrence of senses remain the same across languages. 1 {र प, र पट } {झ ड, व, {small त वर, म, bush} त, प दप} 2 {म घ, अ } {cloud} 3 {, इल क़, इल क, भ ख ड} {geographic al area} {tree} {आक श, आभ ळ, अ बर} {sky} {य, सफ़र} {travel} Co-occurrence of the synsets {cloud} and {sky} is almost same in the Marathi and Hindi corpus.

55 IWSD with parameter projection (Marathi using Hindi) MCL-manual cross linked; PCL: probabilistic cross linked; IWSD=Self: IWSD with own language training data; WFS: wordnet first sense Mitesh Khapra, Sapan Shah, Piyush Kedia and Pushpak Bhattacharyya, Projecting Parameters for Multilingual Word Sense Disambiguation, Empirical Methods in Natural Language Prfocessing (EMNLP09), Singapore, August, Mitesh Khapra, Saurabh Sohoney, Anup Kulkarni and Pushpak Bhattacharyya, Value for Money: Balancing Annotation Effort, Lexicon Building and Accuracy for Multilingual WSD, Computational Linguistics Conference (COLING 2010), Beijing, China, August 2010.

56 EM based unsupervsied Approach 56

57 ESTIMATING SENSE DISTRIBUTIONS If sense tagged Marathi corpus were available, we could have estimated But such a corpus is not available 57

58 Framework: Figure 1 and Figure 2

59 E-M steps

60 Points to note Symmetric formulation E and M steps are identical except for the change in language Either can be treated as the E-step, making the other as the M-step A back-and-forth traversal over translation correspondences in the two languages Does not require parallel corpus only in-domain corpus is needed 60

61 In General.. 61

62 Experimental Setup Languages: Hindi, Marathi Domains: Tourism and Health (largest domain-specific sense tagged corpus) 62

63 Algorithms Being Compared EM (our approach) Personalized PageRank (Agirre and Soroa, 2009) State-of-the-art bilingual approach (using Mutual Information) (Kaji and Morimoto, 2002) Random Baseline Wordnet First sense baseline (supervised baseline) 63

64 Results Performs better than other state-of-the-art knowledge based and unsupervised approaches Does not beat the Wordnet First Sense Baseline which is a supervised baseline 64

65 Error Analysis Non-Progressiveness estimation Some words have the same translations in the target language across senses saagar(hindi) samudra (marathi) ( large water body as well as limitless ) Such words thus form a closed loop of translations In such cases the algorithm does not progress and gets stuck with the initial values Same is the case for some language specific words for which corresponding synsets were not available in the other language Such words accounted for 17-19% of the total words in the test corpus 65

66 have problem of Non Progressive Estimation Results are now closer to Wordnet First Sense Baseline For 2 out of the 4 language domain pairs the results are slightly better than WFS remarkable for an unsupervised approach 66

67 Further error Analysis (1/2) MultiDict related issues: Hindi word sankraman (infection) translates to sansarg (infection) in Marathi However, sansarg (infection) was absent in the corresponding Marathi synset (incomplete Marathi synset) Poor performance on verbs Highly polysemous a common bane for all WSD algorithms Do not form a close loop of translations but share many translations across senses e.g., the Hindi word karna (do) has the same Marathi translation in 8 out of its 21 senses Thus translations do not play a discriminatory role 67

68 Further error Analysis (2/2) Influence of synonyms in a rare sense: Hindi word jab has two senses, viz., when (S1) and if (S2) It is rarely used in the sense S2 (if) However, its other synonyms (yadi (if) and agar(if)) are frequently used in this sense (S2) The same is observed in the Marathi corpus where the translations of yadi (if) and agar(if) in S2 are very frequent As a result, these translations strongly bias the probability towards the if sense of jab 68

69 conclusions An unsupervised bilingual approach for estimating sense distributions using EM Performs a back-and-forth traversal over translation correspondences Performs better than current state-of-the-art approaches When restricted to words not facing the problem of non-progressiveness estimation, the performance was better than WFS for 2 out of 4 language domain pairs An effective way of utilizing untagged corpora in two languages 69

70 Future work Can the problem of non-progressiveness estimation be solved using more than two languages? 70