Debasis Ganguly, Gareth J. F. Jones. CNGL: Centre for Global Intelligent Content School of Computing, Dublin City University Dublin 9, Ireland.

Transcription

1 Fuzzy Queries with Rule-based Normalization for Mixed Script Information CNGL: Centre for Global Intelligent Content School of Computing, Dublin City University Dublin 9, Ireland. December 5, 2014

2 Mixed script information retrieval (MSIR): Retrieve relevant documents for ad-hoc search queries, where both the documents and also the queries could be represented with two different scripts, one native to the language of the document and the other a non-native one (Gupta et. al. 2014). MSIR task at FIRE-2014: Documents are Hindi song lyrics (Devanagari and Roman). Queries are keywords in either Devanagari or Roman script.

3 Challenges Script mixing may necessitate different indexing and retrieval strategies for the terms in two different types of scripts. Example: Stemming will have to be different for the native and the foreign script. Transliteration into a foreign script is usually ambiguous due to linguistic and cultural differences. For example, the Hindi word पहल pehla, pehlaa, pahla, pahlaa etc. Multiple alternatives Vocabulary mismatch between queries and documents, e.g., transliterating पहल into pehla will not be able to retrieve documents that contain any other variants, e.g. pahla.

4 Dictionary-based Term Expansion Rule-based Normalization Index Generation Dictionary-based Term Expansion aa.ndhii chalii to naqshekafepaa nahii.n milaa dil jisase mil gayaa wo dubaara nahii.n milaa ham a.njuman me.n sabakii taraf dekhate rahe apanii tarah se ko_ii akelaa nahii.n milaa aawaaz ko to kaun samjhataa ki duur duur Kaamoshiyo.n kaa dardshanaasaa nahii.n milaa kachche gha.de ne jiit lii nadii cha.dhii huii majabuut kashtiyo.n ko kinaaraa nahii.n milaa आ ध चल त नक़ शएकफ़एप नह मल दल जसस मल गय व द ब र नह मल हम अ ज मन म सबक तरफ़ द खत रह अपन तरह स क ई अक ल नह मल आव ज़ क त क न स झत क द र द र ख़ म शयॲ क दद शन स नह मल क च घड़ न ज त ल नद चढ़ ह ई मजब त क तयॲ क कन र नह मल Figure: An example mixed script document.

5 Dictionary-based Term Expansion Rule-based Normalization Index Generation Dictionary Construction Straight-forward way to construct a dictionary. The dictionary constructed this way has 21, 018 word mappings. Ambiguous transliterations are stored in lexicographically sorted order and the first one is used as the equivalent for the native script word. Example: पहल pahla, pahlaa, pehla, pehlaa ; keep pahla and throw others.

6 Dictionary-based Term Expansion Rule-based Normalization Index Generation Sample words from Dictionary Devanagari रखकर च र स ज़ प क सज न ख र त पख वज प क कमस न क ज़ सज न Roman rakhakar chaaraasaaz patke sajaanaa khairaat pakhaavaj patkaa kamasiin kabze sajaane

7 Dictionary-based Term Expansion Rule-based Normalization Index Generation Normalization for unseen words A simple rule-based character sequence normalization to obtain a single equivalent representation for the ambiguous character sequences in the index. Also performed on the query side. Rules are applied iteratively as a result of which a word can go through a sequence of multiple intermediate transformations before eventually reaching the final normal form, e.g. dhoom doom dum dam.

8 Dictionary-based Term Expansion Rule-based Normalization Index Generation Sample Rules Found Substituted Example aa a laagan lagan ay ai sapnay sapnai ae ai sapnae sapnai ii i mahii mahi ee i mahee mahi oo u pooja puja uu u huzuur huzur q k qayamat kayamat ia ya dooria doorya hh h chhaya chaya v w havas hawas bh b bharat barat cch c iccha ica ch c chaya caya gh g ghungru gungru jh j jharoka jarokha sh s shaan saan th t hathi hati dh d dhoom doom um am hum ham ain ai main mai dh ai main mai

9 Dictionary-based Term Expansion Rule-based Normalization Index Generation Fields for Indexing Separate document content into two individual fields, one for the native script and the other for the foreign, compute document term frequency relative to each individual field. Two more additional fields for each scipt: title and a body; a match of a query term in the title of a song lyric may bear more importance than a match in the body of the song. Indexing tool: Lucene. Word bi-grams to take word order into account, e.g. diwana dil and dil diwana refers to two different songs.

10 Dictionary-based Term Expansion Rule-based Normalization Index Generation Index Statistics Vocabulary is more diverse for the transliterated content due to the inherent ambiguities in the transliteration. Character sequence normalization helps reduce diversity. Transliteration helps expand the document lengths by adding equivalent cross-script word representations. Processing #Collection Frequency EN_Title HN_Title EN_Body HN_Body No pre-processing 106,697 51,074 1,211, ,144 Normalization 78,331 51, , ,144 Translit. + Norm. 89,855 51, , ,058

11 Approximate Query Term Matching Automatic Transliteration Fuzzy Query Term Matching Stemming words written in the foreign script is difficult because of the multiple alternatives. Approximately match the query terms with the index terms in the inverted list instead of exact matches. The utility class of Lucene that we make use of for this is the FuzzyQuery. Edit distances of up to 2 are allowed in the approximate matching. Prefix length: τ max(len(t d, t q )) τ = 0.7 for all the experiments.

12 Approximate Query Term Matching Automatic Transliteration Statistical Machine Translation (SMT) based Approach SMT translates a source language sentence to a target language one Transliteration transforms a source script character sequence to a target script character sequence. Word Phonetic character sequences Equivalent cross-script words from dictionary serves as parallel corpora (aligned at the character level). k h u s h b u u ख श ब Goal of alignment function is to learn correct mapping between the character sequences, e.g. k h ख, u to, s h to श and b u u to ब.

13 Approximate Query Term Matching Automatic Transliteration SMT based Approach (Contd.) Experimental settings: 1. Log-linear phrase based statistical machine translation (PB-SMT) model (Koehn, 2007) implemented in Moses. 2. Maximum phrase length of 5 in translation model and 3-gram (character sequence) for language model. 3. GIZA++ implementation of IBM alignment model with grow-diagonal-final-and heuristics for performing source target character alignment and phrase extraction (Koehn, 2003). 4. No reordering performed and distortion parameter word penalty set to zero.

14 Approximate Query Term Matching Automatic Transliteration SMT Results Table: 10-fold cross validation results on transliterated word prediction. EN-HI HI-EN Character Word Character Word Precision Recall F-score BLEU

15 Development Set Results Test Set (Official) Results Development Set Results Binary relevance judgments: to compare our results with that of (Gupta et. al. 2014). Similar to (Gupta et. al. 2014), we consider documents with manually assessed scores of over 2 (i.e. 3 and 4) out of a 5-point scale from 0 to 4 as relevant.

16 Development Set Results Test Set (Official) Results Parameters Evaluation Metrics Term Expansion Norm. FQ Doc. Query MAP MRR Rcl BPREF no no no no no no yes no no no no yes no no yes yes yes no no no yes no yes no yes no no yes yes no yes yes yes yes yes no yes yes yes yes yes yes y (Dict/SMT) no yes yes y (Dict/SMT) yes

17 Development Set Results Test Set (Official) Results Observations Best Results: Combination of dictionary-based term expansion on both the document and the query-side Rule based normalization and approximate query term matching. Automatically transliterated words degrades retrieval quality.

18 Development Set Results Test Set (Official) Results Comparison with Previously Reported Results Run Description Evaluation Metrics MRR MAP TUVal-2 (Gupta et. al. 13) Deep (Gupta et. al. 14) Our Best Second best results only with a simple term based approach without modeling the equivalence of sub-word features with the help of deep auto-encoders (Gupta et. al. 14).

19 Development Set Results Test Set (Official) Results Run Description Two runs MSIR 14. First run: dictionary alignment to expand index and query terms + rule based normalization for indexing + fuzzy query match for retrieval Second run: PBSMT based query term transliteration.

20 Development Set Results Test Set (Official) Results Results #ret=10 #ret=1000 Run Description MRR MAP MRR MAP Normalization + Doc. & Qry. exp. (dict. based) + fuzzy qry. match Normalization + Doc. & Qry. exp. (dict. based) + fuzzy qry. match + Qry. exp. (Moses) MAP and the MRR values lower for the test set queries in comparison to the training set ones.

21 Development Set Results Test Set (Official) Results Why are the results different? Analyze per-query retrieval results for the test set queries. 8 (out of 35) queries produced very low average precision (AP) values of less than queries produced a zero AP value. Likely reason for the bad performance is due to the compounding effect. Query # Title Relevant Document koi ek taara ek taara 1 इकत र - ikataaraa 2 ओ र मनव... इकत र (म हल वर) - o re manawaa... ikataaraa (mahilaa swar)

22 Development Set Results Test Set (Official) Results Conclusions and Future Work A pure IR approach to the search task. Competitive results in comparison to a more sophisticated approach. Surprisingly, the results degrade with the use of PBSMT based query expansion. In future, we plan to investigate the possible reasons for this degradation.