Language Processing Applications: spelling checkers Nepali spell checker

Transcription

1 Language Processing Applications: spelling checkers Nepali spell checker Bal Krishna Bal Project Manager PAN Localization Project Madan Puraskar Pustakalaya, Nepal URL : bal@mpp.org.np Localization Project. 12 th-16 th January, 2009, Novotel Hotel, Vientiane, Laos 1

2 Contents Background Hunspell and the OpenOffice.org suite Main features of Hunspell Hunspell file prerequisites Fitting the Nepali inflections and derivations into the Hunspell format Nepali Spell checker coverage and robustness Conclusion 2

3 Background Nepali spell checkers do not have a long history. The first spell checkers both for MS Office package and OpenOffice.org suite got released for public usage in the year Spell checking facility for Nepali taken up with great interest. Major beneficiaries publication houses, writers, journalists etc. 3

4 Hunspell and OpenOffice.org suite Hunspell is the default spell checker from OpenOffice.org 2.0 onwards. Hunspell is a spell checker and morphological analyzer library, initially developed for the Hungarian language. Hunspell can be extended to other languages having Unicode support. Hence, the Nepali spell checker is a customized version of Hunspell. 4

5 Main features of Hunspell Extended support for language peculiarities; Unicode character encoding, compounding and complex morphology. Improved suggestion using n-gram similarity, rule and dictionary based pronunciation data. Morphological analysis, stemming and generation. Hunspell is based on MySpell and works also with MySpell dictionaries. C++ library under GPL/LGPL/MPL tri-license. Interfaces and ports: Enchant (Generic spelling library from the Abiword project), OpenXSpell (Mac OS X Enchant port), Delphi, Java (JNA, JNI), Perl, Python, Ruby, UNO. Source: Localization Project. 12 th-16 th January, 2009, Novotel Hotel, Vientiane, Laos 5

6 Hunspell file prerequisites Hunspell consists of language files for different language specific territory. It requires two files in order to define the language that it is spell checking. The first file is a dictionary containing the words for the language and the second is an affix file. The affix file defines the meaning of special flags in the dictionary. These files are located together in one folder ~openofficefolder/share/dic/ooo/ The spell checking is done using the.aff file for the language together with the.dic file. 6

7 Fitting the Nepali inflections and derivations into the Hunspell format A sample entry of the dict file: ãš/r1 A sample of the affix file: Rule r1 contains 4 sub-rules that generates four inflected forms of ãš, which are respectively, ãš, ãš, ãš ` and ãš ^ In the second level, rule r2 is applied to each of the inflected forms. The rules r1 and r2 yield as many as 320 inflections from the single verb like ãš of the dictionary file. 7

8 Nepali spellchecker, coverage and robustness A total of 37,000 head words in the dictionary file. 1,800 affix rules in the affix file. Word coverage of around 6.2 million Nepali words Random tests of the spell checker exhibit: 90% accuracy (43 words unhandled out of 450 words) 94% accuracy (25 words unhandled out of 400 words) 89% accuracy (100 words unhandled out of 923 words) Nepali spell checking in OpenOffice.org Writer 8

9 Conclusion The current version of the Nepali spell checker with substantial enhancements made is believed to have attained the industrial strength or robustness required for the target audience, i.e., publication houses, writers and journalists. Further testing and additional enhancements would be made to the spell checker in the days to come. Localization Project. 12 th-16 th January, Hotel, Vientiane, Laos 9

10 Acknowledgment This work was carried out with the aid of a grant from the Language Resource Association (GSK) of Japan and International Development Research Centre (IDRC), Ottawa, Canada, administered through the Centre for Research in Urdu Language Processing (CRULP), National University of Computer and Emerging Sciences (NUCES), Pakistan. 10

11 Thank You!! 11