Language Identification for Texts Written in Transliteration

Transcription

1 Language Identification for Texts Written in Transiteration Andrey Chepovskiy, Sergey Gusev, Margarita Kurbatova Higher Schoo of Economics, Data Anaysis and Artificia Inteigence Department, Pokrovskiy bouevard, Moscow, Russia Abstract. The probem of identification of natura anguages for the texts written in transiteration is considered. We consider a method of identification of five Savic anguages for texts written with use of a Latin transiteration. We use two ways of creation modes for such texts and compare resuts of those appication. Keywords: statistica text mode; natura anguage identification; transiteration. Introduction In various type of information systems intended for automatic processing of arge amounts of texts in natura anguages various data recognition probems are actua. The requirement of automating textua data processing brings specific importance to the anguage identification probem of a text or a part of a text. At present time the sufficienty accurate anguage recognition methods for ong texts consisting of tens of sentences are known []. Modes based on frequencies of etter combinations are widey used to identify anguage of a text [2, 3]. It was noted in [3] that it is possibe to use rank methods for anguage identification of a text, but they are not suitabe for short texts. Aso in [3] it is concuded that the anguage identification probem for short texts segments is sti actua, and higher accuracy is achieved at the expense of arger mode size and sower processing. In [4] a anguage identification method for texts in natura anguage was studied. This method was appied to texts written in native aphabets for corresponding anguages and good resuts were achieved. In current work we consider using the same method for anguage identification of a text written in Latin transiteration. We use a set of five Savic anguages: Russian, Ukrainian, Byeorussian, Bugarian and Macedonian. A of these anguages use Cyriic aphabet as native, and transiteration must be used to write in Latin. The main probems reated to anguage identification process are: A wide variety of conficting transiteration tabes;

2 4 A. Chepovskiy et a. A frequent use of transiteration rues which do not meet any of standard transiteration tabes. Amount of texts in transiteration is not enough for construction of training sets that are used in anguage identification agorithms. To sove the first probem severa transiteration tabes shoud be used for each of the anguages. Training text sets were created with an automatic text generation process from Cyriic texts by using transiteration tabes. 2 Statistica mode for a natura anguage text The text anguage identification probem is a pattern recognition probem, and its soution can be based on a probabiistic mode. A Bayesian cassifier can be appied to a string of characters assuming that we know statistica characteristics of characters for texts in specific anguage or texts beonging to a given cass. Let s consider a string s which consists of characters с n (n =,, ) that beong to aphabet Σ. Further we wi use the foowing notation: s = с с 2 с - a specific vaue of the string, s i = c i a vaue of a character which is at i-th position in the string. For soving the probem this string must be assigned to one of the casses Y ( =,, K ), where Y denotes one of K anguages. We assume that every cass defines some kind of probabiity distribution on the set of a possibe strings. In that case it is possibe to appy a statistica criterion of maximum ikeihood to determine a cass which contains the string being cassified. The probabiity of a fact that string s wi appear in some anguage equas to product of probabiities that each character of this string wi appear in this anguage provided that a preceding characters wi appear in this anguage too: s... c s ) s s,..., s s 2,..., s 2 )*...* s )* ) () Let s assume that the probabiity distribution for a character at i-th position depends on probabiity distribution of not more than k preceding characters. In this case equation () can be written as foowing: P s s,..., s ) s s,..., s ) (2) ( i i i i i i i k i k i i An estimation of the conditiona probabiities is performed on the training set. For this purpose, the frequencies of a substrings of engths ess than k+2 are cacuated, and the estimated vaue of conditiona probabiity for the next character is a ratio of the frequencies of the corresponding substrings: Y, s i f ( ci m... ci ) i si m i m,..., si i ), m k f ( c... c ), (3) i m i

3 Language Identification for Texts Written in Transiteration 5 f(x) the frequency of substring X in the training set. The estimated vaue of probabiity of string s appearance in cass Y is defined as foows: Y, s) Y, s Y, s s k s k n k k,..., s,..., s 2 2 )* )*...* Y, s ) (4) The cassified string is assigned to the cass with the highest probabiity estimate. 3 Agorithm impementation First, each natura anguage text is converted to a set of words consisting of ower cased characters beonging to the native aphabet of the anguage. It forms a frequency dictionary of substrings having engths in the range [, k+] taken into account the number of word occurrences in the text. This process is executed during the construction of a string mode for a given anguage. This construction is based on training set of texts. The mode is represented as a finite state machine with states marked with preceding character sequences and transitions marked with the next character and corresponding conditiona probabiity. A singe space is appended to the end of each word of the text, then the word is passed as input to the finite state machine. The initia state of the machine corresponds to character sequence consisting of k spaces. According the formuas () (4) a probabiity of transition to the next state from the current one by each of the characters is being cacuated. A probabiity of appearance of the given word is a product of probabiities of a transitions that occurred during the machine operation. The probabiity of a text is a product of probabiities of a its words. For the anguage identification a probabiity of the text appearance for modes of every natura anguage is estimated. The anguage of the mode with the highest probabiity is assigned to the text. 4 Quaity of the text anguage identification For the anguage identification of a text fragment a numeric estimate of its correspondence to a natura anguage text mode can be cacuated. Let the text fragment s consist of characters. A probabiity of its appearance in the text written in -th anguage can be estimated with the formua (4). Then an estimation of this text fragment correspondence to the -th anguage wi be cacuated as: n( Y, s)) E ( s) onst, (5) Y, s) the probabiity of the string s appearance in the anguage Y ;

4 6 A. Chepovskiy et a. the number of characters in the string s; const a normaizing constant. The expected vaue of this estimation doesn t depend on the ength of text fragment. We choose the anguage with a maximum vaue of the estimation E (s). 5 Initia data Transiteration tabes for a of five anguages were constructed. Severa different tabes were used for each anguage from 4 to 0, depending on anguage. Cyriic text sets consisting of at east 500 thousands characters were made for each of the five anguages. These sets were automaticay transiterated into Latin aphabet with each of the transiteration tabes. Some tabes contain ambiguous transation rues (i.e. there are severa versions of Latin character combinations for one Cyriic character). When severa different rues were possibe then ony one of the rues was chosen randomy. These texts subsequenty were used as training texts for creation of text modes. Test sets consisting of at east 50 thousands characters were constructed for each anguage. Test sets are rea texts written in transiteration taken from various sources. We wi ca texts written in transiteration as transiterated texts, and texts written in native aphabets of corresponding anguages as native texts. To compare anguage identification quaity for transiterated texts with anguage identification quaity for native texts the foowing training and text sets of the same size were made: Cyriic sets for the five anguages being considered; Sets for 3 anguages which use Latin aphabet as native. 6 Experimenta resuts We evauate the quaity of our anguage identification method by cacuating precision and reca for individua anguages. When we identify the anguage of a text sampe of known anguage we can determine whether it was identified correcty or not. For a set of test sampes we know the number of correcty identified sampes as we as the number of identification errors for each anguage. For a given anguage precision can be cacuated as a ratio of the number of correcty identified text sampes in this anguage to the overa number of sampes identified to this anguage. Reca is a ratio of correcty identified text sampes in this anguage to the number of a sampes in this anguage in the test set. We use F-measure to combine precision and reca to a singe vaue, which is defined as harmonic mean of precision and reca. To evauate the quaity of the anguage identification method we created a set of anguage modes. Each mode was trained on a text from the training set and marked with the anguage of that text. Then severa sets of text sampes were created from the texts beonging to the test set. Each set incuded 000 text sampes for each anguage

5 Language Identification for Texts Written in Transiteration 7 of the same ength. The sampes were generated as text fragments starting from a randomy seected position inside text with the restriction that the position must be a beginning of a word. Every text sampe was evauated with every mode and the anguage of the mode with the highest estimate was chosen as the identified anguage of the sampe. Then the vaues of F-measure were cacuated for every anguage. The set of modes incuded 5 modes for transiterated texts of Savic anguages as we as 3 modes for native texts with anguages which use Latin as their native aphabet. The test set contained 36 texts for the same anguages. We considered two methods of anguage identification for transiterated texts. The first method uses severa text modes for each anguage. Each mode corresponds to a particuar transiteration tabe for the anguage. The mode is trained on a text which was generated using that transiteration tabe. So the set of modes contains severa different modes for each anguage. If a text sampe receives the highest estimate for any of the modes for a particuar anguage then the sampe is identified as beonging to that anguage. The second method uses exacty one text mode for each anguage. The mode is trained on a text which is a concatenation of a texts generated for the anguage by using a its transiteration tabes. Figures and 2 show dependence of the anguage identification F-measure for transiterated texts on the text sampe ength. Using severa modes for each anguage instead of one mode does not ead to significant improvement of the text anguage identification. Fig.. Dependence of the F-measure on text sampe ength when using severa modes for one anguage.

6 8 A. Chepovskiy et a. Fig. 2. Dependence of the F-measure on text sampe ength when using one mode for one anguage. For comparison figure 3 shows the anguage identification quaity for native texts. Resuts were obtained by training and testing of five anguage modes for the Cyriic sets for the same five Savic anguages. Fig. 3. Dependence of the F-measure on text sampe ength for anguage identification of native texts.

7 Language Identification for Texts Written in Transiteration 9 From figure 3 one can see that anguage identification quaity for transiterated texts is significanty ower than for native texts. The most substantia part of accuracy oss occurs when transiterated text incorrecty cassifies to mode representing transiterated text of another anguage. There are much fewer errors when transiterated text cassifies to a anguage that uses Latin aphabet. To iustrate this fact, we can measure the quaity of separation of transiterated texts from native texts written in Latin. It is much higher than anguage identification quaity for transiterated text. Figure 4 shows F-measure vaues for separation of transiterated texts for various text sampe engths. Fig. 4. F-measure vaues for separation of transiterated texts for various text sampe engths. 7 Concusion The text mode considered in this artice aows to successfuy separate texts written in anguages with Latin-based aphabets from texts written in Latin transiteration for anguages with Cyriic aphabets. The F-measure of such separation reaches 98% for short texts consisting of ony 20 characters. Identification accuracy of anguages using Cyriic aphabet for transiterated texts reaches more than 80% for texts consisting of 40 characters. It was observed that using severa different text modes for a singe anguage does not give a significant advantage over using a singe mode for a anguage.

8 20 A. Chepovskiy et a. References. Mcamee, B.P.: Language identification: a soved probem suitabe for undergraduate instruction Journa of Computing Sciences in Coeges, 20(3), P (2005) 2. Cavnar, W. B., Trenke, J. M.: -gram-based text categorization. In.: Proceedings of SDAIR-94, 3rd Annua Symposium on Document Anaysis and Information Retrieva, Las Vegas, US, P (994) 3. Vatanen, T, Väyrynen, J.J., Virpioja, S.: Language identification of short text segments with n-gram modes. In.: Proceedings of the Seventh conference on Internationa Language Resources and Evauation (LREC'0), P (200) 4. Gusev, G., Chepovskiy, A.: The mode for identification of a natura anguage of the text. Business Informatics, 3(7), P (20)