Morphological Ending based Strategies of Unknown Word Estimation for Statistical POS Urdu Tagger

Transcription

1 IJCSES Iteratoal Joural of Computer Sceces ad Egeerg Systems, Vol., No.3, July 2007 CSES Iteratoal c2007 ISSN Morphologcal Edg based Strateges of Ukow Word Estmato for Statstcal POS Urdu Tagger Waqas ANWAR, Xua WANG, Lu LI 2, ad Xao-Log WANG 2 School of Computer Scece ad Techology,Harb Isttute of Techology Shezhe Graduate School Shezhe, 58055, R.P.Cha E-mal:{waqas,xuawag}@su.ht.edu.c 2 School of Computer Scece ad Techology,Harb Isttute of Techology Shezhe Graduate School Shezhe, 58055, R.P.Cha E-mal:{ll,wagxl}@su.ht.edu.c Abstract Natural laguage processg has wdely used Statstcal based laguage models to solve dsambguato problems. Over the past decades dfferet techques regardg POS taggg have bee proposed for Eglsh, Europea ad East Asa laguages. I ths paper our focus s POS taggg for Urdu due to the facy stage of Urdu laguage based taggg system. We have combed two approaches (Statstcal ad morphologcal edg based techque) to assg the approprate sytactc categores, usg Urdu laguage corpus as our expermetal data. The process of our taggg system cossts of two dfferet stages. I frst stage, we apply statstcal based word level laguage model to compute the probablty ad assg the approprate tag to the word. I secod stage, we extract all the ambguous ad ukow words the corpus ad apply morphologcal edg based rules to resolve these dscotutes, whch arse f statstcal model fals to assg the approprate tag to a gve word. The developmet of ths tagger s a tal step toward the Urdu POS taggg. The expermetal results of the tagger show that the performace of the ukow word s mproved whe we add morphologcal edg based features wth statstcal model. Evaluato method employed shows the sgfcace of expermetal results ad the effectveess of morphologcal edg o statstcal method. Ths s a poeerg work towards buldg a POS tagger for Urdu laguage through Morphologcal edg based strateges Key words: Urdu Laguage, Morphologcal Based Edg Model, Probablstc Model, Part-of-speech Taggg.. Itroducto Research automatc text taggg s a fudametal ad mportat task atural laguage processg, whch s cosdered to be solved for most of the corpus rch Laguages. O the other had uder-prvlege resources laguage the problem of automatc text taggg s stll ts tal stage especally Urdu laguage. The POS taggg provdes prelmary step towards buldg varous atural laguage processg applcatos e.g. Iformato extracto, parsg, word sese dsambguato, text-to-speech coverso etc I Eglsh ad other smlar laguages may mportat results have bee acheved by applyg statstcal based methods. However for the Laguages whch exhbt dfferet behavor tha Eglsh lke Urdu, Hd, Turksh ad Czech, may of the successful techques appear capactated whle solvg the taggg problem [9],[0]. Several dfferet approaches have bee used for automatc part of speech taggg. Geerally t ca be classfed to two dfferet groups: rule based approach ad statstcal approach. But most of the free word ad hghly flected laguages the researcher solve dsambguato problem through hybrd approach. Because morphologcal edg based approach play a vtal role to solve ukow word problem. Eve at some pot the researcher use frst edg based techque ad the use statstcal based techque. Here we wll dscuss some of the tal work about features based techque as secodary strategy wth combato of statstcal method. Itally (Churcg, 989) [2] used the captalzato features to recogze the ukow proper ous that occur durg trag process. I (Meteer et al, 99) pot out that the accuracy rate ad specfcally ukow words accuracy sgfcatly mprove after addg morphologcal edg based Kowledge. They defe dfferet features such as flecto, dervatoal edgs, hypheato ad captalzato. The word edg characterstcs vares from laguage to laguage ad requred laguage kowledge. They also report that the morphologcal edg of the words reduces the error rate o the ukow word by a factor of 3[4]. I (Adams ad Nefeld, 993) arbtrary character edgs s appled alog wth some other mportat laguage features such as captalzato ad puctuato [2]. Also there taggg system uses exteral lexco that s prepared wthout statstcal formato but has POS tag wth every word to assg the correct tag to the word. I (Adam el al, 994) motvates the dea that Mauscrpt receved May, Mauscrpt revsed Jue 5, 2007.

2 68 IJCSES Iteratoal Joural of Computer Sceces ad Egeerg Systems, Vol. No.3, July 2007 prmarly use the edg based strategy ad secodary use whole-word statstcs based approach [],[2]. Oe feels that laguages lke Urdu has ot bee studed -depth to corporate statstcal methods as compared to other laguages. Urdu s the wdely spoke laguage South Asa. It s morphologcally rch ad free word laguage. Though, the part of speech taggg Urdu laguage s stll ts facy stage. The reasos for workg o part of speech taggg Urdu laguage are the followg: The amout of avalable aotated lgustcs resources for Urdu laguage s much smaller tha for well researched laguages. No well-defed Urdu part of speech tag set s avalable that cotas all the formato. Comprehesve Urdu part of speech lexco s very dffcult to fd. All the above meto reasos are major hdrace Urdu laguage processg research ad accout for the fact, why Urdu laguage processg s lackg behd [6]. I fact Urdu s hghly flected laguage ad the lmtatos of the statstcal based methods may be recompesed wth the stregth of rule based methods. Aother problem wth part of speech taggg of Urdu laguage s how to utlze ad alter exstg techques to reap maxmum effcecy ad to detfy the best method to be used cojucto wth lmted aotated corpora. Keepg ths vew we have developed a edg strateges based o statstcal model to fully utlze the correspodg features of Urdu laguage. The ma cotrbuto of ths work s to detfy the techques ad laguage models that are approprate for desgg a Urdu POS taggg system. To the best of our kowledge we are the frst to deal wth POS taggg task for Urdu laguage through ths techque. I ths paper, we wll descrbe morphologcal edg based part-of-speech taggg model for Urdu laguage. The frst step of our model s to compute word level statstcs. The secod step of our model s to add the morphologcal edg-based features. However, our statstcal model s based o three methods (ugram, bgram, backoff). Durg the mplemetato of our statstcal model we observed that the basele algorthm of our tagger does ot performed well. Due to the lmtato statstcal mode ad laguage characterstcs we combe our statstcal based model wth morphologcal edg based features. The Expermetal results of our tagger show that the better performace ca be acheved by combg statstcal model wth morphologcal features stead of pure statstcal model. By applyg these morphologcal features the overall performace crease up to.2% but ukow word error rate s sgfcatly decreased. Ths paper s orgazed as follow: Secto 2 gves a bref overvew of Urdu laguage, Secto 3 descrbes the taggg model (ugram, bgram ad backoff).i ths secto we also descrbes the detal of our two taggg algorthms. Secto 4 presets the expermetal results. Secto 5 presets the model evaluato. Cocluso ad future work are preseted at the ed 2. A Bref Overvew of Urdu Laguage Urdu s a dervatoal word from Turksh ad ts mea horde (Lashkar). Urdu, a Ido-Europea laguage of the Ido Arya famly, s spoke Ida ad Paksta. It s the atoal laguage of Paksta havg eleve mllo speakers. Amog all the laguages the world t s most closely smlar to Hd laguage. Urdu ad Hd both have orgated from the dalect of Delh rego ad other tha the mute detals these laguages share ther morphology. Sce Hd has adopted may words from Saskrt, Urdu has borrowed a large umber of vocabulary tems from Persa ad Arabc Urdu s also borrowg umber of vocabulary from Turksh, Portuguese ad Eglsh. May of the Arabc words have bee borrowed by Urdu laguage through Persa laguage. These words vary slghtly ther toe, cootatos ad feelg. Oe of the oteworthy aspects of Urdu grammar costtuto s ts word order SOV (subject, object, ad verb). Ths order does exhbt some flexblty as the subject proous are frequetly dropped [2]. 3. Laguage Model I atural laguage processg statstcal laguage model s employed to calculate ad assg a probablty Pw ( ) to every word ad to decde the accurate target word sequece w= w, w2,... w. Let s say that gve a partcular word sequece w we wat to fd the most probable tag sequecet. I our algorthm we compute the probablty of dvdual word. Weather a gve tag t s

3 Morphologcal Edg based Strateges of Ukow Word Estmato for Statstcal POS Urdu Tagger 69 approprate for a gve word w ca be aswered by fdg the probablty Pwt (, ). P( wt, ) = P( w) P( t w) () P( t w ) = P( t, t2,... t t0, w, w2,... w) = t0, w, w,... w). 2 t, t0, w, w2,... w)... t... t0, t, w, w2,... w)... pt ( t,..., t, t0, w, w2,... w) = t0, w). 2 t, w2)... t, w)... t, w) P( t w ) = P( t t, w) (2) = I equato () Pws ( ) the ucodtoal probablty ad detcal for all tag sequeces ad that s why we gored Pw. ( ) w) s the codtoal probablty of the occurreces of sequece t gve that a sequece of word w occurred. I ths case we ca make the assumpto that each tag deped oly the mmedate precedg tag wth curret word ad curret tag (bgram model). We called ths s a Markov assumpto. The model use ths paper s dfferet from the oe use [4]. I [4] the authors used the geeratve model to solve the dsambguato problem. The Geeratve dsambguato s a model of the jot probablty P (W,T),where W represets the words ad T represets the specfc tag set as put. I ths model we make the assumpto through Bayes rules to compute the P (T W) ad to fd out most probable tag sequece T.[3] O the other had our model s based o Dscrmatve model or codtoal probablty. I ths model we ca compute P (T W) drectly Because W s already kow as put. The other researchers also pot out covcg causes regardg the use of dscrmatve rather tha geeratve model. I fact, f we leave the ssue of ukow word hadlg whch s more mportat dsambguato problem the dscrmatve dsambguato model s mostly preferred over geeratve dsambguato model. [3] The word sequece probablty value ca be calculated from the tagged corpus whch s obta from trag corpus usg, w) w) (3) Cw ( ), t, w) t, w) (4), w) * t t = arg max P( t w ) (5) t = arg max P( t t w ) (6) * t Cw ( ) Occurreces of the word w Ctw (, ) Occurreces of tag t tagged wth w ) Occurreces of the tag t, t2) Occurreces of the tag bgram t, t 2 tw ) Occurreces of the tag-tag word bgram tw ) that s the tag t followed by the tag t word wth w I our approach ugram tagger s based o covetoal statstcal taggg algorthm whch commts a tag for each toke that s most probable for the word questo. Whle determg the probablty of the most probable tag for a toke we must take to cosderato the relatve frequeces of the tags desgated to the toke. These data set are ot mache tagged rather ths procedure requres huma terveto. Addtoally maxmum lkelhood estmato s used to tag our taggers..i case of ugram taggg the optmal dctoary lst out all words w, w2,..., w ad ther correspodg tags t, t 2,... t.the words ad tags are dcated by w adt. Thus ma task of the tagger s to fd out the best Part of speech tag * t wth maxmzg the codtoal probablty w ) [], [3] [5], [7]. Whle ugram model requred most probable tags, bgram model eeds dual formato. It collects tags for the curret word ad ts precedg tag to calculate the cotext for the toke. Ths cotext s ter used by the tagger to assg most probable tag. Maxmum lkelhood estmato s also employed to tag the data. Table shows buld up of a dctoary wth bgram model. Ths dcates prevous tag wth curret word agast curret word. Ths dctoary s optmal because t cludes all the word ad correspodg tags wth the avalable corpus. 3. Back-off model The Backoff model essetally overcomes the sparse occurreces problems the trag corpus. Due to the lack of eough amout of trag corpus word frequeces dstrbuto s msrepreseted.[8] As we already metoed our bgram laguage model s classfed by the probablty of curret tag based o curret word

4 70 IJCSES Iteratoal Joural of Computer Sceces ad Egeerg Systems, Vol. No.3, July 2007 ad prevous tag. I our algorthm case of bgram model, f the order of word par (cotextual formato) s ot foud wth the defte cotext the trag corpus, the bgram tagger s backed off to the ugram tagger. Also, f the order of the word has too sparse occurrece, the ugram tagger s backed off to the default tagger.the default tagger specfy the etre toke that are ambguous ad ukow. After that, by default assg the same tag to every toke that tag s specfy by the user []. Thus, the Bgram tagger probablty for a word w s defed as: Ctt ( w) Ctt ( w) > U ( ) Ct w) t w = Otherwse w) Procedure Bgram Back-off ( ) U: = Step: Compute Ctt ( w) Step 2: Compute w) Step 3: for to f Ctt ( w) > U, t, w) t, w) =, w) else t w) = w) Ed for Step 4: t * = arg max P( t t w) t Step 5: stop Procedure Our proposed backoff tagger gves better effcecy to the trag corpus by elmatg detcal tag smlar cotext calculated by bgram ad ugram backoff tagger durg trag process. Ths eables bgram tagger to keep mmal umber of staces. Ths umber ca also be user defed as cutoff value []. The cutoff techque of trag corpus prug elmates those ugrams ad bgrams that occur rarely wth the laguage model [8] 3.2 Morphologcal edg strateges-based model I frst stage of our tagger, the statstcal model does ot successfully tag all the words specally ukow words. Also some of the kow words are wrogly tagged. Urdu s oe of the partally free word ad hghly flected laguage. Most of these types of laguages should use orthographc rules to resolve the problem dscussed above. Lgustcally the flectoal chage drectly flueces to the word edg. I secod stage of our taggg system we apply morphologcal edg based strateges to resolve the ukow ad ambguous words those ot hadled by statstcal method. The morphologcal edg based method cossts of umber of feature set. I features set we defe all the sytactc categores rules. Such as, the ou ca be categorzed to dfferet type of character edgs. ),(و, wa o ),(يا, ya ),(وں, w ),(ی, ye ),(ع, a ),(اں, a ),(ہ, he ),(ا, alf ).[ 5 ](ں, u Smlarly we also defe all the other sytactc categores rules ad covert these rules to model form. [6].These rules also cover all the sytactcally affected flecto e.g. ou ca be flected as umber geder ad case. I fgure 3 we defe some of the most frequet edg that has occurred durg taggg process. Each rule s descrbed as character edg wth approprate tag. The character edg sze s ot fxed ad all the edg depeds o the Urdu laguage grammar characterstcs. Durg the model buldg process we extract all the ambguous ad ukow word that has occurred durg the statstcal based model taggg process. After extractg all the words we evaluate all the words edg oe by oe. I morphologcal rules mplemetato the sequece of rules play a mportat role because the sequece of rules depeds o the most frequet error occurrg tags. I our taggg system the frequet error occurrg tags are ou, adjectve, verb etc.the process of choosg the most approprate tag depeds o how well the rules are defed. But whe we mplemet these rules to our tagger the frst we estmate morphologcal edg wth o-frequet error occurrg tags ad at last we estmate the most-frequet error occurrg tags. After ths process we decde that whch tag s most approprate for each word. f Codto s true P( t codto) = 0 Otherwse 4. Expermets I the expermet, we use EMILLE corpus to test our results. The EMILLE corpus was released o 2004 by Lacaster Uversty. Ths corpus has three ma parts (moologal, aotated, parallel) [2].We evaluates the performace of our taggg system to compare two dfferet techques (statstcal ad statstcal wth combato of morphologcal edg). The result of our tagger depeds o three stadardze measures (Precso, Recall, F-measure) [4] ad these measures gve us comprehesve results about tagger performace.

5 Morphologcal Edg based Strateges of Ukow Word Estmato for Statstcal POS Urdu Tagger 7 correct umber of toke tag par occurrece Precso = total umber of toke tag par correct umber of toke tag par occurrece Recall = umber of correct toke tag par that s possble 2*Precso*Recall F-measure = Precso+Recall As we metoed above that our tagger s based o two techques. Fgure ad Fgure 2 demostrate the learg bars usg dfferet sze of trag corpus. The learg bars of the Fgure ad Fgure 2 represet the overall accuracy uder dfferet expermetal codtos (ugram, bgram, backoff) wth morphologcal ad wthout morphologcal edgs respectvely. The otable aspect both Fgures s that after applyg morphologcal edg based strateges the overall performace of the tagger creases to approxmately.2%. The expermetal results suggest that partally free word ad hghly flected laguages, the laguage features has a very mportat role. So geerally speakg morphologcal edg based strateges have better complace tha pure statstcal method. I case of the pure statstcal based model the accuracy rates of ugram ad bgram crease from 79% to 94.30% ad 6.50% to 88.50% respectvely. Here, the backoff model overcomes the sparse data problem ad the accuracy of the tagger s creased to 95.0%.Becuase trag data s ot suffcet eough so the ukow word accuracy of the pure statstcal based model s ot hgher as the case of other laguages. We overcome ths performace degradg drawback by combg the Urdu morphologcal edg based kowledge. BY usg morphologcal based kowledge cojucto, the accuracy rates of ugram ad bgram crease from 90% to 95.0% ad 8% to 93% respectvely. Also whe we apply backoff model through combato of morphologcal edg based strateges the accuracy creases from 90.60% to 96.3%. The ma cotrbutor overall accuracy mprovemet through morphologcal edg based strateges s the laguage features set. I Table 3 we represet the comprehesve result comparso betwee both of the models. I ths table we represet the Overall Precso, Overall Recall, F-Measure, ad kow overall Precso. Ukow word hadlg wth dfferet laguage features s aother mportat characterstc of our tagger. Fgure 3 shows the error rate of ukow word. I ths Fgure we cosder two aspects: oe s sze of trag corpus ad the other s laguage features. We apply the laguage feature set step by step. If we compare pure statstcal method ad statstcal method wth oly ou features there s a bg error rate dfferece. After addg the ou features we troduce more features whle creasg the sze of trag corpus at the same tme. Fgure 3 also shows that after addg all features the ukow word error rate s sgfcatly reduced.

6 72 IJCSES Iteratoal Joural of Computer Sceces ad Egeerg Systems, Vol. No.3, July 2007 R 0 / R [ ( )] *00 r0 = r Where R 0 s error dex for gve model relatve to the base model (statstcal). The Model wth ou features despte 33.44% decrease error relatve to base model. Smlarly model wth (Nou + Adjectve) features ad model wll all features has gve 39.64% ad 48.36% reducto ukow error relatve to statstcal model respectvely 5. Evaluato I table 4 ad table 5 we llustrate that there are some sgfcat dffereces betwee three models. Sgfcat dffereces are determed by usg pared sample t-test. d* t = sd Follows the t-dstrbuto wth degree of freedom d ( d d) Where d = ad sd = Thus, we obta the followg results show table 4 ad table 5 wth level of sgfcat 0.05.The pared t-test results show the comparso of the statstcal based (ugram, bgram,backoff) techque ad morphologcal edg based (ugram, bgram, backoff) techque uder dfferet expermetal data. These results show that there s sgfcat dfferece amog the three models whe we use these models statstcal techque ad morphologcal edg based techque. I case of backoff model the results are gradually creasg at every pot. Eve the values of ugram ad backoff model have o large dffereces but our t-test result shows that both models ( case of statstcal ad morphologcal edg based techques) are hghly sgfcat. 2 Table 6 shows the average Geometrc mea (G.M) of ukow error relatve to statstcal model. Error dces have bee calculated usg uweghted dex umbers to fd the mprovemet models usg laguage features. G.M as a average gves the average of relatves 6. Coclusos I ths paper, we have proposed combato of two methods for assgg approprate part-of-speech tag to

7 Morphologcal Edg based Strateges of Ukow Word Estmato for Statstcal POS Urdu Tagger 73 word level toke Urdu laguage. Also we have compared two dfferet part-of-speech taggg approaches for Urdu laguage, oe statstcal-based ad the other s statstcal wth morphologcal edg based techque. We have tested ad aalyzed these techques our expermets. Although ths s the frst attempt to apply such approaches for Urdu laguage, yet both of the taggg methods performed well. As a result, our morphologcal edg based approach outperformed purely statstcally based approach uder the gve test codtos. We also ote that due to the reasos of hgh flecto, partally free word order laguage features ad lack of Urdu laguage corpus, purely statstcal based taggg approach dd ot perform well. We have observed that the accuracy of the tagger ca be mproved by addg more features morphologcal edg based model. [3] A. Y. Ng, ad M. I. Jorda, "O Dscrmatve vs. Geeratve Classfers: A Comparso of Logstc Regresso ad Nave Bayes", Advaces Neural Iformato Processg Systems (NIPS), Vol. 4, [4] J.Trommer, ad D. Kallull. "A Morphologcal Tagger for Stadard Albaa", I Proceedgs of LREC,2004 [5] M. Humayou, "Urdu Morphology,Orthography ad lexco Extracto", M.S Thess,Chalmers Uversty of Techology ad Goteborg Uversty, Swede,2006. [6] Y. M. A, H. D. Lm, ad Y. H. Seo, "Korea Part-of-Speech Taggg Based o Cotext Iformato", Idustral Electrocs, Proceedgs. ISIE, 200. Refereces [] S. Brd, E. Kle, ad E. Loper, Itroducto to Natural Laguage Processg, Uversty of Pesylvaa, [2] A. Harde, "Developg a Tagset for Automated Part-of- Speech Taggg Urdu",Corpus Lgustcs coferece. Departmet of Lgustcs, Lacaster Uversty,2003. [3] D. Jurafsky, ad J.H.Mart, A Itroducto to Natural Laguage Processg, Computatoal Lgustcs, ad Speech Recogto, Pretce-Hall, [4] M.Meteer, R.Schwartz, ad R.Weschedel, "Studes Part of Speech Labelg", Huma Laguage Techology Coferece,99. [5] J. Carlbergre, ad V. Ka, "Implemetg a Effcet Part-Of-Speech Tagger", Software: Practce ad Experece, Vol. 29, No. 9, 999, pp [6] C. H. Chag; C. D. Che, "HMM-Based Part-of-Speech Taggg for Chese Corpora", I Proceedgs of ACL-93 Workshop o Very Large Corpora USA, 993. [7] S. Brd, E. Kle, ad E. Loper, Natural Laguage Toolkt (NLTK) [8] Rosefeld, "Scalable Backoff Laguage Models", I Proc. ICSLP'96, Phladelpha, October 996. [9] Z. Dlek, Hakka-Tür, K. Oflazer, G. Tür, "Statstcal Morphologcal Dsambguato for Agglutatve", Computers ad Humates, Vol. 36, 2000, pp [0] C. Oravecz, ad P. Dees, "Effcet Stochastc Part-of- Speech Taggg for Hugara", Proceedgs of the Thrd Iteratoal Coferece o Laguage Resources ad Evaluato, LREC, [] E. Neufeld, ad G. Adams, "Part-of-Speech Taggg from "Small" Data Sets". Ffth Iteratoal Workshop o Artfcal Itellgece ad Statstcs. [2] G. Adams, B. Mllar, E. Neufeld, ad T. Phlp, "Edgbased Strateges for Part-of-Speech Taggg", Ucertaty Artfcal Itellgece(UAI), 994, pp. -7