Cross-Language Information Retrieval

Transcription

1 Feature Artcle: Cross-Language Informaton Retreval 19 Cross-Language Informaton Retreval Jan-Yun Ne 1 Abstract A research group n Unversty of Montreal has worked on the problem of cross-language nformaton retreval (CLIR) for several years. A method that explots parallel texts for query translaton s proposed. Ths method s shown to allow for retreval effectveness comparable to the state-of-the-art effectveness. A major problem of ths approach s the unavalablty of large parallel corpora. To solve ths problem, a mnng system s constructed to automatcally gather parallel Web pages. The mnng results are used to tran statstcal translaton models. When a query s translated word by word, the accuracy may be low. In order to ncrease the translaton accuracy, compound terms are extracted and ncorporated nto the translaton models, so that compounds can be translated as a unt, rather than as separate words. Our experments show that ths can further ncrease the CLIR effectveness. I. INTRODUCTION NFORMATION retreval (IR) tres to dentfy relevant I documents for an nformaton need, expressed as a query. The problems that an IR system should deal wth nclude document ndexng (whch tres to extract mportant ndexes from a document and wegh them), query analyss (smlar to document ndexng), and query evaluaton (.e. matchng the query wth the documents). Each of these problems has been the subject of many studes n IR. Tradtonal IR dentfes relevant documents n the same language as the query. Ths problem s referred to as monolngual IR. Cross-language nformaton retreval (CLIR) tres to dentfy relevant documents n a language dfferent from that of the query. Ths problem s more and more acute for IR on the Web due to the fact that the Web s a truly multlngual envronment. In addton to the problems of monolngual IR, CLIR s faced wth the problem of language dfferences between queres and documents. The key problem s query translaton (or document translaton). Ths translaton rases two partcular problems [6]: the selecton of the approprate translaton terms/words, and the proper weghtng of them. In the last few years, researchers have worked on these problems ntensvely. Three man technques for query translaton have been proposed and tested: - Wth an on-the-shelf machne translaton (MT) system; - Wth a blngual dctonary; - Or wth a set of parallel texts. 1 Département d'informatqueet Recherche opératonnelle, Unversté de Montréal, C.P. 6128, succursale Centre-vlle, Montréal, Québec, H3C 3J7 Canada, ne@ro.umontreal.ca The frst two approaches are qute straghtforward. We wll not gve detals about them. Our research efforts have been concentrated on the thrd approach. Ths approach s promsng because t does not requre extensve manual preparaton (n comparson wth the constructon of an MT system); and ts translaton s usually more approprate than wth a blngual dctonary. The major advantages of ths approach are the followng ones: The tranng of a translaton model can be completely automatc. No (or lttle) manual preparaton s requred. The resultng translaton model reflects well the word usage n the tranng corpus. Ths offers the possblty to tran specalzed and up-to-date translaton models. In ths paper, we wll descrbe our approach to CLIR based on parallel texts, as well as some experments. The paper wll be organzed as follows. In Secton II, we wll frst descrbe brefly the tranng process of statstcal translaton models on a set of parallel texts. Then we wll descrbe n Secton III the IR system we use for our experments. Secton IV descrbes our experments wth the translaton models traned on a manually prepared parallel corpus. Secton V descrbes our approach to mnng parallel Web pages, as well as ther utlzaton for CLIR. Secton VI presents our utlzaton of compound terms n CLIR. Fnally, we present our conclusons n Secton VII. II. TRAINING STATISTICAL TRANSLATION MODELS ON PARALLEL TEXTS Let us frst descrbe brefly the tranng of statstcal translaton models on a set of parallel texts. These models wll be used n our experments. Statstcal translaton models are traned on parallel texts. A par of parallel texts s two texts whch are translaton one of the other. Model tranng tres to extract the translaton relatonshps between elements of the two languages (usually words) by observng ther occurrences n parallel texts. Most work on the tranng statstcal translaton models follows the models (called IBM models) proposed by Brown et al. [1]. In our case we use the IBM model 1. Ths model does not consder word order n sentences. Each sentence s consdered as a bag of words. Any word n a correspondng target sentence s consdered as a potental translaton word of any source word. Ths consderaton s oversmplfed for the purpose of machne translaton. However, for IR, as the goal of query translaton s to dentfy the most probable words wthout consderng the syntactc features, ths smple IEEE Computatonal Intellgence Bulletn June 2003 Vol.2 No.1

2 20 translaton model may suffce. In order to tran a translaton model, parallel texts are usually decomposed nto algned sentences,.e. for each sentence n a text, we determne ts translaton sentence(s) n the other language. The prmary goal of producng sentence algnment s to reduce the scope of translaton relatonshps between words: nstead of consderng a word n a source text to correspond potentally to every word n the target text, one can lmt ths relatonshp wthn the correspondng sentences. Ths allows us to take full advantage of the parallel texts and to produce a more accurate translaton model. A. Sentence Algnment Sentence algnment tres to create translaton relatonshps between sentences. Sentences are not always algned nto 1:1 pars. In some cases, one sentence can be translated nto several sentences, and the sentence may even be deleted or a new sentence may be added n the translaton. Ths adds some dffcultes n sentence algnment. Gale & Church [5] propose an algorthm based on sentence length. It has been shown that ths algorthm can successfully algn the Canadan Hansard corpus (the debates n the Canadan House of Commons n both Englsh and French), whch s rather clean and easy to algn. However, as ponted out by Smard et al. [12] and Chen [3], whle algnng more nosy corpora, the methods based solely on sentence length are not robust enough to cope wth the above-mentoned dffcultes. Smard et al. proposed a method that uses lexcal nformaton, cognates, to help wth algnment [12]. Cognates are pars of tokens of dfferent languages, whch share obvous phonologcal or orthographc and semantc propertes, wth the result that they are lkely to be used as mutual translatons. Examples are generaton/génératon and fnanced/fnancé for Englsh/French. In a wder sense, cognates can also nclude numercal expressons and punctuaton. Instead of defnng a specfc lst of cognates for each language par, Smard et al. gave language-ndependent defntons on cognates. Cognates are recognzed on the fly accordng to a seres of rules. For example, words startng wth 4 dentcal letters n Englsh and French are consdered as cognates. Another method ncorporates a dctonary [3]. The translatons contaned n the dctonary serve as cues to sentence algnment: a sentence s lkely to algn wth another sentence f the latter contans several dctonary translatons of the words of the former. In our mplementaton, we use the approach of Smard et al. [12]. B. Model Tranng The prncple of model tranng s: n a set of algned sentences, f a target word f often co-occur wth a source word e n the algned sentences, then there s a hgh chance that f s a translaton of e,.e. the translaton probablty t(f e) s hgh. The tranng algorthm uses dynamc programmng to Feature Artcle: Jan-Yun Ne determne a probablty functon t(f e) such that t maxmzes the expectaton of the gven sentence algnments (see [1] for detals). We brefly descrbe the tranng for IBM model 1 as follows. The translaton probablty functon t s determned such as to maxmze the probablty of the gven sentence algnments A of the tranng corpus. Suppose a sentence algnment e f, and that the sentences e and f are composed of set of words as follows: e={e 1, e 2, e 3,..., e l }, f ={f 1, f 2, f 3,..., f m } where l and m are respectvely the length of these sentences. Then the functon t s determned as follows: t = arg max p( A) t e f = arg max p( f e) t = arg max ε(1 + l) t m m e f j= 1 = 0 l t( f j e ) where ε s the probablty that an Englsh sentence of length l can be translated nto a French sentence of length m, and t(f j e ) the word translaton probablty of e by f j. The probablty t can be determned by applyng the teratve EM (Expectaton maxmzaton) algorthm. We do not gve detals here. Interested readers can refer to [1]. IBM model 1 consders every word n the target sentence to be equvalently possble translaton of any word n the source sentence, regardless to ther poston and to the fertlty of each word (e.g. an Englsh word may be translated by one or more French words). It s obvous that the translaton model does not learn syntactc nformaton from the tranng source and thus cannot be used to obtan syntactcally correct translatons. However, the model s able to determne the word translaton probablty t between words, and ths fts the need of cross-language nformaton retreval of fndng out the most mportant translaton words. III. IR SYSTEM In our experments we use the SMART system. SMART s an IR system, developed at Cornell Unversty [2]. The ndexng process consders every token as an ndex. Indexes are weghted accordng to the tf*df weghtng scheme 2. Ths s a common way to wegh the mportance and specfcty of a term n a document. The prncple s as follows: 1) the more a word occurs n a document, the more t s mportant. Ths s the tf factor. On the other hand, the more there are documents contanng the word, the less the word s specfc to one partcular document. In other words, the word does not allow dstngushng a document from the others. Therefore, the weght of the word s lowered. Ths s the df factor. More precsely, the two factors are measured as follows: 2 tf = term frequency, and df = nversed document frequency. June 2003 Vol.2 No. 1 IEEE Computatonal Intellgence Bulletn

3 Feature Artcle: Cross-Language Informaton Retreval 21 tf ( t, D) = log( freq( t, D) + 1); N df ( t) = log( ) n( t) where freq(t, D) s the frequency of occurrences of the word/term t n the document D; N s the total number of documents n the collecton; n(t) s the number of documents contanng t. The retreval process follows the vector space model [2]. In ths model, a vector space s defned by all the tokens (words or terms) encountered n the documents. Each word/term represents a dstnct dmenson n ths space. Then a document, as well as a query, s represented as a vector n ths space. The weght n a dmenson represents the mportance of the correspondng word/term n the document or query (the tf*df weght). The degree of correspondence between a document and a query s estmated by the smlarty of ther vectors. One of the commonly used smlarty measures s as follows: d q sm( D, Q) = 2 2 d q where d and q are respectvely the weghts of a term n the document D and n the query Q. IV. EXPERIMENTS WITH THE HANSARD MODELS There are a few manually constructed parallel corpora. The best known s the Canadan Hansard, whch contans the debates of the Canadan parlaments durng 7 years, n both French and Englsh. It contans dozens of mllons words n each language. Such a parallel corpus s a valuable resource that contans word/term translatons. Our frst experments are carred out wth translaton models traned on the Hansard corpus- we call the resultng models the Hansard models. We used two test collectons developed n TREC 3, one n Englsh (AP) and the other n French (SDA). Both collectons contan newspaper artcles. The SDA contans 141,656 documents, and AP 242,918 documents. We use two sets of about 30 queres, avalable n both French and Englsh. These queres have been used n TREC6 and TREC7 for French- Englsh CLIR. The queres have been manually evaluated (.e. we know ther relevant documents). Table I shows the CLIR effectveness obtaned wth these translaton models. F- E means usng French queres to retreve Englsh documents,.e. the French queres are frst translated nto Englsh, then the Englsh translaton s used to match the documents. In all our experments, we select the 25 most probable translaton words as the translaton of a query. In Table I, the effectveness s measured by average precson,.e. the average of the precsons over 11 ponts of recall. Ths s a standard measure used n IR. We also show 3 TREC: Text Retreval Conference, a seres of conferences amng to test IR systems wth large document collectons. See the percentage of the CLIR effectveness wth respect to the monolngual IR effectveness (%mono). In comparson wth the state-of-the-art effectveness, whch s usually around 80-90% of the monolngual effectveness (see the reports of TREC at the results we obtaned are qute comparable. TABLE I. AVERAGE PRECISION USING HANSARD MODEL F-E (%mono) E-F (%mono) Trec (74.8%) (67.9%) Trec (97.6%) (93.6%) V. MINING OF PARALLEL WEB PAGES A major problem to use parallel texts s often the unavalablty of large parallel corpora. In order to obtan such corpora, we constructed a mnng system PTMner [4] to automatcally gather parallel Web pages. Although many parallel Web pages exst on the Web, t s not obvous to dentfy them and to confrm that a par of pages s truly parallel. In our mnng approach, we explot several heurstc features. For example, f an Englsh page ponts to another page wth an anchor text French verson or verson françase, ths s a useful ndcaton that the second page s a French verson of the frst page. Although these ndcatons are not fully accurate, and they can produce errors, we wll show later n our experments that a nosy parallel corpus s stll useful for query translaton n CLIR. In the followng subsectons, we wll brefly descrbe our mnng approach. A. Automatc Mnng Parallel web pages often are not publshed n solaton. Most of the tme, they are connected n some way. For example, Resnk [11] observed that parallel Web pages often are referenced n the same parent ndex web page. In addton, the anchor text of such lnks usually dentfes the language. For example, f a home page ndex.html contans lnks to both Englsh and French versons of the next page, and that the anchor texts of the lnks are respectvely Englsh verson and French verson, then the referenced pages are parallel. In addton, Resnk assumes that parallel Web pages have been ndexed by large search engnes exstng on the Web. Therefore, n hs approach, a query of the followng form s sent to Alta Vsta n order to frst retreve the common ndex page: anchor: englsh AND anchor: French Then the referenced pages n both languages are retreved and consdered to be parallel pages. We notce that only a small number of web stes are organzed n ths way. Many other parallel pages do not satsfy ths condton. Our mnng strategy uses dfferent crtera. In addton, we also ncorporate an exploraton process (host crawler) n order to dscover more web pages IEEE Computatonal Intellgence Bulletn June 2003 Vol.2 No.1

4 22 that have not been ndexed by the exstng search engnes. Our mnng process s separated nto two man steps: frst dentfy as many canddate parallel pages as possble, then verfy external features and contents to determne f they are parallel. Our mnng system s called PTMner (for Parallel Text Mner). The whole process s organzed nto the followng steps: 1. Determnng canddate stes Ths step tres to dentfy the Web stes where there may be parallel pages. 2. Fle name fetchng It dentfes a set of Web pages from each Web page that are ndexed by search engnes. 3. Host crawlng It uses the URLs collected n the last step as seeds to further crawl each canddate ste for more URLs. 4. Par scannng by names It pars the Web pages accordng to the smlarty of ther URLs. IDENTIFICATION OF CANDIDATE WEB SITES To determne canddate stes, we assume that a canddate ste contans at least one page that refers to another verson of the page, and the anchor text of the reference clearly dentfes the language. For example, an Englsh Web page contans a lnk to the French verson, and the anchor text s French verson, n French, en franças and so on. So to determne the canddate stes, we send a partcular request to search engnes askng for Englsh pages that contan a lnk wth an anchor text dentfyng another language such as: anchor: french verson, [n french,...] language: Englsh The host addresses we extract from the resultng Web pages correspond to the canddate stes. FILE NAME FETCHING To search for parallel pars from each canddate ste, PTMner frst asks the search engnes for all the Web pages from ths ste they have ndexed. Ths s done by a query of the followng form: host: <hostname> However, a search engne may not ndex all the Web pages on a ste. To obtan a more complete lst of URLs from a ste, we need to explore the stes more thoroughly by a host crawler. HOST CRAWLING A host crawler s slghtly dfferent from a Web crawler or a robot [10] n that a host crawler only explots one Web ste. A breadth-frst crawlng algorthm s used n ths step. The prncple s that f a retreved Web page contans a lnk to an unexplored document on the same ste, ths document s added to a lst that wll be explored later. Ths crawlng step allows us to obtan more web pages from the canddate stes. PAIR SCANNING BY NAMES We observe that many parallel pages have very smlar fle Feature Artcle: Jan-Yun Ne names. For example, an Englsh web page wth the fle name ndex.html often corresponds to a French translaton wth the fle name ndex_f.html, ndex_fr.html, and so on. The only dfference between the two fle names s a segment that dentfes the language of the fle. Ths same observaton also apples to URL paths. In some cases, the two versons of the web page are stored n two dfferent drectores, for example, vs. So n general, a smlarty n the URLs of two fles s a good ndcaton of ther parallelsm. Ths smlarty s used to make a prelmnary selecton of canddate pars. FILTERING AFTER DOWNLOADING The remanng fle pars are downloaded for further content verfcaton accordng to the followng crtera:. Length of the pages: A par of parallel pages usually has smlar fle lengths. A smple verfcaton s then to compare the lengths of the two fles. Note that the length rato changes between dfferent language pars. HTML structure: Parallel web pages are usually desgned to look smlarly. Ths often means that the two parallel pages have smlar HTML structures. Therefore, the smlarty n HTML tags s another flterng crteron. The par-scannng crteron we used only explots the name smlarty of parallel pages. Ths s not a fully relable crteron. Fles wth a segment en_ may be not n Englsh. Therefore, a further verfcaton s needed to confrm that the fles are n the requred languages. In our system, we use the SILC4 system for an automatc language and encodng dentfcaton. Wth PTMner, we have been able to collect several parallel corpora from the Web. Table II shows some of them. TABLE II. SIZES OF THE WEB CORPORA FR-EN DE-EN IT-EN # Text Pars Raw data (MB) Cleaned data (MB) In our further descrpton, we wll concentrate on the French-Englsh par. B. CLIR Wth the Web Models Translaton models are traned on the set of parallel Web pages as descrbed n Secton 2, except that some preprocessng has to be performed on these pages n order to remove HTML tags. Once translaton models (n both drectons) are traned, they are used to produce 25 most probable translaton words that are consdered as the translaton of a query. Table III descrbes the CLIR 4 See June 2003 Vol.2 No. 1 IEEE Computatonal Intellgence Bulletn

5 Feature Artcle: Cross-Language Informaton Retreval 23 effectveness wth the Web models. TABLE III. AVERAGE PRECISION USING WEB MODEL F-E (%mono) E-F (%mono) Trec (72.6%) (70.4%) Trec (74.3%) (71.5%) In comparson wth the Hansard model, we see that the Web models perform slghtly worse. However, consderng the nose that ths tranng corpus may contan, ths effectveness s qute good. It s stll close to the state-of-theart effectveness. Ths test shows that the automatcally mned parallel Web pages are greatly useful for CLIR. VI. INCORPORATING COMPOUND TERMS IN TRANSLATION MODELS In the prevous approach, parallel texts have been exploted to fnd translatons between sngle words. The most obvous problem we can see s that by takng words one by one, many of them become ambguous. The translaton model wll then suggest several translatons correspondng to dfferent meanngs of the word. For example, the word nformaton (n French) wll have many possble translatons because 1) the word denotes several meanngs; 2) t appears very frequently n the parallel corpus. Among the possble translatons, there are nformaton, ntellgence, esponage, etc. However, f the term we ntend to translate s système d nformaton (nformaton system), and f the term s translated as a whole, then many of the meanngs of nformaton can be elmnated. The most probable translaton of ths term wll be the correct term nformaton system. Through ths example, we can see that a translaton model that ntegrates the translaton of compound terms can be much more precse. Ths s the goal of our utlzaton of compounds durng query translaton. To do ths, we have to tran a translaton model that ncorporates compound terms as addtonal translaton unts to words. So compound terms are frst extracted from the tranng parallel corpus, and added to the orgnal sentences. Then the same translaton process s launched. The resultng model contans the translatons for both sngle words and compound terms. To dentfy compound terms, we use both a large termnology database contanng almost 1 mllon words and terms, and an automatc extractor of compound terms. The extractor uses syntactc structures, together wth a statstcal analyss. Frst, word sequences correspondng to predefned syntactc templates are extracted as canddates. If the frequency of occurrences of a canddate s above a certan threshold, then the sequence s consdered as a compound term. The frst problem s the defnton of the syntactc templates. Ths s done manually accordng to the general knowledge on syntactc structures of a language. Usually the extracton s restrcted to noun phrases. For example, the followng template s used n the tool we used - Exterm: ((NC AJ) )*((NC AJ) NC PP) ((NC AJ) )*NC where NC means a common noun, AJ an adjectve, and PP a preposton. Of course, a POS (Part-Of-Speech) taggng s necessary n order to recognze the syntactc category of each word. The tagger we used s a statstcal tagger traned on the Penn Treebank 5. It tres to determne the most probable syntactc categores that ft the best the words of a sentence. Detals on the tranng of such a tagger can be found n [7]. All the terms and words n documents, queres and the tranng parallel corpus are submtted to a standardzaton process on words, as follows: Nouns n plural are transformed nto sngular form (e.g. systems system); Verbs are changed nto nfntve form (e.g. retreves retreve, retrevng retreve); Artcles n a term s removed (e.g. the database system) For example, the expresson adjusted the earnngs wll be transformed nto adjust earnng. Once a compound term s recognzed n a document or a query, t s added nto the document or query. For example, consder a preprocessed text as follows: arm dealer prepare relef supply to sovet unon From ths segment, we can extract two stored terms arm dealer and sovet unon So the followng terms are appended to the orgnal text: arm_dealer sovet_unon Once compound terms are extracted from the tranng texts, the corpus s submtted to the tranng process of translaton models descrbed n Secton 2. However, as compounds are consdered as unts of the texts, the resultng translaton models wll also contan translatons for the compounds, whch are usually more accurate than ther wordby-word translatons. A. Experments on CLIR Table IV shows the CLIR results wth both types of translaton model. These results are obtaned on the same document collecton as the one used earler, but the query set s dfferent. In these experments, we separate sngle words and compound terms nto two separate vectors. SMART has the flexblty of buldng multple vectors for a document and for a query. Then the global smlarty between the document and the query s determned by the weghted sum of the smlartes between the vectors. One can assgn a relatve weght to dfferent vectors of the query to balance ther mportance n the global smlarty. In our experments, we tested several values for the relatve weghts of the sngle-word vector and the compoundterm vector. The above results are obtaned wth the relatve 5 IEEE Computatonal Intellgence Bulletn June 2003 Vol.2 No.1

6 24 Feature Artcle: Jan-Yun Ne mportance of 0.3 to the compound-term vector, and 1 to the sngle-word vector. Ths assgnment gves the best result. We can see a great mprovement n CLIR effectveness once the translaton model ncorporates compound terms, especally for the F-E case. We have not appled the same approach to the Web corpus. However, we could expect smlar mprovements wth the Web corpus when compound terms are ncorporated. TABLE IV. THE CLIR EFFECTIVENESS WITH DIFFERENT MODELS. Word Compounds (change) F-E on AP data set (+76.86%) E-F on SDA data set (+26.72%) VII. CONCLUSIONS In ths paper, we descrbed an approach based on parallel texts that has bee used for CLIR at Unversty of Montreal. Globally, our experments show that the statstcal translaton models traned on parallel texts are hghly useful for CLIR. They can acheve comparable effectveness to the state-ofthe-art approaches. Our further tests wth the parallel Web pages mned automatcally show that we can arrve at a reasonable level of effectveness despte the relatvely hgh rate of nose n the tranng parallel Web pages. Ths seres of experments show that our method based on parallel Web pages s sutable for CLIR. Nevertheless, we also observe several aspects that requre mprovements: We encounter problems for translatng proper names. Proper names are often treated as unknown words, and are added nto the translaton as t s. For some names, the spellngs n all the European languages are the same, whch does not rase partcular problems. For some others wth dfferent spellngs (e.g. Bérégovoy n French, but Beregovoy n some Englsh documents), ths smple approach does not solve the problem. Translaton by common but non stop- words: Very often, among the top translaton words, the common words such as prendre and donner ( take and gve n French) appear wth qute strong probablty. The mned parallel Web pages contan a certan amount of nose. To mprove the translaton accuracy, a further flterng of nose s necessary. We are currently nvestgatng on these problems. [4] J. Chen, J.Y. Ne. Automatc constructon of parallel Englsh-Chnese corpus for cross-language nformaton retreval. Proc. ANLP, pp , Seattle (2000). [5] W. A. Gale, K.W. Church, A program for algnng sentences n blngual corpora, Computatonal Lngustcs, 19: 1, (1993). [6] G. Grefenstette. The Problem of Cross-Language Informaton Retreval. In Cross-language Informaton Retreval. Kluwer Academc Publshers. pages 1-9, 1998 [7] C. Mannng, H. Shultze, Fundamentals of Statstcal Natural Language Processng, MIT Press, 1999 [8] J.Y. Ne, P. Isabelle, M. Smard, R. Durand, Cross-language nformaton retreval based on parallel texts and automatc mnng of parallel texts from the Web, ACM-SIGIR conference, Berkeley, CA, pp (1999). [9] J.Y. Ne, J.F. Dufort, Combnng Words and Compound Terms for Monolngual and Cross-Language Informaton Retreval, Informaton 2002, Bejng, July [10] Prosse J., Crawlng the Web, A gude to robots, spders, and other shadowy denzens of the Web, PC Magazne - July 1996 ( [11] Resnk, Phlp (1998) Parallel stands: A prelmnary nvestgaton nto mnng the Web for blngual text, AMTA'98, Lecture Notes n Artfcal Intellgence, 1529, October. [12] M. Smard, G. Foster, P. Isabelle, Usng Cognates to Algn Sentences n Parallel Corpora, Proceedngs of the 4 th Internatonal Conference on Theoretcal and Methodologcal Issues n Machne Translaton, Montreal (1992). REFERENCES [1] P. F. Brown, S. A. D. Petra, V. D. J. Petra, and R. L. Mercer, The mathematcs of machne translaton: Parameter estmaton. Computatonal Lngustcs, vol. 19, pp (1993). [2] Buckley, C. (1985) Implementaton of the SMART nformaton retreval system. Cornell Unversty, Tech. report [3] Chen, S. F. Algnng sentences n blngual corpora usng lexcal nformaton. Proc. ACL, pp. 9-16, June 2003 Vol.2 No. 1 IEEE Computatonal Intellgence Bulletn