Selecting the Most Highly Correlated Pairs within a Large Vocabulary

Transcription

1 Seleting the Most Highl Correlted Pirs within Lrge Voulr Koji Umemur Deprtment of Computer Siene Toohshi Universit of Tehnolog Astrt Ourene ptterns of words in douments n e epressed s inr vetors When two vetors re similr, the two words orresponding to the vetors m hve some impliit reltionship with eh other We ll these two words orrelted pir This report desries method for otining the most highl orrelted pirs of given sie In prtie, the method requires omputtion time, nd memor spe, where is the numer of douments or reords Sine this does not depend on the sie of the voulr under nlsis, it is possile to ompute orreltions etween ll the words in orpus 1 Introdution In order to find reltionships etween words in lrge orpus or etween lels in lrge dtse, we m use distne mesure etween the inr vetors of dimensions, where is the numer of douments or reords, nd the th element is 1 if the th doument/reord ontins the word or the lel, or otherwise There re severl distne mesures suitle for this purpose, suh s the mutul informtion(churh nd Hnks, 199), the die oeffiient(mnning nd Shueute 85, 1999), the phi oeffiient(mnning nd Shuete 5, 1999), the osine mesure(mnning nd Shueute 85, 1999) nd the onfidene(arrwl nd Sriknt, 1995) There re lso speil funtions for ertin pplitions, suh s then omplimentr similrit mesure (CSM)(Hgit nd Swki, 1995) whih is known s to e suitle for ses with nois pttern All of these five mesures n e otined from simple ontingen tle This tle hs four numers for eh word/lel nd word/lel The first numer is the numer of douments/reords tht hve oth nd We define this numer s The seond numer is the numer of do- uments/reords tht hve ut not We define this numer s The third numer is the numer of douments/reords tht do not hve ut do hve We define this numer s The fourth nd the lst numer is the numer of douments/reords tht hve neither nor We define this numer s An ovious method to otin the most highl relted pirs is to lulte!,,, for ll pirs of words/lels, ompute the similrit for ll pirs nd then selet pirs of the highest vlues Let " e the numer of possile words/lels, nd e the totl numer of douments/reords in orpus/dtse This method requires "# memor spe nd " #$ omputtion time However, its use is onl fesile if " is smller thn!( When " is lrger thn ten thousnd, eeution of this proedure eomes diffiult The method desried here is sed on the oservtion tht there is n upper oundr to the numer of different words in one doument The ssumption of suh oundr ould even mde of lrge sle

2 , orpus For emple, olletive orpus of newspper m eome lrger nd lrger, ut the length of eh rtile is stle It is not likel tht one rtile would ontin thousnds of different words In view of this oservtion nd the ssumption, this method is effetive for otining the most highl orrelted pirs in lrge orpus, nd uses memor spe, nd omputtion time 2 Nottions Severl nottions re introdued in this setion to desrie the method Assuming orpus C, whih is set of sets of words, vlues re ssigned s follows : douments (elements of the orpus),, : lel (elements of doument) : is pled fter in the lphetil order : the totl numer of douments : the numer of douments tht ontin : the numer of douments tht on- tin nd ontin tin ut not tins ut not : the numer of douments tht on- : the numer of douments tht one- tin neither nor : the numer of douments tht on- Prolem Definition When the orpus of set of sets of lels is provided, " nd the funtion! of pir of lels to the numer in the following form is lso provided, we will otin " : the set of pirs of given sie tht stisfies the following ondition # # # where # "%$ '& (& "%$) '&'(& The following re emples of -, / osine funtion 1 -,2 -,2 die oeffiient 4, -, ,52 onfidene,,2 pirwise mutul informtion, phi oeffiient 1 -,2 -,2, 87 -,2, -,2 +* 2 omplementr similrit mesure, ,2 2 # # 2

3 & "# Implementtion of progrm tht requires " # memor spe nd omputtion time is es A progrm of this tpe ould e used to lulte,,, nd for ll pirs of nd, nd ould then provide the most highl orrelted pirs However, ompution with this method is not fesile when " is lrge For emple, in order to lulte the most highl orrelted words within newspper over severl ers of pulition, " eomes roughl, nd eomes The mount of omputtion time is then inresed to 4 Approh In terms of the tul dt, the numer of orrelted pirs is usull muh smller thn the numer of unorrelted pirs Moreover, most of the unorrelted! pirs usull stisf the ondition: &, nd re not of interest This method tkes this ft into ount Moreover, it lso uses the reltionship etween nd to mke the omputtion fesile 5 Reltionship etween nd Proofs of the following equtions re provided elow 7 7! 7 7 '2 Proof: 1 " " is equivlent to " 2 B definition, the sum of,,, nd lws represents the totl numer of douments 2 2 '2 Similrl, the sum of tht ontin This equls nd is the numer of douments Similrl, the sum of tht ontin This equls nd is the numer of douments '2 2 5 These four equtions mke it possile to epress,, nd nd These formuls indite tht the numer of required two-dimensionl tles is not four, ut just one In other words, if we rete tle of nd one vrile for, we n otin,, nd 6 The memor requirement for Let e the mimum numer of different words/lels in one doument The following prop- ert eists in #,

4 The left side of the formul equls the totl numer of ll pirs of words/lels # This nnot eeed This reltionship indites tht if the tle is stored using tuples of where &, the required memor spe is Tuples where & re not neessr euse we know tht & when the tuple for does not eist in memor This estimtion is pessimisti The tul sie of # the tuples will e smller thn, sine not ll douments will hve different words/lels 7 Otining, nd, nd is strightforwrd First, the orpus must e trsformed into set of sets of words/lels Sine this is set form, there re no duplitions of the words/lels of one doument In the following progrm, the hshtle returns for non-eistent item The lgorithm to otin! (1) Let DFA e empt hshtle (2) Let DF e empt hshtle () Let N e The omputtion time for this progrm is less thn # Sine is independent from, the omputtion time is # Agin, is pessimisti estimtion, sine not ll douments will hve different words/lels 8 Seleting Pirs Even though,,, nd n e otined in onstnt time fter preproessing, there re " # vlues to onsider to otin the est N orrelted pirs Fortuntel, mn of the funtions tht re usle s inditors of orreltion nd, t lest, ll five funtions, return lower vlue thn the known threshold if & The osine mesure, the die oeffiient, nd pirwise mutul informtion hve propert 1 nd propert 2 s defined elow This implies tht the vlue for where & is tull the minimum vlue of ll Therefore, the first prt of the totl ordered sequene of is the sorted list of where & The rest is n ritr order of pirs where & Propert 1: the vlue is not negtive Propert 2: when &, the vlue is & The phi oeffiient nd the omplementr similrit mesure hve the following properties 1, 2 nd Therefore, the first prt of the totl ordered sequene where the vlue is positive, is equl to first prt of the sorted list where & nd the vlue is positive Moreover, this list ontins ll pirs tht hve positive orreltion This list is long enough for the tul pplition Propert 1: when! + &, the vlue is negtive Propert 2: when nd re not orrelted, the (4) For eh doument, ssign it to D estimted vlue is & (5) N = N + 1 (6) For eh word in D Propert : when nd tend to pper t the (7) ssign the word to X sme time, the estimted vlue is positive (8) For eh word in D (9) ssign the word to Y It should e relled tht the numer of pirs (1) DFA(X, Y)=DFA(X,Y)+1 where! # & is less thn The sorted (11) end of loop list is otined in # # om- (12) end of loop (1) end of loop puttion time, where is the mimum numer of different words/lels in one doument Sine is onstnt, it eomes, even if the sie ofvoulr is ver lrge It is true tht for the given some fied voulr of sie " #, might e lrger thn " # s we inrese the sie of orpus Fortuntel, the tul memor onsumption of this proedure lso hve the upper ound of " #, nd we will not loose n memor spe When " is not fied nd " m eome ver lrge ompre to # proper nouns, is smller thn " # s is the se for 9 Cse stud of Newspper Corpus The omputtion time of the seline sstem is " # where " is the distint numer of lels in the

5 % & time(se) speed(se/do) Tle 1: The tul eeution time shows liner reltionship to the sie of input dt orpus When we nled lels of nmes of ples in newspper over the ourse of one er, this orpus onsisted of out 6, douments The ple nmes totlled 192 fter morphologil nlsis The mimum numer of nmes in one doument ws 142, nd the verge in one doument ws 42 In this se, the method desried here, ws muh more effiient thn the seline sstem Tle 1 shows the tul eeution time of the progrm in the ppendi, hnging the length of the orpus This progrm omputes similrit vlues for ll pirs of words where & It indites tht the eeution time is liner Our oservtion shows tht even if the orpus were etended er er, whih is the mimum numer of different words in one doument is stle, even though the totl numer of words would inrese with the ongoing ddition of proper nouns nd new onepts 1 For lrge orpus Although the progrm in the ppendi nnot e pplied to orpus lrger thn memor sie, we n otin tle of using sequentil ess to file The progrm in the ppendi stores ever pir in # memor The spe requirement of m seem too gret to hold in memor However, sequentil file n e used to otin the tle, s follows Although the omputtion time for is rther thn, the totl omputtion time remins the sme euse omputtion of is required to selet pirs in oth Eh line orre- ses Consider the following dt sponds to one doument When the pirs of words in eh doument re reorded, the following file is otined Note tht sine!, it is not neessr to reord pirs where This redues the memor requirement Using the merge sort lgorithm whih n sort lrge file using sequentil ess onl, the file n e sorted in omputtion time After sorting in lphetil order, sme pirs ome together Then, the pirs n e ounted with sequentil ess, there providing the! tle An emple of this tle fllows: It should e noted tht the tle n e otined esil etrting lines in whih letter of the first olumn nd tht of the seond olumn re the sme, sine The tle n usull e stored in memor sine it is one dimensionl rr After storing in memor, similrit n e omputed line line The following emple uses the

6 phi oeffiient The first olumn is the oeffiient, followed,,,, nd Sine the phi oeffiient is refletive, the vlue where is not required When the funtion is not smmetri, nd n e ehnged t the sme time The ordered list n e otined sorting this tle with the first olumn This emple shows tht pirs where &, suh s or, do not dd n overhed to either memor or omputtion time 11 Comprison with Apriori There is well known lgorithm for forming list of relted items termed Apriori(Arrwl nd Sriknt, 1995) Apriori lists ll reltionship using onfidene, where is lrger thn speified vlue Using Apriori, the! threshold n e speified in order to redue omputtion, wheres with the proposed method, there is no w to djust this threshold This implies tht Apriori m e fster thn our lgorithm in terms of onfidene However, sine Apriori uses the propert of onfidene to redue omputtion, it nnot e used for other funtions, unlike the proposed method whih n emplo mn stndrd funtions, t lest the five mesures used here inluding onfidene 12 Correltion of All Sustrings When omputing orreltions of ll sustrings in Sine the orpus, " n e s lrge s 7 % memor spe requirement nd omputtion time does not depend on ", this method n e used to generte list of the most hightl orrelted sustrings of n length In ft, in some ses, m e too lrge to ompute Churh, 21) llows for the retion of The Ymmoto-Churh method(ymmoto nd tle using memor spe nd omputtion time, where represents ll sustrings in given orpus Ymmoto s method shows tht lthough there m e 7 % kinds of sustrings in orpus, there is ourene ptterns (or sets of sustrings whih hve sme ourene pttern) t most The omputtionl ost is gretl redued if we del with eh pttern insted of eh sustring Although the order of omputionl ompleit does not depend on ", differs whether the pttern is used or not We hve lso developed sstem using the pttern whih tull redues the ost of omputtion Although the numer of is still prolemti even using the Ymmoto- Churh method, nd lthough the omputtion ost is muh lrger thn using words, the progrm runs muh fster thn the simple method 1 Conlusion This pper desries method for seleting orrelted pirs in memor spe nd omputtion time, where is the numer of douments in orpus, provided tht there is n upper oundr in the numer of different words/lels in one doument/reord We hve oserved tht orpus usull hs this kind of upper oundr, nd hve shown tht we n uses sequentil file for most of our memor requirements This method is useful not onl for onfidene ut lso for other funtions whose vlues re deided,,, Emples of these funtions re mutul informtion, the die oeffiient, the onfidene mesure, the phi oeffiient nd the omplimentr similrit mesure Referenes K W Churh nd P Hnks 199 Word ssoition norms, mutul informtion nd leiogrph Computtionl Linguistis, 16(1):22 29 R Agrwl nd R Sriknt 1995 Mining of ssoition rules etween sets of items in lrge dtses In Proeedings of the ACM SIGMOD Conferene on Mngement of Dt:94 15 N Hgit nd M Swki: 1995 Roust reognition of degrded mhine-printedhrters using omplimentr similrit mesure nd error-orretion lerning Proeedings of the SPIE - The Interntionl Soiet for Optil Engineering 2442:24 244

7 U M Fd, G Pitetsk-Shpiro, nd P Smth The KDD Proess for Etrting Useful Knowledge from Volumes of Dt Communitions of the ACM, 9(11):27 4, Christopher D Mnning nd Hinrih Shuete, 1999 Chpter 85, Semnti Similrit Foundtions of sttistil nturl lnguge proessing:294, The MIT Press Christopher D Mnning nd Hinrih Shuete, 1999 Chpter 5, Person s hi-squre test Foundtions of sttistil nturl lnguge proessing: , The MIT Press Mikio Ymmoto nd Kenneth W Churh 21 Using Suffi Arrrs to Compute Term Frequen nd Doument Frequen for All Sustring in Corpus Computtionl Linguistis, 27(1):1, The MIT Press Appendi Smple of DATA Tkruk Toko Okinw Ytushiro Hiroshim Kurihr Onomihi Yokohm Toko Ihihr Toko Aiuwkmtu Fukushim Toko Mtumoto Shr Utsunomi Toko Aomori Shimokit Smple Progrm(smwk) 1 # Definition of similrit - 2 # Complimentr Similrit Mesure funtion f(,,, d) { 4 return ( * d - * ) / sqrt(( + ) * ( + d)); 5 } 6 # For eh line, ount up oth df() nd df(, ) 7 { for(i=2; i<= NF; i++) { 8 df[$i]++; 9 for(j=i;j<= NF; j++) { 1 df[$i, $j]++; 11 } 12 } 1 } 14 # For ll (, ) where df(, )<>, get the vlue 15 END{ 16 for(k in df) { 17 split(k,, SUBSEP); 18 if([1]!= [2]) { 19 = df[k]; 2 = df[[1]] - ; 21 = df[[2]] - ; 22 d = NR ; 2 r = f(,,, d); 24 printf("%16f\t%s\t%s\n", 25 r, [1], [2]); 26 r = f(,,, d); 27 printf("%16f\t%s\t%s\n", 28 r, [2], [1]); 29 } } 1 } Usge: $ wk -f smwk < mitt sort -nr hed -2