A Method for Calculating Term Similarity on Large Document Collections

Transcription

1 $ A Method for Cacuating Term Simiarity on Large Document Coections Wofgang W Bein Schoo of Computer Science University of Nevada Las Vegas, NV bein@csunvedu Jeffrey S Coombs and Kazem Taghva Information Science Research Institute University of Nevada Las Vegas, NV jscoombs, Abstract We present an efficient agorithm caed the Quadtree Heuristic for identifying a ist of simiar terms for each unique term in a arge document coection Term simiarity is defined using the Expected Mutua Information Measure (EMIM) Since our aim for defining the simiarity ists is to improve information retrieva (IR), we present the outcome of an experiment comparing the performance of an IR engine designed to use the simiarity ists Two methods were used to generate simiarity ists a brute-force technique and the Quadtree Heuristic The performance of the ist generated by the Quadtree Heuristic was commensurate with the brute force ist 1 Background To faciitate the retrieva of OCR documents, the Information Science Research Institute has begun construction of a retrieva system caed Hairetes [7] Hairetes enhances a traditiona retrieva system by incorporating the technique of Retrieva by Genera Logica Imaging (RbGLI) as deveoped by Crestani and Van Rijsbergen [1, 2, ] The basic idea of RbGLI is to retrieve not ony those documents which contain terms in a query, but aso documents which contain terms that are simiar to those in the query but do not contain the query terms themseves In standard retrieva systems ony terms that actuay appear in a query can be used to determine which documents shoud be retrieved Thus, if we enter a query such as Find a documents that discuss magnetic fied intensity tests from fauts at Yucca Mountain, ony documents which contain at east one occurrence of one of the query terms coud be retrieved The aim of RbGLI is to expand the query so that documents The work of this author was supported by ISRI during summer 2002 which do not contain any of the query terms woud sti receive some weight if they contain terms which are simiar to query terms For OCR texts in particuar, we hope that documents which contain terms that are misrecognized by the OCR process might sti be retrieved if the correct counterpart to a misrecognized term is found to be simiar to a misrecognized form Since a user is ikey to enter the correct counterpart, it woud be usefu if the retrieva woud aso pu up documents which contain the misrecognized term We have foowed Crestani in seecting EMIM, the Expected Mutua Information Measure, as our measure of word simiarity [2] EMIM, we wi see in Section 2, requires counting the co-occurrences of terms in a document Since counting co-occurrences is computationay very expensive, we propose a heuristic to find EMIM-simiar terms in a arge document coection in Section 3 The resuts of experiments are reported in Section These indicate that the agorithm performs we when compared to resuts obtained by brute-force computation 2 The Expected Mutua Information Measure We consider a coection of documents which contain keys, where we assume that documents and keys are indexed by natura numbers, ie the set of documents is and the set of keys is For each key, we define the the key-document incidence vector, "!#%$&, where for '(, *) '*+ -, if occurs in ' otherwise The tabe beow shows an exampe of two key-document incidence vectors and /

2 O + R f O Š $ 0 / the occur- For term 12, we define the quantities 35 rence of, and 376 the non-occurrence of 37 9,; '( occurs in '< for = > '( does not occur in '< for =?$ Furthermore, for two terms A@ occurrences C ED F, for D = /G %$& C HDF D 'I 0 ) '*+ =KJL/ ) '*+ MG B, we define mixed co- The vaue CN D is the number of documents in which the two terms both appear and is caed the co-occurrence of the two terms We define the Expected Mutua Information Measure (EMIM) as QP* ) 0@ R C EDF D<UWVXY[Z TS 6 FS 6 O (We note that in the definition of `_ acbed*f QP C EDF D 3 3 ]\ F (1), there sometimes is an extra factor ^ present; however within one data set ^ is a constant, see aso van Rijsbergen [3]) (1) Both and are arge numbers, in practice we might g $h O ; cacuating QP* efficienty is expect therefore non-trivia We can assume that for key i1, the quantity 3j is O known Then, from Tabe 1 it is cear, that to cacuate QP* 0@ + for A@ k, ony the co- ) occurrence C is needed, as a other vaues can be cacuated by simpe additions once C] is known Of course, D D determining CN is a non-trivia task for a arge document D coection and represents the primary difficuty for using EMIM as the simiarity measure for generating simiar term ists for each term in the inverted fie of a typica document retrieva system C D C5m6 D 3j C 6e D Cn66 D 3K6 Tabe 1 The Contingency Tabe It can be assumed that a documents have been preprocessed such that the foowing two queries can be performed in o ) UTVX qp UWVX r+ time 3j 3K6 1 For any Ls, provide a pointer to a ist of a distinct documents, which contain key 2 For any 'I, provide a pointer to a ist of a distinct keys, which are contained in document ' Such preprocessing can be accompished by setting up two B-trees, one for each query type; this takes time o )0) qpt r+ ) UTVX qp UWVX r+0+ For very arge data coections such preprocessing is tedious, abeit not impossibe (as compared with a brute-force o )0) upb + Y ) We are interested in preprocessing the data further to faciitate an efficient (ogarithmic run-time) approximation of the foowing EMIM query v Given a constant w(x pointers to keys y such that yz{ are the keys with highest EMIM to, for a chosen B, provide To appy RbGLI, we require a ist of w simiar terms for each of the terms in a document coection, where w wi tend to be a sma number (w = 5 in experiments) These w terms wi be the most simiar to a given term in the exicon of our retrieva engine in the sense of having the highest EMIM reative to the given term In the next Section we show a heuristic that gives such simiar terms efficienty 3 The Quadtree Heuristic For our heuristic we appy the foowing preprocessing steps to the data coection 1 Find a representative set of reference keywords words } }~, where >o ) UTVX + 2 For each key, create the reference incidence vector ƒ ) &+ ) ƒ ) &+ ƒ ~ ) &+0+, where ƒ QP* ) } +, for = > 3 For each ) = G +, = Gi =r ˆG Initiaize an empty two-dimensiona quadtree with granuarity HD F 6 g The parameter Š is chosen in such a way that 6 there are no more than w eements at the eaves the tree Seect integer, $[Œ Œ{ ŽM 1 5 For each, insert the best pairs (best according to ƒ p ƒ vaue) F ) ƒ ) &+ ƒ FA ) &++,, ) ƒ ) &+ ƒ FA ) &+0+ into their respective quadtrees DFA *DFm We note that these steps can be carried out using the same o )0) qpt r+ ) UTVX qp UWVX r+0+ time compexity as required by the usua preprocessing described in Section 2 for the B- trees setup The co-occurrence query is impemented as foows 2

3 Average Average Reca Precision Interpoated Precision Average Point 3-Point Precision Precision Interpoated Tabe 2 Average Precision and Reca (Top); Overa Averages for Brute Force (Bottom) Average Average Reca Precision Interpoated Precision Average Point 3-Point Precision Precision Interpoated Tabe 3 Average Precision and Reca (Top); Overa Averages for Quadtree Heuristic (Bottom) 1 For key, find the two highest vaues in the reference incidence vector; et = 6 and G 6 be the corresponding indices 2 In the quadtree, find a keys T DFm square of ƒ T ) <+ and ƒ Fm ) <+ 3 From that are in the extract the w best keys If fewer than w are found repeat with new pair =, 6 G 6 We note that the actua query step has o ) UWVX up UTVX +0+ run-time for each key queried Experiments To determine if the Quadtree Heuristic provides a reasonabe ist of simiar terms, the foowing experiment was performed We compared the ists of simiar terms generated by a brute-force method with the Quadtree Heuristic in a retrieva engine designed to use such ists If the two ists performed simiary, this gives some support to the idea that the faster Quadtree Heuristic is the better method for generating the ists of simiar terms A vector-space retrieva system using the cosine measure as defined by Witten [11] was buit using Berkeey DB data structures [6] The document coection indexed consisted of 1055 OCR documents from the Licensing Support Network (LSN), a coection of OCR documents donated to the Information Science Research Institute by the Department of Energy for research purposes [10] The tota number of pages in the 1055 coection was 75,236, and the average number of pages was 71, see [] Simiar term ists were generated in two ways The first used a brute-force approach which compared each unique term in the exicon of the retrieva engine with every other distinct term EMIM was cacuated for each pair, and for each term, the five terms with highest EMIM reative to it were seected for its simiarity ist As this process was very time consuming, we computed EMIM ony for terms occurring in three or more documents The second method used the Quadtree Heuristic (Section 3) to generate the simiarity ists We seected 100 reference keywords randomy from a ist of terms which had a document frequency between 20 and 150 The idea was that the most effective reference keywords woud fa in the cass of medium frequency terms as these had been shown to be most usefu for document retrieva [5] In both cases the top five EMIM-simiar terms were added to the retrieva system A query on the retrieva system was processed as foows Each term in the query was stemmed using a Porter stemmer, and stop words were removed For every document which contained a query term, a weight was cacuated using the cosine measure defined in Witten [11] In addition to this standard way of cacuating document 3

4 y + weights, the weights from terms in s simiarity ist were factored in Let through %š be the terms in s simiarity ist For every document ' which contained one of the simiar terms but not, a fraction of s cosine weight was added to the document s weight This fraction is caed the opinionated factor by Crestani [1] We cacuated the opinionated factor of a simiar term as wœ ) =5 G (2) FS where w was the tota number of terms in the simiarity ist, and = ž ( = w ) was the position of in the ist Since we set w >Ÿ for our experiments, the simiar terms through š woud respectivey add š š, š, š, Y š, and š of the cosine weight to the documents in which they appeared Forty queries were submitted to the retrieva engine augmented by the simiarity ist These queries had been deveoped by researchers famiiar with the LSN document coection [] An exampe of one such query was Find a documents that discuss magnetic fied intensity tests from fauts at Yucca Mountain Five geoogists then formuated reevancy judgments to determine which documents were most reevant to a given query The standard measures of retrieva effectiveness, precision and reca, were cacuated based on these reevancy judgments [] Their percentages were averaged over the forty queries and the resuts presented in Tabes 2 and 3 The first coumn of the top part in each Tabe ists the percentage of the reca vaue The reca vaue is the number of reevant documents retrieved at a given point divided by the tota number of reevant documents Next to the reca vaues arrayed in the standard 11-point dispay appears the average percentage of the precision at the 11 reca points Precision is the number of reevant documents retrieved divided by the tota number of documents retrieved In the 11-point dispay, the precision represents the maximum precision achieved within a given reca range For exampe, in the first row Œ of the top part of Tabe 3, within the range of 0% reca 10%, 5555% was the average percentage of the maximum precision achieved in this range At reca point 0% in the same Tabe, on the average a maximum of 31% precision was achieved At the reca point 100%, the precision at exacty the point when 100% reca was reached is given Interpoated precision takes the maximum precision for a given interva and a higher reca eves [11] Since there are eeven points from 0% to 100% in the reca coumn, the average of a eeven precision vaues is the 11-point average This vaue is typicay used to compare performances of retrieva systems In addition, a 3-point vaue which averages the precision at reca points 20%, 50%, and 0% is another common measure The 11-point and 3-point averages of precision in both its standard and interpoated form are provided for both experiments Comparing the overa averages isted in the bottom of Tabes 2 and 3 show that the simiarity ists created by the Quadtree Heuristic performed as we as those produced by the sower brute-force approach The Quadtree Heuristic required ony a few hours to generate its simiarity ist on a Sun Bade 1000 machine whie the brute-force required severa days 5 Concusion We note that in the formuation of the Quadtree Heuristic of Section 3 there are a number of choices depending on the specific document coection appication A good choice of representative reference keywords is crucia In our experiments we used a choice based on medium frequency ; however, choices based on the specifics of the coection are ikey to improve precision Other choices invove picking good vaues for and Š 6 The resuts obtained here are for a reativey sma coection However, we expect for our method to scae up favoraby Experiments are under way to appy the Quadtree Heuristic to a number of rea-word coections avaiabe at ISRI [9] References [1] Fabio Crestani and CJ Van Rijsbergen A study of kinematics in information retrieva ACM Transactions on Information Systems, , 199 [2] Fabio Crestani, Ian Ruthven, M Sanderson, and CJ van Rijsbergen The troubes with using a ogica mode of ir on a arge coection of documents experimenting retrieva by ogica imaging on TREC In Proceedings of the Fourth Text Retrieva Conference (TREC-), 1995 [3] C J Van Rijsbergen A theoretica basis for the use of co-occurrence data in information retrieva Journa of Documentation, 33(2) , June 1977 [] C J Van Rijsbergen A non-cassica ogic for information retrieva The Computer Journa, 291 5, 196 [5] G Saton, H Wu, and CT Yu The measurement of term importance in automatic indexing Journa of the American Society for Information Science, pages , May 191 [6] Seepycat Software Berkeey DB New Riders, 2001 [7] Kazem Taghva and Jeffrey Coombs Hairetes A search engine for OCR documents In Proc of 5th

5 IAPR Int Workshop on Document Anaysis Systems, Lecture Notes in Computer Science, pages 12 22, Princeton, NJ, August 2002 Springer-Verag [] Kazem Taghva, Thomas Nartker, Juie Borsack, and Aen Condit Determining the usefuness of manuay assigned keywords for a vector space system In Proc of the IEEE Int Conf on Information Technoogy Coding and Computing, pages 22 26, Las Vegas, NV, Apri 2002 IEEE Computer Society [9] Kazem Taghva, Tom Nartker, Juie Borsack, and Aen Condit UNLV-ISRI document coection for research in OCR and information retrieva In Proc IS&T/SPIE 2000 Int Symp on Eectronic Imaging Science and Technoogy, San Jose, CA, January 2000 [10] Kazem Taghva, Tom Nartker, Juie Borsack, Steve Lumos, Aen Condit, and Ron Young Evauating text categorization in the presence of OCR errors In Proc IS&T/SPIE 2001 Int Symp on Eectronic Imaging Science and Technoogy, pages 6 7, San Jose, CA, January 2001 [11] I Witten, A Moffat, and T Be Managing Gigabytes Compressing and indexing documents and images Morgan Kaufmann, 2nd edition,