Clustering documents with vector space model using n-grams

Transcription

1 Clusterg documets wth vector space model usg -grams Klas Skogmar, Joha Olsso, Lud Isttute of Techology Supervsed by: Perre Nugues, Abstract Ths paper descrbe a method to cluster documets usg the lear space algorthm together wth ugrams, bgrams, trgrams ad -grams a attempt to ehace the clusterg performace compared to ugrams oly. Our tests dd ot reveal a mprovemet, but shows that the techque has a potetal, especally certa specfc areas lke fdg ctatos or versos of the same documet. The extra effort requred for mplemetato ad the speed loss could make t less terestg however.

2 Itroducto Ths paper s the result of a project doe for a course laguage processg at Lud Isttute of Techology. The project was madatory, but the topc of the project was optoal. We chose to use our ewly acqured kowledge o the beefts of -grams versus ugrams, but o a dfferet area, amely: documet clusterg. We wated to use basc stadard algorthms for everythg, because we wated to focus o the dffereces that -grams versus ugrams could make for the accuracy of clusterg. Therefore we chose the vector space model for determg smlartes betwee documets, ad k-meas for clusterg. We expected that the beefts of our suggested soluto would be that t ehaced the accuracy of clusterg, that t worked wth all avalable solutos (wth some mor modfcatos), ad that t gave better retrevg of keywords (whch we wll ot cosder ths paper). The accuracy should crease documets, where smlar words occur, but whe the orderg s dfferet. A example s Avod a wrtte door code ad code wrtte to avod the Troja back door, where they would be cosdered qute smlar usg ugrams, but ot as smlar usg bgrams or trgrams. I the example above Troja back door s a example of the beefts of trgrams over bgrams. The proposed algorthm wll ot gve as may beefts laguages where words are compoud words, as Swedsh ( cotrast to Eglsh). As far as we kow, most exstg algorthms oly use ugrams, although the advatages of -grams have prove most successful may laguage processg areas. The problem Clusterg s the ablty to automatcally group smlar documets, oly usg the text the documet tself. To do ths oe eeds to determe the smlarty betwee documets. There s also a eed to create referece documets whe combg two or more documets. Clusterg s doe may areas of computer scece, ad there are a lot of dfferet techques, lke algorthms, eural etworks, geetc algorthms, et cetera. The dfferet techques used deped o the applcato area. There are may applcatos for clusterg documets. Some examples are: Iteret search eges, kowledge maagemet systems or documet databases. Smlartes betwee documets could be used for searchg the local computer for documets or automatcally acqurg meta-data from documets for use verso maagemet systems. Usually clusterg of documets ca be combed wth several other techques to ehace the clusterg accuracy. These techques ca be: determg grammar, omttg commo words or puttg weghts to the words. Our soluto We chose to use the k-meas clusterg algorthm. Ths algorthm requres a dstace value betwee documets. For ths we used oe of the most wdely

3 used techques for determg smlartes: the Vector space model. The reaso for the choce of these algorthms s that they are very commo ad therefore kow by most computer lgusts. The fact that they are well documeted make them easy to mplemet ad easy to read for others. Also, we wated to focus o the beefts of -grams compared to ugrams. We chose to mplemet the project Java. I addto to the fact that t s object oreted ad that we are famlar wth the laguage, we could also show our progress usg a Applet o the Iteret. Sce ths s a commoly used laguage, others wll be able to copy our source code ad modfy t for ther ow purposes. Sce we had used regular expressos the course, we thought t would be good to clude that fuctoalty ths project as well. Ths makes the parsg of the documets very flexble. We used Java s stadard mplemetato of regular expressos (cluded sce Java.4). Vector space model The vector space model s wdely used for determg smlarty betwee documets [5]. Ths method sees each documet as a vector a word-space. The dmeso (umber of axs) of ths word-space s the umber of dfferet words the two documets. The umber of occurreces of each word determes the legth of that axs for that documet. The the vector space model determes the agle (cose coeffcet) or some other value (for example Jaccard or Dce coeffcets) [4] betwee the documets. Ths represets the smlarty betwee the documets. Whe usg -grams stead of ugrams, each axs cossts of more tha oe word. Sce we used Java as the programmg laguage, we dd some smple hertace to solve ths. We chose to use the cose coeffcet, whch determes the agle betwee the documets. [] preseted the algorthm for VSM as follows: r r cos( q, d) = = = q q d = where q s the query, d s the documet ad the umber of dfferet words. Sce we used oly the umber of words as the coeffcet, oly those words, whch have a word cout hgher tha zero both documets, wll produce a value. Therefore we smplfed ths algorthm, so that we would oly eed to multply the commo words the omator. Ths way the algorthm we use does ot requre ay computatos o the words that are ot part of both the documets. r r cos( d, d ) = q = qd = d d d d d =, d where d s documet, d s documet, ad qd the umber of commo words. We extract the words from the documets usg regular expressos, whch make t flexble to redefe the smallest parts that are aalyzed. The we sort usg Java s Mergesort for O(log) performace, stead of puttg the words a sorted lst drectly, whch would gve O( ) performace.,

4 Clusterg Accordg to [], the k-meas algorthm s the coceptually smplest method ad should probably be used frst o a ew data set because ts results are ofte suffcet. Ths summarzes the reasos why we used ths algorthm. The k-meas method s really smple; frst some cluster ceters (ceter of mass) are radomly chose (we pcked radomly chose documets). Each documet s assged to oe of these clusters (defed by the closest cetre). The a ew cluster cetre s calculated for each cluster. I the ext terato each documet s assged to oe of the ew cluster ceters that were prevously calculated. Curretly we terate a gve umber of tmes, stead of havg a stop crtero. The calculato of ew cluster ceters was doe usg oly the words of all the documets that cluster. A alteratve would have bee to clude all words all documets stead, but that would have requred more computatoal power. Results We have oly tested our algorthm o a lmted test, due to tme restrctos. We have also costructed examples where our algorthm outperforms tradtoal (ugram) methods. For example our method ca see dffereces texts wth words that are the same, but that come a dfferet order. I our test texts there are ot may cases where ths arses. I some areas there are two words that are belogg to each other, though, whch make our algorthm work better. Whe calculatg a Vector space model value betwee two documets, a choce has to be made betwee ugrams, bgrams, trgrams, et cetera. Sce we wated to aalyze the beefts of -grams, we dd both separate tests usg them dvdually ad a weghed sum of the ugrams, bgrams ad trgrams. We foud that the Vector space model betwee arbtrary documets s oly applcable usg up to 4-grams or 5- grams, uless you wat to spot ctatos or versos of exactly the same documets. We have chose to oly use the frst three values (up to trgrams) our clusterg tests. Whe we tested to cluster 6 texts from the New Scetst web page, 3 texts about eutros ad 3 texts about trasstors, they were clustered correctly usg both ugrams ad our exteded verso, whch cluded bgrams ad trgrams. Whe aalyzg the output of the Vector space model (the smlartes), we foud that our approach usg oly bgrams chaged the outcome of the clusterg somewhat the wrog drecto to what we had expected. Clusterg wth oly trgrams made t cluster properly aga, as dd the combato of the three. Oe reaso to the falure of bgrams could be the small quatty of texts, or the fact that we dd ot choose texts that were dfferet eough. Cocluso Our lmted testg shows that lookg at more tha oe word at a tme wo t ecessarly gve accuracy, but ca potetally gve other beefts whe comparg documets. I the few texts we have used, there s o obvous advatage of usg our suggested algorthm, although t could be our hadpcked documets that are ot adequately belogg to other areas. The extra tme of mplemetg the

5 algorthm probably makes t eve less terestg. We hope that people ca reuse, ad make use of, our source code, though. Besdes clusterg, -grams could be used to spot ctatos from oe documet to aother, or versos of the same documet, wth some small chages the source code. The method could also be used to extract key -grams, stead of extractg keywords from texts. These would probably descrbe the documet eve better tha sgle words. We thk there are a lot of potetally terestg areas where -grams ca be used, ad clusterg s oe of them. We have show that t has potetal, but the curret beefts are too small. Maybe a combato of -grams ad other techques wll prove to work best? Further testg of our proposed algorthm eeds to be doe. Applet, Java source, ad test texts To make t easer to aalyze our results, ad to preset the code used, a web page was created for the project. It s located at: e. There we have the source code for the project, together wth javadoc, a example Applet, the test text samples we used ad ths report.

6 Refereces [] Chrstopher D. Mag ad Hrch Schütze, 999, Foudatos of statstcal atural laguage processg, MIT Press. [] Adre Z. Broder, Steve C. Glassma, Mark S. Maasse, Geoffrey Zweg, 997, Sytactc Clusterg of the Web, Dgtal SRC Techcal Note , SRC/techcal-otes/abstracts/src-t html [3] Perre Nugues, 00, Itroducto to Laguage Processg ad Computatoal Lgustcs, Lecture Notes, Lud Isttute of Techology. [4] Damarks Tekske Uverstet, 00, Itroducto: Vector Space Model, Techcal report, ultmeda/textmg/ode5.html [5] Gerard Salto, A. Wog, C.S Yag, 975, A Vector Space Model for Automatc Idexg, Commucatos of the ACM, 8(), November. [6] Joh Zukowsk, 00, Java Tech Tps o usg regular expressos Java, Su JDC Techcal otes, per/jdctechtps/00/tt043.html