Term Ranking for Clustering Web Search Results

Transcription

1 Term Rakig for Clusterig Web Search Results Fatih Gelgi Departmet of Computer Sciece ad Egieerig Arizoa State Uiversity Tempe, AZ, 85287, USA Hasa Davulcu Departmet of Computer Sciece ad Egieerig Arizoa State Uiversity Tempe, AZ, 85287, USA Sriivas Vadrevu Departmet of Computer Sciece ad Egieerig Arizoa State Uiversity Tempe, AZ, 85287, USA ABSTRACT Clusterig web search egie results for ambiguous keyword searches poses uique challeges. First, we show that oe caot readily import the frequecy based feature rakig to cluster the web search results as i the text documet clusterig. Next, we preset TermRak, a variatio of the PageRak algorithm based o a relatioal graph represetatio of the cotet of web documet collectios. TermRak achieves desirable rakig of discrimiative terms higher tha the ambiguous terms, ad rakig ambiguous terms higher tha commo terms. We experimet with two clusterig algorithms to demostrate the efficacy of TermRak. TermRak is show to perform substatially better tha frequecy based classical methods. Keywords: Web page clusterig, term rakig, PageRak, radom walk. 1. INTRODUCTION Durig Web search, whe keyword queries are ambiguous such as apple, which appears i various cotexts such as Computers or Fruit, it becomes arduous for a user to idetify the distict seses of their search terms ad fid cotextual search results. This problem is further exacerbated whe the user is uable to guess additioal discrimiative keywords, such as ipod, to filter the matches. For example, there are 544 millio results retured by Google search egie 1 for the keyword query apple. A better way to browse the search results retrieved by a search egie would be to orgaize them ito clusters with their descriptios (like Vivisimo 2, EigeCluster [3], SakeT [4] ad Grouper [15]) to guide the user durig her search. The features i Web documet clusterig are usually geerated from HTML sytax ad text i a give collectio. Sytactic features are out of the scope of this work; this paper focuses o exploitig most useful terms i a give Web Permissio to make digital or hard copies of all or part of this work for persoal or classroom use is grated without fee provided that copies are ot made or distributed for profit or commercial advatage ad that copies bear this otice ad the full citatio o the first page. To copy otherwise, to republish, to post o servers or to redistribute to lists, requires prior specific permissio ad/or a fee. Copyright 200X ACM X-XXXXX-XX-X/XX/XX...$5.00. documet collectio retrieved from a ambiguous keyword search such as apple. Clusterig has bee previously applied o text documets [8]. The commo practice is to use all the terms as features [13] i the documet collectio by geeratig the term vectors for each documet. Especially i large collectios usig all terms explodes the feature space ad gives rise to a wellkow problem, the curse of dimesioality. It slows dow ad reduces the quality of clusterig dramatically. To avoid this problem, oe usual method is to use the top-k best raked terms as features where k is a reasoable dimesio for clusterig techique to be used. I that case, the quality of top-k terms eeds to be assured by a robust term rakig method. Broadly used frequecy based term rakig methods are TF (term frequecy) ad TF/IDF (term frequecy/iverse documet frequecy) [13] that pealizes the weights for commo keywords that appear i large umber of documets such as that, about, the etc. These measures work well o clusterig text documets sice text documet collectios usually cotai focused ad mostly relevat vocabulary for their terms. However, clusterig web search egie results for ambiguous keyword searches poses uique challeges. First, we show that we caot readily import the frequecy based features to cluster the web search results. This is due to the ature of the web search results which (i) usually cotai various kids of irrelevat terms i their avigatioal aids, advertisemets ad liks to other pages ad also due to the observatio that (ii) Web pages are usually ot as comprehesive as text documets ad mostly cover oly fragmets of the relevat cotext. Hece, we propose to orgaize terms foud i web search results ito three categories: discrimiative terms that belog to a specific cotext ad strogly related with a distict sese of the keyword search term, ambiguous terms that have may seses, ad commo terms that appear i may distict cotexts of a keyword search term. Adjustig term weights based o their category is critical for buildig pure clusters of web search results. For example, for the keyword search term apple, the term ipod is discrimiative i determiig that the web documets matchig ipod belogs to the computer sese of the word. Where as, a commo term such as cotact or iformatio should ot carry much weight durig clusterig. I this paper we preset TermRak, a variatio of the PageRak [11] algorithm based o a relatioal graph represetatio of the cotet of web documet collectios. TermRak achieves desirable rakig of discrimiative terms higher

2 tha the ambiguous terms, ad rakig ambiguous terms higher tha commo terms. As idicated i [16], traditioal clusterig algorithms refie terms after clusterig istead of filterig carefully before clusterig [9, 14, 5, 7]. However, a good term rakig method improves the quality of the features ad helps Web page clusterig sigificatly. We use two clusterig algorithms to demostrate the efficacy of TermRak: K-meas, a popular ad efficiet clusterig algorithm, ad SCuBA [1], a state-of-art subspace clusterig algorithm. We show that performaces of classical term rakig methods, TF ad TF/IDF, for both K- meas ad SCuBA are very close. O the other had, TermRak is show to perform substatially better tha the classical methods for both K-meas (purity +9% ad F-measure +11%) ad SCuBA (purity +4% ad F- measure +5%). 2. FREQUENCY BASED TERM RANKING Term rakig is a essetial issue i clusterig documets. Rakig distiguishig terms higher yields better estimatio of similarity betwee documets ad hece higher quality clusterig. Stadard frequecy based term rakig methods i Iformatio Retrieval (IR) are: Term frequecy (TF) is the frequecy of a term amog all the terms i the Web page collectio, ad calculated as T F (t) = t where t is the umber of occurreces of t i the collectio ad is the total umber of terms i the collectio. Term frequecy / iverse documet frequecy TF/IDF [13] is a method to reduce the bias of term frequecy by pealizig with the documet frequecy. It is calculated as T F/IDF (t) = T F (t). lg W where D(t) is D(t) the set of Web pages t appears. 2.1 Aalysis o the Web Data TF/IDF is kow to be a effective method for rakig terms i text documet collectios. However, Web pages are ot composed of oly raw text ad the characteristics of the Web data is differet from text documets. We ca distiguish betwee Web ad text documet collectios by makig the followig key observatios: Observatio 1. Web pages are usually ot as comprehesive as text documets ad mostly cover oly fragmets of the relevat cotext. Observatio 2. Almost all terms appearig i a text documet ca be cosidered as relevat withi the documet s cotext. However, oe ca observe may cotext irrelevat terms which might be part of advertisemets, headers or footers, ad avigatio aids such as back, cotact ad search withi Web pages. Substatial quality differece betwee text documet clusterig ad Web page clusterig is apparet i experimetal results of [2, 9]. Such results strogly supports our observatios. We will detail our aalysis by ivestigatig the frequecy distributios of frequet terms o the Web data. We also observe that, all frequet terms ca be grouped ito three categories: Discrimiative terms: These terms typically belog to a specific cotext, ad they are strogly related with the cotext. Mac, ipod ad recipe are such examples from apple 3 data. Ambiguous terms: These terms appear i more tha oe cotext ad their degree of relatedess might vary depedig o the cotext. For istace, software ad computer appear i both Computers ad Video games categories of the apple data. However, their degree of relatedess would ot be weak due to the overlap i the cotext of categories. Commo terms: These terms appear i may cotexts i the data. Ulike the ambiguous terms, they have weak coectios with the cotext that they appear. Some examples of commo terms are , cotact, ad search. I the rest of the paper importat terms refer to both discrimiative ad ambiguous terms. I Web page clusterig, rakig discrimiative terms higher tha the ambiguous terms, ad rakig ambiguous terms higher tha the commo terms cotribute to high accuracy. Sice cotextual relatedess does ot deped o the term frequecies aloe ad i the Web data commo ad ambiguous terms have higher frequecy characteristics, TF method would mix the raks of differet types of terms. TF/IDF is a good measure for pealizig commo ad ambiguous terms usig iverse documet frequecy i text collectios. However, it becomes ieffective for rakig ambiguous ad commo terms lower tha discrimiatig terms sice they would ot appear i sufficietly large umbers due to Observatio 1 of Web collectios. As a cosequece, TF/IDF gets ito the same pitfalls as the other frequecy based methods such as TF. To illustrate the similarities of term rakigs based o TF ad TF/IDF we raked the top-200 (out of 24,455) terms i the apple data. We observed excessive overlap (97%!) betwee their rakigs ad similar weightig of the terms. Cosequetly, oe caot merely rely o term frequecy based measures to rak amog the three groups of terms i the preferred orderigs described above. Next, we will preset a ew rakig method i order to obtai the desired term rakigs for clusterig Web search results. 3. EXTRACTING RELATIONAL GRAPH A relatioal graph is a weighted udirected graph that captures the co-occurrece relatioship iformatio betwee terms i a give Web page collectio. The odes are the terms i the collectio, ad the weights o the edges represet the associatio stregth betwee the terms. Formally, we defie ad iitialize a relatioal graph as follows: Defiitio 1 (Relatioal Graph). A relatioal graph G is a weighted udirected graph where the odes are the terms ad the edge weights are the couts of the correspodig co-occurrece relatios i the collectio. Assumig w ij as the weight betwee the terms i ad j, w ij is the umber of times the edge (i, j) appears i the etire data. The weight of the ode i is iitialized as the occurrece of the correspodig term i the collectio. 3 The experimetal Web page collectio is gathered from with apple keyword search.

3 Note that the edges are udirected sice associatio stregth betwee terms is a bidirectioal measure. Extractig cotextually related terms precisely from Web pages require deep sytactic ad sematic aalysis. This problem is outside the scope of this paper. Thus we use a simple heuristic that retrieves oly the blocks i which the search keyword appear withi each Web page. Here, block refers to the text fragmets delimited by a set of pre-determied tags such as <div>, <spa>, <table>, <p>, <ul> ad <ol>. The edges are iitialized as the co-occurrece of terms withi such blocks. We observed that usig the etire page for relatio extractio itroduces too may irrelevat term associatios. For example, terms such as copyright, search, back are almost always irrelevat to the keyword search cotext. Our block based heuristic would reduce their associatio stregths sice they would rarely co-occur i the same block with the importat terms. Suppose a relatioal graph of a Web search result collectio is give. We ext propose a efficiet algorithm to utilize the relatioal graph to rak the importat terms. We also observe that differet types of terms preseted above have the followig associatio characteristics: A discrimiative term does ot have may eighbors, but importat oes, ad its associatios are strog. A ambiguous term has may eighbors ad it has strog associatios as well as weak oes. A commo term has may eighbors ad mostly its associatios are weak. Figure 1: The fragmet of the relatioal graph of apple data. Sizes of odes ad thickess of edges are proportioal to their term frequecies ad associatio stregths respectively. A fragmet of a relatioal graph of apple data is preseted i Figure 1 for illustrative purposes. Discrimiative terms such as mac ad recipe have eighbors with strog associatios. Commo terms such as cotact ad have may eighbors with weak coectios. Computer ad software are examples of ambiguous terms that have may eighbors with strog associatios as well as weak oes. Let W = {W 1, W 2,..., W W } be the set of Web pages i the collectio. Cosiderig a Web page is the vector of its terms ad all pairwise relatios betwee them, extractio of the relatioal graph of the collectio requires W i W Wi 2 time. OP 4. TERMRANK ALGORITHM TermRak is a variatio of PageRak algorithm that calculates the raks of the terms i the relatioal graph. This sectio gives a brief backgroud o PageRak ad its ituitive explaatio. The, we preset the TermRak algorithm alogside its justificatio for term rakig. 4.1 PageRak PageRak [11] is a method that calculates the importace of the odes i a lik (citatio) graph of Web pages. The idea is based o the probability of a radom surfer visitatio to a Web page followig the liks of Web pages. As idicated i [11], importace of a Web page is cosidered to be proportioal to the umber importat sources poitig to that page. PageRak computes this recursive defiitio of importace by utilizig a radom walk approach over the Web lik graph. Radom walk propagates the probability of each page to the pages it liks to. Give the lik graph G PageRak score for a ode is calculated as: P R(i) = α X j N (i) P R(j) N (j) + + (1 α) 1 V(G) where V(G) is the set of odes i G, N ( ) + ad N ( ) gives the set of eighbors that are coected to the ode with their outgoig ad icomig liks respectively. (1 α) is a decay factor to avoid of rak siks. Rak siks [11] are defied to be a set of odes which have liks betwee themselves but o liks to the other odes. Hece these odes geerate a loop ad accumulate rak but they ever distribute ay rak outside sice there is o outgoig edges. That decay factor acts as a exit ode with certai probability. A ituitive explaatio for the decay factor is give as the jumpig probability of a radom surfer to other pages. 4.2 TermRak Origially PageRak operates o a directed graph, ad edges have o weights. TermRak adopts the PageRak algorithm to icorporate udirected edges ad edge weights. Hece, all edges are cosidered to be both icomig ad outgoig. Additioally, jumpig to a radom page requiremet i PageRak is ot ecessary for TermRak sice there are o rak siks i udirected graphs. Thus decay factor is ot icluded i our formula. Give a relatioal graph G, TermRak ca be calculated as follows: T R(i) = X j N (i) PT R(j).w ij k N (j) w jk where N ( ) represets the set of eighbors of the ode. Similar to [11, 6], to compute TermRak, we use a very efficiet approximatio method which iterates Equatio 2: T R (0) (i) = T R (t+1) (i) = w i Pj V(G) wj = T F (i) X j N (i) T R (t) (j).w ij Pk N (j) w jk (1) (2). (3)

4 TermRak iteratio: 0 iteratio: 20 TF/IDF mac macitosh game ipod video computer cotact Table 1: Raks of terms based o TermRak ad TF/IDF. The higher value is the higher rak. I Equatio 3, the raks of the odes are iitialized with their term frequecies sice importat terms are assumed to be potetially frequet i the collectio. This iitializatio biases the formula towards the odes with high term frequecies. Please recall that this set of frequet terms cotais importat terms as well as commo oes. ItemRak rus util the differece betwee to iteratios is less tha δ which is a reasoably small value. TermRak satisfies the exteder graph property ad hece its covergece is guarateed [10]. I radom walks, a high rakig ode eeds to satisfy two essetial factors: importat eighbors, strog coectios. Please ote that the umber of liks of a ode is a importat factor i PageRak due to its o-weighted (or uiform) liks. However, i weighted graphs the stregth of the edges are the determiig factor. For istace, a ode with oe edge which has a weight of 10 will receive more rak tha a ode with 10 edges each of which has a weight of 1 whe the rest of their graphs are idetical. The term raks from higher to lower will be sorted as follows: (i) the odes with may importat eighbors ad strog coectios, (ii) the oes with some importat eighbors ad some strog coectios, ad fially (iii) the oes with may eighbors ad may weak coectios. Please recall that the term rak orderigs preseted above would correspod exactly to the desired rakigs of discrimiative, ambiguous, ad commo terms. Figure 2: The fragmet of the relatioal graph of apple data. The umbers uder the ode labels represet the term ad documet couts of the correspodig odes i the Web page collectio. Similarly, the umbers o the edges represet the edge couts. To illustrate TermRak o a example, Figure 2 which is part of the relatioal graph of apple data is adopted from Figure 1. mac, macitosh, game, ipod ad video are discrimiative terms, computer is a ambiguous term ad cotact is a commo term. Node weights are iitialized as term couts ad δ is set to As preseted i Table 1, iitial raks of the terms are TF values which gives a mixed rakig i terms of discrimiative, ambiguous ad commo terms. TermRak coverges i 20 iteratios ad it results with the desirable rakig of the terms; first five are the discrimiative terms, the the ext two are the ambiguous terms ad fially the commo term. Coversely, computer has the highest rak ad cotact has the third rak i TF/IDF rakig. TermRak retrieves each ode ad edge i each iteratio. That meas the total ruig time for k iteratios is O (k( + m)) utilizig hash tables for odes ad their edges. I our experimets we foud that k is a small costat. Eve for 322 millio liks, the PageRak algorithm coverges i about 52 iteratios [11]. 5. EXPERIMENTS Keyword # of # of Categories pages terms apple Computers(463), Fruit(136), Music(21), Locatios(17), Movies(6), Games(5) dell Arts(18), Computers(11), Authors(4) gold Shoppig(471), Miig(151), Movies(28), Motors(11), Games(8), Sports(1) jaguar Cars(78), Video games(48), Aimals(9), Music(3) jorda Coutry(249), Music(42), Authors(38), City(29), Basketball(10), Geealogy(8), Baks(10), Movies(10), Soccer(1), News(4) satur Cars(22), Plaets(21), Aime(19), Video games(9) tiger Sports(35), Video games(28), Aimals(25), Movies(19), Terrorism(3) Table 2: Selected search keywords ad their correspodig categories i data sets. I our experimets, we idetified some ambiguous keywords ispired from [16] o Ope Directory Project 4 (ODP). ODP is the largest, most comprehesive huma-edited directory of the Web. It is costructed ad maitaied by a vast, global commuity of voluteer editors. We selected the search keywords apple, dell, gold, jaguar, jorda, satur ad tiger for data collectio comprisig 2,112 Web pages ad 81,422 uique terms i total. These documet collectio statistics are preseted i Table 2. Durig the preprocessig step, commo data types such as percetages, dates, umbers etc., stop words, ad puctuatio symbols are filtered usig simple regular expressios. We used two clusterig algorithms to demostrate the efficacy of TermRak: K-meas ad SCuBA [1]. K-meas is oe of the most commo clusterig methods preferred for its speed ad quality. I our experimets the actual umber of clusters K is provided accordig to the umber of matchig ODP categories. For each keyword data, K-meas has bee executed 20 times ad the results correspod to the average of all rus. Purity ad F-measure deviate i the iterval of ±5%. SCuBA is a state-of-art subspace clusterig algorithm that efficietly determies clusters ad their related features (subspaces) by aalyzig frequet term sets of documets. It is origially part of a article recommedatio system for researchers. The experimets ru o a Itel Petium4 3GHz CPU with 1GB RAM which had Widows XP operatig system. 4

5 K-Meas SCuBA TF TF/IDF TermRak TF TF/IDF TermRak Keyword P E F P E F P E F P E F P E F P E F apple dell gold jaguar jorda satur tiger overall Table 3: Performace compariso of term rakig methods i K-meas ad SCuBA. P, E ad F refers to purity, etropy ad F-measure respectively. etropy of the clusterig: E = X i i Ei. (6) Figure 3: The overlap of TF/IDF features with the other methods. Keyword Sample features Rel. Category apple cake, puddig, fruit, Fruit recipe, pie, bread dell computer, desktop, busiess, software Computers gold exploratio, mieral, miig, Miig resources, paig, compay jaguar classic, british, club, car Cars jorda atioal, foreig, middle, east Coutry satur playstatio, game, arcade Video games tiger movie, martial, art, hidde, Movies drago, crouchig, chiese Table 4: Sample subspaces ad their related categories. Oe subspace is show for each search keyword. 5.1 Evaluatio Metrics To evaluate the quality of our results, the computed clusters are compared with the actual categories give i ODP. We use the commo evaluatio metrics for clusterig [12]: precisio, recall, F-measure, purity, ad etropy. Precisio, p ij = ij i ad recall, r ij = ij j compare each cluster i with each category j where ij is the umber of Web pages appear i both the cluster i ad the category j, i ad j are the umber of Web pages i the cluster i ad i the category j respectively. F-measure, F ij = 2p ij r ij p ij +r ij is a commo metric calculated similarly to the oe i IR. The F-measure of a category j is F j = max i{f ij} ad similarly the overall F-measure is: F = X j j Fj. (4) Quality of each cluster ca be calculated by purity ad etropy. Purity measures how pure is the cluster i by ρ i = max j{p ij}. The purity of the etire clusterig ca be calculated by weightig each cluster X proportioal to its size as: i ρ = ρi (5) i P where is the total umber of Web pages. The etropy of a cluster i is E i = j pij log pij. Calculatig the weighted average over all clusters gives the etire 5.2 ODP Results First we use a wrapper which seds a give search keyword to ad collects the resultig categories ad the Web pages that belog to those categories. Collected pages are categorized by their ODP categories. Next, all blocks are extracted from the collected Web pages. About 5% of the collected Web pages i ODP are defective such as icorrect or redirected urls, erroeous HTML codes, pages composed of just images, or uder costructio. For all Web search result collectios, TermRak is quite fast ad it rus i less tha a milisecod. The term overlaps i the top-200 for TF/IDF vs. TF ad TermRak are give i Figure 3. TF/IDF ad TF have sigificat overlap about 94% o the average. Whereas TermRak has overlap of oly 76%. Hece it is show TF ad TF/IDF perform very similarly (almost i ±3% rage) as detailed i Table 3. Overall performaces of TF ad TF/IDF for both K-meas ad SCuBA are very close. O the other had, TermRak have performed substatially better for both K-meas (purity +9% ad F-measure +11%) ad SCuBA (purity +4% ad F-measure +5%). The overlaps i the terms geerated by TermRak are the lowest i jaguar ad jorda data sets as show i Figure 3. Discrimiatig terms of the categories are well idetified ad properly raked hece oe ca see the sigificat differece betwee F-measures of TF, TF/IDF ad TermRak i Table 3. TermRak is show to be successful i degradig the commo terms. Table 5 presets some obvious commo terms ad their raks. Commo terms are ideed placed at much lower raks by TermRak tha by TF/IDF. Especially i dell, jorda, satur ad tiger data sets, TermRak very successfully reduces the raks of the commo terms. That s why TermRak performs better tha the other methods i these data sets. The quality of clusterig i a documet collectio is affected by other factors such as cotext domiace ad cotext overlap. If most of the Web pages belog to oe category, the cotext of that category domiates the others. As a cosequece, a large amout of the feature terms are geerated from the cotext of the domiatig category ad their raks are high. I that case, clusterig quality reduces sice the there is ot sufficiet terms to distiguish remaiig small categories. Apple, gold ad jorda are such examples i our data sets. Computer, Shoppig ad Coutry are the domiatig categories i these data sets respectively. I the secod case, Web pages i differet categories have

6 apple dell gold jaguar jorda satur tiger tfidf tr tfidf tr tfidf tr tfidf tr tfidf tr tfidf tr tfidf tr search >1000 > >1000 cotact > liks > > > >1000 ews > olie > > >1000 special > > copyright >1000 >1000 >1000 >1000 rights >1000 > >1000 >1000 >1000 >1000 >1000 >1000 >1000 top Table 5: Raks of some commo terms i differet data sets. tfidf ad tf refers to TF/IDF ad TermRak respectively. top-200 row specifies the umber terms raked i top-200. similar cotexts. This is ot due to the cotext ambiguity or the ambiguity of the terms i the cotext but exactly the same cotext might appear i more tha oe categories. For istace, may Web pages i Shoppig category have iformatio o gold as a material which is commo i Miig. Aother example is Sports ad Video games categories i tiger data set. Tiger Woods is the well-kow famous golf player ad golf is the mai cotext of the Tiger Woods Web pages i Sports category. However, the cotext is agai golf i the Tiger Woods video game pages i Video Games category. That is the mai reaso of the purity of tiger data set to be lower as preseted i Table 3. High quality term features geerated by TermRak ca be very useful i subspace clusterig algorithms to produce precise subspaces. To demostrate the efficacy, some of the subspaces geerated by SCuBA are preseted i Table 4. Subspaces that serve as the cotextual terms of the categories are accurately idetified from the terms geerated by TermRak. For example, i tiger data set the subspace refers to the Web pages of Crouchig Tiger, Hidde Drago, the famous movie with 4 Oscar awards i The movie is origially Chiese ad about martial arts. 6. CONCLUSION I this paper, we show the ieffectiveess of frequecy based term rakig methods such as TF ad TF/IDF for clusterig the Web search results. Istead, we provide a ovel term rakig method, TermRak which utilizes radom walks o a relatioal graph of the give Web page collectio. Our experimetal results illustrate the effectiveess of our algorithm by measurig purity, etropy ad F-measure of geerated clusters based o Ope Directory Project (ODP) data. I this work, we raked the terms without explicitly categorizig them ito the three categories; discrimiative, ambiguous ad commo terms. Future work icludes clearly separatio of terms ito these categories. By categorizig the terms, commo terms ca be easily excluded whereas discrimiative ad ambiguous terms ca be more effectively used i clusterig Web search results. 7. REFERENCES [1] N. Agarwal, E. Haque, H. Liu, ad L. Parsos. A subspace clusterig framework for research group collaboratio. Iteratioal Joural of Iformatio Techology ad Web Egieerig, 1(1):35 38, [2] F. Beil, M. Ester, ad X. Xu. Frequet term-based text clusterig. I SIGKDD, pages , New York, NY, USA, ACM Press. [3] D. Cheg, S. Vempala, R. Kaa, ad G. Wag. A divide-ad-merge methodology for clusterig. I PODS, pages , New York, NY, USA, ACM Press. [4] P. Ferragia ad A. Gulli. A persoalized search egie based o web-sippet hierarchical clusterig. I WWW, pages , New York, NY, USA, ACM Press. [5] E. Gabrilovich ad S. Markovitch. Feature geeratio for text categorizatio usig world kowledge. I Proceedigs of The Nieteeth Iteratioal Joit Coferece for Artificial Itelligece, pages , Ediburgh, Scotlad, [6] M. Gori ad A. Pucci. Itemrak: A radom-walk based scorig algorithm for recommeder egies. I IJCAI, pages , [7] S. Huag, Z. Che, Y. Yu, ad W.-Y. Ma. Multitype features coselectio for web documet clusterig. IEEE Trasactios o Kowledge ad Data Egieerig, 18(4): , [8] A. Leouski ad W. B. Croft. A evaluatio of techiques for clusterig search results. Techical Report IR-76, Uiversity of Massachusetts, Amherst, [9] T. Liu, S. Liu, Z. Che, ad W.-Y. Ma. A evaluatio o feature selectio for text clusterig. I ICML, pages , [10] R. Motwai ad P. Raghava. Radomized Algorithms. Cambridge Uiversity Press, [11] L. Page, S. Bri, R. Motwai, ad T. Wiograd. The pagerak citatio rakig: Brigig order to the web. Techical report, Staford Digital Library Techologies Project, [12] M. Rosell, V. Ka, ad J.-E. Litto. Comparig comparisos: Documet clusterig evaluatio usig two maual classificatios. I ICON, [13] G. Salto, A. Wog, ad C. S. Yag. A vector space model for automatic idexig. Commuicatios of the ACM, 18(11): , [14] A. Strehl, J. Ghosh, ad R. Mooey. Impact of similarity measures o web-page clusterig. I AAAI, pages AAAI, July [15] O. Zamir ad O. Etzioi. Grouper: a dyamic clusterig iterface to Web search results. Computer Networks, 31(11 16): , [16] H.-J. Zeg, Q.-C. He, Z. Che, W.-Y. Ma, ad J. Ma. Learig to cluster web search results. I ACM SIGIR, pages , New York, NY, USA, ACM Press.