Optimizing Web Search Using Social Annotations

Transcription

1 Sesson: Search Qualty and Precson Optzng Web Search Usng Socal Annotatons Shenghua Bao *, Xaoyuan Wu *, Ben Fe 2, Gurong Xue, Zhong Su 2, and Yong Yu Shangha JaoTong Unversty Shangha, , Chna {shhbao, wuxy, grxue, 2 IBM Chna Research Lab Beng, 00094, Chna {feben, ABSTRACT Ths paper explores the use of socal annotatons to prove web search. Nowadays, any servces, e.g. del.co.us, have been developed for web users to organze and share ther favorte web pages on lne by usng socal annotatons. We observe that the socal annotatons can beneft web search n two aspects: ) the annotatons are usually good suares of correspondng web pages; 2) the count of annotatons ndcates the popularty of web pages. Two novel algorths are proposed to ncorporate the above nforaton nto page rankng: ) SocalSRank (SSR) calculates the slarty between socal annotatons and web queres; 2) SocalPageRank (SPR) captures the popularty of web pages. Prelnary experental results show that SSR can fnd the latent seantc assocaton between queres and annotatons, whle SPR successfully easures the qualty (popularty) of a web page fro the web users perspectve. We further evaluate the proposed ethods eprcally wth 50 anually constructed queres and 3000 auto-generated queres on a dataset crawled fro del.co.us. Experents show that both SSR and SPR beneft web search sgnfcantly. Categores and Subect Descrptors H.3.3 [Inforaton Systes]: Inforaton Search and Retreval. General Ters: Algorths, Experentaton, Huan Factors. Keywords: Socal annotaton, socal page rank, socal slarty, web search, evaluaton. INTRODUCTION Over the past decade, any studes have been done on provng the qualty of web search. Most of the contrbute fro two aspects: ) orderng the web pages accordng to the querydocuent slarty. State-of-the-art technques nclude anchor text generaton [2, 2, 34], etadata extracton [37], lnk analyss [34], and search log nng [0]; 2) orderng the web pages accordng to ther qualtes. It s also known as queryndependent rankng, or statc rankng. For a long te, the statc rankng s derved based on lnk analyss, e.g., PageRank [7], HITS [5]. Recently, the features of content layout, user clckthroughs etc. are also explored, e.g., frank[9]. Gven a query, the retreved results are ranked based on both page qualty and query-page slarty. Recently, wth the rse of Web 2.0 technologes, web users wth dfferent backgrounds are creatng annotatons for web pages at an ncredble speed. For exaple, the faous socal bookark web ste, del.co.us [4] (henceforth referred to as Delcous ), has ore than llon regstered users soon after ts thrd brthday, and the nuber of Delcous users have ncreased by ore than 200% n the past nne onths [3]. Socal annotatons are eergent useful nforaton that can be used n varous ways. Soe work has been done on explorng the socal annotatons for folksonoy [2], vsualzaton [], seantc web [36], enterprse search [23] etc. However, to the best of our knowledge, lttle work has been done on ntegratng ths valuable nforaton nto web search. How to utlze the annotatons effectvely to prove web search becoes an portant proble. In ths paper, we study the proble of utlzng socal annotatons for better web search, whch s also referred to as socal search for splcty. More specfcally, we optze web search by usng socal annotatons fro the followng two aspects: Slarty rankng, whch eans the estated slarty between a query and a web page. The annotatons, provded by web users fro dfferent perspectves, are usually good suares of the correspondng web pages. For exaple, the top 5 annotatons of Aazon s hoepage n Delcous are shoppng, books, aazon, usc and store, whch depct the page or even the whole webste exactly. These annotatons provde a new etadata for the slarty calculaton between a query and a web page. However, for a specfc web page, the annotaton data ay be sparse and ncoplete. Therefore, a atchng gap exsts between the annotatons and queres (e.g., between shop and shoppng ). How to brdge the gap reans a crucal proble n further provng the slarty rankng. We propose a new slarty estaton algorth, SocalSRank (SSR) to address ths proble. Statc rankng, the aount of annotatons assgned to a page ndcates ts popularty and ples ts qualty n soe sense, yet tradtonal statc rankng algorths such as PageRank have no way to easure ths knd of qualty. For exaple, thousands of Delcous users collect the popular pattern ntroducton page 2 as ther favorte wth a varety of annotatons, but ths ste s gven a PageRank Copyrght s held by the Internatonal World Wde Web Conference Cottee (IW3C2). Dstrbuton of these papers s lted to classroo use, and personal use by others. WWW 2007, May 2, 2007, Banff, Alberta, Canada. ACM /07/0005. * Part of Shenghua Bao and Xaoyuan Wus work of ths paper was conducted n IBM Chna Research Lab

2 Sesson: Search Qualty and Precson of zero by Google. Furtherore, dfferent annotatons ay have dfferent weghts n ndcatng the popularty of web pages. In the lght of the above ntutons, we propose a novel algorth, SocalPageRank (SPR) to easure the popularty of web pages usng socal annotatons. For each aspect, we wll show one algorth that s guaranteed to converge. The proposed algorths are evaluated on a Delcous corpus whch conssts of,736,26 web pages wth 269,566 dfferent socal annotatons. Prelnary experental results show that SSR can calculate the annotaton slartes seantcally and SPR successfully depcts the qualty of a web page fro the web users perspectve. To evaluate ther effectveness for web search, 50 queres and the correspondng ground truths are collected fro a group of CS students and 3000 queres are generated fro the Open Drectory Proect 3. Experents on two query sets show that both SSR and SPR prove the qualty of search results sgnfcantly. By further cobnng the together, the ean average precson of search results can be proved by as uch as 4.0% and 25.02% on two query sets, respectvely. The rest of the paper s organzed as follows. Secton 2 dscusses the related work. Secton 3 proposes the socal search fraework wth SocalSRank and SocalPageRank n detal. Secton 4 presents the experental results. Secton 5 gves soe dscussons. Fnally, we conclude wth Secton RELATED WORK 2. Research on Web Search Much work has been done on provng user experence of web search, ost of whch focuses on rankng search results. We brefly revew the related work on slarty rankng and statc rankng as follows. Slarty rankng easures the relevance between a query and a docuent. Many odels have been proposed to estate the slarty between the query and the docuent []. In odern search engnes, several ethods have been proposed to fnd new nforaton as addtonal etadata to enhance the perforance of slarty rankng, e.g., docuent ttle [37], anchor text [2, 2, 34], and users query logs [0]. These ethods proved the perforance of web search to soe extent. For exaple, Google s search engne [2] took the anchor text as ts etadata to prove the perforance of search. Fortunately, recent eergng socal annotatons provde a new resource to calculate the query-docuent slarty ore precsely. We propose a new ethod,.e. SocalSRank for effectve use of ths new resource. Snce the publcaton of Brn and Page s paper on PageRank [7], any studes have been conducted n the web county for the statc orderng of Web pages. Recently, Rchardson et al. proposed frank [9] usng features that are ndependent of the lnk structure of the Web. PageRank utlzes the lnk structure of the Web and easures the qualty of a page fro the page creator s pont of vew, whle frank utlzes content-layout and user clck-though nforaton and captures the preference of both page authors and search engne users. In ths paper, SocalPageRank s proposed to explore statc rankng fro socal annotatons and capture the preference of web annotators. 2.2 Research on Socal Annotatons Exstng research on socal annotatons ncludes folksonoy [2, 24], vsualzaton [], eergent seantcs [25], seantc web [36], enterprse search [23] etc. Folksonoy, a cobnaton of folk and taxonoy, was frst proposed by T. V. Wal n a alng lst [2]. Folksonoy was further dvded nto the narrow (e.g. flckr 4 ) and the broad (Delcous) folksonoy n [33]. It provdes user-created etadata rather than the professonal created and author created etadata [2]. In [24], P. Merholz argued that a folksonoy could be qute useful n that t revealed the dgtal equvalent of desre lnes. Desre lnes were the foot-worn paths that soetes appeared n a landscape over te. [27] analyzed the structure of collaboratve taggng systes as well as ther dynacal aspects. Hotho et al. proposed Adapted PageRank and FolkRank to fnd countes wthn the folksonoy but have not appled the to web search []. A general ntroducton of folksonoy could be found n [6] by E. Quntarell. M. Dubnko et al. consdered the proble of vsualzng the evoluton of tags []. They presented a new approach based on a characterzaton of the ost nterestng tags assocated wth a sldng te nterval. Soe applcatons based on socal annotatons have also been explored. P. Mka proposed a trpartte odel of actors, concepts and nstances for seantc eergence [25]. X. Wu et al. explored achne understandable seantcs fro socal annotatons n a statstcal way and appled the derved eergent seantcs to dscover and search shared web bookarks [36]. Dtrey et al. lghtened the ltaton of the aount and qualty of anchor text by usng user annotatons to prove the qualty of ntranet search [23]. Dfferent fro the above work, we nvestgate the capablty of socal annotatons n provng the qualty of web search fro the aspects of slarty rankng and statc rankng wthn the Internet envronent. 3. SEARCH WITH SOCIAL ANNOTATION In ths secton, we ntroduce the socal annotaton based web search. An overvew s presented n Secton 3.. We dscuss SocalSRank and SocalPageRank n Secton 3.2 and 3.3, respectvely. In Secton 3.4, we descrbe the socal search syste utlzng both SSR and SPR. 3. Overvew As shown n Fgure, there are three knds of users related to the socal search, naely web page creators, web page annotators, and search engne users. Obvously, these three user sets can overlap wth each other. The dfferent roles they play n web search are as follows:

3 Sesson: Search Qualty and Precson Search engne users Web page annotators Page content, Anchor text, etc Web page creators clck-through data a Socal annotatons Web pages Socal search engne SSR SPR Fgure. Illustraton of socal search wth SocalSRank and SocalPageRank ) Web page creators create pages and lnk the pages wth each other to ake browsng easy for web users. They provde the bass for web search. 2) Web page annotators are web users who use annotatons to organze, eorze and share ther favortes onlne. 3) Search engne users use search engnes to get nforaton fro the web. They ay also becoe web page annotators f they save and annotate ther favortes fro the search results. Prevous work shows that both web page creators and search engne users contrbute to web search a lot. The web page creators provde not only the web pages and anchor texts for slarty rankng, but also the lnk structure for statc rankng fro the web page creators pont of vew (e.g. PageRank [7]). Meanwhle, the nteracton log of search engne users also benefts web search by provdng the clck-through data, whch can be used n both slarty rankng (e.g. IA [0]) and statc rankng (e.g. frank[9]). Here, we are to study how web page annotators can contrbute to web search. We observed that the web page annotators provde cleaner data whch are usually good suarzatons of the web pages for users browsng. Besdes, slar or closely related annotatons are usually gven to the sae web pages. Based on ths observaton, SocalSRank (SSR) s proposed to easure the slarty between the query and annotatons based on ther seantc relaton. We also observed that the count of socal annotatons one page gets usually ndcates ts popularty fro the web page annotators perspectve and the popularty of web pages, annotatons, and annotators can be utually enhanced. Motvated by ths observaton, we propose SocalPageRank (SPR) to easure the popularty of web pages fro web page annotators pont of vew. Fgure llustrates the socal search engne wth SSR and SPR derved fro the socal annotatons. In the followng sectons, we wll dscuss the two rankng algorths n detal. b d c 3.2 Slarty Rankng between the Query and Socal Annotatons 3.2. Ter-Matchng Based Slarty Rankng The socal annotatons are usually an effectve, ult-faceted suary of a web page and provde a novel etadata for slarty rankng. A drect and sple use of the annotatons s to calculate the slarty based on the count of shared ters between the query and annotatons. Lettng q={q,q 2,,q n } be a query whch conssts of n query ters and A(p)={a,a 2,, a } be the annotaton set of web page p, Equaton () shows the slarty calculaton ethod based on the shared ter count. Note that s TM (q,p) s defned as 0 when A(p) s epty. q A( p) s TM ( q, p) =, () A( p) Slar to the slarty between query and anchor text, the ter-atchng based query-annotaton slarty ay serve as a good copleent to the whole query-docuent slarty estaton. However, soe pages annotatons are qute sparse and the ter-atchng based approach suffers ore or less fro the synonyy proble,.e., the query and the annotaton ay have ters wth slar eanngs but n dfferent fors. In the next secton, we are to solve the synonyy proble by explorng the socal annotaton structures Socal Slarty Rankng Observaton : Slar (seantcally-related) annotatons are usually assgned to slar (seantcally-related) web pages by users wth coon nterests. In the socal annotaton envronent, the slarty aong annotatons n varous fors can further be dentfed by the coon web pages they annotated. ubuntu lnux gnoe Annotatons U a Annotators (a) U b a b c Web pages Query = lnux SocalSRank@st teraton: (ubuntu, lnux) = /3 (ubuntu, gnoe) = /6 (lnux, gnoe) = /4 Fgure 2. Illustraton of SocalSRank Assue that there are two annotators (U a and U b ) as llustrated n Fgure 2(a). Gven the ubuntu s offcal webste b, U a ay prefer usng the annotaton lnux, whle U b would lke ubuntu. Thus, lnux and ubuntu ay have soe seantc relatons connected by ther coonly assocated page. As for web page c, both the annotaton lnux and gnoe are gven by U a, then lnux and gnoe should also assocate wth each other to soe degree. (b) 503

4 Sesson: Search Qualty and Precson In soe cases, soe pages ay contan only the annotaton ubuntu e.g., web page a. Then gven the query contanng lnux, the page that only has ubuntu ay be fltered out properly by sple ter-atchng ethod. Even f the page contans both annotatons ubuntu and lnux, t s not proper to calculate the slarty between the query and the docuent usng the keyword lnux only. An exploraton of slarty between ubuntu and lnux ay further prove the page rankng. To explore the annotatons wth slar eanngs, we buld a bpartte-graph between socal annotatons and web pages wth ts edges ndcatng the user count. Assue that there are N A annotatons, N P web pages and N U web users. M s the N A N P assocaton atrx between annotatons and pages. M (a x, p y ) denotes the nuber of users who assgn annotaton a x to page p y. Lettng S A be the N A N A atrx whose eleent S A (a, a ) ndcates the slarty score between annotatons a and a and S P be the N P N P atrx each of whose eleent stores the slarty between two web pages, we propose SocalSRank(SSR), an teratve algorth to quanttatvely evaluate the slarty between any two annotatons. Algorth : SocalSRank (SSR) Step Int: Let S 0 A (a, a ) = for each a = a otherwse 0 S 0 P (p, p ) = for each p = p otherwse 0 Step 2 Do { S k + A = n= For each annotaton par (a, a ) do C A ( a, a ) = P( a ) P( a ) P( a ) P( a ) S k + P = n= n( M ax( M ( a, p ( a, p ), M ), M ( a, pn )) S ( a, p )) n k P For each page par ( p, p ) do CP ( p, p ) = A( p ) A( p ) A( p ) A( p ) n( M ax( M ( a ( a, p ), M, p ), M ( an, p )) S ( a, p )) }Untl S A (a, a ) converges. Step 3 Output: S A (a, a ) n ( P ( a ), P ( a )) k + A n ( A ( p ), A ( p )) n, (2) Here, C A and C P denote the dapng factors of slarty propagaton for annotatons and web pages, respectvely. P(a ) s the set of web pages annotated wth annotaton a and A(p ) s the set of annotatons gven to page p. P (a ) denotes the th page annotated by a and A (p ) denotes the th annotaton assgned to page p. In our experents, both C A and C P are set to 0.7. Note that the slarty propagaton rate s adusted accordng to the nuber of users between the annotaton and web page. Take Equaton (2) for an exaple, the axal propagaton rate can be acheved only f M (a, p ) s equal to M (a, p n ). Fgure 2(b) shows the frst teraton s SSR result of the saple data where C A and C P are set to. The convergence of the algorth can be proved n a slar way as SRank [9]. For each teraton, the te coplexty of, (3) the SSR algorth s O(N 2 A N 2 P ). Wthn the data set of our experent, both the annotaton and web page slarty atrces are qute sparse and the algorth converges quckly. But f the scale of socal annotatons keeps growng exponentally, the speed of convergence for our algorths ay slow down. To solve ths proble, we can use soe optzaton strategy such as ncorporatng the nal count restrcton [9] to ake the algorth converge ore quckly. Lettng q={q,q 2,,q n } be a query whch conssts of n query ters and A(p)={a,a 2,, a } be the annotaton set of web page p, Equaton (4) shows the slarty calculaton ethod based on the SocalSRank. s SSR = n S A = = ( q, p) ( q, a ), (4) 3.3 Page Qualty Estaton Usng Socal Annotatons Exstng statc rankng ethods usually easure pages qualty fro the web page creators, or the search engne users pont of vew. Recall that n Fgure, the estaton of PageRank [7] s subect to web creators, and the frank [9] s calculated based on both web page creators and search engne users actvtes. The socal annotatons are the new nforaton that can be utlzed to capture the web pages qualty fro web page annotators perspectve SocalPageRank Algorth Observaton 2: Hgh qualty web pages are usually popularly annotated and popular web pages, up-to-date web users and hot socal annotatons have the followng relatons: popular web pages are bookarked by any up-to-date users and annotated by hot annotatons; up-to-date users lke to bookark popular pages and use hot annotatons; hot annotatons are used to annotate popular web pages and used by up-to-date users. Based on the observaton above, we propose a novel algorth, naely SocalPageRank (SPR) to quanttatvely evaluate the page qualty (popularty) ndcated by socal annotatons. The ntuton behnd the algorth s the utual enhanceent relaton aong popular web pages, up-to-date web users and hot socal annotatons. Followng, the popularty of web pages, the up-todateness of web users and the hotness of annotatons are all referred to as popularty for splcty. Assue that there are N A annotatons, N P web pages and N U web users. Let M PU be the N P N U assocaton atrx between pages and users, M be the N A N P assocaton atrx between annotatons and pages and M UA,the N U N A assocaton atrx between users and annotatons. Eleent M PU (p,u ) s assgned wth the count of annotatons used by user u to annotate page p. Eleents of M and M UA are ntalzed slarly. Let P 0 be the vector contanng randoly ntalzed SocalPageRank scores. Detals of the SocalPageRank algorth are presented as follows. Algorth 2: SocalPageRank (SPR) Step Input: Assocaton atrces M PU, M, and M UA and the rando ntal SocalPageRank score P 0 504

5 Sesson: Search Qualty and Precson Step 2 Step 3: Do: U U P A = M P = M A = M + = M = M = M T PU T UA T UA PU Untl P converges. Output: P U A P A U (5.) (5.2) (5.3) (5.4) (5.5) (5.6) P * : the converged SocalPageRank score. Step does the ntalzaton. In Step 2, P, U, and A denote the popularty vectors of pages, users, and annotatons n the th teraton. P, U, and A are nteredate values. As llustrated n Fgure 3, the ntuton behnd Equaton (5) s that the users popularty can be derved fro the pages they annotated (5.); the annotatons popularty can be derved fro the popularty of users (5.2); slarly, the popularty s transferred fro annotatons to web pages (5.3), web pages to annotatons (5.4), annotatons to users (5.5), and then users to web pages agan (5.6). Fnally, we get P * as the output of SocalPageRank (SPR) when the algorth converges. Saple SPR values are gven n the rght part of Fgure 3. Fgure 3. Illustraton of qualty transton between the users, annotatons, and pages n the SPR algorth In each teraton, the te coplexty of the algorth s O(N U N P + N A N P + N U N A ). Snce the adacency atrces are very sparse n our data set, the actual te coplexty s far lower. However, n Web envronent, the sze of data are ncreasng at a fast speed, and soe acceleraton to the algorth (lke [7] for PageRank) should be developed Convergence of SPR Algorth Here, we gve a bref proof of the convergence of the SPR algorth. It can be derved fro the algorth that: where Web page annotators (5.5) (5.2) (5.3) (5.4) Socal annotatons (a) (5.) (5.6) P +, (6) T T + = ( M M ) P = ( M M ) P0 b c a Web pages SocalPageRanks: SPR(a)= SPR(b)= SPR(c) =0.02 (b) (5) M = M PU MUA M. (7) A standard result of lnear algebra (e.g. []) states that f M s s a syetrc atrx, and v s a vector not orthogonal to the prncpal egenvector of the atrx ω (M s ), then the unt vector n the drecton of (M s ) k v converges to ω (M s ) as k ncreases wthout bound. Here MM T s a syetrc atrx and P 0 s not orthogonal to ω (MM T ), so the sequence P converges to a lt P *, whch sgnals the ternaton of the SPR algorth. 3.4 Dynac Rankng wth Socal Inforaton 3.4. Dynac Rankng Method Due to the large nuber of features, odern web search engnes usually rank results by learnng a rank functon. Many ethods have been developed for autoatc (or se-autoatc) tunng of specfc rankng functons. Prevous work estates the weghts through regresson [26]. Recent work on ths rankng proble attepts to drectly optze the orderng of the obects [3, 22, 32]. As dscussed n [5], there are generally two ways to utlze the explored socal features for dynac rankng of web pages: (a) treatng the socal actons as ndependent evdence for rankng results, and (b) ntegratng the socal features nto the rankng algorth. In our work, we ncorporate both slarty and statc features exploted fro socal annotatons nto the rankng functon by usng RankSVM [32] Features We dvded our feature set nto two utually exclusve categores: query-page slarty features and page s statc features. Table descrbes each of these feature categores n detal. Table. Features n socal search A: query-docuent features DocSlarty Slarty between query and page content TerMatchng (TM) SocalSRank (SSR) B: docuent statc features GooglePageRank (PR) SocalPageRank (SPR) Slarty between query and annotatons usng the ter atchng ethod. Slarty between query and annotatons based on SocalSRank. The web page s PageRank obtaned fro the Google s toolbar I. The popularty score calculated based on SocalPageRank algorth. 4. EXPERIMENTAL RESULTS 4. Delcous Data There are any socal bookark tools on Web [30]. For the experent, we use the data crawled fro Delcous durng May, 2006, whch conssts of,736,26 web pages and 269,566 dfferent annotatons. Although the annotatons fro Delcous are easy for huan users to read and understand, they are not desgned for achne use. For exaple, users ay use copound annotatons n varous 505

6 Sesson: Search Qualty and Precson fors such as ava.prograng or ava/prograng. We splt these annotatons nto standard words wth the help of WordNet [35] before usng the n the experents. 4.2 Evaluaton of Annotaton Slartes In our experents, the SocalSRank algorth converged wthn 2 teratons. Table 2 shows the selected annotatons fro four categores, together wth ther top 4 seantcally related annotatons. Wth the explored slarty values, we are able to fnd ore seantcally related web pages as shown later SPR vs. PageRank (Dstrbuton Analyss) Fgure 4 shows the average counts of annotatons and annotators over web pages wth dfferent PageRank values, and for the Unque Annotaton lne, the value eans the count of annotatons that are dfferent wth each other. Fro the fgure, we can conclude that n ost cases, the page wth a hgher PageRank s lkely to be annotated by ore users wth ore annotatons. Table 2. Explored slar annotatons based on SocalSRank Technology related: dubln deban Econoy related: adsense etadata, seantc, standard, owl dstrbuton, dstro, ubuntu, lnux sense, advertse, entrepreneur, oney 00 nuber, drectory, phone, busness Entertanent related: albu chat Entty related: ensten chrstan gallery, photography, panoraa, photo essenger, abber,, acosx scence, skeptc, evoluton, quantu devote, fath, relgon, god 4.3 Evaluaton of SPR Results We obtaned each web page s SPR score usng the algorth descrbed n secton 3.3. In our experents, the algorth converged wthn 7 teratons. Each page s PageRank was also extracted fro the Google s toolbar I durng July, Hereafter, we use PageRank to depct the extracted Google s PageRank by default. Fgure 4. Average count dstrbuton over PageRank To further nvestgate the dstrbuton, counts of annotatons and annotators for all collected pages wth dfferent PageRank values are gven n Fgure 5(a). It s easy to see that the pages wth each PageRank value dversfy a lot on the nuber of annotatons and users. Web pages wth a relatvely low PageRank ay own ore annotatons and users than those who have hgher PageRank. For exaple, soe pages wth PageRank 0 have ore users and annotatons than those who have PageRank 0. We appled the SPR algorth to the collected data. For easy understandng, SPR s noralzed nto a scale of 0-0 so that SPR and PageRank have the sae nuber of pages n each grade fro 0 to 0. Fgure 5(b) shows the detaled counts of annotatons and users of pages wth dfferent SocalPageRank values. It s easy to see that SocalPageRank successfully characterzes the web pages popularty degrees aong web annotators. (a) Annotatons and users over PageRank (b) Annotatons and users over SocalPageRank Fgure 5. Dstrbuton analyss of socal annotatons 506

7 Sesson: Search Qualty and Precson SPR vs. PageRank (Case Study) Table 3 shows case studes of PageRank vs. SPR. Soe web stes have hgh SocalPageRank but low PageRank, e.g., or vce versa, e.g., Soe web stes are both popular for web creators and web users, e.g., and soe are both not, e.g., Fro the case studes we conclude that the web creators preferences do dffer fro the web users (web page annotators) whch are successfully characterzed by SPR. Table 3. Case studes of SPR vs. PageRank Web Pages PR SPR htl Dynac Rankng wth Socal Annotaton 4.4. Query Set We use the data descrbed n Secton 4. to evaluate the effectveness of ntegratng socal annotatons nto dynac rankng. Both anually and autoatcally constructed query sets are used. ) Manual query set (MQ): we asked a group of CS students to help us collect 50 queres and ther correspondng ground truths. Most of the 50 queres are about coputer scence snce ost of the Delcous docuents we crawled are about coputer scence. We also selected soe queres about other felds to guarantee the dversty of queres, e.g., NBA Houston Rockets and Martn Luther Kng. The ground truth of each query was bult by browsng top 00 docuents returned by Lucene search engne. Fnally, the queres were assocated wth 304 relevant docuents n total. The average length of the anual queres s ) Autoatc query set (AQ): we autoatcally extracted 3000 queres and ther correspondng ground truths fro the ODP as follows. Frst, we erged the Delcous data wth ODP and dscarded ODP categores that contan no Delcous URLs. Second, we randoly sapled 3000 ODP categores, extracted the category paths as the query set and extracted the correspondng web pages as the ground truths. Note that the ter TOP n the category path was dscarded. For exaple, the category path TOP/Coputers/Software /Graphcs would be extracted as the query Coputers Software Graphcs. Fnally, we got 3000 queres wth 4233 relevant docuents. The average length of autoatc queres s Syste Setup In our experent, the DocSlarty s taken as the baselne. Ths slarty s calculated based on the BM25 forula [29], whose ter frequency coponent s pleented as follows: k * f ( t, d) TF( t, d) = k *(( b) + b * doclen avgdoclen) + f ( t, d), () where f(t,d) eans the ter count of t n docuent d. In the experent, k and b are set to and 0.3, respectvely. To evaluate the features proposed n Table, we frst extracted the top 00 docuents returned by BM25 for each query and then created fve dfferent rando splts of 40 tranng and 0 testng queres on MQ set, and 2,400 tranng and 600 testng queres on AQ set. The splts were done randoly wthout overlaps. Then, RankSVM s appled to learn weghts for all the features descrbed n Table. The default regularzaton paraeter s set to Evaluaton Metrcs We evaluate the rankng algorths over two popular retreval etrcs, naely Mean Average Precson (M), and Noralzed Dscounted Cuulatve Gan (NDCG). Each etrc focuses on one aspect of the search perforance, as descrbed below. Mean Average Precson: We anly used M to evaluate search perforance. It s defned as the ean of average precson over queres. The average precson for each query s defned as: M average _ precson = p( ) * Δr( ), (9) = where p() denotes the precson over the top results, and Δr() s the change n recall fro - to. NDCG at K: NDCG s a retreval easure devsed specfcally for web search evaluaton [6]. It s well suted to web search evaluaton as t rewards relevant docuents that are top-ranked ore heavly than those ranked lower. For a gven query q, the ranked results are exaned n a top-down fashon, and NDCG s coputed as: N q = M K q = r() ( 2 ) / log( + ), (0) where M q s a specally calculated noralzaton constant for akng a perfect orderng obtan an NDCG value of ; and each r() s an nteger relevance label (0= Irrelevant and = Relevant ) of the result returned at poston Dynac Rankng Usng Socal Slarty Fgure 6 shows the coparson between NDCG of the teratchng and socal-atchng on the AQ set. We can easly fnd that usng ter-atchng to utlze socal annotatons does prove the perforance of web search. By ncorporatng socal atchng, the perforance can be further proved. A slar concluson can be drawn fro Table 4, whch shows the coparson of M on the two query sets. 507

8 Sesson: Search Qualty and Precson Table 5. Coparson of M between statc features Method MQ50 AQ3000 Baselne Baselne +PR Baselne +SPR Dynac Rankng Usng Both SSR and SPR By ncorporatng both SocalSRank and SocalPageRank, we can acheve the best search result as shown n Table 6. T-tests on M show statstcally sgnfcant proveents (p-value<0.05). Table 6. Dynac rankng wth both SSR and SPR Fgure 6. NDCG at K for Baselne, Baselne +TM, and Baselne +SSR for varyng K Table 4. Coparson of M between slarty features Method MQ50 AQ3000 Baselne Baselne +TM Baselne +SSR Dynac Rankng Usng SPR Fgure 7 shows the coparson between NDCG of PageRank and SocalPageRank on the AQ set. Both SPR and PageRank beneft web search. The better result s acheved by SPR. Agan, slar concluson can be drawn fro Table 5, whch shows the coparson of M on the two query sets. Method MQ50 AQ3000 Baselne Baselne+SSR,SPR (+4.0%) (+25.02%) Case Study To understand how these proveents are acheved, we present a case study here. For splcty, the slarty between the query and docuent content s not consdered. Gven the query arfare, 3 web pages are assocated through the socal annotatons and the top-4 web pages accordng to SPR scores are shown n Table 7. Through the revews of the travel stes lke excellent-roantc-vacatons 5, we can conclude that the socal annotatons are useful and SPR are really effectve. For exaple, the ste wth top SPR score s also ranked frst by excellent-roantc-vacatons and called Google of Travel Stes. Table 7. Web pages assocated by annotaton arfare URLs SPR Then by usng SSR rank, we fnd the top-4 slar tags to arfare are tcket, flght, hotel, and arlne. Through the analyss, we fnd ost of the top ranked web stes n Table 7 are annotated by these slar annotatons as well. Besdes, these slar annotatons also ntroduce soe new web pages. Table shows the top-spr URLs that are not annotated by arfare n our corpus. Fgure 7. NDCG at K for BM25, BM25-PR, and BM25-SPR for varyng K

9 Sesson: Search Qualty and Precson Table. Illustraton of seantcally related web pages based on SocalSRank for query arfare Annotaton Seantc Related Web Pages SPR tcket flght hotel arlne Fro the above table, we fnd that ost of the newly ntroduced web pages are relevant to arfare. For exaple, s an nterestng ste that s annotated by both tcket and arlne. The slar tags ay also ntroduce soe nose pages, e.g., and are related to concert tcket and flght sulator, respectvely. However, the nose pages wll not be ranked hgh n our settng as no other slar annotatons wll be assgned to t. 5. DISSCUSSION As we have observed that the socal annotatons do beneft web search. There are stll several probles to further address. 5. Annotaton Coverage Frst, the user subtted queres ay not atch any socal annotaton. In ths case, SSR wll not be appled and SPR wll keep on provdng the ost popular web pages to the user. Second, any web pages ay have no annotatons. These web pages wll beneft fro nether SSR nor SPR. The pages that are not annotated can be roughly dvded nto three categores: ) newly eergng web pages: these pages are too fresh to be annotated or even learnt; 2) key-page-assocated web pages: these pages are not annotated because they can be accessed easly va the key pages such as hub pages and hoepages whle users tend to annotate key pages only; 3) unnterestng web pages: these pages ay nterest no user. The eergence of new web pages usually does not nfluence the socal search a lot snce the socal annotaton systes are senstve to new thngs. For exaple, [] shows that popular annotatons can be found over te. Wth the help of the senstvty of these systes and the SSR algorth, we can quckly dscover new valuable web pages wth a sall aount of annotatons. As for key-page-assocated pages, one feasble soluton s to propagate the annotatons fro the key pages to the. As for unnterestng pages, t s beleved that the lack of annotatons would not affect the socal search on the whole. 5.2 Annotaton Abguty Annotaton abguty s another proble concerned wth SSR,.e., SSR ay fnd the slar ters to the query ters whle fal to dsabguate ters that have ore than one eanngs. For exaple, as has been shown n the case studes, tcket ay refer to ether arplane tcket or concert tcket, and ters wth these two dfferent eanngs wll be xed up. In [36], Wu et al. studed the proble of annotaton abguty by usng a xture odel [3]; however, t s not sutable for the web search due to ts hgh 9 9 coputatonal coplexty. Soe effcent dsabguaton ethods ay be requred for further provng the perforance of SSR. However, the abguty proble does not affect the search a lot snce ths proble can be lghtened by query word collocaton and word senses skewed dstrbuton [20]. 5.3 Annotaton Spang Intally, there are few ads or spas n socal annotatons. However, as socal annotaton becoes ore and ore popular, the aount of spa could drastcally ncrease n the near future and spang wll becoe a real concern for socal search [4]. Both SSR and SPR proposed n ths paper take the assupton that the socal annotatons are good suares of web pages, so alcous annotatons have a good opportunty to har the search qualty. There are generally two ways n preventng the spa annotatons. ) Manually or se-autoatcally deletng spa annotatons and punshng users who abuse the socal annotaton syste. Such work usually reles on servce provders; 2) Flterng out spa annotatons by usng statstcal and lngustc analyss before the use of SSR and SPR. Ths should be the an approach we wll study. 6. CONCLUSION In ths paper, we studed the novel proble of ntegratng socal annotatons nto web search. We observed that the fast eergng annotatons provded not only a ult-faceted suary but also a good ndcator of the qualty of web pages. Specfcally, socal annotatons could beneft web search n both slarty rankng and statc rankng. Two novel teratve algorths have been proposed to capture the socal annotatons capablty on slarty rankng and statc rankng, respectvely. The experental results showed that SSR can successfully fnd the latent seantc relatons aong annotatons and SPR can provde the statc rankng fro the web annotators perspectve. Experents on two query sets showed that both SPR and SSR could beneft web search sgnfcantly. The an contrbutons can be concluded as follows: ) The proposal to study the proble of usng socal annotatons to prove the qualty of web search. 2) The proposal of the SocalSRank algorth to easure the assocaton aong varous annotatons. 3) The proposal of the SocalPageRank algorth to easure a web page s statc rankng based on socal annotatons. It s ust a begnnng to ntegrate socal annotatons nto web search. In the future, we would optze the proposed algorths and explore ore sophstcated socal features to prove the socal search qualty. 7. ACKNOWLEDGEMENT The authors would lke to thank IBM Chna Research Lab for ts contnuous support to and cooperaton wth Shangha JaoTong Unversty. The authors would also lke to express ther grattude to Shenglang Xu, one of our tea ebers, for hs excellent work on data preparaton. The authors also apprecate the valuable suggestons of Le Zhang, Yangbo Zhu, Lnyun Fu, Hapng Zhu, and Ke Wu. In the end, the authors would lke to thank the three anonyous revewers for ther elaborate and helpful coents. 509

10 Sesson: Search Qualty and Precson. REFERENCES [] A. Hotho, R. Jaschke, C. Schtz, and G. Stue. Inforaton Retreval n Folksonoes: Search and Rankng. In: Proc. of ESWC 2006, pp.4-426, [2] A. Mathes. Folksonoes Cooperatve Classfcaton and Councaton through Shared Metadata. Deceber [3] C. Burges, T. Shaked, E. Renshaw, A. Lazer, M. Deeds, N. Halton, and G. Hullender. Learnng to rank usng gradent descent. In: Proc. of the ICML 2005, Bonn, Gerany, [4] Delcous: [5] E. Agchten, E. Brll, and S. Duas. Iprovng Web Search Rankng by Incorporatng User Behavor Inforaton. In Proc. of SIGIR 2006, August 6, 2006 [6] E. Quntarell. Folksonoes: power to the people. Paper presented at the ISKO Italy-UnMIB eetng. June 2005 [7] F. McSherry. A unfor approach to accelerated PageRank coputaton. In: Proc. of WWW 2005, pp , [] G. Golub, C.F. Van Loan, Matrx Coputatons, Johns Hopkns Unversty Press, 99. [9] G. Jeh, J. Wdo,. SRank: A Measure of Structural- Context Slarty. In Proc. of SIGKDD 2002, pp , 200. [0] G.-R. Xue, H.-J. Zeng, Z. Chen, Y. Yu, W.-Y., Ma, W. X, and W. Fan, Optzng Web Search Usng Web Clckthrough Data, In Proc. of CIKM 2005 pp [] G. Salton and M. J. McGll. Introducton to Modern Inforaton Retreval. McGraw-Hll, New York, 93. [2] G. Sth. Atoq: Folksonoy: socal classfcaton. fcaton.htl, Aug 3, 2004 [3] [4] [5] J. Klenberg. Authortatve sources n a hyperlnked envronent. In Proc. of 9th Annual ACM-SIAM Syposu. Dscrete Algorths, pp , 99 [6] K Jarveln and J. Kekalanen. IR evaluaton ethods for retrevng hghly relevant docuents. In Proc. of SIGIR 2000, 2000 [7] L. Page, S. Brn, R. Motwan, and T. Wnograd. The PageRank ctaton rankng: Brngng order to the web. Techncal report, Stanford Dgtal Lbrary Technologes Proect, 99. [] M. Dubnko, R. Kuar, J. Magnan, J. Novak, P. Raghavan, A. Tokns. Vsualzng Tags over Te. In: Proc. of WWW2006, pp , May 23.26, 2006 [9] M. Rchardson, and A. Prakash, and E. Brll. Beyond PageRank: Machne Learnng for Statc Rankng. In: Proc. of WWW2006, May 23-26, [20] M. Sanderson: Retreval wth good sense. Inforaton Retreval (2), pp.47-67, [2] N. Craswell, D. Hawkng, and S. Robertson Effectve Ste Fndng usng Lnk Anchor Inforaton, In: Proc. of SIGIR 200, pp , New Orleans, 200. [22] O. Dekel, C. Mannng, and Y. Snger. Log-lnear odels for label-rankng. In: Advances n Neural Inforaton Processng Systes (6). Cabrdge, MA: MIT Press, [23] P. A. Dtrev, N. Eron, M. Fontoura, and E. Shekta. Usng Annotatons n Enterprse Search. In: Proc. of WWW 2006, pp. -7, May 23.26, [24] P. Merholz. Metadata for the Masses. October 9, php [25] P. Mka Ontologes are us: a unfed odel of socal networks and seantcs. In: Proc. of ISWC pp , Nov [26] R. Herbrch, T. Graepel, and K. Oberayer. Support vector learnng for ordnal regresson. In: Proc. of the 9th Internatonal Conference on Artfcal Neural Networks, pp [27] S. A. Golder, and B. A. Huberan. Usage Patterns of Collaboratve Taggng Systes. Journal of Inforaton Scence, 32(2), pp.9-20, [2] S. Brn and L. Page, The Anatoy of a Large-Scale Hypertextual Web Search Engne, n: Coputer Networks and ISDN Systes, 30(-7) pp. 07-7, 99. [29] S. E. Robertson, S. Walker, M. Hancock-Beauleu, A. Gull, M. Lau. Okap at TREC. In:Text REtreval Conference, pp.2 30, 992. [30] T. Haond, T. Hannay, B. Lund, and J. Scott. Socal book arkng tools () - a general revew. D-Lb Magazne, (4), [3] T. Hofann, and J. Puzcha. Statstcal Models for Cooccurrence Data. Techncal report, A.I.Meo 635, MIT, 99. [32] T. Joachs. Optzng search engnes usng clckthrough data. In: Proc. of SIGKDD 2002, pp , [33] T. V. Wal. Explanng and showng broad and narrow folksonoes. explanng_and_.htl : February 2, 2005 [34] T. Westerveld., W. Kraa., and D. Hestra, Retrevng Web Pages usng Content, Lnks, URLs and Anchors, n: Proc. of TREC0, [35] WordNet: [36] X. Wu, L. Zhang, and Y. Yu. Explorng Socal Annotatons for the Seantc Web. In: Proc. of WWW 2006, pp , May 23.26, 2006 [37] Y. Hu, G. Xn, R. Song, G. Hu, S. Sh, Y. Cao, and H. L. Ttle Extracton fro Bodes of HTML Docuents and Its Applcaton to Web Page Retreval. In: Proc. of SIGIR pp ,