Query classification using topic models and support vector machine

Transcription

1 Query classfcaton usng topc models and support vector machne Deu-Thu Le Unversty of Trento, Italy Raffaella Bernard Unversty of Trento, Italy Abstract Ths paper descrbes a query classfcaton system for a specalzed doman. We take as a case study queres asked to a search engne of an art, cultural and hstory lbrary and classfy them aganst the lbrary catalogung categores. We show how clck-through lnks,.e., the lnks that a user clcks after submttng a query, can be exploted for extractng nformaton useful to enrch the query as well as for creatng the tranng set for a machne learnng based classfer. Moreover, we show how Topc Model can be exploted to further enrch the query wth hdden topcs nduced from the lbrary meta-data. The expermental evaluatons show that ths system consderably outperforms a matchng and rankng classfcaton approach, where queres (and categores) were also enrched wth smlar nformaton. 1 Introducton Query classfcaton (QC) s the task of automatcally labelng user queres nto a gven target taxonomy. Provdng query classfcaton can help the nformaton provders understand users needs based on the categores that the users are searchng for. The man challenges of ths task come from the nature of user queres, whch are usually very short and ambguous. Snce queres contan only several to a dozen words, a QC system often requres ether a rather large tranng set or an enrchment of queres wth other nformaton (Shen et al., 2006a), (Broder et al., 2007). Ths study wll focus on QC n art, culture and hstory doman, usng the Brdgeman art lbrary 1, although our framework s general enough to be used n dfferent domans. Manually creatng a tranng 1 set of queres to buld a classfer n a specfc doman s very tme-consumng. In ths study, we wll descrbe our method of automatcally creatng a tranng set based on the clck-through lnks and how we buld an SVM (Support Vector Machne) classfer wth the ntegraton of enrched nformaton. In (Le et al., 2011), t has been shown that clck-through nformaton and topc models are useful for query enrchment when the ultmate goal s query classfcaton. We wll follow ths enrchment step, but ntegrate ths nformaton nto a SVM classfer nstead of usng matchng and rankng between queres and categores as n (Le et al., 2011). The purpose of ths paper s to determne (1) whether the query enrchment wth clck-though nformaton and hdden topcs s useful for a machne learnng query classfcaton system usng SVM; and (2) whether ntegratng ths enrched nformaton nto a machne learnng classfer can perform better than the matchng and rankng system. In the next secton, we wll brefly revew the man streams of related work n QC. In secton 3, we wll descrbe the Brdgeman art lbrary. Secton 4 accounts for our proposed query classfcaton framework. In secton 5, we wll present our experment and evaluaton. Secton 6 concludes by dscussng our man achevements and proposng future work. 2 Related work Intal studes n QC classfy queres nto several dfferent types based on the nformaton needed by the user. (Broder, 2002) consdered three dfferent types of queres: nformatonal queres, navgatonal queres and transactonal queres. Ths stream of study focuses on the type of the queres, rather than topcal classfcaton of the queres. Another stream of work deals wth the problem 19 Proceedngs of the 2012 Student Research Workshop, pages 19 24, Jeju, Republc of Korea, 8-14 July c 2012 Assocaton for Computatonal Lngustcs

2 of classfyng queres nto a more complex taxonomy contanng dfferent topcs. Our study falls nto ths second stream. To classfy queres consderng ther meanng, some work consdered only nformaton avalable n queres (e.g., (Betzel et al., 2005) only used terms n queres). Some other work has attempted to enrch queres wth nformaton from external onlne dataset, e.g., web pages (Shen et al., 2006a; Broder et al., 2007) and web drectores (Shen et al., 2006b). Our work s smlar to ther n the dea of explotng addtonal dataset. However, nstead of usng search engnes as a way of collectng relevant documents, we use the metadata of the lbrary tself as a reference set. Furthermore, we employ topc models to analyze topcs for queres, rather than enrchng queres wth words selected from those webpages drectly as n (Shen et al., 2006a; Broder et al., 2007). The context of a gven query can provde useful nformaton to determne ts categores. Prevous studes have confrmed the mportance of search context n QC. (Cao et al., 2009) consdered the context to be both prevous queres wthn the same sesson and pages of the clcked urls. In our approach, we wll also consder clck through nformaton to enrch the queres and analyze topcs. In (Le et al., 2011), queres and categores are enrched wth both nformaton mned from the clckthrough lnks as well as topcs derved from a topc model estmated from the lbrary metadata. Subsequently, the queres are mapped to the categores based on ther cosne smlarty. Our proposed approach dffers from (Le et al., 2011) n three respects: (1) we enrch the queres, but not the categores (2) we employ a machne learnng system and ntegrate ths enrched nformaton as features to learn an SVM classfer (3) we assume that the category of a query s closely related to the category of the correspondng clck-through lnk, hence we automatcally create a tranng data for the SVM classfer by analyzng the query log. 3 Brdgeman Art Lbrary Brdgeman Art Lbrary (BAL) 2 s one of the world s top mage lbrares for art, culture and hstory. It contans mages from over 8,000 collectons and 2 more than 29,000 artsts, provdng a central source of fne art for mage users. Works of art n the lbrary have been annotated wth ttles and keywords. Some of them are categorzed nto a two-level taxonomy, a more fne-graned classfcaton of the Brdgeman browse menu. In our study, we do not use the mage tself but only the nformaton assocated wth t,.e., the ttle, keywords and categores. We wll take the 55 top-level categores from ths taxonomy, whch have been organzed by a doman expert, as our target taxonomy. 4 Buldng QC usng topc models and SVM Followng (Le et al., 2011), we enrch queres both wth the nformaton mned from the lbrary va clck-through lnks and the nformaton collected from the lbrary metadata va topc modelng. To perform the query enrchment wth topcs derved from the lbrary metadata, there are several mportant steps: Collectng and organzng the lbrary metadata as a reference set: the lbrary metadata contans the nformaton about artworks that have been annotated by experts. To take advantage of ths nformaton automatcally, we collected all annotated artworks and organzed them by ther gven categores. Estmatng a topc model for ths reference set: Ths step s performed usng hdden topc analyss models. In ths framework, we choose to use latent drchlet allocaton, LDA (Ble et al., 2003b). Analyzng topcs for queres and ntegratng topcs nto data for both the tranng set and new queres: After the reference set has been analyzed usng topc models, t wll be used to nfer topcs for queres. The topc model wll then be ntegrated nto the data to buld a classfer. 4.1 Query enrchment va clck-through lnks We automatcally extracted clck-through lnks from the query log (whch provdes us wth the ttle of the mage that the user clcks) to enrch the query, represented as a vector q, wth the ttle of one randomly-chosen clck-through assocated wth t. To further explot the clck-through lnk, we fnd the correspondng artwork and extract ts keywords: q t kw, where t, kw are the vectors of words 20

3 n the ttle and keywords respectvely. 4.2 Hdden Topc Models The underlyng dea s based upon a probablstc procedure of generatng a new set of artworks, where each set refers to ttles and keywords of all artworks n a category: Frst, each set w m = (w m,n ) Nm n=1 s generated by samplng a dstrbuton over topcs ϑ m from a Drchlet dstrbuton (Dr( α )), where N m s the number of words n that set m. After that, the topc assgnment for each observed word w m,n s performed by samplng a word place holder z m,n from a multnomal dstrbuton (Mult( ϑ m )). Then a word w m,n s pcked by samplng from the multnomal dstrbuton (Mult( ϕ zm,n )). Ths process s repeated untl all K topcs have been generated for the whole collecton. Table 1: Generaton process for LDA M: the total number of artwork sets K: the number of (hdden/latent) topcs V : vocabulary sze α, β : Drchlet parameters ϑ m: topc dstrbuton for document m ϕ k : word dstrbuton for topc k N m: the length of document m z m,n: topc ndex of nth word n document m w m,n: a partcular word for word placeholder [m, n] Θ = { ϑ m} M m=1: a M K matrx Φ = { ϕ k } K k=1: a K V matrx In order to estmate parameters for LDA (.e., the set of topcs and ther word probabltes Φ and the partcular topc mxture of each document Θ), dfferent nference technques can be used, such as varatonal Bayes (Ble et al., 2003b), or Gbbs samplng (Henrch, 2004). In ths work, we wll use Gbbs samplng followng the descrpton gven n (Henrch, 2004). Generally, the topc assgnment of a partcular word t s computed as: p(z =k z, w)= [ V n (t) k, + β t v=1 n(v) k +β v] 1 n (k) m, + α k [ K j=1 n(j) m +α j ] 1 (1) where n (t) k, s the number of tmes the word t s assgned to topc k except the current assgnment; V v=1 n(v) k 1 s the total number of words assgned to topc k except the current assgnment; n (k) m, s the number of words n set m assgned to topc k except the current assgnment; and K j=1 n(j) m 1 s the total number of words n set m except the current word t. In normal cases, Drchlet parameters α, and β are symmetrc, that s, all α k (k = 1..K) are the same, and smlarly for β v (v = 1..V ). 4.3 Hdden topc analyss of the Brdgeman metadata The Brdgeman metadata contans nformaton about artworks n the lbrary that have been annotated by the lbrarans. We extracted ttles and keywords of each artwork, those for whch we had a query wth a clck-through lnk correspondng to t, and grouped them together by ther sub-categores. Each group s consdered as a document w m = (w m,n ) Nm n=1, wth the number of total documents M = 732 and the vocabulary sze V = 136K words. In ths experment, we fx the number of topcs K = 100. We used the GbbsLDA++ mplementaton 3 to estmate ths topc model. 4.4 Buldng query classfer wth hdden topcs Let Q = { q } =1 =N be the set of all queres enrched va the clck-through lnks, where each enrched query s q = q t kw. We also performed Gbbs samplng for all q n order to estmate ts topc dstrbuton ϑ = {ϑ,1,..., ϑ,k } where the probablty ϑ,k of topc k n q s computed as: + α k ϑ,k = (2) + α j where n (k) s the number of words n query assgned to topc k and n (j) s the total number of words appearng n the enrched query. In order to ntegrate the topc dstrbuton ϑ = {ϑ,1,..., ϑ,k } nto the vector of words q = {w,1, w,2,..., w,n }, followng (Phan et al., 2010), we only keep topcs whose ϑ,k s larger than a threshold cut-off and use a scale parameter to do the dscretzaton for topcs: the number of tmes topc k ntegrated to q s round(ϑ scale). After that, we buld a Support Vector Machne classfer usng SVM lght V n (k) K j=1 n(j) 21

4 5 Evaluaton In ths secton, we wll descrbe our tranng set, gold standard and the performance of our system n comparson wth the one n (Le et al., 2011). 5.1 Tranng set Manually annotatng queres to create a tranng set n ths doman s a dffcult task (e.g., t requres the expert to search the query and look at the pcture correspondng to the query, etc.). Therefore, we have automatcally generated a tranng set by explotng a 6-month query log as follow. Frst, each query has been mapped to ts clckthrough nformaton to extract the sub-category assocated to the correspondng mage. Then, from ths sub-category, we obtaned ts correspondng top-cateogry (among the 55 we consder) as defned n BAL taxonomy. The dstrbuton of queres n dfferent categores vares qute a lot among the 55 target categores reflectng the artwork dstrbuton (e.g., there are many more artworks n the lbrary belongng to the category Relgon and Belef than to the category Costume and Fashon ). We have preserved such dstrbuton over the target categores when selectng randomly the 15,490 queres to buld our tranng set. After removng all punctuatons and stop words, we obtaned a tranng set contanng 50,337 words n total. Each word n ths set serves as a feature for the SVM classfer. 5.2 Test set We used the test set of 1,049 queres used n (Le et al., 2011), whch s separate from the tranng set. These queres have been manually annotated by a BAL expert (up to 3 categores per query). Note that these queres have also been selected automatcally whle preservng the dstrbuton over the target categores observed n the 6-month query log. We call ths the manual gold standard. In addton, we also made use of another gold standard obtaned by mappng the clck-through nformaton of these queres wth ther categores, smlar to the way n whch we obtan the tranng set. We call ths the va-ct gold standard. 5.3 Expermental settngs To evaluate the mpact of clck-though nformaton and topcs n the classfer, we desgned the followng experments, where QR s the method wthout any enrchment and QR-CT -HT s wth the enrchment va both clck-through and hdden topcs. Settng Query enrchment QR q QR-HT q HT QR-CT q = q + t + kw QR-CT -HT q HT q : query q : query enrched wth clck-through nformaton t : clck-through mage s ttle kw: clck-through mage s keywords HT : hdden topcs from Brdgeman metadata Table 2: Expermental Settng Hts Settng Manual GS va-ct # 1 # 2 # 3 T op 3 GS QR QR-HT QR - CT QR - CT - HT Table 3: Results of query classfcaton: number of correct categores found (for 1,049 queres) Fgure 1: The mpact of clck-through nformaton wth matchng-rankng (mr) and our approach (svm) To answer our frst research queston, namely whether clck-through nformaton and hdden topcs are useful for ths query classfer, we examne the number of correct categores found by the classfer bult both wth and wthout the enrchment. The results of the experment are reported n Table 3. As can be seen from the table, we notce that the clckthrough nformaton plays an mportant role. In par- 22

5 tcular, t ncreases the number of correct categores found from 311 to 388 (compared wth the manual GS) and from 231 to 266 (usng the va-ct GS). To answer our second research queston, namely whether ntegratng the enrched nformaton nto a machne learnng classfer can perform better than the matchng and rankng method, we also compare the results of our approach wth the one n (Le et al., 2011). Fgure 1 shows the mpact of the clckthrough nformaton for the SVM classfer (svm) n comparson wth the matchng and rankng approach (mr). Fgure 2 shows the mpact of the hdden topcs n both cases. We can see that n both cases our classfer outperforms the matchng-rankng one consderably (e.g., from 183 to 388 correct categores found n the QR-CT-HT method). Fgure 2: The mpact of hdden topcs wth matchngrankng (mr) and our approach (svm) However, n the case where we use only queres wthout clck-through nformaton, we can see that hdden topcs do not brng a very strong mpact (the number of correct categores found only slghtly ncreases by 7 - usng the manual gold standard). The result mght come from the fact that ths topc model was bult from the metadata, usng only clckthrough nformaton, but has not been learned wth queres. 6 Concluson In ths study, we have presented a machne learnng classfer for query classfcaton n an art mage archve. Snce queres are usually very short, thus dffcult to classfy, we frst extend them wth ther clck-through nformaton. Then, these queres are further enrched wth topcs learned from the BAL metadata followng (Le et al., 2011). The result from ths study has confrmed agan the effect of clck-through nformaton and hdden topcs n the query classfcaton task usng SVM. We have also descrbed our method of automatcally creatng a tranng set based on the selecton of queres mapped to the clck-through lnks and ther correspondng avalable categores usng a 6-month query log. The result of ths study has shown a consderable ncrease n the performance of ths approach over the matchng-rankng system reported n (Le et al., 2011). 7 Future work For future work, we are n the process of enhancng our expermentaton n several drectons: Consderng more than one clck-through mage per query: In ths work, we have consdered only one category per query to create the tranng set, whle t mght be more reasonable to take nto account all clck-through mages of a gven query. In future work, we plan to enrch the queres wth ether all clck-through mages or wth the most relevant one nstead of randomly pckng one clck-through mage. In many cases, a clck-through lnk s not necessarly related to the meanng of a query (e.g., when users just randomly clck on an mage that they fnd nterestng). Thus, t mght be useful to flter out those clck-through mages that are not relevant. Enrchng queres wth top hts returned by the BAL search engne: In the query logs, there are many queres that do not have an assocated clckthrough lnk. Hence, we plan to explot other enrchment method that do not rely on those lnks, n partcular we wll try to explot the nformaton comng from the top returned hts gven by the lbrary search engne. Analyzng queres n the same sesson: It has been shown n some studes (Cao et al., 2009) that analyzng queres n the same sesson can help determne ther categores. Our next step s to enrch a new query wth the nformaton comng from the other prevous queres n the same sesson. Optmzng LDA hyperparameters and topc number selecton: Currently, we fxed the number of topcs K = 100, the Drchlet hyperparameters α = 50/K = 0.5 and β = 0.1 as n (Grffths and 23

6 Steyvers, 2004). In the future, we wll explore ways to optmze these nput values to see the effect of dfferent topc models n our query classfcaton task. Explotng vsual features from the BAL mages: The BAL dataset provdes an nterestng case study n whch we plan to further analyze mages to enrch queres wth ther vsual features. Combnng text and vsual features has drawn a lot of attenton n the IR research communty. We beleve that explotng vsual features from ths art archve could lead to nterestng results n ths specfc doman. A possble approach would be extractng vsual features from the clck-through mages and representng them together wth textual features n a jont topc dstrbuton (e.g., (Ble et al., 2003a; L et al., 2010)). Comparng system wth other approaches: In the future, we plan to compare our system wth other query classfcaton systems and smlar technques for query expanson n general. Furthermore, the evaluaton phase has not been carred out thoroughly snce t was dffcult to compare the one-class output wth the gold-standard, where the number of correct categores per query s not fxed. In the future, we plan to explot the output of our mult-class classfer to assgn up to three categores for each query and compute the precson at n. Acknowledgments Ths work has been partally supported by the GALATEAS project ( CIP-ICT PSP ) funded by the European Unon under the ICT PSP program. References Steven M. Betzel, Erc C. Jensen, Ophr Freder, and Davd Grossman Automatc web query classfcaton usng labeled and unlabeled tranng data. In In Proceedngs of the 28th Annual Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval, pages ACM Press. Davd M. Ble, Mchael I, Davd M. Ble, and Mchael I. 2003a. Modelng annotated data. In In Proc. of the 26th Intl. ACM SIGIR Conference. Davd M. Ble, Andrew Y. Ng, and Mchael I. Jordan. 2003b. Latent drchlet allocaton. J. Mach. Learn. Res., 3: , March. Andre Z. Broder, Marcus Fontoura, Evgeny Gabrlovch, Amruta Josh, Vanja Josfovsk, and Tong Zhang Robust classfcaton of rare queres usng web knowledge. In Proceedngs of the 30th annual nternatonal ACM SIGIR conference on Research and development n nformaton retreval, SIGIR 07, pages , New York, NY, USA. ACM. Andre Broder A taxonomy of web search. SIGIR Forum, 36:3 10, September. Huanhuan Cao, Derek Hao Hu, Dou Shen, Dax Jang, Jan-Tao Sun, Enhong Chen, and Qang Yang Context-aware query classfcaton. In SIGIR 09, The 32nd Annual ACM SIGIR Conference. Thomas L Grffths and Mark Steyvers Fndng scentfc topcs. Proceedngs of the Natonal Academy of Scences of the Unted States of Amerca, 101 Suppl 1(Suppl 1): Gregor Henrch Parameter estmaton for text analyss. Techncal report. Deu-Thu Le, Raffaella Bernard, and Edwn Vald Query classfcaton va topc models for an art mage archve. In Recent Advances n Natural Language Processng, RANLP, Bulgara. L-Ja L, Chong Wang, Yongwhan Lm, Davd Ble, and L Fe-Fe Buldng and usng a semantvsual mage herarchy. In The Twenty-Thrd IEEE Conference on Computer Vson and Pattern Recognton, San Francsco, CA, June. Xuan-Heu Phan, Cam-Tu Nguyen, Deu-Thu Le, Le- Mnh Nguyen, Susumu Horguch, and Quang-Thuy Ha A hdden topc-based framework towards buldng applcatons wth short web documents. IEEE Transactons on Knowledge and Data Engneerng, 99(PrePrnts). Dou Shen, Rong Pan, Jan-Tao Sun, Jeffrey Junfeng Pan, Kangheng Wu, Je Yn, and Gang Yang. 2006a. Query enrchment for web-query classfcaton. ACM Transactons on Informaton Systems, 24(3): Dou Shen, Jan-Tao Sun, Qang Yang, and Zheng Chen. 2006b. Buldng brdges for web query classfcaton. In SIGIR