Relevance Ranking using Kernels

Transcription

1 Relevance Ranking using Kernels Jun Xu 1, Hang Li 1, and Chaoliang Zhong 2 1 Microsof Research Asia, 4F Sigma Cener, No. 49 Zhichun Road, Beijing, China Beijing Universiy of Poss and Telecommunicaions, Beijing, China {junxu,hangli}@microsof.com, and clzhong@cn.fujisu.com Absrac. This paper is concerned wih relevance ranking in search, paricularly ha using erm dependency informaion. I proposes a novel and unified approach o relevance ranking using he kernel echnique in saisical learning. In he approach, he general ranking model is defined as a kernel funcion of query and documen represenaions. A number of kernel funcions are proposed as specific ranking models in he paper, including BM25 Kernel, LMIR Kernel, and KL Divergence Kernel. The approach has he following advanages. (1) The (general) model can effecively represen differen ypes of erm dependency informaion and hus can achieve high performance. (2) The model has srong connecions wih exising models such as BM25 and LMIR. (3) I has solid heoreical background. (4) The model can be efficienly compued. Experimenal resuls on web search daase and TREC daases show ha he proposed kernel approach ouperforms MRF and oher baseline mehods for relevance ranking. 1 Inroducion Relevance ranking is sill a cenral issue of research in Informaion Rerieval. Tradiionally, relevance ranking models are defined based on he bag-of-words assumpion, i.e., query and documen are viewed as wo ses of words. These include Vecor Space Model (VSM) [17], BM25 [15], and Language Models for IR (LMIR) [22]. There is a clear rend in IR recenly on sudy of models beyond bag-of-words, paricularly under he noion of erm dependency or proximiy, which assumes erms are dependen and akes ino consideraion in relevance ranking he cooccurrences of query erms in documens. For example, if he query is machine learning book, hen a documen conaining he phrase machine learning and he erm book should be more relevan han a documen in which he erms machine, learning, and book occur separaely. I is obvious from he example ha dependency informaion needs o be leveraged in relevance ranking. For insance, a model for using erm dependency based on Markov Random Fields (MRF) has been proposed [11], in which he informaion on independen erms (unigrams), sequenial erms (n-grams), and dependen erms is exploied. MRF is regarded as a sae-of-he-ar mehod for using erm dependency. In our view, an ideal ranking model using erm dependency should have he following characerisics. (1) I can incorporae various ypes of erm dependency informaion o represen relevance, and hus can achieve high accuracy in ranking. (2) I has srong connecions wih exising models. (This is necessary, as exising models of BM25 and The work was conduced when Chaoliang Zhong was visiing Microsof Research Asia.

2 LMIR already perform quie well.) (3) I is based on solid heory. (4) I can be efficienly compued. Alhough previous work on erm dependencies made progresses, furher sudy on he problem sill appears o be necessary. In his paper, we aim o conduc a comprehensive sudy on relevance ranking using erm dependency, and o propose a general ranking model which has he advanages. Specifically his paper proposes a novel approach o relevance ranking, on he basis of Sring Kernel in saisical learning. I defines he relevance ranking model as a kernel funcion of query sring and documen sring. The original query sring and documen sring are mapped ino vecors in a higher dimensional Hilber Space and relevance (similariy) beween he wo represenaions is defined as do produc in he Hilber Space. As case sudies, we inroduce wo kernel funcions, BM25 Kernel and LMIR Kernel, and use one exising kernel funcion, Kullback-Leibler (KL) Divergence Kernel, in his paper. The former wo are asymmeric kernel funcions and he las is a symmeric kernel funcion. They correspond o differen ways of mapping he original query and documen srings o he Hilber Space. (1) The kernel based models can naurally represen various ypes of erm dependency informaion, for example, n-grams and n-dependen-erms. Since he essence of relevance is o compare he similariy beween query and documen, a richer model on he similariy beween query and documen can hen be realized wih he approach and high performance in relevance ranking can be achieved. (2) BM25 Kernel and LMIR Kernel respecively include convenional BM25 model and LMIR model as special cases. (Noe ha many oher relevance models can also be inerpreed as kernel funcions, for example, he radiional VSM.) Wihin he kernel framework, differen ranking models can be easily sudied and compared. (3) Kernel mehods are becoming very popular in saisical learning. I is aracive because of is powerful represenabiliy. The proposed approach using kernel hus has a solid heoreical background. I bridges he gap beween relevance ranking and machine learning, and provide a framework for auomaically learning ranking model using kernel machines. (4) Kernel funcions are essenially do produc. This enables an efficien compuaion of kernel funcions in search. The implicaion of he work is far beyond using erm dependency. The proposed kernel-based ranking models are acually generalized versions of convenional ranking models. I provides a new view, namely kernel funcion, on relevance ranking. We conduced experimens using daa from a web search engine and TREC daa, and found ha BM25 Kernel, LMIR Kernel, and KL Kernel perform beer han MRF and oher bag-of-words models for relevance ranking. We conclude, herefore, ha he kernel approach proposed in his paper really has advanages. 2 Relaed Work Convenional Informaion Rerieval models such as Vecor Space Model (VSM) [17], BM25 [15], and Language Models for IR (LMIR) [22] were originally proposed based on he bag-of-words assumpion, which is obviously oo srong, as in many cases query erms (words) are relaed, and heir relaions (dependencies) should also be considered

3 in relevance ranking. The ranking models beyond he bag-of-words may be caegorized ino wo groups. One approach considers he use of erm dependency, specifically, i assumes ha he query is composed of several unis of erms (e.g., n-grams) and uilizes he occurrences of he unis in he documen in ranking. For example, Mezler and Crof [11] proposed using Markov Random Fields o characerize differen ypes of erm dependency relaions. For oher relaed work see [1, 2, 4, 13]. The oher approach considers he use of degree of approximae mach of query o documen, for example, i looks a he minimum span of he query erms appearing in he documen, also referred o as proximiy. Tao and Zhai [19] conduced a comprehensive sudy on proximiy measures for relevance ranking and found ha he proximiy measures are useful for furher improving relevance. (Also refer o [8, 6].) Kernel funcion characerizes a kind of similariy beween wo represenaions k(x, y), where x and y denoe represenaions. For example, Sring Kernel is a special ype of kernel funcion defined on ex srings, i.e., x and y are srings [5, 9]. KL Divergence Kernel is a ype of kernel funcion defined based on KL Divergence [12]. Kernel funcions have been applied ino several asks in IR. In [21], he KL Divergence Kernel was used for calculaing he similariy beween documens given a query. In [9], a mehod for ex classificaion using Sring Kernels was sudied. In [16], a kernel funcion for measuring similariy beween shor exs was proposed. In [10], a number of Sring Kernel funcions were surveyed. There are wo ways o creae (or idenify) a kernel funcion. The Mercer s heorem provides a sufficien and necessary condiion of kernel funcion, ha is, any coninuous, symmeric, posiive semi-definie funcion k(x, y) is a kernel funcion. One can also define a kernel funcion in a consrucive way, i.e., hrough definiion of mapping funcion. Usually a kernel funcion is symmeric in he sense k(x, y) = k(y, x). Asymmeric kernel funcions have also been inroduced, which saisfy k(x, y) k(y, x) [20]. Obviously asymmeric kernel funcions are more general han symmeric kernel funcions. 3 Kernel Approach The essence of relevance is o compare he similariy beween query and documen, or he maching degree beween query and documen. Differen ways of defining he similariy or maching degree lead o differen ranking models. In his paper, we use kernel echniques in saisical learning o consruc ranking models. We firs give a general model, and hen some specific examples. 3.1 General Model There are hree issues we need o consider when defining a ranking model: query represenaion, documen represenaion, and similariy calculaion beween query and documen represenaions. Here, we assume ha query and documen are wo srings of words (erms). Such a represenaion can reain all he major informaion conained in he query and documen. We hen define he similariy funcion beween he query and documen as kernel funcion beween query sring and documen sring. Suppose ha Q is he space of queries (word srings), and D is he space of documens (word srings). (q, d) is a pair of query and documen from Q and D. There is

4 a mapping funcion ϕ : Q H from he query space o he Hilber space where H denoes he Hilber Space. ϕ(q) hen sands for he mapped vecor in H from q. Similarly, here is a mapping funcion ϕ : D H from he documen space o he Hilber Space H. ϕ (d) hen sands for he mapped vecor in H from d. The similariy funcion beween q and d is defined as a kernel funcion beween q and d, k(q, d) = ϕ(q), ϕ (d) = ϕ(q) i ϕ (d) i. (1) The kernel funcion is acually he do produc beween vecors ϕ(q) and ϕ (d), respecively mapped from q and d. We call he kernel funcion k(q, d) in Eq. (1) (general) kernel-based ranking model. Noe ha k(q, d) is an asymmeric funcion, which means k(q, d) k(d, q), because ϕ and ϕ are no necessarily idenical. When ϕ and ϕ are idenical, he kernel funcion becomes a symmeric funcion, i.e., k(q, d) = k(d, q). The general kernel-based ranking model can include mos exising relevance ranking models such as VSM, BM25, LMIR, and even LSI [3], as will be made clearer laer. For example, in VSM, he query sring and he documen sring are mapped ino vecors of f-idf values and do produc beween he vecors is calculaed and used, which is a very simple example of he general ranking funcion in Eq. (1). 3.2 Model Consrucion Le us inroduce he way of creaing a kernel-based ranking model. We firs idenify a ype of dependen query erms for which we care abou he occurrences in documens. (For example, if he ype is bigram, we decompose he query machine learning book ino wo bigrams machine learning and learning book, and we look a heir occurrences in documens.) Nex we assume o respecively map he query sring and documen sring ino vecors of he ype in a Hilber Space. (For example, we map query and documen ino vecors of bigrams.) We can hen consruc a kernel funcion for he ype on he basis of he mapping funcions. Finally, we can define a ranking model by combining he kernel funcions in differen ypes using he properies of kernel funcions (cf., Secion 4.3). Tha is o say we acually creae kernel funcions. As ypes, we can consider he occurrences of independen query erms in he documen (unigrams), he occurrences of sequenial query erms in he documen (n-grams), or he occurrences of dependen query erms in he documen (n-dependen-erms). Then we can creae a vecor of unigrams, a vecor of n-grams, or a vecor of n-dependenerms in he Hilber Space. The relaion beween unigrams, n-grams, and n-dependenerms is illusraed in Fig. 1. For each ype, he se of possible unis (unigrams, n-grams, or n-dependen-erms) is fixed, provided ha he vocabulary is fixed. We denoe he se of unis as X and each elemen of i x. In his paper we give hree examples of kernel-based models. They are BM25 Kernel, LMIR Kernel, and KL Kernel. 3.3 BM25 Kernel BM25 Kernel is a generalizaion of convenional BM25 model using Sring Kernel. I has a similar form as BM25, bu is more general in he sense ha i can make use of erm dependencies in differen ypes. I is an asymmeric kernel funcion. i

5 unigram bigram w 1 w 2 w 3 w 1 w 2 w 3 wo-dependen-erms #maxskips unigram n-dependen-erms w 1 w 2 w 3 #erms No 1 bigram Yes order Fig. 1. Illusraion of unigram, n-gram, and n- dependen-erms. Fig. 2. Three facors for erm dependency. The BM25 Kernel funcion beween query q and documen d in ype is defined as BM25-Kernel (q, d) = ϕ BM25 (q), ϕ BM25 (d), where ϕ BM25 (q) and ϕ BM25 (d) are wo vecors of ype in he Hilber Space mapped from q and d. Each dimension of he vecor corresponds o a uni x: ϕ BM25 (q) x = (k 3 + 1) f (x, q) k 3 + f (x, q) and ϕ BM25 (k 1 + 1) f (x, d) (d) x = IDF (x) ( ) k 1 1 b + b f (d) avg f + f (x, d), where k 1, k 3, and b are parameers, f (x, q) and f (x, d) are frequencies of uni x wih ype in q and d, respecively, f (d) is oal number of unis wih ype in documen d, and avg f is average number of unis f (d) wih ype per documen wihin he whole collecion. IDF (x) = log d f d f (x)+0.5 d f (x)+0.5 is inverse documen frequency of uni x wih ype, where d f (x) is number of documens in which uni x wih ype occurs, and d f is oal number of documens ha have unis wih ype. BM25 Kernel in ype becomes BM25-Kernel (q, d) = IDF (x) (k 3 + 1) f (x, q) k 3 + f (x, q) x (k 1 + 1) f (x, d) ( ) k 1 1 b + b f (d) avg f + f (x, d). Finally, BM25 Kernel is defined as a linear combinaion of kernels over all he ypes: BM25-Kernel(q, d) = α BM25-Kernel (q, d), α 0. (2) 3.4 LMIR Kernel We employ Dirichle smoohing as an example. Oher smoohing mehods can also be used. The LMIR Kernel beween query q and documen d in ype is defined as LMIR-Kernel (q, d) = ϕ LMIR (q), ϕ LMIR (d) + f (q) log µ f (d) + µ, where µ is free parameer, and ϕ LMIR (q) and ϕ LMIR (d) are wo vecors of ype in he Hilber Space mapped from query q and documen d, respecively. Each dimension of

6 he vecor corresponds o a uni x: ϕ LMIR (q) x = f (x, q) and ϕ LMIR ( (d) x = log 1 + f (x, d) µp(x ) where P(x ) is probabiliy of uni x wih ype in he whole collecion. P(x ) plays a similar role as inverse documen frequency IDF (x) in BM25 Kernel. Combining he wo, he LMIR Kernel funcion in ype becomes LMIR-Kernel (q, d) = f (x, q) log x ( 1 + f (x, d) µp(x ) ) + f (q) log ), µ f (d) + µ. Finally, we have LMIR-Kernel(q, d) = α LMIR-Kernel (q, d), α 0. (3) 3.5 KL Kernel In his paper, we also use he KL Divergence kernel as a specific kernel-based ranking model, referred o as KL Kernel. Unlike BM25 Kernel and LMIR Kernel, KL Kernel is a symmeric kernel funcion. Anoher difference is ha KL Kernel represens a mapping of he query and documen ino a Hilber Space wih an infinie number of dimensions. KL Kernel was previously used in documen similariy comparison [12], and his is he firs ime i is used in relevance ranking. In KL Kernel in a ype, he query and documen are represened by disribuions. We define a disribuion P (q) = (P(x 1, q), P(x 2, q),, P(x N, q)) of unis x wih ype in query q, where P(x i, q) is probabiliy of uni x i given ype and query q, and N is size of X. Similarly, we define a disribuion P (d) = (P(x 1, d), P(x 2, d),, P(x N, d)) of unis x wih ype in documen d. Then, he KL Kernel funcion beween q and d in ype is defined as KL-Kernel (q, d) = exp { D(P (q) P (d)) D(P (d) P (q))}. Noe ha he kernel funcion represens do produc beween wo vecors in a Hilber Space wih infinie number of dimensions. (We can use Mercer s heorem o prove ha i is a kernel funcion. [12]) D(P (q) P (d)) is he KL divergence of P (d) from P (q): D(P (q) P (d)) = N i=1 P(x i, q) log P(x i, q) P(x i, d). The KL Kernel funcion is defined as KL-Kernel(q, d) = KL-Kernel (q, d) α. (4) According o he properies of kernel funcion, KL-Kernel(q, d) is sill a kernel. For efficiency consideraion, we acually ake logarihmic funcion of he kernel funcion. This will no change he ranking resuls. KL-Kernel (q, d) = log (KL-Kernel(q, d)) = α ( D(P (q) P (d)) D(P (d) P (q))). Smoohing is also needed in esimaion of he probabiliy disribuions in KL Kernel. In his paper we employ he Dirichle smoohing mehod [22]: P(x, d) = f (x,d)+µp(x ) f (d)+µ.

7 4 Advanages of he Kernel Approach 4.1 Represenabiliy The kernel approach o ranking has rich represenaion abiliy. There are several orhogonal facors which one needs o consider when using erm dependencies of differen ypes. They are number of erms, order preservaion, maximum number of skipping erms. Suppose ha he query is machine learning book. Number of erms indicaes wheher he erm unis we use consis of wo erms or hree erms, ec in documens ( machine learning vs machine learning book ). Order preservaion means we care abou he order of query erms ( machine learning vs learning machine ) in documens. Maximum number of skipping erms includes he cases in which here are oher erms occurring in beween he erms in quesion ( machine book ). Combining he facors above can give us differen ypes of dependencies, including some complex ones. The n-gram models belong o he cases in which differen numbers of ordered and non-skipping erms are used. The n-dependen-erms models fall ino he cases in which differen numbers of non-ordered and skipping erms are used. The unigram model is he simples. Fig. 2 shows he relaions among he facors. Each facor is represened as one axis; he firs axis corresponds o order preservaion, he second maximum number of skips, and he hird number of erms. I furher shows he posiions of differen ypes of dependencies. One can immediaely noice ha he kernel approach proposed in his paper can easily uilize he differen ypes of erm dependencies (ypes). In he experimens in his paper, we make use of hree ypes: unigram, bigram, and wo-dependen-erms. 4.2 Relaion o Convenional Models I is easy o verify ha BM25 Kernel and LMIR Kernel respecively include convenional BM25 and LMIR models as special cases. Specifically, when he ype is unigram, he kernel funcions become he exising ranking models. The kernel approach o relevance ranking proposed in his paper, hus, provides a new inerpreaion of many IR ranking models. These include no only BM25 and LMIR, bu also VSM and even LSI. Relevance ranking models are nohing bu do produc beween vecors which are nonlinearly mapped from query and documen. The kernel funcions (BM25 Kernel, LMIR Kernel, and KL Kernel) have similar shapes as funcions of uni frequency in documen (e.g., erm frequency in documen for he case of unigram). (Noe ha uni frequency in query is usually fixed o one, because queries are shor.) Fig. 3 shows plos of he funcions. From he figure we can see ha alhough BM25 and LMIR were derived from differen heories, hey acually have similar shapes as funcions of uni frequency. This can in par explain why BM25 and LMIR perform almos equally well in pracice. This also srongly sugges kernel funcions including KL Kernel would have similar performances in relevance ranking. As will be seen in Secion 5, his is in fac rue.

8 1 0.3 normalized ranking score KL Kernel BM25 LMIR LMIR BM25 KL-Unigram MRF LMIR Kernel BM25 Kernel erm frequency 0.15 MAP NDCG@1 NDCG@3 NDCG@5 KL Kernel Fig. 3. KL Kernel, BM25 Kernel, and LMIR Kernel as a funcion of uni (erm) frequency. Fig. 4. Ranking accuracies on he web search daa. 4.3 Theoreical Background Anoher advanage of aking he kernel approach is ha i has solid heoreical background. As a resul, deeper undersanding of he problem and broader use of he echnique become possible. For example, i is easy o verify ha kernel funcions have he following closure properies, which means several ways of combining kernel funcions will sill resul in kernel funcions. Le k 1 and k 2 be any wo kernels. Then he funcion k given by 1. k(x, y) = k 1 (y, x); 2. k(x, y) = k 1 (x, y) + k 2 (x, y); 3. k(x, y) = αk 1 (x, y), for all α > 0; 4. k(x, y) = k 1 (x, y) k 2 (x, y) are also kernels. Noe ha his is rue even for asymmeric kernel funcions. Tha means we can creae differen ranking models by aking sum or produc of kernel funcions. In recen years, kernel learning[18, 14], which aims o learn a kernel funcion from raining daa, has been proposed. The proposed kernel approach for relevance ranking bridges he gap beween informaion rerieval and machine learning. I provides a framework for learning opimal ranking model using machine learning. We will furher invesigae he problem in our fuure work. 4.4 Efficien Implemenaion The kernel funcions proposed above can be calculaed efficienly. This is because kernel funcions are essenially do producs and heir compuaions only need o be performed on unis which occur in query or documen. (The values of he oher unis are jus zero). For example, for D(P (q) P (d)) (wih Dirichle smoohing) in KL Kernel we acually only need o calculae = x: f (x,q)>0 f (x,d)>0 x: f (x,q)>0 f (x,d)>0 P(x, q) log P(x, q) log where µ is he smoohing parameer. P(x, q) P(x, d) + x: f (x,q)=0 f (x,d)=0 P(x, q) P(x, d) + µ µ + f (q) log f (d) + µ f (q) + µ µp(x ) µ + f (q) log f (d) + µ f (q) + µ x: f (x,q)>0 f (x,d)>0 (1 P(x )),

9 5 Experimens We conduced experimens o es he effeciveness of he proposed kernel approach, using a daase from a web search engine and TREC daa ses of OHSUMED [7] and AP. Specifically, we esed he performances of BM25 Kernel, LMIR Kernel, and KL Kernel. In he kernels, as ypes of erm dependencies we used unigram, bigram, and wo-dependen-erms. The final ranking score is defined as linear combinaion of he kernel funcion scores: Score = (1 λ 1 λ 2 ) kernel unigram + λ 1 kernel bigram + λ 2 kernel wo dependen erms, where λ 1 and λ 2 are parameers. (I is sill a kernel funcion, c.f., Secion 4.3.) BM25 and LMIR (wih Dirichle smoohing) were seleced as baselines in he experimens, as represenaives of bag-of-words models. We also viewed KL-Unigram as a baseline, in which only unigram is used in KL Kernel. MRF was also chosen as a baseline of using erm dependencies. For MRF model he same erm dependency informaion (unigram, bigram, and wo-dependen-erms) was used for fair comparison. We adoped mean average precision (MAP) and NDCG@N (N = 1, 3, and 5 ) as evaluaion measures. The web search daase has five levels of relevance: Perfec, Excellen, Good, Fair, and Bad. We considered he firs hree as relevan when calculaing MAP. The OHSUMED daase have hree levels of relevance: definiely relevan, parially relevan, and no relevan. We considered definiely relevan and parially relevan as relevan when calculaing MAP. 5.1 Experimens on he Web Search Daase In his experimen, we used he web search daase. The daase conains abou 2.5 million web pages, and 235 long queries randomly seleced from query log. Each query has a leas 8 erms. The average query lengh is erms. We use long queries because he paper focuses on he erm dependency in search. For each query, he associaed documens are labeled by several annoaors. In oal, here are 6,271 labeled query-documen pairs. On average each query has documens labeled. We esed he accuracies of ranking models and obained he resuls in Fig. 4. We can observe an overall rend ha he kernel based ranking models perform beer han he bag-of-words models and he MRF model. Specifically, he kernel-based models ouperform heir counerpars (for example, BM25 Kernel works beer han BM25, ec). There are always kernel-based models working beer han MRF in differen seings. In he experimens, we uned wo parameers λ 1 and λ 2 for each kernel funcion. We invesigaed he sensiiviy of he performances wih respec o he wo parameers. Specifically, we changed he values of a parameer and fixed he oher a is opimal value (he opimal values are λ 1 = 0.4 and λ 2 = 0.1). The performances in erms of MAP wih respec o parameer values are repored in Fig. 5. The KL kernel has anoher parameer µ. We also carried ou experimens o invesigae how his parameer impacs he performances. We changed he values of he parameer and observed he model s performances in erms of MAP. Fig. 6 shows he performance curve. We can see ha here is a fla region near he opimal value (µ = 4.0), which means he model s performances are no sensiive o µ.

10 MAP 0.16 λ1 MAP λ parameer value µ Fig. 5. Ranking accuracies of KL Kernel w.r.. λ 1 and λ 2. Fig. 6. Ranking accuracies of KL Kernel w.r.. µ unigram 0.2 MAP unigram + bigram uingram + wo-dependenerms all MAP 0.15 LMIR LMIR Kernel BM25 Kernel LMIR Kernel KL Kernel query lengh Fig. 7. BM25 Kernel, LMIR Kernel, and KL Kernel wih differen erm dependencies. Fig. 8. Ranking accuracies of KL-Unigram and he KL Kernel by query lenghs. We conduced experimens o verify how he use of erm dependency informaion can improve ranking accuracy. Specifically, we esed he performances of BM25 Kernel (and also LMIR Kernel and KL Kernel) wih unigram only, wih unigram plus bigram, wih unigram plus wo-dependen-erms, and wih unigram plus bigram and wo-dependen-erms (denoed as all ), in erms of MAP and NDCG. Fig. 7 shows he resuls in erms of MAP. We can see ha bigram and wo-dependen-erms really help o improve he ranking accuracies in erms of MAP (we observe he same endency for he resuls in erms of NDCG). If hey are combined ogeher, he ranking accuracies can be furher improved. This indicaes ha he kernel-based model really has he abiliy of incorporaing differen ypes of dependency informaion. We also conduced experimens o invesigae how KL Kernel can improve he ranking accuracies, especially on long queries. We calculaed MAP (and NDCG) for KL- Unigram and KL Kernel on each query, and grouped he queries by lengh. Fig. 8 show he resuls. From he resuls, we can see ha KL Kernel improve MAP scores when queries become longer. This indicaes ha he kernel-based models are more effecive when queries become longer. The same endency can be found for mehods of BM25 Kernel, LMIR Kernel and for measures of NDCG. 5.2 Experimens on he OHSUMED and he AP Daases In his experimen, we used he OHSUMED [7] and TREC AP daases o es he performances of he ranking models. The OHSUMED daase consiss of 348,566 doc-

11 LMIR BM LMIR BM KL-Unigram MRF KL-Unigram MRF 0.45 LMIR Kernel BM25 Kernel KL Kernel 0.4 LMIR Kernel BM25 Kernel KL Kernel 0.4 MAP 0.3 MAP Fig. 9. Ranking accuracies on he OHSUMED. Fig. 10. Ranking accuracies on he AP daa. umens and 106 queries. There are in oal 16,140 query-documen pairs upon which relevance judgmens are made. The relevance judgmens are eiher d (definiely relevan), p (possibly relevan), or n (no relevan). The AP collecion conains 158,240 aricles of Associaed Press in 1988 and queries are seleced from he TREC opics (No.51 No.100). Each query has a number of documens associaed and hey are labeled as relevan or irrelevan. In oal, here are 8,933 query-documen pairs labeled. On average, each query has labeled documens. We used he Lemur oolki 3 as our experimen plaform. The daases were indexed and queries were processed wih Lemur. The experimenal resuls are shown in Fig. 9 and Fig. 10. Again, we can see ha he kernel-based models work beer han he bagof-words models and MRF. However, he improvemens are smaller han hose on he web search daa. This is probably because he query lengh of daa is shor (on average 4.96 erms per query on OHSUMED and 3.22 erms per query on AP). 6 Conclusion In his paper, we have sudied relevance ranking models, paricularly, hose using erm dependencies. Our work is new and unique in ha we employ he kernel echnique in saisical learning in he analysis and consrucion of ranking models. We have defined hree kernel based ranking models: BM25 Kernel, LMIR Kernel, and KL Kernel. The former wo are generalizaion of BM25 and LMIR, and he laer is a new model. The general ranking model employed in he our approach has (1) rich represenabiliy for relevance ranking paricularly for ha using erm dependencies (differen ypes of erm dependencies are summarized in he paper, and he kernel based ranking model can naurally represen hem.), (2) srong connecions wih exising models, (3) solid heoreical background, and (4) efficien calculaion. Experimenal resuls on web search daa and TREC daa show ha (1) he kernel based ranking models ouperform he convenional bag-of-words models and he MRF model using he same informaion, (2) he kernel funcions can really effecively incorporae differen ypes of erm dependency informaion (from unigram o bigram and wo-dependen-erms), (3) he kernel funcions work paricularly well for long queries. There are several issues which need furher invesigaions as fuure work. (1) The relaionship beween he proposed approach needs more sudies. (2) Kernel funcions 3 hp://

12 can be learned in machine learning. A fuure sep would be o learn he ranking model by plugging i ino some kernel machines. References 1. Bai, J., Chang, Y., Cui, H., Zheng, Z., Sun, G., Li, X.: Invesigaion of parial query proximiy in web search. In: Proc. of 17h WWW. pp (2008) 2. Bücher, S., Clarke, C.L.A., Lushman, B.: Term proximiy scoring for ad-hoc rerieval on very large ex collecions. In: Proc. of 29h SIGIR. pp (2006) 3. Deerweser, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by laen semanic analysis. J. of American Sociey for Informaion Science 41, (1990) 4. Gao, J., Nie, J.Y., Wu, G., Cao, G.: Dependence language model for informaion rerieval. In: Proceedings of he 27h annual inernaional ACM SIGIR conference. pp (2004) 5. Haussler, D.: Convoluion kernels on discree srucures. Tech. rep., Dep. of Compuer Science, Universiy of California a Sana Cruz (1999) 6. Hawking, D., Thislewaie, P.B.: Proximiy operaors - so near and ye so far. In: Proceedings of he 4h Tex Rerieval Conference (1995) 7. Hersh, W., Buckley, C., Leone, T.J., Hickam, D.: Ohsumed: an ineracive rerieval evaluaion and new large es collecion for research. In: Proc. of 17h SIGIR. pp (1994) 8. Keen, E.M.: Some aspecs of proximiy searching in ex rerieval sysems. J. Inf. Sci. 18(2), (1992) 9. Lodhi, H., Saunders, C., Shawe-Taylor, J., Crisianini, N., Wakins, C.: Tex classificaion using sring kernels. J. Mach. Learn. Res. 2, (2002) 10. Marins, A.: Sring kernels and similariy measures for informaion rerieval. Tech. rep., Priberam, Lisbon, Porugal (2006) 11. Mezler, D., Crof, W.B.: A markov random field model for erm dependencies. In: Proceedings of he 28h annual inernaional ACM SIGIR conference. pp (2005) 12. Moreno, P.J., Ho, P.P., Vasconcelos, N.: A kullback-leibler divergence based kernel for svm classificaion in mulimedia applicaions. In: NIPS 16. MIT Press (2003) 13. Na, S.H., Kim, J., Kang, I.S., Lee, J.H.: Exploiing proximiy feaure in bigram language model for informaion rerieval. In: Proc. of 31s SIGIR. pp (2008) 14. Ong, C.S., Smola, A.J., Williamson, R.C.: Learning he kernel wih hyperkernels. J. Mach. Learn. Res. 6, (2005) 15. Roberson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M., Gaford, M.: Okapi a rec-3. In: Proceedings of he 3rd Tex RErieval Conference (1994) 16. Sahami, M., Heilman, T.D.: A web-based kernel funcion for measuring he similariy of shor ex snippes. In: Proc. of 15h WWW. pp (2006) 17. Salon, G., Wong, A., Yang, C.S.: A vecor space model for auomaic indexing. Commun. ACM 18(11), (1975) 18. Sonnenburg, S., Räsch, G., Schäfer, C., Schölkopf, B.: Large scale muliple kernel learning. J. Mach. Learn. Res. 7, (2006) 19. Tao, T., Zhai, C.: An exploraion of proximiy measures in informaion rerieval. In: Proc. of 30h SIGIR. pp (2007) 20. Tsuda, K.: Suppor vecor classifier wih asymmeric kernel funcions. In: European Symposium on Arificial Neural Neworks. pp (1999) 21. Xie, Y., Raghavan, V.V.: Language-modeling kernel based approach for informaion rerieval. J. Am. Soc. Inf. Sci. Technol. 58(14), (2007) 22. Zhai, C., Laffery, J.: A sudy of smoohing mehods for language models applied o informaion rerieval. ACM Trans. Inf. Sys. 22(2), (2004)