NTCIR-6 CLIR-J-J Experiments at Yahoo! Japan

Transcription

1 NTCIR-6 CLIR-J-J Experiments at Yahoo! Japan Sumio FUJITA Yahoo Japan Corporation, Roppongi Hills Mori Tower, , Roppongi, Minato-ku, Tokyo , Japan sufujita AT yahoo-corp DOT jp Abstract This paper escribes NTCIR-6 experiments of the CLIR- J-J task, i.e. Japanese monolingual retrieval subtask, at the Yahoo group, focusing on the parameter optimization in information retrieval (IR). Unlike regression approaches, we optimize parameters completely inepenent from retrieval moels so that the optimize parameter set can illustrate the characteristics of the target test collections. We aopte the genetic algorithm as optimization tools an cross-valiate with 4 test collections, namely NTCIR- 3,4,5, an 6 CLIR-J-J. Keywors: Information retrieval, Test collections, Parameter calibration, Genetic algorithm. 1. Introuction The choice of scoring function is crucial for better search effectiveness. In our past NTCIR-4 CLIR-J-J an NTCIR-5 CLIR-J-J experiments, we ene by choosing BM25TF*IDF runs for official submissions. In fact, the choice was foun to be goo for previous NTCIRs. Our choices of NTCIR-X were base upon the pre-submission experiments using the test collection of NTCIR-(X-1). The scoring functions have some coefficient parameters to be etermine uring the pre-submission experiments. Failing to well calibrate these parameters results in poor effectiveness in the official evaluation even the scoring function is goo enough. Then the calibration of parameters becomes a major work in pre-submission experiments of the NTCIR tasks. Such coefficient parameters make the scoring function aaptable to iverse environments, where requiring the efforts of calibration. Given available four test collections in the NTCIR-6 CLIR task, we try to evaluate if an automatic calibration oes work for such limite number of training ata. In orer to compare with retrospective runs, we aopte the same system as for the NTCIR-5 experiments. Optimization of scoring function is stuie by several regression approaches to information retrieval [3][6]. We consier the calibration process as a simple optimization problem an we aopte the genetic algorithm (GA) to optimize the parameters to given test collections. The process is challenging because: 1) The optimization process is easily fallen into local maximum points an it faile to fin the global maximum. 2) The optimize parameters might be overfitte to the training collection an they o not perform well for other collections. Aopting the GA, the main concern seems to be the secon issue rather than the first one. In this paper, we present our NTCIR-6 CLIR-J-J task experiments, i.e. Japanese monolingual runs, focusing on the possibility of automatic calibration of search parameters. The rest of the paper is organize as follows: Section 2 escribes our experiment environment an retrieval system. Section 3 explains briefly the genetic algorithm an Section 4 reports our official runs an post submission experiments. Section 5 conclues the paper. 2. System Description Our evaluation environment: YLMS system is implemente base on Lemur toolkit 4.0 for inexing system [12], which is being evelope by the Lemur project. 2.1 Inexing language Chasen version Japanese morphological analyzer with IPADIC ictionary version are utilize for Japanese text segmentation an output single wors are use as inexing units. Stop wor lists for newspaper ocumentation are prepare. 2.2 BM25TF*IDF A Retrieval Status Value between a ocument an a query q is calculate as a ot prouct between the ocument term vector an the query term vector, where each term is weighte by TF*IDF [15]. Okapi BM25 TF [13][14] is use. RSV ( q, ) = w( q, w( t q w( = TF( IDF( w( q, = TF( q, IDF(

2 ( k1+ 1) freq( TF( = l k1((1 b) + b ) + freq( avl N IDF( = ( k4 + log ) f ( : ocument or query t : term N : total number of f ( : number of ocuments where t appears freq( l : number of : ocument length of ocuments in the collection occurrences of t in avl : average ocument length in the collection 2.3 Feeback strategies The strategy of feeback from top k ocuments in a pilot search is applie. The Rocchio feeback for TF*IDF is aopte as term extraction metho where the term precision measure to select salient terms is calculate as an element of the centroi vector of pseuo-relevant ocuments. Finally, an upate query vector Q was compute from the original query vector Q an a set of (pseuo) relevant ocument vectors R. 1 Q' = Q + poscoeff R R Instea of using a linear mean to average through feeback ocument vectors, we also use an alpha average, which is introuce by information geometry society to mixture probability istributions [1]. Alpha averaging is also use by an experimental IR system [8][9] to mixture ocument feature vectors. 1 Q' = Q + poscoeff R R 2 1 α 1 α 2 Giving -1 to alpha, this is equivalent to a linear mean. 2.4 Parameter calibration In our past experiences in NTCIR-4 an 5 [4][5], where we compare BM25TF*IDF with KL- Divergence approach. The BM25TF*IDF scoring metho is usually a very goo function to achieve better effectiveness against iverse test collections but it is subject to some coefficients to be calibrate base on training collections, namely k1, b, k4. As for k1 an b of the query sie TF, these are fixe to 1000 an 0 respectively, reucing the query TF function to a simple count of in-text term occurrences. Feeback effectiveness is also subject to the feeback parameters, namely feeback ocument number (fbdoccn, feeback term number (fbtermcn an feeback term coefficient (fbposcoeff). 3. Genetic Algorithm for IR GA is applie to information retrieval systems mainly on relevance feeback contexts as has been note by Lopez-Pujalte et al.[11]. Fan et al. propose to irectly learn ranking functions by applying genetic programming, i.e. an extension of GA [2]. 3.1 Genetic algorithm at a glance From the metaphor of the organic reprouction systems, the genetic algorithm is generally applie to optimization problems in iverse omains, consiering each trial point in the search space as an iniviual. Iniviuals to be examine are generate by applying genetic operations on each chromosome, representative of an iniviual, on which parameters to generate a particular iniviual are encoe. Given a population of iniviuals, genetic operations are applie iteratively in orer to prouce a new generation of the population. For each generation, each iniviual is evaluate by the given fitness function an the process terminates when a targete Parameter Range K B K fbdoccnt (Integer) fbtermcnt (Integer) fbposcoeff Table 1: Parameter range of GA search space fitness value is achieve or the preefine number of generations are processe. On top of the traitional genetic operations, namely selection, crossover an mutation, the istribute genetic algorithm aopts the migration operation: a population is ivie into several islans an GA is performe in each islan where the migration operation moves a certain number of iniviuals to another islan as escribe in Hiroyasu et al. [7]. We use their implementation. 3.2 GA process an operations Unlike many applications of GA to IR in literature, we o not encoe term vectors irectly to chromosomes. Instea we encoe the parameters of our retrieval moel, namely k1, b, k4, fbdocnum, fbtermnum, fbposcoeff. These integer or real numbers are encoe on 45 bits of Boolean strings in the ranges shown in Table 1. We carrie out GA optimization on a 8-noe cluster of Xeon 3.00GHz Dual CPU machines running a Free BSD operating system. The process is ivie into 8 islans an each islan contains 10 iniviuals. Initial

3 T T-01-N T-01-N4 T-01-N Table 2: Effectiveness of CLIR-J-J 1 st stage, the official title only run an corresponing 2 n stage runs D D-02-N D-02-N4 D-02-N Table 3: Effectiveness of CLIR-J-J 1 st stage, the first official escription only run an corresponing 2 n stage runs D D-03-N D-03-N4 D-03-N Table 4: Effectiveness of CLIR-J-J 1 st stage, the secon official escription only run (wor + char-bigram fusion) an corresponing 2 n stage runs T-01 D-02 D-03 bigram K1 b K4 #fb #fb Fb pos FB Inexing Docs Terms coeff Alpha Wors Wors Bigrams/ Wors Table 5: Parameters of CLIR-J-J official runs Fusion 0.5:0.5 populations are ranomly generate an the following operations are sequentially applie in each islan Migration The migration operation moves ranomly chosen iniviuals from an islan to another islan. This operation continues from the islan to the next islan an finally returns to the first islan so that the population in each islan remains the same.

4 Recall-precision curves Sensitivity of (feebackposcoefficien Precision No feeback T-03 KL-Dir No feeback KL-Dir 5 BM25TF*IDF Recall Feeback positive coefficient Figure 1 (Lef: Recall-precision curves of NTCIR-5 T-03, its no feeback baseline, KL-Dir run an its no feeback baseline Figure 2 (Righ: Sensitivity of to feeback positive coefficient (coefficient of positive term weigh in NTCIR-5 CLIR-J-J Sensitivity of to feeback positive coefficient 4 Sensitivity of to feeback Alpha alpha=-1.2 alpha= feeback positive coefficient Figure 3 (Lef: Sensitivity of to feeback positive coefficient (coefficient of positive term weigh in NTCIR-6 CLIR-J-J, T-01 Figure 4 (Righ: Sensitivity of to feeback alpha (coefficient of alpha averaging operation in positive ocument vector aggregation) in NTCIR-6 CLIR-J-J, T-01 Alpha Crossover We use two point crossover operator with the crossover ratio 1.0. It takes couples of iniviuals, chooses ranomly two positions an exchanges the part between positions of each couple Mutation The mutation operation consists of reversing ranomly chosen one bit on each chromosome Evaluation by fitness function Each iniviual is evaluate by the fitness function, i.e. the that the retrieval system using the ecoe parameters achieve against the training collection. As we are trying to maximize the against test collections, this fitness function is naturally aopte. However there may be other strategies, e.g. combining several measures in orer to avoi overfitting to the training collection Elitism Given the number of elite, the elite group of the previous generation an the same number of the best fitte iniviuals in the current generation are merge. The best fitte iniviuals as the same number as elites in this group are save in the current generation Selection As recommene in [7], we use tournament selection with the tournament size 4, i.e. select ranomly 4 iniviuals from the islan an take the best fitte iniviual to the next generation an repeat this until the next generation is complete. 4. CLIR Experiments The etails of the NTCIR-6 CLIR task are escribe in Kishia et al.[10].

5 Collections Test collections N3 N4 N5 N6 N N N N3,N4,N N Table 6: of GA parameter optimization of CLIR-J-J runs, the training collection an the test collection are the same in shaowe cells collectio n Parameters K1 b K4 fdoccnt fbtermcnt fbposcoeff N N N N3,N4,N N Avg Offic. run Table 7: Coefficient parameter sets optimize by 4 test collections of CLIR-J-J Population 80 Chromosome length 45 or 24 Number of Islans 8 Mutation rate 1/45 or 1/24 Population / Islan 10 Tournament size 4 Elite/Islan 5 Migration rate 0.5 Crossover rate 1.0 Migration interval 5 Table 8: Control parameters of GA process an GA operations Collections Test collections N3 N4 N5 N6 N N N N3,N4,N N Table 9: of GA parameter optimization of CLIR-J-J runs without feeback, the training collection an the test collection are the same in shaowe cells collectio n Parameters K1 b K4 fdoccnt fbtermcnt fbposcoeff N N/A N/A N N/A N/A N N/A N/A N3,N4,N N/A N/A N N/A N/A Avg N/A N/A Table 10: Coefficient parameter sets optimize by 4 test collections of CLIR-J-J without feeback parameters for the moels are calibrate by using 4.1 CLIR official runs for J-J SLIR NTCIR-5 CLIR-J-J test collections. The exactly We submitte a title only run, two escription only same parameter sets are applie for corresponing runs for the 1 st stage of Japanese monolingual 2 n stage runs. retrieval subtask. All the official runs are using the T-01 is our title only run an YLMS-J- TF*IDF metho with BM25 TF an a Rocchio J-D-02 is our escription run. D-03 is the feeback with a top k ocuments strategy. The fusion of D-02 an other escription only

6 run using a bigram inexing language. Table 2,3 an 4 show effectiveness of these three official runs in 1 st stage an their corresponing 2 n stage runs. Table 5 shows parameters use in the official runs. 4.2 Sensitivity of feeback effectiveness to test collection characteristics We emphasize that the top k ocument feeback strategy was exceptionally successful with the NTCIR-5 CLIR-J-J test collection as shown in Figure 1. Figure 2 illustrates the situation where the higher the fbposcoeff parameter, the better the results obtaine. Because our official submission T-01 is calibrate to NTCIR-5 CLIR-J-J, the feeback positive coefficient is ajuste to 0.8, a comparatively higher value. Figure 3 shows the sensitivity of to feeback positive coefficient (coefficient of positive term weigh in NTCIR-6 CLIR-J-J (T-01). The value arrives at its best when feeback poscoeff is 0.6. The situation is quite ifferent from NTCIR-5 CLIR-J-J. From Figure 4, we can see that giving any value other than -1.0 to feeback alpha causes the egraation. For the following experiments, we fixe the feeback alpha parameter to GA optimization of NTCIR-6 runs We trie to re-optimize the parameters of the title runs of NTCIR-6 CLIR-J-J by applying a istribute GA on a 8 noe cluster servers. NTCIR-3,4,5 an 6 CLIR-J-J test collections (N3,N4,N5,N6) are use for cross-valiation. Each of these test collections an combination of N3,N4 an N5 are use for training. Table 6 shows the of runs where coefficient parameters are optimize by the training collection. Table 7 shows optimize parameter set against each training collection. In our GA optimize run to N5, the feebackposcoeff parameter is ajuste to , whereas in GA optimize to N3,N4,N5. This is consistent with our observation that the impact of feeback is comparatively large in N5. Unlike regression approaches, where optimize coefficients are ifficult to be interprete or genetic programming approaches, which generate increibly complicate formulae, optimize parameters are interprete by referring to our experiences of past empirical experiments. See the column of N5, using N5 itself for training, the of as high as is achieve whereas using N4 for training, the of 616 is even much worse than our best official title run in NTCIR- 5, which achieve the of In effect, this NTCIR-5 official run was manually calibrate with N4 [5]. GA optimization processes seem to be fining approximately the maximum fitness against the training collection. The problem is rather overfitting to the training collection. Human calibrators o much work than simply looking for the best fitness setting in the search space but they carefully o avoi overfit problems. 4.4 GA optimization without feeback In orer to illustrate ifferent characteristics of test collections, we carrie out optimizations by GA without feeback as well. As shown in Table 9, overfitting to the training set seems to be alleviate. Overfitting is mainly cause by the feeback parameters like the number of terms/ocuments to use for feeback or feeback coefficient, which are presumably collection epenent. 4.5 Optimization costs As can be seen from Table 6, GA optimize N6 runs are very close to our official title only run, which is set manually. The GA optimization using N5 took 36 hours for 20 generation reiteration operate on a 8 noe cluster of Xeon 3.00GHz Dual CPU machines. These harware environments are not at all cheap even toay. But it presumably pays for efforts to optimize these parameters manually from scratch. We stoppe the iteration at the 20 th generation because no rastic improvement is observe even at the 100th generation. 5. Conclusions We reporte our NTCIR-6 evaluation experiments of the CLIR-J-J task. We aopte a TF*IDF approach an applie GA optimization of coefficient parameters of the retrieval moel. GA optimize runs achieve almost the same effectiveness as human calibrate runs, although GA runs overfitte to the training collections. Automatic calibration also illustrates ifferent characteristics of the test collections especially such as feeback effectiveness. These optimize parameters can be utilize to help experts in system calibration. Acknowlegments We thank NTCIR management/secretariat members an CLIR task organizers for proviing us of the ata. References [1] Amari, S. Alpha-integration of Stochastic Eviences. In Proceeings of 2 n International Symposium on Information Geometry an its Applications, Tokyo, [2] Fan, W., Goron, M.D. an Pathak, P., A generic ranking function iscovery framework by genetic programming for information retrieval, Information Processing an Management, vol. 40, issue 4, , July [3] Fuhr, N. an Pfeifer, U. Probabilistic information retrieval as combination of abstraction, inuctive learning an probabilistic assumptions. ACM Transactions on Information Systems, 12(1), , 1994.

7 [4] Fujita, S. Revisiting the Document Length Hypotheses --NTCIR-4 CLIR an Patent Experiments at Patolis. In Working notes of the fourth NTCIR workshop meeting, , [5] Fujita, S. A Decae after TREC-4 NTCIR-5 CLIR- J-J Experiments at Yahoo!Japan. In Proceeings of NTCIR Workshop 5 Meeting, , [6] Gey, F. C. Inferring probability of relevance using the metho of logistic regression. In The proceeings of seventeenth annual international ACM SIGIR conference on research an evelopment in information retrieval, , [7] Hiroyasu, T., Yoshia, J., Sano, M., Fukunaga, T. an Kataura, T. Distribute Genetic Algorithms ga2k (ver. 1.3) Specification, Intelligent System Design Laboratory, Doshisha University, Sep [8] Kanazawa, T., Takasu, A., an Aachi, J. The effects of the Relevance-base Superimposition Moel for Effective Information Retrieval. IEICE Transactions, Vol. E83-D, No. 12, pp , Dec [9] Kano, N., Kanazawa, T., an Miyazawa A. Retrieval of Web Resource Using a Fusion of Ontology-base an Content-base Retrieval with the RS Vector Space Moel on a Portal for Japanese Universities an Acaemic Institutes. In Proceeings of the 39th Hawaii International Conference on System Sciences (HICSS 39), [10] Kishia, K., Chen, K.-H., Lee, S., Kuriyama, K., Kano, N., Chen, H. -H. an Myaeng., S. -H. Overview of CLIR task at the sixth NTCIR workshop. In Proceeings of the Sixth NTCIR Workshop, [11] Lopez-Pujalte, C., Guerrero-Bote, V.P. an Moya- Anegon, F. e, Genetic algorithm in relevance feeback: a secon test an new contributions, Information Processing an Management, vol 39, issue 5, , September [12] Ogilvie,O. an Callan,J. Experiments Using the Lemur Toolkit, In NIST Special Publication : The Tenth Text REtrieval Conference (TREC 2001), , [13] Robertson, S. an Walker S. Some Simple Effective Approximations to the 2-Poisson Moel for Probabilistic Weighte Retrieval. In Proceeings of the 1994 ACM SIGIR Conference on Research an Development in Information Retrieval, , [14] Robertson, S. E., Walker, S., Jones, S., M.Hancock- Beaulieu, M., an Gatfor M. Okapi at TREC-3. In NIST Special Publication :Overview of the Thir Text REtrieval Conference (TREC-3), , [15] Salton, G. Automatic Text Processing The Transformation, Analysis, an Retrieval of Information by Computer, Aison-Wesley publishing company, Massachusetts,1988.