TDT-2004: ADAPTIVE TOPIC TRACKING AT MARYLAND

Transcription

1 TDT-2004: ADAPTIVE TOPIC TRACKING AT MARYLAND Tamer Elsayed, Douglas W. Oard, Davd Doerma Isttute for Advaced r Studes Uversty of Marylad, College Park, MD Cotact author: telsayed@cs.umd.edu Gary Kuh Natoal Securty Agecy 9800 Savage Road, Sute 6514 Fort Meade, MD ABSTRACT A topc trackg system that combes elemets from vector space ad laguage modelg frameworks to compute documet scores s descrbed. The model s used for both the tradtoal TDT topc trackg evaluato desg ad the ew supervsed adaptve topc trackg evaluato. Results dcate that supervsed adaptato ad score ormalzato should be more closely coupled, ad that curret techques for detecto error trade aalyss may be of lmted utlty whe supervsed adaptato s performed. 1. INTRODUCTION The adaptve topc trackg task at TDT-2004 ers a useful evaluato framework for a project that we have recetly tated whch ole learg of user eeds wll be a mportat compoet. Our goals for partcpato TDT-2004 therefore focused o frastructure buldg; specfcy: (1 to tegrate our basele rakg fucto to a adaptve topc trackg system, ad (2 to explore the desg of evaluato measures that are sutable to our ultmate tasks. Tme costrats precluded parameter tug usg the TDT-4 collecto, so the results reported below should be cosdered prelmary. We submtted oe o-adaptve topc trackg ru for the requred codto (oe o-topc trag. At the tme we submtted our o-adaptve ru, score ormalzato was ot yet corporated to our system. As we show below, ths adversely affected both our Detecto Error Trade (DET curves ad the resultg mmum detecto error cost. Our actual detecto error cost s below the computed mmum cost pot o the DET curve because a mplemetato lmtato at the tme of submsso resulted a hard decso of NO for a few topcs for whch the trag epoch was very ear the begg of the TDT-5 collecto. Ths s the frst year whch adaptve topc trackg has bee cluded the Topc Detecto ad Trackg (TDT evaluatos. We submtted two rus, oe wth sgle-pass score ormalzato, ad oe whch score ormalzato was dsabled (for cotrast wth our o-adaptve ru. Both used oly a sgle o-topc trag ; topcs wth early trag epochs were hadled approprately these rus. Adaptato reduced the actual detecto error cost for the uormalzed codto from 0.65 to Sgle-pass score ormalzato (wth z-scores computed from the tal o-topc trag creased the actual detecto error cost from 0.24 to 0.38, suggestg a eed for reormalzato each tme the topc represetato s adjusted. I pror topc trackg evaluatos, we foud the DET curves to be useful because our terests focused more o score computato tha threshold selecto [1,2]. Adaptve topc trackg troduces a atural depedece o threshold selecto, ad that depedece does ot seem to be reflected well by the preset way whch DET curves are costructed. We dscuss ths ssue greater detal below. The remader of ths paper s orgazed as follows. I the ext secto, we descrbe the score computato fucto that was used for three submtted rus ad the mplemetato detals for each ru. Secto 3 the presets our cal results ad some tal post-hoc aalyss, cludg dscussos of score ormalzato ad DET curve terpretato for adaptve topc trackg. Fy, we coclude wth a few remarks about future work. 2. MODELING TOPIC TRACKING I ths secto we dscuss the desg of our trackg systems. We start wth the laguage model Secto 2.1 followed by a overvew of the system compoets Secto The Laguage Model The "-gram logodds" laguage model s a term frequecy model, cosstg of a lexcographcy ordered lst of character strgs ad log lkelhood ratos. The strgs the lst are exactly characters log, hece the ame -gram, where the legth s a model parameter. These strgs occurred the o-topc ad (assumed -topc trag stores for the topc beg tracked, T, ad they may overlap. Ay -gram begg wth a character code value less tha or equal to that of a ASCII space s ot cluded the model. Other model parameters are the mmum umber of occurreces of each N-gram, separately for the o-topc trag set

2 (MCout o, the (assumed -topc trag set (MCout, ad over (MCout, -grams that do ot meet these mmum couts are excluded from the model. log odds(, T log o orm orm (9 The model s traed o two sets of stores. Oe set T o cotas o-topc stores, ad the other set T cotas (assumed -topc stores. I each set, the umber of occurreces (term frequecy of each -gram s couted. Those -grams that meet the preset mmum couts are cluded the model. The relatve frequecy of each -gram s computed for the o-topc, the -topc, ad the over sets: o tf (, T j T o j T o tf ( j, T tf (, T j T tf ( j, T o tf (, T tf ( j, T where tf(,x s the term frequecy of the set X. A weght, λ, s computed for each N-gram the model, based o ts over cout: (1 (2 (3 2 λ 1 (4. (, 1+ k tf T e The expermeter chooses k ths equato. As k s creased from 0, λ reaches 0.5 at er ad er values of the over cout. λ tells us how much weght to gve to the potety very (ad therefore less relable class-depedet relatve frequecy for -gram, ( o, ad ad the potety very large (ad therefore more relable over relatve frequecy for -gram,, the followg covex combatos T o ad T, whch defe the oothed "lkelhood" of -gram both of the trag sets. Note that these values eed to be ormalzed after beg oothed. o λ (1 ( To + λ p T (5 p λ + (1 λ (6 ( o orm orm j T o j T j j o Fy, the "logodds" or log lkelhood rato for each -gram s computed from the class-depedet lkelhoods: o (7 (8 The score of S for topc T was the computed as follows: tf _ score(, S L 1.5 L s S S avg tf (, S + tf (, S (10 1 score ( S, T log odds(, T * tf _ score(, S (11 where s a -gram occurred S, S s the umber of dfferet -grams S, L S s the total umber of -grams S, ad L avg s the average legth of stores see so far ( both trag ad testg sets. I a earler verso of ths model, we used the raw tf value; for our TDT-2004 expermets we replaced that wth the Okap BM25 term frequecy compoet equato (10 [3] System Desg Buldg topc model Fgure 1: System overvew. As llustrated Fgure 1, the core system cossts of three ma modules: buldg the topc model usg the techque descrbed Secto 2.1, computg the factors that wll be used the cotext of cross-topc ad cross-source score ormalzato, ad fy trackg the o-topc stores the comg data stream. Buldg the Topc Model Sample -topc stores o-topc term -topc term Computg ormalzato factors relatve Fgure 2: Buldg the topc model. Topc trackg logodds A two-sded trag set s eeded to buld our model for each topc. The requred codto provdes oe o-topc trag ad o cofrmed -topc stores. We assumed that stores that substaty predate the o-topc trag wll be -topc. To mmze the possblty of falsely selectg a was actuy o-topc, we formed a trag epoch from the begg of the TDT-5 corpus to the kow o-topc ad the

3 radomly sampled N stores (f avalable from the frst 80% of that trag epoch 1. A cose larty measure was the computed betwee each of the N stores ad the oe o-topc trag. The top half of the most lar stores costtuted the -topc trag set for that topc. Ths procedure was desged to detfy -topc stores that were farly lar to the oe o-topc the way whch they used terms. The logodds value for each term s the computed as descrbed above. The whole process s depcted Fgure 2. No-adaptve Topc Trackg Testg set ew term score Pre-computed logodds Fgure 3: The o-adaptve trackg approach. The approach we adopted for the o-adaptve trackg task ply treats each depedetly ad does ot take the tmg attrbutes to accout. I other words, oce the topc model has bee traed as descrbed above, each the comg stream s scored wth respect to the tal model (represeted by the pre-computed logodds as show Fgure 3. Ay gve would therefore receve the same score regardless of ts order the stream. A hard (yes-o decso for each was made usg a statc threshold Th Track. Because the o-adaptve system dd ot perform ay usupervsed updates to the topc model, the computed score for each does ot deped o the specfc value of Th Track. Supervsed Adaptato for Topc Trackg Make hard decso Th Track The ew (supervsed adaptve trackg task s lar to the o-adaptve task, except that the true state (o-topc or -topc of a S wth respect to the curret topc becomes kow to the system mmedately f the system makes a hard decso that the s o-topc. Ths s teded to ulate a teractve applcato whch the user provdes feedback for stores that the system elects to dsplay. We adopted a straghtforward adaptato approach (llustrated Fgure 4 whch the ew formato was leveraged to ehace the curret topc model. A judged to be o-topc was added to the o-topc trag set by mergg ts -gram couts wth the curret couts the topc model ad the re-computg the 1 I some cases, ths resulted fewer tha N stores the trag epoch. For our adaptve rus, we added the etre TDT-4 corpus to the trag epoch order to avod that problem. relatve frequeces ad the logodds values for each term. A judged to be -topc was hadled larly; sce our tal model was bult usg assumed -topc stores (as descrbed above, we removed oe of the assumed -topc stores each tme a ewly judged -topc became avalable ad aga retraed the logodds. I ether case, the modfed model would be used utl the ext tme the system elected to declare a o-topc (at whch tme a ew judgmet would become avalable. We aga used a statc threshold, Th Track, to make these hard decsos for each. Testg set ew Fgure 4: The adaptve trackg approach. Computg Normalzato Factors I our secod adaptve trackg ru, we augmeted our supervsed topc trackg model by corporatg score ormalzato. I order to get comparable scores across topcs, laguages ad ews sources, we adopted a varat of the z-score ormalzato method troduced [4]. The method assumes that the true scores of -topc stores would follow a commo Gaussa dstrbuto regardless of codto; therefore, a score ca be ormalzed gve estmates for the mea ad stadard devato of -topc stores. We assumed for ths purpose that most TDT-4 stores would be topc for ay TDT-5 topcs. For each ews source that exsted both TDT-4 ad TDT-5, we sampled N orm stores uformly dstrbuted across data fles for that source ad computed scores for each usg each topc model separately; that resulted tal estmates of µ ad σ for each topc. I order to guard agast the possblty that a few o-topc stores mght exst the TDT-4 corpus, we the removed ay sampled wth a score greater tha µ by more tha Th Norm, * σ ad the recomputed term fal estmates for µ ad σ score Normalzato factors Logodds Relevace judgmets Th Track Make hard decso f yes Add to topc model. Those estmates (we c them ormalzato factors are the used to ormalze the raw scores as descrbed below. For those cases whch TDT-5 cotaed a source that was ot preset TDT-4, we averaged the ormalzato factors for TDT-4 sources the same source laguage ad used those values as a estmate for what we would have obtaed had trag data for the ew source bee avalable. Fy, the ormalzed score for each was computed as follows:

4 score orm ( S, T score ( S, T µ σ ( src ( S, T ( src ( S, T (12 where src(s s the source of the S, ad score(s,t s computed as the o-adaptve approach. Note that the ormalzato factors were computed oly oce before the trackg phase ad used throughout the testg process. Implemetato ad Parameters Settgs Cotext Parameter Value Parameter Value N 6 K 0.37 Model MCout o 0 MCout 0 MCout 1 Normalzato Th orm 5 N orm 2000 Trag N 100 Table 1. System parameters Our systems were mplemeted Java2. We dd t make use of ay prevously developed formato retreval system compoets, so the frastructure has bee bult from scratch. Table 1 summarzes the specfc values of our systems parameters. All these values were statcy used submtted rus. 3. RESULTS Ths year, we submtted 3 systems, oe for the o-adaptve trackg task, ad two for the adaptve supervsed task. Here we dscuss the performace of these systems. Fgure 5. DET curve for the o-adaptve system. abormal result. Our system bult a statc model traed wth oly oe o-topc, so we beleve that that uque mght ot be suffcet for our approach to buld a relable topc model. Had more tme bee avalable, we expect that a parameter tug usg the TDT-4 collecto, would have yelded a stroger basele. Score ormalzato s also expected to have a major effect. 3.2 Supervsed Adaptato Systems Actual M DET Task System Th Track Cost Cost No-adaptve UMD UMD Adaptve UMD Table2. Actual ad mmum DET costs for the submtted rus 3.1 No-adaptve System At the tme of the o-adaptve task submsso, the ormalzato phase was ot read so the cost show Table 2 ad the DET curve llustrated Fgure 5 are acheved the absece of score ormalzato. The real cost s less tha the computed cost because of the hard No decso take for 4 topcs whose uque o-topc was located at the very begg of the TDT-5 corpus. Ths problem s solved the adaptve rus. Surprsgly, our actual cost was less tha the mmum DET cost. The reaso s that our system provded o score for the stores whose legth s oe word of less tha 6 characters. Ufortuately, the evaluato set cludes more tha a hudred stores (o average per topc. Despte beg judged -topc, those stores caused that Fgure 6. DET curve for the o-ormalzed adaptve system. Fgure 6 llustrates the performace of o-ormalzed supervsed adaptve system. The actual cost dropped to 0.24 as show Table 2, about a 63% mprovemet over the o-adaptve cost. We attrbute ths to the ablty of our system to lear evely from both o-topc ad -topc stores. Ths result demostrated the stregth of our adaptato techque regardless of score

5 ormalzato. reteto behavor; we therefore pla to explore possbltes for erchg the collecto ways that may also be of terest to other TDT partcpats. Before we commt to dog so, however, we wll eed to better uderstad the effect of complete judgmets o the utlty of the TDT-5 collecto as a bass for system comparso. The large umber of topcs the TDT-5 collecto makes t somewhat better suted to evaluato desgs that requre extedg the trag epoch (because some topcs wth few relevat documets would be lost, but f there are cocers about the stablty of effectveess measures we mght ultmately choose to use the more extesvely aotated TDT-4 collecto stead. We are, therefore, also partcularly terested characterzg the effect of complete judgmets o the stablty of both the TDT effectveess measures ad other measures that mght er a lar degree of sght (e.g., bary preferece [5]. We look forward to meetg Gathersburg to dscuss these ssues! ACKNOWLEDGMENTS Fgure 7. DET curve for the ormalzed adaptve system. The DET curve show Fgure 7 llustrates the performace of our adaptve system that ormalzes the scores. We dd oly oe ormalzato pass, before startg the evaluato epoch. Uexpectedly, the performace cost creased about 37% wth respect to the o-ormalzed ru, but was stll mproved over the o-adaptve system (about a 50% cost drop. Three reasos could be behd that degradato. Frst, the ormalzato sample sze mght ot be large eough to obta good estmates of the -topc dstrbuto. Secod, scores resultg from our approach for -topc stores mght ot follow a Gaussa dstrbuto, falsfyg a basc assumpto the Z-score ormalzato techque. Fy, as the model chages, the pre-computed ormalzato factors become less ad less represetatve because they were computed gve a dfferet model represetato. We expect better performace to be acheved f we recompute these factors perodcy. The scores reported here are threshold depedet, ply because the topc model chages dyamcy 2. If we use aother threshold, we expect to have dfferet scores, hece probably dfferet DET curve ad mmum cost. Ths makes the usefuless of DET curve the evaluato of adaptve systems doubtful. 4. CONCLUSION AND FUTURE WORK Our partcpato TDT-2004 has yelded both a deeper uderstadg of evaluato ssues for adaptve topc trackg ad a substatal part of the evaluato frastructure that we pla to leverage as we explore the desg of adaptve topc trackg systems. We are partcularly terested the effect of alteratve tal codtos (e.g., explct queres ad/or substatal umbers of marked o-topc ad -topc documets ad modelg mperfect evdece of user preferece (e.g., readg ad/or The authors are grateful to Jo Fscus for hs exceptoal efforts ad fte patece. Ths work has bee supported part by DoD cooperatve agreemet N REFERENCES 1. Levow, G.-A. ad Oard, D.W. Sgal Boostg for Traslgual Topc Trackg, Alla, J., ed.., Topc Detecto ad Trackg: Evet-Based Iformato Orgazato, Kluwer Academc Publshers, Bosto, pp , He, D., Park, H.R., Murray, G.C., Subot, M., ad Oard, D.W., TDT-2002 Topc Trackg at Marylad: Frst Expermets wth the Lemur Toolkt, TDT-2002 Workg Notes, Aval from Robertso, S.E., Walker, S., Joes, S., Hacock-Beauleu, M.M. ad Gatford, M., Okap at TREC-3, Proceedgs of TREC-3, Gathersburg, MD, pp , Leek, T., Schwartz, R., ad Ssta, S., Probablstc Approaches to Topc Detecto ad Trackg, I James Alle, edtor, Topc Detecto ad Trackg: Evet-based Iformato Orgazato, chapter 4, pp Kluwer Academc Publshers, Buckley, C. ad Voorhees, E.M., Retreval Evaluato wth Icomplete Iformato, Proceedgs of SIGIR-2004, Sheffeld, UK, pp , Assumg the topc model has bee chaged betwee.