Analyzing Query Reformulation Data using Multi-level Modeling

Transcription

1 Analyzng Query Reformulaton Data usng Mult-level Modelng Kun Lu 1, Soohyung Joo 2, Taehun Lee 1, Rong Hu 3 1 Unversty of Oklahoma 2 Unversty of Kentucky 3 Jang X Unversty of Fnance and Economcs Abstract Ths study explores the adopton of mult-level modelng to analyze query reformulaton data. Thus far, the dependency among query reformulatons wthn the same search sesson has not been adequately treated n the expermental desgn. Ths has lmted the analyss of users query behavor. Ths study ntroduces mult-level modelng to query reformulaton data analyss. Mult-level modelng s capable of handlng the correlatons among query reformulatons and provdes an avenue to analyzng the nested data structure. A demonstraton of fttng query reformulaton data to two types of mult-level models s provded. The method ntroduced n ths study provdes a potental soluton to the analyss of query reformulatons. Keywords: Query reformulaton; mult-level modelng; data analyss; nteractve nformaton retreval do: /16473 Copyrght: Copyrght s held by the authors. Contact: kunlu@ou.edu 1 Introducton Understandng users query behavor has been one of the mportant topcs n the feld of nformaton retreval. Early studes examned the characterstcs of user queres based on transacton logs (Jansen et al., 2000; Spnk et al., 2001). Also, nvestgators looked at how users reformulate ther queres (Reh & Xe, 2006; Jansen et al., 2007; Jansen et al., 2009), and more recent studes have explored the effects of contextual factors (e.g., search task types, cogntve status, doman knowledge, and search sklls) on users query reformulaton behavor (, 2010; Hu et al., 2013). When analyzng query reformulatons, one hurdle that mpedes the adopton of tradtonal statstcal analyss methods has been that query reformulatons are nested wthn a search sesson, and thus they are prone to be correlated wth each other. Tradtonal statstcal analyss methods, such as regresson or analyss of varance (ANOVA), requre the observatons to be ndependent wth each other. Ths has lmted the use of nferental statstcs n query reformulaton studes. For example, Joo and Lee (2011) analyzed only the frst occurrng reformulaton types to avod the dependency problem among query reformulatons when usng ANOVA. (2010) vewed each ndvdual reformulaton as a unt of analyss, and dsregarded the dependency among reformulatons n the analyss. Smlarly, Hu et al. (2013) analyzed query reformulaton wthout consderng the sesson effect by aggregatng query reformulatons across sessons. As s shown n these examples, researchers have had dffculty n controllng for the nterdependency among observatons n the analyss of query reformulatons. Mult-level modelng can be a compellng soluton to analyzng ths type of nested data structure (Gelman & Hll, 2007). In the context of mult-level modelng, query reformulatons can be consdered as level one varable (ndvdual level) that s nested wthn search sessons (level two varable, or group level). It s capable of handlng the correlatons among query reformulatons wthn each search sesson. The purpose of ths study s to ntroduce mult-level modelng to query reformulaton data analyss. The study wll demonstrate how dfferent types of mult-level models can be appled to analyze query reformulaton data. 2 Mult-level Modelng The hstory of mult-level modelng goes back to the work of Robnson (1950), whch dscussed the fallacy of ecologcal correlaton, the papers by Blau (1960) and Davs et al (1961) on the contextual effects, and the paper by Esenhart (1947) on the dstncton between fxed and random effects. After ther semnal papers, there has been a bloom of methodologcal artcles amng at showng that serous nferental errors may result from applyng fxed parameter regresson models to the analyss of herarchcally structured data (Lard & Ware, 1982; Blalock, 1984; Atkn & Longford, 1986; Bryk & Raudenbugh, 1987; Kenny & Judd, 1986; Hoffman, 1997). Behavoral and socal scences data commonly have herarchcally structured systems. For example, workers are nested wthn frms n organzatonal/management research, and students are nested wthn schools n educatonal research. In the analyss of such data,

2 Conference 2016 mult-level models provde a set of ntegrated methods for modelng dependence due to the herarchcally nested structures. 3 Data Analyss 3.1 Expermental Data The data was collected from a user study n whch forty fve subects were recruted from a state unversty n the Unted States to test two dfferent search nterfaces. One search nterface (SmpleMed) s smlar to Google wth a smple search box and users can clck a search result to pop up the full document. The other search nterface (MeshMed) provdes addtonal components, the MeSH tree browser and the MeSH term browser, to allow users to nteract wth MeSH thesaurus whle searchng (Fgure 1). The tree browser dsplays the herarchcal structure of MeSH that allows a user to navgate the thesaurus. The term browser allows a user to search related MeSH terms and ther defntons wth natural language queres. The data corpus used n the experment was Ohsumed test collecton ( Sx search topcs were randomly selected from the 106 topcs n Ohsumed. Each partcpant searched sx topcs on two search systems: three on SmpleMed and three on MeshMed. The sequences of the search topcs and search systems were randomzed to balance the learnng effects. B A C Fgure 1. MeshMed search nterface (A s the search browser, B s the tree browser, and C s the term browser). 3.2 Codng The study collected a total number of 270 search sessons, wth 135 from SmpleMed and 135 from MeshMed. The total number of sessons wth at least one reformulaton s 199, wth 112 from SmpleMed and 87 from MeshMed. The query reformulaton data was then ndependently coded by two of the authors usng the codng scheme n Table 1. The nter-coder relablty turned out to be accordng to Cohen s kappa, whch suggests a hgh level of relablty. Facets Sub-facets Example Content changed Specfcaton(SPE) Generalzaton(GEN) lymphoma lymphoma defnton Dabetc gastroparess gastroparess 2

3 Conference 2016 Content unchanged Format Parallel movement(par) Synonym(SYN) Format(FOR) lymphoma defnton lymphoma small bowel Menopausal menopause defnton:menopausal defnton menopausal Error Error(ERR) COPT COPD Table 1. Codng scheme Reformulaton type Frequency Percentage ERR % FOR % GEN % PAR % SPE % SYN % Table 2. Frequency of reformulaton types 3.3 Mult-level Models for Query Reformulaton In ths study, a smple mult-level modelng wth one level 2 varable (search sesson) and two level 1 varables (query reformulaton and search performance) s employed to demonstrate the use of mult-level modelng for query reformulaton data. As s shown n Fgure 2, the model hypotheszes that query reformulatons affect search performance wth search sesson as the level 2 varable (query reformulatons are nested n search sessons). As for the search performance, p@10 (precson value at the top 10th retreved documents) s used. The model used the search sesson as the level 2 ndcator. However, due to the lmted space, the model does not nclude any level 2 predctors n ths poster. It should be noted that level 2 predctors, such as search system types or search topc famlarty, can be easly ncorporated n mult-level modelng to mprove the nference. Fgure 2. Dagram of a mult-level model for query reformulaton data. There are three approaches that mult-level modelng models the random effects: random ntercepts (dfferences n the overall level of level 2 unts), random slopes (dfferences n the effects of predctors across level 2 unts), or both random ntercepts and slopes. In ths study, we demonstrate the random ntercepts and both random ntercepts and slopes. In random ntercepts model, only the coeffcent that vares across groups s the ntercept. The model can be wrtten as: Level 2: 3

4 Conference 2016 Level 1: y [ ] x where y s the p@10 (precson at top 10th retreved documents) of the th observaton, x s th query reformulaton, β s the regresson coeffcent, and α s the group level ntercept that can vary across dfferent groups ( refers to the th sesson). The random ntercepts and slopes model can have varyng ntercepts and slopes, whch can be wrtten as: Level 2: Level 1: y [ ] [ ] x where α s the varyng ntercept, β s the varyng slope, other varables are the same as those n random ntercepts model. 4 Intal Results Table 3 lsts the average search performance after dfferent types of query reformulatons as s measured by p@10. As Table 3 shows, SPE showed the hghest p@10 among all, followed by ERR, SYN, PAR, FOR, and GEN. Then, the two types of mult-level models were appled. The SYN reformulatons were excluded from the mult-level modelng snce there were only three observatons. SPE was selected as the reference level as t had the hghest p@10. Wth the random ntercepts model, the p@10 after FOR was sgnfcantly lower than that after SPE (t=-3.879; d.f.=600; p<0.05), the p@10 after GEN was sgnfcantly lower than that after SPE (t= ; d.f.=600; p<0.05), the p@10 after PAR was sgnfcantly lower than that after SPE (t=-3.829; d.f.=600; p<0.05). There was no sgnfcant dfference between the p@10 after ERR and p@10 after SPE (t=-0.399; d.f.=600; p>0.05). The data was then ftted wth the random ntercepts and slopes model where both the ntercept and slope can vary across search sessons. Accordng to the statstcal test results from random ntercepts and slopes model, the p@10 after FOR was sgnfcantly lower than that after SPE (t= ; d.f.=586; p<0.05), the p@10 after GEN was sgnfcantly lower than that after SPE (t=-5.619; d.f.=586; p<0.05), the p@10 after PAR was sgnfcantly lower than that after SPE (t=-2.418; d.f.=586; p<0.05), whle the p@10 after ERR was not sgnfcantly dfferent from that after SPE (t=-0.100; d.f.=586; p>0.05). Reformulaton type P@10 ERR FOR GEN PAR SPE SYN Table 3. Search performance after dfferent reformulaton types 4

5 Conference Concluson Ths study ntroduces mult-level modelng to query reformulaton data analyss. Query reformulatons are generally nested wthn search sessons, whch lmts the adopton of tradtonal statstcal analyss such as regresson or ANOVA. Mult-level modelng offers a compellng method to analyze such nested data n query reformulaton studes. Wth mult-level modelng, predctors at dfferent levels (sesson level, move level) can be ntegrated nto a sngle research model. It also avods the volaton of ndependent assumpton. Future study wll further explore more complex mult-level models for query reformulaton. 6 References Atkn, M., & Longford, N. (1986). Statstcal modellng ssues n school effectveness studes. Journal of the Royal Statstcal Socety, 149(1), Blalock, H. M. (1984). Contextual-effects models: Theoretcal and methodologcal ssues. Annual Revew of Socology, 10, Blau, P. M. (1960). Structural effects. Amercan socologcal revew, 25(2), Bryk, A. S., & Raudenbush, S. W. (1987). Applcaton of herarchcal lnear models to assessng change. Psychologcal Bulletn, 101(1), 147. Davs, J. A., Spaeth, J. L., & Huson, C. (1961). A technque for analyzng the effects of group composton. Amercan Socologcal Revew, 26(2), Esenhart, C. (1947). The assumptons underlyng the analyss of varance. Bometrcs, 3(1), Gelman, A., & Hll, J. (2007). Data analyss usng regresson and multlevel/herarchcal models. New York: Cambrdge Unversty Press. Hofmann, D. A. (1997). An overvew of the logc and ratonale of herarchcal lnear models. Journal of management, 23(6), Hu, R., Lu, K., & Joo, S. (2013). Effects of topc famlarty and search sklls on query reformulaton behavor. In Proceedngs of the Assocaton for Informaton Scence and Technology 2013 (ASIST 2013). Montreal, Canada. Jansen, B. J., Spnk, A., & Saracevc, T. (2000). Real lfe, real users, and real needs: A study and analyss of user queres on the Web. Informaton Processng and Management, 36(2), Jansen, B. J., Spnk, A., & Narayan, B. (2007). Query modfcatons patterns durng Web searchng. In Proceedngs of the Internatonal Conference on Informaton Technology 2007 (ITNG 07), pp Jansen, B. J., Booth, D. L., & Spnk, A. (2009). Patterns of query reformulaton durng Web searchng. Journal of the Amercan Socety for Informaton Scence and Technology, 60(7), Joo, S., & Lee, J. (2011). Assessng effectveness of query reformulatons: Analyss of user-generated nformaton retreval dares. In Proceedngs of the Assocaton for Informaton Scence and Technology 2011 (ASIS&T 2011), New Orleans, LA, USA. Kenny, D. A., & Judd, C. M. (1986). Consequences of volatng the ndependence assumpton n analyss of varance. Psychologcal Bulletn, 99(3), 422. Lard, N. M., & Ware, J. H. (1982). Random-effects models for longtudnal data. Bometrcs, 38(4), Lu, C., Gwzdka, J., & Belkn, N. J. (2010). Analyss of query reformulaton types on dfferent search tasks. In Proceedngs of the Conference 2010, Urbana Champagn, IL. Metzler, D., & Croft, W. B. (2004). Combnng the language model and nference network approaches to retreval. Informaton Processng and Management, 40(5), Mu, X., Lu, K., & Ryu, H. (2014). Explctly ntegratng MeSH thesaurus help nto health nformaton retreval systems: An emprcal user study. Informaton Processng and Management, 50(1), Reh, S. Y., & Xe, H. (2006). Analyss of multple query reformulatons on the Web: The nteractve nformaton retreval context. Informaton Processng and Management, 42(3), Robnson, W. S. (1950). Ecologcal Correlatons and the Behavor of Indvduals. Amercan Socology Revew, 15(3), Spnk, A., Wolfram, D., Jansen, B.J., & Saracevc, T. (2001). Searchng the Web: The publc and ther queres. Journal of the Amercan Socety for Informaton Scence and Technology, 52(3),