New Measures for the Evaluation of Interactive Information Retrieval Systems: Normalized Task Completion Time and Normalized User Effectiveness

Transcription

1 New Measures for the Evaluation of Interactive Inforation Retrieval Systes: Noralized Task Copletion Tie and Noralized User Effectiveness Jing Cheng Graduate School of Library and Inforation Science, University of Illinois at Urbana-Chapaign 50 E. Daniel Street Chapaign, IL 6820, U.S.A. Xiao Hu Graduate School of Library and Inforation Science, University of Illinois at Urbana-Chapaign 50 E. Daniel Street Chapaign, IL 6820, U.S.A. P. Bryan Heidorn School of Inforation Resources & Library Science, The University of Arizona 55 East First Street Tucson, AZ 8579, U.S.A. ABSTRACT User satisfaction, though difficult to easure, is the ain goal of Inforation Retrieval (IR) systes. In recent years, as Interactive Inforation Retrieval (IIR) systes have becoe increasingly popular, user effectiveness also has becoe critical in evaluating IIR systes. However, existing easures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new easures to evaluate IIR systes, the Noralized Task Copletion Tie (NT) and the Noralized User Effectiveness (NUE). The two easures overcoe liitations of existing easures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two easures and the user satisfaction and effectiveness of a given IR syste. The learning effects described by NT, NUE, and the task copletion tie were also studied and copared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of syste effectiveness than task copletion tie, and both new easures are superior to task copletion tie in describing the learning effect of the given IR syste. Keywords Measureent, experientation, huan factors. INTRODUCTION Inforation Retrieval (IR) syste evaluation has been studied in the IR research counity for decades. There is ASIST 200, October 22 27, 200, Pittsburgh, PA, USA. Copyright of this work is owned by the authors. ASIS&T has the perission to use and display digital or hard copies of all or part of this work. still a heated debate regarding easures of syste evaluation (e.g., Borlund & Ingwersen, 997; Cooper, 973; Geva, Kaps, Peters, Sakai, Trotan, & Voorhees, 2009). In addition, as there have been ore and ore eerging Interactive Inforation Retrieval (IIR) systes. Evaluations of IR systes consequently need to take into account users interactive processes of inforation searching and retrieval (Beaulieu, Robertson, & Rasussen 996; Belkin, 2008; Kelly, 2009). The ore effective a syste is, the less tie a user will need to spend on it to fulfill their inforation needs. The whole interactive inforation search process is also the process of users learning to utilize IR systes with increasing proficiency. Consequently, for a given IR syste, it is iportant to support non-expert users to iprove their search perforance over tie. This study focuses on evaluating IIR systes by proposing two new easures -- Noralized Task Copletion Tie (NT) and Noralized User Effectiveness (NUE). These easures strive to evaluate the effectiveness, efficiency as well as the learning effect of IR systes. Here we define learning effect as systes support for non-expert users iproving search perforance over a series of search sessions. Using a standard task copletion tie and a standard user effectiveness easure as noralization factors, the two new easures ai to overcoe deficiencies of existing easures such as precision, recall and task copletion tie. A user study of a real-world IR syste was conducted to exaine whether the proposed easures were good indicators of syste effectiveness, user satisfaction and learning effect. RELATED WORK It has been a half-century since the Cranfield odel of laboratory-based tests were invented, and it is still the doinant approach to IR evaluation, with precision and recall still widely used as IR evaluation easures (Borlund, 2003). However, the literature in IR evaluation has reported ixed results regarding whether precision or recall is a

2 significant factor of user satisfaction towards IR systes (e.g., Turpin & Hersh, 200; Kelly, Fu, & Shah, 200). In addition, precision and recall are insufficient for evaluating IIR systes, because while users ay odify or develop search queries and strategies during search processes, the two easures cannot quantify the inforativeness of interactions (Borlund & Ingwersen, 997). Cooper (973) and other researchers believe that an ideal evaluation ethodology ust soehow easure the ultiate worth of a retrieval syste to its users in ters of an appropriate unit of utility. But both user satisfaction and utility are difficult to define and quantify. Cleverdon and Keen (966) listed tie lag (response tie of the syste to user requests) as one of the six iportant criteria that could be used to evaluate IR systes. Su (992) stated efficiency-tie as the ost iportant search success category reported by users (also see Jones, 997). With the developent of IIR systes, researchers then started to pay attention to the dependency between task copletion tie and interface. Dunlop (997) proposed a new easure, called the expected search duration and established an interface-based predicted-tie odel, which easured how long it took to view a set of docuents before a nuber of relevant ones were found. This odel used task copletion tie as a criterion and integrated user interface into IR syste evaluation. Researchers also began to realize the influence of learning in the IR process. Allen (994) studied the relationship between perceptual speed, learning, and inforation retrieval perforance by end users. He found that inforation syste usability was deterined by interactions between characteristics of the users and syste features. Borlund (2003) also argued that IR evaluation should pay attention to the learning process of users and take into account odifications ade by users during the IR process. Therefore, it is desirable to have easures that can gauge the learning effect of an IR syste (i.e., the support for non-expert users becoing proficient users) and its effectiveness at the sae tie. Deficiencies of Existing Measures Precision and recall are the ost popular easures of IR evaluation, but researchers (e.g., Su, 992; Hearst, 2009) have found that users ay not always care about precision or recall. For exaple, depending on their inforation seeking tasks, users ay not be concerned about retrieving all the docuents relevant to their search tasks; for any users, they are happy if they can get a good answer in a short aount of tie. In other cases, users do not only care about precision and recall; other attributes of the syste, user and task ay also play iportant roles in the search process. Therefore, additional easures are needed to better understand the interactive nature of IIR. It is coonly agreed that user satisfaction is the ain goal of IR systes. However, it is difficult to precisely define satisfaction or objectively evaluate systes by subjective feelings reported by users (Belkin & Vickery, 985). According to Su (992), users consider task copletion tie as critical to successful inforation retrieval, and tie-based easures are often used in Web searches (Xu & Mease, 2009) and IIR evaluations (Kelly, 2009). Here we forally define the task copletion tie as the tie period fro when the user begins a search to the point he or she stops the search. The task copletion tie is not only deterined by an IR syste s paraeters, e.g., the syste s response speed, its user interface, and ranking of results, but it is also influenced by the users and the coplexity of search tasks (also called task difficulty (Gwizdka, 2008)). This akes the task copletion tie incoparable across different systes, users and search tasks. To reduce users influence on the task copletion tie, researchers norally divide users into different groups, based on their knowledge background. To reduce the influence of task difficulty, researchers often perute search tasks through a Latin-Square arrangeent to ensure all search tasks were perfored in different orders by the users. Note that this ethod only works when the saple pool of search tasks is large. Should we have a standard task copletion tie, e.g., the tie taken by an expert, as a reference, the influence of task difficulty could be eliinated without requiring a large task saple pool. In this way, the cost of IR evaluation can be greatly reduced. The two easures proposed in this study are based on a standard task copletion tie and thus have the erit of being ore cost-effective than task copletion tie. NEW MEASURES AND HYPOTHESES Definition of NT and NUE The new easures NT and NUE are proposed to solve the aforeentioned deficiencies of existing easures, using a standard task copletion tie and a standard user effectiveness easure as noralization factors. NT is defined as: t e NT i = () tni Where t e is the average tie expert users spend on a search task, t ni is the tie that a novice user, i, spends on the search task. Unlike the task copletion tie, NT is not an absolute value of copleting a task, but the proportion of tie that a perfect user spends on one single search task relative to the tie that a novice user spends on the sae search task. We define the perfect user using three criteria: ) being good at searching for inforation in general and thus able 2

3 to correctly interpret search tasks and eploy sound search strategies; 2) having the best knowledge of a given IR syste and thus able to choose the optial way to do a search using this syste; 3) having the best expertise in the doain of a given search task and thus able to create good queries and assess relevance for the task. In the real world, the perfect user does not exist, but the tie spent by a perfect user could be approxiated by aggregating the tie spent by a nuber of expert users. We use the ean as the aggregation function in order to give each expert an equal weight. Therefore, supposing there are expert users, forula () can then be written as: NT t ek k = i = (2) tni For exaple, let us copare two systes. If a novice user spends 2 inutes conducting one search task by using syste, while the average tie of expert users is inute, then the NT of syste would be 0.5; if the novice user spends.5 inutes by using syste 2, and the average tie of expert users is still inute, the NT of syste 2 would be Based on the saple data, we ay conclude that syste 2 is better than syste for the novice user. By dividing the tie spent by the perfect user (i.e., a standard task copletion tie), NT is expected to be ore stable than the original task copletion tie when the difficulty level of the search tasks varies. For instance, if one search task is very difficult, both expert users and novice users ay spend a long tie searching for it, so the task copletion tie will be longer than that of conducting other search tasks using the sae syste. Here, NT will not change as uch as the task copletion tie, because the nuerator and the denoinator are both increasing. The sae applies if a search task is easy. The task copletion tie ay be very sall, while the NT will still be relatively stable. This feature of NT (as well as NUE, see below) focuses on the evaluation of IR systes and avoids distractions by the difficulty levels of search tasks, which cannot be easily controlled. Despite the aforeentioned advantages, NT is, in fact, a siplified calculation. Although a novice user spends ore tie than an expert user, the quality of the forer s search results ay still not be as good as that of the latter s. The proportion of relevant docuents retrieved by the two users can indicate the difference of the effectiveness of their searches. In the above exaple, using syste, the novice user ay have retrieved 5 relevant docuents while the expert user retrieved 0; and using syste 2, the novice user ay have retrieved 3 relevant docuents, while the expert user retrieved 8. In this case, the difference in search quality between the two users cannot be reflected by NT. To solve this proble, NUE is proposed: NUE = NT i i k= I I ni ek = t ek k= ni I k= where I ni stands for the nuber of relevant results retrieved by the novice user, i, and I ek stands for the nuber of relevant results retrieved by the expert user, k. The nae of NUE coes fro user effectiveness. Al- Maskari & Sanderson (200) defined user effectiveness as the accuracy and copleteness with which users achieve certain goals (p. 860) and easured it by the nuber of relevant docuents retrieved and the tie users take to coplete the task. NUE is the ultiplication of NT and the nuber of relevant docuents retrieved by a user noralized by the nuber of relevant docuents retrieved by a perfect user. Again, the perfect user is represented by the average of a nuber of expert users. In this study, the notion of binary relevance is adopted, i.e., a docuent is either relevant or irrelevant to a search task. According to the definition, in the above exaple, the NUE of syste is 0.25, and the NUE of the syste 2 is also In ters of NUE, the two systes are the sae. NUE contains ore inforation than NT, as it also easures the effectiveness of retrieval by calculating the noralized nuber of retrieved relevant docuents. NUE is specifically useful when recall is critical for users, i.e., when users need to retrieve as any relevant docuents as possible (e.g., when doctors ay want to find all prescriptions for one disease). As entioned above, because NT and NUE are not affected by the coplexities of different search tasks as uch as the task copletion tie is, calculating the only requires a sall saple pool of search tasks. This efficiency in calculation is also an advantage over task copletion tie. Hypotheses on NT and NUE This study ais to find etrics that can evaluate both the effectiveness and the efficiency of IR systes, while being a good indicator of user satisfaction. Meanwhile, as learning is an indispensable part of an IIR process, the etrics also ai to easure systes learning effect (i.e., their support for non-expert users iproving search perforance over search sessions). This section states the four hypotheses exained in this study. Hypothesis : NT and NUE are highly correlated to user satisfaction. As user satisfaction is the ain goal of IR systes, it is desirable to have easures with strong correlations to user satisfaction. We expect NT and NUE are such easures. t I ni ek (3) 3

4 Hypothesis 2: NT and NUE are better easures of user satisfaction than the task copletion tie, as defined by fit to regression odels. Due to the aforeentioned advantages of NT and NUE over the task copletion tie, we also expect that NT and NUE will have a higher correlation to user satisfaction than the task copletion tie. Hypothesis 3: NT and NUE are highly correlated to the F easure. To investigate whether NT and NUE are good easures for syste effectiveness, we exaine the correlations between each of the two new easures and the F easure, a cobination of precision and recall that is coonly used to easure the effectiveness of IR systes. The F easure is forally defined as: F = 2 (precision recall)/(precision + recall). If NT and NUE can evaluate the effectiveness of an IR syste, they should be strongly correlated with the F easure. Hypothesis 4: NT and NUE are better than the task copletion tie in easuring the learning effect of an IR syste. Learning effect can be depicted by a learning curve, which is a graphical representation of search perforances over a series of search sessions. In the aeronautical industry, learning curves is based on repeating tasks perfored by a worker, and task copletion tie is a coon easure to study learning curves (e.g., Wright, 936). However, this is not quite the sae situation in IR research where there is no point asking a user to repeatedly search for the sae task. Thus, different search tasks are assigned to the sae users to test a syste. Therefore, the task copletion tie is very likely to be influenced by the difficulty levels of search tasks. It was hypothesized that NT and NUE can easure the learning effect of an IR syste better than the task copletion tie, because, as stated in last section, NT and NUE are expected to be ore stable than the task copletion tie when the difficulty level of search tasks varies. EXPERIMENTS Research Settings An experient was conducted to exaine the proposed hypotheses by coparing NT and NUE with other selected easures. This experient was conducted on a state university capus fro January 2007 to March The details of the experiental settings are explained in the following sections. Testing Syste In this study, LexisNexis Acadeic was the selected testing syste. As a coercial database, though it is accessible to novice users, it is ostly used by experienced users (e.g., inforation professionals) because this powerful database is not as easy to use as Web search engines (e.g., Google) for the general public. Therefore, this syste is desirable for recruiting both novice and expert users. LexisNexis Acadeic provides coprehensive coverage of five sections: news, business, legal inforation, edical inforation and reference publications, with full text and abstracts. The search tasks assigned to participants covered all of the above five sections. Subjects were asked to use the Guided News Search, rather than the Quick News Search option, because the latter one is too general and could return too any docuents. Subjects There were a total of 20 subjects, including 5 novice users and 5 experts who voluntarily participated in this experient on the capus of the state university. The subjects were selected based on their failiarity with the LexisNexis Acadeic search engine. The expert users were university librarians who used LexisNexis Acadeic extensively. The novice users were staff, undergraduate and graduate students at the university. Their ajors varied fro science and social science to huanities. Aong the 5 novice subjects, 7 were ale and 8 were feale. The subjects were given an introduction letter specifying the experiental procedure, and the researchers reiterated the procedure to the subjects before starting the experient(s). Subjects were assigned an ID code as their identity. Before the novice subjects started searching, they were asked to answer an entry-questionnaire to ake sure they were qualified for this research, i.e., they had soe knowledge of the Internet, but had not used the testing syste before. The novice users all received basic training on how to use this syste prior to perforing searches. The training lasted for about 0 inutes. They were first asked to study the instructions for LexisNexis Acadeic by theselves. The researchers answered their questions if they had any. Search Tasks Each participant was assigned 0 search tasks. These search tasks were based on real-life reference questions fro the university library. The 0 search tasks covered the five ajor sections of LexisNexis Acadeic (see the list of search tasks below). The search tasks were not doainspecific, and thus the last criterion of a perfect user was relaxed in this experient. Specifically, the perfect user was represented by 5 experts (librarians) who were experienced in searching for inforation, were very failiar with the testing syste, but were not required to be experts in any specific doain. 4

5 The researchers ran a pilot study before finalizing the search tasks to ake sure that LexisNexis Acadeic could retrieve the relevant docuents for all these search tasks. Each search task was assigned an ID code, and the 0 tasks were given to the subjects in different orders distributed by a Latin-Square arrangeent to reduce the influence of the search tasks difficulty levels on the task copletion tie. For exaple, the following are the search tasks given to one subject, who conducted the searches in the listed order fro to 0. The nubers in parentheses were the search task IDs, e.g., the first search task given to this subject was search task 4 (Q4).. Who are the announced Deocratic 2008 Presidential candidates? (Q4) 2. Find an obituary for Princess Diana fro the Washington Post. (Q9) 3. Find reviews of the ovie Brokeback Mountain. (Q0) 4. According to recent research, which vitain is iplicated in skin cancer developent with the intake of folate? (Q3) 5. Find data on the urder rates for each state in the USA after (Q7) 6. How any points did Yao Ming score in the gae after his return fro his ost recent injury? (Q6) 7. Who was the auditor of Best Buy Co. in 2004? (Q) 8. Find articles about President Bush s ost recent noinee as the next US envoy to the United Nations? (Q5) 9. What are keywords that ay result in a webpage being blocked by the Chinese governent? (Q2) 0. What is the phone nuber for Google s ain corporate office? (Q8) Measures To test the proposed hypotheses, this study copared the following seven easures:. Precision (P): The proportion of retrieved docuents that are relevant, where P = the nuber of retrieved relevant docuents / the nuber of retrieved docuents 2. Recall (R): The proportion of relevant docuents that are retrieved, where R = the nuber of retrieved relevant docuents / the nuber of relevant docuents It is difficult to know how any relevant docuents for a search task exist in a large-scale IR syste like LexisNexis Acadeic. Because of this well-known difficulty of calculating recall, a odified relative recall was used to substitute. We followed the pooling ethod used in the Text Retrieval Conference (TREC) (Voorhees & Haran, 200). For a given search task, all docuents retrieved by the 20 users were erged into a single set, and only those docuents were judged on relevance. Docuents that were not retrieved by any of the subjects were not considered. We eployed one assessor to ake relevance judgents, because the experience in TREC indicated that judgents ade by a single judge and group judgents did not affect the relative perforance of the retrieval systes (Voorhees, 998). The assessor in this experient was an IR researcher in the library school at the state university and was failiar with the nature and issues of relevance judgents. 3. F easure (F): The weighted haronic ean of precision and recall, also called the traditional F-easure or the balanced F-score, where F = 2 (precision recall)/(precision + recall) 4. Task copletion tie (T): The tie fro the start until the copletion of a retrieval task. The task copletion tie is recorded in seconds, but note that the inute was the unit in the calculations. The subjects stopped each search session by theselves, either once their search needs were satisfied or when they gave up. 5. User satisfaction (S): An ordinal nuber indicating the level of user satisfaction. A questionnaire was provided to the users after each search was coplete. It asked the subjects to rate their satisfaction level towards the syste in regard to supporting the accoplishent of the search task, using a scale fro (not satisfied at all) to 5 (extreely satisfied). 6. NT (see the Definition of NT and NUE section) 7. NUE (see the Definition of NT and NUE section) Data Collection The whole experiental process lasted between 45 inutes to.5 hours for each subject. For each search task, the subject stopped the search session when either the inforation need was satisfied or he or she gave up the task. The researchers observed the subjects perforing their searches in order to ensure that the correct procedures were being followed. The researchers also recorded the task copletion tie of each task, and asked the subjects to answer a questionnaire after each task. As a subject ay odify queries during a search session, the retrieved docuents were defined as those displayed on the screen in response to the last query in a search session. As required in the instructions of the experient, the subject sent the researchers his or her search results by e-ail at the end of each search session. After gathering the e-ails fro all subjects, the relevance of all the retrieved docuents was judged by the independent assessor. 5

6 RESULTS The 0 novice users and the 5 expert users each perfored the 0 search tasks, and thus there were 50 search sessions. Aong all the 50 search sessions, 7 were unsuccessful in that no relevant docuents were retrieved. In these cases, the users gave up the task after a few futile attepts. As the tasks were never copleted, the task copletion tie in these cases would be infinite, and thus the values of NT and NUE would be 0. For 8 other sessions, the users copleted their search when they believed they had retrieved relevant docuents, but in fact, they did not retrieve any relevant docuents. For these cases, we still used the tie they spent on the sessions as their task copletion tie and calculated NT accordingly. However, the value of NUE would be 0 because the nuber of retrieved relevant results was 0. For a coparison of statistics of novice users and expert users, Table lists the average task copletion tie and the nubers of retrieved relevant docuents of the two user groups on each search task. Statistics Analysis on User Satisfaction To test the first hypothesis, NT and NUE are correlated with user satisfaction; correlation analysis and regression were applied. The correlations between user satisfaction and other easures, such as precision, recall and task copletion tie, were also studied as coparisons to NT and NUE. Here, user satisfaction is the dependent variable, while task copletion tie, precision, recall, NT and NUE are the independent variables. The saple size for the analysis is 50, where each of the 5 novice users conducted searches on each of the 0 search tasks. Search Tasks Correlation Matrix The degree of user satisfaction has five levels, fro (not satisfied at all) to 5 (extreely satisfied). Spearan's rank correlation coefficient was selected for the correlation analysis because it can be applied to ordinal variables such as user satisfaction. Unlike the ore coonly used Pearson correlation coefficient, the Spearan's rank correlation coefficient does not require the assuption that the relationship between the variables is linear, nor does it require the variables to be easured on interval scales. Table 2 presents the Spearan's rank correlation coefficients for all the variables we studied. It shows that NT had the strongest correlation with user satisfaction aong all the tested easures, followed by task copletion tie (T). However, the correlation between NUE and user satisfaction could only be described as oderate at best. This result partially supports the first hypothesis that NT and NUE are highly correlated to user satisfaction. As a coparison, it is noteworthy fro Table 2 that precision (P) and recall (R) were weakly correlated with user satisfaction (S). Avg. T by Novice Users Table. Statistics of Two User Groups across Search Tasks T P R NT NUE S T P R NT Note: N = saple size Avg. T by Expert Users Avg. # of Ret. Rel. Docs by Novice Users.7952 Avg. # of Ret. Rel. Docs by Expert Users Q Q Q Q Q Q Q Q Q Q Total Table 2. Spearan Correlation Matrix for Covariates of User Satisfaction (N=50) Regression Models Five ultinoial logistic regression odels were run to further study the relationships between user satisfaction and the five easures respectively. As an ordinal variable, user satisfaction does not obey the assuption that the independent and dependent variables have a linear relationship. Thus the ultinoial logistic regression odel was used, for it assues neither a linear relationship, nor does it require norally distributed variables. Based on our saple size (N=50), we set the significance level at 6

7 0.0, i.e., two variables are not recognized as significantly correlated to each other unless p 0.0. A suary of the five regression odels is presented in Table 3. Based on these odels, NT, NUE and the task copletion tie were all significantly correlated with user satisfaction at the p level. According to the log likelihood of the three odels, the odel of NT had the best fit, i.e., NT was the best predictor of user satisfaction, while NUE was not better than task copletion tie to predict user satisfaction. Hypothesis 2, which posited that NT and NUE are better easures of user satisfaction than task copletion tie, was partially supported. According to Table 3, precision and recall were not significantly correlated with user satisfaction (p > 0.0), which indicated that the two easures were not good indicators of user satisfaction. This is in accordance with the weak correlations between precision, recall and user satisfaction found in Table 2. Statistics Analysis on F Measure To test hypothesis 3, three linear regression odels were used to study the relationships between the F easure and each of NT, NUE, and the task copletion tie (shown in Table 4). Unlike user satisfaction, which is an ordinal variable, F easure, NT, NUE, and task copletion tie are interval variables, and thus the linear regression odel is suitable for the. According to the three odels, NT and the task copletion tie were not significantly correlated with the F easure (p > 0.0), while NUE has a significant correlation with the F easure (p < 0.0). This finding partially supports hypothesis 3 that states that NUE is correlated with the F easure, but rejects that NT is correlated with the F easure. Analysis on Learning Curves The learning curve discloses the relationship between users experiences with the syste and search perforance. As individuals get ore experienced at using a syste, their search perforances usually iprove. In IR evaluation, we can copare the slopes of different systes learning curves to study the learning effect of IR systes. The steeper the slope, the better the syste can support its novice users iproving their search perforances over tie. A learning curve was drawn using the 5 novice users average task copletion tie of each search task. The x- coordinate is the successive sequence of the 0 search tasks conducted by the novice users. As entioned in the experient design section, the search tasks were peruted through a Latin-Square arrangeent, such that each subject had a unique sequence of search tasks. Figure -3 show the learning curves explained by the task copletion tie, NT, and NUE respectively. I. V. Model () T (.0339) Model (2) P.4350 (.435).334 D. V.: S Model (3) R.9220 (.4737).056 Model (4) NT.474 (.6240) Model (5) NUE.330 (.3854) Log Note: N = saple size D.V.=Dependent Variable I.V.=Independent Variable Log.=Log Likelihood Table 3. Suary of the Five Multinoial Logistic Regression Models for User Satisfaction (N=50) Independent Variable T (.0037) Note: N = saple size Dependent Variable: F Model (i) Model (ii) Model (iii).222 NT.0038 (.0439).932 NUE.077 (.0359).0027 Table 4. Suary of Three Linear Regression Models (N=50) Fro Figure, we can observe that, on average, the ore search tasks the users conducted, the faster the users becae. This eets the general iplication of the learning curve. However, the coefficient of deterination, R 2 was That is, 48.42% of the variation in the response variable could be explained by the task copletion tie, 7

8 and the reaining 5.58% could be explained by unknown, lurking variables or inherent variability. For exaple, the fluctuation in the iddle part of the curve ight be due to changes of interest or concentration level of the huan subjects in the iddle of the experient. In any case, the curve shows task copletion tie is a weak easure for learning effect. The learning curves on NT and NUE (Figures 2, 3) show that the ore search tasks the users practiced, the higher the NT and NUE were, i.e., the novice users becae closer to the experts. This also eets the deduction of learning curves. According to R 2, 74.32% of the variation in the response variable can be explained by NT, and 76.% of it can be explained by NUE. NUE and NT explain the learning curves uch better than the task copletion tie. Therefore, hypothesis 4 was supported: NT and NUT are better easures for describing systes support for novice users iproving search perforance over search sessions. DISCUSSION ON RESULTS Based on the experient results, we conclude that: NT was strongly correlated with user satisfaction and was a better indicator of user satisfaction than task copletion tie; NUE was a better indicator of syste effectiveness (represented by the F easure) than task copletion tie; and the two proposed easures were better than the task copletion tie in depicting the learning effect over the search sessions. The experient results also confired that precision and recall had weak correlations with user satisfaction and were not good indicators of user satisfaction. This is consistent with previous studies that found no substantial relationship between syste effectiveness and user satisfaction (e.g. Turpin & Hersh, 200). Also, the task copletion tie was not significantly correlated with syste effectiveness (F easure), based on our linear regression odels. In addition, NT and NUE described the syste s learning effect uch better than task copletion tie. Therefore, we conclude that, for factual search tasks in the general doain (such as the ones in this experient), NT and NUE can be used together as overall easures in evaluating IR systes in ters of user satisfaction, effectiveness and learning effect. Calculating NT and NUE requires participation of expert users, but it only requires a sall saple pool of search tasks. This is because NT and NUE are not affected by the coplexities of different search tasks as uch as the task copletion tie is. Therefore, NT and NUE are cost effective easures copared to task copletion tie. CONCLUSIONS AND FUTURE WORK User satisfaction, syste and user effectiveness, as well as support for users iproveent are all critical to IIR systes evaluation. However, existing easures such as precision, recall and task copletion tie are not sufficient tie (in.) sequence y = x R 2 = Figure. Learning Curve Explained by Task Copletion Tie LER y = x R 2 = sequence Figure 2. Learning Curve Explained by NT MLER sequence y = x R 2 = 0.76 Figure 3. Learning Curve Explained by NUE for easuring these critical aspects. In particular, precision and recall copletely leave users out of the picture while task copletion tie, to a large extent, depends on search tasks and users. This study proposes two new easures, NT and NUE, which can overcoe these liitations of existing easures by coparing novice users searches to those of experts. A user study was conducted to confir that NT is good for easuring user satisfaction of IR systes, NUE is a good easure for syste effectiveness, and both are good for describing systes learning effect. A single NT or NUE can evaluate the user effectiveness of a syste, and a series of successive NT or NUE can exaine the syste s support for novice users iproving search perforances by 8

9 observing the learning curve slope (i.e., how fast a novice user can catch up to an expert user when using the sae syste). In suary, we believe that NT and NUE are copleentary to existing easures. It is noteworthy that the search tasks in this experient are doain general, and thus future studies are needed to validate whether the conclusions are applicable to other doain-oriented search tasks. To confir the generalizability of the two easures, a coparison of two or ore IR systes is needed, and a larger saple of testing search tasks and subjects is also desirable in the future. ACHKNOWLEDGMENTS We thank the anonyous reviewers for their suggestions and help on iproving this paper. REFERENCES Allen, B. (994). Perceptual speed, learning and inforation retrieval perforance. In Proceedings of the 7 th ACM SIGIR Conference on Research and Developent in Inforation Retrieval (pp.7 80). Al-Maskari, A. & Sanderson, M. (200) A review of factors influencing user satisfaction in inforation retrieval. Journal of the Aerican Society for Inforation Science, 6(5): Beaulieu, M., Robertson, S. & Rasussen, E. (996). Evaluating interactive systes in TREC. Journal of the Aerican Society for Inforation Science, 47(): Belkin, N.J. (2008). Soe(what) grand challenges for inforation retrieval. ACM SIGIR. Foru Archive, 42 (): Belkin, N.J. & Vickery, A. (985). Interaction in inforation systes: A review of research fro docuent retrieval to knowledge-based systes. (LIR Report No 35). London: British Library Borlund, P, (2003). The IIR evaluation odel: a fraework for evaluation of interactive inforation retrieval systes. Inforation Research, 8(3), Paper No.52. Borlund, P. & Ingwersen, P. (997). The developent of a ethod for the evaluation of interactive inforation retrieval systes. Journal of Docuentation, 53(3): Cleverdon, C. W. & Keen, E.M. (966). Factors deterining the perforance of indexing systes. Vol. : Design. Vol 2: Results. Cranfield, U.K: Aslib Cranfield Research Project. Cooper, W. S. (973). On selecting a easure of retrieval effectiveness. Part. Journal of the Aerican Society for Inforation Science. 24(2): Dunlop, M. (997). Tie, relevance and interaction odeling for inforation retrieval. In Proceedings of the 20th ACM SIGIR Conference on Research and Developent in Inforation Retrieval (pp ). Geva, S., Kaps, J., Peters, C., Sakai, T., Trotan, A. and Voorhees, E. (editors) (2009). In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation. Boston, MA, USA. Gwizdka, J. (2008). Revisiting search task difficulty: behavioral and individual difference easures. In Proceedings of the 7th Annual Meeting of the Aerican Society for Inforation Science and Technology. Hearst, M. (2009). Search User Interfaces. Cabridge University Press. Jones, K. S. (997). Readings in Inforation Retrieval, San Francisco: Morgan Kaufann. Kelly, D. (2009). Methods for evaluating interactive inforation retrieval systes with users. Foundations and Trends in Inforation Retrieval, 3(-2): Kelly, D., Fu, X. & Shah, C. (200) Effects of position and nuber of relevant docuents retrieved on users evaluations of syste perforance. ACM Transaction on Inforation Syste, 28(2): -29. Su, L. T. (992). Evaluation easure for interactive inforation retrieval. Inforation Processing and Manageent, 28(4): Turpin, A. H. & Hersh, W. (200). Why batch and user evaluations do notgive the sae results. In Proceedings of the Annual International ACM SIGIR Conference on Research anddevelopent in Inforation Retrieval (pp ). Voorhees, E. (998). Variations in relevance judgents and the easureent of retrieval effectiveness. In Proceedings of the 2th ACM SIGIR Conference on Research and Developent in Inforation Retrieval (pp ). Voorhees, E. & Haran, D. (200). Overview of the sixth Text REtrieval Conference (TREC-6). Inforation Processing and Manageent, 36(): Wright, T. P. (936). Factors affecting the cost of airplanes. Journal of Aeronautical Sciences, 3, Xu, Y. & Mease, D. (2009). Evaluating web search using task copletion tie. In Proceedings of the 32 nd annual international ACM SIGIR conference on Research and Developent in Inforation Retrieval (pp ). 9