Training for Web Search: Will It Get You in Shape?

Transcription

1 Training for Web Search: Will It Get You in Shape? Wendy Lucas and Heikki Topi Department of Computer Information Systems, Bentley College, 175 Forest Street, Waltham, MA Given that time is money, Web searching can be a very expensive proposition. Even with the best search technology, the usefulness of search results depends on the searcher s ability to use that technology effectively. In an effort to improve this ability, our research investigates the effects of logic training, interface training, and the type of search interface on the search process. In a study with 145 participants, we found that even limited training in basic Boolean logic improved performance with a simple search interface. Surprisingly, for users of an interface that assisted themin forming syntactically correct Boolean queries, performance was negatively affected by logic training and unaffected by interface training. Use of the assisted interface itself, however, resulted in strong improvements in performance over use of the simple interface. In addition to being useful for search engine providers, these findings are important for all companies that rely heavily on search for critical aspects of their operations, in that they demonstrate simple means by which the search experience can be improved for their employees and customers. Introduction Being able to conduct effective searches has become more and more of a requirement for many of us, with companies relying on e-commerce to market and sell their products, and e-services rather than human operators to provide support. Companies also depend on the searching skills of their employees for finding information from knowledge management systems, corporate intranets, and the Web. While the effectiveness of the search technologies employed in these applications is critical, those technologies can only be effective if people are able to use them. To that end, searchers must understand how to express their information needs in a form that is both understood by the search engine and adequately conveys the intent of their search. Yet, users queries are often inadequate for expressing their search criteria, containing only a few search terms that Received June 11, 2003; revised September 15, 2003; accepted January 22, Wiley Periodicals, Inc. Published online 13 July 2004 in Wiley InterScience ( DOI: /asi may bear little relevance to the information they are seeking (Lucas & Topi, 2002; Spink, Wolfram, Jansen, & Saracevic, 2001). Searchers rarely use Boolean operators and even more rarely use them correctly, negating any possible benefit they might achieve. Search features labeled as advanced by search engines are seldom used, and hints and tips are usually ignored, partly due to an (often misguided) assumption that they are intended for advanced users (Pollock & Hockley, 1997). In addition, advanced search features are less accessible and therefore more likely to be overlooked than the simple search box offered as the primary search interface. In fact, many usability experts explicitly advise against implementing a search interface that supports Boolean queries as the primary interface (Nielsen, 1997; Petersen, 2000). The purpose of the study reported here is to investigate the effects of training on users Web search performance. It is not the search technology that is under investigation here but rather the users interactions with that technology. We consider the effects of both training in Boolean logic and training in the specific characteristics of the search interface on user performance and attitudes. Study participants sought answers to information requests by searching a known set of documents. All links had been removed from the documents, forcing users to formulate and reformulate queries rather than navigate from one page to another in search of answers. This study explicitly excludes issues related to broader search strategies (Brajnik, Mizzaro, Tasso, & Venuti, 2002) or paradigms (Dennis, Bruza, & McArthur, 2002). Two of the experimental treatments were based on the type of training the users received. The first of these treatments was the presence or absence of training on the principles of the use of Boolean operators, henceforth referred to as logic training. The second was the presence or absence of training that focused on the specific characteristics of the assisted search interface, which we will refer to as interface training. The third treatment was the type of interface used by the participants, namely, simple or assisted. The impacts of logic and interface training and the type of search interface used on correctness, time, satisfaction, and confidence were measured and analyzed. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 55(13): , 2004

2 The next section of this article reviews work related to the research presented here. This is followed by our research model and hypotheses, study methodology, and results. A discussion of those results, directions for future work, and conclusions are then presented. Related Work The majority of studies related to Web searching focus on the content of users queries or the characteristics of search engines rather than factors affecting the search experience and the impact of those factors on search performance. Two exceptions are Topi and Lucas (2003) and Te eni and Feldman (2001). In the former study, participants were presented with one of two interfaces: a simple search tool like those used by the majority of search engines or an interface designed to assist the users in formulating correct Boolean queries. Half of the participants received training on Boolean operators, while half did not. For those receiving no training, the quality of the search results (as measured by the number of correct responses to information requests) was significantly higher for users of the assisted search tool than for users of the simple one. This difference in correctness between interface treatments did not exist for users who received logic training. In all cases, the use of the assisted interface had a positive impact on both user satisfaction and confidence. The latter study evaluated user performance but concentrated on searching by browsing rather than by querying. They found that user performance, measured by time and accuracy, improved when the user s view of the Web site was adapted to the search task. User satisfaction with the adapted sites, however, did not increase, despite their improved functionality. The authors postulate that users prefer Web sites with consistent appearances. The selection of terms used in a query has been shown to have a significant effect on search performance (Lucas & Topi, 2002). Relevance feedback, which refers to the modification of queries by adding new terms and reweighting existing ones based on user feedback, is an accepted means for improving search results in traditional information retrieval (IR) systems (Buckley, Salton, Allan, & Singhal, 1995; Harman, 1992; Salton & Buckley, 1990). Yet Web searchers are loath to take advantage of such features. A transaction log analysis of 51,473 queries from 18,113 users of the Excite search engine found that only about 5% of users queries made use of the provided relevance feedback mechanism (Jansen, Spink, & Saracevic, 2000). The queries used an average of 2.35 terms, and less then 10% included any Boolean operators. These findings are consistent with those of similar studies (Abdulla, Liu, & Fox, 1998; Silverstein, Henzinger, Marais, & Moricz, 1999; Spink et al., 2001), which found that users typically submit short queries containing two or three terms. Although users do tend to change some terms when modifying their queries, the total number of terms often remains unchanged (Spink, Jansen, Wolfram, & Saracevic, 2002). In addition, logical operators are used infrequently (and used correctly even less frequently), and only the first page of search results are typically viewed (Jansen et al., 2000; Silverstein et al., 1999). Expert Web searchers do use more terms and advanced search operators than the average user (Hölscher & Strube, 2000), but far fewer terms than is typically found in queries to traditional IR systems, which are three to seven times longer than an average Web query (Jansen et al., 2000). Given the difficulty users have in formulating queries, natural language processing (NLP) technology, in which the user provides a description of the information need rather than keywords connected by logical operators, would seem to hold great promise for Web searches. In the IR community, it is generally accepted that users achieve better results with natural language interfaces than with explicit Boolean searches (Borgman, 1986; Anick et al., 1990; Turtle, 1994). Empirical evidence, however, is contradictory, with Hersh, Turpin, Price, and Kraemer (2001) finding that Boolean queries can be at least as successful as natural language searches. The application of NLP technology to Web searching is in its infancy, with commercial search engines providing limited capabilities in this area (Lucas, 2001). For example, although the AskJeeves search engine permits users to enter natural language queries, it does not attempt to interpret them. Rather, it returns a list of matching questions from its database. The user then chooses from this list and is taken to the Web page with an answer to the selected question. Kwok, Etzioni, and Weld (2001) built a questionanswering system called MULDER on top of the Google search engine. MULDER parses the user s natural language query, building several different keyword queries that are automatically submitted to Google. Answers are extracted from the returned results using a voting procedure to select the best candidates. In comparisons with AskJeeves and Google, MULDER outperformed both in terms of required user effort at every level of recall. User effort was significantly less than that required with Google, and 34% of questions had the correct answer as their top-ranked results (as compared to 1% for Google). MULDER also demonstrated a threefold advantage in recall when compared to AskJeeves. The IntelliZap System of Finkelstein et al. (2002) implements a context-based approach to searching. Users can highlight text in a document about which they want more information. The context around that text is captured using NLP techniques, including semantic keyword extraction and clustering, and new augmented queries are automatically generated. These queries are submitted to general and domain-specific search engines, and the results are reranked based on their semantic proximity to the original context. The level of performance achieved by IntelliZap, which requires very limited user involvement, was comparable to that achieved by participants using major search engines. Research efforts have also been devoted to providing users with graphical representations of their queries, with the expectation that these will be easier to understand than the underlying textual queries. Anick et al. (1990) developed an interface that translates users NLP queries into a visual 1184JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004

3 representation of their Boolean equivalent. Users can then manipulate this representation to improve its expressiveness. This approach was not tested with any actual users. Young and Shneiderman (1993) developed an interface that uses the metaphor of water flowing through filters to represent Boolean queries. In a comparison with a text-only SQL interface, 20 study participants performed better in terms of query correctness when using the visual interface. These findings are not, however, directly applicable to Web search because the study was implemented in a structured database context. The study described in this article is not about user interface design; our assisted search interface provides standard assistance with Boolean operators, using dropdown lists like those found in the typical advanced search pages of commercial search engines. Rather, our focus is on the user s ability to conduct searches using standard search features and how that ability can be affected by training. For Web searches to be effective, it is critical that the queries entered by the user accurately captures her information needs. In this article, we explore the impacts of logic training, interface training, and the search interface on the user s ability to successfully translate those needs into executable queries. Theoretical Framework The success of a search is commonly defined in terms of its performance (correctness or accuracy, time) and the searcher s attitudes (confidence and satisfaction) (Chan, Tan, & Wei, 1999; Te eni & Feldman, 2001). The theoretical framework for our research focuses on those factors that ultimately affect the successfulness of the search process via their effects on performance and attitudes, as represented in Figure 1. First and foremost, a searcher must understand the information request to initiate a successful search. This understanding will affect the entire search process, from the quality of the searcher s conceptual query to his ability to understand query results. The conceptual Boolean query is also affected by the searcher s understanding of Boolean constructs. As an example, consider the case where the searcher is asked to find information about coffee beans that do not come from Colombia or Brazil. In conceptualizing this query, the searcher must recognize that coffee beans, Colombia, and Brazil are the key terms or phrases that define the context for this search. The searcher must also understand the nature of Boolean constructs that allow search criteria to be associated with search terms and phrases. That is, coffee beans must be included in the search results, but neither Colombia nor Brazil should be. The conceptual Boolean query can therefore be expressed as: coffee beans AND NOT (Colombia OR Brazil). The searcher must then translate this conceptual query into a form that can be parsed by the search engine to which it is being submitted. Understanding the syntax rules governing proper query formation will therefore impact the quality of the query implementation. If submitting to Google, for example, the use of AND or is not required, because all terms in the query must appear in every retrieved document by default. This is not always the default interpretation for other search engines. In Lucas and Topi (2002), it was shown that errors involving query operators, caused by either their incorrect usage or their absence, had a significant negative effect on search results. The success of a query, as defined by performance and attitudes, ultimately rests on the quality of the query implementation and the ability of the searcher to understand and interpret the results. In addition, there are numerous individual and contextual characteristics that play a role in all aspects of the search process. These include the searcher s experience and domain knowledge (Hölscher & Strube, 2000), cognitive abilities and style (Allen, 2000; Ford, Miller, & Moss, 2001), problem-solving style (Kim & Allen, 2002), self-efficacy (Ford et al., 2001), maturity, and ability to concentrate, among other factors. Conceptual Understanding of the Information Request Understanding of Boolean Constructors Quality of Conceptual Boolean Query Quality of Query Implementation Ability to Understand and Interpret Results Ability to Use Search Interface Performance and Attitudes Additional Factors: Individual and Contextual Characteristics Search Interface Understanding of Search Engine Characteristics FIG. 1. Model of factors affecting user s search experience. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November

4 Search Interface Simple Assisted Operator Usage Correctness Time Training Type Logic Assisted Interface Term Usage Satisfaction Confidence User Support Success Measures FIG. 2. Model of study treatments. Research Model The theoretical framework for our research lays the groundwork for evaluating the effects of the users ability to formulate correct Boolean queries and to transform a conceptual Boolean expression into a syntactically correct query. Topi and Lucas (2003) found that the use of their assisted search tool had a significant positive effect on the quality of search results over the use of their simple search tool. Boolean training, however, removed the assisted search tool advantage by compensating for the lack of direction and assistance in the simple search environment. This study extends that work by considering the effects of assisted interface training, in addition to those of logic training and the search interface, on the searcher s performance and attitudes, as shown in Figure 2. Performance is defined here in terms of the correctness of the response to an information request and the elapsed search time, while attitudes include satisfaction and confidence. In accordance with the model of Figure 1, training in Boolean logic should improve the searcher s understanding of Boolean operators, which should have a positive effect on the quality of the conceptual query. We do not expect this training to have any impact on the selection of search terms, which also affects quality. Standard experimental procedures should ensure that errors related to search term selection are evenly distributed amongst the study participants, limiting their effects on our results. The choice of the search interface affects the quality of the query implementation, and can be influenced by training in the use of that interface. Because the simple interface is one with which all search engine users are expected to be familiar, and an understanding of how to form Boolean queries is sufficient for its successful use, no additional training is needed. For the assisted interface, searchers should benefit from training specifically aimed at understanding its functionality. Hypotheses Our first four hypotheses concern the users of the assisted interface. Based on the results of prior related research (Topi & Lucas, 2003), we do not expect logic training to affect the performance of these users. We do anticipate, however, that their performance can be improved by training them on the specific characteristics of that interface. We posit that interface training improves the participant s ability to translate the conceptual idea of a Boolean query into the implemented query that is ultimately submitted to the search engine. The search results retrieved by the search engine should be more relevant to the information request because of the improvement in the users ability to formulate queries. Increased relevancy of search results leads to an increase in the likelihood that the participant will find the correct answer to the information request. In addition, the time required for finding that answer should be reduced because training is expected to improve the speed of query formulation as well as the quality of the retrieved documents. We also predict that interface training will improve the participants confidence in their results because they will have a better understanding of the search interface. Having better command of the search tool will reduce the effort they need to expend, thereby improving their satisfaction with the search environment. These predictions lead to the following hypotheses: H1: For the users of the assisted interface, interface training will lead to improved correctness of search results. H2: For the users of the assisted interface, interface training will lead to reduced time required to complete the search task. H3: For the users of the assisted interface, interface training will lead to improved confidence in the search results. H4: For the users of the assisted interface, interface training will lead to improved satisfaction with the search environment. As previously mentioned, logic training on Boolean constructors was shown to lead to improved search performance, as measured by the correctness of answers to information requests, for participants using a simple, nonassisted search interface (Topi & Lucas, 2003). This earlier research also implied that logic training is not effective if the participants use an interface that supports the user in constructing 1186 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004

5 Boolean queries. In the absence of training, users of the assisted interface performed at a higher level than users of the simple interface. We expect to replicate these findings in this research with simple interface users and assisted interface users who received interface training. H5a: For the users of the simple interface, logic training will lead to improved correctness of the search results. H5b: The users of the assisted interface will perform at a higher level of correctness than the users of the simple interface who did not receive logic training. Results of that earlier research also suggest that logic training will not have an impact on the time it takes to perform the search tasks. This is understandable because a relatively brief training session is not extensive enough for participants to be able to automate any of the routines required for producing correct Boolean queries. Thus, training may, in fact, increase the required search time. We do predict, however, that users of the simple interface who receive logic training will be more confident about their results than those who do not receive this training. This is because logic training will improve both the participants understanding of the process of query writing and the quality of the search result, leading to an increase in the participants perception of the quality of their work. These same factors should also cause an improvement in the participants satisfaction with the search process. H6: For the users of the simple interface, logic training will lead to improved confidence in the search results. H7: For the users of the simple interface, logic training will lead to improved satisfaction with the search environment. Methodology Experimental Search Environment The experimental environment used in this study was developed for an ongoing research program on Web searching. It permits the integration of survey instruments and search interfaces to measure the effects of a variety of factors on user performance and satisfaction in conducting Web searches. The modular design of this three-tiered architecture enables components to be swapped in and out, depending on the specific needs of the experiment being conducted. In a typical session using this research environment, a study participant is first guided through a number of survey and test instruments. The former are used to obtain information on computer skills, educational background, and various attitudes, while the latter test the participants understanding of such concepts as Boolean operators or the use of a search interface. The participant then attempts to find the answers to several information requests through a repetitive process of forming queries, submitting them, and viewing the retrieved pages. Searches are conducted over a predefined search space containing a snapshot of 21,890 documents from a FIG. 3. Simple search interface. university intranet Web site. All of the hypertext links have been removed from those documents so that participants must search, rather than browse, for information. Two search interfaces have been integrated into the experimental search environment to date. Each participant is assigned to one of these interfaces. The simple search interface is a text box with a submit button. An assisted search interface is also integrated into this environment. Dropdown menus allow the participant to select the appropriate Boolean operators. Multiple terms entered to a search box are treated as phrases, so that quotation marks are not required. Parenthesis may be specified by clicking the buttons appearing to the left and right of the search boxes. Figures 3 and 4 show the simple and the assisted interface, respectively (please see the Appendix for a description of the usage of the assisted interface). A query submitted from either interface is processed and forwarded to the Microsoft Index Server search engine. Search results are returned in groups of 10, with a title and a brief abstract displayed for each of the pages included in the answer set. The participant can click on links to view the full pages listed in the set of retrieved results but cannot navigate from any of those pages to another page. If the participant does not find the answer being searched for, then new queries can be entered until either the answer is found or the maximum time allotted for each information request has been reached. Responses to surveys, keystrokes entered to the search interfaces, answers to information requests, and corresponding time stamps are all stored in a database for later analysis. Participants and Setting The participants for this study were recruited from students enrolled in an introductory information systems course at a small university located in the northeastern United States. Table 1 contains background information and Internet-related experience for the 145 participants. The experiment was conducted as an in-class search exercise, and participation was voluntary. As an incentive, $50 prizes were offered to the two best performing participants based on the accuracy of their responses to the information requests and their total search times. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November

6 No Logic Training Logic Training No Assisted Search Training I. Simple Interface II. Simple Interface No Assisted Search Training III. Assisted Interface IV. Assisted Interface Assisted Search Training V. Assisted Interface VI. Assisted Interface FIG. 5. Research design. interface, with those in cell II receiving logic training. Participants in cells III through VI used the assisted search interface. Cell III participants received no training, cell IV participants received only logic training, cell V participants received only assisted interface training, and cell VI participants received both logic and assisted interface training. It is important to note this is not a complete design, because there is no meaningful interface training treatment for the users of the simple interface. This will, naturally, be taken into account in the statistical analysis of the data. TABLE 1. FIG. 4. Demographic characteristics of study participants. Mean Min Max Age: Male Assisted search interface. Design Figure 5 shows the segmentation of study participants by search interface, assisted interface training, and logic training. The participants in cells I and II used the simple search Female Gender: 48.28% 51.72% Freshman Sophomore Junior Senior Class status: 87.59% 11.03% 0.69% 0.69% 6 mos 6 12 mos 1 3 yrs 4 5 yrs >5 yrs Online search exp: 4.83% 1.38% 36.55% 35.86% 21.38% Internet exp: 3.45% 2.76% 35.17% 37.93% 20.69% Procedure and Task The experimenter provided all of the treatment groups with a general introduction to the experimental stages to be completed. Then the participants were directed to the URL of the first stage. As participants completed a stage, they were individually directed to the URL for the subsequent stage. This process continued until each participant had completed all stages of the experiment. Participants receiving both types of training completed four stages in the following order: (1) interactive Web-based training in Boolean logic, (2) interactive Web-based assisted interface training, (3) experimental search system overview using PowerPoint slides, and (4) experimental task. Participants in the other treatment groups skipped stage 1, stage 2, or both, depending on the training they received. All participants completed stages 3 and 4. Boolean logic training. The logic training teaches participants about the basic Boolean operators (i.e., AND, OR, and AND NOT), how to group query terms into phrases, and how to form query clauses using parentheses. Participants complete several exercises in which they must form a Boolean query corresponding to a particular information request. Please see the Appendix for a description of this training. Assisted interface training. The interface training familiarizes participants with the technical components of the interface and how they work, and demonstrates how to form queries using the interface. The participants then complete 1188 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004

7 several exercises in which they are provided with textual queries that they must represent using the assisted interface. The Appendix also contains a description of the interface training process. Please note that the interface training is clearly different from the logic training because it focuses solely on the characteristics of the technical interface, whereas the logic training emphasizes how to express a conceptual idea of a query using Boolean logic. Experimental task. The experimental task used the experimental search environment previously described. Participants first completed a background questionnaire, followed by a manipulation check on Boolean operators for use in determining the effectiveness of the logic training. Those participants who would be using the assisted interface then completed another manipulation check measuring the effectiveness of interface training. All participants completed several more questionnaires on computer experience and attitudes before moving on to the information requests. Each participant was presented with six questions in random order, and was allotted a maximum of five minutes per question in which to search for and enter an answer. The queries required for each information request fall into three levels of complexity, based on their structural characteristics. These complexity categorizations were validated in a pilot study and have been used in an earlier study (Topi & Lucas, 2003). Those queries in the first category were the simplest, requiring the use of the conjunctive AND operator between two single search terms. The second category required AND operators for joining three terms in the first case and two terms and a phrase in the second case. The two queries in the third category were the most complex, requiring four or five terms or phrases and the use of the negation operator, NOT. In one of these cases, a more compact query could be formed by applying the NOT operator to a group of two terms joined with a disjunctive OR. The six information requests used in the experimental task, the optimal Boolean queries for finding answers to those request, and the complexity of those queries appear in Table 2. After entering an answer for an information request, each participant entered answers to four instrument items on confidence levels before being directed to the next information request. Those who reached the five-minute time limit were taken directly to the next request. When all six requests had been completed, participants filled in some final questionnaires, after which they were dismissed. Independent Variables Logic training. The first independent variable is logic training, that is, whether or not the participant received Web-based training on the use of fundamental Boolean constructors. Operators AND, OR, and NOT were covered in the training; in addition, the use of quotation marks to construct phrases and parentheses to group query structures together were reviewed. Logic training is described at a more detailed level in the earlier section on Procedure and Task. TABLE 2. Information requests, queries, and complexity. Information request and query Comp. R: Who is the chair of the philosophy department? Q: chair AND philosophy R: What former chair who established Bentley s English program died in Weymouth? 1 Q: chair AND Weymouth R: What faculty member of the management department whose first name is John attended Syracuse University? Q: John AND management AND Syracuse University R: From what institution did Professor Davis of the Management Department receive his BSEE? 2 Q: Davis AND Management AND BSEE R: What faculty member has an MA from Indian University, is not in the philosophy or management department, and is interested in politics? Q: Indiana University AND MA AND NOT (philosophy OR management) AND politics 3 R: What faculty member of the management department whose first name is Joseph but whose last name is not Byrnes taught and worked in the Middle East? Q: Joseph AND management AND NOTByrnes AND Middle East Interface training. The second independent variable, interface training, was also implemented on the Web as a series of screens that provided detailed step-by-step instructions on the use of the assisted search interface. Interface training is also described at a more detailed level in the Procedure and Task section. Search interface. Finally, the third independent variable is the search interface. As described above, participants used either the simple interface, consisting of only a text box, or an assisted interface containing user interface elements intended to help participants convert their conceptual query into a syntactically correct query implementation (see Figure 4). Dependent Variables Correctness. The most fundamental performance variable is correctness, i.e., the searcher s ability to provide a correct answer to an information request, which is used in this study as a measure of the quality of search results. All of the information requests were designed so that they had only one correct answer, and the results were evaluated to be correct or incorrect based on the match between the participant s answer and the correct one. Answers containing an error that clearly was typographical were evaluated as correct. A measure that aggregates individual correctness scores at the participant level is called score, which varies from zero to six (that is, the total number of information requests). Time. The system included automatic mechanisms for measuring the time the participants used for formulating the queries, viewing the results, and entering their chosen answer. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November

8 Each user action was time stamped, so that very detailed information about user behavior is available. In this analysis, the performance variable time is the sum of the times the participant used from first seeing the information request to entering an answer to that request. As noted earlier, the system imposed a five-minute time limit, gave a warning 30 seconds before that time was up, and moved the user automatically to the next question if the five-minute limit was exceeded. Satisfaction. An ease-of-use measure adapted from Venkatesh (2000) was used to guage participants satisfaction with the search environment. This scale demonstrated acceptable psychometric characteristics in a reliability analysis (Cronbach s alpha 0.937). Confidence. The participants confidence in their answers was measured with a one-item scale after each question. The item is the same at that used in the latest Text REtrieval Conference (TREC) studies (see projects/t9i/qforms.html). Results Manipulation Checks Two 2 2 ANOVA analyses were used to perform the manipulation checks, the purpose of which was to measure the effectiveness of the interface training and logic training manipulations. In the first analysis with participants using the assisted interface (because there was no interface training for the users of the simple interface), the independent variables were logic training (training vs. no training) and interface training (training vs. no training). The dependent variable was interface test, a five-item instrument developed for this research program that measured the participants ability to translate Boolean constructs into Web queries using the assisted interface. This analysis reveals a very strong main effect for interface training (F(1,93) , p 0.001) and a weaker main effect for logic training (F(1,93) 4.293, p 0.041). The main effect for logic training is not surprising because this training also exposes the participants to issues related to Boolean operators and increases the participants familiarity with them (the difference between the means was relatively modest, with a mean of 3.43 out of a maximum of 5 for those with training and 2.89 for those without it). The interface training effect is, however, significantly stronger and indicates that the interface training was effective in improving the participants performance in the use of the assisted interface (the means were 4.27 for the interface trained group and 2.06 for the group without interface training). The other manipulation check was also a 2 2 ANOVA test, with logic training and interface as independent variables and the pretest, which is a five-item instrument that has been designed for this research program to measure the participants ability to use Boolean logic constructs, as the dependent variable. In this test, as in later hypotheses testing, the analysis included users of the simple interface and those users of the assisted interface who received interface training. The results indicate a strong main effect for logic training (F(1,92) 31.55, p 0.001). Hypothesis Testing Whenever possible in the hypothesis testing, statistical analysis was performed at the individual information request level with the complexity category associated with each information request used as an additional independent variable. As will be seen, the explanatory role of complexity is essential because it improves the quality of the analyses even though it is not one of the independent variables used in the hypotheses (for a more detailed description of the complexity categories, please refer to the prior section on experimental task). The sample sizes and the means and standard deviations for the dependent variables score, time, ease-of-use (satisfaction), and confidence are shown in Table 3. Hypothesis 1 suggested that the interface training will improve the participants performance in the search tasks, as measured by the number of correct answers to the information requests. To test this hypothesis, a binomial logistic regression analysis was performed in which only the participants that utilized the assisted interface were included (because there was no interface training manipulation for the users of the simple interface, as noted earlier). This analysis was conducted at the individual information request level with the two training manipulations (Logic Training and TABLE 3. Numbers of participants, means, and standard deviations of dependent variables per treatment. Score Time Ease-of-Use Confidence Assisted interface N Mean SD Mean SD Mean SD Mean SD No logic tr. No interface tr No logic tr. Interface tr Logic tr. No interface tr Logic tr. Interface tr Simple interface No logic tr. No interface tr Logic tr. No interface tr JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004

9 Interface Training) and the complexity category of the information request (Complexity), along with all possible interactions between them, used as independent variables. Correctness was the dependent variable with binary values of Correct or Incorrect. The hierarchical analysis was started by entering complexity as a predictor. The resulting model was highly significant (x , 1 df, p 0.001) suggesting that Complexity is a strong predictor of Correctness. In the next step, Logic Training was added to the model and improved its predictive capability, with a change in x 2 of (1 df, p 0.003). A closer evaluation of cell frequencies suggests that Complexity and Logic Training each had a negative impact on Correctness. The third step added Interface Training, which resulted in a x 2 change of (1 df, p.267), showing that Interface Training did not influence the participants ability to find correct answers. In the last two steps, all interactions between logic training, interface training, and complexity were added; neither one of these steps improved the model s fit. Based on these results, Hypothesis 1 was not supported. Hypothesis 2 predicted that interface training will reduce the time required by participants using the assisted interface to complete the information search task. This hypothesis was tested using a ANOVA analysis, with Complexity, Logic Training, and Interface Training as independent variables and Time as the dependent variable. The main effect for Interface Training was only marginally significant (F(1,568) 2.771, p 0.097) and thus Hypothesis 2 was not supported. The rest of the analysis reveals that complexity had a strong main effect on Time (F(2,568) , p 0.000). There were no other significant effects. Hypothesis 3 predicted that participants using the assisted interface who have interface training will be more confident about their results than those without interface training. The results indicate that this was, indeed, the case. Interface Training had a main effect on Confidence in an ANOVA analysis with the same independent variables as above (F(1,504) 3.970, p 0.047), and an inspection of the means reveals that this effect was positive. Hypothesis 3 was therefore supported. In addition, both Complexity (F(2,504) , p 0.001) and Logic Training (F(1,504) 9.096, p 0.003) had main effects on Confidence, and both of these effects are negative (i.e., Logic Training and increased Complexity led to lower Confidence). The results concerning Logic Training are surprising and will be examined further in the Discussion section. There were no significant interaction effects. Hypothesis 4, the last one related to interface training, predicted that users of the assisted interface who receive interface training will be more satisfied with the search environment than those who do not. This hypothesis was tested using a 2 2 ANOVA, with Logic Training and Interface Training as independent variables and Ease-of-Use as the dependent variable. There were no significant differences and thus, Hypothesis 4 was not supported. Hypothesis 5a posited that logic training will lead to improved correctness of the search results for the users of the simple interface. Hypothesis 5b suggested that, in the absence of logic training, users of the assisted interface, both with and without logic training, will perform at a higher level of correctness than users of the simple interface who have not had logic training. These hypotheses were tested using data at the individual information request level, thereby requiring regression analysis. Furthermore, because the dependent variable, Correctness, is either right or wrong, analysis was performed using binomial logistic regression. The independent variables were Complexity, Logic Training, and Interface. The hierarchical analysis was started by entering Complexity as an independent variable. The resulting model was highly significant (x , 2 df, p 0.001), providing further support for the role of Complexity as a strong predictor of correctness. In the next step, Logic Training and Interface were added to the model. Given that the change in x 2 was (2 df, p ), these variables did not add to the predictive capability of the model. The third step added two interactions, Logic Training by Complexity and Interface by Complexity. The resulting change in x 2 of (4 df, p 0.697) indicates that these interaction effects were not significant. The fourth step added the most important interaction, i.e., Logic Training by Interface, because the hypotheses testing results will be based on it: If this interaction is significant, it suggests that the effects of logic training are different depending on the interface that is used. The x 2 change was (1 df, p 0.002) and thus, the interaction is highly significant. An inspection of the cell counts reveals that logic training benefited those who used the simple interface and was harmful for those who used the assisted interface. Inspection of the average results for the four Logic Training by Interface quadrants shows that the users of the assisted interface performed at the highest level but only if they did not receive logic training. With logic training, the correctness of the users of the simple interface is relatively close to the correctness of the assisted interface users who did not receive logic training. Therefore, Hypotheses 5a and 5b were supported, although the negative effect of logic training on participants using the assisted interface was not initially predicted. As a last step, the three-way interaction among Logic Training, Interface, and Complexity was added to the model and improved its fit significantly (x 2 change was 6.661, df 2, p ). This result signifies that there is further variation in the nature of the interaction between Logic Training and Interface based on the complexity category. Hypothesis 6 proposed that the users of the simple interface will feel more confident about their results if they have Logic Training before performing the tasks. This hypothesis was tested with a ANOVA at the information request level with Complexity, Logic Training, and Interface as independent variables, and Confidence as the dependent variable. The results of this analysis show a significant Complexity main effect (F(2,492) , p 0.001) and a significant Logic Training by Interface interaction effect (F(1,492) , p 0.001). An evaluation of the cell means reveals that logic training has a positive effect on the confidence of the participants using the simple interface and a negative effect on the participants using the assisted interface. Therefore, Hypothesis 6 was supported. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November

10 Finally, Hypothesis 7 suggested that the users of the simple interface will be more satisfied with the search environment if they receive logic training. This hypothesis was tested at the participant level with a 2 2 ANOVA with Logic Training and Interface as independent variables and Ease-of-Use as the dependent variable. Although an inspection of the cell means suggests a strong interaction effect, this is not statistically significant, and thus Hypothesis 7 is not supported. Summary of the Results One of the hypotheses related to interface training was supported: As Hypothesis 3 suggested, users of the assisted interface had more confidence in their search results if they received training in the use of the search interface. Hypotheses 5 and 6 were also supported, i.e., logic training had a positive effect on the search result correctness and confidence in search results among the users of the simple interface. An analysis of the experimental data also indicates that, for users of the assisted interface, logic training had a negative effect on correctness and confidence. Complexity was also found to have a negative effect on correctness, time, and confidence for all users, regardless of interface and training. Discussion First, it is important to point out that the results of this study support prior research (Topi & Lucas, 2003) suggesting that even relatively limited training in formal logic can improve human performance in information search tasks for users of an interface that does not provide assistance in the use of Boolean logic constructs. This is an important and highly relevant finding because most Web searchers rarely use advanced search features and limit their queries to a few search terms and even fewer operators (Spink et al., 2002). In addition, participants using the simple interface who had Boolean training were justifiably more confident in their search results. Although the characteristics of the interface were not the main focus of this study, our findings also support earlier evidence that a user interface providing assistance in formulating Boolean logic can lead to the same performance improvements as Boolean logic training. Our results suggest that relatively small, well-designed interventions (either user interface manipulations or training) can significantly improve the effectiveness of interfaces that require the direct use of Boolean constructs by end users. This is a much simpler alternative to the development of interfaces that offer increased support via NLP or graphical representations of query constructs. Although the correct use of Boolean operators may help searchers gain better results, many sites are following advice by Nielsen (1997), Petersen (2000), and others who suggest that primary search tools should not include Boolean search. Our results suggest that inexperienced users Boolean searches can be effectively supported with simple tools that do not necessarily require the use of natural language (Anick et al., 1990) or a visual representation of the Boolean structure (Young & Shneiderman, 1993). We do not have empirical evidence about the differences in effectiveness between these approaches; this has to be left for future research. We believe that our results provide justification for further research on the effectiveness of both search interface manipulations and training approaches and methods. Much of the recent research on Web search has focused on search strategies and other aspects of the broader search process (Brajnik et. al, 2002; Cothey, 2002; Dennis, Bruza, & McArthur, 2002; Sutcliffe, Ennis, & Watkinson, 2000). While this research is highly useful and has greatly improved our understanding of the factors affecting the ultimate success of the process of fulfilling information needs, we should not ignore the more narrow search process. Effects of training were not, however, the same in all contexts in this study. An unexpected result was that logic training had a clearly negative impact on the performance of users of the assisted interface; that is, those participants using the assisted interface and receiving logic training performed significantly worse than those using the same interface but not receiving the same training. While we can only speculate about the reasons underlying this phenomenon, there are several possibilities. The combination of the assisted interface and logic training could have led to conceptual confusion even though these manipulations have different goals: The logic training is intended to support the formulation of conceptual query constructs, whereas the purpose of the assisted interface is to help users translate a conceptual construct into the correct query implementation. A related possibility is that the combination of Boolean training and assisted interface usage caused information overload, i.e., too many cues within a short period of time. In addition, logic training is likely to have had a better fit with the simple interface than with the assisted one because the former allows participants to enter Boolean queries in exactly the same way as they appeared in the logic training, whereas the latter requires the use of multiple text boxes separated by interface-provided operators. Giving the logic training immediately before the task may have prompted the participants to use the assisted interface in a less effective way (e.g., by entering combinations of terms and operators as phrases). The negative impact of logic training on both satisfaction with the search process and on user confidence for assisted interface users, as opposed to the positive impact of logic training on confidence for simple interface users, provides some support for these suppositions. A major goal of this study was to investigate the effectiveness of and the need for training that focuses on the characteristics of the assisted interface. The purpose of interface training was to help the participants close the gap between the conceptual query and the practical query implementation by pointing out available mechanisms for building queries with the assisted interface. Our results suggest that, at least in this study, the ability to use the assisted search interface was not the roadblock that prevented users from getting correct results. This is indicated by the fact that interface training, which was shown to be successful by the manipulation 1192 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004

11 check, did not have any impact on correctness, time, or satisfaction with the search environment. Implicitly, this also suggests that the main benefit of the assisted interface was that it reminded the participants of the existence of the Boolean constructs and provided hints about the correct structure. Once they became actively aware of them because of the presence of the interface, they appeared to have been able to express their intent equally well (or poorly) with or without the interface training. Although interface training did have a positive impact on the level of the users confidence in the search results, the benefits of this perceptual change are questionable because they are not linked to a change in actual search performance. It is therefore necessary to search for other sources underlying the less than perfect observed search performance. First of all, as a background fact, it is important to remember that query complexity was a very important determinant of Correctness 82% of the simplest queries, 60% of the moderately complex queries, and only 33% of the complex queries were correct. The effects of all the factors discussed here were significantly moderated by query complexity. As indicated in our research model, human performance in Web search tasks is affected by the selection of query terms, the selection of query operators and their correct syntactic use, the understanding of the characteristics of the search engine, the searchers ability to understand the initial information request, and the ability to find a correct answer to an information request using the documents retrieved in response to submitted queries. In addition, results are affected by individual and contextual factors, including experience and domain knowledge (Hölscher & Strube, 2000). Implications This study and the factors that were manipulated focused primarily on the selection and correct syntactic use of query operators. Our results and prior research indicate that training in Boolean logic can help users of simple query interfaces improve their search results. This concurs with research supporting instruction in Boolean logic for helping users conduct effective searches (Bellardo, 1985). The role of the search engine, however, cannot be overlooked. Although Johnson and Szabo (1998) also found that training in both keyword selection and Boolean logic led to the correct usage of keywords and operators in constructing advanced search statements, they did not find a corresponding increase in searcher success when participants submitted queries to the Excite search engine. This can be attributed to the fact that Web search services seek to accommodate the typical search behaviors of their users, who rarely use Boolean operators. Adding advanced operators may therefore have little effect on query results (Jansen, 2000; Lucas & Topi, 2002), depending on how the search engines interpret queries. It is also important to note that in this study even those participants in the best-performing treatment groups were able to find only about 70% of the correct answers on average, and the worst groups performed below the 50% level. This finding is supported by a survey of online end users at the Ohio State University business library, which found the success rates for end-user searches to be quite low (Ankeny, 1991) and by a recent study that found the overall performance of medical students performing MEDLINE searches to be poor (Sutcliffe et al., 2000). Based on observations that will be investigated more formally in future research, we believe that our participants did not find the right answers because of one or more of the following factors: (1) incorrect or suboptimal search term usage, (2) inability to find answers that were available in the resulting document set, (3) misunderstanding of the information request, and (4) misunderstanding of the entire task, as demonstrated in a few cases by participants who entered a Boolean query rather than a fact into the answer box. Choosing the correct terms, patiently evaluating the resulting document set beyond the first few documents, knowing how to search for keywords within returned documents, and carefully evaluating the search task and its requirements are all essential for successful search performance. Limitations Our study participants were college students enrolled in a required introductory computer course and were therefore likely to have more experience, exposure to search engines, and comfort with searching than the general public. In addition, they represented a limited range of ages. Although the homogeneity of the participant population is a limitation from the perspective of external validity, it should improve internal validity, which was, as in almost all laboratory studies, the main concern of this research. The logic and assisted interface training participants received was of relatively short duration. Although our manipulation checks did show strong effects of training on test performance, it is possible that the amount of time devoted to the self-paced Web-based instructions was not sufficient for fully absorbing the information presented and that longer training could have had a stronger effect on performance. We would also like to point out that the main focus of this research was not on evaluating the user interface characteristics of a specific interface, but on evaluating how two different types of training in conjunction with the availability of certain user interface features affect the participants performance and attitudes. Therefore, a detailed evaluation of the assisted interface using methods of usability research was outside the scope of this study. Finally, to focus solely on Web queries, we created an artificial environment that prohibited users from the common information-gathering approach of combining searching with browsing (Catledge & Pitkow, 1995). This undoubtedly led to increased user frustration with both search interfaces. Conclusions and Future Work This study confirms that even a limited amount of logic training can lead to improved correctness of search results when a simple interface is used. The fact that an assisted JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November

12 interface can achieve an even higher level of correctness without any training is a significant result for search service providers. Also of interest are the findings that interface training with the assisted interface had no effect on performance and logic training actually hurt performance, perhaps by confusing the search participants. Although confidence in search results increased with assisted interface training, this is not a desired outcome because it was unaccompanied by a corresponding improvement in performance. Training manipulations and the choice of search interface were not the only determinants of performance. The complexity of the queries required to retrieve the desired documents had a strongly significant negative effect on performance and confidence. It was also postulated, and further research is expected to confirm, that search effectiveness depends on the searcher s ability to understand the information request, identify terms and phrases relevant to that request, understand the search task, and correctly interpret the results returned by the search engine. We therefore believe that future research on Web search must pay a great deal of attention to issues related to assisting the searcher in term selection, providing relevance feedback mechanisms, imparting meaning through the presentation of search results, and correcting for or preventing simple mistakes in elementary tasks. Acknowledgments This study was supported in part by a Bentley College Faculty Affairs Committee grant. We gratefully acknowledge this support. An earlier version of this article was included in the proceedings of the 2003 IEEE International Conference on Information Technology: Coding and Computing in Las Vegas, Nevada. We are thankful for the insightful comments by the editors and reviewers of all versions of this article. References Abdulla, G., Liu, B., & Fox, E.A. (1998). Searching the World-Wide Web: Implications from studying different user behavior. Paper presented at the WebNet98 Conference, Orlando, FL. Allen, B. (2000). Individual differences and the conundrums of usercentered design: Two experiments. Journal of the American Society for Information Science, 51(6), Anick, P.G., Brennan, J.D., Flynn, R.A., Hanssen, D.R., Alvey, B., & Robbins, J.M. (1990). A direct manipulation interface for Boolean information retrieval via natural language query. Paper presented at the 13th International ACM SIGIR Conference on Research and Development in Information Retrieval, Brussels, Belgium. Ankeny, M.L. (1991). Evaluating end-user services. Journal of Academic Librarianship, 16(6), Bellardo, T. (1985). An investigation of online searcher traits and their relationship to search outcome. Journal of the American Society for Information Science, 36, Borgman, C.L. (1986). Why are online catalogs hard to use? Lessons learned from information retrieval studies. Journal of the American Society for Information Science, 37(6), Brajnik, G., Mizzaro, S., Tasso, C., & Venuti, F. (2002). Strategic help in user interfaces for information retrieval. Journal of the American Society for Information Science and Technology, 53(5), Buckley, C., Salton, G., Allan, J., & Singhal, A. (1995). Automatic query expansion using SMART: TREC 3. Paper presented at the NIST Special Publication : The Third Text REtrieval Conference (TREC-3). Catledge, L.D., & Pitkow, J.E. (1995). Characterizing browsing strategies in the World Wide Web. Paper presented at the Third International World Wide Web Conference, Darmstadt, Germany. Chan, H.C., Tan, B.C.Y., & Wei, K.K. (1999). Three important determinants of user performance for database retrieval. International Journal of Human-Computer Studies, 51(5), Cothey, V. (2002). A longitudinal study of World Wide Web users information-searching behavior. Journal of the American Society for Information Science and Technology, 53(2), Dennis, S., Bruza, P., & McArthur, R. (2002). Web searching: A processoriented experimental study of three interactive search paradigms. Journal of the American Society for Information Science and Technology, 53(2), Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2002). Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1), Ford, N., Miller, D., & Moss, N. (2001). The role of individual differences in Internet searching: An empirical study. Journal of the American Society for Information Science and Technology, 52(12), Harman, D. (1992). Relevance feedback revisited. Paper presented at the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Hersh, W., Turpin, A., Price, S., & Kraemer, D. (2001). Challenging conventional assumptions of automated information retrieval with real users: Boolean searching and batch retrieval evaluations. Information Processing & Management, 37(3), Hölscher, C., & Strube, G. (2000). Web search behavior of Internet experts and newbies. The International Journal of Computer and Telecommunications Networking, 33(1 6), Jansen, B.J. (2000). The effect of query complexity on Web searching results. Information Research, 6(1). Retrieved from net/ir/6-1/paper87.html Jansen, B.J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: A study and analysis of user queries on the Web. Information Processing & Management, 36(2), Johnson, B.L., & Szabo, M. (1998, June 20 25). Instruction, gender, search success, and search satisfaction on the World Wide Web: A research study. Paper presented at the World Conference on Multimedia and Hypermedia Edmedia/Edtelecom, Freiburg, Germany. Kim, K.-S., & Allen, B. (2002). Cognitive and task influences on Web searching behavior. Journal of the American Society for Information Science and Technology, 53(2), Kwok, C., Etzioni, O., & Weld, D.S. (2001). Scaling question answering to the Web. ACM Transactions on Information Systems, 19(3). Lucas, W. (2001). Search engines, relevancy, and the World Wide Web. In A.G. Chin (Ed.), Text Databases and Document Management: Theory and Practice (pp ). Hershey, PA: Idea Group Publishing. Lucas, W., & Topi, H. (2002). Form and function: The impact of query term and operator usage on Web search results. Journal of the American Society for Information Science and Technology, 53(2), Nielsen, J. (1997). Search and you may find. Retrieved June 6, 2003, from Petersen, C. (2000). Simplify & sort for better searches. Retrieved June 6, 2003, from Pollock, A., & Hockley, A. (1997, March). What s wrong with Internet searching. D-lib Magazine. Salton, G., & Buckley, C. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4), Silverstein, C., Henzinger, M., Marais, J., & Moricz, M. (1999). Analysis of a very large Web search engine query log. SIGIR Forum, 33(1), Spink, A., Jansen, B.J., Wolfram, D., & Saracevic, T. (2002). From e-sex to e-commerce: Web search changes. IEEE Computer, 35(3), JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004

13 Spink, A., Wolfram, D., Jansen, B.J., & Saracevic, T. (2001). Searching the Web: The public and their queries. Journal of the American Society for Information Science, 53(2), Sutcliffe, A., Ennis, M., & Watkinson, S.J. (2000). Empirical studies of enduser information searching. Journal of the American Society for Information Science, 51(13), Te eni, D., & Feldman, R. (2001). Performance and satisfaction in adaptive Web sites: A laboratory experiment on search tasks within a task-adapted Web site. Journal of AIS, 2(3). Topi, H., & Lucas, W. (2003). Searching the Web: Operator assistance required. Information Processing and Management (forthcoming; available electronically in final form from ScienceDirect). Turtle, H. (1994). Natural language vs. Boolean query evaluation: A comparison of retrieval performance. Paper presented at the SIGIR 94. Venkatesh, V. (2000). Determinants of perceived ease of use: Integrating control, intrinsic motivation, and emotion into the technology acceptance model. Information Systems Research, 11(4), Young, D., & Shneiderman, B. (1993). A graphical filter/flow representation of Boolean queries: A prototype implementation and evaluation. Journal of the American Society for Information Science, 44(6), Appendix: Logic and Interface Training Logic Training The purpose of the logic training is to familiarize the study participants with the basic Boolean operators; the concepts of conjunction, disjunction, and negation that relate to their use; the use of quotation marks for forming phrases; and the use parentheses to group query clauses. The trainee is first guided through several exercises focusing on the principles of Boolean logic. In these exercises, the trainee is shown a sample information request and the corresponding Boolean query. Then, the trainee is prompted to form a query for another information request that is similar in structure to the sample one (see Figures A1 and A3). After pressing the submit button, the correct Boolean query is shown along with the one entered by the trainee (see Figures A2 and A4). After completing the practice exercises on the AND, OR, and AND NOT operators, as well as on the usage of parentheses and quotation marks within queries, the trainee is presented with a series of information requests and prompted to enter Boolean queries for each. These final queries serve as a review of all the information presented in the training sessions. As in the individual exercises, the trainee is shown the ideal search string and the string that was entered for each information request. Interface Training The purpose of the interface training is to familiarize the study participants with the look and feel of the interface and help them understand the technical functionality of the interface. The participants are first shown screen shots of the interface, which highlight the fields and buttons for entering terms and phrases, selecting operators, adding parenthesis, and submitting a query. The screen shots shown during training that describe these operations have been incorporated into the one view shown in Figure A5. Please note that the FIG. A1. Training in the use of the AND operator. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November

14 FIG. A2. Solution to AND operator practice exercise. FIG. A3. Training in the use of parentheses. FIG. A4. Solution to parentheses operator practice exercise JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004

15 FIG. A5. Overview of assisted interface components. actual training consists of multiple slides, with each focusing on one interface characteristic. The next step in the training is a review of the basic rules concerning interface usage. Figure A6 contains the information from the slides shown during training about how to form phrases and choose the AND operator. Figure A7 describes how to select the OR and AND NOT operators, as well as parenthesis for applying conditions to multiple terms and phrases. The preceding example would send this query to the search engine: Southwest Airlines AND Herbert K. Kelleher AND retirement AND year The preceding example would send this query to the search engine: airline AND NOT (Northeast OR Transatlantic) In the last phase of the training, the participant interacts with the assisted interface to form queries matching the FIG. A6. Summary of default usage rules. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November

16 textual examples shown in a total of five exercises. One of these exercises, which is representative of the other four, is shown in Figure A8. The participant is shown a textual query and asked to form that query using the assisted interface. After pressing the submit button, the participant can see if the formed query, as translated by the interface, matches the original textual one. If it does not, the participant can reform and resubmit the query until it has been entered correctly. FIG. A7. Summary of operator and parenthesis usage rules. FIG. A8. Sample training exercise JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2004