Effects of Advanced Search on User Performance and Search Efforts: A Case Study with Three Digital Libraries

2012 45th Hawaii International Conference on System Sciences Effects of Advanced Search on User Performance and Search Efforts: A Case Study with Three Digital Libraries Xiangmin Zhang Wayne State University xiangmin.zhang2@wayne.edu Yuelin Li Nankai University yuelinli@nankai.edu.cn Abstract This study investigated into the effects of the Advanced search feature of three digital libraries on users search performance and search efforts. Three operational digital libraries, i.e., the ACM digital library, the IEEE Computer Society digital library, and the IEEE Xplore digital library, were used in this study. Thirty-five students participated in an experiment and completed the assigned search task, using their preferred search feature(s). The results demonstrate that for ACM and IEEE CS, the use of Advanced search did not have a significant effect on improving search performance. Only in Xplore, the use of Advanced search significantly improved search performance. The search efforts increased significantly for the combined-use of Advanced search and Basic search in ACM and IEEE CS. The reasons leading to the results are discussed. 1. Introduction Interaction design for search systems, including digital libraries (DLs), has drawn much attention in recent years. It is a norm for today s websites in general and search systems in particular to provide at least two search buttons on the user interface: Basic or Simple search and Advanced search. Basic search does not require the user to be knowledgeable about the system and the search process. It enables the user to conduct a quick search but restricts the user to simple keyword searches. It is supposed to be easy to understand and to use by novice users. Advanced search, on the other hand requires more searching knowledge and skills from the user. Although many websites, and nearly all search systems provide these two search features, what the user can do with them differ from system to system. Despite the popularity of this Basic search Advanced search paradigm in user interface, there is a lack of understanding of how these features are actually being used and what would be the effects of using them. This paper reports a study that investigated the effects of Advanced search in DLs on the user s search performance and search efforts. The study used three operational DLs: the Association for Computing Machinery digital library (ACM), the Institute of Electric and Electronic Engineering Computer Society digital library (IEEE CS), and the Institute of Electric and Electronic Engineering Xplore digital library (Xplore). The three DLs presented different search interaction designs, providing an excellent opportunity to explore user experiences in different system context. Table 1 summarizes the differences of search features among the three DLs. As Table 1 shows, the major difference is the search method, full-text vs fielded search. These two search methods can typically be found in DLs [16]. All three DLs supported both fielded search and full-text search. However, the three DLs had different interface designs for the search function. Both ACM and IEEE CS used full-text search as the default search method for both Basic search and Advanced search. In ACM, if a user needed to conduct a fielded search, the user had to manually code fields in the search query in the Advanced search page. Fielded search was very limited in IEEE CS: it provided only two (author and title) fields in Simple search and four (author, title, date, ISBN) fields in Advanced search. However, Xplore used fielded search as the default search method for both Basic and Advanced search. Only in Advanced search the user could choose fulltext search. It would be interesting to see if users would exhibit different usage patterns with these different designs. Our purpose was to find out how users would use Advanced search in these systems. Specifically, we addressed the following questions: How and to what extent do users use different search features (Basic search and Advanced search) in different DLs? How does the use of Advanced search affect the user s performance? Does the use of Advanced search cost more search effort compared to the use of Basic search? 98-0-695-4525-/12 $26.00 2012 IEEE DOI 10.1109/HICSS.2012.234 1605

Basic search Advanc ed search System default Default Boolean operator User search options System default Default Boolean operator User search options Table 1. Major differences of search interaction design in three DLs * Fields have to be manually coded in full-text mode, using special syntax. ** Although the Help says and, it actually performs or. 2. Literature review Dependin g on document type Title, abstract, review Title, autho r ACM IEEE CS Xplore Full-text/ Fulltext Fielded Fielded* OR OR** AND Fulltext/Fielde d OR Fulltext NON E Title, autho r, date, ISBN / ISSN Title, author, source title, and other metadat a fields Fielded AND All fields & fulltext, title, author, source title, abstract, index field Hearst et al. [1] pointed out that some features in search user interface were found working well but others did not across studies. They developed a browse-and-search interface framework which was featured by presenting the hierarchically faceted metadata to guide users toward possible choices and to organize the results of keyword searches. This interface provided an easy way to assist users to formulate queries with ANDs or ORs. Belkin et al. [2] examined whether a query-entry line or a scrollable query-entry box was more effective in eliciting longer queries. Belkin et al. [3] presented the findings that users input significantly longer queries when they were asked to do so. Nevertheless, the difference in the length of queries in these two studies did not lead to significant differences in search performance and user satisfaction. Hearst [4] emphasized the importance of small details for interface design, such as the size of search box, spelling suggestions, and so on. These details can deeply affect users information-seeking behavior. The above-mentioned works are certainly helpful to understand the context of the design issues for search features. However, they did not address specifically the issues we intended to investigate in this study. Some usability evaluations of DLs did touch search features at a general level. A consensus seems to be that users prefer to have both the Basic and the Advanced search features. Xie [5] found that having both Basic search and Advanced search ( multiple search mode ) was one of the user s requirements for a search system. Shiri and Molberg [6] found that Basic search could not well support more precise and complicated search queries. Hartson, Shivakumar, & Pérez-Quinones [] report an inspection on the usability of NCSTRL. Part of the usability inspection was the graphic design of the Advanced search button. The inspection revealed various problems with the design of the Advanced search button in the system investigated. The study, however, did not test the use of this feature. Though these studies pinpointed problematic designs in search features of DLs and suggested how to improve the design, they did not touch upon how these different designs would impact users search performance and efforts from the user s perspective. None of the studies reviewed so far has provided empirical evidence on the use of search features. Use of Advanced search might be reflected by the structure of the queries the user used. Complex queries that are multiple words, including Boolean operators and some other advanced features are more likely constructed by using Advanced search. Jansen, Spink & Saracevic [8] found that among the total queries examined, only about 24% were complex queries that used the Boolean operators, +/- operators, phrase (quotation marks) and parenthesis, implying a very low use of Advanced search among the Web search engine users. Jansen [9] found that averagely the difference between the results from simple queries and from complex queries was not large. The complex queries made only 2. (among the first 10) new results. The author then questioned if it was worth learning and utilizing the more complex searching operators. These studies suggest that simple search or Basic search is far more popular than Advanced search for general users. This may be because users may not effectively use the operators in the search [10] or Advanced search cannot 1606

help too much, as Ruthven [11] pointed out that such interfaces could only help cutting down the number of returned results rather than generating new ideas for the searchers to formulate new queries. Topi and Lucas [12,13] found that although in general the interface supporting advanced search operators could improve search queries, the improvement was not consistent. Also, most Web searchers do not have any professional training on searching. They may not really understand the meaning of the operators. Unfriendly interface and the requirement of know-how may be other reasons that made the advanced search operators not as effective as expected [14]. Literature review indicates that most current studies are concerned with the usage of advanced search techniques (such as complex Boolean operators) rather than with the use of Advanced search available on a system s user interface. We think the use of Advanced search is an important issue and the investigation into the issue should provide valuable input for the design of interfaces for DLs and other web-based systems. 3. Methodology The three operational DLs, i.e., ACM, IEEE CS, and Xplore presented different styles of interface design. Although the focus of content differs among the three, they all cover, more or less the field of computer science. This overlap on computer science literature was another reason that the three DLs were selected: they made a comparison among the three on the same ground possible. ACM provides citation and full text articles from ACM journals, magazines, transactions, and proceedings sponsored by ACM, covering various areas of information technologies, including computer science. IEEE CS DL provides a one-step access mainly to 15 society magazines, 9 transactions, and about 51 conference proceedings in computer science. Xplore, as a collaboration between the IEEE in the US and the IEE in UK, provides access to over 100 periodicals and 4149 conference proceedings, covering approximately 40 specialties relating to electrical engineering, computing sciences and their various applications. They all provide both Basic or Simple search and Advanced search. In addition to Basic search and Advanced search, there are also other search features, such as Author search, Proceedings search, CrossRef, and so on. We focus on Basic search and Advanced search in this study. 3.1. Participants Thirty-five students from a large state university participated in the study. Among them 12 were undergraduate engineering or computer science (UE) students, 12 graduate engineering or computer science (GE) students, and 11 masters of library and information science graduate (LIS) students. These participants were considered as the end users of the DLs investigated because the computer science and engineering students were part of the targeted users on campus, and the LIS students would be librarians serving as intermediaries helping users use these DLs. The participants as a whole were a very computer literate group, with over 94% self-reported above the medium level (scale 4) of computer experience on a - point scale, and nearly 50% considered themselves experts (rating of 6 or ) in using computers. They were also experienced users of search engines. About 83% considered themselves very experienced with searching on the internet. In contrast to their experience with internet search engines, the majority (23 out of 35, 66%) of the participants did not have any experience with ACM and Xplore. An even higher percentage of participants, 80% (28 out of 35), never used IEEE CS. In general the participants as a whole were new or infrequent users of the three DLs. 3.2. User tasks Two information seeking tasks were used in the study: one search task and one browsing task. The two tasks represent the two major interaction activities any DL is supposed to support. This paper focuses on the results of the search task. Ideally, different types of search tasks should be used in a study to represent different searching context. However, in reality, there are always constraints on the number of tasks to be used in a user experiment. We initially designed 4 tasks for this research. Our pilot study, however, proved the participants could only complete 2 of them on each of the three DLs in about 2 and half to 3 hours. Beyond that timeline, the participants simply could not do more due to the physical exhaustiveness. Therefore, the number of systems tested in this study limited the number of user tasks to only one search task. We recognize the data collected through this only search task is very limited in reflecting real world user experience. The search task was a typical topic search task: it required the participants to conduct a search on each of the three systems for relevant information about protecting the on-line repository from fraudulent activity by watermarking. The participants could use any search feature(s), in any order, as they wanted. As part of the search task, 160

participants were asked to save the top ten ranked retrieved documents from the search result list with which they were most satisfied (they might have multiple search sessions that generated more than one results list), and to make relevance judgment (in relation to the search topic) on these saved search results. The relevance judgment was made on a 3-point scale: relevant, partially relevant, and not relevant. 3.3. Variables 3.3.1. Independent variables. The independent variable in the study was the use of the search features with each of the three DLs. Given the two search features: Basic search and Advanced search, this variable described how participants used these features to accomplish the search task. The number of participants who used a particular feature was recorded. 3.3.2. Dependent variables. There were two sets of dependent variables: the user s search performance and the user s search efforts. Search performance. This variable was a description of the user s (and the system s) ability to find relevant items from each of the systems. Average Precision (AP) [15] was used as the specific measure. AP is the mean of the precision obtained after each relevant document is retrieved by a search query: Ra Average Precision = Pi Ra i= 1 where Pi is the precision of the relevant document at the ith position in the ranked list; Ra is the number of relevant documents retrieved by the query. This measure takes into account of the number of the relevant documents retrieved and the ranking of these relevant documents in the result list. We chose to use this measure because for an effective search, it is important that the participant should not only find relevant documents but also be able to use a query that can have the relevant documents ranked high in the result list. Search efforts. The user s search efforts included the following measures: Average query length: The mean number of words in all queries issued by a participant. The number of queries issued: The number of queries a participant submitted to the system for completing the search task., Amount of search time: The amount of time a participant worked with a system in order to get the satisfactory results. It was measured as the time period starting from the participant submitting the first query and ending at copying the first ten results. The number of search steps: The steps the participants took before they finally obtained the results (i.e. copying the first ten results from the screen), including choosing a search feature, inputting a query, changing search fields, clicking Back and Forward in the browser for navigation between different pages, clicking the button for submitting a query, or using the Enter key, and so on. Intuitively, entering more words in a query and/or submitting more search queries would need more efforts, at least physically. The more search steps the user needs to go through the more mental decisions the user has to make, the more effort is needed. A short search time means the user found the needed information quickly in fewer query sessions, if not a single search session, without much effort. However, longer search time would indicate that the user has either to conduct multiple search sessions, or working hard to work out a good query that can find the needed information. In either case, more search efforts the user would be making. Our hypothesis regarding the relationship between the use of Advanced search and search performance was that the Advanced search feature would enable the user to be able to more effectively retrieve relevant documents. Therefore, this use would increase average precision. This hypothesis was tested in the experiment. For the relationships between the independent variable and different search effort variables/measures, we did not have hypotheses because the effect could be either increasing or decreasing efforts. For example, the use of Advanced search could either increase search time because its use would involve more choices or options that would take more time. Or the use could decrease search time because the search would be more effective and the task could be completed with fewer query sessions. We intended to reveal the possible relationships between these two sets of the variables. 3.4. Experimental design We used a blocked Latin square design to randomize the system and the task order to avoid possible bias. There were 12 orders of searching and browsing tasks combined with the system order. The 1608

orders were assigned to the participants that there would be three participants who would follow the same order, among all the participants. 3.5. Procedure The participants were invited individually to a usability laboratory on campus to take part in the study. Upon arrival, the participant would first be asked to sign the consent form and to fill out the user background questionnaire. A brief instruction session then followed to inform the participant of the tasks that needed to be completed. All the participants were asked to perform the same tasks. The user interactions with the system were recorded by the Morae software. A post-task questionnaire was administered to collect information regarding the participant s interaction experience with the systems. While the whole experimental session was limited to two and half hours (including the browsing task), no time limit was set to a particular search/system session. 4. Results 4.1. Use of Advanced search Participants could use any one of the search features available to accomplish their search task. One participant did not complete the task with ACM. Therefore, the total number of participants for ACM was 34, rather than 35 as for the other two DLs. There appeared to have three distinctive ways in using the search features: 1). Basic: the participants used only Basic search; 2). Advanced: the participants used only Advanced search; and 3). Combined: the participants used both Basic search and Advanced search. This combined use could be Basic search first then switching to Advanced search or vice versa. Figure 1 describes the participants use of the search features. The use of Advanced search differed from system to system. In general, the uses in ACM and IEEE CS exhibited a similar pattern: only a low percentage of participants used Advanced search, much less than those who used Basic search. In ACM, 26% (9 out of 34) used Advanced search and in IEEE CS, 20% ( out of 35). In contrast, the pattern for Xplore was different: a relatively high number of the participants (3% (13 out of 35)) used Advanced search, more than those who used Basic search. # of Users 18 16 14 12 10 8 6 4 2 0 16 9 9 15 ACM IEEE CS Xplore Figure 1. Use of search features in three DLs 13 13 15 Basic Advanced Combined Regarding the combined use of Basic and Advanced search, more participants started their task by using Basic search. Specifically, 56% (5 out of 9) in ACM, 9% (12 out of 15) in Xplore, and 62% (8 out of 13) in IEEE CS started with Basic search first, they then switched to Advanced search. It seemed that Basic search in general was the start point of the participants when searching in digital libraries. If not successful, they would switch to Advanced search. 4.2. Effects of the use of search features on search performance The results of the search outcome from the three systems are depicted by Figure 2. In the figure, there are two mean AP scores for each system: one is the general AP (indicated by AP ) including both relevant and partially relevant judgments and the other one (indicated by Hard AP ) including only relevant judgment, without partially relevant ones. Mixed results can be observed for ACM and IEEE CS. Descriptively the scores from the Advanced group were higher than that from the Basic group in the general AP case. In the hard AP case, the use of Advanced search actually decreased the performance slightly in both DLs. The combined-use increased performance slightly in ACM, but decreased performance from Basic search in IEEE CS. The combined-use did increase performance a little from the Advanced group in IEEE CS. For Xplore, a pattern seemed to be clear in both AP and hard AP cases: The use of Advanced search dramatically increased performance from the Basic search case, and the combined-use increased performance as well. Our hypothesis was that the use of Advanced search should result in higher AP scores. To test this hypothesis, for each system, the scores from different 1609

use cases were compared using the one way ANOVA procedure. The results of ANOVA analyses did not support this hypothesis for ACM and IEEE CS, on both AP and hard AP scores. The changes in the performance measure in ACM and IEEE CS were not significant. Nevertheless, this hypothesis was supported for Xplore. Significant differences were found among the three use scenarios for both general AP (F(2, 32)=3.399, p<.05) and hard AP (F(2, 32)=.321, p<.01) scores. The Post Hoc Tukey tests found that for general APs, the Advanced case s scores were significantly higher (borderline, p=.068) than that of the Basic case. No significant differences were found between the Basic and the combined-use, and between the Advanced and the combined-use. For hard AP scores, the difference was even more significant: the Advanced case was significantly higher than that of the Basic (p=0.003) and than that of the combined-use (p=0.036). There was no difference between the Basic and the combined-use. M ean Averag e Precision 1 0.8 0.6 0.4 0.2 0 Overall AP Hard AP Overall AP Hard AP Overall AP Hard AP ACM IEEE CS Xplore Baisc 0.65 0.4 0.5 0.6 0.69 0.42 Advanced 0.69 0.45 0.91 0.4 0.93 0.4 Combined 0. 0.4 0.1 0.53 0.1 0.56 Baisc Advanced Combined Figure 2. Effects on search performance in the three DLs The mixed results indicate that the use of Advanced search increased search performance for certain DLs but not for others. 4.3. Effects of the use of search features on search efforts Table 2 presents the mean values of the search effort measures for the three DLs. Since the use of the search features showed different patterns among the three DLs, the results of data analysis are presented in the following subsections for each of the three DLs. The number of queries. Significant differences were found with the combined group on the number of queries issued (F(2, 31)=11.51, p<.0001). The combined group issued significantly more queries than the Basic group (p=.000) and than the Advanced group (p=.000). No significant difference was found between the Basic group and the Advanced group. Search time. There were significant differences among the three groups in terms of search time (F(2,31)=3.35, p<.05). The combined group spent significantly more time than the Basic group (p=0.000) and than the Advanced group (p=0.000). There was no significant difference between the Basic and the Advanced group. It is noticed that for ACM, the participants who used the combined mode spent much more time than others who used only one search feature, either Basic search or Advanced search. The amount of time used by the combined group was about four times of that used by the Basic search group and by the Advanced group alone. This was quite different from results for the other two DLs. It may be related to the significantly more queries (also about 4 times more) issued when using both search features. Using both Basic search and Advanced search in ACM seemed having increased search effort dramatically. The reason might be that the differences between Basic search and Advanced search in interface design in ACM were significant that the user would need to make more efforts to get used from one to the other. Several participants clicked on search tips for help. In contrast, the other two DLs designs were relatively concise, which might have saved participants much time to figure out. Search steps. There were also significant differences among the three groups in terms of search steps (F(2, 31)=14.65, p<.0001). The combined group made significantly more steps than the Basic group (p=0.000) and than the Advanced group (p=0.001). There was no significant difference between the Basic and the Advanced group. The use of Advanced search itself did not seem to put much more burden to the user on search. However, when the use of Advanced search was combined with the use of Basic search, the combined-use did not increase the search performance but greatly increased user efforts in ACM. This result was consistent with the results on the number of queries and the amount of search time. 4.3.1. ACM 1610

ACM IEEE CS Xplore B A C B A C B A C Average query length Mean # of queries issued Mean search time (Seconds) Mean # of search steps 3.51 4.25 3.36 3.83 2.90 3.04 1.1 2.55 3.32 1.44 2.22.8 1.33 2.5 4.08 3.43 6.08.4 135.19 152.33 663.33 120.93 205.86 39.92 164.00 339.15 394.6 3.88 10.11 2.8 4.3 11.5 20.84 11.14 25.23 26.53 Table 2. Mean values of user efforts in three DLs (Notes: B = Basic search; A = Advanced search; C = Combined-use) 4.4. IEEE CS 4.4.1. The number of queries. In terms of the number of queries, there were significant differences among the three groups (F(2, 32)=13.11, p<.0001). The Tukey test found that the combined group had significantly more queries than the Basic group (p<.0001) and than the Advanced group (borderline, p=.05). 4.4.2. Search time. The differences were also significant for search time (F(2, 32)=.3, p=.002). The Tukey test found that the combined group spent more time than the Basic group (p=.002). 4.4.3. Search steps. There were also significant differences in terms of search steps (F(2, 32)=13.42, p=.000) among the three groups. The combined group had more steps than the basic group (p=.000) and than the Advanced group (p=.056, borderline). Again, the results for IEEE CS showed that the combined-use significantly increased the user s search efforts. However, this increase was less than the ACM case. 4.5. Xplore The results for Xplore were different from that for ACM and IEEE CS. The only significant difference detected was among the three groups in query length (F(2, 32)=.9, p=.002), which was not found in both ACM and IEEE CS. The Tukey test found that the combined-use had significantly longer queries than the Basic case (p=.001) and than the Advanced case (borderline, p=.08). No significant differences were found on other measures. Descriptively, the results of the combined used were similar to those of Advanced search. In all the three DLs, the use of Advanced search itself did not have significant effects on search efforts. However, the combined-use, which included Advanced search, showed significant effects on search efforts on all the measures in ACM and IEEE CS and on one of the measures in Xplore. 5. Discussion 5.1. How was Advanced search used? The data showed that for all the three DLs, over 50% participants used Advanced search, either exclusively or in part. This result is different from the results reported in [9] and [10]: low percentage of complex queries using advanced searching techniques by Web search engine users. Considering that some of the complex queries may be constructed by using just Simple search, the potential use of Advanced search on the Web could be even lower. The difference between the result from the current study and from the previous two studies may be attributed to the growth of search knowledge and skills among general public users. When the previous studies were conducted, Web searching was still in its early stages. Searching was still a skill needing training. In the current study, however, as mentioned in Section 3.1, most participants considered themselves very experienced search engine users, whose search knowledge and skills certainly enabled them to use more advanced features of search systems. This result may also reflect the difference between Web search and the searches on commercial databases and digital libraries. For database searches, the result confirmed the findings from previous studies that the users desired both Basic and Advanced search on the interface (e.g., [6]). This high percentage of use found in the current study should validate the presence of Advanced search on 1611

the user interface of DLs. However, it is noticed that more users started from Basic search, rather than from Advanced search. This is reasonable because most users did not have any formal search training and did not understand why and how Advanced search functioned. This result also suggests that for digital libraries it is necessary to enhance the search capability of Basic search, which may be an approach to improving usability of DLs. The use of Advanced search, nevertheless, seemed to be related to specific DLs, or a type of DLs. The results showed two different patterns. ACM and IEEE CS represented one pattern and Xplore represented another pattern. In the ACM and IEEE CS case, the use of Advanced search was less than the use of Basic search. While the use of Advanced search significantly increased search performance in Xplore, it did not have significant effects in ACM and IEEE CS. The results may be due to the different search methods, coupled with the default Boolean operators, employed by the three DLs, as described in the Introduction section. It seemed that Basic search in Xplore strictly limited the user s capabilities of constructing effective queries. The limitations included no full-text search, only one text entry box for search term(s), and the default AND Boolean operator. The Advanced search feature, however, provided the user with the opportunity to search in full-text, and a user interface that could construct more accurate and complex queries, which in turn increased the possibility to retrieve relevant items. The availability of full-text search was a substantial addition to Basic search. In ACM and IEEE CS, there was a lack of a substantial change from Basic search to Advanced search in terms of search method. 5.2. Effects on search performance Only in one DL the use of Advanced search showed significant improvement on search performance. The effects of use seemed to be related to the search method employed by this DL. For the full-text search systems, Advanced search did not really provide more functions than Basic search already did, as far as the query construction is concerned. That perhaps was the reason why the use of Advanced search in ACM and IEEE CS did not have significant effects on search performance. This result is similar to the findings reported in [9], in which the use of advanced searching techniques in 5 Web search engines did not make a big difference from the Basic search in terms of the search results. Advanced search in Xplore, on the other hand, provided quite different result from Basic search. Advanced search in Xplore provided full-text search that was not available in Basic search. It also provided more fields for entering multiple query terms, connected by explicitly selected Boolean operators, which was impossible in Basic search. These additions dramatically enhanced the user s ability to construct effective queries. This may explain why the query length was significantly longer when using Advanced search in Xplore than using Basic search. The capability of doing full-text search and being able to construct more detailed queries apparently were the reasons that the use of Advanced search increased search performance significantly. 5.3. Effects on search efforts The effects on search efforts were also different between the two types of DLs. Although the use of Advanced search itself did not increase search efforts, the combined-use did increase search efforts significantly for the full-text search systems. For both ACM and IEEE CS, the combined-use significantly increased the number of queries issued, the search time, and the number of steps. Since the use of Advanced search itself did not have an effect, we suspected that the effects might come from the switch from one search feature to another feature, which cost the user time and steps to adapt from one to another. This was particularly obvious in the ACM case. For Xplore, or fielded search systems, on the three key measures of search efforts: the number of queries issued, the number of steps to get search results, and the amount of time spent on a search, there were no effects at all. Advanced search was definitely a big plus for the users of this DL: its use increased search performance significantly without costing extra search efforts. This contrasted the effects with ACM and IEEE CS, which used the full-text search method as the default. 5.4. Differences between/among different types of users Possible differences between/among different types of users were also investigated. The use of search features by the three types of participants is depicted in Figure 3. In general, the LIS participants used the basic search least across the three DLs. The UE participants tended to rely on the basic search. This difference may reflect the fact that the LIS participants were more knowledgeable about search and the Advanced search feature. 1612

# of users 12 10 8 6 4 2 0 6 2 4 3 2 3 4 4 4 1 6 4 2 2 2 UE GE LIS UE GE LIS UE GE LIS ACM IEEE CS Xplore 6 1 5 1 4 0 8 3 Baisc Advanced Combined Figure 3. Use of search features by different types of users ACM IEEE CS Xplore Hard AP Hard AP Hard AP AP AP AP LIS 0.5 0.84 0.54 0.80 0.61 0.9 GE 0.31 0.59 0.40 0.3 0.55 0.9 UE 0.30 0.5 0.53 0.1 0.4 0.1 Table 3. Different groups of participants search performance in the three DLs In terms of search performance and search efforts, no statistically significant differences were found between/among the three groups of users across all the three DLs. Descriptively, the LIS students had better search performance in terms of AP scores. Table 3 presents the AP scores from the three groups. MLIS participants obtained slightly higher scores in general than the other two groups. In most cases, the GE group did better than the UE group. These could be explained by the search skills the three types of users had: MLIS students were trained searchers. GE students should have more searching experience in general than UE students because they had involved in more research activities. The more searching experience/skills the users had, the better their search performance. 5.5. Implications The result of no significant effect on search performance for full-text search systems can be interpreted in two ways from the systems design s point of view. On one hand, it might tell us that Basic search was doing as good as Advanced search in terms of search performance in these systems, that Basic search should be able to satisfy the user s search needs. On the other hand, as straightforward as the results could tell, Advanced search either was not that necessary, or had to be improved to show substantial advantage over Basic search, at least for the DLs involved in the study. Furthermore, considering that the combined use of Advanced search increased the search efforts significantly, this feature warrants further re-examination of its necessity. If the extra efforts could not result in more accurate results, why do users need to spend extra effort? Jansen [9] challenged if it was worth for the user of Web search engines to make efforts learning and using advanced search operators to construct complex queries. If the user does make extra effort, it is the failure of the systems design that does not reward the user s effort. For the fielded search systems represented by Xplore, the results can be seen as very positive for Advanced search because it improved search performance significantly. It can also be seen as a negative point for Basic search currently implemented: it was simply hard to use to accomplish the search task for inexperienced users. Perhaps some efforts have to be made on improving the Basic search design. From the user s point of view, especially inexperienced users, the findings indicate that: 1) Performing full-text search, Basic search could be sufficient to satisfy the user s need. Advanced search did not do better than Basic search, as demonstrated in ACM and IEEE CS, and the combined use cost more search efforts; 2) Performing fielded search, Basic search might not be able to satisfy the user s need. Advanced search could do a much better job, if Advanced search offered full-text search option, as demonstrated in Xplore. In addition, using Advanced search, either exclusively or in a combined way, would not increase search efforts very much. 6. Conclusion While the findings from this study are important and interesting, there are also limitations for the study. First, the user population was limited to inexperienced users of the DLs investigated. The results may not be applicable to all users, particularly to expert users of these DLs. for future studies it is necessary to enlarge the sample size and recruit more participants from different disciplines for robust research results. Second, limited by the experiment time, there was only one search task. The variety of tasks in real life was not present in the study. This lack of tasks may limit the generalization of the findings. Tests on more and different types of search tasks, with experienced users are desired. Yet another limitation is that the study was conducted on three DLs. Given the phenomenal design of Basic and Advanced search in 1613

nearly all search system interfaces, the use of Advanced search in other systems, particularly in search engines, should be investigated. Web search engines are typical full-text search systems. It should be interesting to explore how users would use Advanced search, and what effects the use of it would have on search performance. The results from such studies should be able to provide more direct evidence whether Advanced search is a helpful feature for obtaining more useful search results. These limitations outline the future research directions for the current study.. Acknowledgement This study was partially founded by a grant from the IEEE, Inc. 8. References [1] M. Hearst, A. Elliot, J. English, R. Sinha, K. Swearingen, and K. P. Yee, Finding the Flow in Website Search, Communications of the ACM, 45(9), 2002, pp. 42-49. [2] N. J. Belkin, C. Cool, J. Jeng, A. Keller, D. Kelly, J. Kim, H.-J. Lee, M.-C. Tang, and X.-J. Yuan, Rutgers TREC 2001 Interactive Track Experience. In E. M. Voorhees and D. M, Harman (Eds.). The tenth text retrieval conference, TREC 2001 (pp.465-42). NIST, Gaithersburg, Md, USA, 2002. [3] N. J. Belkin, C. Cool, D. Kelly, G. Kim, J.-Y. Kim, H.-J. Lee, G. Muresan, M.-C. Tang, and X.-J. Yuan, Query Length in Interactive Information Retrieval. Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, ACM, New York, NY, 2003. [4] Hearst, M., Search User Interface. Cambridge University Press, Cambridge, UK, 2009. [5] H. Xie, Supporting Ease-of-use and User Control: Desired Features and Structure of Web-based Online IR Systems, Information Processing and Management, 39(6), 2003, pp. 899 922. [6] A. Shiri, K. Molberg, Interfaces to Knowledge Organization Systems in Canadian Digital Library Collections, Online Information Review, 29(6), 2005, pp. 604-620. [] H. R. Hartson, P. Shivakumar, M. A. Perez-Quinones, Usability Inspection of Digital Libraries: A Case Study, International Journal of Digital Libraries, 4, 2004, pp. 108-123. [8] B. J. Jansen, A. Spink, T. Saracevic, Real Life, Real Users, and Real Needs: A Study and Analysis of User Queries on the Web, Information Processing and Management, 36, 2000, pp. 20-22. [9] B. J. Jansen, An Investigation into the Use of Simple Queries on Web IR Systems, Information Research: An Electronic Journal, 6(1), 2000. [10] C. M. Eastman, B. J. Jansen, Coverage, Relevance and Ranking: The Impact of Query Operators on Web Search Engine Results, ACM Transactions on Information Systems, 21, 2003, pp. 383 411. [11] I. Ruthven, Interactive Information Retrieval, Annual Review of Information Science and Technology, 42, 2009, pp. 43-91. [12] H. Topi, W. Lucas, Mix and Match: Combining Terms and Operators for Successful Web Searches, Information Processing & Management, 41, 2005, pp. 801 81. [13] H. Topi, W. Lucas, Searching the Web: Operator Assistance Required, Information Processing & Management, 41, 2005, pp. 383 403. [14] I. Xie, S. Joo, Transitions in Search Tactics during the Web-based Search Process, Journal of the American Society for Information Science and Technology, 61(11), 2010, pp. 2188 2205. [15] Korfhage, Information Retrieval and Storage, New York: John Wiley & Sons, 199, p.80 [16] Lesk, M., Understanding Digital Libraries, 2 nd ed. Morgan Kaufmann Publishers, San Francisco, CA, 2005. 1614