Evaluation of Visual Aid Suite for Desktop Searching

Transcription

1 Evaluation of visual aid suite for desktop searching Foo, S., & Hendry, D. (2007). Proc. 11 th European Conference on Research and Advanced Technology for Digital Libraries (ECDL2007), Budapest, Hungary, September 16-21, Lecture Notes in Computer Science (LNCS) 4675, Evaluation of Visual Aid Suite for Desktop Searching Schubert Foo, Douglas Hendry Division of Information Studies, School of Communication and Information Nanyang Technological University, Singapore {assfoo, Abstract. The task of searching for documents is becoming more challenging as the volumes of data stored continues to increase, and retrieval systems produce longer results list. Graphical visualisations can assist users to more efficiently and effectively understand large volumes of information. This work investigates the use of multiple visualisations in a desktop search tool. These visualisations include a List View, Tree View, Map View, Bubble View, Tile View and Cloud View. A preliminary evaluation was undertaken by 94 participants to gauge its potential usefulness and to detect usability issues with its interface and graphical presentations. The evaluation results show that these visualisations made it easier and quicker for them to find relevant documents. All of the evaluators found at least one of the visualisations useful and over half of them found at least three of the visualisations to be useful. The evaluation results support the research premise that a combination of integrated visualisations will result in a more effective search tool. The next stage of work is to improve the current views in light of the evaluation findings in preparation for the scalability and longitudinal tests for a series of increasingly larger result sets of documents. Keywords: Query result processing, query reformulation, tree view, map view, bubble view, tile view, cloud view, evaluation, search engine, user interface 1 Introduction The worldwide explosion in digital information due to rapidly increasing computer usage and the development of the Internet has been widely noted and documented [1]. As the volume of information stored electronically has grown, so has the need to be able to search this information resource in order to be able to answer individual 1

2 2 Schubert Foo, Douglas Hendry information needs [2]. This has led to the dramatic increase in the development of search tools to meet the growing demand to locate information more easily. The dramatic growth in web search technology, pioneered since 1998 by Google, has led to the widespread use of such tools by most computer users. Mainstream development to-date has concentrated on providing tools for searching the World Wide Web. Desktop search tools, for searching documents held on local computers hard drives, have been slower to develop. It is notable that these tools are now being incorporated into the latest desktop Operating Systems - Spotlight in Apple s OSX and Microsoft s Vista also incorporates similar functionality. Therefore, these desktop tools are starting to reach a much larger user base. In a classic search engine, the users enter their search terms and then request the system to search for matching results. These are then returned as a list of resources that best matches the users queries. Typically, a large text-based list of ranked results is returned to the users who scan through them to identify deemed relevant documents, and read through the contents to extract information to satisfy their information needs. As the volumes of data to be searched increase and result lists becomes increasingly longer, the challenge is now to maintain the efficiency and effectiveness of the search and result selection process. Well-presented graphical views can convey large amounts of complex information in a simple and easy to understand manner [3]. It is, therefore, not surprising that graphical visualisations have been employed in search engines to assist users in the searching process. 2 Related Work Much research in Information Retrieval Systems (IRS) has been focused on the algorithms used for searching and relevance ranking. This includes work on augmenting search results by the use of metadata such as thesauri for aggregating results. However, most of the deployed systems (including search engines) are based on returning textual list of results for users to review. Some graphical tools have been developed to visually assist the user in formulating their query. These include the use of Venn Diagrams, Filter-Flow Visualisations and Block Orientated Visualisations, which have been documented elsewhere. However, the principal area where visualisation techniques have been applied is in the results review process. Here a visualisation is used to replace or augment the results list and/or the document preview functions. A number of tools have been created that use a variety of 2D and 3D graphical visualisations in order to allow the user to explore and understand the results of their query. In order to support query reformulation visualisations usually present terms that are related to the query terms being used. These related terms often come from a controlled dictionary, thesaurus or other metadata held in the system. The user can review these new terms and use them to modify their query. One such tool is the AquaBrowser Library [4] which shows a visual word cloud that suggests words similar or related to the users query terms. Established online search engines such as Google have a very traditional user interface. While the search and ranking algorithms behind these search engines are

3 Schubert Foo, Douglas Henry 3 very sophisticated their interfaces have remained traditional and text based. However, a number of new online search engines have started to offer more graphical interfaces to assist users. Grokker and Ujiko are examples of visualisations being used to present the results of web searches. Whilst some evaluation studies have reported mixed results [5] many have found positive support that the visualisations have aided user performance [6, 7]. Even in cases where performance has not improved users often report better satisfaction with tools incorporating visualisations [8]. Visualisations seem to be particularly effective where the complexity of the task and volumes of data are at their highest [9, 10]. They also seem to work well when they are kept as simple as possible [11]. 3 Design A Java based lightweight desktop search engine was developed that could index and search content on a desktop computer. The indexing and searching sub-systems were designed based on traditional IRS principles and incorporates stop word removal, stemming and is based on Boolean logic. The design of the user interface was based on the following research premises: - Visualisations can assist users to search for documents [7, 10] Different visualisations can be used to support different elements of the searching process (results review and query reformulation) Different graphical techniques can be used to assist users to visualise different kinds of information Visualisations work best when they are kept simple [11]. The search engine GUI has a plug-in view architecture that allows different views to be created independent of the searching mechanism. Six views were constructed for use and evaluation. While most of the individual views have been used elsewhere as standalone views in different forms of IRS, the novelty in our work lies in the bubble view, and the provision of a suite of related views thereby providing synergy through flexibility to switch views to visualise, infer and process the result sets differently. 3.1 List View The List View (see Fig 1) is the classic search results view. It contains a list of files that, based on Boolean logic, match the users query. Each file name is shown along with the number of hits from the query. A hit is defined as one occurrence of one query term in the file contents. The files are listed in descending order of the total number of hits found.

4 4 Schubert Foo, Douglas Hendry Fig 1. List View This view also contains a File Viewer window. This window will display the textual contents of a selected file in order to allow quick review of the file contents. Any nontextual content, such as images etc. in a Word or PDF document, are not displayed. Similarly, some of the source document formatting will be lost as only line and paragraph breaks are preserved in the text extraction process. 3.2 Tree View The Tree View is similar to the List View (see Fig 2). The files are organised based on their underlying folder structure. For each file in the results list, all of its parent folders are added to the folder hierarchy (avoiding duplicates). The Result files are then added into the tree at the appropriate folder for their physical location. Fig 2. Tree View

5 Schubert Foo, Douglas Henry 5 This view is very similar to the Microsoft Windows Explorer view. However, only files that match the query string are displayed and only the parent folders of these files are included in the tree. The purpose of this view is to use the physical file structure as part of the results display. If users have taken the time to organise their documents into meaningful folders and hierarchies then this information may be useful when reviewing results. This view is particularly suited for thesaurus or taxonomy based folder organisations where documents are stored in the respective nodes of this organisation scheme. 3.3 Map View The Map View (see Fig 3) provides an overview of the relationship between the query terms and the result files. Each query term is depicted as a blue rectangle and each result file as a green ellipse. Lines link related query terms and results files. These are annotated (in red) with the number of occurrences of the query term in the result file. The view can be zoomed and rotated and individual shapes can be moved around on screen. In addition, the shapes can be reduced to mini icons if the display is overcrowded. The purpose of this view is to provide an overview of the query and how the individual query terms are influencing the results. Therefore, this bird s eye view can be used to detect problems in the query specification if the required results are not as expected. It will clearly show the relative influence of each query term in producing the result files and therefore help the user in deciding whether the query needs to be reformulated and how to do so. Fig 3. Map View

6 6 Schubert Foo, Douglas Hendry 3.4 Bubble View Boolean logic systems make it difficult to judge the relevance of a result file. The total number of hits may not be a good guide to relevance especially when document length is taken into consideration. Therefore, it is desirable to normalise this measure to take into account document size. In this work, this is done by calculating a hit density for each result file expressed as the number of hits per 1,000 searchable terms (non stopwords) in the document. Fig 4. Bubble View The intention of the Bubble View (see Fig 4) is to help the user better assess the relevance of different documents. The axes of the graph are the number of hits and the calculated hit density. These measures are used to distribute the documents along each axis as they provide good document discrimination in order to achieve a better visualisation. The diameter of the bubble is determined by the number of query terms present in the result file more query terms found results in a larger bubble diameter. Quadrant 1 is expected to contain the most relevant documents as both the number and density of hits is greatest. Correspondingly, quadrant 4 will be expected to contain the least relevant documents, as both the hit count and density are smallest. The Bubble View provides an overview of document relevance for a given query and aids the review of documents most likely to be relevant to the query. 3.5 Tile View The Tile View (see Fig 5) presents each result file as a coloured tile using a Treemap. A Treemap is a space-constrained visualization of hierarchical structures [12]. The size of each tile is determined by a measure such as Total Number of Hits, File Size, and Hit Density (Hits per 1,000 searchable terms). Using the control panel the user can change the measure used to determine the size of a tile.

7 Schubert Foo, Douglas Henry 7 Fig 5. Tile View In addition, the colour of a tile is determined by its file type. The display can be restricted to certain file types or all can be shown. The Tile View can optionally include the folder hierarchy of the results files. In this variant, all the result files in a specific folder are grouped together in a super tile. Each folder is enclosed within a tile representing its parent so that the entire folder structure of the results files can be displayed. The purpose of the Tile View is to allow users to review the results visually and judge their relevance based on different criteria with larger tiles denoting the most relevant documents. They can then decide which files to review in detail by looking at their contents. 3.6 Cloud View The Cloud View (see Figure 6) is adapted from the Tag Clouds popular on social networking sites such as Flickr [13]. A Tag Cloud is a weighted list which contains the most popular tags used on that site and the relative popularity of each tag is indicated by changing its font. It is thus easy to see the most popular tags. The Cloud View creates a Word Cloud based on the (indexable) content of the result files. The contents of these files are examined and stop words and nonindexable terms are removed. The words are then stemmed and a simple term count made of all the terms in the documents. The top 300 terms are then displayed in a Word Cloud as they represent the most common indexable terms in the documents. Only files selected in the Results List (in the left hand window) have their contents included in the Word Cloud. If the user changes the selection of files, the Word Cloud is dynamically refreshed with information based on the new selection of files.

8 8 Schubert Foo, Douglas Hendry Fig 6. Cloud View When the user clicks on a word in the Word Cloud a popup menu appears offering the choice to expand, restrict or exclude the word from the current query or to create a new search using the selected word. The purpose of the Cloud View is to help users to reformulate their queries based on the searchable terms found in the results documents. This could be used to expand or restrict the search in order to better refine the results set. 4 Evaluation The next stage was to undertake a preliminary system evaluation in order to determine the usefulness of the views in the system. The principal aim was to determine if the availability of such visualisations was useful and which of them would be worth developing further. As such, the work reported in this study is mainly intended as a proof of concept and to have a first gauge of usability and usefulness of the views. The evaluation was carried out through a user survey based on the questionnaire technique. A questionnaire comprising 51 questions was developed to gather user opinion about the visualisations and their usefulness. The evaluation was split into five tasks each task required the user to perform a search in support of a given information need and the participants worked through these in sequence. For each task they were given five minutes to interact with an interface to find the most relevant documents to satisfy the information need. Following this, they were asked to complete a series of questions pertaining to the interface used. Prior to the evaluation, a handout describing the search engine and views was provided to the evaluators, followed by a briefing and demonstration. Evaluators were also invited to download the search engine to familiarise themselves with it and the various views. Each evaluator had a minimum of one week of familiarity prior to the evaluation, and each spent 45 to 60 minutes in completing the evaluation.

9 Schubert Foo, Douglas Henry 9 Since the purpose of this evaluation was to review the visualisation aspects of the search tool, it was important to remove other factors that could influence user responses. Therefore, for each task the search terms to be used were pre-specified. The evaluation system was based on a small collection of 30 documents since this was a preliminary evaluation to test the proof of concept of these visualisations. At the same time, the search engine has the provision for users to define and show the top n documents. This filtering system effectively restricts the volume of results displayed to the user, so even if a larger collection were used for indexing, the user would only see a controlled subset of documents when reviewing the results. Based on the results of this evaluation, we expect future evaluations to be more comprehensive and extended to test for scalability to larger document collections. The documents covered topics on information retrieval systems, the World Wide Web, indexing and programming languages and a portion of the ERIC thesaurus was used to create a hierarchical folder structure. The sample documents were then stored into the folders based upon their categorisation in ERIC. The students of the Nanyang Technological University M.Sc. Information Studies class of 2006/07 participated in the evaluation. There were a total of 94 participants 57% were female and 43% were male. 4.1 List View Results This is the classic search engine results view with no additional visualisation. The responses received (see Table 1) showed strong support that the List View is both easy to use and useful in reviewing the results. These results confirm that the basic design and operation of the desktop search engine is effective and useful. Table 1. List View Results # Question % Who Agree / Strongly Agree 1 The List View was easy to use. 89% 2 The List View was useful in reviewing my results. 86% 4.2 Tree View Results The results strongly indicate that the Tree View (see Table 2) was easy to use and useful in reviewing the results. The results are very similar to those of the List View. As would be expected from these results a large majority (93%) of the evaluators found the function of the Tree View clear and obvious. The evaluators also indicated that they organise and structure their own folders so most of them could potentially benefit from a search engine that uses the folder structure to present search results. This confirms the design premise that the user s folder structure would be a useful aid to presenting the results documents as well as a means to logically organise information in thesaurus/taxonomy-like structures that can support browsing as well as searching in this instance.

10 10 Schubert Foo, Douglas Hendry Table 2. Tree View Results # Question % Agree / Strongly Agree 6 The function of the Tree View in the left hand window was clear and obvious. 93% 7 The Tree View was easy to use. 91% 8 The Tree View was useful in reviewing my results. 85% 9 I organise my documents logically in folders so this view would be useful to me. 87% 4.3 Map View Results The evaluation results for the Map View (see Table 3) showed that slightly over half of the evaluators (51%) agreed or strongly agreed that the Map View was useful in reviewing their query results and reformulating their query. The distribution of responses for ease of use and usefulness are very similar. Table 3. Map View Results # Question % Agree / Strongly Agree 13 The Map View was easy to understand. 53% 14 The popup windows were useful. 75% 15 The Map View was easy to use. 58% 16 The Map View was useful in reviewing my query results and reformulating my query. 51% Comment analysis indicated that the most useful aspect noted by the evaluators (33) was the ability to see an overview of the relationship between the query terms and the results files. This was the design premise for the Map View to provide a clear overview of the query. However, the view can become very crowded when a large number of items are displayed resulting in overlapping of the graphic objects. A significant number of evaluators (34) indicated that this caused confusion. 4.4 Bubble View Results The evaluation results for the Bubble View (see Table 4) show that around half the evaluators (46%) found this view useful in reviewing their query results. The majority of evaluators found the position (65%) and size (59%) of the bubbles gave them useful information, which supports the concept of this view as a means to convey several dimensions about the relevance of the results documents. The comments analysis showed that useful features were the ability to get a quick and easy overview of the relevancy of the results and the ability to see the hit density. The major confusion factors reported were related to the display of a large number of result documents. In this case, the document titles overlap and become unreadable and the evaluators found the display to be very cluttered and messy. Some evaluators (10) did not understand that the size of the bubble related to the number of different query terms found in the result document.

11 Schubert Foo, Douglas Henry 11 Table 4 Bubble View Results # Question % Agree / Strongly Agree 20 The position of a Bubble on the Chart gave me useful information. 65% 21 The different sizes of the Bubbles made sense and was useful. 59% 22 The Bubble View was easy to use. 51% 23 The Bubble View was useful in reviewing my query results. 46% 4.5 Tile View Results The results of the evaluation of the Tile View (see Table 5) show over half the evaluators (59%) agreed or strongly agreed that the Tile View was useful in reviewing their results. Over two thirds found the tiles to be obvious and easy to understand (68%) and the ability to use different criteria to control their sizing was found to be useful (69%). This supports the design objective for this view to easily support the use of different criteria for judging the relevance of the results documents. The ability to group files by folders also received strong support with 75% of evaluators agreeing or strongly agreeing that this was useful. The comments analysis indicated that useful features were the ability to change tile size based on different criteria, the ability to group files by folder and the use of colour to distinguish file types. Table 5 Tile View Results # Question % Agree / Strongly Agree 27 The purpose of the Tiles and the Information they display was obvious and easy to understand. 68% 28 The ability to change Tile Size based on different criteria was useful. 69% 29 The ability to group files by Folders was useful. 75% 30 The Tile View was useful in reviewing my query results. 59% 4.6 Cloud View Results The results for the Cloud View (see Table 6) showed that nearly two thirds of the evaluators found the Cloud View useful in reformulating their query (63%) and easy to use (61%). However, the distribution profile for question 37 (usefulness of Cloud View) is different it has a bi-polar distribution, with a peak for disagree and agree. This implies that the evaluators were split into two groups. A review of the comments strongly supported this conclusion. Those evaluators who scored the usefulness of the Cloud View very low (strongly disagree or disagree) reported a lot of confusion as to the contents of the Cloud. In other words, they did not find the view useful because they did not understand what it does. In order to support this conclusion, the responses to question 34 (about understanding the contents of the view) were compared to those of question 37 (usefulness of the view). Of those who found the view difficult to understand (disagree or strongly disagree with Q34) only 29% found it useful. However, 85% of those who found the view easy to understand (agree or strongly agree with question

12 12 Schubert Foo, Douglas Hendry 34) found it useful. Therefore, it is reasonable to conclude that the reported usefulness of the Cloud View would increase if the users better understood what it does. The comments imply that some users had not seen this type of visualisation before. Table 6. Cloud View Results # Question % Agree/ Strongly Agree 34 The contents of the Word Cloud were obvious to me and easy to understand. 56% 35 The popup menu on the Word Cloud was easy to use. 66% 36 The Cloud View made it easy to reformulate my query. 61% 37 The Cloud View was useful in reformulating my query results. 63% 4.7 General Results After evaluating each of the views, the evaluators were asked for feedback that was more general about the search engine and its visualisations. There was strong support that they made it easier (83%) and quicker (86%) to find relevant documents. 4.8 Limitations of Evaluation As this was intended to be a preliminary evaluation, we did not address the issue of scalability in terms of document collection sizes (hence result list sizes), as the main aim was to obtain a first set of evaluation results to validate the ease of use and usefulness of the various views to display and aid users to effectively use the search results. As such, we have left the testing of scalability for future evaluation. Along with this, we expect to carry out a series of longitudinal studies as research has found that complex interfaces summarising lots of data in comparison with simple interfaces will invariably fall short of users satisfaction metrics initially but stand to have the potential to be gradually accepted when used for longer periods of time [14]. Another potential limitation is in the form of learning effects in using a small document set in the evaluation. In order to minimise this, we have used different scenarios in each of the 5 tasks with different queries resulting in different relevant documents. With hindsight, we might have used a larger set of documents in different domains for the evaluation to ensure that this effect is eliminated. Nonetheless, we do not view this as a serious limitation due to the context of our current evaluation objectives. 5 Conclusion The results of the evaluation indicate that overall the users found the visualisations useful and easy to use. They also felt that it would help them find their desired results quicker. In particular, the Tree View and Cloud View were rated highly by the evaluators. The Tree View takes advantage of the users own defined hierarchies (their folder structures) to present the search results in a format that significant numbers of the evaluators found useful.

13 Schubert Foo, Douglas Henry 13 The Cloud View is a much more novel interface and some of the evaluators had difficulty understanding its principle of operation. A large majority of those users who understood the concept found the view to be very useful. In conclusion, this work has shown that the inclusion of multiple visualisations would be likely to increase the usefulness and ease of use of a search tool and that different visualisations can be integrated together to provide support for different elements of the search process. REFERENCES 1. Moore, F., Digital Data's Future - You Ain't Seen Nothin' Yet! Computer Technology Review, 20, 10 (2000) Mostafa, J., Seeking Better Web Searches. Scientific American, 292, 2 (2005) Tufte, E.R., The visual display of quantitative information. 1983, Cheshire, Conn. (Box 430, Cheshire 06410): Graphics Press Kaizer, J. and A. Hodge, AquaBrowser Library: Search, Discover, Refine. Library Hi Tech News, 22, 10 (2005) Heo, M. and S.C. Hirtle, An empirical comparison of visualization tools to assist information retrieval on the Web. Journal of the American Society for Information Science and Technology, 52, 8 (2001) Sebrechts, M.M., et al., Visualization of search results: a comparative evaluation of text, 2D, and 3D interfaces. Proceedings of SIGIR: International Conference on R&D in Information Retrieval, 22, (1999) Veerasamy, A. and R. Heikes. Effectiveness of a graphical display of retrieval results. in Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval Philadelphia, Pennsylvania, United States: ACM Press. 8. Heflin, J., et al. WebTOC: Evaluation of a Hierarchical Browsing Interface for the World Wide Web [cited 7-Nov-2006]; Available from: 9. Morse, E., M. Lewis, and K.A. Olsen, Testing visual information retrieval methodologies case study: comparative analysis of textual, icon, graphical, and "spring" displays. J. Am. Soc. Inf. Sci. Technol., 53, 1 (2002) Wingyan, C., H. Chen, and J.F. Nunamaker Jr, A Visual Framework for Knowledge Discovery on the Web: An Empirical Study of Business Intelligence Exploration. Journal of Management Information Systems, 21, 4 (2005) Chen, C. and Y. Yu, Empirical studies of information visualization: A meta-analysis. International Journal of Human-Computer Studies, 53, 5 (2000) Bederson, B.B., B. Shneiderman, and M. Wattenberg, Ordered and Quantum Treemaps: Making Effective Use of 2D Space to Display Hierarchies ACM Transactions on Graphics (TOG), 21, 4 (2002) Yahoo Inc. Popular tags on Flickr photo sharing [cited 11-Nov-2006]; Available from: Shneiderman, B. and C. Plaisant. Strategies for Evaluating Information Visualization Tools: Multi-dimensional In-depth Long-term Case Studies. in AVI workshop on Beyond time and errors: Novel evaluation methods for information visualisation Venice.