FacetBrowser: A User Interface for Complex Search Tasks

Transcription

1 FacetBrowser: A User Interface for Complex Search Tasks Robert Villa Dept. of Computing Science University of Glasgow Glasgow, UK villar@dcs.gla.ac.uk Nicholas Gildea Dept. of Computing Science University of Glasgow Glasgow, UK ngildea@dcs.gla.ac.uk Joemon M. Jose Dept. of Computing Science University of Glasgow Glasgow, UK jj@dcs.gla.ac.uk ABSTRACT With the rapid increase in online video services, multimedia retrieval systems are becoming increasingly important search tools to users in many different fields. In this paper we present a novel retrieval interface, FacetBrowser, which supports the creation of multiple search facets, to aid users carrying out complex search tasks involving multiple concepts. Each facet represents a different aspect of the search task: an assumption of this work is that search facets are best represented by sub-searches, providing users with flexibility in defining facets on the fly, rather than using predefined categories or metadata information as used in many other exploratory search interfaces [3, 8, 17]. Such facets can be organised into stories by users, facilitating users in building up sequences of related searches and material which together can be used to satisfy a work task. The interface allows more than one search to be executed and viewed simultaneously, and importantly, allows material to be reorganized between the facets, acknowledging the inter-relatedness which can often occur between search facets. The design of the FacetBrowser interface is presented, along with an experiment comparing it to a tabbed interface similar to that on modern web browsers. The results suggest that the FacetBrowser has the potential to aid users in exploring and structuring their searching effort when carrying out broad search tasks. Categories and Subject Descriptors H.3 [Information Storage and Retrieval]: Information Search and Retrieval General Terms Human Factors, Design 1. INTRODUCTION With the rapid increase in online video services, video retrieval systems are becoming increasingly important search Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MM 08, October 26 31, 2008, Vancouver, British Columbia, Canada. Copyright 2008 ACM /08/10...$5.00. tools to many users in many different fields. As the use of multimedia retrieval systems increase, there is a continuing need for interfaces which maximise the ability of users to find relevant results, and which better reflect real-world use cases. In this paper we present a novel multimedia retrieval interface, which supports the creation of multiple search facets, to aid users carrying out complex, multi-faceted search tasks. The emphasis is on allowing the user to explore and organise a data collection within the searching interface, in an attempt to address these dual requirements, of maximising the potential of content based retrieval systems, and matching better real world needs. Access to many current multimedia databases is often based on metadata, requiring effort from users to manually tag and organise their own and other people s data. Such techniques tend to leave some information indiscoverable due to the lack of user supplied tags. This is exacerbated in some domains, such as the media industry (e.g. film and television production), where the number of users with access to video and image collections is relatively small in comparison to the quantity of data present. Production rushes, for example, the raw footage shot during the making of a film or television programme, may together be much longer than the final production and may be viewed by only a handful of people. This can potentially lead to a large quantity of video data which will be untagged and irretrievable, by such methods. Content-based retrieval methods, such as those systems involved in the TRECVid effort [18], do not require user involvement in the search process. Unfortunately, the performance of current content-based video systems is very low when compared with equivalent state of the art text systems. For example, in TRECVid 2007 the best performing automatic run had a MAP score of 0.088, on average returning just over 2 relevant shots for the top 10 results [15]. This relatively low performance can be attributed to the problem of the Semantic gap, which is the large difference between the low level features which can typically be extracted from image, video and audio data, and the semantic concepts which users typically want to search for. A problem as difficult as the semantic gap is unlikely to be solved by a single magic bullet, but rather by a range of techniques, such as improving automatic annotation of images and video content. One approach to towards bridging the semantic gap is by providing user s with searching environments which allow them to better explore, browse, and organise their searching materials. By providing a richer search environment, and supporting exploration through multiple simultaneous search paths, 489

2 we hope to both aid the user in expressing their information needs, and provide them with an environment in which they are more likely to come across the material they are searching for. Additionally, in structuring their searching, a user may provide more feedback to the underlying retrieval system which it can then use to improve the user s search results. We present a faceted retrieval interface the FacetBrowser, a screenshot of which is shown in Figure 1 which aims to support such facets, aiding the user in organizing queries and associated search results, and supporting exploratory search. The interface allows multiple searches to be executed and viewed simultaneously, and allows material to be reorganized between the facets. The system is based solely on content-based retrieval techniques, and does not use manual annotation - rather, it aims to exploit content based systems as far as possible, acknowledging that their lower performance when compared to state of the art text systems will inevitably require a greater explorative effort by the user when using the system. Additionally, by allowing the user to easily split a broader search into more than one narrow search, we hope that the feedback provided to content-based retrieval systems by users will be more amenable to automatic search systems based on low-level features. Each facet within the FacetBrowser is modeled as an individual search, which exists within a context of other searches, which together represent the user efforts in satisfying a task. Additionally, we make the assumption that facets are often inter-related, and that this relatedness is natural in complex search tasks, with many different yet related aspects. For example, researchers and journalists in the broadcasting industry often need to investigate topics in which they are not expert, requiring them to explore and learn about a topic while searching. Through the process of searching, the journalist will gradually learn about the subject, resulting in many different searching directions. We aim to support the user in taking many different directions, and to support their exploration through a collection. The rest of this paper is structured as follows: first we present a scenario describing how a user can could potentially use the FacetBrowser system, with a discussion of the key advantages the system is intended to provide. A description of previous related work then follows. The software architecture of the system is then described, followed by the FacetBrowser interface. A user study comparing FacetBrowser to a tabbed interface is then described, and results presented. 1.1 Example scenario: Review of the 33rd G8 summit in Heiligendamm The example scenario is based on the news domain, and involves a user who must simultaneously research and learn about a topic: Being a journalist employed on a monthly newspaper, Bob has been given the task of reviewing the last G8 summit in Heiligendamm, Germany, to find good video material showing the politicians who took part in the summit. Bob has a list of the main political leaders who took part in the summit, and starts by carrying out a search for George Bush, G8, Heiligendamm. This returns many results of George Bush, including many clips which include more than one of the leaders who were present: e.g. there are shots of George Bush and Angela Merkel together. Based on the results from this initial search, Bob decides to create a facet for each of the eight leaders who took part, and starts to reorganise the material from the initial facet, into each of the other facets, one per politician. There are no shots, however, of one leader, Canada s Stephen Harper. For each of the facets that do have relevant material, Bob uses the relevance feedback functionality to find better shots for each individual politician, selecting for each facet a single shot which will become it s exemplar. In carrying out these searches, Bob discovers a picture of Stephen Harper when searching for Tony Blair, and is able to use this as a starting example for the Stephen Harper facet. Once Bob has found material about each of the individual leaders, he starts to reorganise the facets, so that the order of the facets will reflect the order in which he will present the material in his report. In reorganising the facets, he decides to place the George Bush facet first, followed by Vladimir Putin, then Angela Merkel, etc. In doing this, he notices material showing George Bush talking to Vladimir Putin, which would be appropriate as a intermediate sequence between these two leaders. He then creates a new facet, placing it between the George Bush and Vladimir Putin facets, drags and drops the relevant material into this facet, and calls it George Bush meeting with Vladimir Putin. Based on this initial material, Bob can then use relevance feedback to search for more shots. He then creates similar intermediate facets between other leads, such as Tony Blair and Nicolas Sarkozy, in order to find material linking these leaders. At the end of the search process, Bob has a sequence of facets, in the same order in which he plans to talk about the leaders. As well as the facets for each leader, he also has a selection of linking facets, showing combinations of leaders meeting together. This above scenario illustrates a number characteristics which FacetBrowser aims to support, of which perhaps the most striking of which is the splitting of a search in to a sequence of sub-searches, all of which can be seen at the same time. Visually laying out different searches in facets can be seen as a way of naturally visualising a temporal progression of a search, yet still allow the previous search facets to be instantly available. The interface allows a user to see multiple facets simultaneously on screen, allowing their visual comparison. This allows the user to get an overview of multiple facets at the same time, providing a better overview of the state of the user s task. This also makes it easy to create new facets from old ones, leveraging the query by example functionality in content-based retrieval systems. Facets allow users to take advantage of material found in old searches, and use that material as a starting point of a new search. Importantly, new search facets do not destroy the current facet s search history - new facets can therefore be spun out 490

3 Figure 1: A screenshot of the FacetBrowser interface (Section 4) from older facets, allowing the user to investigate multiple directions simultaneously, while keeping the user s searching history intact in the older searches. This is difficult in most current search systems, where the user s search history is typically a single strand, and jumping back into the user s history will loose all the searching effort the user has carried out up until that point. History can be viewed as a user s past search effort, and this searcher effort should, we argue, be maintained as a resource to the user. The scenario also shows the use of categorisation while searching: by interacting and searching the user creates new facets as a byproduct. This is also different to grouping interfaces such as [21], where the creation of groups is a separate process from the search process. Another important feature is the ability to re-organise material between facets. This reorganisation is an important aspect of the FacetBrowser: current image and video retrieval systems are not always good at retrieving highly relevant results. Allowing the user to reorganise material in this way allows the user to take advantage of relevant material, no matter what search it is found. This can be difficult in existing search systems; for example, it can be difficult to use a good image found via google s image search interface in a new search with a different system. This presents a barrier to reuse. Finally, it should be stressed that the interface should not require explicit annotations by a user to operate: giving a name or a key frame to a facet is an optional operation which user s don t have to use if they don t find it necessary. Indeed, one possible avenue of research is investigating how best to automatically label facets, either with automatically generated text, or by selecting a single exemplar from the facet. Creating a new facet should be as easy as creating a new tab in a modern web browser - it should be a fast operation which does not get in the way of the user s searching, requiring only a single key or button press. In summary, FacetBrowser is designed to provide facilities for the organisation of facets, re-organisation of material between facets, and maintenance of search histories. With the combination of these features, we hope to provide the user with the ability to tell a story via the search interface and to take advantage of any serendipity in the search process. 2. PREVIOUS WORK Given the importance of exploration in this work, the area of exploratory search is an important and emerging area of past work. [3] is an early example of a paper proposing exploratory search, and gives a definition of facet based on categorical metadata, from [3]: It can be useful to think of category metadata as being composed of facets: orthogonal sets of categories, which together can be used to describe a topic. A facet is an attribute which can be used by the system and user to split a database into facets, allowing a user to browse an information space by selecting facets. An example given in [3] is a recipe finding task, where a user initially selects Poultry as a main ingredient of the recipe which they wish to make. The interface then provides a set of subsequent facets, such as Appetizers, Brunch, Sauce, etc. which the user can select to refine their search. The set of retrieved documents are those which contain the conjunction of the selected metadata. This is contrasted to themorefixedbrowsingpossibleinahierarchy,wherethe 491

4 user must select categories in the order in which they are defined in the hierarchy. A great many modern e-commerce websites, such as amazon.com and ebay.com use such techniques, and in the domain of video retrieval, the Open Video Digital Library project [9, 7] provides another example of a system. Relation Browsers [8, 7] partition a data collection in into slices, based on metadata attributes. In [7] a relation browser for the open video digital library is shown, which allows users to explore a video collection using attributes such as genre, format, language, etc. A third example of this type of system is the work of [23], who carried out an evaluation of their system with 32 art history graduates, which resulted in a very positive preference for faceted search. It must be noted, however, that the concept of facet used in such systems is very different to that in FacetBrowser: it is an attribute which splits a data collection into one or more orthogonal groups. The definition of aspect as used in the TREC interactive track can be considered as similar to the definition of facet used in this work. The interactive track in TREC-5 [13] defines an aspect as... roughly one of many possible answers to a question which the topic in effect posed, similar topics also being used in TREC-7 and TREC-8 ([14, 4]. For example, topic 408i from [4] has description What tropical storms (hurricanes and typhoons) have caused property damage and/or loss of life?, and in it s associated instances section asks the user to... find as many different storms of the sort described above as you can.... Also relevant are grouping interfaces [21, 19, 10, 11, 12]. One such example is EGO [21, 19], which was designed to provide a media professional (such as graphic designers) a workspace in which they could organise and characterise their visual information needs. The interface provided a searching system, which presented the list of results from an image retrieval system, and a separate workspace, where images could be dragged and dropped, and then bundled together into groups. Groups were used as the main user classification system, and existed in a zooming workspace which could be viewed at different levels of detail, allowing a searcher to gain a bird s eye view of their groups. A recommendation system was also provided which found new images similar to those of the group, and which allowed these images to be integrated into existing groups through drag and drop. A similar system to this is ImageGrouper [11, 12] which also provides grouping functionality. In the text retrieval domain, [2] describes a system that allows the user to structure their information seeking environment by grouping documents into piles. Each pile can be used as a source of relevance information for executing new queries. From the information discovery literature, [6] describes combinformation, an interface which allows users bring together to image and text summaries to provide users with an environment to promote idea generation and discovery. While providing a space for users to organise information, the focus of this system is more general than the others mentioned so far, however, not being focused solely on search tasks. The main difference between these grouping type of systems and FacetBrowser is the use of a single global search history, which is divorced from the grouping mechanism itself. In FacetBrowser, these are combined, each facet carrying with it a separate search subsystem, and associated history. Server Profile Database Visual Index Text Index Profiling Module FacetBrowser Interface Web browser clients Retrieval Module Figure 2: FacetBrowser system architecture 3. SYSTEM ARCHITECTURE The overall architecture of the FacetBrowser system is shown in Figure 2, and can be split into three main parts: the video and image retrieval system, which implements the underlying content-based retrieval system, the FacetBrowser interface as shown in Figure 1, and a profiling subsystem, which stores the persistent user profiles and facet information. A web based architecture is used, with the retrieval backend and profiling modules running on a web server, and the interface implemented using the Google Web Toolkit (GWT), an AJAX toolkit 1. The interface is accessible from a web-based login, allowing multiple users to remotely search the indexed video or image databases. The backend retrieval system is responsible for the retrieval of videos and images 2, and is based on the system described in [20], using typical content-based indexing techniques. Videos are segmented into short video sequences called shots, which are used as the element of retrieval: each shot is separately indexed by the system, and the results of searches are presented as a list of shots. For each shot, one or more keyframes are extracted, which are single frames from the video used to represent the shot. In this work only the textual retrieval component of the system was used, which searches automatic speech transcripts extracted from video content. This textual context can then be searched using normal text queries entered by the user. The profiling module handles the storage of the persistent user profiles, including all data pertaining to the facets created by users, and the organisation of facets within particular sequences. User profiles are organised into a hierarchy of objects: Each user is provided with a user profile, which contains one or more facet storyboards The system supports both image and video collections, although in this paper we focus on the video retrieval mechanisms 492

5 A storyboard is a sequence of one or more facets, or panels, providing a container for facets. Facets have an order within each storyboard, and can be moved and deleted within a storyboard. Each storyboard is intended to represent a particular work task in which the user is engaged A facet, or panel, represents a single topical aspect of the user s information seeking needs, contextualised by the storyboard and user profile in which it exists This profile information is persistently stored across user sessions, being automatically loaded by the interface code when a user logs in. In the next section, we describe the FacetBrowser interface, concentrating our interest on how facets are represented to the user. 4. THE FACETBROWSER INTERFACE A screen shot of the FacetBrowser interface is shown in Figure 1. It is split into one or more vertical panels, each panel representing a single facet of a larger work task. When the system is initially started up, a single empty panel is displayed on the left of the screen, ready for the user. New panels can be created using the Add new item button on the top left of the screen, appearing at the far right. Following the yellow numbers on Figure 1, each panel contains: 1. A name for the panel, which can be optionally supplied by the user 2. Delete icon, which removes the entire panel 3. A key shot, intended to be the visual exemplar for the panel, optionally selected by the user. This can be changed by dragging and dropping a shot from a relevant list onto the key shot area 4. Left and right arrows, which will move the whole panel left one place in the sequence, or right one place. For the example given in Figure 1, clicking the left arrow will swap the order of the last two panels 5. The search box and button, allowing the user to enter a textual query and start a search 6. A pull down list of the searches already carried out in that panel. The user can re-execute an old query by selecting an item on this list. The associated history button will pop up a window containing deleted shots, which allows a user to undelete shots. 7. The list of relevant shots, as selected by the user. These shots are selected by dragging and dropping shots from other panels, the current panels result list, or an open video browser window. 8. The list of search results, if any, ranked from most relevant (top left) to least relevant (bottom right) The interface makes extensive use of drag and drop. Shots on the search result list can be dragged and dropped onto the relevant shots area, which will add the shot to the panel s list of relevant shots. There is no restriction on what panel a result can be dragged onto, therefore it is possible to drag a result from one panel directly onto the relevant list of a different panel. This action is a copy rather than a move the shot will still remain in its source location after being dropped elsewhere. Relevant shots can also be dragged and dropped between different panel s relevant shots, allowing the reorganisation of material across the sequence. This action is, once again, a copy rather than a move. Relevant shots can be removed from the relevance lists using a delete button given on the bottom left of each shot s keyframe. This will remove shot from the panel, and add it to the list of deleted shots, appearing in the history pop-up window. The shot can be played by clicking the play button on the bottom right, which brings up the video player. One interface complication is that the shot s must be dragged by the grey bar at the top, which also provides the number of the shot within the video, designed to enable the drag and drop to function more smoothly. When a shot s play button is pressed, the built-in video player will pop up. The video for the shot itself plays in the center of the window, with keyframes of the shots temporally before and after displayed on the left and right; clicking on a keyframe will move that shot to the centre of the player, and start it playing. In this way a user may navigate temporally through a video, backwards or forwards. At the bottom of the video player is the automatic speech transcript of the playing shot, followed by the date at which the video was broadcast. This text can also be dragged and dropped on any panel, to mark the currently playing shot as relevant to apanel. In addition to being able to organise shots between the panels, a user is also able to re-order the panels themselves, using the green left and right arrows on either side of the panel keyframes. The icons move the corresponding panel left or right in the overall sequence, allowing the user to reorder their searches to best match the story or multi-faceted query which they are engaged. 5. USER STUDY To evaluate the effectiveness of the FacetBrowser interface, a user study was carried out, with two main research questions: Research Question 1: Does the FacetBrowser interface allow a user to better explore an information space compared to a conventional search interface? Exploration is an important aspect motivating FacetBrowser, and the broad search tasks for which is it designed. Research question 2 concerns the other important design aspect of the FacetBrowser interface, the ability to organise material into and between facets: Research Question 2: Will the organisation facilities of the FacetBrowser, such as the ability to re-organise material between facets, be exploited by users? This re-organisation of material is also central to the FacetBrowser design, and the utilisation of the re-organisation facilities in the FacetBrowser interface would be an indication that the FacetBrowser can aid users when searching. To investigate these two research questions, we created a baseline version of the FacetBrowser interface which aped the tabbed behaviour of current web browsers, functionality which is likely to be familiar to most users. This baseline interface (described in Section 5.1), restricts the visualisation of facets by allowing only one to be viewed at a time, 493

6 selected via the tabs. Similarly the re-organisation of material is also restricted in the baseline interface. For research question 1, three different measures of exploration were defined: the number of shots explicitly marked as relevant by the user; the number of shots played using the video player; and the number of queries executed. All three of these measures aim to estimate the degree of exposure of the user to the collection of the whole, where we assume that users who carry out more searching, play more shots, and mark more shots relevant, have had a greater exposure to the collection s content, and have therefore explored the collection to a greater degree. Our hypothesis for research question 1 is: Hypothesis 1: user s will explore the video collection to a greater degree with the FacetBrowser interface than the baseline interface (mark more shots as relevant, play more shots in the video browser, and execute more searches) Research question 2 is a more open ended question than the first. Direct usage comparisons of re-organisation functionality with the baseline interface is not possible. One potential measure of organisation can be considered, however, which is the number of facets created. Both the baseline and FacetBrowser interfaces allows the user to structure their search into facets, represented by either tabs or FacetBrowser panels. A greater number of facets may represent a greater ability of the user to structure their searching, and therefore the increased utility of the interface under these conditions. Hypothesis 2 for research question 2 is: Hypothesis 2: Users will create more facets using the FacetBrowser interface than the baseline interface. We assume that a user who defines more facets is making a greater effort in structuring their information seeking task, and that this effort will be reflected in the number of facets created. In addition to this hypothesis, we also track the usage of the different re-organisation facilities which the user can take advantage of in the FacetBrowser interface, in order to get a better picture of how these facilities are used. 5.1 Baseline interface The baseline interface created models the tabbed browsing behaviour of many current web browsers, such as Mozilla Firefox 3 and Microsoft s Internet Explorer version 7 4.This system is motivated by the popularity of the tabbed model of browsing, and it s ability to allow users to express multiple searches in parallel, albeit without the organisational and overview features of the FacetBrowser interface. A screenshot of the tabbed interface is shown in Figure 3. Each tab of the baseline interface is similar to an individual panel of the faceted interface, where each panel takes up the whole of the user s search screen, and is associated with a tab. A similar query box, history functionality, list of relevant shots, and list of results is shown on the screen. Shots can be dragged and dropped between these areas as in the FacetBrowser interface, although unlike the FacetBrowser interface, shots found in one tab cannot be copied to a different tab. A single button on the top left of the display Figure 3: A screenshot of the baseline interface allows a user to create new tabs, which appear on the right hand side of the list of tabs. These tabs cannot be renamed, and cannot be deleted, unlike in the FacetBrowser interface. 5.2 Collection and tasks The TRECVID 2006 dataset was used for the evaluation [16]. The vast majority of this data is news programming, although there are also some music and entertainment programs. The videos are broadcast in three different languages, English, Chinese, and Arabic, and automatic speech recognition transcripts of all videos are also provided as part of the data set along with automatic translations to English. Two tasks were defined, given in the Appendix, aiming to reflect two separate broad user needs. These were created following the situated work task framework described by [1]. A situated work task has two parts: a work task situation, and an indicative request. The work task situation describes a task scenario for the user, including the larger work context in which the search is taking place. The indicative request defines example requests for the task. Task A is the more open of the two tasks, and asks the user to discover material reflecting international politics at the end of 2005 (the period of time covered by the TRECVID 2006 data). Task B asked for a summary of the trial of Saddam Hussein to be constructed, including the different events which took place and the different people involved (such as the judge). Both of these tasks were defined to be open and broad, in keeping with the aims of the FacetBrowser interface. 5.3 Procedure A within-subject design was used for the study, each user carrying out a task with either the FacetBrowser or baseline tabbed interface. Twenty four subjects took part in total, 10 female and 14 male. Most subjects were students at Glasgow university, with all but three having university degrees or higher. All had good English (nine were native speakers), and seven subjects were also native Chinese speakers. None of the users could understand Arabic, the third of the languages of the test collection. The average age of the subjects was 26, with the youngest subject being 19, and oldest 43. All users stated that they used the web and standard textual web search engines at least once a day, with the exception 494

7 of a single user who stated they did not use search engines. Each participant read and signed a consent form before being briefed on the overall structure of the experiment. They were then given a pre-experiment questionnaire to fill in. Before a task was carried out the user was given a 10 minute, pre-recorded tutorial on the interface being administered. Two tutorials were created, one for each interface, both of which were an annotated recording of a user s session with the interface, and both of which lasted the same length of time. Once the tutorial was finished, the user was then given a further five minute training period in which he or she could user the interface, and execute queries of their choice. At this training stage each user was guided, when required, by the experimenter. When the system training was finished, the user was presented with the search task. After 30 minutes, the user was then informed that the task was finished, and the final results and logs would be stored. In addition to the logs recorded by our search system, a screen reading program was also used to record the user s action with the interface, although the analysis of this data is not reported in this paper. All but two users took the full 30 minutes for all tasks. On completion of the task, the user was then presented with a post-task questionnaire. After the first task was finished, the user was given a chance to take a break before starting the second task, which would then proceed in the same way as the first. After both tasks were administered, the user would be given a final, short, exit questionnaire, before being thanked for their help. All users were paid 10 pounds for taking part (roughly $20). 5.4 Results Research question 1 from Section 5 has an associated hypothesises 1 based on three measures of user exploration within a video collection: the number of shots marked as relevant, the number of shots played, and the number of searches carried out. Tables 1, 2, and 3 show the mean and standard deviation for these three measures, by task and system. NotethatforTable3,wetakethesumofthree interface events: shot play, where the user has clicked the play button on a shot which has been marked relevant or is a search result; next shot on the video player, which will play the next shot in the video; and previous shot which will play the previous shot in the video. An ANOVA indicated a significant interaction between shots marked relevant and system (F=4.658, p < 0.05), while no significant interaction was found between number of searches and number of shots played with system. It must be noted, however, that the numbers of shots which were marked by users varied greatly - for example, the median values for Task A in Table 1 are 26 (range 10-48) and 26.5 (range ). From the tables, it can be seen that the two tasks turned out to have very different characteristics. Table 1 shows that users working on Task A with the FacetBrowser interface found on average more shots than users working with the tabbed interface, with the proviso that there was a great variation between the user who marked the most from the user who marked the least; for Task B many more shots were found with the baseline interface. In Table 2 there is a trend for users to execute less queries with FacetBrowser, although this difference is not significant. There is, on the other hand, a larger difference between the degree of searching between the two tasks, with users searching more, on average, on Task A than Task B. Lastly, in Table 3, there is a very slight increase in the number of shot s played when using the FacetBrowser interface on Task A, and a slight decrease on Task B (again, this is not statistically significant). Table 1: Number of shots found (Mean and SD) Tabbed Faceted Task A (12.87) (32.99) Task B (34.40) (24.69) Table 2: Number of searches (Mean and SD) Tabbed Faceted Task A (9.43) (4.31) Task B 8.58 (6.20) 7.83 (5.15) Table 3: Number of shots played (Mean and SD) Tabbed Faceted Task A (49.89) (72.01) Task B (50.99) (64.89) Research question two is concerned with the degree of organisation carried out by users, and has an associated hypotheses 2. Table 4 shows the average number of facets created by users and existing at the end of the users session s, by task and system. For the FacetBrowser interface, the number of facets was taken to be the number of panels created by the user and present at the end of the task (i.e. this does not include deleted panels). For the tabbed interface, the number of tabs opened by the user is counted as the number of facets. Using an ANOVA, no significant variation was found between the number of facets and system. A similar situation is shown in this table as with the previous results, with more facets being created by users using FacetBrowser on Task A, with the number of facets being roughly equal for Task B. Within the FacetBrowser interface, there are three unique actions which were tracked, and which can be considered as organisational: moving a panel either left of right within the sequence of panels (events MovePanelLeft and MovePanelRight), and providing a panel with a keyframe image (event ChangeKeyframe). A fourth action, that of providing a panel with a name, was unfortunately not recorded in the interface logs. The number of MovePanelRight events in all user logs is 2, while MovePanelLeft was used a total of 11 times across all logs, by four different users. One user, user 6, accounted for most of this usage. In total, 10 users gave their panels keyframe images, across all users this accounts for 22% of the facets created, although as shown in Figure 4 this varied wildly by user. Figure 4 shows the number of facets created and still existing at the end of the user s task (in light grey) next to the number of those facets which were given keyframe images (in dark grey). It can be seen that some users, such as users 8 and 9 give keyframe images to a majority of their facets while others, such as user 10, do not provide any keyframes whatsoever. 495

8 Table 4: Number of facets created (Mean and SD) Tabbed Faceted Task A 3.33 (1.77) 5.50 (2.88) Task B 3.75 (2.77) 3.50 (1.08) Average number of shots Task A Average number of shots Task B Video player Same results Other results Other facets History Tabbed Faceted Tabbed Faceted Figure 4: The number of facets created, and number given keyframes, for each user Figure 5: Sources of shots marked relevant We also looked at where user s found their marked shots for example, whether a shot came from the video player, the search results, the history mechanism, etc. or the search results or relevance list of another facet. We were able to identify five different sources: From the video player: shots which were dragged from the video player onto a facet From the facet s own search results: shots which were dragged to the relevance list of a facet from that facet s ownsearchresults From a different facet s search results: shots which were dragged from the search results of one facet to the relevance list of a different facet. For example, a result in facet 3 can be dragged to the relevance list of facet 1. This functionality is not supported by the tabbed interface From another facet s relevance list: shots which were dragged from the relevance list of a different facet, for example, the user dragging a shot from panel 5 to panel 2. This functionality is not supported by the tabbed interface From the history mechanism: shots which were dragged from the history panel back onto the facet s relevance list. This is equivalent to an undo, where a user has perhaps accidentally deleted a shot, and then wishes to undo this action The source of each shot was found as follows: first, the list of final results selected by the user was computed from the logs. For each of these results, the log file was then searched backwards for the last action which resulted in the shot being placed in the facet s relevant shot list. When found, the source of the shot was then extracted from the action. This method will give precedence to the last action which selected the shot, ignoring any previous actions which may have selected the same shot for that particular facet. Figure 5 gives the results of this method, split into two for the two different tasks. It can be seen that the majority of the shots marked as relevant are selected from the video player, for both tasks. On the FacetBrowser interface, the next most popular method of finding shots was moving the shot from a different facet s relevance list. For the tabbed interface, selecting search results was, as could be expected, the next most popular way of selecting relevant shots. The history functionality was only used a single time by one user during the 24 runs. Table 5: Number of unique terms used (Mean and SD) Tabbed Faceted Task A (7.82) (5.87) Task B (7.84) 8.25 (3.54) Lastly, we looked at the query vocabulary used when searching, counting the number of different unique terms used in queries. Table 5 shows the average number of unique query terms used in the two interfaces, by task. A large task difference can be seen, with more terms used in Task A. No significant difference between the number of unique terms used and system was found, although when using the FacetBrowser interface user s did tend to use a more restrictive search vocabulary. This backs up Table 2 where similar results were found - more searches were executed on task A, with a trend (not significant) for users to execute less searches with the FacetBrowser interface. 5.5 Discussion The results presented in section 5.4 show some surprises, perhaps the greatest of which is the difference between the 496

9 two tasks. It was intended, at the experimental design stage, for both of these tasks to represent broad, complex, search needs, to reflect the usage scenarios outlined in Section 1.1. As it turned out, the two tasks in fact appeared to operate very differently, with Task A being much broader than Task B. This is suggested by Table4wheremorefacetswere created with FacetBrowser on Task A, but not Task B. To check this, we looked at the level of agreement between the sets of found shots, making the assumption that there will be less agreement between users for broader tasks which require a greater degree of interpretation. For Task A, 634 unique shots were found by all users, of which 94 were selected by 2 or more users (almost 15%). For Task B, 557 unique shots were found, 259 of which were selected by 2 or more users (46%). This greater agreement between users for Task B is consistent with Task B being the more focused task. Looking at the results of the measures of exploration (Tables 1 to 3), we find mixed results: for Task A, the broader task, some users were able to find more shots using the FacetBrowser interface, while there was also a negligible increase in the number of shots played by users on Task A. The number of searches executed, however, were actually slightly reduced when compared to the tabbed baseline, suggesting that while users were possibly finding more they were searching less. This reduction in searching is also reflected in the size of the search vocabulary on Task A, shown in Figure 5, possibly indicating that the organisation facilities of the FacetBrowser interface allowed users to search less. This organisation was one of the aims of the interface, and is backed up by Figure 5, where copying shots from other facets is the second most likely source of shots for users. Figure 5 however, shows clearly that it is the video browsing functionality which accounts for most relevant results, probably due to user s checking the relevance of a shot by playing it, before marking it. Unfortunately, we were not able to tell whether results found using the video player were transfered to multiple, various facets. One encouraging aspect from the study was the use of the organisational features. Figure 4 shows the number of facets created by users, alongside the number of those facets which were given keyframes, showing that 10 users utilised the keyframe functionality to represent panels in the FacetBrowser interface. For those 10 users, 73% of the panels were given a keyframe image, suggesting that users who did use this function, used it allot. All users but one created facets with FacetBrowser, the median number of facets created being 4, and with one subject using 12 facets. The flip side to the reasonably encouraging results on Task A, however, is the decrease in the exploration on Task B: users found considerably less shots using the FacetBrowser interface on this task than the baseline tabbed interface. A possible reason for this decrease in performance is the design of the respective interfaces. On a tabbed interface, the entire screen can be used for a single search, while on the FacetBrowser interface, each panel fills a much smaller section of the screen, typically only one quarter of the screen space. The larger screen space of the tabbed interface provides for a less constrained searching experience on a single search, with the user able to see a greater number of search results. The results here suggest this extra space, especially coupled with the relatively low performance of content based video retrieval systems, provides a considerable advantage when the task being undertaken is not multifaceted, as appeared to be the case with Task B. In such tasks, the overview and organisation facilities provided by the FacetBrowser interface do not provide enough of a benefit to compensate for the lack of space for search results. On Task A, this disadvantage of the FacetBrowser does appear to be compensated for by it s extra benefits. 6. CONCLUSIONS AND FUTURE WORK In this paper, we have presented a new faceted search interface which considers a facet as a sub-topic of a larger search task. The system is designed to provide users with a flexible content-based multimedia search system, where complex and exploratory search needs can be sketched out by the user, and stored for future use by others. A user study was carried out, which provided evidence that the FacetBrowser interface can potentially enable users to explore more of a collection to a greater degree than a baseline tabbed interface, for broad multi-faceted search tasks. However, the results also show that the FacetBrowser can reduce exploration when the search task is relatively narrow. The organisational facilities provided by the FacetBrowser were used by all but one user, with the re-organisation of material between facets being the second most common source of relevant shots after the video player. Based on the results of this study, there is an ongoing effort to improve the FacetBrowser system. This has started with the implementation of focus functionality allowing the user to expand a single FacetBrower panel to cover the whole screen, aiming to integrate the advantages of the tabbed interface into the FacetBrowser interface. Secondly, given the importance of the video browsing functionality, we are at the moment working on an updated video browser, which aims to allow a user to quickly view and interact with a video in a much more flexible manner than that provided in the system described here. Thirdly, we hope to carry out further formal and informal evaluations of the interface with media professionals, to further investigate how search can be better integrated into real-world work tasks. Additionally, FacetBrowser provides considerable scope for utilising search context: each panel in the interface can be thought of as being contextualised by the sequence it is part of. To this end, the work reported in [5] further analyses the log files generated in this study, aiming to improve the search results of individual facets by using the relevance information of neighbouring facets, while [22] investigates the automatic generation of facets through clustering. 7. ACKNOWLEDGMENTS This research was supported by the European Commission contracts, FP SALERO and FP K-SPACE. 8. REFERENCES [1] P. Borlund. The iir evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8(3), [2] D. J. Harper and D. Kelly. Contextual relevance feedback. In IIiX: Proceedings of the 1st international conference on Information interaction in context, pages , New York, NY, USA, ACM. [3] M. Hearst. Next generation web search: Setting our sites. IEEE Data Engineering Bulletin, Special issue on Next Generation Web Search, September

10 [4] W. Hersh and P. Over. Trec-8 interactive track report. In The Eighth Text REtrieval Conference (TREC 8), [5] F.Hopfgartner,T.Urruty,R.V.N,Gildea,and J. Jose. Exploiting log files in video retrieval. In Proceedings of JCDL 2008, [6] A.Kerne,E.Koh,S.M.Smith,H.Choi,R.Graeber, and A. Webb. Promoting emergence in information discovery by representing collections with composition. In Proceedings of the 6th ACM SIGCHI conference on Creativity & cognition, June [7] G. Marchionini. Exploratory search: from finding to understanding. Commun. ACM, 49(4):41 46, [8] G. Marchionini and B. Brunk. Toward a general relation browser: A gui for information architects. Journal of Digital Information, 4(1), [9] G. Marchionini and G. Geisler. The open video digital library. dlib, 8(12), December [10] M. Nakazato and T. S. Huang. Extending image retrieval with group-oriented interface. In Proceedings of IEEE ICME2002, [11] M. Nakazato, L. Manola, and T. S. Huang. Group-based user interface for content-based image retrieval. In Proceedings of Advanced Visual Interfaces (AVI 2002), [12] M. Nakazato, L. Manola, and T. S. Huang. Imagegrouper: a group-oriented user interface for content-based image retrieval and digital image arrangement. Journal of Visual Languages and Computing, 14(4): , August [13] P. Over. Trec-5 interactive track report. In The Seventh Text REtrieval Conference (TREC 7), [14] P. Over. Trec-7 interactive track report. In The Seventh Text REtrieval Conference (TREC 7), [15] P. Over, G. Awad, W. Kraaij, and A. F. Smeaton. Trecvid 2007 overview. In TRECVid Text REtrieval Conference TRECVid Workshop, [16] O. P, T. Ianeva, W. Kraaij, and A. F. Smeaton. Trecvid an overview. In TRECVid Text REtrieval Conference TRECVid Workshop, [17] M. C. Schraefel, M. Wilson, A. Russell, and D. A. Smith. mspace: improving information access to multimedia domains with multimodal exploratory search. Commun. ACM, 49(4):47 49, [18] A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and trecvid. In MIR 06: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pages , New York, NY, USA, ACM Press. [19] J. Urban. An Adaptive Approach for Image Organisation and Retrieval. PhD thesis, Department of Computing Science, University of Glasgow, UK, January [20] J. Urban, X. Hilaire, F. Hopfgartner, R. Villa, M. Jose, S. Chantamunee, and Y. Gotoh. Glasgow university at trecvid In TRECVid Text REtrieval Conference TRECVID Workshop, MD, USA, National Institute of Standards and Technology. [21] J. Urban and J. M. Jose. Ego: A personalised multimedia management and retrieval tool. International Journal of Intelligent Systems (Special issue on Intelligent Multimedia Retrieval), 21(7): , July [22] T. Urruty, F. Hopfgartner, R. Villa, N. Gildea, and J. Jose. A cluster-based simulation of facet-based search. In Proceedings of JCDL 2008, [23] P. Yee, K. Swearingen, K. Li,, and M. Hearst. Faceted metadata for image search and browsing. In Proceedings of ACM CHI ACM, April Appendix: Experimental Tasks Task A: Reflections on international politics at the end of 2005 Imagine you are a student working towards a media studies degree at the Open University, during the last few months of As part of your 2rd year politics and the media course, you have to produce a video program which presents a review of international politics at the end of 2005, as reported on the television news. You must now find material for this video presentation, to use in illustrating the important people, events, meetings, and situations which have occurred. Your task is to find, using the system, shots which reflect the important political events and people during the end of Material to find may include shots of politicians, speeches, interviews, panel discussions and in particular shots linking the different people and events together. For instance, searches may include famous leaders such as George Bush or Tony Blair, and include thematic situations in which they are involved together (for example, the in war in Iraq is of common relevance to both of the above leaders). Other international organisations such as the UN and EU, and shots illustrating events involving these organisations are also of significance to your video report. Choose as many shots from as many different videos as possible. If any individual video story is split across multiple shots, please mark all shots you feel are necessary in the video. Don t worry about the order of your marked shots you may assume that a separate video editing package is available which would allow you to edit your found shots into a summary, a task which will not be carried out in this experiment. Task B: A Summary of the trial of Saddam Hussein You work as a researcher for a television company, and as part of your remit, you work on a bi-monthly news program, which summarises and reviews major news stories. The latest episode of the program is to summarise the trial of Saddam Hussein, occurring during the final months of 2005 From the given database containing news from the final months of 2005, discover shots of the main news events which describe the story of the trial. This should, as far as possible, include all relevant material, including discussions and interviews about the trial reflecting contemporary views from all sides. For instance, aspects include the victims of his regime, his capture, allegations in court (e.g. torture) and shots of the other court players such as the judge. (The final paragraph was the same as for task A) Last paragraph is as for Task A 498