A User-Centric Media Retrieval Competition: The Video Browser Showdown

Size: px

Start display at page:

Download "A User-Centric Media Retrieval Competition: The Video Browser Showdown"

Neil Wilson
6 years ago
Views:

1 Preprint of article (author s version) to appear in IEEE MultiMedia (MM) issue (Oct-Dec. 2014), IEEE 1 A User-Centric Media Retrieval Competition: The Video Browser Showdown Klaus Schoeffmann, Alpen-Adria-Universität Klagenfurt Abstract The Video Browser Showdown is an international competition in the field of interactive video search and retrieval. It is held annually as a special session at the International Conference on MultiMedia Modeling (MMM). The Video Browser Showdown evaluates the performance of exploratory tools for interactive content search in videos in direct competition and in front of the audience. Its goal is to push research on user-centric video search tools including video navigation, content browsing, content interaction, and video content visualization. This article summarizes the first three editions of the competition (VBS2012-VBS2014). Index Terms Video Browsing, Video Interaction, Video Retrieval V I. INTRODUCTION ideo browsing is an interactive process where a user inspects the content of a single video or video collection. This exploratory search approach is required when a user wants to know what content or information is contained. Moreover, it can also be used for retrieval scenarios where users want to find a particular content but cannot describe it through a query ( I will know it when I see it ). During the browsing process, however, based on the learned knowledge while browsing, a user may want to perform automatic content retrieval to filter content or to find similar content. Video browsing tools help a user to efficiently solve these tasks and employ various techniques and methods from several research fields in multimedia. Video browsing tools [1][4] use content-based indexing methods to understand the structure of the videos in order to consider it for content visualization and to provide on-demand automatic search and filtering features. content visualization methods to provide users a comprehensive overview of the content that allows to understand the content structure. content navigation techniques to allow for quick Submitted on April 4, 2014 Acknowledgements. The organization of the Video Browser Showdown has received funding from the European Union s Seventh Framework Programme (FP7/ ) under grant agreement n entitled TOSCA-MP. The content used in the Video Browser Showdown has been made available by the EBU MIM/SCAIE project. Moreover, the author would like to thank his colleague Werner Bailer from JOANNEUM RESEARCH, Graz, Austria ( werner.bailer@joanneum.at), for co-organizing the Video Browser Showdown from jumps in a video/collection. content abstraction and summarization approaches to support visualization and navigation. video surrogates to more quickly communicate the content of a video to the user. content interaction techniques to enable quick and flexible skimming through the content and to inherently support content understanding. video retrieval methods to provide different kinds of automatic query features (e.g., query-byexample and query-by-sketch). Video browsing can be considered as an interactive video retrieval approach. However, the primary focus is not on automatic querying but on content exploration. Video browsing tools provide various features but put the user in the center, who will know best what feature to employ in the current situation to satisfy a search demand [5]. Therefore, video browsing tools can cope with different user needs and also adapt to different needs of a single user, based on the current context. An important requirement to enable flexible search scenarios is to provide the user (as the search engine ) maximum insight to the video content structure. The Video Browser Showdown (VBS) is a live evaluation competition where international researchers, in the field of video browsing, evaluate and demonstrate the efficiency of their exploratory video search tools. It is hosted as an annual special session at the International Conference on MultiMedia Modeling (MMM) and contains a moderated public part, where search tasks are performed in front of the audience. In order to participate, potential teams submit an extended demo paper of their systems for peer-review. The first VBS was hosted at MMM2012 in Klagenfurt, Austria, where 8 international teams evaluated their video browsers against each other. The second edition of the VBS was held at MMM2013 in Huangshan, China, with 6 international teams. The most recent competition (VBS2014) was performed in Dublin, Ireland, also with 6 participating teams (see Fig. 1). Table I lists all participating research institutions of the last three years. In this article we review the last three VBS editions and discuss some interesting observations and findings. K. Schoeffmann is with Alpen-Adria-Universität Klagenfurt, Institut für Informationstechnology (ITEC), Universitätsstrasse 65-67, A-9020 Klagenfurt, Austria ( ks@itec.aau.at).

Preprint of article (author s version) to appear in IEEE MultiMedia (MM) issue (Oct-Dec. 2014), IEEE 2 TABLE I PARTICIPANTS OF THE VIDEO BROWSER SHOWDOWN 2012-2014.

1: The moderated public session of the Video Browser Showdown 2014 at the 20th International Conference on MultiMedia Modeling (MMM2014) in Dublin, Ireland. II.

However, instead of evaluating the automatic retrieval performance in a video collection (e.g., in terms of recall and precision) the VBS focuses on usercentric video search.

2 Preprint of article (author s version) to appear in IEEE MultiMedia (MM) issue (Oct-Dec. 2014), IEEE 2 TABLE I PARTICIPANTS OF THE VIDEO BROWSER SHOWDOWN THE ENTRIES IN BOLD FONT OUTLINE WINNERS OF THE CORRESPONDING YEARS. Research Institution Year(s) Fig. 1: The moderated public session of the Video Browser Showdown 2014 at the 20th International Conference on MultiMedia Modeling (MMM2014) in Dublin, Ireland. II. RULES The idea of the VBS was originally inspired by the VideOlympics [1], which is an international competition of video retrieval tools. However, instead of evaluating the automatic retrieval performance in a video collection (e.g., in terms of recall and precision) the VBS focuses on usercentric video search. As a consequence, automatic search tasks with textual queries are not allowed. Instead, the users have to interactively find the target segments by navigation, filtering, browsing, or other interactive query methods (e.g., query-by-sketch). Target segments of about 20 seconds duration, which do not necessarily begin and end at shout boundaries, are presented on a projected wall for each search task. The participants of the VBS first have to interpret and understand the seen information and then based on their knowledge and experience use the best way with their tool to find the target segment as fast as possible. This setting should simulate Known-Item-Search (KIS) tasks (also previously used in TRECVID [6]), where users know about a desired segment, know it is contained in a video, but do not know where to look for it. Dublin City University (DCU), Ireland 2014, 2013, 2012 Institute of Information Technology (ITEC), 2014, 2013, 2012 Alpen-Adria-Universität Klagenfurt, Austria JOANNEUM RESEARCH, Austria 2014, 2013, 2012 Information Technologies Institute / Centre for 2014 Research and Technology Hellas (CERTH), Greece SIRET Research Group, Charles University in 2014 Prague, Czech Republic National Institute of Informatics (NII), Japan / 2014, 2013 University of Information Technology (UIT), Vietnam / The Graduate University for Advanced Studies, Japan / University of Science, Vietnam Université de Mons, Belgium 2013 France Telecom Research & Development 2013 Beijing, P.R. China / Beijing University of Posts and Telecommunications, P.R. China School of Computing, National University of 2012 Singapore, Singapore Institut Télécom, Télécom SudParis, ARTEMIS 2012 Department / Alcatel-Lucent Bell Labs France / Mines ParisTech, France Technical University of Catalonia (UPC), Spain 2012 Institute of Informatics Systems (ISYS), Alpen Adria-Universität Klagenfurt, Austria by the teams for a task. More precisely, the server checks if the submitted segment (which could be a single frame as well) is within the desired target segment. If so, a thumbnail with green border is shown on top of the column of the team in the overview visualization, together with the achieved points for the task (cf. Fig. 2). If the submitted segment is outside the target segment a thumbnail with red border is shown and the counter of false submission is incremented for the team. A successful submission is rewarded with points, which are based on the number of previous false submissions and the required search time. At the end of the competition, the winner is selected according to the achieved total points and receives a certificate for the best performing video browser. The bold entries in Table I outline the winning teams for the last three years. Fig. 2: The VBS Server shows a live overview of submissions (green=correct, red=wrong) and achieved points, in an own column for each team (total on top of each column). The second line of the window shows the IDs of the videos tested so far in the session. The equipment of the participants is connected to the VBS Server via a dedicated network. The server is responsible for evaluating and visualizing search results that were submitted III. DATA SET, TASKS, AND PERFORMANCE The data set used in the VBS consists of video files with an average duration of 1 hour in PAL resolution, encoded with H.264/AVC. The size of the data set varies over the years (2012: 30 videos, 2013: 10 videos, 2014: 76 videos). The data set was made available to the participating teams about one month before the competition; the search tasks were randomly selected and presented on site (i.e., during the competition). The competition is split into an Expert Run and a Novice Run. While in the Expert Run the developers themselves act as searchers, volunteers from the audience are selected as searchers in the Novice Run. The rationale behind the Novice Run is to inherently target on video browsing tools that are easy to use. It should be noted, however, that the audience of VBS consists of participants of the MMM conference. Thus, the novices are actually researchers in the field of multimedia. In the first two editions, the VBS focused on visual KIS

3 Preprint of article (author s version) to appear in IEEE MultiMedia (MM) issue (Oct-Dec. 2014), IEEE 3 Year TABLE II KNOWN-ITEM-SEARCH TASKS IN THE VBS (VISUAL KIS IN SINGLE VIDEOS ONLY) Number of Tasks Average Task Solve Ratio Average Task Solve Time in Seconds / / / (s.d / 16.77) / / / (s.d / 11.13) / / / (s.d / 14.65) tasks within single videos only. Hence, for each search task the corresponding video file name was also presented when the target segment was presented at the beginning of a task. Moreover, all search tasks were performed in a public onehour session. For each task the participants were given 3 minutes to solve the task. In the latest edition (VBS2014) we extended the Video Browser Showdown to a special workshop running over 5 hours, where only the last hour was public to the audience. This enables testing more search tasks in order to collect more data for a later statistical analysis of the search performance. Moreover, in addition to KIS tasks in a single video, which were issued as visual queries, we also tested KIS tasks in a video collection as well as two different kinds of query formulation (visual and textual presentations of the target segment). Therefore, VBS2014 consisted of the following sessions: Visual KIS tasks in single videos (same as in VBS2012 and VBS2013) Textual KIS tasks in single videos: instead of presenting the target segment as a video clip, a textual description is shown at the projector. Visual KIS tasks in a video collection: the target segment is randomly selected from a subset of 20 video files but the corresponding video file name is not mentioned for the task. Textual KIS tasks in a video collection: similar to the scenario above but instead of the target clip, a textual description of it is shown at the projector. Table II shows some details of the visual KIS tasks in single videos (first entry in the enumeration above) performed in all three editions of the VBS. While in 2012 we tested only 8 tasks with experts and 6 tasks with novices for the visual KIS in single videos session, we increased the number of tasks to 10/6 (experts/novices) in 2013 and to 20/10 in Additionally, in 2014 we tested 10 textual KIS tasks in single videos as well as 10 visual and 10 textual KIS tasks in a video collection (a detailed evaluation is in work). The third column of Table II shows the average task solve ratio, which is the average percentage of tasks solved by all participating teams. We can see that for the Experts Run this measure strongly increased over all three years from 0.61 in 2012 to 0.92 in 2014, which is a remarkably high value. Although significantly more tasks were performed for the visual KIS in single videos session in VBS2014, most of the teams scored in every task. More specifically, 14 out of 20 tasks for the experts could be solved by all teams. Hence, in terms of task solve ratio the video browsers used in the VBS became significantly better over the years when used by the experts (i.e., developers) themselves. However, from the average task solve ratio for the Novice Run (only 0.77 in 2014) we can see that there is still potential to improve the usability of the systems, such that also third-party users can efficiently use it. During the VBS the performance of each team is logged at the VBS Server for each task. This log data are used for a detailed evaluation of the performance of all teams, which is published in a joint effort a few months after the VBS has taken place. This is an important aspect to provide detailed insight on the performance of proposed systems and should foster improved versions or adaptions in future editions of the VBS. Additionally, it gives the authors of the systems the possibility to get a comparative evaluation among all teams in order to further improve their systems. Evaluations of the first two editions can be found in [1] and [4]. An evaluation of VBS2014 is still in work but will be submitted later this year. The last column of Table II shows the average task solve time (and standard deviation) achieved by all teams over the years. It can be seen that in the latest Video Browser Showdown competition the experts were clearly faster in solving tasks and also more steady than in previous years, when taking a look on the average standard deviations. This is, however, not fully true for the novice users, where the average task solve time in VBS2014 is similar to the one in VBS2012 and more unstable than in previous years (standard deviation 16.77). This result reflects the need for video browsing systems with simple but powerful interfaces, which can also be used by non-experts efficiently. IV. THE IMPACT OF THE USER In all three editions of the VBS performed so far, many different systems have been tested. Some systems use sophisticated content analysis methods with frame-based indexing of color, texture, visual concepts (e.g., faces, cars, landscape etc.), and salient points in images. A few systems focus mainly on rich content visualization and flexible navigation in videos. Others focus on convenient content interaction with special devices or touch-enabled tablets. Interestingly, the winners of VBS2012 [1] use a system without any content analysis (see third picture in the first row of Fig. 3). Instead, their system relies only on the knowledge of the user and his/her ability to quickly navigate in videos and filter for relevant content. Their system provides parallel playback windows with a combined seeker-bar for simultaneous content navigation in different parts of the video. In VBS2012 their system could easily outperform other systems with sophisticated content indexing methods. This result demonstrates the highly important role of the user in video browsing systems. Similarly, the winning system of VBS2014 relies on the ability of the user to draw appropriate sketches to query for relevant content and to quickly browse the result lists (topleft picture in Fig. 3). Their system is built around a content indexing method with color-based signatures, which is used for describing the sample frames from the videos as well as the sketch created by the user. Hence, after the user has drawn the sketch, the signature of it is compared with all the signatures from the videos and a result list is created, where

Preprint of article (author s version) to appear in IEEE MultiMedia (MM) issue (Oct-Dec. 2014), IEEE 4 the results are ranked by distance. This is similar to methods used in video retrieval tools.

4 Preprint of article (author s version) to appear in IEEE MultiMedia (MM) issue (Oct-Dec. 2014), IEEE 4 the results are ranked by distance. This is similar to methods used in video retrieval tools. In VBS2013 the situation was slightly different. Due to the fact that the data set mainly consists of news videos, a few systems in 2013 implemented face detectors. Since several queries of VBS2014 contained close-ups of persons or showed taking peoples, these teams could efficiently filter for faces and therefore narrow down the visual search of the user to a handful of segments. The system of the winners in 2013 could also take advantage of a face filtering approach (second picture in first row of Fig. 3). In order to better assess the performance of the video browsing systems used in VBS2013, we performed a baseline study with all expert tasks used in 2013 [4]. For that purpose we invited 16 non-expert users (master students from different disciplines) to use an HTML5 video player to find the same 10 target segments by simple seeker-bar navigation. Although the atmosphere in this study was different to the VBS competition, where several teams compete against each other, the settings (data set, tasks, scoring) were the same. We compared the achieved search time to the results of the experts in VBS2013 [4]. These results show that even with a simple video player and rather linear search behavior most users simply navigated over the video by quickly dragging the seeker-bar from the beginning to the end the users of the study were only about 50% slower in terms of average search time than the experts in VBS2013. This is a remarkable result when keeping in mind that the users of this study were real nonexperts, although self-declared daily computer users, that used only a seeker-bar for quickly skimming over the video content (i.e., no information from automated content analysis, no advanced visualization or navigation features). We conclude from that finding that the tools in the Video Browser Showdown should focus more on the users and their ability for quick and flexible content interaction, as also stated in [5]. Video browsing tools should support the user in translating his/her knowledge to actual interaction that could help to find the needed content. For example, in the baseline study we could observe that for a target segment showing a weather forecast, which was contained in a 1-hour news video, most of the users immediately started their navigation-based search in the last part of the video and, hence, could quickly find the desired segment. Interestingly, many video browsing systems do not contain such a simple but effective feature for allowing the user to pursue his/her intention. Fig. 3: Impressions of the Video Browser Showdown over the last three years. The first three pictures in the first row show screenshots of the winning systems from 2012, 2013, and 2014, respectively. High resolution pictures as well as links to demos of these systems are also available on the website of the VBS: V. THE FUTURE OF VBS We plan to continue the Video Browser Showdown competition as a special workshop hosted at the International Conference on MultiMedia Modeling (MMM). The next VBS will be held on January 4th, 2015 in Sydney, Australia, in conjunction with MMM2015. The deadline for extended demo papers will be in fall We plan to change the name to Video Search Showcase (VSS) and to extend the rules in order to allow more teams to participate in the competition. Future editions will also include plenary presentations of the systems at the beginning of the public part, accompanied by posters, to give the audience more information about the details of the video browsers. While the VBS is also an entertaining event (see impressions in Fig. 3), the long-term goal is to push research on highly interactive video retrieval tools that enable quick retrieval tasks in videos. We will also keep the special focus on ease of use, such that non-experts could efficiently use the proposed video browsers as well. VI. CONCLUSION In this article we have described the idea and rules of the Video Browser Showdown competition and summarized the results from the last three editions. Overall, 20 systems from different international research institutions competed in the VBS over the last three years. While a few systems were improvements of systems used in previous editions, also a few new teams joined for every year with remarkable performance, like in 2014 where a new team won the competition. Although the evaluation of all sessions of VBS2014 is not completed yet and first indications show well performance, we assume that there is still great potential for improving the performance of the video browsing systems. Our baseline study with the tasks of VBS2013 has shown that in terms of average search time the expert systems were only twice as fast as non-expert

5 Preprint of article (author s version) to appear in IEEE MultiMedia (MM) issue (Oct-Dec. 2014), IEEE 5 users with a simple video player. We expect that the performance of the video browsers could be further improved if more focus is put on the user as well as on easily understandable and simple-to-use features, which are especially important in the Novice Run. We will continue hosting VBS as a special workshop at the MMM conference in order to advance research on the interesting topic of interactive search in videos and we hope to increase the number of participating teams too. REFERENCES [1] W. Bailer, K. Schoeffmann, D. Ahlström, W. Weiss, M. del Fabro, Interactive Evaluation of Video Browsing Tools, Advances in Multimedia Modeling, Springer Berlin Heidelberg, LNCS, vol.7732, pp.81-91, Jan., 2013 [2] C. Snoek, M. Worring, O. de Rooij, K. van de Sande, R. Yan, A. Hauptmann, Videolympics: Real-time evaluation of multimedia retrieval systems. Multimedia, IEEE, vol.15, no.1, pp.86-91, Jan.- Mar., [3] K. Schoeffmann, F. Hopfgartner, O. Marques, L. Boeszoermenyi, J.M. Jose, Video browsing interfaces and applications: a review, SPIE Reviews), SPIE, vol.1, no.1, pp , Mar., 2010 [4] K. Schoeffmann, D. Ahlström, W. Bailer, C. Cobârzan, F. Hopfgartner, K. McGuinness, C. Gurrin, C. Frisson, D.-D. Le, M. del Fabro, H. Bai, W. Weiss, The Video Browser Showdown: a live evaluation of interactive video search tools, International Journal of Multimedia Information Retrieval, Springer London, ISSN , pp.1-15, Dec., 2013 [5] M. Worring, P. Sajda, S. Santini, D.A. Shamma, A.F. Smeaton, Q. Yang, Where Is the User in Multimedia Retrieval?, Multimedia, IEEE, vol.19, no.4, pp.6-10, Oct.-Dec., 2012 [6] A.F. Smeaton, P. Over, W. Kraaij, Evaluation campaigns and TRECVid, Proc. of the 8th ACM International Workshop on Multimedia Information Retrieval, ACM Press, New York, NY, USA, pp , 2006 Klaus Schöffmann is assistant professor at the Institute of Information Technology (ITEC) at the Alpen-Adria- Universität Klagenfurt, Klagenfurt, Austria. His research interests include interactive video retrieval, multimedia applications, and visual content analysis. Together with Werner Bailer from JOANNEUM RESEARCH, Graz, Austria, he organized the Video Browser Showdown for the last three years. Contact him at ks@itec.aau.at.

The Multi-Stripe Video Browser for Tablets

The Multi-Stripe Video Browser for Tablets Marco A. Hudelist* and Qing Xu** *Klagenfurt University, Universitaetsstr. 65-67, 9020 Klagenfurt, Austria **School of Computer Science and Technology, Tianjin