Short Video Metadata Acquisition Game

Transcription

1 Short Video Metadata Acquisition Game Ales Masiar Institute of Informatics and Software Engineering Slovak University of Technology Bratislava, Slovakia Jakub Simko Institute of Informatics and Software Engineering Slovak University of Technology Bratislava, Slovakia Abstract Gathering proper descriptive metadata for multimedia resources is nowadays essential for effective information processing, recommendation and personalization. And, we still need to employ human workforce in crowdsourcing scenarios for solving particular metadata acquisition tasks. In this paper we present a human computation game, which acquires metadata for short videos found in the Vine social media service. Player s task is to watch a given short video (without any further description) and formulate a search query, which will make that video appear in top results of the search. The player queries are then processed for keywords characterizing the videos. Our experiments show, that these keywords are both correct and from the great part, represent new information atop existing video descriptions. I. INTRODUCTION Number of multimedia resources shared through social websites is growing rapidly every day. It is important to have the resources described and organized using metadata, so they can be easily searched and recommended. There are many video sharing sites and some of them are based on sharing the videos via mobile devices, where users usually do not have much time and proper ergonomic tools for describing the videos. Moreover, many users do not consider proper video description to be important. This leads to videos being lost among others, because they are effectively not searchable. The goal of our work is the metadata acquisition for short videos a very popular format of user-generated content. In particular, we work with the videos of Vine social media service 1, where users share the content as video-tweets. They usually do so using mobile devices and do not pay much attention to providing useful descriptions or metadata. To achieve our goal, we opted for a crowdsourcing approach and specifically, through a human computation game. The crowdsourcing has emerged as an alternative to automated and manual (expert) approaches to metadata acquisition. It is a compromise between the lack of quality (associated with automated approaches) and lack of scalability (attributed to manual-expert work). Crowdsourcing utilizes the human computation work of large groups of people. Through platforms like CrowdFlower or Amazon Mechanical Turk, it can produce high quantities of results of sufficient quality. Yet crowd-based approaches still need to incentivize their workers, who often demand monetary rewards. To find alternative incentive schemes a specific branch of crowdsourcing approaches, known as human computation games (or sometimes: games with a purpose), has emerged. They are specially designed 1 games, where players generate useful artifacts as a by-product of the gameplay. Applying the principles of human computation games, we created the Vine Search Game 2 a search game that harnesses manpower for acquisition of keywords to short videos. In this game, players formulate search queries that best match videos previously shown to them. In a typical game round, a short video is firstly shown to a player. Then, the player (repeatedly) formulates a search query with a goal of getting the given video into top N results of the search (a winning condition of the game). A natural choice for the players in such setting is to use query words that distinctly characterize the given video. This allows us to process the player-generated queries for video-describing keywords after the game. As we show in our experiments, keywords acquired this way are indeed correct. For execution of the queries, the game uses indexes built from existing (but vastly imperfect and incomplete) video descriptions and metadata found in the original Vine service data. They are still good enough to make the game challenging and entertaining. However, it often happens that some player queries, despite being relevant to videos, do not win the game. From player s perspective, they are not useful, but from the metadata acquisition perspective, they are the most prized assets, because they represent new information, not found in original descriptions. As we show in our experiments, our game discovers a substantial portion of new metadata through these queries. II. MOTIVATIONAL STUDY: SHORT VIDEO METADATA ARE MISSING All multimeda resources require proper semantics and plethora of approaches focus on the general task of metadata acquisition. Video resources are no exception and many human computation games have been devised to acquire metadata for them [1] [3]. Yet the video resources (e.g. those found on the Web) are heterogeneous from many perspectives and not all content type segments are sufficiently covered with metadata and/or approaches that sufficiently acquire them. One of such segments are short user videos, shared through social services like vine.co. Despite having less than 7 seconds, they became a popular form of user content sharing and the amount of this type content in sharing services grows rapidly (in 2013, 12 million videos were daily uploaded to Vine). 2

2 However, the videos are poorly described with metadata that would allow the content to become searchable and more accessible. To investigate the scale of this issue, we conducted a small study, where we randomly selected 20 videos from the Vine service (drawing from the most popular ones). After seeing these videos, we assigned 10 most important keywords to each one. Then we looked at the actual textual information associated with these videos and evaluated, whether the created keywords are represented in them (we looked at semantics, so very similar concepts or synonyms were accepted as positive cases). We found out, that out of the 200 keywords, only 23 (11.5%) were represented in descriptions or tags. This is definitely not a favourable state. We can also say that existing video descriptions and tags are often short or missing. And, when they exists, they are often too specific (e.g. carrying person names). We also looked at the social comments to the videos, as they may also serve as potential source for keywords [4] (although not as reliable because comments include much more informationless gangue). For each keyword, we evaluated, whether it is (at least theoretically) discoverable within 20 randomly selected existing comments. Out of 200 keywords, only 29 (14.5%) were somehow represented in comments. This is not favourable either, but at least we can report, that keywords found in comments were from great part different than those found within descriptions and tags: overall 46 (23%) keywords were found in at least one of these textual resources. The poor quality of existing metadata to videos reflects a widespread phenomenon in social media: users do not put much effort in describing of their multimedia. Considerng this, we see a great space for applying semantics acquisition techniques for user short videos. III. GAME BASED METHOD FOR VIDEO METADATA ACQUISITION The main contribution of this paper is Vine Search Game a human computation game, which is used for obtaining correct and new keywords for video resources on the Web. Our game specifically focuses on short videos, found in services like Vine, where users upload six-second-long videos mostly from mobile devices with only little or no textual description. The game s output is one or more keywords relevantly describing the videos on input. The game requires existing (although imperfect) textual descriptions or other texts associated to videos from which it bootstraps to provide new keywords. A. Player s perspective From the player s perspective, the game consists of one minute play rounds, each round featuring one video. The game has the following assignment: what query would you use to find this video?. The goal of the player (and the winning condition of the game) is to come up with such search query, that would place the given video (i.e. task video) into top N results of the search. The game takes place in a simple environment, shown in the figure 1. A typical game round follows this scenario: Fig. 1. User interface of the game. The task video to find occupies the left half of the screen. The player can enter his queries into text box below it. On the right center, top six results for the current query are displayed. Player may also review, what is the current position of the task video within the result ranking (bottom right), his score and time left. 1) Player is shown an empty interface (video without thumbnail ready to play) along with the game instructions. 2) The player starts playback the video for the video (for example: a video with a white cat sitting on a worktable is played). No other information on the video are shown to the player. 3) The countdown timer starts immediately (the player has to finish the task in a time limit, in our experiments we used one minute). 4) Player watches the video. She may pause or replay it anytime. 5) Player comes up with a search query, which (he thinks) characterizes the video and enters it into the search field (in our example, she uses query cat white table computer ). 6) The game responds by executing the query, displaying top N results (in our experiments, we chose N=6) and informing the player about the position of the task video in the result ranking (if the task video appears in the top N results, player wins the round, however, in our example, we do not let him yet: the given query puts the task video on the 15th position). 7) Because player was not successful with his query, she tries to reformulate it several times. She helps herself by replaying the video and reviewing other found videos through small thumbnails. Finally, she finds right keywords (e.g. stare ) and gets the task video into top N results shortly before the time limit expires. 8) If the player was successful, she is scored by a constant score. 9) After the game round, either successful or unsuccessful, the player still watch the video (if he enjoys it) and eventually, start another round. From the game design perspective, the players are challenged primarily by the task itself. Finding of the right query words is not always easy, sometimes, there are too many videos that match the proposed query, sometimes, the video is not indexed

3 with given keywords. The challenge is also strengthened by time limit. Players thus need to reformulate their queries and apply different strategies to promote the task video. These strategies include (based on our observations during experiments) replacing individual keywords with synonyms, introduction of more specific or more general keywords or use of different combinations of previously used keywords. To further strengthen player motivation to play, the game includes competition with other players. We have built a global player ladder, where players are compared according to their level, which is computed from their total score accumulated from all successful game rounds. Moreover, game also keeps record of player s winning ratio as percentage (which is presented to players returning to the game). Finally, players are also motivated to play, because they like to watch the video content itself. After all, the videos are drawn from the most popular ones on Vine. This was also confirmed throughout our experiments: players we observed, tended to watch the videos also after the game and some even asked us for links to these videos (in the game, we have hidden this information for gameplay reasons). B. Keyword extraction The search queries used by players serve as the material for keyword extraction. Through these queries, players express what semantics do the videos carry (i.e. each query represents one or more keyword suggestions). We extract the keywords from queries through simple NLP techniques and cross player validation (voting), similar to other human computer games [5], [6]. More specifically: 1) Each query is transformed to a set of keywords. We consider queries as natural language texts (as the players are free to use any expressions). They are therefore tokenized, downcased and individual words lemmatized. Furthermore, stopwords (of the language) are removed. In our implementation, we sticked to this simple approach, but we understand that more sophisticated keyword extraction approaches or named entity recognition techniques could be used here. 2) Keywords of each query are considered as suggestions of a particular player to a particular video. If a player entered more queries for one video, repetitive keyword suggestions are ignored (i.e. each suggestion is counted only once). Ultimately, a set of (unique) player-video-keyword tripples is created from the whole game log. 3) To this set of tripples, a simple voting scheme is applied: if a video receives two or more keyword suggestions (e.g. multiple players independently agree on it), it is considered relevant and is put into game s output. The principal backend mechanism fueling our game is a search engine that executes the search queries. In the game implementation it was realized by ElasticSearch library. The indexing was done over three textual fields associated with each video: description, tags and comments. This of course imposes a prerequisite of already existing base of textual resources associated with the videos, on which the search indexes could be built. Our game cannot function, if we only input multimedia resources without any texts associated with them. This does not mean, we need high quality video descriptions prior to the game. We need only that much, so we can execute queries with reasonable player satisfaction. Fortunately, the input texts do not have to be clean or complete and may even contain a lot of misleading and irrelevant concepts (which is often the case with user-generated texts, as we have also seen in our motivational study). The game is, however, able to work with these. From this perspective, our game can be seen as a method for cleaning up existing textual descriptions (it separates relevant keywords from other less relevant or irrelevant texts) but also, as a method of acquiring new keywords (in case when they were not present in original descriptions). In this perspective, the query keywords that actually didn t worked for the players (because they were not indexed for the task videos and thus, did not pushed the task video upwards in the ranking) are very valuable for us: they carry a completely new information, previously unrelated to the video. IV. EVALUATION OF ACQUIRED KEYWORD QUALITY To evaluate our method, we conducted an online, uncontrolled gameplay experiment followed by manual metadata evaluation. We let players to play the game at will and collected the gameplay data, which were afterward processed for keywords. Then, for sample keywords, we asked expert judges to evaluate correctness (i.e. whether the acquired keywords are relevant to respective videos) and novelty (i.e. do keywords carry new information according to already existing descriptions and metadata). Hypotheses and metrics. We have set up the following hypotheses: 1) Our method is able to acquire relevant keywords for videos (correctness). We measured the correctness as precision (i.e. a ratio of video-relevant keywords to all keywords). We expected the overall correctness to be high (over 0.9, which is in line with similar semantics game acquisition works). 2) Our method acquires new keywords (novelty). Here, we measured the novelty as ratio of correct new keywords to all correct keywords. We had no prior expectation considering this measure. Methodology. Regarding the keyword correctness, two judges were asked to use three-value measure of keyword correctness: the keyword could be either relevant (e.g. a major feature of the video, main topic, important contextual information, dominant background feature, object in the foreground, concepts relevant to the story etc.), less relevant (e.g. less important features of the video, background, related terms, context etc.) or irrelevant (having marginal or no relevance to the video). The correctness (precision) was then computed two times: for relevant keywords only and for relevant and less relevant keywords together. We chose the three value relevancy instead of binary, because we wanted more insight into the keyword quality and we also wanted to make the relevance judging easier: it is sometimes hard to decide, whether a keyword, which is distantly related to a video, is relevant or not. In such cases, the middle category offers a judging

4 comfort, while at the same time, it does not burden the judges with unnecessary wide range of choices. Regarding the keyword novelty, a new keyword is defined as keyword that represents information about the video which is not yet represented in existing video description, metadata or social comments. For example, if a video depicting a dog eating his dinner was only described by a description special easter dinner, but the players supplied the word dog, then this word is considered as new. To assess the newness of each game-generated keyword, we asked the judges to evaluate, whether the meaning is already contained in existing textual resources. First, we asked for the presence in description and tags (which were supplied by video creators themselves). Secondly, (similarly to our motivational study), we also asked the same for comments (which were created by other users of the Vine service). For both cases, we asked the judges to judge strictly semantically: if the meaning was already represented in the original text, but for example, by a synonym, the keyword was not counted as new. Data. As the game data, we acquired 800 videos along with all available texts associated with them through Vine service API. The videos were no longer than 7 second long and were mostly recorded by mobile devices. We took videos from several categories to make the dataset more diverse: comedy (200), animals (200), family (200), music (100), places (50) and sport (50). For each category, we drew the actual popular videos, because we expected they will be more entertaining for our players and also that they will have more textual resources associated with them (e.g. comments). After we drew the video descriptions, we built our search indexes upon them and the game was ready to play. Participants. The players were recruited on social networks. They were mostly students and young people. We accepted anyone, who was willing to play. We have not incentivized the players any further, apart from joy of gameplay itself. In total, 35 players tried the game. Process. Once recruited, players were familiarized with the rules of the game and then they played the game freely, for as long as they wanted. We logged all of the player activity. For two weeks, we let the game open for play. During this time 19 players have played more than 10 games and 6 players played more than 100 games each. Overall, we recorded 1671 games. Afterward, the keyword extraction procedures were run, resulting in 601 extracted keywords, distributed among 302 videos. Each resulting video-keyword association was judged by two independently working judges, who evaluated correctness (relevant, less relevant or irrelevant) of the association and also, its novelty (whether the keyword meaning was semantically present in the original textual descriptions of the video). When judges evaluations for each keyword matched, we accepted their decision. If they disagreed, we accepted the less favourable decision. Results. The correctness evaluation yielded 535 (89.02%) keywords as relevant, 60 (9.98%) as less relevant and 6 (1.00%) as irrelevant. A marginal number (7 1.2%) of the extracted keywords were grammatically incorrect, but semantically correct. We deem such results as positive and in line with other semantic acquisition game performance [5], [6]. However, the more important measure for us was the novelty of the correctly acquired keywords, as it quantifies the effects of our method in regard of expansion of an existing video metadata base (e.g. Vine data). The novelty of our tags was 82.5%, if we only considered original vine descriptions and tags. When only comments were considered, the novelty was 77.7%. When both sources were combined, the resulting novelty was 66.0%. In every perspective we consider these results very good our method delivers substantial portion of new information even in the worst case (when comments are included). V. RELATED WORK Many human computation games have been made for multimedia metadata acquisition, predominantly for images, but also for audio and video resources. Human computation games are mostly unique and are tailored for every area of use. However, they share the same problems and adopt similar solutions to them. The ESP Game, created by Louis von Ahn [7] was the first and one of the most successful human compuation games. This game was used by Google for annotation of images, where it gained great popularity and successfully obtained lot of quality metadata. The game was multi-player and it was based on mutual output agreement. Two players were given the same image and with no possible means of communication, they had to agree on a word. The best strategy to win the game for both players was to write the words describing the image. Following this pattern, several video annotation games were devised. In the OntoTube game [1], players must agree on answers to certains questions. They earn more points when they agree on something content-based. In the game Waisda? [3] the agreement tags are fixed to specific timeframe of the videos (therefore the game is able to acquire descriptions for specific time segments). The multi-player methods work very well, once they become popular, but before that, they risk being stuck in a cold start problem. If the game needs more than one player, especially in the early stages of deployment, it can be difficult to find an opponent and therefore the game cannot start. One of the advantages of game presented in this paper, is its single-player scheme, which is much less prone to cold start problems. Single-player games are generally harder to create, because they need to somehow check the correctness of players output (while at the same time, provide the feedback to the player immediately and based on the correctness of their actions). In these cases, metadata are usually acquired indirectly from players behaviour in the game. Interesting single-player approach is used in the game Akinator 3. In this game, player thinks of famous character or person and then computer (in role of web genie) is asking the player yes-or-no questions, trying to guess the person. In the end, the Akinator asks, if the guess is correct or not and strengthens his model accordingly. In the case of incorrect guess, player writes the name of the character and the game updates its database as well. This approach is very innovative, but it requires very good starting data 3 en.akinator.com

5 and lot of played games to give satisfactory results. Another game, Filmillion 4 attempted to apply similar principles not on characters, but on movies. However it does not work as well, probably because of poor starting data set and lesser popularity. Single-player game schemes in human computation games are often based on bootstrapping, when game builds upon some known data to gain more. Also delayed player feedback can be used (i.e. when the created artifacts are confirmed by other players later on). As example of this, we can list the work of Pinto and Viana [2]. Their TAG4VD is a single player game, where tag videos within given time frames (to enable content seeking within the video) and where players confirm metadata, created by other players, later on. The incorporation of the search into a human computation game, similarly asi in Vine search game, presented in this paper, was done in the past as well. Little search game [6] is a single-player game, which uses search engine for indirect validation of player output. At the beginning of the game, the player is given a term and his task is to lower the number of results returned by the search engine. This can be achieved by entering terms that have high co-occurrences with the specified term on the Web. The game is then able to construct term relationship network from acquired game logs. This approach satisfyingly reduces the cold start problem, because the only artifact used for output validation is result returned from search engine, which is independent from the game. Using search engine for validation of player input is interesting way of constructing human computation games, and we are also using modification of this approach in our method. Outside of the domain of human computation, there were also attempts to mine human search behavior for acquisition of metadata to video resources. An interesting approach was devised by Yao et al. [8]. Their method analyses users browsing logs from a video hosting website. Based on their analysis, it is possible to discover similarity between two videos and then mathematically acquire tag assignment. Authors achieved significant results, but this method works only when there is an existing metadata layer available and it is also based on the assumption that users always visit related videos, which is not definitive. VI. CONCLUSION This work was motivated by the lack of descriptive metadata for user-generated short videos, found on social media, such as vine.co. Our brief exploratory study pointed on a great space of missing descriptive metadata to videos found in this particular service. Following this, we presented the Vine search game a human computation game, that is able to acquire correct and (for a great part) new metadata in the form of keywords to such short social videos. Moreover, since our game is very similar to real search situations, we implicitly get exactly those keywords that user would use to search the specific videos. Our approach has the advantage of being a single-player game, so it does not suffer from potential cold start problems. Limitation of our approach lies in the need for existing base of textual descriptions (or other textual resources) associated to videos. The texts we experimented with, were not of the good quality, but the game performed well. However, it is worth of future investigation, what degree of quality of existing texts is sufficient, what type of information they must carry and how these aspects affect the overall gameplay and player enjoyment. Another open question is the typological composition of the acquired keywords and furthermore, whether players could be incentivized for providing specific (more valuable) types of keywords. Also, we have not yet investigated, how large video space should be used. In this work, we experimented with only small sample of 800 videos. With larger spaces, it could be harder for players to come up with queries, that satisfy winning conditions, as there are more similar videos to the task video. REFERENCES [1] K. Siorpaes and M. Hepp, Games with a purpose for the semantic web, IEEE Intelligent Systems, vol. 23, no. 3, pp , May [Online]. Available: [2] J. P. Pinto and P. Viana, Tag4vd: A game for collaborative video annotation, in Proceedings of the 2013 ACM International Workshop on Immersive Media Experiences, ser. ImmersiveMe 13. New York, NY, USA: ACM, 2013, pp [Online]. Available: [3] M. Hildebrand, M. Brinkerink, R. Gligorov, M. van Steenbergen, J. Huijkman, and J. Oomen, Waisda?: Video labeling game, in Proceedings of the 21st ACM International Conference on Multimedia, ser. MM 13. New York, NY, USA: ACM, 2013, pp [Online]. Available: [4] O. Alonso, S. Bannur, K. Khandelwal, and S. Kalyanaraman, The world conversation: Web page metadata generation from social sources, in Proceedings of the 24th International Conference on World Wide Web Companion, ser. WWW 15 Companion. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee, 2015, pp [Online]. Available: [5] J. Simko and M. Bielikova, Personal image tagging: A game-based approach, in Proceedings of the 8th International Conference on Semantic Systems, ser. I-SEMANTICS 12. New York, NY, USA: ACM, 2012, pp [Online]. Available: [6] J. Simko, M. Tvarozek, and M. Bielikova, Little search game: Term network acquisition via a human computation game, in Proceedings of the 22Nd ACM Conference on Hypertext and Hypermedia, ser. HT 11. New York, NY, USA: ACM, 2011, pp [Online]. Available: [7] L. von Ahn and L. Dabbish, Labeling images with a computer game, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI 04. New York, NY, USA: ACM, 2004, pp [Online]. Available: [8] T. Yao, T. Mei, C.-W. Ngo, and S. Li, Annotation for free: Video tagging by mining user search behavior, in Proceedings of the 21st ACM International Conference on Multimedia, ser. MM 13. New York, NY, USA: ACM, 2013, pp [Online]. Available: 4