Video Annotation and Retrieval Using Vague Shot Intervals

Transcription

1 Master Thesis Video Annotation and Retrieval Using Vague Shot Intervals Supervisor Professor Katsumi Tanaka Department of Social Informatics Graduate School of Informatics Kyoto University Naoki FUKINO Feburary 6, 2004

2 Video Annotation and Retrieval Using Vague Shot Intervals Naoki FUKINO Abstract In this paper, we propose a fast content description method to describe metadata for video which is broadcasted live. Recently, multichannel digital broadcasting and hard-disk video recorder are becoming widespread. Consequently, the amount of available video content becomes to be enormous. Although the time we can watch video contents is not longer than before. Therefore A system need to support users by abstracting high-value parts automatically. The system have to understand video content for abstracting proper video interval which is meet user s request. The metadata which describes video content is needed for the system to understand content. Content description of video is called video annotation. Annotation costs much. Therefore, there are few video contents which have metadata of the content. Especially, it is rare that metadata is sent with live broadcasting. In this paper, we propose content description method which emphasizes speed of annotation to make more video contents which have metadata. Concretely speaking, the goal of our research is that a sports video is described metadata in real-time. There are two approaches for video annotation. They are segmentation approach and stratification approach. In segmentation approach, the cost of annotation is lower but it is not suited for annotating some kind of videos in which semantic unit such as scene in movie is not provided. In stratification approach, intervals correspond tokeywordsarespecified in several. These intervals can have overlap with each other and an interval can contain other interval. This approach can describe keywords in detail, but this approach costs much because the boundary of intervals has to be specified. Each approaches are not suited for describing sports video in real-time. Consequently, we use a new approach which is neither segmentation approach nor stratification approach. In this approach we describe only the recognized timing of event without specifying the boundary of event s interval pre- i

3 ii cisely. Recognized timing of an event can be described by using manpower. In addition, sports commentator repeat realizing an event and speaking the event, so recognized timing of an event can be described by using speech recognition for commentator. In our method Vague Shot Interval and Vague Modifier are used. Vague Shot Interval is used to estimate interval of event and relation between events. This expresses distribution of event s interval and original point of distribution is recognized timing of the event. Vague Modifier is used to estimate an event which is modified by a keyword. This expressed distribution of event s interval and original point of distribution is a keyword which modifies the event. Starting point and terminal point of interval of each event are expressed by probability distribution, so the boundary is described vaguely. We can reduce cost of content description by using Vague Shot Interval and Vague Modifier. For retrieval for metadata described using Vague Shot Interval and Vague Modifier, an algorithm suited to the form of annotation is needed. In this paper, we describe the search algorithm too. We implemented a prototype system which uses proposed method of annotation and retrieval, and we annotated a soccer video. In annotation, we did not stop or rewind video on the assumption that we must annotate movie in real-time. In result, we can describe practical number of keywords in two iterations. In addition we retrieve scene from the described metadata. In result, the recall ratio is high but precision ratio is low. Roughly half of the wrong abstracted scenes come from mistakes in annotation. From this implementation, we verified that two annotator can describes practical number of keywords in real-time, and retrieval from the metadata is practical for a sort of applications in which the accuracy of retrieval is not so important. In this paper we describe an application whose demand will be higher if the real-time annotation is popular.

4 iii 曖昧なショット区間を用いたビデオ映像の内容記述と検索方式吹野直紀内容梗概 本稿で, 我々は生放送映像にメタデータを付与するための高速な内容記述手法を提案する. 近年, 放送のデジタル多チャンネル化やハードディスクレコーダーの普及, インターネット回線の高速化による動画配信サービスの増加により, 我々は膨大な映像コンテンツを得ることができるようになった. しかし, 個人が映像を視聴できる時間は限られている. ユーザが大量の映像の中から好きな時に好きな部分のみを視聴するためには, システムによる映像区間の選択支援が必要になる. ユーザの要求に応じてシステムが適した映像区間を取り出すためには, システムが映像の内容を理解する必要がある. システムが映像内容を知るためには映像中のどの部分に何が映っているかという情報を記述したメタデータが必要である. ビデオ映像の内容を記述する事を映像のアノテーションと呼ぶ. アノテーションには通常時間的コストが多くかかる. そのため, 現状ではメタデータが付与された映像はあまり多く提供されていない. 特に, 生放送の映像にメタデータが付与されている事は極めてまれである. 本研究ではできるだけ多くの映像にメタデータがつくよう, 高速性を重視した内容記述手法を提案する. 具体的には生中継される事が多いスポーツ映像に対してリアルタイムにアノテーションを行う事を目標とする. アノテーションには大きく分けると構造化法と層状化法がある. 構造化法とはビデオをまず複数の区間に区切り, それぞれの区間に対してキーワード等を付与する方法である. この方法はアノテーションのコストは低くなるが, スポーツ映像等, はっきりとした句切れ目の無い映像には向かない. 一方の層状化法はキーワード毎に該当区間を設定する方法である. 自由度は高いがそれぞれの区間の指定と終点を指定しなければならず, アノテーションに時間がかかってしまう. 生中継スポーツ映像に対してリアルタイムに内容記述を行うにはどちらの方法も向いていない. そこで, 本研究では層状化法でも構造化でもない新たな方法として, イベントを認識した瞬間のみを記述しイベント区間の始点と終点ははっきり指定しな

5 iv い方法をとる. イベントを認識した瞬間のみの記述は人手を用いて行う事もでき, また, 実況アナウンサーはイベントを認識して発話するサイクルを繰り返すのでその発話を頼りに記述する事ができる. イベントを認識した瞬間を基にシステムがイベント区間を推測したりイベント同士の関係を推測したりするために, 本手法では曖昧なショット区間 (Vague Shot Interval) と曖昧な修飾子 (Vague Modifier) という2つの概念を用いる. Vague Shot Interval はイベントの区間と関係を推測するために使うもので, イベントを認識した瞬間を基準にイベント区間がどのあたりに分布しているかを表している. 一方,Vague Modifier は修飾語が修飾しているイベントがどれかを推測するために使うもので, 修飾語を認識した瞬間を基準にイベント区間がどのあたりに分布しているかを表している. 各々のイベント区間は始点と終点を確率分布で表現しており, 境界が曖昧になっている. これらを利用して認識に従って内容記述を行う事で, 内容記述にかかるコストを大幅に低減する事ができる. Vague Shot Interval 及び Vague Modifier を用いて記述されたメタデータに対して検索を行うにはその形式に応じたアルゴリズムが必要である. 本稿では, そのアルゴリズム手法についても述べる. 本稿で提案した手法を使ったプロトタイプを実装し, サッカー映像に対してアノテーションを行った. アノテーションは実際のリアルタイムアノテーションをする場合を想定し, 映像が終わるまで映像の停止や巻き戻しはしなかった. 結果,2 回の反復で実用的な数のキーワードを記述する事ができた. また, この作業で作られたメタデータに対していくつかの検索を行った. 結果, 再現率は高いものの適合率は低めな結果となった. 適合していない検索結果のうち, 約半分はアノテーションの際のイベントの認識ミスであった. 結果として, 本手法ではアノテーションをする人が2 人以上いればリアルタイムアノテーションが可能である事がわかり, その手法で作られたメタデータに対する検索では適合率がやや低くなるものの検索精度が重要ではないアプリケーションでは実用的と思われた. 本稿ではリアルタイムアノテーションが一般的になった際に需要が高まると考えられるアプリケーションについても触れる.

6 Video Annotation and Retrieval Using Vague Shot Intervals Contents Chapter 1 Introduction 1 Chapter 2 Basic Items and Related Work BasicItems Segmentation Approach and Stratification Approach MPEG RelatedWork... 4 Chapter 3 Vague Descriptors VagueShotInterval Vague Modifier VagueSceneStructure Chapter 4 Search Algorithm OverviewofSearchProcess QueryForm UseofThesaurus FundamentalBinaryRelations AdditionalBinaryRelations QueryEasingforMultiKeywords Chapter 5 Experiment and Evaluation SystemOverview Creatingthesaurus CreatingTagInformation AnnotatingMovie SearchResult Evaluation Chapter 6 Discussion Multi-modalAnnotation... 40

7 6.2 QueryGenerationfromNewsArticle Chapter 7 Summary and Conclusion 45 Acknowledgments 47 References 48

8 Chapter 1 Introduction For several years, multichannel digital broadcasting and hard-disk video recorder are becoming widespread. By diffusion of multichannel digital broadcasting, we become to be able to get various video contents we are interested in. By diffusion of hard-disk video recorder we become to be able to accumulate video contents and watch them at our leisure. Consequently, the amount of available video content becomes to be enormous. Although the time we can watch video contents is not longer than before, so we have to select a content which have a lot of value for us. Besides, we have to abstract high-value parts from a video content. It is hard for watchers to abstract all high-value parts from all video contents, so a system needs to support users by abstracting high-value parts automatically. However video s bit stream cannot be an object of search engine as it is. Metadata of video content is needed for the search engine to understand details in each part. Metadata contain information what is reflected in each part of a video content. Recently a technology of metadata has gotten a lot of attention, and then MPEG7 is standardized as a format of metadata. We can describe various characteristics of a video content by using MPEG7. Although the number of video contents which have metadata describing details is not so much as it is now. One reason is thought that describing metadata cost much time. At this time, it is hard to recognize meanings of the scene by image recognition, voice recognition, and so on. For this reason, contents description using manpower is needed to efficient retrieval. However contents description using manpower costs much time, so it is hard to describe metadata for video contents produced continuously. Especially the metadata of sports video is hard to be described because the sports video is often broadcast live, so it has to be described in real time. However there is unevenness of importance in sports video, so sports video is in high demand of metadata for watching favorite parts. Therefore we propose a method of contents description at low cost on the assumption that we have to describe metadata in real time for sports video 1

9 contents, especially for soccer video contents. In this method, we describe eventsoccurredinvideobyusingmanpowerintextform. Howeveritcosts too much to describe details, so we reduce the cost of contents description by describing information vaguely when the information is no need to be described precisely. When attach a keyword to certain part of video, we have to specify the starting point and the terminal point of the part. But it costs too much to specify them. In addition, it is impossible in some cases to specify interval in real time because the starting point of interval corresponds to an event is before the moment which the event is recognized. So we take the following method. In the process of contents description, we describe only the recognized timing of events. In the process of retrieval, the search engine estimates the actual intervals correspond to events and the relations between the intervals by using knowledge which is arranged in advance. This knowledge consists of two elements. First one is the distribution of past intervals of an event. The center of this distribution is the time point which the event is recognized. Second one is the degree of impression of an event on surrounding time. This knowledge is accumulated in a dictionary in the form of Vague Shot Interval and the form of Vague Modifier. In addition, knowledge of alternative scene structure of an event is accumulated in the form of Vague Structure for reflecting difference of subjective definition of a scene of an event. The remainder of the paper is outlined as follows: In section Chapter 2, we describe basic items of contents description and related work. In section Chapter3,wedescribeVagueShotIntervalandVagueModifier. These ideas are used for contents description at low cost. In addition, we describe Vague Structure which is used in search process. In section Chapter 4, we describe search method for metadata which is described by using Vague Shot Interval and Vague Modifier. in section Chapter 5, we describe a prototype system which implements search algorithm shown in section Chapter 4, and we evaluate the cost of contents description and the effectiveness of the search algorithm. In section Chapter 6, we discuss alternative method of contents description using Vague Shot Interval and Vague Modifier. In addition, we discuss application using proposed technology. In section Chapter 7, we describe conclusion. 2

10 Chapter 2 Basic Items and Related Work For retrieving scenes from video, we have to recognize and describe content. Content description of video is called annotation of video. In this section, we describe basic items and related work about annotation. 2.1 Basic Items Segmentation Approach and Stratification Approach There are two approaches for description of video content. They are segmentation approach and stratification approach. These approaches is represented in figure 2.1 Figure 2.1: Segmentation Approach and Stratification Approach In segmentation approach, video is divided into segments and keywords are attached to each segment. Divided segments have no overlap with each other. This approach has advantage to make the annotation cost low, but this approach is not suited for annotating some kind of videos in which semantic unit such as scene in movie is not provided. In stratification approach, intervals correspond to keywords are specified in several. These intervals can have overlap with each other and an interval can contain other interval. This approach can describe keywords in detail, but this approach costs much because the boundary of intervals has to be specified. We aim at annotating sports video in real-time. Sports video do not have distinct semantic unit, so we can not use segmentation approach. However if we use stratification approach directly, we can not annotating in real-time because 3

11 of the cost. Consequently, we use stratification approach without specifying the boundary of interval precisely MPEG7 In response to increase of demand for metadata techniques, MPEG7 [6] was designed. Official name for MPEG7 is Multimedia Content Description Interface, and a standard description tool for multimedia content is defined. MPEG7 defines notation method for metadata. The process from creation of metadata to use of metadata is shown in figure 2.2. Method of creating metadata and method of using metadata is not standardized by MPEG7. We propose a method for creating and using metadata for annotating sports video in real time and using this metadata. Figure 2.2: Standardized part by MPEG7 2.2 Related Work There are various methods for annotation of video. Informedia project [1] [2] in CMU is famous system for that annotate video by using technologies of image recognition, speech recognition and natural language processing. This method use segmentation approach and total length of annotated video is more than 1000 hour. NHK [3] develops real-time speech recognition technology for closed caption of live broadcasting. This technology use manpower for adjusting recognized text in real-time. By digital broadcasting, broadcasting station can send scenario of drama. In [8], a matching method between a part of scenario and a part of video is proposed. This use DP matching method and image/speech 4

12 recognition. A method, which describes keyword using scenario and structuralize video by clustering keywords, is proposed[14][15]. GDA(Global Document Annotation)[4] is xml tag set which is used for express natural language to be understandable for machines. A tool that converts natural language to GDA is useful in annotation. In analysis of sports video, Snoek et al. [12] propose a method which use support vector machine for multi modal recognition. Nalagawa et al. [16] propose a method that recognizes play and camera work using image recognition and store its result in score book. 5

13 Chapter 3 Vague Descriptors When we annotate video by keyword or search scene from video by keyword, we come up against a problem which is caused by vagueness. For example, when we want to watch goal scene in soccer video, we input goal in search engine. However, definition of goal is various by people. A certain man may image goal scene as shoot scene, but the other man may image one as whole attack scene. This problem arises as long as recognition of an annotator and a user is different. There is another problem. It costs too much to annotate video. To annotate video by use keyword, we have to specify the starting point and the terminal point of the interval which is applicable to the keyword. This problem is serious especially when an object is a sport video. A sport video is broadcast live in many cases. If we want to put metadata on broadcast, we have to annotate movie in short time. To cope with problems described above, we introduces following three ideas into video annotation and retrieval. Vague Shot Interval Vague Modifier Vague Scene Structure Vague Shot Interval and Vague Modifier are used in order to solve the problem of cost. Vague Scene Structure is used to reflect the definition of the scene which is different with people. We describe details in following chapters; 3.1 Vague Shot Interval When we must annotate video in real time ornearcondition,wecannotspecify the starting point and the terminal point. Especially, it is impossible to specify the starting point in real time depending on the kind of keyword. It is because that the recognized timing of the event is after the starting point of the event. If we describe only the recognized timing of the event, the cost of annotating video is mitigated. It is impossible that specifying starting point and terminal 6

14 point of the event by the recognized timing of the event, but we can roughly estimate the starting point and the terminal point by thinking about characteristic of event. For example, we expect that the starting point and the terminal point of the interval correspond to the event pass is near recognized timing, and its length is 1 or 2 seconds. We also expect that the length of the interval correspond to the event long pass is longer than the length of the interval correspond to the event pass. If this knowledge is accumulated in a dictionary, the system can predict the boundary of interval based on the recognized timing. We accumulate this knowledge in the form of probability distribution of a set of the past intervals. This knowledge is generated in following two steps. First, we accumulate starting point and terminal point of past intervals. The instance of result is shown in table 3.1. Starting point and ending point are measured considering the time point becoming a standard as 0 second. The recognized timing of the event is suitable for this standard time point in realtime annotation. However it may not be suitable when video is annotated by another method. (E.g. Image recognition) starting point terminal point Past Interval second 3.9 second Past Interval second 3.1 second Past Interval second 5.0 second Past Interval second 4.3 second Table 3.1: Past intervals correspond to long pass Secondly, the probability distribution of starting point and terminal point is generated from past intervals (figure 3.1). We call the interval whose starting point and terminal point expressed by probability distribution Vague Shot Interval. In the process of annotating movie, annotator describes only the recognized 7

15 Figure 3.1: Making of probability distribution from past intervals timing of the event. Next, in the process of retrieval, the search engine predicts intervals correspond to the event by using the event s Vague Shot Interval. This processisshowninfigure 3.2. Vague Shot Interval of the event in the dictionary is introduced into the time line in video as a center is the timing that the event described by annotator. The search engine determines starting point and terminal point of this event from introduced Vague Shot Interval. If probability distribution of starting point is expressed f start (t) and probability distribution of terminal point is expressed f end (t), the probability I(t) thatthetimepointt is contained by interval is calculated by following formula. I(t) = Z t f start (t)dt Z t f end (t)dt (3.1) Next, the system set the threshold and determine interval of the event. The probability that the time point t is contained by interval is higher than threshold in the determined interval. If the threshold is set lower, the length of interval 8

16 Figure 3.2: Determin interval s boundary 9

17 becomes longer. The probability distribution of Vague Shot Interval is used not only for determine interval, but also estimate relation between a pair of intervals which is expressed by Vague Shot Interval. 3.2 Vague Modifier Vague Modifierisused to predictkeywords which modifythe intervalof a certain event. Intervals correspond to an event can be modified by various keywords. For example, a interval correspond to a event pass is modified by a keyword zidane if the event pass is released by zidane, and a event shot is modified by a keyword goal if the shot leads to goal in consequence. It is difficult to describe these keywords in real time because of the cost of description. Therefore we describe only the recognized timing of these keywords and the system estimate an event which is modified by the keyword. The scene which should be modified by keyword goal tends to distribute before the time point that goal is recognized. Because the timing we can recognize goal is near the last timing of goal scene. On the other hand, we can recognize event cornerkick before the scene which should be modified by keyword corner-kick. If this knowledge is kept in dictionary, modifying keywords of the interval of the event can be predicted. Vague Modifier is expressed by interval whose boundary is expressed by probability distribution like Vague Shot Interval, but in Vague Modifier the standard point is recognized time of modifier keyword and the interval is modified event. The process of making Vague Modifier is shown in figure 3.3. The system predicts which event is modified by the keyword. This process is shown in figure 3.4. First, Vague Modifiers of keywords are introduced into the time line in video. In the instance in the figure, Vague Modifier, which expresses distribution of event Pass from the recognized timing of player(zidane), is introduced into the time line. Second, Vague Shot Intervals of events are introduced into the time line in video. In the instance in the figure, Vague Shot Interval, which expressed distribution of event Pass from the recognized timing of pass, is introduced into the time line. Finally, the system determines the 10

18 Figure 3.3: Vague Modifier of GOAL 11

19 event which the keyword modifies. The event whose Vague Shot Interval has the largest interval to Vague Modifier is selected. In the instance in the figure, the system estimate that the second pass event is modified by the keyword Zidane because the Vague Shot Interval of the second pass event and the Vague Modifier of the keyword Zidane have the largest overlap. Figure 3.4: Vague Modifier of GOAL By using Vague Modifier, we reduce the cost of describing modifiers of events. 3.3 Vague Scene Structure Vague Scene Structure represents alternative structures of an event. This is used to reflect the difference of scene definition with a user. When a keyword is entered by a user to search it, image of the scene of the keyword may vary from user to user. For example, when user search scene by keyword goal, a user may image shot scene, other user may image shot scene and pass scene which leads to goal, and other user may image whole attack scene which leads to goal. This difference exists in annotators. The difference with annotator is can 12

20 be eliminated if annotators have common definition of scene. However it is not impossible to expect users to have common definition. For this reason, alternative structures of a keyword is stored in dictionary in the form of graph shown in figure 3.5 Figure 3.5: Alternative structures of Goal and Zidane The top-left graph means event shot which is modified by a keyword goal. The event which is in shaded circle is a component of scene. In search process, search engine use alternative structures, and respective results are shown to user. If user select one structure, search engine use it after that time. If a user enters multiple keywords to search engine, the structure imaged by user can be guessed from combination of keywords. 13

21 Chapter 4 Search Algorithm In this section we describe method to retrieve video scene from metadata. In section 4.1, we describe flow of retrieval. In section 4.2, we describe query form which expresses user request. In section 4.3, we explain about thesaurus which is used in retrieval to enhance accuracy of retrieval. In section 4.4, we describe fundamental relations between a pair of intervals which is expressed Vague Shot Interval. In section 4.5, we describe additional relations. 4.1 Overview of Search Process Search process can be divided into following steps. 1. Enter a query 2. Generate query graph by entered query 3. Abstract combination of events from metadata 4. Calculate goodness of each combination 5. Show abstracted scene for the user In step 1, user entered a query in textual form. In step 2, search engine generate query graph as shown in section from entered textual form. In step 3, search engine abstract events from metadata of video. Metadata is a set of tags. The tag contains following data. Time instant that this tag is recorded Tag information A tag information contains following data. Name Type (Modifier or Interval) Ifthetypeismodifier, tag information contains membership function. If the type is interval, tag information contains time distribution of boundary. The interval is expressed by Vague Shot Interval. Search engine abstract combinations of tags in according with query. In step 4, search engine calculate goodness of each combination abstracted in step 3. The goodness is the probability that the combination of tag meets 14

22 conditions described in query. In step 5, search engine shows candidate scenes for user. Candidate scenes are sorted in descending order of goodness. 4.2 Query Form Query form must be able to represent following information. semantic units which the user want conditions which the scene has to meet In common web page retrieval, users do not need to specify the semantic units. It is because that the semantic unit which the user gets after retrieval is essentially one page. But in video scene retrieval, such common semantic unit does not exist. Almost all the video have cuts. But there is no guarantee that a cut in video have some means. Especially in the sports video, it is rare case that a cut contains enough information unlikely in the movie video. Therefore the semantic units in the video need to be specified by the query. Asemanticunitwhichisspecified in the query is an event which has interval or a set of events. In addition to the semantic units, conditions the scene has to meet need to be specified. Conditions are described as relation between events. For example, we describe about a case that a user want to get attack scene which contains pass by zidane and this pass leads to goal. First the semantic unit which user wants is the following event. Attack Conditions which the scene has to meet are following item. This event of attack contains an event of pass This event of pass is modified by zidane This event of pass is modified by goal A graph which represents these conditions is shown in figure 4.1. In figure 4.1, ellipse means a semantic unit which is an interval corresponds to an event. Arrow means condition between events. In this instance, arrow means that the interval of attack event contains the interval of pass event. Keywords surrounded by rectangle means keywords which modifying events. In 15

23 Figure 4.1: Query Form(Query Graph) this instance, an event of pass is modified by keyword goal and keyword zidane in the sense that the pass is by zidane and the pass leads to goal. Thus the query form is expressed by a following graph. Node This expresses an event. Information of node consists of an event keyword, modifying keywords, and a flag. Flag specifies that this event is played or not. Edge This expressed a relation between keywords. Information of edge consists of the relation only. Available relations are shown in following chapter. In search process, search engine find the combination of events which meets the condition of query, and play the interval of each event which is to be played. But when the events which are to be played are more than two events and there is gap between events, system has to determine if the gap between events is played or not. This depends on a length of gap and a kind of the event. In current system, if total length of gap is larger than total length of event interval, gap is not played. 4.3 Use of Thesaurus In real-time video annotation, it is hard to describe abundant metadata because of the limited time. So it is more important to accept differences from keyword of query to keyword of metadata by using thesaurus. Thesaurus is a dictionary which has information of synonym, hypernym, and hyponym. Thesaurus can 16

24 be expressed graph structure shown in figure 4.2 Figure 4.2: Structure of thesaurus In this graph, each node means a set of synonym. Cross, Crossing Pass, and Center have the same meaning. Each arrow means the relation between nodes. Thenodepointedbythearrowisahypernyminrelationtothenodeinthe root of the arrow. By using information of synonym in thesaurus, the search engine can abstract cross tag even if the entered keyword is center. By using information of hypernym, the search engine can abstract cross tag even if the entered keyword is pass. If there are no tag whose keyword is synonym or hyponym of entered keyword, search engine abstract hypernym of the entered keyword from metadata. It is because that the tagged event, whose keyword is hypernym, may synonym of the entered keyword. 4.4 Fundamental Binary Relations We express relation between events by temporal relation between intervals corresponds to the events. There are thirteen fundamental relations, known as the Allen primitives, between pairs of time intervals. These relations are shown in figure 4.3. Every relation between a pair of time intervals belongs to one of Allen primitives, and Allen primitives are exclusive with each other. A relation between a pair of interval depends on position of each interval s starting point and terminal point. For example, the starting point of interval i 1 is before the starting point of interval i 2 and the terminal point of interval i 1 is after the terminal point of interval i 2, then the relation between i 1 and i 2 is Contains which means 17

25 Figure 4.3: Allen Primitives 18

26 the interval i 1 contains i 2. If the interval of event Attack is i 1 and the interval of event Pass is i 2,therelationbetweenAttack and Pass is Contains. However, if the starting point and terminal point are represented by Vague Shot Interval, the relation between intervals cannot be specified because the starting point and the terminal point cannot be specified. The instance is shown in figure 4.4. Figure 4.4: Possible relations between interval 1 and interval 2 In this case three relations are possible. The relation depends on the actual starting point and the actual end point of interval which is expressed by Vague Shot Interval. If the actual starting point of interval i 1 is after the actual terminal point of interval i 2,therelationbecomesAfter as a result. It is impossibletodeterminetherelationfromthepossible relations, so use probability of meeting the relation. In the case shown in figure 4.4, if the probability that the actual interval i 1 is after the actual interval i 2 is 0.2, the relation is expressed by following form. After(i 1,i 2 )=0.2 (4.1) Inthecaseshowninfigure 4.4, possible relations are After, Contains, Overlapped 19

27 by. So following formula has to be met. After(i 1,i 2 )+Contains(i 1,i 2 )+OverlappedBy(i 1,i 2 )=1 (4.2) The combination of intervals, whose probability of meeting condition is high is shown to the user preferentially. The relation between a pair of interval is determined by the relations of the starting points and the terminal points of intervals. For example, the relation between interval A and B is turned out to be Overlaps if the starting point of A is before the starting point of B, the terminal point of A is after the starting point B and the terminal point of A is before the terminal point of B. These decision processes is shown in 4.5 Figure 4.5: Binary tree for determining relation In this figure A start B start expresses that the starting point of A is before the starting point of B. By following the graph, the relation between a pair of intervals is determined from the relation between a pair of time point. Inthisgraph,thecasethateachtimepoint is simultaneous. Because the probability, that the time points which distribute according to probability distribution are simultaneous, is 0. Therefore the relation between a pair of interval which is expressed by Vague Shot Interval is expressed by 6 probabilities 20

28 of Af ter, Bef ore, Contains, During, Overlaps, Overlapped by. A sum of 6 probabilities must be 1. For calculate these six probabilities, the probability that a time point is before or after another time point have to be calculated. As shown in figure 4.6, if a time point T 1 follows the probability distribution f 1 (t) andatimepoint T 2 follows the probability distribution f 2 (t), the probability P (T 1 <T 2 )that the time point T 1 is before T 2 is calculated by following formula. P (T 1 <T 2 ) = (P (T 1 >T 2 ) = Z Z Z T f 2 (T ) f 2 (T ) Z = 1 P (T 1 <T 2 )) T f 1 (t)dtdt f 1 (t)dtdt Figure 4.6: Relation between time point and time point expressed by probability distribution Using this probability, each relation between a pair of interval is calculated. Each probability is calculated following formula. Before(i1,i2) = P (i1 start <i2 start ) P (i1 end <i2 start ) Overlaps(i1,i2) = P (i1 start <i2 start ) P (i1 end >i2 start ) P (i1 end <i2 end ) Contains(i1,i2) = P (i1 start <i2 start ) P (i1 end >i2 start ) P (i1 end >i2 end ) 21

29 After(i1,i2) = P (i1 start >i2 start ) P (i1 start >i2 end ) OverelappedBy(i1,i2) = P (i1 start >i2 start ) P (i1 start <i2 end ) P (i1 end >i2 end During(i1,i2) = P (i1 start >i2 start ) P (i1 start <i2 end ) P (i1 end <i2 end In these formula, i1 andi2 means interval which is expressed by Vague Shot Interval. i1 start means the starting point of interval i1. For describing conditions between events, use these relations as fundamental binary relations between intervals which are expressed by Vague Shot Interval. 4.5 Additional Binary Relations For express the relation between a pair of intervals, Fundamental Binary Relations are described in previous section. But in search process, other relations may be required. In this section, we describe additional binary relations. Additional binary relations are created by following operation on Fundamental Binary Relations. Combine Introduce error By combining of relations, we create new relation. New relations created by this operation are following relations. These relations treat equally each interval. ContainsorDuring HaveOverlap OverlapsorOverlapped by StandOff ContainsorDuring means that intervals are in inclusive relation. In this relation, two intervals are not distinguished. HaveOverlap means that intervals have overlap. OverlapsOrOverlapped by means that intervals have overlap and intervals are not in inclusive relation. StandOff means that two intervals are not have overlap. The probabilities that interval i 1 and interval i 2 meet 22

30 these relations are calculated by following formula. ContainsOrDuring(i 1,i 2 ) = Contains(i 1,i 2 )+During(i 1,i 2 ) HaveOverlap(i 1,i 2 ) = Contains(i 1,i 2 )+During(i 1,i 2 ) +Overlaps(i 1,i 2 )+OverlappedBy(i 1,i 2 ) OverlapsOrOverlappedBy(i 1,i 2 ) = Overlaps(i 1,i 2 )+OverlappedBy(i 1,i 2 ) StandOff(i 1,i 2 ) = Before(i 1 )+After(i 2 ) Next, we describe a method to introduce error into the relations. In section 4.4, we do not think about seven relations in Allen primitives. These relations are Meets, MetBy, Starts, StartedBy, Finishes, FinishedBy,and Cotemporal. It is because the probability, that a pair of time points expressed by probability distribution is simultaneous, is 0. But if the error introduced into the relations, these seven relations can be used. For example, if the error of 1 seconds are allowed, the intervals i 1 and i 2 in figure 4.7 do not meet the relation Meets(i 1,i 2 ). But if the error of 3 seconds are allowed, the intervals i 1 and i 2 in figure 4.7 meet the relation Meets(i 1,i 2 ). Figure 4.7: Gap between intervals In this instance, the relation of intervals depends on the relation of the terminal point of i 1 and the starting point of i 2. Other relations depend on the relation of starting point and terminal point too. So the relation of a pair of time points needs to be decided with the error. When the error of e secondsisallowed,therelation between the time point T 1 and T 2 is determined by following rule. 23

31 If the time points meet the following formula, the time point T 1 is before T 2. T 1 <T 2 + e (4.3) If the time points meet the following formula, the time point T 1 is simultaneous with T 2. (T 1 + e>t 2 ) (T 1 <T 2 + e) (4.4) If the time points meet the following formula, the time point T 1 is after T 2 T 1 + e>t 2 (4.5) If the time points are expressed by probability distribution, the probability that a pair of time point meets the relation is calculated by following formula. In following formula, P (T 1 <T 2,e) means the probability that T 1 is before T 2 on condition that the error is allowed to e seconds. P (T 1 <T 2,e) = P (T 1 >T 2,e) = P (T 1 = T 2,e) = Z Z Z Z T f 2 (T e) f 1 (T e) f 2 (T e) Z T Z T f 1 (t)dtdt f 2 (t)dtdt f 1 (t)dtdt = P (T 1 <T 2,e) P (T 1 >T 2,e) Z Z T f 1 (T e) f 2 (t)dtdt By using these relations, the relation between a pair of intervals is determined. The seven relations which have a relation of simultaneous time points are determined by following rules. meets If the terminal point of interval i 1 is simultaneous with the start point of interval i 2, the relation i 1 meets i 2 ismet. met-by If the start point of interval i 1 is simultaneous with the terminal point of interval i 2, the relation i 1 met by i 2 ismet. 24

32 starts If the start point of interval i 1 is simultaneous with the start point of interval i 2 andtheendpointofintervali 1 is before the terminal point of interval i 2, the relation i 1 starts i 2 ismet. started-by If the start point of interval i 1 is simultaneous with the start point of interval i 2 andtheendpointofintervali 1 is after the terminal point of interval i 2, the relation i 1 started by i 2 ismet. finishes If the start point of interval i 1 is after the start point of interval i 2 and the end point of interval i 1 is simultaneous with the terminal point of interval i 2, the relation i 1 finishes i 2 ismet. finished-by If the start point of interval i 1 is before the start point of interval i 2 and the end point of interval i 1 is simultaneous with the terminal point of interval i 2, the relation i 1 finished by i 2 is met. cotemporal If the start point of interval i 1 is simultaneous with the start point of interval i 2 andtheendpointofintervali 1 is simultaneous with the terminal point of interval i 2, the relation i 1 cotemporal i 2 ismet. Therefore, when the error is introduced, the probabilities of the seven relations are calculated by following formula. In following formula, e means the error which is allowed. i 1 and i 2 means the interval which is expressed by Vague Shot Interval. i1 start means the starting point of interval i 1. P (T 1 <T2,e)means the probability that the time point T 1 is before the time point T 2 on condition that the error is allowed to e seconds. Meets(i 1,i 2,e) = P (i1 end = i2 start,e) MetBy(i 1,i 2,e) = P (i1 start = i2 end,e) Starts(i 1,i 2,e) = P (i1 start = i2 start,e) P (i1 end <i2 end,e) StartedBy(i 1,i 2,e) = P (i1 start = i2 start,e) P (i1 end >i2 end,e) 25

33 Finishes(i 1,i 2,e) = P (i1 start >i2 start,e) P (i1 end = i2 end,e) FinishedBy(i 1,i 2,e) = P (i1 start <i2 start,e) P (i1 end = i2 end,e) Cotemporal(i 1,i 2,e) = P (i1 start = i2 start,e) P (i1 end = i2 end,e) Similarly, Fundamental Binary Relations can be introduced error. Before(i1,i2,e) = P (i1 start <i2 start,e) P (i1 end <i2 start,e) Overlaps(i1,i2,e) = P (i1 start <i2 start,e) P (i1 end >i2 start,e) P (i1 end <i2 end,e) Contains(i1,i2,e) = P (i1 start <i2 start,e) P (i1 end >i2 start,e) P (i1 end >i2 end,e) After(i1,i2,e) = P (i1 start >i2 start,e) P (i1 start >i2 end,e) OverelappedBy(i1,i2,e) = P (i1 start >i2 start,e) P (i1 start <i2 end,e) P (i1 end >i2 end,e) During, e(i1,i2) = P (i1 start >i2 start,e) P (i1 start <i2 end,e) P (i1 end <i2 end,e) 4.6 Query Easing for Multi Keywords When a user enter a query to retrieve scenes, the search engine can be religious about retrieving the scene which the user want if the user use query graph described in section 4.2. But it is better to reduce the cost to enter the query. Then a mechanism,that the search engine automatically decides the query without the user specifies it, is needed. The easiest method to enter query is lining up the keyword without specifying any conditions. In searching web, an information unit (one web page) which contains all keywords is abstracted. Although in searching video scene, there is no such information unit, so search engine have to abstract a scene by only the relation between keywords. Consequently the search engine selects the 26

34 scene whose intervals is related to each other, and show united intervals to user. In this research, the degree of relationship of a pair of intervals is estimated by following order. a pair of intervals is in inclusive relation apairofintervalshasanoverlap a pair of intervals do not have any overlap By using this order, when the user request the scene by lining up the keywords, the search engine retrieve scene by following process. 1. set the query that all keywords are in inclusive relation with each other 2. search scenes by the query 3. if the number of abstracted scene is not sufficient, relax the query and return to step 2. For determine the order to relax the query, set a score to each relation. In prototype, set a score 3 for the relation ContainsOrDuring, 1 for the relation HaveOverlap, and 0 for the relation BeforeOrAfter. For example, ContainsOrDuring(k1,k2) and HaveOverlap(k2,k3) are true, the score of scene which contains k1,k2,and k3 is 4. The search engine relax query in the order which the score is higher. The process of relax query is shown in figure 4.8 In this figure, the tightest query is Query 1. If sufficient number of scene is not abstracted by this query, search engine relax the query to Query 2, Query 3, and Query 4. The score of these queries is 7. The query is relaxed till the number of the abstracted scene is sufficient. After the number of scenes is sufficient, the search engine sorts the abstracted scenes. The order depends on the goodnessofthesceneandthedegreeof severity of query. The rate which is used to order the scenes is calculated by following formula. rate = ScoreOfQuery GoodnessOfScene (4.6) In the instance is shown in figure 4.9. The search engine shows user the abstracted scene in order of rate. 27

35 Figure 4.8: Process of query relaxation Figure 4.9: Calculate rate for each result 28

36 scene rate 1 Scene Scene Scene Scene Scene Scene Table 4.1: Past intervals correspond to long pass 29

37 Chapter 5 Experiment and Evaluation In this section, we describe a prototype system. After describing an overview of the system in section 5.1, we describe details of the system. In section 5.6, we evaluate effectiveness of the system. 5.1 System Overview Figure 5.1 represents an overview of system. The system can be divided into two parts. First part is Annotation part and second part is Search part. The search part can be divided into 3 steps. 1. Create thesaurus 2. Create tag information 3. Annotate movie At first, we created a thesaurus. There is some thesaurus which we can get fromtheweb,butweneedthedomainspecific thesaurus. In this prototype, we created a thesaurus for soccer video. Secondly, we created tag information. Tags are used in annotating movie. The information of tag consists of a keyword and tag type. The keyword is expressed by specifying a node of thesaurus. Tag information contains additional information depending on tag type. Tag types and additional information is described section 5.3. Finally, we annotated movie by using tag information. we simulated a situationthatwehavetoannotatemovieinreal-time. The search part can be divided into 3 steps. 1. Input query 2. Generate query graph 3. Match query graph from metadata At first, input query by lining the keywords. Secondly the system generate query graph from keywords. User can input query graph directly. After user input the query graph, the system matches query graph from metadata and show abstracted scenes to user in order the goodness of the scene. Environment of implementation is as shown below. 30

38 Figure 5.1: Overview of System 31

39 Development Software :VisualC#.NET OS : Windows XP SP1 CPU :Pentium42.53GHz Memory :512MB 5.2 Creating thesaurus We created thesaurus for soccer video. The captured image when we were creating thesaurus is shown in figure 5.2. The form of created thesaurus is tree structure. Each node has synonyms, and each edge represents the hypernym/hyponym relation. In this process, we made 37 nodes for common keyword of soccer and 16 nodes for player name and team name. We made player and team nodes about only Japan team. The number of synonym of each node is from1to5. Figure 5.2: Captured image in the process of making thesaurus 5.3 Creating Tag Information Figure 5.3 is a captured image when we were creating tag information. Tag information is used for simplifying the process of annotation. In annotation, we 32

40 describe only the ID of tag information at each time point. In search process, the search engine refers the tag information from ID. Tag information consists of following element. Figure 5.3: Captured image in the process of making tag information ID keyword tag type additional data Keyword is expressed by the node of thesaurus. So the keyword has to be in thesaurus. There is following tag types. Vague Interval Vague Modifier Starting Point Terminal Point Separator 33

41 If the tag type is Vague Interval, the tag is used for describing event interval by Vague Interval. The tag information contains the probability distribution of the starting point and the terminal point. This probability distribution is stored as additional data. We created probability distribution by past intervals in other soccer video. If the tag type is Vague Modifier, the tag is used for describing modifying keyword by Vague Modifier. The tag information contains distribution of interval of event which is modified by the modifier as additional data. If the tag type is Separator, the tag is used for separating video. For example, this is used to describe cut timing. An interval of the keyword of this tag information is a section enclosed with this Separator. The tag information contains accuracy as additional data. In the prototype, the interval of event Attack is described by Separator tag. If the tags whose type is Starting Point and Terminal Point are used to describe the starting point and terminal point of an interval in several. The tag information contains accuracy as additional data. 5.4 Annotating Movie We annotated movie by using created tag in tag information. Figure 5.4 is a captured image when we were annotating movie. In prototype, we described the second half of Japan vs. Belgium in World Cup The length of annotated segment is about 45 minutes. This process consists of following step. At first, tags are set corresponding to keyboard key. For example, set Pass tag to key P, and Goal tag to key G. And then play the movie and we describe tags by pushing the corresponding keyboard key when we recognize an event. For simulating the situation that we have to annotate in real-time, once we start playing the movie we did not stop playing. But we annotate movie in two operations because it is hard to describe enough tags at once. We assign different tags for each iteration on the assumption that movie is annotated in real time by two annotators. In first iteration, we describe events. Used tags whose type is Vague Interval 34

42 Figure 5.4: Captured image in annotation are Pass, Long Pass, Side Change, Clear, Cross, and Foul. Used tags whose type is Vague Modifier are Goal, Free Kick, Corner Kick, Off Side, Throw In, Goal Kick, Yellow Card, and Red Card. Used Tag whose type is Separator is Attack. In this instance we did not use tags whose types are Start Point and End Point. In second iteration, we describe player names. They are Suzuki, Yanagisawa, Nakata, and so on. We described the player of Japan team only. If we know well about both teams, describing both teams by one annotator is thought to be possible. We describe 450 tags in first iteration, and we describe 208 tags in second iteration. Second iteration is harder than first iteration because of difficulty of recognizing players. 5.5 Search Result We evaluate effectiveness of search algorithm by using metadata created in section 5.4. A captured image of search process is shown in figure 5.5. In this figure, user enters the query graph directly. User make node by entering 35

43 keywords and set relation between nodes. The result is shown in list view and a scene selected by user is played. Figure 5.5: Overview of System At first, we search goal scene. Entered query is shown in figure 5.6 and result is shown in table 5.1. Query graph for searching goal scene has a node whose keyword is Shot and a Modifier of Goal. As a result of search, 5 scenes are abstracted. There are 4 goal scenes in video. Recall ratio was 100 % and precision ratio was 80 %. One scene was not a goal scene. In this scene annotator describes goal because ball is in-goal, but the goal is cancelled by referee because of a foul. Secondly, we search a cross scene which leads to shot. Entered Query is shown in figure 5.7 and result is shown in 5.2. For searching a cross scene which leads to shot, we set condition that the relation of cross interval and shot interval is Meet. Query graph have two node, cross and shot, and one edge. TheedgeexpressesMeet relation and set the error to 2 seconds. As a result of Figure 5.6: Query 1 hits recall ratio precision ratio 5 4/4 4/5 Table 5.1: Result 1 36

44 Figure 5.7: Query 2 hits recall ratio precision ratio miss in annotation miss in retrieval 5 3/3 3/5 1 1 Table 5.2: Result 2 Figure 5.8: Query 3 search 5 scenes is abstracted. Precision ratio is 60 %. Two unsuccessful scenes are abstracted. In one scene the annotator mistake the shot scene for a cross scene. The rank of this scene is 3rd. In another scene a gap between cross and shot is too wide. The rank of this scene is 5th. Finally, we search Inamoto s pass scene. Entered Query is shown in figure 5.8. Number of abstracted scene is 62, and in 10 scenes whose goodness is higher than 0.4 is in these scenes. The result of top 10 scenes is shown in table scenes are unsuccessful because a miss in annotation and 2 scenes are unsuccessful because of a miss in retrieval. Scenes whose rank is lower than 11th contains many overlapping scenes. 37

45 hits precision ratio miss in annotation miss in retrieval 10 6/ Table 5.3: Result Evaluation As for the cost of annotation, we nearly reached our goal. We made metadata in two operations. In first iteration, we described 450 tags in 45 minutes. Tags are mainly representing occurrences of events. Tagging pace is about 6 seconds per tag, and there is margin to describe rare event. In second iteration, we described 208 tags in 45 minutes. Tags are mainly representing player s names of Japan team. Tagging pace is about 13 seconds per tag, but it was harder than first iteration. It is because of the difficulty of recognizing player. Describing both teams by one annotator is thought to be difficult. But its one reason is window size of video player( ), so if window size is more large or annotation is done in stadium, the difficulty of describing player name is thought to be reduced. In each operation we did not stop or rewind movie, so it is thought that two annotators can make metadata for practical use in real-time. As for the accuracy of annotation, there are some mistakes. In abstracted scenes after retrieval, roughly half of the wrong scenes come from misses in annotation. Misunderstanding of scene is more serious than deviation of timing whichwedescribetagat. Weoftenrecognized misunderstanding immediately, so operation which erases adjacent tag is needed. As for the accuracy of retrieval, there is room to make improvement. Roughly half of wrong scenes come from miss in search algorithm. The problems in result are following items. In some scenes the Vague Modifier is not work well. In abstracted scenes whose rank is lower, some scenes overlapped. The calculated relations between Vague Shot Intervals were reasonable. But in some scenes the Vague Modifier did not modify intervals well. The cause is 38

46 thought that membership function of Vague Modifier is determined subjectively. The membership function needs to be determined using some sort of statistical method. Overlapped scenes have to be combined if their rank is near with each other. If their rank has a big difference, the low rank scenes are thought that they have to be eliminated from abstracted scenes as result. 39

47 Chapter 6 Discussion In this section, we discuss the method to reduce the annotation cost. For this purpose we think about using image recognition and speech recognition. In addition, we discuss an application which will be more important when the technique of real-time annotation grow popular. 6.1 Multi-modal Annotation In prototype system, we annotate movie by using keyboard and attach an event tag at the time the event is recognized. But the method to attach tags is not only one way. The cost of annotation can be reduced by using other methods. Typical methods and its nature are shown below. Description Method Reliability of Timing Reliability of Meaning Button and Keyboard middle high Image Recognition high low Speech Recognition low middle Table 6.1: Past intervals correspond to long pass By using image recognition, we can recognize some information. For example, boundary of one cut can be recognized precisely. In addition, caption is recognized automatically which helps understanding of situation. By using audio recognition, we can estimate the importance of scene by the sound volume of audience or announcer, and we can recognize the content which the announcer says. There are various annotation techniques which use multimodal method [12][13]. In the technique we propose, using audio recognition will be effective. In our method, we annotate movie by describe the timing which the event is recognized. In the sports video, announcers often say a keyword when they recognize an event. So a part of the annotation process can be substituted by recognizing announcer s voice. When we use only the announcer s voice, we can search important scenes be- 40

48 cause the announcer often explain the scene if the scene is important. But there is gap between the timing announcer say and the timing announcer recognize, so the Vague Shot Interval have to be stretched to absorb the gap. 6.2 Query Generation from News Article By the growth of available digital video contents, the demand for summarizing video is increasing. There are various techniques to summarize video contents by using metadata. In Informedia[11], video skimming technique is achieved by using TF-IDF(term frequency/inverse document frequency) weighting. In this technique, if a certain keyword appears frequently in a segment and appears rarely in other segments, the keyword is considered to represent the segment. After getting representative keywords of segments, search intervals in which the announcer talks about the keyword by using image recognition and speech recognition. And then the skimmed video is generated by abstracting important segments of video. These methods are successful in summarizing video to understand overview in shorter time. But the summarized video is inferior in scene organization than a summarized video produced using manpower. Owing to this, we discuss a summarizing method using news article in this section. We often know the result of sports contents in news article. Once we know the result, we rarely watch whole contents even if it is recorded in our recorder. Incidentally, a sports article is good summary which is produced by a writer. So if a system can specify the scene correspond to a part of news article, it will be a good summarized video play abstracted scene along the article. The structure of a common news article is shown in figure 6.1 In the part of overview, the overview of this article is described. In sports news about match, this part consists of result, location, date, and so on of the match. The video scene correspond to the part of overview should be an overview of the match. So the corresponding scene is abstracted in order of level of importance. For example, the scene of goal, shoot, red card, and so on. In the part of body text, details of important scene are described. the part 41

49 Figure 6.1: Common structure of new article of body text has following structure shown in 6.2. A paragraph often has following nature. In a paragraph, the order of sentences matches the order of scenes in the video. In a paragraph, each sentence has common element. By using the keyword in the sentence and these natures of the paragraph, the scene correspond to a sentence is abstracted. A query which is used for specifying corresponding scene of the sentence can be generated by following process. 1. Abstract keywords which expresses an event from a sentence. 2. Abstract scenes using the query which is a set of abstracted keywords in step If the abstracted scenes are too many, narrow the scenes using common element of the paragraph. The thesaurus (described in section 4.3) is used forabstractcommonelement. 4. Select the combination of the scene which meet the order and the goodness of each scene is high as much as possible. The order of the scene corresponds to sentence in video have to match the order of the sentences in a paragraph. 42