MetaSMIL : A Description Language for Dynamic Integration of Multimedia Content

Transcription

1 Master Thesis MetaSMIL : A Description Language for Dynamic Integration of Multimedia Content Supervisor Professor Katsumi TANAKA Department of Social Informatics Graduate School of Informatics Kyoto University Takayuki YUMOTO February 6, 2004

2 MetaSMIL : A Description Language for Dynamic Integration of Multimedia Content Takayuki YUMOTO Abstract Content integration gets many attentions as a new style of content use. Content integration is to make new content by presenting several content spatially and temporally. However, many of research about content integration do not consider about existence of various users, such as aggregator, author, etc. Moreover, users demand is more diverse. Static integration, which is achieved by SMIL and other multimedia description language, can not satisfy this. (Static content integration is integration hand by hand.) Therefore, dynamic content integration is necessary. Dynamic content integration is integration to generate integrated content dynamically by using query. However, there is no framework to describe dynamic integration easily. Moreover, when we make integrated content, we must negotiate with authors who make materials for integration. It is very laborious. However, there is no framework to describe the authors intentions for content integration in the machine processable way. In this paper, I propose a description language for dynamic content integration, called MetaSMIL. I classify author side users to two types of users : Authors : Right holders of content which is used as materials for integration. Aggregators : Users who create integrated content. MetaSMIL aims at both users. MetaSMIL has two features : Integration script : This is for aggregators to design integrated content. Integration constraints metadata : This is for authors to describe constraints for integration. Its major characteristics can be summarized as follows: 1. Dynamic unit identification, 2. Integration constraints for content which are integrated with, and 3. Integration query including dynamic retrieval using content, which is dy- i

3 ii namically decided. By dynamic unit identification, we can define a flexible unit such as Its URI is p 1 and its duration is less than 30 seconds. Flexibleunitsleavesomeoptions of unit for aggregators when integration query is executed. Integration constraints aim at describing constraints for integration situation which can not be known when constraints are described. Integration constraints consist of : req/prh relationships compositional constraints xreqystands for if unit x is used, unit y must be used, and xprhystands for if unit x is used, unit y must not be used. These relationships have some axioms and theorem. By using them, implicit conditions are derived and verbose conditions are removed. By using compositional constraints, we can express constraints for content which is integrated with. Integration query is the selection rule of presented content. This query consists of three parts : search, restrict, and rank. In this query, content, which is decided dynamically, can be used as criteria of retrieval. I designed MetaSMIL as XML-based language with reflecting on these technologies. I developed prototype system for MetaSMIL. This system converts MetaSMIL script to SMIL content with reflecting on MetaSMIL metadata. By using MetaSMIL, various applications are realized. Contributions of MetaSMIL are summarized as follows: MetaSMIL provides beneficial technologies for both users who provide content and users re-use content. Therefore, MetaSMIL promotes content re-use and distribution. MetaSMIL has higher expressive power than conventional constraint expression in digital right management. High expressive power of MetaSMIL enables authors to express various conditions and protects the right of authors. Although content right problem is not focused in information integration research, I focus on content right problem. MetaSMIL provides technology to solve right problem on integration.

4 iii MetaSMIL : マルチメディアコンテンツの動的統合のための記述言語 湯本高行 内容梗概コンテンツ統合はコンテンツ利用の新たな形態として注目を集めている. コンテンツ統合とは複数のコンテンツを空間的, 時間的に配置することによって, 新たなコンテンツを生成することである. しかし, コンテンツ統合に関係する研究の多くは Aggregator や Author などさまざまなユーザの存在を考慮していない. さらにユーザの要求は多様であり,SMIL やその他のマルチメディアコンテンツの記述言語で実現されるような静的なコンテンツ統合ではその要求を満たせない.( 静的なコンテンツ統合とは手作業でコンテンツの選定や統合を行う統合手法である.) ユーザの要求を満たすには動的なコンテンツ統合が必要である. 動的なコンテンツ統合とはクエリなどを用いて動的に統合コンテンツを生成する手法である. しかし, 現在, 動的なコンテンツ統合を簡単に実現するフレームワークは存在しない. また, コンテンツ統合の際には素材となるコンテンツを作成した制作者との交渉が必要であるが, それは非常に面倒である. しかしながら, 特に統合に関して制作者の意図を自動処理可能な形で記述するフレームワークは現在, 存在しない. 本稿では, 動的なコンテンツ統合のための記述言語 MetaSMIL を提案する. 制作者サイドのユーザを以下の 2 つに分類する. Author : 統合の素材に用いられるコンテンツの権利保持者 Aggregator : 統合コンテンツを制作するユーザ MetaSMIL はこの双方のユーザを対象にしている.MetaSMIL には以下の 2 つの役割がある. 統合スクリプト : Aggregator が統合コンテンツを設計するのに用いる. 統合制約メタデータ : Author が自分のコンテンツが統合に用いられるときの制約を記述する. MetaSMIL の主要な特徴は以下のとおりである. 動的な情報単位 ( ユニット ) の同定 統合される相手のコンテンツに対する統合制約

5 iv 動的に決定されるコンテンツを用いた検索機能を含む統合クエリ動的なユニットの同定では柔軟なユニットを定義できる. 柔軟なユニットとは例えば, URI が p 1 で表されるビデオのうち, 長さが 30 秒以内の任意の区間 などのユニットである. 柔軟なユニットにより Aggregator が利用するユニットに自由度を持たせることができる. 統合制約は制約を記述した時点では未知である統合時の状況について制約を記述することを目的としている. 統合制約は以下からなる. req/prh 関連 composition 制約 req/prh 関連において,x reqyは ユニット x を使用する場合はユニット y も使用しなくてはならない,x prhyは ユニット x を使用する場合はユニット y は使用してはならない ことを意味する. これらの関連には公理や定理が存在し, これらの公理や定理を利用することによって, 明示的には示されていない制約の発見や冗長な制約の除去が可能になる. また,composition 制約を用いることによって, 統合される相手のコンテンツに対する制約を記述することが可能である. 統合クエリは呈示されるコンテンツの選択ルールである. このクエリは Search,Restrict,Rankの3つの部分からなる. また, クエリの中で動的に決定されるコンテンツのキーワードなどを検索に用いることも可能である. MetaSMIL はこれらの技術を反映した形で XML を用いて設計した. また, MetaSMIL のスクリプトから統合制約メタデータの内容を考慮した形で SMIL コンテンツを生成するプロトタイプシステムを実装した.MetaSMIL により, Contenxtual News や WebTelop などさまざまなアプリケーションが実現できる. この MetaSMIL は以下のような意義がある. コンテンツを提供する側と再利用する側の双方にメリットがある技術を提供することによって, コンテンツの活発な再利用 / 流通を促進している. ディジタル権利処理において, より表現能力の高い制約記述手法を導入することによって, 新たな利用を促進している. 情報統合において, 従来注目されていなかったコンテンツの権利問題に注目し, 統合時に権利問題を解決するための技術を提案している.

6 MetaSMIL : A Description Language for Dynamic Integration of Multimedia Content Contents Chapter 1 Introduction 1 Chapter 2 Related Work MultimediaContent InformationIntegration DigitalRightManagement... 4 Chapter 3 Problems and Approach Background Problems Approach DynamicContentIntegrationLanguageMetaSMIL OverviewofMetaSMIL MetaSMILMetadataDescription MetaSMILScriptDescription Chapter4 TechnologiesforMetaSMIL Dynamic Unit Identification Requirement for Unit Identification Definition and ClassificationofUnit DecisionofUniqueUnit Unit Modification IntegrationConstraints Requires/ProhibitsRelationships VerificationofIntegrationPossibility CompositionalConstraints IntegrationQuery RetrievalFacilityofMetaSMIL RetrievalProcessofMetaSMIL Reflection of Integration Constraints to Integration Query 28

7 Chapter 5 Implementation and Applications DesignofMetaSMIL IntegrationConstraintsMetadata IntegrationScriptDescription PrototypeSystem Architecture Example Applications ContextualNews WebTelop Chapter 6 Discussion SocialContributionofMetaSMIL ExpressivePowerofMetaSMIL Query Constraints Chapter 7 Conclusions 44 References 47

8 Chapter 1 Introduction Although various forms of web and video content are currently available, user demand is more diverse. To satisfy this, I adopt dynamic content integration that produces new content by presenting several types of content simultaneously or sequentially. In SMIL[1] and other multimedia description languages, we can describe content integration statically. (i.e. we can integrate content hand by hand.) However, static content integration cannot yet satisfy the diversity of user demands. If content complements itself, a greater variety of content becomes available. I introduce dynamics to content integration and this means we do not need to describe the actual URIs, but the rules that will decide what content will be integrated dynamically. Content is retrieved and presented basedontheserules. Presentedcontent,however,isnotonlyweb-basedbutalso video content. Presented units of video content must be dynamically decided in dynamic content integration. Although dynamic content integration is useful, it is difficult to produce such content in a conventional multimedia description language such as SMIL. We need a language that will achieve dynamic content integration easily. Moreover, in content integration, the authors intentions on what content can be used for integration is needed. However, there is no framework to describe the authors intentions on content reuse, especially in content integration. In this case, we must negotiate with authors who make materials for integration. It is obstacles for content integration. To overcome these obstacles, we need a framework to express authors intention for integration. In this paper, I propose a description language for dynamic content integration, namely MetaSMIL[2]. MetaSMIL has two features, a content integration script and a metadata description of the users intentions (constraints for integration). MetaSMIL script is written by users who wish to integrate content (I call them Aggregators), and MetaSMIL metadata is written by content authors (I call them Authors). Its major characteristics can be summarized as follows: 1. Dynamic unit identification of content that is integrated into synchronized content, 1

9 2. Integration constraint for content which are integrated with. 3. Integration query including retrieval of content through pre-defined retrieval criteria The rest of the paper is organized as follows: The chapter that follows describes the work related to this research. Chapter 3 describes problems and approach for them. Chapter 4 explains technologies for MetaSMIL. Chapter 5 explains implementation of MetaSMIL and its prototype system, and applications of MetaSMIL. In chapter 6, I discuss about contribution of MetaSMIL and expressive power of it. Chapter 7 is the conclusion. 2

10 Chapter 2 Related Work 2.1 Multimedia Content SMIL (Synchronized Multimedia Integration Language)[1] is a language to describe multimedia content. In this language, users describe spatial and temporal layout of materials of multimedia content and design multimedia content. However, SMIL has no retrieval facility. This can only describe static content integration, not dynamic content integration. MetaSMIL script is added to retrieval facilty to SMIL. MPGS[3] is an interactive tool to generate of multimedia presentations. With this tool, we can create multimedia presentation based on spatio-temporal constraints through the GUI. This system, however, does not support authors intentions. Multimedia Formatting Vocabulary[4] is used to create a multimedia style sheet. Although we can describe some queries and constraints in Multimedia Formatting Vocabulary, MetaSMIL can be used to describe a greater variety of queries and constraints such as conditions about surrounding content. 2.2 Information Integration There is some work that permits dynamic content integration. Ma et al. propose WebTelop[5]. WebTelop synchronously presents web pages, which complement video content, with it. Presented web pages are dynamically retrieved by using queries which are generated from tv closed caption. Figure 1 is the screen shot of WebTelop. Henzinger et al. research query-free search[6]. In query-free search, queries are also automatically generated from tv closed caption and retrieval is executed by using these queries. Terada et al. propose active karaoke[7]. The active karaoke system retrieves suitable pictures in the database by using the song lyric and presents the song and the pictures synchronously. Most of these work can only be applied to limited applications. We have no framework to design easily integrated content except for MetaSMIL. 3

11 Figure 1: Screen shot of WebTelop 2.3 Digital Right Management The TV-Anytime Forum[8] has been trying to standardize various specifications for technologies on personal TVs. Various working groups independently prescribed the specifications at this forum. The working group related to our research is the Metadata Working Group, which is currently discussing content description and user profiles[9]. The TV-Anytime Forum has also a working group that is concerned with digital rights protection and management. Recently,muchattentionhasbeenfocusedonmanagingthedigitalrightsof digital content. MPEG-21 (Multimedia Framework)[10], in particular, is aimed at a digital content framework for the next generation. Its specification consists of eight parts and they includes digital item declaration, rights protection and operation. Integration constraints metadata of MetaSMIL corresponds to Part 5 the Rights Expression Language (REL)[11] and Part 6 Rights Data Dictionary (RDD)[12] in MPEG-21. The current MPEG-21 prepares a description 4

12 framework just for conventional reuse of content such as playing, copying, and printing. It has not yet provided an adequate description framework concerned with the author s intentions on the reuse of stream content and a content composition mechanism. These metadata, which supports digital right management, considers only conventional content use (copy, print, re-distribution, etc.), but does not consider new content use, i.e. content integration. Creative Commons[13] provides RDF[14]-based license metadata for digital work. In Creative Commons, users express the eleven condition as the combination of following four license (refer to Figure 2). Attribution : If they give you credit, copy, distribution, display are allowed. Nocommercial : Copy, distribution, display are allowed for nocommercial purpose. No Derivative Works : Copy, distribution, display are allowed, but derivative works based on it is not allowed. Share Alike : Distribution of derivative works is allowed under the same license as original work. Figure 2: License of Creative Commons In Creative Commons, we can easily get condition metadata by answering three questions in the homepage of Creative Commons. This metadata are not also fully prepared for reuse of digital work. In other words, it does not fully describe the authors intentions. In our work, we can describe the authors intentions by a description similar to predicate logic. RuleML[15] is a markup language to describe predicate logic. In our work, we use a more intuitive method of description way than this. We have no framework to describe the author s intention based on the way content should be reused on integration except for MetaSMIL. 5

13 These D.R.M descriptions are sometimes provided as metadata. Therefore, I compare conventional metadata work with MetaSMIL metadata. Although MPEG-7[16] is major metadata to describe content description of videos, it has no expressive power to describe D.R.M description. MPEG-21 aims at all digital works. It has expressive power to describe both, content description and D.R.M description. Creative Commons can not describe content description but it provides simple right description for D.R.M. D.R.M description of MPEG-21 and Creative Commons aims at conventional content-use, such as copy, print, distribution, and so on. MetaSMIL metadata is specialized in right description on content integration. Therefore, some description, which are easily described in MPEG-21 and Creative Commons, are difficult to describe in MetaSMIL metadata. However, MetaSMIL metadata has enough expressive power to describe right description on content integration, which can not be described in any other metadata. Figure 3 shows differences between MetaSMIL metadata and other metadata. Figure 3: Differences between MetaSMIL metadata and other metadata Furthermore, MetaSMIL is the framework which realizes both purposes: Design of integrated content by using retrieval facility, and Description of integration constraints for material of integration. Otherworkcannotsatisfybothpurposes. 6

14 Chapter 3 Problems and Approach 3.1 Background Various and great many content are available by the web. Many of research focus how to get valuable information. There are following types of research to get information from the web: Knowledge Discovery : Its purpose is to extract important information from the web. Information Integration : Its purpose is to integrate information in the web, and enhance them. Some of information integration research integrate knowledge gotten by knowledge discovery. Information integration and knowledge discovery are not exclusive each other. I focus the role of content (for example, html content, video content, etc.) in information integration. Content is one of the ways to express information, sometimes includes some presentment. It is difficult to discover all information of content without the lack for conventional knowledge discovery. So, content integration is very important. I define content integration is to generate new content by presenting several content temporally and spatially. Content integration is the good way to integrate information and present them effectively. In theconventional ways, integrated contentiscreated hand by hand. It is very laborious to gather material for integration and make integrated content. Furthermore, in order to satisfy diverse demand of users (I call them the consumers), dynamic content integration is especially important. In the view of this definition, HTML[17] and SMIL[1] can be said to be content integration language. Their differences with MetaSMIL, which I propose, and these languages are summarized as Figure 4. 7

15 Figure 4: Aspect as Integration Language In this figure, a horizontal axis is expressive power of languages and a vertical axis is supported user model. Frame facility of HTML enables us the simplest content integration, which is to present several content in the same time. Furthermore, SMIL enables us to synchronize several content. We can integrate content syntactically as above. However they are not enough. 3.2 Problems First problem is user model, which is premised in HTML and SMIL, is not enough. In HTML and SMIL, right problems about content which is used to make new content, are thought to be solved. In this case, we can regard author side authors as only one. (single author model. Refer to Figure 5 left.) However, there are many types of author side users, such as the authors of the material, the character in the material, authors to make the new content by aggregation of the materials, etc. Therefore, it is very hard to apply to single author model. I adopt the model to classify author side users to two types of authors: Author : A right holder about materials of content. Aggregator : A user who makes integrated content. Figure5rightshowsthismodel. 8

16 Figure 5: Differences of User Model (Consumer is a user who browses and uses content.) Semantic integration is important with mediation of authors and aggregators. Second problem is that it is very laborious to make integrated content. Dynamic content integration is a useful new way of utilizing web and video content. In conventional languages and systems, however, it is difficult to express dynamic content integration based on languages. In particular, there are no frameworks to describe users intentions regarding integration of web and video content. I need these functionalities: Metadata description language that describes users intentions based on both the conditions for the content and conditions that are concerned with the surrounding content as well as A content integration mechanism that composes dynamic content based on their intentions 3.3 Approach I focus on following technologies as integration technologies for both authors and aggregators. unit identification integration constraints integration query 9

17 Authors describe conditions to use their content as integration constraints. Aggregators describe integration query to generate integrated content. Content are integrated by the execution of integration script. Integration constraints are reflected to units, which are used for integration. Units are identified when integration constraints are reflected to integration query. To achieve them, I propose MetaSMIL[2]. MetaSMIL consists of : 1. Integration constraints metadata : Authors describe constraints for their content about integration. 2. Integration script : Aggregators design integrated content by using this. Integration constraints metadata is for authors to describe constraints about integration as metadata. Integration script is for aggregators to design integrated content. By using it, they can describe stream-type content like SMIL. Furthermore, retrieval facility is available. By using retrieval facility, dynamic integration is realized. Although MetaSMIL looks like SMIL, it differsfromthisinitsdynamic unit identification and retrieval capabilities. Indeed, MetaSMIL works just like the meta-mechanism for conventional SMIL. MetaSMIL and mechanism can provide the framework to integrate multiple information sources. Each description realizes following applications. Example of integration constraints metadata: constraints for neighboring commercial video Beer company A wants not to present beer commercial video of company K before or after its video. (refer to Figure 6) Figure 6: Constraints for Neighboring Content 10

18 Example of integration script: dynamic integration of video and web pages Ma et. al propose WebTelop[5]. WebTelop retrieves web pages which are related to given tv-program video, and synchronize them with video scenes. By using integration script of MetaSMIL, we can express simplified algorithm of WebTelop, and dynamic integration of video and web pages. 3.4 Dynamic Content Integration Language MetaSMIL Overview of MetaSMIL MetaSMIL consists of: Integration script and Integration constraints metadata Script description describes the scenario for content integration, such as the timing for the presentation of content and the URI of presented content. Aggregators describe script description. Instead of describing URI, query description makes content integration dynamic. In script description, query description is a selection rule for presented content. Metadata description is used to describe constraints for integration. If authors describe metadata descriptions about their content, their intentions are reflected in deciding presented content. Figure 7 shows overview of MetaSMIL. Figure 7: Overview of MetaSMIL 11

19 3.4.2 MetaSMIL Metadata Description We can not only create multimedia content in SMIL 2.0[1] but also attach meta information to it. This meta information depends on Dublin core[19] metadata sets. In many cases, materials for SMIL content are assumed to have been made by the author of SMIL content. This meta information is suitable for being embedded in SMIL content itself. However, as materials and MetaSMIL content can be separately described by different authors in MetaSMIL, we can describe the authors intentions with referential approach. The authors intentions (integration constraints) consists of : Conditions for integrated units (Internal conditions) and Conditions for context in content integration (External conditions). Internal conditions include the dynamic unit identification of content and external conditions have restrict description by content authors MetaSMIL Script Description In MetaSMIL script, we describe following information: spatio-temporal layout of material for integration, and query for integrated content. Figure 8 shows an image of script. Figure 8: Image of script description 12

20 Spatio-temporal layout can be described in SMIL. However, query can not be described in SMIL. In SMIL, we specify material for integration by the URIs. In MetaSMIL, we can also use them. Furthermore,wecanusequeryinsteadof the URIs. This query is selection rule of presented content. This query facility realizes dynamic content integration. 13

21 Chapter 4 Technologies for MetaSMIL To archieve purposes of MetaSMIL, there are following three technologies: unit identification integration constraints, and integration query Unit identification technology enables us to identify unit dynamically and realize various way of unit use. Integration constraints technology enables us to express various constraints about content integration. It can also describe context of integration. (How and What content is integrated with?) By using integration query technology, we can express various type of query function. 4.1 Dynamic Unit Identification Requirement for Unit Identification Authors have a variety of constraints placed on the use of their content. Some of these may be concerned with time, and others may be concerned with content. For example, an author may permit his/her content to be reused under conditions where any content substream is allowed to be integrated if the length of the stream is less than 15 seconds. Another kind of time-related constraint may be when his/her content is allowed to be integrated during an author-specified time period. An example of a more property-based constraint is where an author-specified portion of the substream is included whenever his/her content has been used. These types of constraints are concerned with where and which portion of an author s content may be used. We can express not only which unit can be integrated (this is absolute unit identification), but also the variety of identified units that can be declared for integration. For example, Units A and B should always be reused together, An arbitrary substream within 30 seconds of unit C can be used, and An arbitrary unit with the keyword= ichiro can be used. Unit identification is expressed as metadata described by content authors Definition and Classification of Unit Idefine basic information unit (I call it unit ) asfollows: 14

22 resources which can be expressed by URI, unit split by time, and collection of them. Each unit has properties such as author, updated data, media type, keyword, and so on. It denotes u.property. And there are relationships such as temporal and spatial relationships between units. Types of units can be classified to two criteria: atomic or compound : one unit or collection of units absolute or conditional : unit is decided to be unique or unit is decided to be unique In this paper, we call the unit which satisfies following conditions unique unit. the case of unit unrelated to time : URI is static. the case of unit related to time : URI, start time and finish time are static. At the combination of two criteria, I can classify types of units to four types: atomic-absolute : unique unit which is defined by URI and time Example : Its URI is p 1, its start time is 10 seconds, finish time is 30 seconds. (This situation sometimes occurs, when sport video is used for broadcasting use.) atomic-conditional : an unit which is defined by URI and some conditions or queries Example : Its URI is p 2 and its duration is less than 30 seconds. compound-absolute : a set of units by enumeration of units Example : Their URIs are p 1,p 2,..., p n compound-conditional : a set of units by conditions or queries Example : a set of units which have keyword k. All atomic-absolute units are unique. Not all atomic-conditional units are unique. If compound-absolute unit includes atomic-conditional units, it is not always unique. If compound-absolute unit does not include any atomicconditional units, it is unique. Not all compound-conditional unit is unique. I call a unit which is not unique a flexible unit. Flexible units leave some options of unit when integration query is executed. 15

23 4.1.3 Decision of Unique Unit Integration query decides an unique unit from a flexible unit. I show an example. An author defines a flexible unit whose URI is p 1 anddurationislessthan30 seconds. This is an atomic-conditional unit. Some parts of content, whose URI is p 1, have keywords. (Refer to Figure 9 upper part.) An aggregator describes following integration query: choose a unit, whose keyword is ichiro and duration is maximum. By the execution of this query, a flexible unit is uniquely decided. (Refer to Figure 9 down part.) Figure 9: Example of Decision of Unique Unit Unit Modification A unit that is specified by an author (represented as u Au ) should be used in content integration. If this does not match the unit defined by the aggregator (represented as u Ag ), we need to adjust u Ag by using u Au. u is a part of video content, st(u) andfin(u) are the start and finish times, and dur(u) is duration. S Au is a set of units defined by the authors. A suitable unit is a unit that includes u Ag, a member of a set S Au and whose duration is minimum. Match Au (u Ag ) returns a suitable unit and L represents candidates for suitable units. For each u Au S Au, which satisfies st(u Au ) st(u Ag ), when fin(u Ag ) is greater than st(u Ag ), candidates are added to L as follows. If fin(u Au ) fin(u Ag ), u A u is added to L. If fin(u Au ) <fin(u Ag ), u Au 16

24 Match Au (u Ag(2) ) is added to L. u Ag(1) = split(u Ag,st(u Ag ),fin(u Au )) u Ag(2) = split(u Ag,fin(u Au ),fin(u Ag )) Afunctionsplit(u, t 1,t 2 ) means extracting a part of u on t 1 <t<t 2. When fin(u Au ) fin(u Ag )(refer to a) in Figure 10), because of u Au u Ag = u Ag,itisclearthatu Au L. When fin(u Au ) <fin(u Ag )(refer to b) in Figure 10), if u Ag is split by axis t = fin(u Au ), area t fin(u Au )resultin a) of Figure 10, because of fin(u Au ) fin(u Ag(1) ). u Au is a candidate in area t fin(u Au ). About area t>fin(u Ag ), Match Au (u Ag(2) ) is obtained in the same way. Therefore, u Au Match Au (u Ag(2) )isaddedintol. Match Au (u Ag )isu, which has minimum dur(u) inl. Figure 10: Derivation of candidates for suitable units 4.2 Integration Constraints The major purpose of integration constraints is to describe constraints for integrationnsituationwhichcannotbeknownwhenconstraintsaredescribed. Integration constraints consists of: request/prohibition for unit using, and conditions of unit Iexpressfirst condition as compound unit or by using requires/prohibits relationships. I express second condition as compositional constraints by using relationships between units and property of unit 17

25 4.2.1 Requires/Prohibits Relationships Definition Idefine requires relationship (req) and prohibits relationship (prh) to express use conditions and prohibition conditions. x, y are units and req, prh denotes: xreqy:ifyouuseunitx, you must use unit which includes unit y. xprhy:ifyouuseunitx, you must not use unit which includes unit y. However, I define as follows: xprhy6 xprhy 0 (y 0 y) (1) Under the following supposition (2), the following constraint can be described. If you want to use unit x, you must not use unit whose duration is greater than 30 seconds in y (1), but you are allowed unit whose duration is smaller than 30 seconds in y (2). xprhy xprhy 0 (y 0 y). (2) I will prove that we can not describe above constraint under the definition (2). The underlined constraint (1) is described as follows: xprhy 0,y 0.dur 30s, y 0 y Isupposethaty 00 y and y 00.dur 30s. y 00 is sometimes included y 0, i.e. y 0,y 00 y 0. I get following formula. xprhy 0,y 0.dur 30s, y 0 y xprhy 00,y 00 y 0,y 00 < 30s xprhy 00,y 00 y 0,y 00 < 30s conflicts with the underlined constraint (2). We can not describe above constraint under the definition (2). Therefore, I adopt the definition (1). If you want to prohibit any y 0 such that y 0 y, you describe as follows: xprhy 0,y 0 y (3) 18

26 Characteristics of requires/prohibits relationships req satisfies as follows: ref lectivity : x req x (4) transitivity : (xreqy) (y reqz) xreqz (5) (x, y, z are units, and y 0 y) Figure 11 shows an image of transitivity. Figure 11: Image of transitivity req also satifies following formula. Figure 12 is an image of the formula (6). xreqy xreqy 0 (6) Figure 12: Image of derivation of req When x 0 x, req satisfies following formulas. partial reflectivity : xreqx 0 (7) partial transitivity : (xreqy) (y 0 req z) xreqz (8) 19

27 Figure 13 shows an image of partial reflectivity, and figure 14 shows an image of partial transitivity. Figure 13: Image of partial reflectivity Figure 14: Image of partial transitivity I show a proof of partial transitivity. (x reqy) (y 0 req z) (x reqy) (y reqy 0 ) (y 0 req z) (partial transitivity) (x reqy) (y reqz) (transitivity) xreqz (transitivity) When x 0 x, x 0 req x, x is defined a minimum unit of integration. (Refer to Figure 15.) From the formula (6), following abbreviation is gotten. (x reqy) (x reqy 0 ) xreqy (9) 20

28 Figure 15: Image of a minimum unit Next, I explain about characteristics of prh. prh satisfies following formulas. x 0 prh y xprhy (10) xprhy 0 xprhy (11) (x reqy) (y prhz) xprhz (12) (x reqy) (y 0 prh z) xprhz (13) Figure are images of the formulas (10) (12). Figure 16: Image of the formula (10) Figure 17: Image of the formula (11) 21

29 Figure 18: Image of the formula (11) I show a proof of the formula (12). (x reqy) (y 0 prh z) (x reqy) (y reqy 0 ) (y 0 prh z) (partial reflectivity) (x reqy) (y prhz) ((12)) xprhz ((12)) From the formula (11), following abbreviation is gotten. (x prhy) (x prhy 0 ) xprhy 0 (14) Because of the definitions, following formula are always false. xprhx 0 (15) (x prhy 0 ) (x reqy) (16) Figure 19 is images of the formulas (15),(16). Figure 19: Image of formulas (15), (16) Under the condition : v, w are units, conditions set F is gave, we have the formulas (15) or (16) by using the formulas (4) (14). Then, unit x cannot be integrated. 22

30 4.2.2 Verification of Integration Possibility v, w are units and conditions set F consists of condition vreqwor vprhw. Relationship units x and y can be integrated is defined F does not logically imply xprhy. This relationship is described as x r y. I also define unit x and y cannot be integrated as F logically implies xprhy. This relationship is described as x r y. Now, I explain about the algorithm to judge whether we have x r y. First,Idefine as follows: req (u) = {v u req v, v is logically implied, v is a unit} (17) prh (u) = {u u prhvorvprhu, u is logically implied, v is a unit} (18) u is a unit. Next, calculate req (x),req (y),prh (x),prh (y) by using the formulas (17), (18). If (req (x) prh (x) 6= φ) (req (y) prh (y) 6= φ) (req (x) prh (y) 6= φ) (req (y) prh (x) 6= φ), then x r y. We have following formula: req (x) prh (x) 6= φ v, x r v req (y) prh (y) 6= φ v, y r v Compositional Constraints In 4.2.1, targets of requires/prohibits relationships are explicit units. Units which satisfy some conditions can also become targets of them. These conditions are expressed by using spatio-temporal relationships and property. In spatial relationship, we express relative locations between units. Spatial relationships consist of three types of relationships: neighbor-of: units are in the same window, separate: units are in a different window, and overlap: units overlap each other. Figure 20 shows the spatial layout. In temporal relationships, we express temporal relative relationships by using Allen s time interval relationships[18] expanded by time diffrence parameter diff. MetaSMIL prepares following properties: 23

31 Figure 20: Spatial layout keyword : primal keywords, author : authors of unit (content), dur : duration of unit (if a unit is related to time, such as video, music, etc.), and etc. MetaSMIL also prepares following functions: sim(u 1,u 2 ):similarityofu 1 and u 2, st(u),fin(u) :starttime,finish time, diff(t 1,t 2 ):difference of t 1 and t 2,and etc. By using these properties and functions, we can describe unit condition. For example, I suppose the condition that if you use the unit u 1, unit which has more than 0.5 similarity must synchronously be presented with u 1. This condition is described as follows: u 1 req u, u :(sim(u 1,u) 0.5) (u neighbor of u 1 ) If a unit u satisfies conditions, we can describe as follows: u : conditions (19) Authors can describe constraints by using requires/prohibits relationships and compositional constraints. I define following description as primal form of the constraints. u 0 req u, u : conditions (20) u 0 prh u, u : conditions (21) Constraints which authors can describe are logical sum or product of these 24

32 forms. The formula (20) is equivalent to the formula (22). The formula (21) is equivalent to the formula (23). (u 0 req u 1 ) (u 0 req u n ) (u i : conditions) (22) (u 0 prh u 1 ) (u 0 prh u n ) (u i : conditions) (23) In verification of integration possibility, constraints are translated to form of the formulas (22), (23) from the formulas (20), (21). These constraints are verified by algorithm explained in section Integration Query Retrieval Facility of MetaSMIL In MetaSMIL, we can choose unit by using query instead of describing URI. MetaSMIL provides query using explicit keywords. For example, if you want to get a unit whose keyword is ichiro, you should describe following query: u = geturi( ichiro 00 ) AfunctiongetURI retrieves content with conventional search engines by using argumentsaskeywords. Furthermore, in MetaSMIL, we can retrieve by using properties of other content. For example, I suppose that a unit u, u has the same keyword as neighboring content v, is result of retrieval. We can describe this as follows: u = geturi(v.keyword), v:(vneighbor of u) In this case, you can use keywords of u as query, even if you do not know keywords of u. I call it implicit keyword. If v is changed, then u is also changed. So, MetaSMIL provides templates for content integration. Figure 21 shows an image of integration template. 25

33 Figure 21: Image of integration template Retrieval Process of MetaSMIL Dynamic retrieval of content is executed by users who wish to integrate content to determine presented content dynamically.thisisexecutedinthreesteps. 1. Search : obtain a set of candidates for the presented content 2. Restrict : filter through restrict conditions 3. Rank : sort with the value ranking function These steps are represented by the following formula. u = f Rank (f Restrict (S Search )) (24) Here, we will describe logical function L(u) in the restrict process, and ranking function R(u) in the rank process. Functions f Restrict and f Rank satisfy f Restrict (S) = {u L(u) = 00 true 00,u S} (25) f Rank (S) = {u j R(u j )=max, u j S} (26) Figure 22 shows an image of these process. Figure 22: Image of process 26

34 I show an example. I suppose that a unit u is a retrieval result. u has the same keyword as a unit v, which is presented in neighbor of u. The author of u is A. u is the latest content. This query is described as follows: u = geturi(v.keyword), v:(vneighbor of u) u :(u.author = 00 A 00 ) (u.date = max) Table 1 shows correspondence of MetaSMIL process and the model description. Table 1: Correspondence of process and description Process Description Meaning Search u = geturi(v.keyword) collect units whose keyword is the same as v. Restrict u.author = 00 A 00 The author of u is A. Rank u.date = max The latest unit is chosen. These three steps are expressed as script described by aggregators. Moreover, by using restrict description, content authors can express their intentions about the context of their integrated content. Search In this process, we obtain a set of candidates of presented content from the web or local storage. In other words, we choose the domain of presented content. There are two functions, i.e., geturi and getvideosegment. Function geturi is used to obtain a set of web content using a web search engine. Function getvideosegment is used to obtain video units with video-content metadata [8, 16]. Restrict In this process, we check the conditions that integrated content should satisfy. In other words, we check the critical condition for presented content. By using content properties with these, we can express various types of context. For example, similarity of contents A and B must be more than 0.5 is described as sim(a, B) > 0.5. We can also use description similar to predicate logic to express more complex conditions. By this description, we can describe conditions that a unit, 27

35 which satisfies some conditions, exists or not. For example, we can describe the prohibition of integration with content that has keyword K. Logical operations such as and, or, not, can be used to specify these conditions. We can also refer to the dynamic decided attribute value in condition description. For example, we can describe the same keyword as content presented right before. Rank In this process, content is sorted by a ranking function defined by users and that with a maximum ranking value is selected. In other words, we check preferable conditions for presented content. The ranking function is described as a composition of primitive functions, which are used to express complex conditions in the restrict and rank processes. Primitive functions consist of functions concerning content description (similarity, keyword, etc.), temporal functions (start time, finish time, duration time, etc.), arithmetic operations, logical operations. For example, if we want content A, which is similar to both content B and C, the ranking function could be expressed as 0.5 sim(a, B)+0.5 sim(a, C). Although ranking description makes content authors express various intentions, it is sometimes annoying. In search process, we use existing search engines, and candidate content has some ranking score. If ranking description can be omitted, these scores are used to select content. Therefore, authors of integrated content can choose either the original ranking algorithm or the popular ranking algorithm Reflection of Integration Constraints to Integration Query By reflecting integration constraints to restrict process of integration query, authors intentions and aggregators intentions are integrated. u Ag is a unit which aggregator demands. The restrict process by aggregators is expressed as follows: u Ag Ru,u: conditions (R isreqprh) S search is a set of candidates generated in search process. Here, let U S Search. I suppose that : u s(i) S Search (1 i n) (27) 28

36 In the formula (27), if u Ag is replaced by u s(i), we get following condition: u s(i) Ru,u: conditions (R isreqorprh) (28) When u s(i) can not be integrated is logically implied from a condition set, which consists of this condition and constraints defined by authors, then U U \{u s (i)}. After the execution of above process for each u s(i) (1 i n), then f Restrict (S Search )=U. By this process, integration constraints by authors are reflected to integrated content designed by aggregators. 29

37 Chapter 5 Implementation and Applications 5.1 Design of MetaSMIL MetaSMIL consists of integration constraints metadata and integration script. I explain the method to express them by XML tags Integration Constraints Metadata In the integration constraint metadata, authors describe constraints on their content. This metadata consists of two parts : internal condition : target of metadata, and external condition : constraints for content which is integrated with. Internal condition corresponds to unit identification technology. Therefore, in internal condition, a flexible unit is available. External condition corresponds to integration constraints technology. In MetaSMIL, this condition is described as follows: Internal condition is described in target tags. In MetaSMIL, atomic-absolute units and atomic-conditional units are available. If you describe start tags and finish tags and describe static value, an atomic-absolute unit are defined. If you describe other tags or the range of value, an atomic-conditional unit are defined. condition is described as follows: 30

38 This part corresponds to external condition. External condition of u Au is described as follows: u Au Ru,u: conditions (R isreqorprh) u Au req u corresponds to following description. <unit>(relationship or P roperty) + </unit> u Au prh u corresponds to following description. <not><unit>(relationship or P roperty) + </unit></not> Compositional constraints is described in the part of Relationship or P roperty. I show an example. The constraint for commercial video in chapter 3 is described as follows: Integration Script Description Integration script consists of two pars: a layout description part and an integration query part. In a layout description part, we describe spatio-temporal layout of units. In order to describe spatial layout, we use relationship shown in Figure 20. And, in order to describe temporal layout, we use following relationships: seq : present several units sequentially. par : present several units synchronously. Figure 23 shows an image of temporal layout. 31

39 Figure 23: Temporal Layout A Layout description of MetaSMIL is simplied description of spatio-temporal layout of SMIL. By using this, integration script is expressed as follows: In an integration query part, we describe URI or queries which consist of search, restrict, rank. In MetaSMIL, functions are described by XML tags. A name of tags indicates a name of function (f name), and arguments are described between these tags. Condition is described in the same way with integration constraints. Simplied WebTelop in chapter 3 is described as follows: 32

40 33

41 5.2 Prototype System Architecture I developed a prototype system for MetaSMIL. This system converts a MetaS- MIL script to SMIL content. It consists of two subsystems, Layout Tags Converter (LTC), and Presented Content Decider (PCD). The LTC converts the layout description of the MetaSMIL script to SMIL tags. The PCD determines the presented content including the query description of the MetaSMIL script and MetaSMIL metadata. The system first obtains candidates for content by obeying search part in the query description. Second, the system checks the conditions of the integration constraints metadata. Next, the system checks condition of metadata of candidates content and filters candidates. Finally, system selects presented content through the ranking value. Figure 24 shows the system for this prototype. Figure 24: Prototype system Figure 25 shows the interface of this system. In this system, we can select the location of a script file and metadata file. Then it converts a MetaSMIL script to SMIL content. 34

42 Figure 25: Interface of prototype system Example I show an example of the execution. Figure 26 shows input file of the script. Meaning of this script is as follows: 1. First, present the opening scene of the tv-program video alone. 2. Next, present commercial video related to opening scene of video. Commercial videos are stored in local storage. 3. Finally, present the main scene of tv-program video and web page related to main scene. I prepared two input files : Video about travel in Copenhagen. (Its keywords are travel and Copenhagen.) Video about cooking using a sea bream. (Its keywords are cooking, food, and a sea bream.) Figure 27 shows the execution images of examples. If the tv-program video is only changed in script of Figure 26, the presented commercial video and the presented web page are also changed. I confirmed that : retrieval by content which is integrated with is available, and MetaSMIL script works as template for content integration. 35

43 Figure 26: Example of input script 36

44 5.3 Applications Figure 27: The timeline of the execution I explain two applications of MetaSMIL, i.e., contextual news and WebTelop. They are also expected to used in personal TV Contextual News The purpose of contextual news is to complement the main news with background and follow-up news. Background news and follow-up news are collectively called related news. Background news is right before the main news. Follow-up news is the latest news, which has the same keyword as the main news. The duration of related news is less than 30 seconds. I stored news program with closed captioning broadcast by NHK (Japan Broadcasting Corporation) from September 9, 2002 until April 25, We use closed-caption data as the metadata. The original content had the following constraint: This description means the duration of the composable unit was less than 30 seconds, and therewerenounitswhoseauthorswere not NHK before or after this unit. This constraint is described as follows: 37

45 In this case, the conditions concerning authors did not conflict. I try to complement the news on North Korea on April 17 with our prototype system. Background news was found on April 16, and follow-up news was discovered on April 25. The region surrounded by unit tags whose id is bg is background news, main and fu correspond to main news and follow-up news. The meaning of each region is as follows: In the search process, candidates for related news are selected by using the same keyword as main news has. In the restrict process, candidates for background news are filtered by condition date(main news) date(candidate), and candidates of follow-up news are filtered by condition date(main news) date(candidate). In rank process, the latest content is selected as related news. The MetaSMIL script for contextual news is as follows: 38

46 39

47 5.3.2 WebTelop WebTelop[5] is an application that retrieves web pages concerning on video scenes dynamically, and presents video scenes and web pages synchronously. WebTelop can be realized by several algorithm. I show one of the simplified algorithm. This application presents web pages whose similarity with video scenes are greater and whose similarities with previous presented web pages are smaller to provide new information for users. Figure 28 is a representation of WebTelop. Figure 28: Image of WebTelop Scene i is a part of video the content. Scene k is a part of video the content that must be used. We can easily describe WebTelop in the MetaSMIL. (Refer to Chapter 3.) The region surrounded by unit tags whose id is scene means the video unit is decided by the metadata. The region surrounded by unit tags whose id is web means the following: In the search process, candidates for presented content are selected by using the same keyword as a corresponding the video unit. In the restrict process, HTML content is filtered. In the rank process, content whose 1 sim(web i,web i 1 ) value is the greatest, i.e., content that is not similar to web i 1 is selected. 40