Approach to Metadata Production and Application Technology Research

Transcription

1 Approach to Metadata Production and Application Technology Research In the areas of broadcasting based on home servers and content retrieval, the importance of segment metadata, which is attached in segment units such as per shot or scene, is rising. Advances are being made at the Science & Technical Research Laboratories on technologies research to efficiently index such segment metadata with the application of image recognition and natural language. We are also pioneering broadcasting services that utilize metadata. This article introduces the status of the research being done on metadata production and application. 1. Introduction Metadata plays an important role in areas such as broadcasting based on home servers and content retrieval. This is especially becoming more notable in relation to segment metadata, which is attached in segment units, such as per shot or scene. With the use of segment metadata, broadcasting based on home servers will enable the viewing of digest versions of programs, making possible the faster retrieval of desired scenes. While this indicates the potential of segment metadata attachment in promoting the reuse of program content due to easier content retrieval, segment metadata production is an extremely time-consuming process. NHK operates the NHK Archives for the archiving and application of its program content, having introduced the BEATUS system to aid with the content segment metadata indexing process. Although BEATUS currently systemizes program information during the program production phase, with the purpose of accumulating information that will become the basis for segment metadata, segment metadata indexing is still very difficult to perform for some programs, including live broadcasts. One of our laboratories' ongoing lines of research involves realizing efficient metadata production through the incorporation of image recognition technology and language technology. The inserted Figure 1 shows a conceptual image of this metadata production application that our research envisions attaining. It will attach semantic metadata to NHK's massive amount of archived video content, with the aim of providing video content in various presentation forms based on the metadata. The service closest to realizing the use of segment metadata is broadcasting based on home servers. In addition to this service, we have been conducting technological examinations on assumed services that retrieve and present extracted partial program content in a form similar to that of a multimedia encyclopedia. While the main features of current web services involve textual and still image presentation, the incorporation of broadcast programs will make it possible to include video. Some application examples may involve, "viewing a particular type of animal caring for its young in a nature program," "viewing a recipe for a particular ingredient Metadata production technology Image Image recognition Sound Speech recognition Natural language Multimodal integrated Knowledge Automatic metadata indexing Metadata extraction technology NHK's massive amount of video content Content Program archives Content provision over broadcast-wave, the Internet, etc. Provision Advanced content service using metadata Multimedia encyclopedia Broadcasting based on home server Object-linked data broadcasting AdapTV TV4U Professional production know-how provision Program production Knowledge base Intelligent production support system Figure 1: Goals for metadata production and application technology research 3

2 using a cooking program," or "obtaining information on treatment methods for a specific illness using a health program." Broadcast program content produced with solid professional newsgathering offers highly reliable information. Conventional methods of producing mini programs and video clips can no longer meet the everbroadening range of demands. If a relevant portion of a program can be automatically cropped out upon request for presentation, it will be feasible to meet such diversified requests. Another avenue of metadata application under examination is the pioneering of new types of broadcasting services with the use of metadata. These involve a viewing environment adaptive service, AdapTV, which adaptively presents content according to an individual viewer's needs and viewing situation, an objectlinked data broadcasting service that allows a user to obtain related information by content object unit, and a TV4U (TV for You) system, through which an amateur user can easily produce, publish, and view programs. During the process of advancing this research, we will develop a database containing program production expertise obtained through an analysis of professionallyproduced content, together with an intelligent program production support system using this knowledge base, with the aim of providing further improved programming. The following part of this article will introduce the approaches being made for the research on these metadata production applications. 2. Metadata Production Technology 2.1 Metadata extraction using image recognition The first step in segment metadata indexing is dividing a shot or scene into segments. We constructed a cut detection method, which divides a shot into rectangular fields and employs the patterned information changes to determine dividing points. This technology was also applied to create a summary picture sequence that extracts signature scenes from a drama in still picture format. With regard to research on metadata indexing for segmented data, attempts are being made to extract event data for target images from baseball and soccer game programs. In baseball game video content, we examined the differences in shooting sequence patterns for different events, such as a hit or a homerun, and extracted images of such events from program data. Specifically speaking, this process involves producing a video sequence pattern using a texture characteristic fractal sequence, which was then analyzed with the application of a hidden Markov Model to make a decision. In the case of soccer game video content, event detection, such as of goal and corner kick scenes, was performed through an analysis of formations from extracted player images. These developed techniques, such as object extraction and formation analysis, were applied in the construction of the ball trajectory plotter system B-Motion for baseball and an offside-line display system for soccer, which have been implemented in the broadcasting field. Since information related to who is in the image is important as metadata, research is advancing on face recognition technology. Unlike security authentication, an application to broadcast content requires a highlyaccurate determination of the identities of people from images with a wide range of conditions in terms of size, angle, facial expression, and lighting. To meet such a unique demand, we have constructed a recognition method that reflects the statistical characteristics of the variations in facial images for an individual, or between individuals. Other developments are being made on the research for a soccer player identification system using data fusion, or a combined application of these techniques (face recognition, uniform number recognition). 2.2 Metadata acquisition using sound and natural language Work is also underway on segment division based on sound and natural language. Using acoustic analysis, a technique was fabricated to accurately extract segments where the audience paid more attention based on a sound analysis of cheering in a soccer stadium. Research related to natural language realized the discrimination of portions of a running account and commentary in a game using sentence pattern information, to extract events and the agents related to the events from the running commentary in a broadcast. Using a combination of these methods, the duration and content of important events were extracted. While it seems that the employment of closed-caption data and speech recognition results would yield a perfect extraction of event content, this is not always the case. Broadcasts include cases where the announcer does not mention specific things about an image because it is visually obvious, or makes mention of a point even though it is not shown in the image. For instance, an onsite report may mention that "a monkey ransacked a house, and avoided capture," without showing an image of the monkey. A keyword search of video footage like this will output a result describing an image of a monkey, which will be inaccurate. To eliminate inaccurate result output, a technique was developed to extract only video footage with the target object in it, by means of automatically learning sentence characteristic differences from sentences with the relevant image and those without it. Due to the fact that an accurate extraction of significant metadata will also be greatly affected by the context of a sentence, advances are being made on context analysis. While speech recognition has already been implemented in producing subtitles for NHK news programs, studies to further improve recognition accuracy in sports broadcasting are also underway. Conventional closed-caption service in sports programming was 4

3 Feature Extraction module Cut detection by scene analysis Slowed-motion replay detection by video analysis Speech recognition of announcer's speech Player name and keyword extraction from announcement content Agent and event extraction by natural language Metadata server Program metadata (MPEG-7) Primitive metadata Primitive metadata Integrated metadata Editing module Metadata editor (integrated ) Segment integration + Contents integration Event extraction by video analysis Lawn detection by image Usage module Player name extraction by face recognition Important event segment extraction by cheering sound analysis Metadata conversion MPEG-7 TV Anytime Receiver for broadcasting based on home server Metadata for digest viewing, etc. Figure 2: Metadata production application system provided mainly through a re-speak method (a method that recognizes the speech of a separate speaker who repeats the regular commentary, called a closed-caption announcer). This was needed because a large portion of regular sports broadcast commentary includes exclamatory comments by an excited announcer, which had lowered the recognition accuracy. To attain higher recognition accuracy, an acoustic model designed for these exclamatory segments was incorporated to deal with this problem, and metadata extraction is being performed using this model. Content integration Segment integration Confirmation while watching program video 2.3 Metadata production framework The abovementioned types of alone will not be adequate to complete high-quality accurate metadata indexing. It is necessary to recognize the importance of multimodal integrated combining various methods, including image recognition and speech recognition. Our laboratories are working on this integrated with the goal of acquiring metadata with enhanced accuracy. The annual open house of the Science & Technical Research Laboratories in 2005 presented a demonstration of a metadata production application system, which included integrated, for J League football game video footage. This exhibition system, as shown in Figure 2, extracts metadata (called primitive metadata) using nine types of extraction modules, edits them with a metadata editor, and converts the edited results (called Content data for selected metadata Primitive metadata track Integrated metadata track Figure 3: Prototype metadata editor screen integrated metadata) into TV Anytime metadata for broadcasting based on home servers using a usage module. With the application of this converted metadata output for digest viewing, we presented a demonstration of digest viewing on a broadcasting receiver based on a home server. Multimodal integrated was incorporated into a metadata editor. The inserted Figure 3 describes a screen example of this metadata editor. The metadata editor realized multimodal including the temporal direction, by means of combined basic handling functions. For example, a scoring scene consists 5

4 of a series of event sequences, it may start with loud cheering from the audience, followed by a slow-motion replay, and a close-up shot of the scoring player's face. Multimodal that includes the temporal direction, consisting of image recognition, speech recognition, and language, makes the detection of event content in a series of sequences feasible. This metadata production application system was constructed based on an interface definition, with a view toward the easy incorporation of various future extraction technologies. This definition governs the interface standards for the metadata model and the metadata server shown in orange in Figure 2. We determined and implemented the following interim standard specifications: for the metadata model, a standard based on MPEG-7 CDP (Core Description Profile) with a restriction on derivation and nesting, and for the metadata server interface, a standard for handling XML messages in SOAP (Simple Object Access Protocol). Constructing future modules in compliance with these standards will allow for the easy integration of new techniques, which are to be developed by various research institutions independently, resulting in highly-accurate metadata production through the mutual employment of know-how and expertise from universities and other research organizations. We believe that the combined application of modules produced by experts in the area will maximize the effectiveness of multimodal integrated. While the present standards for the metadata model and interface are interim specifications, a near-future release of the information to the public, along with a metadata editor, is planned based on the results being obtained from the prototype implementation model. 3. Metadata Application Services 3.1 AdapTV Research is advancing on a viewing environment adaptive service, AdapTV, as a new broadcasting service using metadata. AdapTV will present the same content in different formats, adaptively adjusted to individual viewers' needs and diverse viewing situations. One example of this service is the adjustment of the display method for a broadcast, in relation to resolution and other differences based on the different terminals used to watch the content, as an example, from the viewer's living room or on a train. Knowledge differences between adults and children will also necessitate changes in the wording used or the way explanations are given. To realize this type of service, AdapTV transmits additional metadata for adaptive display conversion along with regular content, as shown in Figure 4, which is used by the receiver to perform an intelligent content conversion for display, considering the user's viewing environment/situation (viewing display size/resolution, viewing location such as at home or outside the home, hobbies and preferences, knowledge level, etc.). This "conversion" involves not merely the signal level but also the content meaning level. Our current examinations include adaptive displays involving display resolution, barrier-free adaptive displays, and knowledge-level adaptive display systems. As for display resolution adaptive displays, we are studying conversion schemes that will enable the display of the important portion of a video image when using terminals with different resolutions, such as a car TV or portable TV, through a consideration of the program content when adjusting the display reduction, along with the metadata necessary to realize this service. In research on barrier-free adaptive displays, a Content adaptive conversion display that corresponds to user's viewing situation Converted display through media conversion, content conversion in content meaning level. AdapTV content (essence + metadata) Broadcasting network, cellular phone network, the Internet, etc. Transmission channel AdapTV engine Intelligent conversion display Viewing environment profile TV i Knowledge base (intelligent conversion rule, etc.) Viewing situation (home, outside home, etc.), terminal (display size, display capacity, etc.), and user (hobby, preference, knowledge level, comprehension level, etc.) Figure 4: AdapTV conceptual image 6

5 Feature Preview A Visit to Osore-zan The second report on... Let's make a program like this based on my last trip. I wonder if anybody finds it interesting? My Broadcasting Station (publishing) Internet This program sounds interesting. Let me watch it in my favorite program viewing style. Program request Upload Download Viewing screen When we got off the bus, I was at... Production screen TV Creator (production) TV Browser (viewing) Figure 5: TV4U prototype conversion display system that utilizes voice read-out, an enlarged font size option, and re-layout display (for the weak-sighted) was constructed for data broadcasting content that is difficult for visually-impaired users to see. For the prototype model construction, we also fabricated a conversion scheme that takes a step inside the meaning of the displayed content, using the data broadcasting description language BML and templates, such as for binary tables. Studies on knowledge level adaptive displays advanced in examining and implementing a scheme that changes the presentation style based on a viewer's knowledge level, by repeating or skipping part of the content of the language programming material used. The degree of a user's understanding was determined through statistical of the operation record for a remote control device. Future work in this research will involve determining the metadata needed for AdapTV, developing an automatic user profile construction method, and the construction of a knowledge-base. 3.2 Object-linked Data Broadcasting Object-linked data broadcasting is a broadcasting service that allows a user to specify a subject (image object) in an image, such as a person or an object, by pointing at it with a mouse or a finger, in order to obtain detailed information related to that specified image object. Object-linked data broadcasting transmits metadata describing object shape and location, and its related information, synchronized with video and sound data to the receiver, which are then used to display related information upon an object specification by a user, using the metadata of the corresponding object. This research was conducted as the research project "R&D on an object-linked data broadcasting system" entrusted by the National Institute of Information and Communication Technology during fiscal 2000 and fiscal TV4U TV4U is a system that enables users, even those with no experience in program production, to easily produce a program with CG and synthesized voice by simply writing script on a word professor. This produced program can then be published over the Internet. The letter U in the TV4U system represents the fact that it is User friendly, Unique, Upon request, and Useful. TV4U will make TV production and publishing, which had only been available to a few professionals, more common and familiar to the general public. As Figure 5 shows, the TV Creator, which has easy data entry capability similar to that of word processor, can produce a TV program with appropriate direction from a simple textual script. TV Browser allows a user to search through the programs accumulated in the server according to individual preferences, and then view these programs. It also has the capability of presenting the program based on the viewer's preferences, such as in the production style of choice, or selecting only the desired content in the program. Metadata indexing is provided for the retrieval of materials and direction styles. As for TV Creator's subset, we have made a content production viewing tool available to the general public (freeware) as the TVML (TV Program Making Language) Player-Mini, and recommend that everybody try it. 4. Conclusion The research related to metadata production and application being conducted at the Science & Technical Research Laboratories was introduced in this article. Metadata extraction is not a task that will be accomplished immediately. We feel that a desirable approach to the advancement of this research involves, not only the effort being put forth at our laboratories but also collaboration with universities and other research institutions. Therefore, we believe that it is important to disclose the framework for metadata production and application. (Nobuyuki YAGI) 7