An Information-integration Approach to Designing Digital Video Albums

Transcription

1 An Information-integration Approach to Designing Digital Video Albums C. Madhwacharyula*, W. Jun*, Y. Weiqi*, J. Yi*, A.S.V Radhakrishna*, S. Bissol*, J. Charlson Yu*, Z. Qiuying*, S.H Srinivasan*, H. Hassan Abdulredha*, P. Mulhem**, M. S. Kankanhalli* School of Computing*/IPAL-CNRS** National University of Singapore Abstract In this paper we present the architecture of the DVA (Digital Video Album) system. Information-integration is the key principle utilized in this system to allow for content-based indexing, intuitive access and retrieval of digital video. Our implementation of the system has demonstrated the efficiency and efficacy of such a design approach. 1. Introduction The last few decades have witnessed a dizzying rate of technological innovation in the areas of computing and communication. While the effects of cheaper and faster computation power manifest explicitly in many places, slightly less obvious are the technological advances in sensor and signal processing technologies. Their impact is being increasingly felt in the form of digitization of all forms of communication and media. These advances have directly led to increased inexpensive communication bandwidth, which in turn has spurred the rapid acceleration of the Internet globally. In terms of consumer electronics devices, we are currently witnessing the mass-scale switchover to digital cameras and video players from the traditional analog ones. The main aim of the DVA (Digital Video Album) system described in this paper is to develop techniques for contentbased indexing, and retrieval of digital video. While conceptualizing the project we made two important assumptions. Firstly that people want to share their home videos and secondly video presentations need to be customized for maximum impact. Based on these assumptions we formulated the basic design philosophy of DVA which is mainly based on the following two rules. Rule 1 emphasizes on minimal user intervention through the use of semi automatic methods and learning. Rule 2 states that information integration and propagation should be implemented to maximize reuse. This project focuses on solving anticipated problems due to increasing affordability and proliferation of digital cameras and digital camcorders as well increase in the digitization of traditional content. The central thrust for the basic research is to use an "object - based" approach for bridging the gap between the content feature space (which are a result of applying basically automated techniques for compactly characterizing images and video) and the semantic space (a human being's idea of what a particular image or a video represents -- could be buildings, scenery, people or events). The project aims to aggregate content features like color, texture, shape and motion vectors to obtain intermediate level objects (which may or may not correspond to real - world objects). Then computer - vision and pattern recognition techniques can be applied to attach meaningful descriptions to these objects which can subsequently be used for indexing and retrieval of images and videos. Promising results in human face detection, image segmentation using color and automated class - based indexing have been obtained. The paper will follow the following outline. Section 2 gives an overview of the related existing video album systems. Section 3 discusses architecture of the DVA system. Finally, Section 4 presents the conclusions and directions for future work. 2. Related Works A number of inexpensive and easy to use video editing tools like Ulead Video Studio 6.0[15], Movie Maker 2.0[18], Adobe Premiere 6.0 [1] etc. are commercially available which help in capturing the video, building a media library of clips, trimming the clips, assembling them in order, adding transitions between clips, adding text titles, applying special effects, and adding sound, music, and voice-overs. Hitchcock [4] is a home video editing system, which presents a user interface that supports semiautomatic video editing. This system describes the problems that non-professionals have in using existing video editing tools and provides users with an interactive system for composing video that does not require manual selection of the start and end points for video clips. Zhang and Ma[8] detail a system for indexing and browsing home videos. The system is capable of extracting both the structure information and the semantic objects. FlyAbout [3] is a system uses spatially indexed panoramic video for virtual reality applications. The VideoAnnEx annotation tool [16] developed by IBM assists authors in the task of annotating video sequences with MPEG-7 metadata. Most of these systems do not have features like object tracking, face detection, summarization tools and automatic annotation of videos based on these features which are handled by our system. 1

2 3. System Architecture The DVA (Digital Video Album) system contains three layers of architecture as shown in Figure 1. The first layer is the storage layer which contains the video metadata database which stores the annotated information of the videos in the form of XML files, the video database which stores the actual videos being entered into the system and images database which stores the details and the still images which form a part of the Digital Video Album. The second layer is designated as the Function Layer. The function layer uses the information available in the first layer to perform several low and high level operations on the video data. Like face detection, object tracking, video annotation and query processing, presentation and summarization etc. Figure 1: Software Structure of Digital Video Album (DVA) System The workflow of the system is as follows. When a new video is entered into the system, the first step is to annotate the video details in the form of an XML file. The annotation process stores the details of not only the video but also the different objects or people occurring in the video, the relevant frames in which they occur in the video, the sequences in the video etc, This detailed annotation system helps in efficient querying and retrieval of results using our query system for retrieving videos or sequences based on a particular query. The system also allows operations like face detection, object tracking, shot detection and audio video mixing. A presentation and summarization module allows the user to create video presentations and summaries from existing videos based on the desired criteria. The Function Layer consists of the following modules. 3.1 Face Detection/Identification Model References [13][5][6] detect human faces from video stream and save them into video metadata database. Face detection is a learning process wherein once the face of a particular person is recognized, the next time a video containing the same face is entered into the system, it automatically recognizes the person s face. The DVA system performs the above process and prompts the user asking if the person in question is the same one whose details are already available in the database. If the user answers in the affirmative then the details of the person are automatically annotated for the latest video. This process saves the user the tedious job of reentering the information about the same people who occur in different videos and follows the rules of DVA design philosophy by reflecting the intelligent semi automatic annotation processes of the DVA system with learning. We have also presented a novel experiential based sampling technique[9] which has the ability to focus on the analysis's task by making use of the contextual information and past experiences. Based on this, a sampling based dynamic attention model is built by sensing the experiential environment. Sensor samples are used to gather information about the current environment and attention samples are used to represent the current state of attention. Face detection is only performed on the attended samples to achieve robust real time processing. Experimental results have been presented to demonstrate the efficacy of our technique [9][20]. Object tracking[12] is very important for automatic indexing of video content. Video has in the past been indexed in terms of its constituent shots, or in a more sophisticated manner into context-based strata. One of the relatively difficult jobs is to index the content of the video especially in terms of the objects of interest present in the video. For this the indexing system would record the frame numbers and the screen-coordinates where the object(s) of interest is (are) present in a video. Such information related to the content of video can be made available to the indexing system by means of an object tracker. An object tracker is an indexing tool that tracks objects of interest (as indicated by user or otherwise), through the frames of a video clip along the objects optic axis. Two sensors, one using motion vectors and the other using DCT coefficients obtained from compressed video stream, provide measurements for the location of the object being tracked. The optimal estimate from the two measurements is found using Kalman filtering based state vector fusion approach. In the DVA project the tracker plays an important role for tracking objects chosen by user, tracking faces, shot detection and video summarization. 3.2 Indexing and Annotation Model The architecture of the DVA indexing module is as shown in Figure 2. The indexing/annotation system [11] shown in Figure 2 is a stratification model which captures objects and events. It consists of two types of metadata sources, the user manual input and the automatically extracted features. The former is the text input according to our predefined schema to describe video, the latter includes object tracking features, human faces and shot boundaries which are generated by the computer with the launching action of human. XML is chosen as the annotation medium because considering the increasing popularity of XML; its usage will facilitate interoperability with other metadata standards in the future. 2

3 Figure 2: Architecture of DVA Index/Annotation model Video annotation standards like MPEG-7 [17] set XML (Extensible Markup Language) as the language of choice for the textual representation of content description. Our approach provides a simpler and more customized method of video annotation with respect to home videos as compared to MPEG7 because MPEG-7 contains additional complicated features not necessary for the DVA application and tools for comprehensive utilization of MPEG-7 are not yet available. Since we use XML-based metadata storage, it is relatively easy to write a translator to convert our format to MPEG-7 if such a need arises. The indexing annotation module consists of the following sub modules: Annotator enables users to annotate video data and objects appearing in the video interactively. It can communicate with the video metadata database by performing the operations save, extract, modify and delete with respect to the video metadata. Description schema is the predefined XML schema for describing video and object. It will validate the format and content of the video/object description information in Annotator during XML parsing. Object linker performs the function of maintaining the relationship between low level features and its object, that is, make the features belong to the specific object in video metadata database. Data Storage is composed of two parts, one is video metadata database which includes the information about video, objects/people and their low level features, and the other is video stream database. There is a match between both of them so that data could be synchronized. For example, if a new video clip is added in video stream database, there will be a corresponding video description file in metadata database which records the annotation information. Whenever the system opens a video file, the annotator extracts all the information about this video from metadata database. If this video is a new file to data storage, then the annotator creates a corresponding description XML file. When annotator gets the user manual input annotation, it keeps the information in the XML description file dedicated for that video. After parsing and validating the file by means of description schema and XML parser, the annotator saves this file into metadata database. If user launches other operations such as delete or modify, the annotator does the matching process with the XML metadata. When users perform operations like face detection and object tracking, the system generates the related features. Those low level features do not indicate the high level meaning such as what/where the object is present etc. Thus the object linker presents the information about all the objects and people in the current database to the user. With the help of user or face recognition technique, the object linker links the low level features and their object. Then the meaningful features can be accepted by annotator which saves all the information. The Video Query Model makes use of the annotated information to retrieve results based on the queries put forward by the user. The architecture of the query model is as shown in Figure 3: Figure 3: Query Model of DVA system The user interface in the application layer contains the following models as shown in Figure 3: Query Composer obtains user s query expression when he/she searches for specific videos and translates it into a normalized final expression Annotator and Data Storage are defined in the previous section. Video Player decodes MPEG stream and displays it for the user from video database. Presentation Tool is a multimedia editor that allows user to combine and organize different multimedia elements such as texts, images, audio and videos into a presentation. The Query Composer normalizes the query expression and calls the query processing function which first searches the video metadata database, and then calculates the distances between the query and videos. The ranked results are saved according to the distance and displayed in the interface. The query kernel provides the relevant information to other models. It extracts the related video metadata when the user performs an operation like running or manipulating video clips. The Application layer as shown by Figure 1 is the final layer of the DVA architecture. It provides functional tools to the end users. It calls the function models in Layer 2 without accessing Layer 1 directly. This layer consists of the DVA user interfaces through which the end user 3

4 interacts with the system. Interfaces have been designed to allow the user to perform all the functional operations present in the Functional Layer like Annotation, Object tracking, Query Processing etc. Tools are also provided for playing the video clips and for creating presentations and summaries. A user interface has been developed which integrates all the functionalities of the DVA system in a simple manner to encourage any user without specialized computer knowledge to use it. A screen shot of the general DVA interface is shown in Figure 4. It has three main components. The color motion activity bar for non linear navigation can be seen as shown by (1) in Figure 4 gives information about the dominant colors present in the frames and the level of the motion activity. (2) shows the annotation tool which is used to annotate the video and (3) is the DVA video player which plays the selected videos and is used to select the objects in the video for face detection and object tracking. 3 Figure 4: Screen Shot of the Digital Video Album (DVA) System Interface The Application Layer consists of the following modules in Figure DVA Summarizer and Presentation Tools The DVA summarizer and presentation tool [7] comprises of 5 major sub modules as shown in Figure 5. The first one is the Application Front-end module that deals with providing an interface for the summarizer and thus handles interaction between the user and the system. Second module is the MPEG codec, which is further divided to two parts: the MPEG decoder and encoder. The MPEG-2 decoder serves two purposes: it extracts features from the input video as well as decodes the video frames for display in the program. Whereas, the MPEG-2 Encoder exports the summary video to a MPEG-1 or MPEG-2 file. Thirdly, the Video meta-data contains various video descriptions. Fourth is the Project Management module that deals with the saving and loading of summarization projects. Lastly, Feature Extraction and Adaptive Clustering modules work together to summarize the input video using various algorithms. 2 1 Figure 5: System Architecture of Semantic Summarizer The Semantic summarizer is closely tied with two major modules in the DVA system: the annotation module and the presentation tool. The annotation module is the input to the summarizer, thus it provides the necessary information for semantic processing such as the frame sequences of a certain person or object specified. The presentation tool, on the other hand, makes use of the summarizer to facilitate the creation of interesting presentations for the user. This is made possible by the fact that the video summarizer can be used to select certain relevant frames. In creation of a presentation, users may wish to focus on certain people or events, and the video summarizer provides that flexibility. Moreover, the video summary may vary in length depending on the required length of the presentation. Thus the information from varied sources is integrated and reused in accordance to Rule 2 of the DVA design philosophy. The Presentation module provides an editor that allows user to combine and organize different multimedia elements like texts, images, sounds and videos into a presentation. It basically integrates all the information generated by the video album into a user friendly and relevant form. It consists of two modes, manual and automatic. In the manual mode the user can himself choose the structure of the presentation and add the chosen music to it. In the automatic mode the user just selects a pre defined template and the presentation is automatically created according to the template. Examples of templates can be birthday, marriage, meeting, holiday etc for which different backgrounds, music etc are picked accordingly by the system. It uses the summarization module for abstraction of videos. It allows the video to be fit in the presentation time and adds effects to the presentation like text and music, based on the type of presentation being made. The output is in the form of a SMIL[19] file which is a W3C standard and allows developers to mix many types of media, text, video, graphics, audio and vector based animation together and to synchronize them to a timeline. Audio Visual Mixing [10] is also an important component of the presentation tool. The video and audio features are matched at the semantic level. Then picture-sound matching criteria in cinematography are used to do automatic audio mixing. 4

5 Text is automatically extracted, manipulated and added to the presentations by our XML transformation module [2] which takes input from the video editor automatically whenever the video sequences are merged or chopped for making presentations. The XML metadata is also automatically changed and attached to the created presentation. Hence the presentation tool encompasses most of the modules of the DVA system and makes use of all of its features to create state of the art presentations customized to suit the need of the user thus encompassing the underlying rules governing the DVA project. Another interesting component of the Digital Video Album is the Artifact Handling module [14]. Although this is not a direct component of the DVA system but it is valuable in increasing the quality of the videos and presentations in the system. In digital video albums, most of videos are accompanied with artifacts. Most of the videographers of home videos are amateur and they are short of equipment, conditions, time, space etc. Thus, the quality of home videos is obvious inferior to that of professional videos. The artifacts of home video are generally classified into two categories: The artifacts from camcorder improper setup and the artifacts from incorrect operations. Both of the artifacts are computable and adjustable from the viewpoint of digital signal processing. The former artifacts, such as lighting, contrast, hue and sharpness, depend on camera setup. We are able to detect and remove these artifacts based on digital image and video processing. The latter kinds of artifacts are caused by incorrect operations or improper control, such as shaking, fogging, color-loss and motion blurring. We have identified three kinds of these artifacts namely video lighting and shaking adjustment, video colorizing and video rhythm adjustment. We implement video correction by using rule based techniques, matching based techniques and video analogies based techniques. Video Analogy refers to the ability to improve the quality of videos by utilizing computable video features. Thus when a new video is entered into the system, it first undergoes the process of artifact handling described above to increase the quality of the video. Then the rest of the modules like annotation, querying, presentation etc can subsequently be utilized. 4. Conclusion In this paper, we show the novel system architecture of the DVA system. The architecture is based on the DVA design philosophy which emphasizes on minimal user interaction through semi automatic methods of learning and maximum information integration and reuse. We will use the knowledge and experience gained from DVA project for further research on video analogies and media understanding. References [2] C. L. Madhwacharyula, "XML Document Instance Transformations," Master Degree Thesis, National University of Singapore, [3] FX Palo Alto Laboratory (FXPAL), FlyAbout [4] Girgensohn, A., Boreczky, J., Chiu, P., Doherty, J., Foote,J., Golovchinsky, G., Uchihashi, S., and Wilcox, L. A semi-automatic approach to home video editing. CHILetters: Symposium on User Interface Software and Technology (UIST), 2(2):81 89, [5] J. Wang, "Detecting and tracking human faces in compressed video for content based video indexing," Master Degree Thesis, National University of Singapore, [6] J. Wang, R. Achanta, M.S. Kankanhalli and P. Mulhem, "A hierarchical framework for face tracking using state vector fusion for compressed video," ICASSP 2003, Hong Kong,,pp 28-38, April 2003 [7] J.C.S. Yu, M.S. Kankanhalli and P. Mulhem, "Semantic video summarization in compressed domain MPEG video," Proc. IEEE ICME 2003, Baltimore, July [8] Ma, W.Y. and Zhang, H. J.. An indexing and browsing system for home video, EUSIPCO'2000, 10th European Signal Processing Conference (Tampere, Finland, Sept ), pp.5-8. [9] J. Wang, M.S. Kankanhalli Experience based Sampling Technique for Multimedia Analysis, ACM Multimedia, Berkeley, California, Dec 1-2, 2003 [10] P. Mulhem, M.S.Kankanhalli, H. Hassan and J. Yi, "Pivot vector space approach for audio-video mixing," IEEE Multimedia, April-June pp [11] Q.Y. Zhang, M.S. Kankanhalli and P. Mulhem, "Semantic video annotation and vague query," Proc. 9th International Conference on Multimedia Modeling (MMM 2003), Taipei, January 2003, pp ,. [12] R. Achanta, J. Wang and M.S. Kankanhalli, "A sensor fusion based object tracker for compressed video," Proc. Sixth International Workshop on Advanced Image Technology IWAIT2003, Nagasaki, Japan, January [13] T.S. Chua, Y.L. Zhao, M.S. Kankanhalli, Detection of Human Faces in Compressed domain for Video Stratification, The Visual Computer, Vol. 18, No. 2, April 2002, pp [14] W.Q. Yan and M.S. Kankanhalli, "Detection and removal of lighting & shaking artifacts in home videos," Proc. ACM Multimedia 2002, Juan Les Pins, France, December pp [16] Ulead, [15] VideoAnnEx, Annotation Tool, IBM [17] [18] [19] [20] [1] Adobe Systems, Inc., Adobe Premiere 5