Four Promising Multimedia Databases and Their Embodiments

Transcription

1 Multimedia Tools and Applications 5, (1997) c 1997 Kluwer Academic Publishers. Manufactured in The Netherlands. Four Promising Multimedia Databases and Their Embodiments Y. YAGINUMA, T. YATABE, T. SATOU, J. TATEMURA AND M. SAKAUCHI Institute of Industrial Science, University of Tokyo, Roppongi , Minato-ku, Tokyo 106, Japan Abstract. Multimedia database systems have become more and more important as the tool to extract and generate additional values from multimedia Contents. In this paper, four multimedia database systems are proposed from the view point of promising contents sources; the Network multimedia databases, the Stream MM database systems, the Library MM database systems, and the Real world MM database systems. Important problems to be solved, i.e., what to do, are also discussed for each databases. Three concrete multimedia systems by authors research group, are then introduced and discussed as the embodiments of these multimedia systems; (1) the open Global Image Retrieval and Linking System, GIRLS, for mediation WWW data pace as the network MM database systems, (2) the flexible multimedia database platform GOLS, and (3) the higher level authoring system for the Stream MM environments. Keywords: muiltimedia database, image retrieval, multimedia platform, authoring 1. Introduction Multimedia systems are expected as promising carrying vehicles for creating new services and businesses in the coming twenty-one century. Generally, they are consisted of three major components; Contents in the form of video, voice and images, Distribution for communication tools and Platform for various types of utilization. Especially, Contents which include multimedia soft, video soft, etc., are important to generate additional values directly for the users. On the other hand, Multimedia Database Systems have been defined in the narrow sense as tools for querying efficiently multimedia data. However, the concept and role of Multimedia database systems should be extended here in order to be responsible for contents to generate new services. In this paper, we consider Multimedia Database Systems as the tools to extract and generate additional values from multimedia Contents with putting emphasis on the mediator functions between users and contents. Firstly, promising multimedia data contents sources of On the network, In the digital broadcasting stream, In the library, and In the Real World will be discussed. Four types of multimedia database systems are then proposed corresponding each contents source environments; the Network multimedia databases, the Stream MM database systems, the Library MM database systems, and the Real world MM database systems. Important problems to be solved, i.e., what to do, are also discussed for each databases. Three concrete multimedia systems by authors research group, are then introduced and discussed as the embodiments of these multimedia systems; (1) the open Global Image Retrieval and Linking System, GIRLS, for mediation WWW data pace as the network MM database

2 66 YAGINUMA ET AL. systems, (2) the flexible multimedia database platform GOLS, and (3) the higher level authoring system for the Stream MM environments. 2. Four promising multimedia database systems What are the basic functions (what to do) of the multimedia database systems to provide users with additional values from contents? Before discussing this, we need to imagine where promising multimedia contents in the coming multimedia society are generated and accumulated. Though it s difficult to predict these Where s precisely, we propose here the four Where s (fields) considering the various related situations; namely, On the network, In the digital broadcasting, In the library and In the real world, and define four types of multimedia database systems corresponding to them. (1) On the network: The internet data services including the WWW are now becoming more and more popular. Tremendous amount of severs provide multimedia information even in the form of video and images. This means that most databases hereafter will be scattered where contents are generated and such situations will grow uncontrollably. Such dispersed databases should be called here the Network Multimedia DB as one of our most promising targets as shown in figure 1. There are many what to do for this network multimedia DB. We believe, however, mediator functions, which inform users what kind of data are in the network, or where Figure 1. The network multimedia database environment.

3 FOUR PROMISING MULTIMEDIA DATABASES AND THEIR EMBODIMENTS 67 is their useful or desired information, and help to collect and relate them to the user s applications, are the most important. Such contexing of the network multimedia data should be the main function in the database. Their typical and first step trials examples are the Yahoo directory service or another search engines for text data. Various trials and functions, however, will be required for more general multimedia data in the network. As the embodiment of this network multimedia database system GIRLS(Global Image Retrieval and Linking Systems) for images on the WWW by authors group is introduced in 3. (2) In the digital broadcasting: Needless to say, one of the typical leaders of multimedia contents providers is broadcasting. Commercial-based satellite digital broadcasting with over 100 channels have already started or will soon start in USA, Japan, Europe and Asia. Another digital broadcasting in the form of CATV, ordinary TV, or Fibernet Communication also have already been or will be planed within less than ten years. In such situations, where we ll be able to enjoy hundreds or thousands of broadcasting channels, much more user-oriented and intelligent access to the tremendous amount of contents stream will be required as shown in figure 2. Let s call this situation as Stream-type Multimedia DB. In this stream-type Multimedia DB, the mediator functions would be also most important for users, which help them to pick up useful and required data from the stream and to make their own customized contents for new service and business. Figure 2. The stream multimedia database environment.

4 68 YAGINUMA ET AL. As the embodiments of these stream multimedia database systems, two concrete systems by authors research group will be introduced in 4 and 5. (3) In the Library: Such library-type databases as digital libraries, electronic catalogs et al., are of course important targets of the Multimedia DB in the narrow and ordinary sense. We call this type as Library Multimedia DB. This type of systems has been main targets for Multimedia DB. Many researches including flexible and powerful retrieval functions etc., have been done during two decades [1, 17]. (4) In the Real World: We Japanese, suffered sever earthquake damage at Kobe in January, This tragedy taught us importance of realtime acquisition of our city information for disaster mitigation. Multimedia communication technology enables us to establish a new type of databases which collect and analyze realtime situations (video information) to tell us What s going on in the city. Let s call this type as Real World Multimedia DB, as shown in figure 3. Though this type of systems have not been considered as multimedia databases, they have promising possibility to realize new services and businesses with real time integration of real world situations and existing another databases. Expecting fields in this category include disaster mitigation, intelligent transport systems (ITS), advanced environment management, advanced utilization of high resolution satellite Figure 3. The real world multimedia database environment.

5 FOUR PROMISING MULTIMEDIA DATABASES AND THEIR EMBODIMENTS 69 images etc. Author s group started, in 1994, the research project to realize so called Multimedia Geographical Information System by integrating ordinary map data and realtime city video data from robot cameras. Some details are reported in Ref. [7]. 3. Global image retrieval and linking system in the network MM environments In the network multimedia database systems shown in figure 1, contexing of the data space is the most important function as described in 2. GIRLS (the Global Image Retrieval and Linking System) for the image data in the WWW [13] has been developed as the embodiment of such database systems. In GIRLS, a search robot named BOYS automatically gathers typical images and layout information and URL from the WWW to construct the database for retrieval and linking based on the image contents as shown in figure 4. Though over 100 search robots are working for text data, a very few robots including BOYS are for images and video. The system provides users with open functions including retrieval of images or Logo s in the WWW data space, linkage to the original home pages with images imagined by users, feedback to the retrieval performance, or participating in the retrieval command creation etc. In the example of the GIRLS operation in figure 5, similarity retrieval results to the given layout home page are shown. GIRLS are now growing towards more friendly mediator functions in the network multimedia database environments. 4. Flexible multimedia platform GOLS for stream-type multimedia data environments The second concrete example is a flexible Multimedia Platform GOLS supporting various functions for the stream-type multimedia databases. GOLS [9, 10] provides many functions including database acquisition from images and videos, object-oriented database Figure 4. Open image retrieval system GIRLS.

6 70 YAGINUMA ET AL. Figure 5. The similarity retrieval results to the given layout home page. managements, and powerful database presentations with unique GUI. Acquisition of a desired set of video scene from live TV broadcasting programs (Live Hypermedia) is one typical example of GOLS functions. Figure 6 gives the concept of the live hypermedia in GOLS. The model in GOLS for recognition accepts a set of segments in a shot, detects the scene categories, and extracts objects in the shot of video. In the live hypermedia, information nodes and links between them are dynamically determined corresponding to the structure of the recognition models.

7 FOUR PROMISING MULTIMEDIA DATABASES AND THEIR EMBODIMENTS 71 Figure 6. The concept of the live hypermedia. A state-transition type expression [11] is proposed to describe flexible and compound models for recognition on live hypermedia. The system can detect not only TV program categories but also their contents. The model for recognition is implemented by Video Scene Description Language (VSDL) and the state-transition model. The VSDL realizes primitive description of shots. The statetransition model infers contents of shots using the primitive description. Object-oriented approach is adopted in the model construction process, in order to improve the modularity and reusability of the models. VSDL [1] was developed for automatic detection of TV shots. Users can describe TV scenes or objects they want to pick up or select by the language VSDL. VSDL predicates include description of video scene by features of color segments (color, area, position, etc.), relations to another segment, motion vector, and camera effects. A user describes what to pick up with VSDL. In real time, the GOLS checks for matches between this description and the given live TV images. When it found a match, it picks the corresponding video scenes up into the system. The state-transition model was originally proposed by our research group in order to realize flexible and reusable models for recognition of drawings [11]. In GOLS, this model has been improved and extended to adopt video data recognition. A primitive data (e.g., a segment for video scene generated by color image processing) is called as a token. Each token is given a state. The state is changed from one to another according to arrows between the states.

8 72 YAGINUMA ET AL. Figure 7. Example of database object manipulation using GOLS. Let s consider the human TV caster recognition in the shot for example. The state can be changed to seg skin, seg face,..., caster, etc., in accordance with bottomup and/or top-down recognition rules given for each state-transition link. In this manner, the structure of segments can be recognized by this type of models. The model consists of a set of the transition rules. Each rule describes a condition to change from one state to another. The condition consists of the VSDL predicates and Prolog predicates. The inference engine interprets the rules and updates the state of each token. Interpretation strategy is a combination of bottom-up and top-down analysis. At first, bottom-up rules are Table 1. The retrieval accuracy. Strict Coarse Model Recall Precision Recall Precision News Sumo Golf

9 FOUR PROMISING MULTIMEDIA DATABASES AND THEIR EMBODIMENTS 73 Figure 8. Results of the content-based retrieval. invoked and token states are updated until no more state can be changed. Then top-down rules are invoked to further more updating. Merit of this model expression lies in its modularity: rule sets can be easily replaced according to a recognition target. In fact, over 80% of the experimental models for commercial TV broadcastings are found reasonable. Figure 7 presents an example GOLS operation. GOLS picks up the desired newscaster video scene from the live TV broadcast programs shown in the lower right window. Then it recognizes the contents of the scene and generated video parts, which are shown in the upper left window. A GOLS user can utilize this video parts to design the new card type hyper-media shown in the upper right window. To demonstrate the effectiveness of the GOLS or Live Hypermedia, experimental evaluation of the Recall rate and the Precision rate (video scene pick-up performance) for commercial TV broadcasting programs using the GOLS models has been done successfully, whose results are shown in Table 1.

10 74 YAGINUMA ET AL. Figure 9. Example of content-based editing. 5. A higher-level authoring system for stream-type MM environments The third example is also related to the stream-type MM database. In order to realize more flexible and user-oriented access and utilization of stream-type multimedia DB s, advanced methods of making higher data models for video data in the DB are required. Many trials using image or video recognition techniques have been done for this targets. They have not necessarily, however, been successful because of difficulty of image understanding. A new approach to overcome this problem using multiple media synchronization proposed by author s research group, is introduced here. Many video data have corresponding video, sounds and relevant documents. TV drama programs, for example, have related video sounds and scenario documents. Our idea is to realize logical synchronization between these related multiple media using DP matching techniques. Several numbers of related patterns are first extracted from each medium, and then optimal matching for these patterns is carried out using modified DP matching technique [14, 15]. The scene change pattern in the video, voice or female voice patterns and word arrangement pattern in the scenario document are selected for this related patterns.

11 FOUR PROMISING MULTIMEDIA DATABASES AND THEIR EMBODIMENTS 75 Once, these logical synchronization has been carried out, document data, which have much higher symbolic and abstracted representation than video data, can be used as a kind of structure or content navigator for video. For example, intelligent access to video scenes including the specified person (character) can be successfully realized with the Recall rate of about 90 97% for typical TV dramas. Figure 8 shows the retrieval results of a specified person who is laughing. The user interface of the system is written using HTML. In this version, the retrieval is realized not only by the name of the person but also by the action of the person. Figure 9 shows another example of user-oriented content-based editing for TV drama video. A user can easily realize his/her own video authoring just using much easier wordprocessor editing functions, because sentences in the scenario documents have now tight timing synchronization with video stream. This user-oriented access and authoring system for video is now growing toward more flexible and effective video database mediator. 6. Conclusions Four types of promising multimedia databases are introduced and importance of the mediator functions in these databases have been identified as important research directions. Three concrete multimedia database systems with unique mediator functions, the image data mediator GIRLS on the WWW, the stream-type multimedia database platform GOLS and the intelligent access and authoring system using multiple media synchronization have been introduced with experimental evaluation results and concrete multimedia database applications. We expect various trials will be done toward this multimedia DB mediators. References 1. Yiong Gong and Masao Sakauchi, A method for color image classification using the color and motion features of moving images, Proceedings of Second International Conference on Automation Robotics and Computer Vision (ICARV) 92, N. Nakamura, S. Abe, Y. Ohsawa, and M. Sakauchi, The MD-tree: An efficient data management structure for spatial objects, IEEE Trans. on Knowledge and Data Engineering, Vol. 5, No. 4, pp , M. Sakauchi, Image retrieval techniques, The Journals of the IEICE, Vol. 71, No. 9, M. Sakauchi, Image and multimedia data-base, The Journal of the Institute of Television Engineers of Japan, Vol. 46, No. 11, pp , M. Sakauchi, Image retrieval and image understanding, IEEE Multi-media Computing Magazine, Vol. 1, No. 1, pp , M. Sakauchi and S. Satoh, Data model generation in image database system, Transaction of IEICE of Japan, Vol. J74-D-I, No. 8, pp , M. Sakauchi, Y. Ohsawa, and T. Sagara, Construction and application of multimedia geographical database, Journal of Institute of Geographical Information Systems of Japan, Vol. 3, No. 2, pp , M. Sakauchi, T. Satou, and Y. Yaginuma, Multimedia database systems for the contents mediator, Transaction of IEICE, Vol. E79-D, No. 6, pp , T. Satou and M. Sakauchi, Data acquisition in live hyper media, Proc. of IEEE International Conference on Multimedia Computing and Systems, pp , T. Satou and M. Sakauchi, Video information acquisition on live hypermedia, Transaction of IEICE, Vol. J79- D-II, No. 4, pp , 1996.

12 76 YAGINUMA ET AL. 11. S. Satoh, Y. Ohsawa, and M. Sakauchi, A proposal of drawing image understanding system applicable to various target drawings, Transactions of Information Processing Society of Japan, Vol. 33, No. 9, pp , S. Satoh, H. Moh, and M. Sakauchi, A drawing image understanding system cooperating with rule generation supporting using man-machine interaction, IEICE Transaction on Information and Systems, Vol. E77-D, No. 7, H. Takaha, T. Yatabe, T. Satou, and M. Sakauchi, Open image retrieval system GIRLS on WWW, Technical Report of IEICE, Vol. IE95-11, pp. 1 8, Y. Yaginuma and M. Sakauchi, A proposal of a video editing method using synchronized scenario document, Transaction of IEICE, Vol. J79-D-II, No. 4, pp , Y. Yaginuma and M. Sakauchi, A proposal of the synchronization method between drama image, Sound and Scenario Document Using DP Matching, Transaction of IEICE, Vol. J79-D-II, No. 5, pp , Y. Yaginuma, M. Kageyama, and M. Sakauchi, Synchronization between image, sound and scenario document using DP matching, Proc. of 1994 IAPA Workshop on Machine Vision Applications, pp , J. Yamane and M. Sakauchi, A construction of a new image database system which realizes fully automated image keyword extraction, IEICE Transactions on Information and Systems, Vol. E76-D, No. 10, pp , Yoshitomo Yaginuma received the B.Eng. degree in Phys. Instr. in 1988, and the M.Sci. degree in Physics in 1990, both from the University of Tokyo. He is now a research associate, Institute of Industrial Science, University of Tokyo. He is a member of Information Processing Society of Japan. His research interests include data structure, multimedia database and database acquisition. Tomoyuki Yatabe received the B.Eng. degree in information and communication engineering from the University of Tokyo in He is now a master course student at the Department of Information and Communication Engineering, the University of Tokyo. His research interests include hypermedia systems, data structure and multimedia database.

13 FOUR PROMISING MULTIMEDIA DATABASES AND THEIR EMBODIMENTS 77 Takashi Satou received the B.Eng. degree in electronics engineering in 1991, and the M.Eng. degree in information engineering in 1993, both from the University of Tokyo. He is currently a graduate student in the Department of Information Engineering at the University of Tokyo. He is a fellow of the Japan Society for the Promotion of Science for Japanese Junior Scientists His research interests include hypermedia systems and visual information systems. Junichi Tatemura received the M.S. degree on Information Engineering in Faculty of Engineering, the University of Tokyo in 1991, and the Ph.D. degree on Information Engineering in Faculty of Engineering, the University of Tokyo in He is now a Lecturer in Institute of Industrial Science, University of Tokyo. He is a member of IEEE/CS, ACM, IEICE, and Information Processing Society of Japan. His research interests include information visualization, user interfaces, interactive systems, and parallel processing. Masao Sakauchi received the B.Sci. degree in Electrical Engineering from the Univ. of Tokyo in 1969 and the M.S. and Ph.D. degrees in Electronics Engineering from the Univ. of Tokyo in 1971 and 1975, respectively. He is now a Professor and Director, Center for Conceptual Processing of Multimedia Information, Institute of Industrial Science, Univ. of Tokyo since Dr. Sakauchi is the executive committee member of IEEE computer society s Multimedia computing committee, Program chairman of the 3rd IEEE International Conference on Multimedia Computing and Systems, and Associated Editor in Chief of IEEE Multimedia Magazine. He acted as the general chairman of 5 IEEE or IAPR International conferences and workshops, and also the chairman of the technical committee on Image Engineering, and information group s vice-president of IEICE. He has over 150 refereed papers in the research fields of multimedia databases, multimedia systems, image processing and understanding, spatial data structures, and geographical information systems and fault tolerant computing.