Index Generation and Advanced Search Functions for. Muitimedia Presentation Material. Yahiko Kambayashi 3 Kaoru Katayama 3 Yasuhiro Kamiya 3

Transcription

1 Index Generation and Advanced Search Functions for Muitimedia Presentation Material Yahiko Kambayashi 3 Kaoru Katayama 3 Yasuhiro Kamiya 3 Osami Kagawa 33 3 Department of Information Science, Kyoto University Sakyo Kyoto , Japan f yahiko,kaoru,kamiya g@isse.kuis.kyoto-u.ac.jp 33 Department of Computer Science, Hiroshima Denki Institute of Technology kagawa@isse.kuis.kyoto-u.ac.jp Abstract In this paper we will discuss advanced search functions for multimedia data produced by presentation. Such multimedia data consist of video of the speaker and slides used for explanation. In conventional systems, during playback these two components are synchronized and the presentation is played like video tape recorder. Basic functions are play, fast-play, skip, rewind and go-to-the previous-slide. Since the slide part contains characters showing the contents of the explanation, we can use these character strings to nd out proper portions to playback. In our system, two more components are considered, movement of pointer during the explanation, and additional lines for correction, underlines etc. produced during the presentation. These information is also used to nd proper portion in the presentation data. 1 Introduction One of the serious problem for using multimedia data is how to nd appropriate portion to play. As scene analysis technology cannot be used for reliable index generation and it is time-consuming to generate index terms for each portion of multimedia data manually, one possible way is to utilize sentences describing the contents such as subtitles of video. If such information is available we do not need to prepare index. We can use partial matching facilities used for bibliographic information retrieval. We developed a exible retrieval system for such kind of data. Such captions are prepared for TV news or movies. When we prepare multimedia materials it is rather dicult to prepare such sentences and correspondences between portions of video and such sentences. In this paper, we discuss our system to generate hypermedia material during presentation and the system with exible retrieval facilities. Fig. 1 shows the concept of hypermedia generation during presentation. The speaker need to prepare slides in advance and give presentation using the slides. During the presentation correspondence between a portion of slide called a block and a portion of video corresponding to the explanation is generated. As each block is linked to the corresponding portion of video, we can use slide part as index for video part. The speaker can stop video recording or instruct to re-record during presentation in order to avoid recording unnecessary contents. Functions to retrieve arbitrary part of video is mainly realized by keyword search for slide part. We can also specify slides by "slides having pictures", "slides with long explanation", or "slides when the speaker talked on something else (shown by coee symbol". 2 A Generation of Multimedia Data during Presentation In order to discuss search functions, we need to discuss how data are generated during the presen-

2 SLIDES Movement during presentation Hypermedia Figure 1: Generation of Hypermedia Combining Slides and Its Usage Figure 2: User Interface for a Speaker tation. Fig. 2 shows typical display interface for a speaker. The speaker prepares slides in advance. During the presentation, one slide and video image of the speaker are shown. Each audience sees images shown in Fig.3 together with the voice of the speaker. We need the following functions for better presentation. 1. Pointer : It shows the place of explanation. 2. Pen : During the presentation, usually a speaker wants to put emphasis for some part of sentences by adding underlines, or to add some sentences (even simple gures) for a better explanation. Eraser is also available. In our system we assume that the data taken during the presentation are stored and used forplayback for people who could not attend the presentation. To store various data under synchronization constraints is rather expensive, and a reasonable solution for realization is to store video images and links to each slide. Video images are divided and each portion corresponds explanation of each slide. During playback, slides are replaced according to the video of the presentation. We can skip explanation by selecting a proper slide and we can listen the corresponding explanation. If we want to show movement of the pointer and pen together, we need to synchronize all data. We can, of course, use video for the slide part, then all the movement of the pointer and the pen can be recorded. In MPEG data compression mechanisms, identical parts of the consecutive images are stored only once ideally. We can use MPEG data compression mechanisms for good reduction of data size, since in all ames the background slide part is identical. Only movement of the pointer and pen should be stored. The method

3 Figure 3: User Interface for an Audience is rather straightforward and easy to implement, but still the amount of data is too much to be transmitted through internet. For further reduction of data, we will use hybrid approach as follows. 1. Movement of mouse : Instead of recording all the movement of the mouse, we record locations of special points and time required for moving the previous special point to the current point. Mouse movement is generated by these data. As special points, we select the locations where mouse stayed for some predetermined time, or change of mouse direction more than predetermined angle. 2. Movement of the pen : It is not recorded. Although the movement of the pen is similar to the movement of the pointer, we need to store a lot of locations to playback the movement of the pen since resolution required for pen is much ner than mouse. Instead we store all drawing results of the pen for each slide and the slide contents and the drawing results are shown together. When a new slide is shown, we will see the result of the modication by the pen. Since the pointer shows the location of current explanation, the precise place is not required. For the pen, on the other hand, precise location is required otherwise the resulting drawing will be too much simplied. Sometimes when the drawing process is important. In such cases, we can use the same procedure for the mouse movement for the pen, since the drawing result is already overlaid. By these method the most data consuming part is the video of the speaker. By using MPEG mechanisms and use small number of pixels determined by the network capacity, we can transmit the presentation through networks. The reduction of data also contribute to store the whole presentation in a reasonable amount of data. In order to realize powerful retrieval functions, in our system it is assumed that each slide is divided into blocks. The separation codes for the blocks are embedded in the slide. In Fig. 2, the white portion shows the current block. During

4 the presentation the speaker selects the symbol for block transition, then the next block will be shown. By this way, we can nd the correspondence between the blocks and the portions of the video. There are three tools shown in the rectangle shown in the right middle part of Fig.2. These are, pointer, pen and eraser, respectively. The left symbol of Block shows block transition. If it is selected when the current block is the last block of the current slide, then the rst block of the slide is selected. The right symbol shows rerecording of the same block. If a speaker wants to record explanation of the current block again, he will select this button. The speaker sometimes likes to talk about something which is not strongly related the contents of the slide. In such cases, the speaker selects the coee symbol, to show the current explanation is not related to any block of the slide. At the end of such talk, the speaker selects the coffee symbol again to return to the normal mode. In the slide for audience, coee symbol will be shown as seen in Fig. 3. During the presentation the speaker like to select one of the other slides. It can be realized by the following three methods. 1. Selection of a proper symbol under the slide. By selecting Prev button, we can go back previous slides. By selecting Next button, slides which are not explained yet can be selected. As a result the identical slide may be used more than twice. We duplicate the slide so that the correspondence can be independently stored. 2. Selecting a proper slide by the sequence of the miniature slides. It is shown as the slide list on the back of the current slide and the current slide is high-lighted (in Fig. 2 it is surrounded by black lines). 3. Using selection functions to be discussed in the next section. During the presentation, the speaker can select proper slides by specifying keywords etc. The speaker remembers all the slides used for the current presentation, he may use functions 1 or 2, thus function 3 is required to nd proper slides from other presentation. 3 Advanced Search Functions In conventional systems, video (speaker's face and voice) and slides are synchronized for playback. This is exactly like a video equipment. If the user wants to see the presentation it is enough. There are requirements for more powerful functions to retrieve a proper slide or block. The user may want to see some previous slides to understand the material. By stopping or continuing the current explanation by video, the user can see some other slide. The control button for this purpose are shown under the slide in Fig. 3. The current button in the middle shows when the user wants to go back the slide corresponding to the current video position. The video can be stopped temporarily since the user will be back eventually by using the current button. In other case the audience may want to listen the explanation of the corresponding video for the selected slide. It is realized by selecting interlock button under video. Search by the slide list is similar to the case for the speaker. The only dierence is that the identical slide may appear more than once, if it is used more than once during the presentation. Each of identical slides corresponds to dierent explanation video. Search functions are classied as follows. 1. By video: By using the fast forward button of the video, the audience can nd a proper location. 2. By slide: By nding a proper slide, the corresponding video explanation will be played. 3. By block, by sentence: Instead of nding a proper slide by the contents of slide, we can use block as a unit of search. we can also specify a sentence as a unit of search. Possible video search mechanisms are as follows. 1. Find by the scene. By using fast-forward button or rewind button, we may be able to nd

5 the required scene. 2. Find by voice. For example, we can nd video locations where quiet sound continues over 3 seconds. 3. Find by scene switch. Each time when transfer to a new block is made, scene switch symbol is put to the video. Furthermore, if a camera switches to show other materials beside the speaker, scene switch symbol is automatically put. We have various functions for searching by slides. 1. Find by the contents of the slide. By selecting Prev or Next button, proper slide can be selected. 2. Select by the slide list. By looking at miniature versions of slides, a proper slide is selected. 3. Select by partial matching retrieval. By specifying search conditions to the SEARCH window, a set of slides satisfying the condition are selected. The results are shown as highlighted slides in the slide list. Final selection is made by checking all these candidates. The following three operations are prepared for partial match retrieval. AND : AND operations of keywords. OR : OR operation for keywords. * : Don't care strings. It corresponds to arbitrary character strings. For example,\distributed AND DATA- BASE" means \Find slide containing both DISTRIBUTED and DATABASE". \DISTRI- BUTED OR DATABASE" means \Find slide containing DISTRIBUTED or DATABASE". \DISTRIBUTED * DATABASE" means \Find slide containing DISTRIBUTED and DATA- BASE in this order". We can further specify the domain of the search, sentence, block and slide. If sentence is specied, both DISTRIBUTE and DATABASE should be included in one sentence. Using ( and ), we can express complicated conditions. If one of the slide satisfying the condition is selected, the words (in this case, DISTRIBUTED and DATABASE) satisfying the condition are highlighted. Once a condition is applied a set of slides are selected and these are high-lighted in the slide list. If the number of slides satisfying the condition is too many, we can apply another search condition to this selected set. This function is useful when the system is used for education and there are a lot of teaching materials. We can further combine other conditions. Coee symbol can be used for search condition. We can nd slides which has talk shown by coee symbols. Furthermore, slides with pictures can be selected. 4 Concluding Remarks Indexing multimedia documents is expensive but it is required for exible retrieval of contents. In this paper, a method for indexing presentation video using the slides is given and powerful mechanism for play back is shown. We believe it is a practical solution to generate index for video almost automatically. We believe a similar approach can be used for news video with subtitles. References [1] M.Taini D. Ponta, A. Scapolla. Telematics for education. In ED-TELCOM Conf., pages 252{257, [2] J.Tang E.Isaacs. What video can and cannot do for collaboration, a case study. In Multimedia Systems, volume 2, pages 63{73, [3] O. Kagawa S. Konomi and Y. Kambayashi. View media : A multiuser hypermedia system for interactive distance presentation. In CSCW'94, Workshop on Collaborative Hypermedia Systems, Oct [4] O.Kagawa W.Winiwarter and Y. Kambayashi. Applying language engineering techniques to question support facilities in viena classroom. In Lecture Notes in Computer Science, volume 1134, pages 613{622, Sept 1996.