N1 N2 N3 The narratives and slides are specified as a parallel-last

Transcription

1 Building A Framework for FLexible Interactive Presentations James A. Schnepf Computer Science Department College of St. Benedict/St. John's University Collegeville, Minnesota, USA Yen-Jen Lee, David H.C. Du, Lan Lai y Distributed Multimedia Research Center z Department of Computer Science University of Minnesota Minneapolis, Minnesota, USA Liang-Wei Kang x Computer & Communication Research Laboratories Industrial Technology Research Institute Hsinchu, Taiwan, ROC Abstract As presentation technology advances, it is possible to incorporate a wider range of media including variable duration media such as simulations and animations. At the same time, users are able to take more control over presentations by controlling the rate and selection of media being played. To make full use of these advances, multimedia systems must support exible presentations that incorporate many variations in the way they are played. As a result of content-based searches and other methods of selections such as a selection from an index of slides, viewers may skip to dierent parts of a presentation or start viewing a presentation in the middle. These presentations must remain semantically coherent in the face of these interactions. This paper presents the work we have done to design and implement a framework to support the inclusion of diverse media displayers into a presentation and to support user interaction. The implementation maintains a consistent and coherent presentation in the face of user interactions such as skipping forward and backward to dierent points within the presentation or modifying the speed of a single media segment. The implementation is modular and allows easy inclusion of new media types and displayers. The framework allows the displayer to control the navigation within an object (pause, fast-forward, reverse) while synchronization among objects is handled by the framework. The implementation also provides an easy-to-use authoring tool to compose a presentation. Keywords: FLIPS, multimedia, synchronization, authoring, content-based search This work was conducted while the author was a PhD candidate in Department of Computer Science, University of Minnesota, Minneapolis, Minnesota, USA y Authors' addresses are: schnepf@csbsju.edu, fylee,du,kang,laig@cs.umn.edu z Support provided by US WEST, Honeywell, IVI Publishing, Computing Devices International, and Network Systems x This work was conducted while the author was an MS student in Department of Computer Science, University of Minnesota, Minneapolis, Minnesota, USA 1

2 1 Introduction The fusion of audio, video, text, images, and animation into multimedia presentations provides an opportunity to create more eective and ecient communications of ideas. As multimedia technology advances, it is possible to incorporate a wider range of media into presentations including variable duration media such as simulations and animations. The use of such presentations in an academic environment is particularly promising as instructors transform educational presentations into a process where learners control the presentations to best suit their learning style. Course presentations can be captured and stored to allow students access to the university according to their own schedules and at locations of their choosing [1]. Viewers control the pace of their learning by pausing, skipping forward and backward in the presentation, and interacting with dynamic media displayers 1 that incorporate viewer interaction into the presentation. Hypermedia objects, for example, allow a user to explore a presentation segment in greater or lesser detail. Inclusion of these dierent media implies that objects in a presentation will have unknown or variable durations. We have designed and implemented a framework that supports these exible interactive presentations. A critical component of these presentations is the temporal relations between multiple media that describe the relative timing of their display. This information can be dened in advance by an author of a multimedia document specifying the temporal relations that dene the relative timing of the dierent components. Alternatively, this information may be captured during a presentation (and possibly modied post-capture). As the session progresses, all of the material presented including slides, videos and simulations are captured with relative timing information for later review. The inclusion of user interactions and variable duration objects implies that there is not a single presentation available, but rather, a range of playbacks of a presentation that are subject to constraints that maintain the integrity of the message. Viewer interactions such as setting simulation parameters or adjusting the speed of a particular display can aect the duration of some objects within a presentation but not others. Viewers may also jump to any spot in the presentation such as making selections from an index of slides or nding a point in a video clip based on a content-based search. These interactions may aect the display of some associated media objects while not aecting others ( such as background music) based upon the temporal constraints dened for the presentation. To support these exible presentations, several issues must be resolved in the presentation environment: The user interface must allow the viewer to skip to any object within the presentation. Flexible presentations require that the user be in control of the presentation and be able to freely move to dierent segments of the presentation without being limited to a linear start to nish presentation. User interactions with a single media object must not disrupt the display of other objects unnecessarily. Simulations, rate of play variations and other factors can result in variable durations of objects. Such variations should be allowed without disrupting or changing the display of other objects unnecessarily. 1 We dene a displayer to be any program that can generate an output in the presentation and media objects as the information required for that application program, e.g., video frames, audio samples or parameters for a simulation program 2

3 The system must determine and maintain a valid global state that is consistent with the presentation in the face of variable duration objects and user interactions. When a presentation changes state, the presentation manager must identify which objects to modify to maintain a valid global state that is consistent with the specication. Equally important, is to not modify objects unnecessarily in order to promote continuity in the playout of a presentation. The inclusion of dierent types of displayers should be supported with minimal eort whenever it is possible. Media display capabilities are changing dynamically and a monolithic display package cannot easily take advantage of new media types and capabilities. Our implementation provides the ability to view and interact with a exible presentation that can incorporate standard media displayers as well as many nonstandard displayers such as simulations and visualizations. Our framework supports an event-based model, FLexible Interactive Presentation Synchronization (FLIPS) [4], that can handle the synchronization of variable duration objects and a wider range of specications than the typical binary equality relation of many models. This model, together with the enforcement mechanism, provides a richer presentation environment that supports user interaction in a multimedia presentation. The enforcement mechanism maintains a consistent and coherent playout of a presentation within the constraints specied. Jumps to dierent objects in the presentation and interactions with a single presentation segment aecting the duration of that object are supported without unnecessarily aecting the display of media that are loosely synchronized. A viewer can at any time select an object to view, and the presentation coordinator determines a valid global state that is consistent with the constraint specications. The presentation coordinator module identies which objects require state modications and which objects should not be modied to allow continuity in the presentation. Fine-grain synchronization (e.g. lip-synching) is not handled directly, but is integrated by incorporating composite objects representing ne grain synchronization along with an appropriate displayer such as the Berkeley CM Player [5]. The organization of the paper is as follows. Section 2 relates our design to other work. In section 3, we provide an example of a exible presentation and the basic constructs used in the FLIPS model. Section 4 describes our design of a presentation system based on the FLIPS model. Section 5 shows the platform-dependent implementation on both UNIX and Microsoft Windows environments, and compares the relative merit for the implementation on both environments. Section 6 demonstrates an authoring tool to compose a FLIPS presentation using a user-friendly graphical user interface. Section 7 summarizes the results and describes the direction we are proceeding. 2 Comparison with Related Work The specication, modeling and enforcement of temporal synchronization has been the subject of much research. Much of this research has addressed static presentations with xed objects of known duration [6, 7, 5, 8, 9, 10]. In many schemes, if two sets of objects are specied to be displayed in parallel, their durations not only must be known, but must be exactly equal [11, 6, 7, 10]. Li, et al. [12] suggest that this expectation is unreasonable since the duration information may not be known at the time of specication. They propose a model that handles the specication of media with unknown duration, but it is assumed that the durations are available prior to the presentation time. (i.e., when running their scheduling algorithm prior to presentation). Implementations for coarse grain synchronization have been built by several research groups including the MODE project [13], Firey [14], CMIFed [15], Maestro [16], Xavier [17], and Eventor [18]. [13, 15, 17, 18] all provide media displayers built into the presentation system. While this 3

4 supports ecient implementations, it does not allow for quick and easy incorporation of new and varied media types. As in the models discussed above, the support of user interaction and skips presume that all concurrent media are aected by a jump. Drapeau in [16] allows for varied media displayers to be incorporated, but these applications must be modied to work with the Maestro system. The work of Buchanan and Zellweger[14] is the closest to ours. They also handle variable duration objects and asynchronous events such as user interactions. Their scheme diers from ours in their mapping of events to a timeline. It is not clear whether they handle jumps to previously viewed objects. 3 Flexible Presentations and the FLIPS Model An example illustrates the exibility that can be part of the playout of a presentation. 2 An architecture presentation describes the transition of European architecture from Gothic to Renaissance. The presentation starts with a video clip of churches of dierent architectures. At the end of the video clip, an audio narrative begins which is divided into three parts. The rst part describes the distinguishing characteristics of Gothic architecture. The second part describes the transition from Gothic to Renaissance and the nal narrative summarizes the key points that characterize Renaissance architecture. Within each of these narratives, buttons appear on the screen to allow viewers to select more expanded explanations of key points. For each narrative, there is a set of slides that is displayed while the narrative is played. As the rst narrative starts, the rst set of slides also start, showing representative examples of Gothic architecture. The second set of slides show the transition from Gothic to Renaissance. The third set of slides highlights the characteristics of Renaissance architecture. These slides are not precisely matched to the audio track but instead are loosely synchronized so that the rst set matches to the rst audio narrative, the second set to the second audio narrative and the third set to the third audio narrative. Each slide has a default duration after which the next slide is shown, but the user has control to hold a slide, skip ahead to the next slide, or skip backwards. To reinforce the correspondence of the slides to an architectural style, a dierent style of background music plays for each of the two styles of architecture. A Gregorian chant piece is played while Gothic slides are displayed. A Bach selection is played while Renaissance architecture slides are displayed. The music plays continuously as long as slides of a particular genre are displayed. Viewer accessing this presentation are provided a presentation user interface that allows them to select a particular slide to view. If the slide corresponds to the same narrative (or background music) currently being played, no change is made to the concurrent media. If the slide corresponds to a dierent narrative, the appropriate narrative (or background music) is started. This diers from the timeline model such as is used in the CM Player [5], where any shift in the presentation aects all the corresponding objects. In addition, each displayer could have its own user interface that allowed the viewer to control and interact with the individual object. The FLIPS model supports the specication of exible presentations. The FLIPS model contains four major components: media objects, events, barriers, and enablers. The key events that need to be synchronized in a multimedia presentation are the beginning and ending of the display of objects. Barriers and enablers specify the coarse-grain synchronization among the set of media objects and events. If event A is specied as a barrier to event B, event B is delayed from occurring until event A occurs. If event A is specied as an enabler to event B, the occurrence of event A 2 It is important to recognize that this example includes both a set of media objects and a set of synchronization specications. The same media objects could be used to dene a dierent presentation by specifying dierent synchronizations. 4

5 results in event B occurring as soon as all of its barrier relations are satised. In general, an event occurs when at least one of its enable relations is satised and all of its barrier relations are satised (it is barrier-free). The end of an object is enabled when it either completes its display or an event external to the object occurs that enables the end of the object. In this way, the relative timing of events can be established by dening barrier and enabler links between the dierent events in the presentation. The entire specication is in eect a network of media objects connected by enabler and barrier links to the begin and end events of the objects. A detailed description is beyond the scope of this paper and can be found in [4]. A possible specication for the architecture example is shown in Figure 1. Enablers are represented as A! B and barriers are shown as A a B. Start G1 N1 N2 N3 The narratives and slides are specified as a parallel-last G2 G3 G4 G5 R1 R2 R3 R4 R5 The slides and background music have a "master-slave" relationship, the slides dominate Gregorian chant Bach Narrative End Slides Background Music Figure 1: Specication for the architectural presentation 4 Systems Design The system design has a modular architecture built around a few simple data structures. As shown in Figure 3, the front end processor reads the presentation specication le and sets up the internal data structures The specication is turned into a graph of media objects (connected by enabler and barrier relations) and a corresponding user interface that displays the objects available for selection. The presentation user interface allows the user to control the display point in the presentation and communicates with the presentation coordinator to implement user actions and to provide feedback to the user. The presentation coordinator manages these objects and is responsible for the overall presentation. The media displayers display media, and present media-specic user interfaces (e.g., a volume control for audio). The front end processor translates the content of the presentation specication script and instantiates the status information for media objects in a graph data structure. The graph nodes consist of static information such as the media type, displayer, and a pointer to the data, and dynamic information such as the state of the object and the link status for each graph edge. There are two types of edges between nodes, representing the barriers and enablers that form the relationships between objects. The presentation user interface does not provide user interaction within an object. Instead, that control is deferred to the media displayers. This modular implementation allows the inclusion of dierent media displayers in dierent presentations without aecting the presentation user interface. It also allows the presentation user interface to be easily replaced by another that can provide more semantic meaning to the selections. The heart of the system is the presentation coordinator. The presentation coordinator maintains the status of each object in the presentation. It maintains a valid global state in response to each event by propagating changes in the network of objects until a consistent state is reached (the algorithm is described in [4]). The presentation coordinator is event driven. When an event occurs, the presentation coordinator determines a valid global state and sends messages to the appropriate 5

6 Figure 2: Typical presentation screen layout media displayers to achieve that state. The global synchronization provided by the presentation coordinator is orthogonal to the ne-grain synchronization and internal control mechanisms provided by the media displayers. The media displayers are a set of programs that can display media to the user. Typical media displayers include applications such as an audio player, an MPEG video player, an image displayer, a text viewer, etc. However the modular nature of the design allows for any medium for which a displayer can be dened. These displayers can be external, as implemented in the UNIX environment, or integrated into the application as a set of class objects, as implemented in the Windows environment. The message protocol between the presentation coordinator, the media displayers, and the presentation user interface consists of four simple messages that are sucient to play a presentation: START, STOP, DONE, and JUMP. START and STOP are messages from the coordinator to the displayers to tell the displayers to start or stop displaying a particular object. DONE is a message from a displayer to the coordinator indicating that a particular object's play-out has completed. JUMP is a message from the user interface to the coordinator that the user requests a specic object be played. The user interface can also monitor START, STOP, and JUMP messages to display status information to the user. The presentation specication is a text le that maps objects to locations, denes the media 6

7 Specification File Front end Processor Object Graph Audio Wrapper Audio Displayer 3 Presentation User Interface 1,2 Presentation Coordinator 1 2 Video Wrapper Video Displayer Data Flow Process generation 1 {Start, Stop} 2 Done 3 Jump Image Wrapper Image Displayer Figure 3: The FLIPS systems design type of each object, and denes the temporal relationships between objects using the enablers and barriers of the FLIPS model. It contains all of the information needed to set up a presentation. Users compose a FLIPS presentation by editing the text le of presentation specication as shown in Figure 4 or by using the authoring tool described in section 6. The authoring tool enables viewer to graphically edit a FLIPS presentation and dene enablers and barriers by using a pointer devices such as a mouse. Further development will incorporate the authoring tool with specication verication module to dynamically check for the inconsistencies and potential deadlocks in a presentation. 5 Implementation The FLIPS model has two platform-dependent implementations based on the system design in previous section. One runs in the UNIX environment and the other runs under Microsoft Windows. The script nature of the FLIPS presentation specication is transparent across platforms. As long as appropriate media displayers are available, a FLIPS presentation can be orchestrated on either UNIX or Windows without modication. The two implementations take advantages of the individual platform's capabilities but do not necessarily represent the only approaches for the implementation. 5.1 FLIPS in the UNIX Environment The UNIX implementation is designed to support the integration of any external displayer without modication. Each module of the applications as well as each display process is an independent process. This architecture makes it easy to call any display application through the use of media displayer wrappers which are described later. Communication between modules is handled through the use of BSD sockets. This mechanism allows the displayers for an application to reside on dierent machines and to display on multiple X windows displays. In a large presentation, for example, the presentation user interface may be on one display, a video may occupy an entire second display, and a set of images may occupy a third. The presentation user interface, the media displayers, and the presentation coordinator, are the most critical components in FLIPS presentation. In the following subsections, we describe these components in further details. 7

8 Figure 4: Display of FLIPS script Presentation User Interface The Presentation User Interface is a graphical user interface where viewers start, stop, or jump to dierent objects within the presentation. It is implemented using tcl/tk. When a presentation is selected, each object is mapped to a button. Buttons of same media type are aligned. A start button precedes the entire presentation and an end button is appended to the end of the presentation. A viewer can jump to any object in the entire presentation by pressing a button. When a button is pressed, the object selection is sent to the presentation coordinator. The Presentation Coordinator propagates back to the presentation user interface the status of objects and associated buttons will be highlighted by distinct colors to reect their status Media Display To allow the incorporation of external media displayers, we have dened media displayer wrappers that are a set of proxy agents. They provide a consistent communication interface to the presentation coordinator and control the external displayers. The wrappers receive START and STOP messages from the presentation coordinator, which they convert into appropriate messages to control displayers, and they send DONE messages to the presentation coordinator when the displayers have nished displaying an object. The wrappers handle both the mapping between media objects and les and the mapping of command sets. 8

9 The inclusion of media displayer wrappers allows the FLIPS implementation to operate on any medium for which a displayer can be dened, as long as the displayer can indicate when it is done displaying an object. The lowest-level wrapper, implemented on a UNIX platform, is a process that spawns a new displayer whenever it receives a START message, and then waits for either the termination of that displayer process or another message. If the process terminates, it sends a DONE message to the coordinator. If the wrapper receives a STOP message, it kills the displayer process and waits for further messages. This architecture has been used to support a wide range of media players including audio, image, and video media. The modular plug-and-play capability provides exibility and ease of inclusion of new media types and displayers. The price is the startup overhead of applications. When the start-up latency of a media displayer is too slow, the media displayer wrapper can be congured to send specic messages to the displayer to instruct it to stop and start playing (with a le option). We implemented an MPEG player with this extension to speed up transitions between MPEG objects. The open architecture and simple protocol allow easy extensibility. As important, the media displayers still retain their own user interfaces and underlying display mechanisms. Accordingly, the MPEG player still allows navigation within an object (pause, fast-forward, reverse) while it defers to the presentation coordinator for synchronization among objects. The use of media displayers is totally transparent to the viewer. Dierent media displayers can be adapted and optimized for specic media formats Presentation Coordinator The presentation coordinator uses BSD sockets to set up a listening port that waits for messages from either the user interface or any of the display processes. When a change occurs in one of the displayers or the user interface, the appropriate module sends a message (DONE or JUMP) to the coordinator notifying it of that change. When a message arrives, the coordinator navigates the presentation graph, updating each node according to the synchronization algorithm [4]. When a valid global state is achieved, it generates and sends STOP and START messages to the appropriate media displayers to achieve that state. This can be better understood by viewing an example. In Figure 5, Slide R2 is holding on display until the completion of N2 while Bach music is playing in the background. When N2 nishes playing, the narrative wrapper sends a DONE message to the presentation coordinator. The presentation coordinator determines that both the narrative and the slide have reached their synchronization point and determines that the valid global state to change to will have N3 and R3 display and will cause no change to the Bach music. It then sends the appropriate STOP and START messages to continue the presentation as detailed in Table 1. N1 N2 N3 Narrative Start G1 G2 G3 G4 G5 R1 R2 R3 R4 R5 End Slides Gregorian chant Bach Background Music Presentation status prior to completion of N2 Presentation status after completion of N2 Figure 5: The change in the objects displayed as a result of the completion of N2 9

10 Sending Module Message Receiving Module Narrative Wrapper DONE, N2 Presentation Coordinator Presentation Coordinator STOP, N2 Narrative Wrapper Presentation Coordinator STOP, R2 Slide Wrapper Presentation Coordinator START, N3 Narrative Wrapper Presentation Coordinator START, R3 Slide Wrapper Presentation Coordinator STOP, N2 Presentation User Interface Presentation Coordinator STOP, R2 Presentation User Interface Presentation Coordinator START, N3 Presentation User Interface Presentation Coordinator START, R3 Presentation User Interface Table 1: Message sequence resulting from the completion of N2 Similarly, when the presentation coordinator receives a JUMP message from the presentation user interface, it responds by trying to nd a consistent global state. It considers three cases. 1. If the object selected is currently being played, it does nothing since the state is already consistent. 2. If the object selected has yet to be played (i.e., it is in the future), then the coordinator determines a set of changes that will result in a global consistent state where the selected object's beginning is enabled and barrier free. It then sends the appropriate set of START and STOP messages to match each display with revised state of the corresponding objects. 3. If the object selected has already nished playing (i.e., it is in the past), then the coordinator determines a set of events to roll back to achieve a global state where the beginning of the object is enabled and barrier free, and its end is not enabled. It again sends the appropriate set of START and STOP messages to match each display with the revised state. The presentation coordinator incorporates a two-phase approach where it navigates to a consistent state and then generates the START and STOP messages needed to reach that state. In this way, it avoids sending multiple messages that refer to the same object. Figure 6 shows the change in the objects displayed as a result of a user selection of R3. Table 2 shows the resulting messages that are sent as a result of the user selection. N1 N2 N3 Narrative Start G1 G2 G3 G4 G5 R1 R2 R3 R4 R5 End Slides Gregorian chant Bach Background Music Presentation status prior to user selecting slide R3 Presentation status after user selection of slide R3 and propagation of changes Figure 6: The change in the objects displayed as a result of a user selection of the slide R3 A similar process takes place if the viewer chose to jump to a previously displayed object. 10

11 Sending Module Message Receiving Module Presentation User Interface JUMP, R3 Presentation Coordinator Presentation Coordinator STOP, N2 Narrative Wrapper Presentation Coordinator STOP, G5 Slide Wrapper Presentation Coordinator STOP, Gregorian Chant Music Wrapper Presentation Coordinator START, N3 Narrative Wrapper Presentation Coordinator START, R3 Slide Wrapper Presentation Coordinator START, Bach Music Wrapper Presentation Coordinator STOP, N2 Presentation User Interface Presentation Coordinator STOP, G5 Presentation User Interface Presentation Coordinator STOP, Gregorian Chant Presentation User Interface Presentation Coordinator START, N3 Presentation User Interface Presentation Coordinator START, R3 Presentation User Interface Presentation Coordinator START, Bach Presentation User Interface Table 2: Message sequence resulting from the user selection of the slide object R3 5.2 FLIPS in Microsoft Windows Environment The implementation for FLIPS model in the Microsoft Windows 3.1 Environment uses the same system design concepts which is fairly straight forward on a multitasking platform such as UNIX, and customizes the system components into an integrated presentation environment to suit the non-multitasking desktop personal computer (PC) environment. A typical presentation running under Microsoft Windows 3.1 is shown in Figure 7. The distinct features for the implementation on Windows against UNIX are the richer presentation user interface and the integration of media displayers with the other system components, which demonstrates the platform-dependent way of FLIPS implementation. The core of the system code such as the presentation coordinator and the front-end processor on UNIX are ported to the Windows environment. We will describe the most critical components in Windows corresponding to the UNIX implementation in the following subsections Presentation User Interface The graphical user interface follows the Windows convention and is implemented using Microsoft Visual C++. The pull-down menus, tool bar, and keyboard shortcuts in the user interface provide easy access to the functionalities which described in section 4. The presentation user interface encapsulates the media displayers within a single parent window boundary and denes the initial placement for the media displayers. Viewer start a presentation by selecting a presentation specication le from the File menu. The Presentation menu allows the viewer to stop, resume, or quit a presentation. Each media object in the presentation is mapped to the corresponding media menu such as Text, Graphics, Video, and Audio. Viewer can select an object from the media menus, or from the Object menu which collectively displays a list to objects to choose from. Media menus provide user interaction for individual media object during a presentation. The tool bar is specically for the playout control for video object. The status bar in the bottom of the window produces instant feedback to the user regarding the operation in progress. The tool bar and status bar can be optioned on or o in the View menu so that the drawing area within the window shrinks or enlarge accordingly. 11

12 Figure 7: FLIPS presentation in the MS Windows 3.1 environment Media Display The implementation integrates the media displayers with the other system components. Since Visual C++ development kit provides object-oriented abstraction, the media displayers are dened as C++ class objects which encapsulate status information for the media displayers and export member functions to receive control messages from presentation coordinator. The implementation eectively abstracts away the media displayer wrapper using C++ class objects so that the detail implementation for media displayers is hidden from the other system components. This strategy eases the future extension to incorporate external media displayers within the Windows environment by simply overriding existing member functions in media displayer wrappers. All the ne-grain controls for media displayers are media object specic and can be activated through the media menus. The cassette player-like tool bar provides fast forward/backward, step forward/backward, pause, and resume for video control. Multiple audio streams such as background music and verbal introduction can be multiplexed and played back through the speakers simultaneously Presentation Coordinator The presentation coordinator is functionally equivalent to the implementation on UNIX except that the inter-process communication facility is replaced by intra-application event to send and receive 12

13 Figure 8: A sequence of operations to select a media object to jump to during presentation. messages. Viewer can stop, resume, or quit the entire presentation through the Presentation menu. The event processing is best understood by the following example. To JUMP from one point in the presentation to another, the viewer selects which media object to select via the pull-down menu for the specic media. When the presentation coordinator receives the JUMP message, it navigates through the object graph to determine the valid global state for all objects and via the message facility brings the media displayers to the dened states. Figure 8 demonstrates the sequence of operations for a JUMP message from presentation coordinator to video displayer: 1. viewer activates video object selection option through the pull-down menu (upper-left), 2. viewer selects the video to display (upper-right) in a pop-up window which generates a JUMP event in presentation coordinator, and 3. the presentation coordinator send messages to the media displayers and the user interface shows the desired video object and the other media objects which synchronize to the moment of video object display. 13

14 5.3 Comparison for FLIPS Implementation on UNIX and MS Windows The two implementations of FLIPS, whose benets and limitations, stem from the nature of the operating systems. The prototype implementation for FLIPS presentation on UNIX addresses the exibility and extensibility of FLIPS framework. On the other hand, the implementation on Windows 3.1 environment is tailored down for a single user desktop PC environment. The platform-dependent features are tabulated in Table 3. Platform UNIX (SunOS 4.1.3) Microsoft Windows 3.1 User Interface Toolkit Tcl/Tk and X11R5 Visual C++ and Video for Windows Processes multiple single Window Placement independent of each other encapsulated within a parent window Media Displayer external to user interface internal to user interface Control Communication BSD socket internal to user interface Remote Display X Window not applicable Table 3: Platform-dependent features for FLIPS presentation. The multi-user multi-tasking nature of UNIX, the exible interprocess communication mechanism, and the large amount of freely available source for media displayers and media formats make the implementation for FLIPS presentation on UNIX adhere to the framework. System modules for FLIPS map directly to individual processes in UNIX. Each display process runs independently with communication channels to the presentation coordinator. The mechanism is no dierent if the processes were on a single machine or each were running on an independent device with its own display monitor. However, in taking advantage of the external media displayers on UNIX, a coherent user interface may not be possible for the prototype and the placement of windows is somewhat uncontrollable. The Windows implementation is a fully integrated application program. It presents a nice coherent user interface and a single executable binary code which is ready for any FLIPS presentation. In coordination with the prototype FLIPS authoring tool developed in the Windows environment, a user has a full-edged FLIPS implementation on a single platform. The inability of interacting with external media displayers for the Windows implementation is not a constraint of FLIPS, but largely due to the amount of work to implement inter-application communications within the platform. Since the media displayers are part of the application program, the prototype supports limited amount of, but sucient, media formats on the PC. 6 FLIPS Authoring The authoring tool provides a user-friendly graphical user interface to create and modify presentation specication using a graph of media objects. The user interface provides graphic primitives which represents dierent media objects, enablers, and barriers in FLIPS model. Creating synchronization relationships between media objects can be done by highlighting the media objects, and then selecting the primitive for an enabler, a barrier, or their combination. The tool reduces the time to compose a presentation, and hides the resulting presentation specication from users so that users can easily put together the presentation materials without knowing the details of the specication le format. An example specication created by the authoring tool is shown in Figure 9, where user can visually inspect the presentation ow and perceive the resulting presentation 14

15 specication. Presentation tools on the UNIX and Windows platforms can both share the same specication le generated by the authoring tool transparently. Figure 9: Authoring a FLIPS presentation specication in the MS Windows environment. The implementation of the authoring tool is under Microsoft Windows 3.1 environment using Visual C++. The window convention for the authoring tool is consistent and coherent with the FLIPS presentation in Windows where menu items and status feedback to the user are available through the tool bar and status bar, respectively. The tool bar and pull-down menus provide an easy access to create (or remove) media objects and relationships between media objects (enabler or barrier), and to dene (or modify) the relationship and state information for media objects. The most frequently used menu items are grouped and mapped to the tool bar. The status bar in the bottom of the window produces instant feedback to the user regarding the manipulation that the user is performing. Error checking is presented to user through a pop-up alarm window. The tool bar and status bar can be optioned on or o so that the drawing area within the window shrinks or enlarge accordingly. In the following subsections, we will describe the mechanisms to work with media objects, enables, barriers, and presentation specication. 6.1 Object Creation and Denition Whenever a user wants to include a new media object, the object's properties must be dened. An object denition window is invoked whenever the user create a new object via the pull down menus or toolbar. Alternatively, user can change the object denition by selecting the Parameter in Dene menu or the corresponding toolbar icon in Figure 10. The parameter window corresponding to the selected object is shown in Figure 11. In the parameter window, the user can dene and modify the properties for a selected object including dening the exact location for the media le through the dialog entry (File Selected) and mark its media type by selected the appropriate radio button. The Open File button provides an alternative method of object selection where a le can be selected through a pop-up window and reected in the dialog entry. The Object Name in the parameter window provides a metaphor for users to dene a symbolic name to represent the media le. The presentation specication generated through the authoring 15

16 Figure 10: Menu items and corresponding toolbar icons to dene and preview media objects. Figure 11: Parameter window to dene/modify media object. 16

17 tool then uses that object name for the presentation interface. The Media Specic check box in the parameter window implements Alternative Actions dened in the FLIPS model [4]. The user can dene the duration of display and the action taken after the display comes to an end but must wait for barriers to be removed. Common alternative actions for a media object include holding the display until the barrier is satised or looping through the object again. A user can also preview any object within the graph of media objects by highlighting the object and then selecting the Object Display item in the Dene menu. 6.2 Dening Object Relationships One of the most critical tasks in authoring this type of presentation is to dene the synchronization of media. There are thee levels of synchronization denition dened within a FLIPS authoring and presentation. At the lowest level, ne grain synchronization is handled by the displayers themselves and the synchronized media are treated as a single composite object. As discussed earlier, the synchronization between objects is handled by the barrier and enabler relationships dened between objects. The relationships among media objects can be specied through direct drag-manipulation in the drawing area when a graphical icon in the tool bar or Link menu item representing an enabler or barrier is selected. Though the use of these tools are sucient to dene the dierent types of relationships, a third level of abstraction can be used to provide authoring support. The FLIPS model does not presume a stream-based presentation but instead supports a variety of presentation styles. While this provides for a great deal of latitude in creating presentations, it can also make the task of presentation creation more dicult. To circumvent this and to support this third level of abstraction, special tools have been incorporated into the toolbar that group sets of object relations into a single operation to support common presentation abstractions. These tools allow a user to easily select a set of media objects and dene relationships that encompass multiple enablers and barriers. As shown in Figure 12, the authoring operations dened include: Sequence: to have a set of media objects follow one after another. Master/Slave: to have one object control the start or end of another. Parallel First: to have two events occur in parallel whenever the rst occurs. Parallel Last: to have two events occur in parallel whenever the last occurs. These abstractions are covered in more detail in [4]. The authoring tool maintains the consistency between objects and their relations with other objects. When an object is removed from a presentation, the state information associated with this object and all relationships associated with the object are removed simultaneously. 6.3 Presentation Specication Through the authoring tool, a script le for a composed presentation is generated that can then be read by the presentation tool. In addition, an authoring le is saved that holds the graphical information pertained to the authoring tool. The user is always able to come back and modify the contents of a presentation using the existing authoring le and create a revised edition of the script le. In addition, this paradigm of reusing authoring le is well suited for novice users who would like to get their hands on creating FLIPS presentations without doing it from scratch. 17

18 Figure 12: objects. Menu items and corresponding toolbar icons to create relationship between media 7 Discussion and Future Work In this project we have dened a set of conditions that must be supported in order to have exible presentations that include variable duration objects and support user interaction. We have created a framework that provides the ability to enforce presentations where dierent applications can be synchronized in a coarse grain fashion with a wide range of constraints. One of the strengths of this application is that it can support presentations that are NOT linear in nature, and in fact, can incorporate sets of objects that are unreachable during the normal playout but can be reached through user interaction. Presentations could use this feature as a help facility or for creating alternative information tracks. An area where the application has great potential is to integrate this presentation application with content-based search tools. Researchers have been developing schemes for content-based searches. Recent work on searches on video databases using metabases include the work of Little et al. [2] and Rowe et al. [3]. Rowe et al. have dened query interfaces that provide relational, hierarchical browsing, and keyword search operations to the metadata indices. But until now, there was not a reasonable way to jump into the middle of a presentation of multiple media streams based on these searches. Our implementations can use the results of the these content-based searches and determine an appropriate state for each of the media objects that are part of the presentation composition. The specication of exible presentations is not an exact science and it is possible that specications can be made that do not allow the completion of presentation. With the inclusion of the barrier relation, it is possible for an object to wait for an event that will never occur, or for two events to wait for each other either directly or indirectly. We have developed an algorithm that can evaluate a specication and determine if the presentation will reach a conclusion (barring the viewer deliberately holding the presentation). This verication process is done prior to running the presentation and will verify that a presentation will complete even though a viewer may make many skips forward or backward in the presentation or interact in ways that alter the duration of a given object. We plan to incorporate this algorithm into the application in the near future. We are also investigating ways to incorporate a hierarchical structure into the model. This structure will allow groups of objects to be synchronized as a composite object without limiting the ability to specify synchronizing relationships between sub-objects and objects outside of the composite object. This extension will increase the expressive power of FLIPS to better handle multi-object looping. 18

19 References [1] J. Schnepf, V. Mashayekhi, J. Riedl, and D. Du. Closing the gap in distance learning: Computer-supported, participative, media-rich education. ED-TECH Review, fall/winter [2] Little T.D.C. etal. A digital on-demand video service supporting content-based queries. In Proceedings of ACM Multimedia 93, pages 427{436. ACM, [3] L.A. Rowe, J. S. Boreczky, and C. A. Eads. Indexes for user access to large video databases. In Proceedings of Storage and Retrieval for Image and Video Databases II, IS&T/SPIE Symposium on Elec. Imaging Sci. & Tech., February [4] J. Schnepf, J. Konstan, and D. Du. Doing FLIPS: FLexible Interactive Presentation Synchronization. IEEE Journal on Selected Areas in Communications. To be published in special issue on multimedia synchronization. [5] L.A. Rowe and B.C. Smith. A continuous media player. In Proceedings Third International Workshop on Network and Operating Systems Support for Digital Audio and Video, pages 237{249, November [6] T.D.C. Little and A. Ghafoor. Spatial-temporal composition of distributed multimedia objects for value-added networks. IEEE Computer, October [7] T.D.C. Little and A. Ghafoor. Synchronization and storage models for multimedia objects. IEEE Journal on Selected Areas of Communications, April [8] B. Prabhakaran and S.V. Raghavan. Synchronization models for presentation with user participation. In Proceedings of ACM Multimedia 93, pages 157{165. ACM, [9] L. Hardman, G. vanrossum, and D. Bulterman. Structured multimedia authoring. In Proceedings of ACM Multimedia 93, pages 283{289. ACM, [10] D. Wijesekera, D. Kenchamanna-Hosekote, and J. Srivastava. Specication, verication, and translation of multimedia compositions. Technical report, University of Minnesota, [11] A. Poggio. CCWS: A computer-based multimedia information system. IEEE Computer, Oct [12] L. Li, A. Karmouch, and N.D. Georganas. Multimedia teleorchestra with independent sources: Part 1 - temporal modeling of collaborative multimedia scenarios. ACM Multimedia Systems, 1(1):143{153, [13] G. Blakowski et al. Tool support for the synchronization and presentation of distributed multimedia. Computer Communications, 15(10):611{618, December [14] M. C. Buchanan and P. T. Zellweger. Scheduling multimedia documents using temporal constraints. In Proceedings Third International Workshop on Network and Operating Systems Support for Digital Audio and Video, pages 237{249, November [15] G. van Rossum, J. Jansen, K. Mullender, and D. Bulterman. CMIFed: A presentation system for portable hypermedia documents. In Proceedings of ACM Multimedia 93, pages 183{188. ACM,

20 [16] G. Drapeau. Synchronization in the maestro multimedia authoring environment. In Proceedings of ACM Multimedia 93, pages 331{339. ACM, [17] R. Hamakawa and J. Rekimoto. Object composition and playback models for handling multimedia data. ACM Multimedia Systems, 2(1):26{35, June [18] S.Eun, E.S. No, H.C. Kim, H. Yoon, and S. R. Maeng. Eventor: an authoring system for interactive multimedia applications. ACM Multimedia Systems, 2(3):129{140, September