Submitted to the 11th International Symposium on Visual Languages, Darmstadt, Germany, September David Wetherall

Transcription

1 An Interactive Programming System for Media Computation Submitted to the 11th International Symposium on Visual Languages, Darmstadt, Germany, September 1995 David Wetherall Telemedia Networks and Systems Group Laboratory for Computer Science Massachusetts Institute of Technology Abstract This paper presents PAVES, a direct manipulation system that combines aspects of visualization and multimedia systems to form an interactive video programming environment. PAVES extends the VuSystem media processing toolkit with dataow-style views for controlling compute-intensive media applications as they run. Thus video may be manipulated graphically and interactively both program and data are visual and live. PAVES is also novel in its approach to extensibility. Users may freely combine graphical and underlying VuSystem textual programming methods to restructure and reuse applications. This cooperative programming is available across sessions as well as within them. It is implemented with an object-oriented programming foundation that automatically translates between multiple program representations and maintains them in synchrony. PAVES (for Programming Active Video with an Embedded System) is an interactive and graphical environment for controlling such applications. It allows running programs to be directly manipulated via a dataow-style interface. The liveness and visual aspects of the programming system thus match the characteristics of the data. We feel that this is important for the problem domain, since visual data feedback is provided as the graphical program is manipulated. For example, the parameters of the motion segmentation program may be tuned by observing their eects on the processed video, and alternative algorithms may be tried by editing the ow graph. A novel aspect of PAVES is its support for using graphical and textual programming interfaces cooperatively. Like multiple-view editors, changes to one interface may aect the others, and dierent interfaces 1 Introduction As digital video is manipulated by increasingly powerful computers, new applications that perform significant computation on media are becoming viable. In addition to presenting media for conferencing, monitoring, or playback, programs may act on its content to become more responsive to their users. Figure 1, for example, shows a motion segmentation sub-program developed as part of the COMMA project [11]. It can be used in a variety of intelligent applications, such as automatically noting oce visitors while the user is absent. The author can be reached at: MIT Laboratory for Computer Science, Room 505, 545 Technology Square, Cambridge, MA 02139; Tel: ; djw@lcs.mit.edu. Figure 1: Bluescreen Program with Flow Graph 1

2 may be used to accomplish a given task. PAVES goes further, however, in translating graphical actions to their textual equivalent to help the user form an association, and integrating interactive (graphical) sessions with traditional (textual) development cycles. From the user's point of view, PAVES appears as a set of programming windows: ow graphs, control and code panels, and interpreters. Because these windows are embedded in an application toolkit, they are available across many video programs as a consistent higher level interface for control. We chose to provide a dataow representation both because it is a natural model of the media processing component of our applications, and because it has successfully been used in related domains [13, 15, 12, 9]. Other windows complement the dataow representation. Control and code panels allow the user to focus on the detailed conguration of particular processing modules. Interpreters allow the program to be controlled with a textual language based on Tcl [10]. Underlying PAVES is the VuSystem media processing toolkit [7]. It is the programming foundation, designed for compute-intensive media applications, from which PAVES inherits its timely manipulation of video. It also provides the interpreted programming level that is leveraged to present textual facilities to the user. This paper investigates PAVES by starting with the VuSystem, since it largely denes the application domain. Much of the strength of PAVES as an interface, however, lies in the way it handles multiple representations. We formalize this behavior in terms of a cooperative model. The PAVES interface is then presented from a user's perspective, followed by a discussion of the design issues we encountered. Finally, we place PAVES in perspective by comparing it with other work. 2 The VuSystem The VuSystem [7] software development environment focuses on compute-intensive multimedia applications, where the computer not only manipulates media but also digests it and performs independent actions based on media content. VuSystem applications are modeled as a graph of reusable processing modules through which video and other data ow. Some modules correspond to video capture from the outside world, some to computations such as compression, some to storage and retrieval, Commands Callbacks Out of band code Source Filter Sink Data In band code Data Figure 2: Organization of a VuSystem Program and some to presentation to the user. Program structure thus has a natural representation as ow that PAVES presents for interactive programming. Several levels of programming are available to the application developer, each suited to a dierent task. The basic programming division arises from the model of a program as separated into an in-band data stream and an out-of-band control message queue. This is shown in Figure 2. The components of the time-critical data stream, where video frames are directly manipulated, are implemented by a library of ecient C++ modules. Generic ltering modules may be specialized as a simple but constrained way of implementing new computations. Alternatively, new modules may be written from scratch, with few operational constraints. Programming of the control message processing, in which user events are decoded and user feedback is displayed, is expressed with an extended version of the Tcl [10]. The system contains an embedded Tcl interpreter, which provides great exibility in reconguring and extending the runtime environment. Tcl scripts are used to combine modules into applications, as well as describe standardized user interface panels for conguring each module. 3 Cooperative Approach At the outset of design, we considered PAVES to be an interface implemented on top of the VuSystem, one which would be more accessible to users who are also programmers. The resulting spread of programming levels is shown in Figure 3, with PAVES interactive 2

3 Accessibility simple to use interactive facilities detailed coding cycles visual flow graphs visual control panels command interpreters application module scripts scripts filter modules graphical methods Tcl C/C++ Generality general modules Figure 3: Spread of VuSystem Programming Methods Text Form source code.. describe execute Internal Form Program Database evaluate draw Figure 4: Model of Representation Shifts Interface Form facilities shaded. We were strongly motivated, however, to ensure that these levels could be used in combination. For example, PAVES ow graphs should be applicable to VuSystem programs developed textually, and programs developed graphically should not be locked into the graphical interface. This cooperative strategy is modeled in Figure 4. It addresses a weakness of many visual programming systems: that they are often well-suited to their specic tasks, but can lock users to their interface for a larger set of tasks, and become frustrating to use after a promising beginning. For example, graphical shells as exemplied by the MacIntosh Finder are apt for small le management tasks, while command-line shells as provided by UNIX are appropriate for more complex tasks. Yet there is no easy way to combine the strengths of both methods. The abstract cooperative model is quite general, and places few restrictions on the way users may work with the system; the diculty is the implmentation issues that this raises, which are discussed in a later section. Forms of View In Figure 4, three forms of representation of a single video application can be seen. Each generic form is presented to the user through dierent types of views that capture observable information about the executing program image. Many dierent views can reveal dierent aspects of the application. In the middle, the internal form represents the video program as it evolves from moment to moment. Its views are depicted by the program database because they provide momentary direct access to the running VuSystem application. A command interpreter is a view of internal form. The interface form models the executing program image across the graphical user interface. Its views are depicted by windows because they provide a direct manipulation representation of the application. The media ow graph, showing the pattern of processing currently implemented by the application and allowing it to be altered, is a view of interface form. The text form models the running video program in terms of the source code that corresponds to it. It is extended Tcl code that is interpreted by the VuSystem shell to run the application, and is edited as the traditional means of programming. Views of text form are depicted by lines of source code because they provide a snapshot of a portion of the program database at the moment they were constructed. Mappings The path moving from left to right represents the course of action for traditional applications: stored code is interpreted to yield a program image, which presents a standard interface to its users. It is the reverse path, from right to left, that enables cooperative programming between dierent interfaces. Cooperative use requires a means of mapping between the interface, text, and internal forms, since any view can be used to drive the others. The mappings are represented in Figure 4 with the addition of the evaluate and describe operations. The evaluate operation represents the interactive reprogramming of the application via its graphical interface, such as edit actions on the media ow graph. 3

4 Since the edit actions are not articially restricted, they may eect general program changes. For example, a user might remove components of the application, rearrange them, or substitute entirely new components. This requires the user interface to be updated to match the altered program, complicating the usually straightforward draw operation. A describe operation represents the serialization of the program into equivalent source code, code of the same type originally interpreted to launch the program. This code may be rened as well as reused, in contrast to lower level representations, such as a suspended process image, that are opaque to further development. However, determining the source code that is in some sense equivalent to a running application raises the issue of what state should be observed. Figure 5: Peering Inside the BlueScreen Group 4 A Tour of PAVES PAVES is targeted at customization and experimentation tasks, as well as prototyping and more general program development. It is intended to be accessible to inexperienced programmers as well as act as an overview facility that is available on demand for experts. We present its interfaces by using the BlueScreen video application as an example. BlueScreen segments moving objects from a stationary background by assuming a xed camera position, and is named because of its similarity to the chroma-key technique used to combine analog video 1. Its user programmable interface is shown in Figure 1, with the input and output video streams showing a moving person segmented from the oce scene in the background. We chose it as an example because it performs a non-trivial task, is implemented with a small number of more primitive processing modules, and is itself used as a subsystem in larger applications. Media Flow Graphs Figures 1, as well as 5 and 6, contain ow graph representations of the media processing component of the program. In terms of the cooperative model, the ow graph is an interface form of view. It presents a direct manipulation interface with a straightforward interpretation: vertices represent processing modules; and edges represent the ow of media between them, from left to right. Media ows are not visually dierentiated, and no restrictions are made on the connections between modules. Each ow may contain payloads of any type, and modules process only those types they recognize, passing the rest without error. It is thus allowable, but not useful, to pass video data to an audio processing device. Beyond edit actions for creating, destroying and rearranging modules, modules may be grouped to form a new type of module. The resulting abstraction is rst class, that is, it has all the properties of a primitive modules. A grouping is seen in Figure 5, which shows the ow graph of Figure 1 after abstracting the ve modules that comprise the segmentation eect. The internal arrangement of the group is also presented, and appears in a separate window to emphasize the existence of the abstraction barrier. With these operations a library of primitives, a means of combination, and a means of abstraction ow graph manipulations form a language and support a method of programming. Interpreters For textual programming, an interpreter window is provided in the lower section of the ow graph window. In terms of the model it corresponds to an internal 1 In chroma-keying, a specially colored background is eliminated and substituted by a dierent background. Saturated blue is traditionally chosen as the special color because it is unlikely to occur in natural scenes. 4

5 User Interface Boundary presenters Flow Graph Executing Program Image Control Panel other views recognizers Figure 6: BlueScreen views generated by PAVES Figure 7: Presentation-Style Synchronization Model form of view, recording queries and commands at the moment they aect the executing program. As well as providing direct access to the underlying VuSystem program, the interpreter monitors ow graph activity. When a graph operation is completed and executed, its textual equivalent is displayed in the interpreter window as though it had been typed. Thus not only may either interpreter or ow graph to be used as appropriate, but the user is aided in forming an association between equivalent textual and graphical commands. 5 Design Issues Our goal of using graphical and textual programming cooperatively, both in a single interactive session and across oine development sessions, raised several design issues. Primarily, automatic mechanisms were needed for: synchronizing multiple program views; drawing a graphical representation of an application that was developed textually; and serializing equivalent source code for an application that was developed with graphical tools. Synchronization of Views Control Panels and Code Fragments Control panel and code fragement windows are the remaining views, generated automatically by PAVES to complement ow graphs. An example of each, coresponding to the highlighted modules, is seen in Figure 6. Control panels are a view of interface form, and contain controls for dynamically adjusting the internal parameters of modules. Code fragments are a text form of view that reveal the internal state of the modules as a source code serialization of the running application. They provide a means of linking the graphical views of the program with the underlying textual representation, and the code can be saved for later renement. A presentation-style synchronization model, after that of Ciccarelli [2] and shown in Figure 7, is used to maintain the program and interface in synchrony. In Figure 7, each view is linked to the executing program by recognizer and presenter activities. The recognizer translates interface directives, such as edit actions on the ow graph, into program commands. The presenter monitors the program database for relevant changes and translates in the reverse direction. Implementing the recognizer activity is accomplished by using the event translation functionality of the Xt widget set [1], which batches sets of primitive window events into higher level actions. Implementing the presenter activity potentially requires feedback from each recognizer to all presenters. 5

6 Many ad hoc schemes can be invented to cater to particular situations, but they will work at the cost of restricting exibility. Instead, PAVES build presenter and view management support into its object infrastructure. The outcome of each observable change to a module is packaged and signalled to a view manager, which distributes it to the relevant presenters. Presenters themselves are responsible for compressing multiple module change messages into the appropriate incremental interface change. For example, destroying a media processing module causes all of its ports to be destroyed too. Multiple messages are generated, yet only one screen widget (corresponding to the entire module) should be destroyed. PAVES takes advantage of the generality of this synchronization framework in two other ways. Formalizing the feedback pathways allows for monitoring the trac along them. This is the mechanism by which the textual translation of graphical ow graph edits is displayed in the interpreter window. Programs changes that do not originate with an interface will also be correctly handled. For example, a program may internally alter media ow connections as a normal part of its operation, and these changes will be externally reected by the graph. The main programming convention is the organization of an application as a tree with an embedded graph. The tree indicates functional containment, while the graph connects its leaves according to media ow. This structure can be seen in Figure 8 for the Bluescreen example. We have tried to adopt structuring conventions that provide insight about the organization of the application but that do not overly constrain the programmer. The conventions then provide a generic model for applications, so that operations which are safe for the model are safe across a variety of real applications. The structuring conventions also serve to embed PAVES interface functionality into applications. Applications are expressed in an object-oriented extension to Tcl, Object Tcl [14], that was developed as part of PAVES. Inheritance mechanisms on objects ensure that all modules have a default method to draw and describe themselves. The programmer can then use class and object delegation, along with multiple inheritance with method combination, to specialize operations. This is how abstractions (groups of objects acting as a single object) and proxy objects (representing remote computation) are accommodated. vs Drawing Interfaces and Describing Programs vs.source.video vs.source vs.filter vs.sink vs.filter.stat vs.filter.dup vs.filter.mg vs.filter.mo vs.filter.color vs.sink.raw Figure 8: Program Structure Structuring Conventions and Objects vs.sink.bluescreen The translation of running programs into graphical interfaces and equivalent source code is facilitated by object-oriented programming techniques and structuring conventions. The structuring conventions and inheritance mechanisms, plus the primitive object library, provide all the framework needed to draw interfaces and serialize source code. Beyond this, some special issues arise for each type of view. A default ow graph layout is needed because applications that were developed textually have no userspecied positioning information. We use a simple topological sort algorithm to ensure that media ows from left to right, and allow the user to improve upon this. A means of deciding what state information should be serialized to accurately describe the program is needed. Some state information captures the intent of the application, such as the sequence of media processing operations it performs. Other state information pertains only to a particular run of the program, being derived from the content of the video and other environmental bindings. These types of state are, in the general case, dicult to separate. PAVES solves this problem by leveraging the structuring conventions, effectively letting the user decide. 6

7 6 Perspective PAVES addresses the domain of compute-intensive multimedia applications. We have tried to build on successes in related domains: the Cantata interface to KHOROS for image processing [15]; AVS for scientic visualization [13]; and MAVIS for active computer vision [9]. With them, we share with them the basic idea of image data owing through a graph of processing elements as introduced in HI-VISUAL [5]. Unlike them, we cater to wider audience than researchers. This is reected in choices such as: an emphasis on video presentation rather than resource control; embedding of the interactive facilities as secondary tools available on demand; and visually undierentiated ows of data with unconstrained module connections. PAVES is fully live in the sense of Tantimoto's VIVA [12], but achieves this through the timeliness of its video data, rather than requiring a separate constraint-based propogation system. This is because video is always owing, with resource scheduling accomplished by the underlying VuSystem, so program changes naturally lead to presentation updates that appear to be responsive. Timeliness also distinguishes PAVES from inherently oine video programming environments, such as VideoScheme [8]. PAVES is also notable for its support of multiple view editing. It allows graphical and textual programming methods to be used in combination, as do applications such as SchemePaint [3]. Its synchronization and free interchange between textual and graphical methods, however, match the generality of generic multi-view frameworks such as MViews [4], and take them a step further with a cooperative model that spans interactive and traditional development cycles. Finally, PAVES has supports extensibility, an area which is weak in many dataow systems. Like VPL [6], modules can be grouped interactively to dene rstclass abstractions, which may then be hierarchically composed and interrogated with the graphical interface. 7 Results and Conclusions Half a dozen users have experimented with PAVES through a variety of video programs and over the course of a year 2. We have found it most convenient 2 Actually, many more have used PAVES. The TNS World Wide Web technology demonstrations are available on the Inas an embedded facility that is consistently available across VuSystem applications for expressing throwaway customizations. That is, programming is often a secondary concern to application use. As in similar domains, a dataow-style interface proved an eective formalism, this time for expressing the media processing component of computeintensitve multimedia programs. We believe this is promising area for future applications, and that interactive programming facilities coupled with video feedback can be an eective user tool. We also found the cooperative approach, which allowed graphical and textual methods to be used in combination and across sessions, to be promising and worthy of further exploration. While we did not fully achieve the goal of automatically integrating interactive and oine development cycles (manual intervention was typically involved to verify that the generated code was both necessary and sucient), we enabled a useful synergy of developement modes. In implementing our model, the object foundation provided by Object Tcl proved particularly useful. An object naming scheme with inheritance is well-suited for expressing the patterns of similarities and dierences between program components in a modular fashion. Acknowledgements I am grateful to my supervisor, Prof. David Tennenhouse, for his support and encouragement. I am also fortunate that this research was supported by the Advanced Research Projects Agency of the Department of Defense, monitored by the United States Air Force (AFSC, Rome Laboratory) under contract No. F C-0019, and by the University of Western Australia under a Hackett Studentship. References [1] Paul J. Asenta and Ralph R. Swick. X Window System Toolkit: The Complete Programmer's Guide and Specication. Digital Press, [2] Eugene C. Ciccarelli. Presentation based user interfaces (revised Ph.D.). Technical Report MIT/AI/TR 794, Articial Intelligence Lab., ternet and include several video programs from which PAVES is accessible. The feedback gained from this exposure was positive, but not detailed enough to include here. 7

8 Massachusetts Institute of Technology, August [3] Michael Eisenberg. Programmable Applications: Interpreter Meets Interface. Technical Report AI Memo 1325, MIT, October [4] John C. Grundy and John G.Hosking. Constructing Multi-View Editing Environments Using MViews. In Proceedings of 1993 IEEE Symposium on Visual Languages, pages 220{224, Bergen, Norway, August [14] David Wetherall and Christopher J. Lindblad. Extending Tcl for Dynamic Object-Oriented Programming. Submitted to the Tcl/Tk Workshop 1995, Toronto, Ontario, July [15] C. Williams and J. Rasure. A visual language for image processing. In IEEE Computer Society Workshop on Visual Languages, Skokie, IL, IEEE Computer Society. [5] M. Hirakawa et al. HI-VISUAL Iconic Programming. In Proceedings of the IEEE Computer Society Workshop on Visual Languages, Linkoping, Sweden, [6] David Lau-Kee et al. VPL: An Active, Declarative Visual Programming System. In Proceedings of VL '91, pages 40{46, Kobe, Japan, October [7] C. J. Lindblad, D. Wetherall, and D. Tennenhouse. The VuSystem: A Programming System for Visual Processing of Digital Video. In Proceedings of ACM Multimedia 94. ACM, October [8] James Matthews, Peter Gloor, and Fillia Makedon. VideoScheme: A Programmable Video Editing System for Automation and Media Recognition. In Proceedings of ACM Multimedia 93, pages 419{426, Anaheim, CA, August ACM. [9] Thomas J. Olsen et al. MAVIS: A Visual Environment for Active Computer Vision. In Proceedings of Workshop on Visual Languages, pages 170{176, Seattle, WA, September IEEE. [10] John Ousterhout. Tcl: An Embeddable Command Language. USENIX, [11] William Stasior. Visual Processing for Seamless Interactive Computing. in MIT/LCS/TR-590, November [12] Steven L. Tantimoto. VIVA: A visual language for image processing. Journal of Visual Languages and Computing, 1(2), [13] C. Upson et al. The Application Visualization System: A computational environment for scientic visualization. IEEE Computer Graphics and Applications, pages 30{42, July