MediaLisp -- A System Design for Multimedia Augmented Transition Network in Lisp

Transcription

1 MediaLisp -- A System Design for Multimedia Augmented Transition Network in Lisp -- for CIS6931 Term Paper Assignment March 2001 Yanli Sun (50%) Xiangyu Ye (50%) Schools of Computer Science, Florida International University, Miami, FL.

2 ABSTRACT This paper presents a draft design of MediaLisp, a feasible approach to compose multimedia presentations and to control the procedure of the presentation by using Lisp based system. A multimedia presentation consists of a cluster of multimedia objects. Due to the complexity of multimedia object, it is very difficult to describe the complex temporal, structural and behavioral properties of a multimedia presentation and all underlying objects by using conventional approaches. Though Lisp language is very simple, it is expressing to meet the requirements for describing temporal, structural and behavioral properties of multimedia objects and presentations. Based on Lisp convention, MediaLisp provides an integrated system to evaluate and process presentation. MediaLisp extends the semantics to define the temporal, structural and behavioral properties of multimedia functionality by using MATN semantic model and property set. The MediaLisp interpreter provides fundamental functionality to control and synchronize media streams during the procedure of presentation and servers as a framework over the component based hierarchy to manage all multimedia related functional components. 1

3 0. Introduction Multimedia has gained significant popularity and extensive interest nowadays. Unlike traditional information processing (text number ect), multimedia information is very complex in nature and usually time-dependent. ations and to control the procedure of the presentation by using Lisp based system. A multimedia presentation consists of a cluster of multimedia objects. Due to the complexity of multimedia object, it is very difficult to describe the complex temporal, structural and behavioral properties of a multimedia presentation and all underlying objects by using conventional approaches. Though Lisp language is very simple, it is expressing to meet the requirements for describing temporal, structural and behavioral properties of multimedia objects and presentations. Based on Lisp convention, MediaLisp provides an integrated system to evaluate and process presentation. MediaLisp extends the semantics to define the temporal, structural and behavioral properties of multimedia functionality by using MATN semantic model and property set. The MediaLisp interpreter provides fundamental functionality to control and synchronize media streams during the procedure of presentation and servers as a framework over the component based hierarchy to manage all multimedia related functional components. 2

4 1. MATN Model Multimedia Augmented Transition Network (MATN) is a variant of Augmented Transition Network (ATN). This model is introduced by Shu-ching Chen in The idea of MATN is to describe the behavior of multimedia streams by defining states and transitions at the change (begin or end) of a stream. MATN model is defined as follow: - The building blocks: MATN consists of nodes and arcs. Each node represents a state, and each arc represents the transition of a state. The states can have state names. A new state is created whenever there is any change of media streams in the presentation. There are two situations for the change of media streams: any media stream finishes to display; any new media stream joins to display. The arcs represent the time flow from one state to another. The arc labels represent the media streams being displayed in the time duration between the states the arc connects. P X 1 P/X X 2 1 P/X X 3 2 P/X X 4 3 P/X X 5 4 P/X 5 Time Instant t 1 t 2 t 3 t 4 t 5 t 6 Time Duration d 1 d 2 d 3 d 4 d 5 Figure-1 An MATN example 3

5 - The inputs: The inputs for MATNs are modeled by multimedia input strings. A multimedia input string consists of several input symbols and each of them represents the media streams to be displayed at a time interval. In a multimedia input string, the symbol & between media streams indicates these two media streams are displayed concurrently. P/X i denotes the state after X i has been read. In the example shown in Figure 1, there are five occurrences of media stream combinations at each time-duration: Duration d 1 : V 1 & I 1 Duration d 2 : V 1, T 1, I 1, & A 1 Duration d 3 : T 2, I 1 & A 1 Duration d 4 : V 2, T 2, I 2 & A 1 Duration d 5 : V 2 & A 1 4

6 2. Lisp Language Lisp language was invented in late 1950 s as a formalism of reasoning about the use of recursion equations as a model for computation. LISP stands for LISt Processing. Its development history has often been associated with symbolic processing and with both computer and human languages. A heterogeneous list data type has always been built into the language in order to efficiently deal with arbitrary and changing models. Besides its many other advantages, the following features caught our attention and made us choose Lisp to represent MATN: Decoupled syntax from semantics: Lisp defines a simple regular syntax that is used by default for reading programs. It also provides a means to change that syntax, and an internal data representation for programs that are not test based. Flexibility: Lisp can be combined with other languages to produce wide variety of applications. Heterogeneity: Lisp supports powerful data type (numeric, as well as heterogeneous objects), polymorphic functions, logical pathnames, timing and date utilities, sophisticated control structure. Object-oriented: Lisp was the first ANSI standard to incorporate object oriented programming. It has strong support of Meta Object Protocol (MOP). 5

7 Rapid prototyping: Lisp is comprehensive. It supports modular design, supports interactive development, debugging and update. Lisp is a very good vehicle for rapid prototyping, which enables early review and discovery of problems. Database accessing support. Now let s move to the system design of MediaLisp. 6

8 3. Multimedia Data Types in MediaLisp Based on traditional lisp convention, MediaLisp achieves the functionality concerning multimedia composition by extending a set of data types and corresponding functions designated to describe and create multimedia objects. The following illustration gives an overview of those data types: MultimediaObject UnitMediaStream CompositMediaStream MediaStream Delay Sequence Parallel Figure 2.1. Multimedia Data Types In above figure, MultimediaObject, UnitMediaStream and CompositStream are virtual data types used internally in MediaLisp. Though they are not reachable by users, they do have semantic meanings and tightly related to internal implementation. MediaStream, Delay, Sequence and Parallel are concrete data types used to define media streams in multimedia representation. Every data type has a property set that can be constructed at run-time and bound to individual instances. There are two property types: system preserved property and user 7

9 defined property. All the properties can be read and written at run time. The property set functions as hash table and bound to individual media stream. System preserved properties are properties whose names and values are defined by MediaLisp system used to specify common features regarding to multimedia representation, such as name, time duration and etc. Those properties can be recognized by MediaLisp and used to retrieve the media stream and control its display User defined properties can be any name-value pairs without name conflict with system preserved properties. They are bound to the multimedia object to store information used at run time. Users can feel free to use this feature to impose advanced control features at run time Description of Multimedia Data Types: ModiaStream: The ModiaStream data structure severs as basic unit that represents a single media stream within a multimedia representation. Each ModiaStream has a name within the multimedia representation and a set of properties. In Media Lisp, the name of a ModiaStream need to be distinct so that it can be referred by name. The system preserved properties of ModiaStream are listed in following table: System Preserved Properties Name Type Description Name String The name of MediaType String The media types, such as Text Video Audio Image 8

10 ect. Duration Integer Location String Priority Integer DisplayPolicy Integer Channel Integer Player String The number of basic time intervals. The negative value means the property needs to be determined at run time. This property tells the location to retrieve the media stream, it adopts Uniform Resource Location (URL) convention. This value is used to determine the priorities among media streams played in parallel. This value can be one of the predefined valued: 1. To be displayed exactly one time 2. To be repeatedly displayed at least one time 3. To be repeatedly displayed any times 4. This value specifies the channel to physically load huge media objects (e.g. video stream). It is assigned by system and is read-only. User can use this value to refer the run-time properties of the channel so as to programmatically control the display. It allows users to specify their desired device to display the media stream. If this value is not specified, MediaLisp can assign a proper one according to MediaType property MediaStream Creation: By using function Define to define a MediaStream object. The function interface is as follows: Define (Define stream_name) Stream_name The name of the media stream. This value is Nil, then MediaLisp would automatically assign a name Return Value: The reference of the MediaStream Onject Temporal Relationship: For such objects, the time duration is calculated as follows: 9

11 For non-looping object, the same value as it Duration property. For looping object, if it is placed in a sub-net of MATN, the duration if the sub-net's overall duration, otherwise is infinite Explicit Delay: Sometimes, it's necessary for user to impose an explicit delay for some duration so as to control (synchronize) the multimedia representation. For this purpose, we introduce the Delay data type in MediaLisp. The property set of such data type contains "Name" and "Duration" with regarding system preserved properties. Delay object can be created by the following clause: Delay (Delay interval) Interval The number of unit time interval to wait for Return Value: The reference of the Delay object Sequencing: Semantics of Sequencing: To describe a sequence of media streams, we adopt the operator <<<. A sequence is a tuple of variable length surrounded by parenthesis with the first element is<<<. For 10

12 example, to represent a sequential presentation of media streams x1, x2 and x3, we can use: (<<< x1 x2 x3). Sequencing allows nested structures, each element can be A single media stream A sequence of media streams A parallel of media streams The property set of Sequence contains "Name" and "Duration" with regarding system preserved properties Temporal Relationship Regarding to Sequencing: The time duration is the summation of all time duration of contained streams within it. The temporal relationship among streams within a sequence can be obtained by simply applying "meet" relationship sequentially over all contained media streams. For example, given a sequence (<<< x1 x2 x3), then the time duration of the sequence is τ τ τ Where x1 x2 x3 τ x1 + x2 τ + τx 3 Duration of x1 Duration of x2 Duration of x3 The temporal relationship according to MATN: x1 x2 x3 11

13 Parallel(Concurrence): Semantics of Sequencing: To describe media streams played in parallel, we adopt the operator. A parallel is a tuple of variable length surrounded by parenthesis with the first element is. For example, to represent presentation of media streams x1, x2 and x3 palyed in parallel, we can use: ( x1 x2 x3). Parallel allows nested structures, such that each element can be A single media stream A sequence of media streams A parallel of media streams The property set of Parallel contains "Name" and "Duration" with regarding system preserved properties Temporal Relationship Regarding to Parallel: The time duration is the maximum all time duration among all contained streams within it. The temporal relationship among streams within a sequence can be obtained by simply applying "meet" relationship sequentially over all contained media streams. For example, given a sequence ( x1 x2 x3 x4 x5), then the time duration of the sequence is τ τ τ τ τ Where x1 x2 x3 x4 x5 τ τ 2 τ 3 τ x4 x5 (max x1 x x Duration of x1 Duration of x2 Duration of x3 Duration of x4 Duration of x5 τ ) 12

14 The temporal relationship depends on the mutual relationship among all media streams' duration. We can apply use Bucket-Sort algorithm to calculate the temporal relationship as follows: 1. Calculate the number of all distinct time duration 2. Construct a bucket according to duration in ascending order 3. Put each media stream into the buckets in which the duration values are less on equal to its duration 4. For each bucket, combine all names of media streams inside to build the multimedia string for an edge of MATM For above example, suppose the time duration distributed as follows: τ x1=5, τ x2=3, x3 τ =7, τ x4=7, τ x5=5. The total number of distinct time duration is 3. The buckets and the multimedia strings are shown as below: Buckets Contents Multimedia String of MATN 3 x1, x2, x3, x4, x5 x1&x2&x3&x4&x5 5 x1, x3, x4, x5 x1&x3&x4&x5 7 x3, x5 x3&x5 The corresponding MATN is drawn as follows: x1&x2&x3&x4&x5 x1&x3&x4&x5 x3&x5 13

15 3.2. Related API Functions: In order to manipulate multimedia object, we define a small set of API functions described as follows: Query Name by Reference:? (?stream_ref) stream_ref The reference of the media stream Return Value: The name of the media stream Getting Reference by Name:? (? stream_name) Stream_name The name of the media stream Return Value: The reference of the media stream. Nil means no media stream exists under this name Property Query: GetProperty (GetProperty stream_ref property_name) (GetProperty stream_name property_name) Stream_name Or Stream_ref The name or reference of the underlying media stream Property_name The name of the property Return Value: If the property exists, then return the corresponding value, otherwise return Nil. 14

16 Property Update: SetProperty (SetProperty stream_name property_name property_value ) or (SetProperty stream_ref property_name property_value ) Stream_name Or Stream_ref The name or reference of the underlying media stream Property_name The name of the property Property_value The value of the underlying property Return Value: None. Description: For convenience, this function supports variable input. The input must contain exact integer number of "name-value" pairs. For example: (SetProperty stream_name name1 value1 name2 value2 name3 value3) 15

17 4. Multimedia Representation Control Except for functionality supported in general Lisp language, MediaLisp also provides support for media stream branching and composing, as well as precise control for multimedia presentation. The MediaLisp maintains a set of system properties. Those properties can be used to configure the behavior of the overall environment for the multimedia presentation. Dor the sake of flexibility, a concept of Phase for fine-tuning of run-time behavior of one or a group of media streams: As in the MATN model, media stream branching occurs whenever user choice or automatic choice based on condition table occurs. Media stream choice provides great flexibility to deal with unpredictable situations in real life. We define three branching functions for multimedia presentations, they are Case and Select System Property: The are two types of system properties. They are defined by MediaLisp (thye are recognizable by system) so as to describe and the control the runtime behavior of the whole presentation. The set of system properties should contain at lease: Name Type Description Physical_Interval The physical time duration of unit time Number interval. Error_Handling Integer The predefined error handling behavior 1 Ignore error 2 Prompt message 3 End presentation 4 16

18 Accordingly, we define can use functions GetSProperty and SetSProperty to access system properties: System Property Query: GetSProperty (GetSProperty property_name) Property_name The name of the underlying system property Return Value: If the property exists, then return the corresponding value, otherwise return Nil System Property Update: SetSProperty (SetSProperty property_name property_value ) or (SetSProperty stream_ref property_name property_value ) Property_name The name of the system property Property_value The value of the underlying system property Return Value: None. Description: For convenience, this function supports variable input. The input must contain exact integer number of "name-value" pairs. For example: (SetSProperty name1 value1 name2 value2 name3 value3) 4.2. Phase: The introduction of concept Phase provides detailed control over multimedia presentation. A phase is a sub runtime environment under environment MediaLisp. 17

19 Similarly, phase maintains a set of properties similar to system properties. Actually, the runtime system can be treated as an implicit phase. If there is no phase specified, MediaLisp uses system properties. Also, used can use any user defined phase properties as long as there is no conflict with the preserved phase properties. Whenever a phase is specified, MediaLisp uses the properties of the phase instead. When a phase finishes, MediaLisp switches back to the default system properties to control remaining presentation. The following table lists some basic phase properties: Name Error_Handling Type Integer Description The predefined error handling behavior 1 Ignore error 2 Prompt message 3 End current phase Semantic of Phase Creation: Phase (Phase phase_name) Phase_name Return Value: The reference of the phase Optional. If it's not specified, the system will automatically assign an unique one Phase Property Query: GetPProperty (GetPProperty phase_name Property_name) or (GetPProperty phase_ref Property_name) phase_name Name of the specific phase phase_ref Reference of the specific phase Property_name The name of the underlying system property 18

20 Return Value: If the property exists, then return the corresponding value, otherwise return Nil Phase Property Update: SetPProperty (SetPProperty phase_name property_name property_value ) or (SetPProperty phase_ref property_name property_value ) phase_name Name of the specific phase phase_ref Reference of the specific phase Property_name The name of the phase property Property_value The value of the underlying phase property Return Value: None. Description: For convenience, this function supports variable input. The input must contain exact integer number of "name-value" pairs Media Stream Branching Case: The Case function is similar to "switch" function in C++, its decisions are based on integer value. Usually, the integer value should be the choice sequence number from a user interactive choice. For this purpose, we also define ChoiceList data type to describe user prompts Semantic of Case: Case (Case num stream1 stream2 ) Num The integer on which the decision should be based at runtime 19

21 Stream i The corresponding media stream reference Return Value: The reference of media stream (unit stream, parallel or sequence) that is determined by num at runtime. Description: If there is no stream defined corresponding to the choice number, an exception will be passed to the phase or system ChoiceList: A ChoiceList object is simply an array of prompt string. The sequence number starts from 1. In MediaLisp, ChoiceList object triggers a dialog that shows all prompts in order to user and returns the choice number. The semantics of ChoiceList is shown as follows: ChoiceList (ChoiceList prompt1 prompt2 prompt3 ) prompt i The corresponding prompt string reference Return Value: The reference of choice object Select An alternative of media stream branching is Select. Compared with Case, Select provides much more flexible and detailed detail. This function can be used to serve automatic decision-maker depending on various system or phase properties at run time. The semantics of Select is shown as follows: Select (Select condition1 stream1 condition2 stream2 ) Condition i The condition which should be of boolean data type 20

22 Stream I The media stream reference corresponding the underlying condition Return Value: The reference of choice object The input parameter consists of a list of "boolean-stream" pairs. This function is executed as follows: The MediaLisp evaluates the input list pair by pair from left to right. If the condition is true, then return the corresponding media stream reference. If no condition is true, then raises an exception and passes it the underlying phase of system. To make sense, we can always put an explicit "true" condition (value T in Lisp) paired with a stream reference to specify a default value Temporal Relationship in Media Branching All media branching functions can be regarded as a specific media stream (can be simple media stream, parallel or sequence of media streams) whose actual time duration need to be determined at runtime. To keep track their actual time, we need to utilize MATN model to describe and control the branching. The detailed technique concerning this issue will be discussed in later section Presentation After all media streams and their temporal and behavioral properties has been defined, the work left is to bind all those information together and add them to the MATN based execution tree implemented within MediaLisp. This job can be done by the Present function call. The semantics of Present is listed as below: 21

23 Present (Present stream) or (Present phase stream) stream phase Return Value: None. The reference of media stream The reference of underlying phase 22

24 5. Comprehensive Examples So far we have briefly discussed all necessary conventions about MediaLisp. In this section, we will demonstrate MediaLisp to by an example Example1: In this example, a bunch of multimedia objects of different types need to be displayed time by time. They are listed in following table: Object Type Start Time Duration Location V1 Video T1 Text 0 5 C:\temp\text1.txt I1 Image 3 7 D:\iamges\image1.jpg A1 Audio T2 Text 5 2 C:\temp\text2.txt The corresponding MediaLisp script: (Define "v1") (setproperty "v1" "Duration" 5 "Mediatype" "video" "location" " (Define "T1") (setproperty "T1" "Duration" 5 "Mediatype" "text" "location" "C:\temp\text1.txt") (Define "I1") (setproperty "I1" "Duration" 7 "Mediatype" "image" "location" "D:\imges\image1.jpg") (Define "A1") (setproperty "A1" "Duration" 8 "Mediatype" "audio" "location" " ") (Define "T2") (setproperty "T2" "Duration" 2 "Mediatype" "text" "location" "C:\temp\text2.txt") (setf x1 ( (? "v1") (? "T1"))) (setf x2 (<<< (delay 3) ( ( (? "I1") (? "A1")) ) (setf x3 (<<< (delay 5) (? "T2") )) (present ( x1 x2 x3) ) 23

25 5.2. Example2: In this example, we use a sequence of images to simulate an animation. For each image there are two versions of source files: compressed version and uncompressed version. Suppose for j th image Compressed version: Uncompressed version: Whether to use the compressed or uncompressed version, depends on the bandwith of the system. Let N be the total number of images included in this animation, then the script is: (setf ANI NIL) (setf i 1) (while (le N) (setf x1 (define NIL)) (setf x2 (define NIL)) (setproperty x1 "duration" 1, "Mediatype" "image") (setproperty x2 "duration" 1, "Mediatype" "image") (setf s1 (+ " "compressed_" i "jpg")) (setf s1 (+ " "uncompressed_" i "jpg")) (setf unit (select (getsproperty "Bandwidth") x1 true x2)) (setf ANI (if (eq ANI NIL) unit (<<< ANI unit))) ) (present ANI) 24

26 6. Issues Concerning MediaLisp Interpreter To support multimedia related functionality, MediaLisp must provide: Multi-threaded execution and management support. Network communication ability Real-time management and synchronization. Intelligence to handle the semantic heterogeneity of multimedia representation. The ability to display various multimedia objects via various devices. So the in internal implementation of MediaLisp interpreter is much more complex than common Lisp interpreters, it has four sub-systems that work together to process a multimedia presentation. They are: General Lisp interpreter MediaLisp interpreter Real-time media presentation control sub-system Real-time media retrieval management sub-system 6.1. General Lisp Interpreter: This part severs as common list interpreter that evaluates (interprets) instructions in MediaLisp program except for those in MediaLisp extension. It manages all variables and control flows. It also passes the MediaLisp instruction to MediaLisp Interpreter MediaLisp Interpreter: 25

27 This interpreter is designed to evaluate MediaLisp instructions. It creates some extra data structures will be created to hold all the media object's internal properties as well as their mutual temporal and behavioral relationship, such as: Vector of unit media objects: Vector of phase objects (the outermost phase is the A MATN graph data structure whose edges are multimedia strings and nodes are time stamps and choice markers. The pointers (reference relationship) among those data structures will alos be built up within this part Real-time media presentation control sub-system: This sub-system plays a crucial role in multimedia representation. It serves as a central coordinator to control (track and synchronize) the media retrieval, display and error handling during realtime presentation based upon the MATN graph, Starts from the start node of the MATN graph, the coordinator periodically (every unit time interval) looks one more steps ahead to predict what to do next time interavl. Before a multimedia object starts to display, it needs to initiates a pre-retrieval thread to build up the network transmission channel, caches the whole object or its header depending on the properties of the object. Whenever a choice is made, it can correctly lead the media stream to the certain branch. It also manages media displayers and assign different media objects to certain players. A small knowledge needs to be involved to help making decision. 26

28 7. Conclusion and Future Works From the proposed draft design of MediaLisp in this paper, we have presented a feasible approach to describe and control multimedia representations. By adopting the MATN semantic model, this approach is expressive, flexible and convenient enough to handle multimedia presentations. Though Lisp was developed in 1950s, it has gained extensive interests in recent 10 years since its simplicity and flexibility. The semantics is very suitable to define tree-like data structures. By extending Lisp with multimedia support, this approach can easily describe all static and dynamic properties and conditions with respect to multimedia presentation. By adopting the MATN semantic model, MediaLisp can precisely control and keep tracking of both static and dynamic temporal relationship among all multimedia objects in a presentation. The ability to model dynamic temporal relationship as well as the flexibility of scripting technique, MediaLisp can handle any complex situations concerning multimedia presentation. 27

29 8. Reference [1]. Shu-Ching Chen, Mei-Ling Shyu, and R. L. Kashyap, "Augmented Transition Network as a Semantic Model for Video Data," accepted for publication, International Journal of Networking and Information Systems, Special Issue on Video Data. [2]. Sheng-Tun Li, Shu-Ching Chen, and Mei-Ling Shyu, "A Snapshot Browsing Model for Distributed Surveillance Systems," accepted for publication, Journal of Applied Systems Studies, Special Issue on Distributed Multimedia Systems with Applications. [3]. Sheng-Tun Li, Shu-Ching Chen, and Mei-Ling Shyu, "A Live TV-Quality Distant learning Multimedia Presentation System for Education," the 34th Hawaii International Conference on System Sciences (HICSS-34), CD- ROM, IEEE Press, 9 pages, January 3-6, 2001, Maui, Hawaii. [4]. Sheng-Tun Li, Shu-Ching Chen, and Mei-Ling Shyu, "A Presentation Semantic Model for Asynchronous Distance Learning Paradigm," ACM Multimedia 2000 Conference, pp , October 30 - November 3, 2000, Los Angeles, California. [5]. Mei-Ling Shyu and Shu-Ching Chen, "A Bayesian Network-Based Expert Query System for a Distributed Database System," IEEE International Conference on Systems, Man, and Cybernetics, Nashville, Tennessee, USA, pp , October 8-11, [6]. Shu-Ching Chen, Mei-Ling Shyu, and Naphtali Rishe, "Modeling Interactive Multimedia Presentation Systems Using Augmented Transition Networks," First International Workshop on Intelligent Multimedia Computing and Networking (IMMCN'2000), pp , February 27-March 3, 2000, Atlantic City, NJ, U.S.A. [7]. Shu-Ching Chen, Srinivas Sista, Mei-Ling Shyu, and R. L. Kashyap, "Augmented Transition Networks as Video Browsing Models for Multimedia Databases and Multimedia Information Systems," 11th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'99), pp , November 9-11, 1999, Chicago, IL, U.S.A. [8]. Guy L. Steele, "Common Lisp the Language, 2nd Editio", Digital Press 1990 paperbound ISBN [9]. Sean Luke, "Java for Lisp Programmers",at AAAI99 (the 1999 conference of the American Association for Artificial Intelligence). [10]. Heiko Kirschke, "PERSISTENT LISP OBJECTS!", Version 2.09 of May 22, " [11]. (March 2001). 28