Checking Temporal Integrity of Interactive Multimedia Documents

Transcription

1 Checking Temporal Integrity of Interactive Multimedia Documents I. MIRBEL 1, B. PERNICI 2, T. SELLIS 3, S. TSERKEZOGLOU 4, M. VAZIRGIANNIS 4 1 Laboratoire I3S, CNRS-UNSA, Sophia Antipolis, 06560, France. mirbel@unice.fr 2 Politecnico di Milano, Milano, Italy. pernici@elet.polimi.it 3 Dept. of El. & Comp. Engineering, National Technical University of Athens, Hellas. timos@cs.ntua.gr 4 Dept of Informatics, Athens University of Economics & Business, Hellas. mvazirg@aueb.gr Abstract. When authoring multimedia scenarios, and in particular scenarios with user interaction, where the sequence and time of occurrence of interactions is not predefined, it is difficult to guarantee the consistency of the resulting scenarios. As a consequence, the execution of the scenario may result in unexpected behavior or inconsistent use of media. The present paper proposes a methodology for checking the temporal integrity of Interactive Multimedia Document (IMD) scenarios at authoring time at various levels. The IMD flow is mainly defined by the events occurring during the IMD session. Integrity checking consists of a set of discrete steps, during which we transform the scenario into temporal constraint networks representing the constraints linking the different possible events in the scenario. Temporal constraint verification techniques are applied to verify the integrity of the scenario, deriving a minimal network, showing possible temporal relationships between events given a set of constraints. Keywords. Multimedia presentation, Constraint networks, Temporal integrity 1

2 1. Introduction An Interactive Multimedia Document (IMD) involves a variety of individual multimedia objects presented according to the presentation sequence specified in an IMD scenario. During the presentation, multimedia objects that participate in the IMD are transformed, either spatially or temporally, in order to be presented according to author's requirements [Vazirgiannis 96] specified in the interactive multimedia document description. At authoring time, the author defines the spatial and temporal order of objects within the application context and the relationships among them. In addition, the way the user will interact with the application as well as the way the application will treat application or system events have to be defined. The related application domains are quite challenging and demanding, such as interactive TV, synthetic movies, animation etc. In such interactive documents we consider the following constituents: the Objects that participate in the document along with their spatial and temporal transformations (i.e. cropping, scaling) to align with the authors requirements. the Events, considered as the fundamental carriers of interaction in the context of an IMD, raised by user actions, by objects participating in the IMD or by the system. They may be simple (i.e. not decomposable in the IMD context) or complex and have attached a spatiotemporal signature (i.e. the space and the time they occur). Spatiotemporal composition: represents the spatial and temporal ordering of the participating objects in the IMD, i.e. when, and where each object is presented. Scenario: the integrated behavioral contents of the IMD, i.e. which events will trigger the spatiotemporal compositions defined above. Real world large IMDs can be very complex with respect to (i) the number of involved objects, (ii) transformations of the objects in the scope of an application, and (iii) relationships among them. It is obvious that in a complex IMD it may be very difficult, at authoring time, to keep track of all paths the application may follow, i.e. to decide if the document is temporally consistent. Interaction, with its inherent temporally indeterminate elements, may complicate the process for 2

3 temporal integrity checking for verifying the temporal consistency of the scenario. As temporally consistent we define a document that under any allowed interaction will not lead to runtime errors and moreover eventually lead to a state where all the participating objects end their presentation. In the present work, we propose a methodological approach for IMD temporal integrity checking focusing on temporal properties of the presentation. An interesting issue is the temporal consistency of the presentation, that provides the possibility to decide in advance, i.e. before the scenario execution, if the order of presentation actions is consistent according to a set of constraints. These constraints may be either generic (i.e., an idle object cannot be stopped ) or application specific (i.e., video clip A should not overlap with sound B or two sounds should not be active at the same time ). The scenario integrity issue is crucial when designing a complex scenario. Moreover the multitude of events and the related interactions that may occur in an IMD session may create a lot of potential IMD evolution paths which are difficult to follow at authoring time, considering all potential implications. Therefore there is the necessity to provide a formal and sound background for building tools to support authors in designing consistent scenarios and to verify the integrity of existing ones. The approach we elaborate on in the following sections is based on previous work of some of the authors for modeling interactive multimedia scenarios [Vazirgiannis and Boll B; Vazirgiannis et al. 98] which is briefly illustrated in section 2. In this approach, the scenario consists of a set of self-standing functional units (scenario tuples) that include: triggering events (start and stop events), presentation actions (in terms of spatiotemporal compositions of actions on multimedia objects) to be carried out in the context of the scenario tuple, and related synchronization events. In the rest of the paper we will interchangeably use the terms integrity and consistency referring to the correctness of the temporal aspects of an IMD scenario. The methodology for checking interactive multimedia document consistency presented in the paper has the following main characteristics: The methodology is based on the transformation of scenario specifications into networks representing the temporal constraints between multimedia objects and 3

4 those concerning their evolution. During the transformation, the effect of possible user interactions during the scenario execution has to be taken into account. The method is based on a set of rules for deriving constraint networks. Techniques from temporal reasoning [Brusoni et al. 97] are applied in this context to analyze the resulting networks and examine their consistency and potential problems. The structure of the paper is as follows. In Section 2, we present the scenario model that serves as the basis of our integrity checking methodology. In Section 3, we discuss properties of a consistent scenario with respect to temporal aspects. The integrity checking (Section 4) consists of a set of discrete steps which transform scenarios into temporal networks that correspond to the fundamental functional units of the scenario model mentioned above. In Section 5, we present a way to use the approach in order to evaluate queries and additional constraints that may be posed to the scenario structure. In Section 6, we discuss the methodology in terms of time complexity and potential optimizations. In Section 7, we present related work and compare it to our approach. 2. The Scenario Model In this section we present briefly the scenario model that serves as the basis of the temporal constraint checking scheme. A multimedia scenario is defined as the high level definition of a multimedia application functionality that covers the spatial and temporal synchronization of participating media objects and the handling of interaction between the user, the system and the application. In the present paper, we are going to exploit an existing model for IMDs [Vazirgiannis and Boll 97; Vazirgiannis et al. 98]. According to that model an IMD consists of the spatial and temporal composition of the participating media objects, the definition of interaction between the user and the IMD (external interaction), the interaction between internal IMD entities (internal interaction) and the specification of media synchronization in the temporal and spatial domains. The term scenario in the context of IMDs stands for the integrated behavioral contents of the IMD, i.e. what kind of events the IMD will consume and what actions will be triggered as a result. The scenario consists of a set of autonomous 4

5 functional units (scenario tuples) that include the triggering events (for starting and stopping the scenario tuple), the presentation actions to be carried out in the context for the scenario tuple, related synchronization events and possible constraints. More specifically, a scenario tuple has the form: (Start_event, Stop_event, Action_list, Synch_events) where: Start_event represents the event expression that triggers the execution of the actions described in the Action_list. Stop_event represents the event expression that terminates the execution of this tuple (i.e. it terminated the execution of the actions described in the Action_list before its expected termination). Action_list represents the list of synchronized media-presentation actions that takes place when this scenario tuple becomes activated. The expressions included in this attribute are in terms of compositions as described in the following and as in [Vazirgiannis et al. 98]. Synch_events is a pair of events generated at the beginning and at the end of the current tuple execution. These events may be used for synchronization purposes. Temporal Relationships Hereafter we present a set of Temporal Access Control (TAC) operators [Little and Ghafoor 93] that represent the temporal composition of media objects, together with the causality of temporal relationships among presentation intervals. For more details refer to [Vazirgiannis et al. 98]. These operators correspond to the well-known TAC actions: start (>), stop(!), pause ( ), resume ( >). Therefore: TAC_operation :== >! > For instance, assume a multimedia object A, then A> represents the start action of the multimedia instance, A! the stop, A the pause and A > the resume actions. We have to illustrate the events arising from the temporal state changes of a multimedia object. i.e. when the object A starts its presentations then the A> temporal event is raised. Special attention should be paid to the event generated when the object finishes its execution ( < ) and to distinguish this event from the TAC operation!. Therefore: t_event :== > < > 5

6 Following, we define a temporal composition scheme based on the one presented in [Vazirgiannis et al. 98], to represent the temporal composition of media objects in the context of an IMD scenario. Let time be the length of a temporal interval between two events or TAC operations. Therefore: temporal_composition :=, $ (object (t_event TAC_operation) [{time object TAC_operation}] where, is the spatiotemporal origin of an IMD session. Additionally the symbol $ represents the start of the tuple. For instance: the expression: A> 4 B! 0 C> conveys Start A, after 4 time units, stop B, and just after this start C. Finally, we define the natural duration d A of a multimedia object A as the temporal interval between the temporal events A> and A< where no other TAC operation related to A occurs in between. The above-defined set of operators is minimal. Event Modeling & Composition As mentioned before events are considered are carriers of interaction, and hence a cornerstone of the proposed IMD model. An event occurs due to an action, caused by an object (subject) that may be one or a group of presentation objects participating in the IMD. Each event occurs at a specific point in time and/or position to which we refer to as the event s spatiotemporal signature. In principle, the event may affect or could be related to one or to a group of media presentations, further called object of the event. In [Vazirgiannis and Boll 97] an event classification and composition model has been presented. For event composition we distinguish between algebraic composition (composition of events according to algebraic operators) and spatiotemporal composition (reflecting the spatial and temporal relationships between events). In the following, we briefly describe some of the composition operators defined in the scenario model exploited in the current work: e = OR(e 1,,e n ): This event occurs when at least one of the events e 1,,e n occurs. For instance we may be interested in the event e occurring when button A (e 1 ) or button B (e 2 ) is pressed. e = ANY(k,e 1,,e n ): This event e occurs when at least any k of the events e 1,,e n occur. The sequence of occurrence is irrelevant. For example, in an interactive 6

7 game a user proceeds to the next level when she/he is successful in two out of three tests that generate the corresponding events e 1,e 2,and e 3. e = SEQ(e 1,,e n ): This event occurs when all events e 1,,e n occur in the order appearing in the list. For example, in another interactive game the user proceeds to the next level when she/he succeeds in three tests causing the events e 1,e 2,and e 3 one after the other. e = TIMES (n,e 1 ): This event occurs when there are n consecutive occurrences of event e 1. This does imply that other events can occur in-between occurrences of e 1. e = Simultaneously(e 1,e 2 ): This event is raised when e 1, e 2 both occur and the time stamp of e 1 is equal to the corresponding signature of e Consistency of IMD Scenarios In the present section, we define the main notions of consistency in multimedia scenarios that are used in our approach. As stated in previous sections, we focus on defining at authoring time scenarios which are consistent from the point of view of the temporal sequence of actions on the multimedia objects, according to the characteristics of each object and the characteristic of the described scene. As it has been introduced in other efforts [Courtiat and De Oliveira 96; Layaida and Keramane 95] two categories of consistency are recognized: the qualitative and the quantitative one. The former is related to the validity of the combination of temporal relationships among media objects regardless of their duration, while the latter assumes the first and checks whether the specific objects duration cause inconsistency problems. A scenario is temporally consistent at authoring time if i. the temporal compositions of media objects in scenario tuples, as presented in the previous sections by the designer, are fulfilled all together, (ii) the sequences of TAC actions in the scenario are legal according to a set of rules that are further introduced. We introduce the concept of consistency at two levels: single media objects scenario level, where the internal representation of scenarios is based on temporal constraints. 7

8 3.1 Consistency of Single Media Objects A time-dependent media object may be in one of the following temporal states: active, idle, suspended. Moreover there is a set of actions that may be applied that change the object status (referred to as Temporal Access Control actions introduced in Section 2). When a TAC action is applied to an object its temporal state changes. For instance when an object is idle and the action > is applied to, it falls into the active state. All the allowed transitions between states and the corresponding actions appear in Fig. 1. It has to be stressed that the object s initial and final state is always the idle one. Active! Idle < >! > Suspended Figure 1: State transition diagram for time-dependent media object In a multimedia scenario there may be a set of actions that are applied to a media object due to scenario expressions. The execution of these actions must obey to some constraints. For instance, it is not possible to stop an object which is not active (i.e. in idle state). In the following table, we list the runtime constraints that emanate from the above diagram (i.e. which conditions must be fulfilled in order to be able to apply an operator): OPERATOR/EVENT NAME CONSTRAINT > Start The object must be in IDLE state! Stop The object must be in ACTIVE or SUSPENDED state Pause The object must be in ACTIVE state > Resume The object must be in SUSPENDED state < Natural end Occurs when the object finishes execution. Starts from ACTIVE state. Table 1: The TAC operators & events and the related constraints 8

9 In a multimedia scenario a TAC operator may be applied to a time-dependent object resulting in a change of state (e.g. from IDLE to ACTIVE). As the scenario evolves, another TAC operator may be applied to the same object resulting in another change of its state. Not all the sequences of TAC actions on media objects are legal: therefore, at authoring time, we can accept only the ones that are represented in the state transition diagram of Figure 1Fig. 1 Let us assume the expression: A op t A op, where A is an actor, op is one of the operators: >,!,, >, and t the time elapsed between the two operations. The first general constraint is that : t should be such that the time of execution of the second action is not before the start of the scenario execution. Moreover it is important to stress the fact that some combinations are not feasible ( illegal ) and, among the rest of the combinations, some are feasible under some constraints. An exhaustive combination of TAC operations and their features, at authoring time, appears in Table 2. In this table the sequence of actions and the time t have to respect the constraints imposed by state transition diagram of Fig. 1. An important clarification here refers to the fact that the consecutive actions appearing in each line of Table 2 refer to the same instance (i.e. presentation) of each object. This means that a second presentation of the same object is considered as another instance. To start again an object which is already active, a second instance of the object has to be started. This is possible in our model. The sequence start-start would lead to a cycle, which is not allowed in the proposed methodology. Moreover, we consider time as linear. A second start action on a specific object in the current execution plan would result in potential shift back in time. Therefore, concurrent play outs of the same object are allowed but considered as different objects. We also assume that t can also be a negative value. ACTION ACTION Legal Constraint 1 > > No 2 >! Yes 0<t<d 3 > Yes 0<t<d 4 > > No 5! > Yes -d<t<0 6!! No 9

10 7! Yes t <0 8! > Yes -r<t<0 9 > Yes -d<t<0 10! Yes t>0 11 No 12 > Yes t>0 13 > > No 14 >! Yes 0<t<r 15 > Yes t<0 16 > > No d: duration of object presentation r: remaining time of object presentation Table 2: The constraints on pairs of consecutive TAC actions on the same object Summarizing the above table, the sequences of action in lines: 1,4, 6, 11, 13 and 16 violate the state transition diagram of Fig. 1 and thus are considered as illegal. One could argue that, for instance, in line 1 a second > operator would not harm. This is true but it would imply that a second instance of the same object should be started, whereas in our approach consecutive application of operators are applied to the same instance. Thus we do not allow TAC actions that cannot change the temporal state of the object starting from the current one Scenarios represented as Networks of Temporal Constraints As a basis for our methodology for consistency checking and scenario analysis support, we adopt an internal representation of scenarios based on constraint networks. The derivation of constraint networks is a non trivial problem, in particular when considering different possible user interactions during the scenario execution. The methodology for deriving temporal constraint networks from scenario descriptions is the core of the present work and is described in the following sections. In the present section, we introduce the basic assumptions on the type of networks we use for our analysis. 10

11 We assume to derive from scenario descriptions a set of temporal constraint networks which represent the constraints between the different actions performed within a particular scenario execution, and representing also multimedia objects internal properties, such as for instance the maximum allowed duration for an object execution. Constraints derived from scenario tuples are represented internally as conjunctions of bounds of differences between variables representing events [Dechter et al. 91; Brusoni et al. 97]. Such differences represent the minimum and maximum possible temporal distances between events, as derived from the specification of actions in tuples and from possible concatenations of tuple executions. For instance, if event a is at least 10 units of time before event b, but b occurs no later than 50 time units after a, we write 10 b-a 50. We build networks (or network fragments), where nodes represent events in tuples (internal and external events), and arcs express the minimum and maximum possible distances between two nodes n 1, n 2 in the form b 1 n 2 - n 1 b 2. In such a network, a variable name can appear only once. For each network (built according to the rules illustrated in next section), the consistency of the conjunction of bounds of differences can be checked using temporal reasoning techniques such as the STP framework [Dechter et al. 91] in O(n 3 ) time, where n is the number of nodes in the network. The algorithm checking the consistency produces also the minimal network, representing all possible solutions, i.e., possible assignments to the variables satisfying the constraints. The minimal network contains, for each pair (n i, n j ) of variables, the maximal range [c,d] that includes all and only the values for n j - n i satisfying all the constraints in the network [Dechter et al. 91]. According to this approach a network is inconsistent if there are negative cycles, i.e. a cyclic path from a node to itself which results in negative bounds of differences. In the following, we adopt a graphical representation of the bounds of differences, using a graph where the nodes correspond to the network nodes, and arcs are labeled with the minimum and maximum distances between the events represented by the nodes. 11

12 In the following sections, we show how to build constraint networks, how to verify local properties of portions of the scenarios using reasoning on network fragments, and how to pose queries to the minimal networks to examine scenario properties during authoring. The issue of considering the complex problem of examining all possible scenarios executions is also considered in the methodology and solutions provided according to a defined set of hypothesis on the scenario being authored. 4. IMD Scenario Integrity Checking The temporal integrity of an IMD scenario may be considered at two levels: At scenario tuple level, we check if the sequence of events, as they are generated by the action list of the scenario tuple, is temporally consistent according to the criteria defined in previous sections. At scenario level, we examine the various execution paths that the IMD may follow according to the events generated. For each path, we verify its consistency. Thus we find the consistent execution paths. The consistency at scenario level is checked using the STP framework Dechter & al. 91]. The consistency checking methodology may be summarized in the following. The IMD scenario as defined in the previous section consists of a set of scenario tuples. Each of them is transformed into a constraint network fragment (cnf). A cnf represents the temporal constraints between the start/stop events of the tuple and the TAC actions/events in the action part of the scenario tuple. Due to the interactive nature of the IMD scenarios there are a lot of alternative sequences of events that will start/stop tuples. Each of these sequences (paths) is an execution plan for the IMD scenario. Thus after each event we have a different set of tuples that are active (i.e. their action part is being executed). Each such set of active tuples defines a state for the IMD scenario. The transition between states is caused by occurring events and defines an execution plan. There are roughly as many execution plans as many as paths. In the case of pre-orchestrated scenarios (i.e. no interaction is assumed), only one execution plan is possible, while in interactive scenarios several alternatives for execution have to be considered. Each execution 12

13 plan defines a sequence of tuples, which is mapped in a composition of the corresponding cnfs. The result is a composite constraint network (cn). The temporal consistency of the IMD scenario is carried out in the following consecutive stages: Qualitative checks, in which we verify if the sequences of TAC actions are legal according to the state diagram in Table 1. Quantitative checks, in which we verify whether the temporal intervals between legal sequences of TAC actions are not creating inconsistencies, according to the criteria set in Table 2. Temporal consistency check, that verifies the global temporal consistency of the scenario. This verification is carried out exploiting the framework presented in [Dechter et al. 91]. Through the stages of the methodology, we distinguish warnings from errors. An error is detected when a certain runtime error occurs as a result of an action expression. Then the scenario cannot be further executed. For instance, the stop of an object which is not anymore running lead to an execution error. As regards warnings we distinguish two cases. In the first one, there may be cases in which there is an action expression that locally causes an error that may be fixed when combining it with other actions from other tuples in the process of constraint networks building. These cases raise warnings. Also, paths leading to the undefined state generate to potential errors, and should be signaled by a warning.. The second case in which warning is raised is when the scenario may not run properly or may lead to an execution error. For instance, a scenario tuple which never ends is considered only as a warning because it may be required by the designer of the scenario. The methodology consists of four steps (see Fig. 2): Step 1: Network Fragment Design: this step aims at translating each tuple of the scenario in one or several constraint network fragments (cnfs), expressing constraints on the tuple events (Start Event, Stop Event, Synch Event) and the TAC events (generated by the TAC actions : >, <,, >). 13

14 IMD scenario Step 1 Transformation Constraint Network Fragments Preliminary consistency h k Step 2 Transformation Preliminary consistency check Execution plans path Composition of cnfs Constraint networks cn Step 3 Preliminary consistency h k Step 4 Consistency check Valid paths Figure 2: The architecture of the IMD scenario consistency checking methodology Step 2: Alternative Execution Plans: this task is carried out independently of step 1. The aim here is to analyze the event sequences in the scenario, in order to define the different possible execution plans (also called paths in the following). The global states that can be reached in the scenario are identified, and corresponding transitions between states are derived from the tuple execution. The result of this step is the execution plans diagram. Step 3: Composition of Network Fragments: in this step the aim is the composition of cnfs built in step 1 with regards to the different paths elaborated in the second step to allow one to check the consistency in each of the possible execution plans. The composition is carried out by building constraint networks (cns) representing the temporal constraints of a scenario session following a path. Step 4: Temporal Consistency Check: this last step aims at checking the consistency of the cns derived in Step 3 for each execution plan. Then, the results of the consistency check allow the author (i) to verify the correctness of the execution plans and (ii) to submit queries related to the scenario to verify further properties. 14

15 In the following sections, we elaborate on the steps of the methodology and we illustrate them through a running example. 4.1 Step 1: Network Fragment Design This first step of the methodology aims at translating the tuples of the scenario into temporal networks, called constraint network fragments (cnfs). These cnfs represent the temporal constraints of each tuple through bounds of differences in order to check the temporal consistency of the scenario. A preliminary check of consistency will also be discussed at the end of this step Building the cnfs The cnfs represent the constraints among events (tuple events and TAC events) in the tuple. As the check is based on conjunctions of bounds of differences, if there are alternatives in the tuple (for instance complex start/stop events, such as e 1 OR e 2), different cnfs have to be created, in order to check the consistency on each possible alternative. Let NT=<N,A> be the cnf, where: N is the set of the nodes of the network. A node represents a start/stop event or a TAC event. TAC events are taken from the Action list. A is the set of labeled arcs between the nodes of the network. An arc represents a temporal constraint between two events. It can, for example, link the TAC start event and the TAC stop event of a time dependent object (for instance a video clip). On each arc there is a label of the form (x, y) where x,y are the minimum and the maximum temporal duration values between the connected nodes. For instance, the duration associated with an arc linking the start and stop events of a time dependent object is 0-ObjectDuration. Thus for each tuple we create a cnf starting with the start event and continuing with the TAC events. After that we have to connect the stop events to the actions that have to be stopped. As mentioned before the start/stop events may be composite [Vazirgiannis and Boll 97]. In this case they have to be translated into cnfs according to the following mappings: OR(e 1,..., e n ): In this case, n cnfs have to be created, each of them representing one alternative of the disjunction. 15

16 ANY(k, e 1,..., e n ): Each possible subset of k events from the n given in the clause have to be represented in different cnfs. There are different cnfs resulting. Durations on arcs are unknown, so labels are of the form (0-). SEQ(e 1,..., e n ): The sequence of events has to be represented by a path from e 1 to e n, with unknown durations (0-) on their arcs. TIMES(n, e 1 ): In this case, n occurrences of e 1 (with different names for each occurrence to distinguish them) have to be defined, all linked to a same node representing the composite event. Durations on the arcs are unknown (0-). SIMULTANEOUSLY(e 1,e 2 ): In this case, e 1 and e 2 are starting points for arcs ending on a common node representing the composed event. Arcs have 0-0 durations. n k The mapping of the complex events to cnfs are summarized in Fig. 3, where the tuple is represented as (Start event, Stop event, Actions List, Synchronization events) and different composite starting events are considered. Tuple 1, in which action a is executed, is triggered by event e 4 or e 5 ; Tuple 2 is triggered by two of the events e 1, e 2, e 3 ; Tuple 3 is triggered by the event sequence e 1, e 2 ; Tuple 4 is triggered by three consecutive occurrences of e 1 ; and Tuple 5 is triggered by simultaneous occurrence of e 1 and e 2. Previously, we introduced the symbols θ and $. Their mapping into cnfs is performed as follows: θ: This symbol refers to the temporal origin of the scenario. Usually, there is a StartApp event to indicate the start of the scenario. Therefore this symbol θ has to be replaced by an arc starting from the StartApp event. If the tuple contains the StartApp event, the arc can be drawn during this step of the process, otherwise it will be done later during the composition step. $: This symbol refers to the start of the tuple. Therefore it has to be replaced by an arc starting from the Start Event of the tuple. 16

17 Tuple 1 (OR(e 1,e 2 ), -, a, -) e 4 a Fragment 1 a Fragment 2 e 5 Tuple 2 (ANY(2, e 1, e 2, e 3 ), -, a, - e 1 0- a e 2 0- Fragment 1 Fragment 2 e a e 3 e 2 0- Fragment 3 0- a e 3 Tuple 3 Tuple 4 e1 1 e1 2 (SEQ(e 1, e 2 ), -, a, -) 0- e 1 e 2 Figure 3: Mapping of the composite events to cnfs e 2 e1 3 Fragment 1 (TIMES(3,e 1 ), -, a, -) 0-0- a 0- Fragment 1 Tuple 5 (SIMULTANEOUSLY(e 1, e 2 ), -, a, -) e 1 a a Fragment 1 Appendix A presents a sample IMD scenario. The scenario represents a simple application related to ancient Olympic Games. We use this example to illustrate the four steps of the methodology we propose. An example of one of the constraint networks fragments corresponding to the Stage1 tuple of the Olympic Games example is given in Fig. 4. The tuple follows: (StartApp, ANY(1; _DoubleClick; _KeyEsc; _IntroStop), TITLE> 0 INTRO> 3 IMG1>). As three alternatives are provided as stop events (_IntroStop, _DoubleClick or _KeyEsc) three cnfs have to be created. In Fig. 4 we present the cnf built in the case that _IntroStop stops the execution of Stage1. StartApp TITLE> 0- TITLE< INTRO> 3-3 IMG1> IMG1< INTRO< Figure 4: Sample cnf In the network depicted in Fig. 4, the constraints derived from the action list of the Stage1 tuple are represented according to the graphical representation presented in previous section. The network represents also the constraints on the durations of time-dependent actors. For instance INTRO audio clip lasts 47 seconds (see actor specification in Appendix A). This temporal constraint is shown by the label 0-47 on the corresponding arc between INTRO start (INTRO>) and INTRO end 17

18 (INTRO<) nodes. On the other hand, images do not have intrinsic duration. For instance we use the constraint 0- to indicate that there is no temporal constraint between the start and the end of the presentation of IMG1 media object Mapping into Bounds of Differences The mapping from each cnf to a set of bounds of differences consists in translating each arc into a bound of differences. For instance the arc between INTRO> and INTRO< nodes in Fig. 4 will be translated into the following set of inequalities: 0 (INTRO<) (INTRO>) 47 where 47 is the duration of INTRO. On the other hand, the duration on the arc between TAC TITLE> and event TITLE< nodes has no maximum bound, as it is a text object and thus it is not time dependent. This arc is translated into the following set of inequalities: 0 (TITLE<) - (TITLE>) Preliminary Consistency Check It is already possible to do a preliminary consistency check on cnfs for two reasons: to detect as soon as possible inconsistencies: some are already present in the cnf and it is useless to compose them together following a path (the global network will be inconsistent). to better locate the inconsistencies: it is easier to find an inconsistency and to correct it at the cnf level than on a whole network representing a whole scenario session (therefore bigger and more complex). This preliminary check verifies the constraints presented in Table 1 and Table 2. We distinguish two cases: qualitative constraints: the consistency of the action sequences is checked (Table1). For instance sequences of the form : A!tA! may cause an error, if another event from another tuple (such as A>) does not rectify this error sequence. Therefore, if this situation occurs, it is detected as a warning while building the cnf. Another situation that may cause errors is when the same event appears in the start event of the tuple as well as in its action list. Such a tuple is: (A>,_, A>5A!,_). 18

19 The result is that there will be an endless loop since the action A> starts the tuple recursively: a warning is raised. quantitative constraints: legal sequences of TAC actions may cause errors if the temporal intervals between them exceed specific limits. This set of constraints on bounds of differences (bods) has to be checked for each of the objects involved in the tuple. If they are not satisfied, a warning is raised. The constraints appear in Table 2 and are translated as follows: If bod = x (A! - A> ) y, then x 0, y d (lines 2 & 5 of Table 2) If bod= x (A - A>) y, then x 0, y d (lines 3 & 9 of Table 2) If bod= x (A! - A ) y, then x 0 (lines 7 & 10 of Table 2) If bod= x (A! - A >) y, then x 0, y r (lines 14 & 8 of Table 2) If bod= x (A > - A ) y, then x 0 (lines 12 & 15 of Table 2) Such kinds of inconsistencies are raised as warnings, because they may not appear anymore when the tuple under consideration is combined with other tuples in the scenario (for a complete check including these constraints see Section 5). This check is performed on the minimal network obtained through temporal reasoning, from the constraint networks (i.e. their corresponding sets of bounds of differences). It shows the possible temporal relationships between represented events which satisfy the constraints. For instance, the minimal network corresponding to the fragment shown in Fig. 4 constraints the duration of TITLE actor to be the same as INTRO video clip. At the end of this step, each tuple of the scenario is translated into one or more cnfs, and thus a preliminary consistency check has been carried out on each of them 4.2 Step 2: Alternative Execution Plans The second step of the methodology, which is independent (i.e. it does not need to follow) of the first one, has the goal of identifying possible event sequences resulting in different orders in the tuple execution. The sequence of execution of 19

20 the different tuples of the scenario is driven by events. Depending on the sequence of occurring events, the different tuples of the scenario start and by this way define global states in the scenario. The goal of this step is to identify these different global states, and the events (transitions) leading from one state to another. The events belong to the ones appearing in the scenario. The result of this step is the set of all the possible execution plans (paths) of the scenario. For this purpose, an automaton is created, representing the different states of the scenario while the events allow transitions between states. Then, with regards to this automaton, we can already carry out some preliminary checks (that result in potential errors, indicated as warnings), which are discussed after the presentation of the way to build the automaton Automaton Building A state is defined as a set of executing (active) scenario tuples. A state change occurs when a new tuple is started. All the start events (Start event, Synch events) have to appear on at least one transition. A transition is caused by at least one of the start events. The resulting state-transition diagram must at least contain the Idle state (no tuples are running), from which all the paths start, through the StartApp event. If there are stop events that do not lead to a new state, then they have to be written on a transition ending in the pre-defined state called undefined state. In Fig. 5, the execution plans diagram for the example presented in Appendix A is depicted. Each path starting from the Idle state ending to the Idle state represents an execution plan of the scenario. From Fig. 5, we deduce two alternative paths : P1: StartApp - _DoubleClick/_KeyEsc. In this path the presentation starts with the occurrence of StartApp event, which leads the scenario to State 1. The next event is _DoubleClick or _KeyEsc, which leads the scenario to Undefined state. P2: StartApp - _IntroStop - _Timer10 - e1/_apptimerevent. StartApp event is raised first, leading the application to State 1, followed by _IntroStop, leading the scenario to State 2. Then _Timer10 event starts Stage3. When user presses EXITBTN, _ExitEvent occurs stopping Stage2 and triggering e1 synchronization event. This leads to State4. Alternatively, if the user does not press EXITBTN until 20

21 time 01 minute and 30 seconds, _AppTimerEvent occurs also leading anyway the application to State 4. Then the ExitApplication event is raised and the application goes back to the Idle state. ExitApplication Idle StartApp State 1 _DoubleClick\_KeyEsc _IntroStop Undefined State 2 State 1: State 2: State 3: State 4: Stage1 running ApplTimer running Stage2 running ApplTimer running Stage2 running Stage3 running ApplTimer running Stage3 running ExitTuple running ApplTimer running _Timer10 State 3 State 4 e1/_apptimerevent Figure 5: Sample execution plans diagram Preliminary Consistency Check At this step of the process, consistency checks can already be done. The forthcoming step of the methodology consists in composing the sets of bounds of differences defined in step 1, with regards to the different paths elaborated through the second step of the methodology, to know if such paths are consistent or not. 4.3 Step 3: Composition of cnfs The goal of this step is to group the different cnfs with regard to each path, to check its consistency. Composition is done in three steps: 21

22 Merging the sets of bounds of differences corresponding to the different fragments under consideration, Adding additional information if necessary, Adding constraints to express the order in the occurrence of events. The composition of different cnfs to obtain a cn is done by following a path (an execution plan) of the scenario. Therefore, for each path deduced from step 2, we compose the cnfs representing the tuples involved in the path, to get the cn corresponding to a whole scenario execution plan (path) Building cnfs When there are alternative cnfs corresponding to a tuple, each of the alternatives has to be composed with the cnfs representing the other connected tuples. Each possibility of composition between the alternative cnfs of each tuple of the scenario under consideration leads to a constraint network (cn): we must take into account the various instantiations of the (event_in, event_out) pairs, where event_in is an event leading to the state in which this tuple is active, and event_out is an event leading out of that state. In the sequel, we present the composition of the cnfs corresponding to path P2 of the sample scenario. All the tuples are involved in the path. We must find which cnfs should be involved in the cn representing path P2. There is only one cnf corresponding to each of the tuples Stage2 and Stage3, according to the rules presented in section On the other hand, there are three different cnfs corresponding to tuple Stage1, one for each alternative stop event. From the definition of path P2, Stage1 is stopped by the event _IntroStop. This means that the cnf having _IntroStop as stop event (depicted in Fig. 4) is involved in the composition. The ExitTuple is mapped into two cnfs, one for each of the two different start events (e1 and _AppTimerEvent). Both of them are involved in the composition. Hence, two alternative cns represent the composition of the cnfs involved in path P2. Fig. 6 depicts the first cn, in which we included the cnf having e1 as start event for tuple ExitTuple. 22

23 The tuples have been linked together in the following way: Tuple Stage1 is stopped by the _IntroStop event, which also starts tuple Stage2. Event _Timer10 starts tuple Stage3. The sequence of _Timer10 and _ExitEvent events stops tuple Stage2 and generates synchronization e1 event which starts the last tuple of the path, ExitTuple. Then, according to step ii, arcs have to be added, for each cn, between events generated by TAC actions dealing with the same object. If the object is timedependent, the arc duration has to fulfill the rules presented in Table 2. If the object is not time-dependent, the maximal bound of the arc duration has to be unbound. In Fig. 7, three additional arcs have been drawn (shown with slashed bold lines). The three arcs connect start and stop actions performed on YMNOS1, YMNOS2 and KAVSOUND media objects. The duration on the first two arcs (labeled as arc 1 and arc 2) has unknown maximum bound, while the arc representing the constraint between start and stop actions performed on KAVSOUND actor (arc 3) is labeled by duration Then, the events starting the different tuples have to be linked together through arcs indicating their sequence of occurrence (following the path under consideration). Since the occurrence time of interactions is not predefined, the duration on these arcs have unknown maximum bound (0-). In Fig. 7, two additional arcs have been drawn (shown with dotted bold lines). The first arc (labeled as arc 4) connects start action on actor EXITBTN with _ExitEvent. The second arc (arc 5) connects start action on timer _TIMER1 with event _Timer10. The duration on the first arc has unknown maximum bound while the second arc is labeled by duration 10-10, indicating that the event _Timer10 will occur 10 seconds after the initiation of timer TIMER1. 23

24 Stage1 StartApp TITLE> 0- TITLE< INTRO> 3-3 IMG1> IMG1< INTRO< Stage2 KAVALAR> 0-20 KAVALAR< YMNOS1> TIMER1> EXITBTN> YMNOS1< TIMER1< EXITBTN< _ExitEvent 0- _Timer10 ExitTuple e1 KAVSOUND! EXITBTN>2 KAVSOUND> Stage3 EXITBTN<2 KAVSOUND< YMNOS2! ExitApplication YMNOS1! 6-6 YMNOS2> 0- YMNOS2< Figure 6: Sample cns (part 1) 24

25 StartApp TITLE> 0- TITLE< Stage1 INTRO> 3-3 IMG1> IMG1< INTRO< Stage2 KAVALAR> 0-20 KAVALAR< YMNOS1> TIMER1> EXITBTN> YMNOS1< TIMER1< EXITBTN< _ExitEvent 0- _Timer10 Arc 5 Arc ExitTuple e1 EXITBTN>2 0- KAVSOUND! 0-30 KAVSOUND> 0-30 Arc 3 0- YMNOS2! YMNOS1! 0- Arc ExitApplication YMNOS2> 0- Arc 2 Stage3 EXITBTN<2 KAVSOUND< YMNOS2< Figure 7: Sample cns (part 2) Preliminary Consistency Check A tuple is considered as terminated either (i) after the stop event occurs (if it exists), (ii) when all the objects presented in this tuple are in idle state, having finished their presentation as defined in the action part. The objects which are not time-dependent stop only if explicitly required through a stop event. Therefore, at this state of the process, a first consistency check can be done with regards to the 25

26 ending tuples: a warning is raised when an object does not explicitly finish in a given path. Moreover, cns are evaluated with regards to the set of constraints representing the allowed sequences of TAC actions for a given object as it has already been explained at the tuple level in step 1. For each actor A, A and A > cannot appear if A> is not present: one cannot pause or resume an object which has not been started. Similarly, A > can not appear without A. These constraints are fulfilled through a set of constraints for each object involved in the cn. Some of these are: x, y such that x (A! A>) y x, y such that x (A A>) y x, y such that x (A > A>) y x, y such that x (A A >) y where x, y are positive values for bod as defined above. These checks have to respect also the state transition diagram of the object temporal state (Fig. 1). If such constraints are not fulfilled, an error is raised. Although these constraints have been checked at the tuple level (cnf), they have to be evaluated again at the scenario level (cn), since it is not possible to check them fully at the tuple level, as a given object can be, for example, started in a tuple and stopped in another one. Therefore, such inconsistencies were detected as warnings at tuple level and are detected as errors at scenario (path) level. It is important to note that the satisfaction of the previous constraints is not sufficient but necessary to insure the consistency of the scenario. At this point of the methodology, we have different sets of bounds of differences, each of them corresponding to an alternative in a possible path of the scenario. Preliminary consistency checks have been performed at the cnf and cn level. The global consistency check can now be performed. 26

27 4.4 Step 4: Temporal Consistency Check The consistency check is performed according to the STP framework [Dechter et al. 91] for each of the cns derived by the cnf composition step described above, which takes place for each alternative path. During the consistency check, for each cn, the corresponding minimal network [Dechter et al. 91] is derived. From the consistency check, we can find paths considered as consistent (without errors) or inconsistent. Inconsistent paths have to be corrected to avoid their occurrence at runtime. The consistency check provides also constraints on unbound variables (unknown durations on arcs). It can for example derive the maximum duration necessary between two events to assume the consistency of the scenario for a given path. Therefore, the scenario can then be enriched with this set of new constraints in order to improve its robustness. In the example under consideration, we built four cns. We presented the two alternative cns corresponding to path P2 in the previous section. The remaining two cns represent the temporal constraints for path P1, which leads to the undefined state as shown in section 4.2. After performing the consistency check to the first of the two cns for path P2 (presented in Fig. 7) we conclude that the network has no negative cycles (a cycle composed of nodes with arcs adding up to negative bounds). This means that the path is consistent [Dechter & al. 91]. Table 3 presents a portion of the minimal network that step 4 produces for the cn presented in Fig. 7. For each pair of events e i and e j, the cell MN[i, j] in Table 3 contains the minimum and the maximum time distance between the occurrences of the two events. For example event _ExitEvent must occur more than 2 seconds and less than 30 seconds after the occurrence of the event _Timer10 (the cell MN[4, 2] in Table 3 contains the value [2, 30]). As we can see there are no negative cycles: Every diagonal cell (MN[i, i]) contains the value [0, 0]. 27

28 Timer1> Timer10 _e1 _ExitEvent KAVSOUND! YMNOS2! Timer1> [0, 0] [10, 10] [12, 40] [12, 40] [12, 40] [12, 40] _Timer10 [-10, -10] [0, 0] [2, 30] [2, 30] [2, 30] [2, 30] e1 [-40, 12] [-30, -2] [0, 0] [0, 0] [0, 0] [0, 0] _ExitEvent [-40, -12] [-30, -2] [0, 0] [0, 0] [0, 0] [0, 0] KAVSOUND! [-40, -12] [-30, -2] [0, 0] [0, 0] [0, 0] [0, 0] YMNOS2! [-40, -12] [-30, -2] [0, 0] [0, 0] [0, 0] [0, 0] Table 3: A portion of a consistent Minimal Network (MN) The second constraint network (in which ExitTuple starts with _AppTimerEvent) corresponding to path P2 produces a network with negative cycles. For instance, the diagonal cell for the KAVSOUND! event has the value [-3, -3], which implies that the sounds stops before it starts, thus the corresponding constraint network is found to be inconsistent [Dechter et al. 91] Advanced features In this section, we provide some advanced features of the methodology presented previously. First, we address the issue of partial execution of paths in scenarios and we discuss the problem of operators appearing several times in the tuples of the scenario. Then, we explain how to take into account complex events using the composition operators S_CON, IN, NOT while checking the consistency of scenarios Partial execution of paths In an IMD session, we distinguish events which occur definitely (for instance raised by time-dependent media), from events that occur conditionally (e.g. raised by user-interaction). If a path includes events from the latter category, the path has also to be checked for consistency in the case this kind of event does not occur. In addition, when a stop event is used in a tuple and when this stop event is raised by user interaction, its occurring time is unknown. It may be wrong to directly link it to a TAC stop event, especially if the stopped object is a time-dependent 28

29 one (it has to be stopped only if it is still in active state). Depending on the occurrence time the stop event, objects could be idle or active and therefore could be stopped or not. Therefore, an additional arc and an additional node between the TAC stop event and the arc starting from the stop event have to be provided, with a variable duration depending on the time the stop event is raised. As the network is different with regards to the time of occurrence of the stop event, the duration between the start of the tuple and the stop event (the duration of the tuple) is also represented through a variable called "d". In Fig. 8, depicting the second cnf for Stage1, an additional node has been created in-between the stop event of the tuple (_KeyEsc) and the TAC end event of INTRO audio clip (INTRO<). The duration of the tuple is represented by the arc connecting StartApp and _KeyEsc events, labeled by the variable d. (d) StartApp TITLE> 0- TITLE< INTRO> 3-3 IMG1> INTRO< IMG1< 0- _KeyEsc Figure 8: The second cnf for Stage1 for d 3 Depending on the occurrence time of the stop event (i.e. the duration d of the tuple) media objects may be in Active or in Idle state and should or should not be stopped. To represent consistently the temporal constraints of the cnf, a network has to be derived for each situation. Each situation corresponds to a range of variable d. For instance, assume the action part of Stage1: TITLE> 0 INTRO> 3 IMG1>. It produces two alternatives depending on the value of d. In the former the value of d is between 0 and 3 seconds (i.e. IMG1 is not yet shown), while in the latter situation d is greater than 3 seconds. In both situations INTRO audio clip is still running when _KeyEsc event is raised. We have already presented the cnf for the second alternative (Fig. 8). Fig. 9 shows the cnf when d is varying between 0 and 3 seconds. As we can see, IMG1 actor has not yet been presented and therefore should not be stopped. 29

30 (d) StartApp TITLE> 0- TITLE< INTRO> 0-47 INTRO< 0- _KeyEsc Figure 9: The second cnf for Stage1 for 0 d 3 The ranges which are deduced from the different possible situations may be numerous. Therefore, if the author knows about the range in which the stop event is reached, he/she can give meaningful times for the cnf execution to guide the consistency process and avoid derivation of all the possible networks, in an exhaustive search for constraint violations Duplicate event names We might have problems with duplicate event names when merging cnfs, if, for instance, a video is started several times in the scenario. It is not allowed by the adopted methodology to use more than once a node name in the network. In order to tackle this problem we propose: Renaming of action events. Let us consider a specific state of the execution plan diagram corresponding to the scenario under consideration. If identical names are used among the objects of tuples involved in the state under consideration, it means they are working on the same objects. Therefore, no renaming has to be performed. On the contrary, if tuples belonging to different states of the scenario refer to objects through identical names, it is a problem, because they are not referring to the same objects. Therefore, a renaming has to be performed. Renaming of start and stop events. Start events provide links between the different states of the scenario in the execution plan diagram. Therefore, if events with identical names correspond to stop and start events of tuples belonging to states that follow each other, they have to keep their common names. On the contrary, if identical names refer to events which do not belong to tuples belonging to states that follow each other, a renaming has to performed. 30

31 As all the renaming situations can not be detected and treated in an automatic way, the author s assistance is required through this renaming procedure Complex composition operators The event composition operators S-CON, IN and NOT [Vazirgiannis and Boll 97] need special mappings into constraint networks in order to integrate them in the integrity check. The mappings of these events follow Mapping of complex events into cnfs The expression S_CON(e 1,, e n ) represents consecutive events. So, the sequence of events has to be represented by a path from e 1 to e n, with unknown duration (0- ) on their arcs (See Fig. 10). To map the expression NOT(e, e min, e max ), one needs to represent the boundaries e min and e max of the interval under consideration, as it is illustrated in Fig. 10. To map a IN(e, e min, e max ), one needs to deal with the boundaries e min and e max of the interval under consideration, as it is illustrated in Fig. 10. S_CON(e 1,, e n+1 ) x 1 y 1 x 2 y 2 x n y n e 1 e 2 e n+1 NOT(e, e min, e max ) x y e min e max IN(e, e min, e max ) z - t e min e 2 x y u - v e max Figure 10: Complex event modeling 31

32 Mapping complex events into bounds of differences. The mapping for the complex operators are the following. S_CON(e 1,, e n ) Let us consider the network fragment of Fig. 10. In the minimal network, each of its arcs corresponds to bounds of differences of the form: x n (e n+1 e n ) y n, x n+1 (e n+2 e n+1 ) y n+1, x m-1 (e m e m-1 ) y m-1, S_CON deals with a sequence of strictly consecutive events, therefore we consider the sum of the bounds of differences involving the pairs of events of S_CON (e 1 and e 2, e 2 and e 3, ). As for the boundaries of the time interval delimited by e n and e m (i.e., the sum) are the minimum value of the bounds involved in the sum; it is represented by X', while the maximal value of the bounds involved in the sum is represented by Y'. X e m e n Y with X =min(x n,x n+1,, x m ) and Y =max(y n, y n+1,, y m ). NOT(e, e min, e max ) To be able to represent the semantics of this operator, we represent the boundaries e min and e max of the interval under consideration, as it is illustrated in Fig. 10. Then, the corresponding bound of differences is: x e max e min y 32

33 IN(e, e min, e max ) To represent the semantics of this operator, we also represent the bounds e min and e max of the interval under consideration, as it is illustrated in Fig. 10. Then, the corresponding bounds of differences are the following: x e max e min y, u e max e v, z e e min t Preliminary Consistency Check S_CON, NOT and IN need additional constraints, not expressed by cnfs, to ensure they have been correctly represented: S_CON needs additional constraints to ensure the events are consecutive; NOT needs additional constraints to ensure no event e occurs in between e min and e max ; IN needs additional constraints to insure e occurs between e min and e max. In the following, for each of the three composite events, we provide a set of additional constraints, which are added to the cnfs, and therefore to the cns before being checked. Such additional constraints are part of the methodology and are added to the cns allowing thus correct check of scenarios. Fig. 11 illustrates the additional constraints associated with operators S_CON, NOT and IN. The possible time stamps (time ranges) for e n, e m, e min and e max are shown in between the dotted lines. The arrows indicate the possible time stamps for e, if it exists. S_CON(e 1,, e n ) Insuring that the event e does not occur between the occurrences of e n and e m means that the interval delimited by e and e n (or e and e m ) has to be longer than the interval delimited by e n and e m. To be sure it is longer, the minimum boundary of the interval delimited by e and e n (or e and e m ) has to be greater than the maximum boundary of the interval delimited by e n and e m. Therefore, to be sure there is no event e between e n and e m, as we deal with consecutive sequences of events, the following constraint has to be fulfilled: 33

34 e (not in S_CON) such that z e e n t or z e m e t then z < where Y' represents the maximal value of the bounds involved in the sum, as presented above. In each path under consideration (corresponding to a cn), each event appearing in the cn and not in S-CON has to be checked with regards to the previous constraint. NOT(e, e min, e max ) As it has been explained for the operator S_CON, in order to ensure that e is not present in the interval bounded by e min and e max, the following constraint must hold: e such that z e - e min t or z e max - e t then z > y In each path under consideration (i.e. corresponding to a cn), each event e appearing in a NOT(e, emin, emax) statement has to be checked with regards to the previous formula. Of course, if a NOT(e, emin, emax) statement on a given event e is used several times in a scenario, it will be translated in the corresponding cn into different NOT(e,emin, emax) statements, each of them dealing with an instance of the event. And all the NOT(e, emin, emax) statements will be checked. IN(e, e min, e max ) In order to ensure that e is in the interval bounded by e min and e max the following must hold: t y, z x, v y, u x. By using these mappings while building cnfs, and using these additional constraints while checking the consistency of the cnfs and cns, the methodology supports the full modeling language presented in Section 2. 34

35 S_CON NOT IN e e n e m e e e min e max e e min e e max time time Figure 11: Constraints on complex composite events time The advanced features presented allow to better exploit the richness of the model of IMD scenario. 5. Further checks on IMD scenarios based on queries As mentioned already in previous sections, for interactive scenarios verification, the computation of minimal networks does not solve entirely the problem of verifying a scenario, since some additional constraints should be checked on all produced networks, or in some subsets of the generated networks. In addition, the computation of the consistency of the minimal networks is useful to verify if a network is consistent, but does not give indications about where errors occur, in case the network is inconsistent. For these reasons, additional support must be provided to the designer, to help him in finding errors and in performing further verifications of the scenario. In addition, the computation of minimal networks for the different paths can also be the basis for further checks performed on the scenario directly by the author. He/she can check whether some of the desired properties, such as specific requirements of the author, (e.g., the fact that two given videos should not be running at the same time) are fulfilled. In the approach proposed in the present paper, further checks on the scenarios may be performed posing queries on the scenarios. These queries can be answered by the system based on the results of the computations of minimal networks. Queries may be classified into the following categories: 35

36 temporal queries: to ask about temporal properties of the scenario, such as its minimal and maximal duration, or the possible occurrence time of a given event; conditional queries: to investigate about properties of the scenario which are linked to the occurrence of given events; this type of queries is particularly important since it allows the design to check scenarios according to the assertions which are defined in association with some constraint network construction; layout queries: to assess temporal properties of objects at given instants of time. These queries can be posed separately or in conjunction. For instance, two queries may represent two properties P1 and P2, where for instance P1 requires that sounds do not overlap and P2 requires that Video V1 precedes video V2. The author may check a possible path of the scenario, to see if these two properties may be verified separately. In this case, if the result is ok, the author knows that one of the two properties may be verified in any possible execution of the scenario. This result does not imply that both properties may be verified at the same time, since one of the two can be incompatible with the other one. If the author wants to verify if a given property P1 is compatible with P2, she must verify if either property is necessarily verified in any possible execution, or verify the scenario with a query which is the conjunction of the two queries for the two properties. In general, the following possible modalities of posing queries are possible: posing queries separately on a single cn: the modality can be that of asking whether the result necessarily holds, i.e., all possible executions of the network imply that the property is satisfied, or that it possibly holds, i.e., that it is possible to perform an execution in which the property is verified, but it does not necessarily occurs. composed queries: simple queries, i.e., of the types defined in the following section, can be composed in a conjunctive form, to query scenarios about properties which hold at the same time. 36

37 queries on sets of cn: queries can be posed on a single cn, or the author can ask for the results of a given query on a set of cns; in case a query is posed on a set of scenarios, the author can ask whether the result of the query necessarily holds, i.e., it holds necessarily in each considered cn, or whether it holds possibly, i.e., it possibly holds on at least one of the considered cns. Combining the modalities of posing queries illustrated above, the author may investigate about properties held by the scenario been authored. On the other hand, to identify errors in inconsistent scenarios, the author can also base its activity on queries on related cns, e.g., cns that are related to the one which is inconsistent, such as, for example, those resulting from partial execution paths of the scenarios, or sets of alternative scenarios to the one being investigated. We discuss now each type of simple query which may be posed to investigate scenarios properties. 5.1 Temporal queries Temporal queries deal with temporal features of the presentation and are further classified in point queries and relationship queries. Point queries: These queries are useful when the author is interested in finding relationships between events. For instance: "Does object YMNOS2 start before KAVSOUND?" or "Does object YMNOS2 end after the start of object KAVSOUND?". Relationship queries: In this category we are interested whether the temporal interval corresponding to the presentation of an object fulfills a required relationship. Such a query would be: "Are the presentations of objects IMG1 and KAVALAR overlapping?" or "Is IMG1 presented before KAVALAR?". This category of queries is handled as follows: The cn is evaluated by parsing and computing the temporal difference between the desired events (i.e. YMNOS2 start as compared to KAVSOUND start). The result, depending on the modality in which the query is posed, is obtained examining the obtained temporal differences. 37

38 5.2 Conditional queries This query category deals with queries, which capture the higher consistency level of the scenario according to author-defined constraints and conditions, in terms of relationships between events in the scenario. Two types of conditional queries are considered: constraints and conditions. Constraints: when constraints are considered, the author is interested in a set of constraints related to temporal features of the presented objects. Usually, these constraints represent general properties which are required in the scenario. For instance, a requirement can be that audios cannot overlap due to hardware constraints. Other constraints can be specific to scenario being investigated. Such constraints could be as follows: In any execution of the scene, i.e., independently of any user interaction, "Audio YMNOS2, if started, must start before Audio KAVSOUND", or "KAVALAR must be temporally shorter that audio KAVSOUND". A conjunction of all related queries guarantees that if the network is consistent, all constraints are taken into consideration at the same time. Conditions: Here, authors are interested in queries based on the assumption that a specific set of events has occurred in a specific order. This is a way of testing scenarios based on given sequences of events. This is particularly useful to investigate properties of scenarios which have errors in some of the paths. Examples of such queries can be as follows: "Will the presentation reach IMG1 before KAVALAR given the sequence of events StartApp, _IntroStop, _Timer10, e1?". Such queries are handled in the following way: given the event sequence (i.e. StartApp, _IntroStop, _Timer10, e1), the corresponding network(s) is examined (if it exists) and then it is (or they are all) evaluated against the requested condition (e.g. IMG1 starts before KAVALAR). 5.3 (Temporal) layout queries This kind of queries results in graphical representations, depicting the temporal relationships among presented objects. Such kind of queries is a necessity that is recognized among the authoring community [Vazirgiannis et al. 98]. An example of such queries can be: Q1 "Show the temporal layout of the IMD between the 38

39 2nd and 10th second of the presentation" or Q2 "Show the temporal layout of the IMD between the events StartApp and KAVSOUND<. The general approach to answer the aforementioned queries is to consider those cns which present the characteristics mentioned in the query. For instance, Q2 implies that both StartApp and KAVSOUND are reached (necessarily or possibly, depending on the modality in which the query is posed). If a specific time value is mentioned, such as in Q1, all possibly occurring events between time 2 sec. and time 10 sec. are retrieved as a result of the query. When the author wants to know what will be the precise scenario evolution at a specific time, values can be associated to the events and cns can be examined based on the given time values. This type of queries gives the authors the possibility of examining the different possible evolutions of scenarios, which is particularly useful to detect the source of possible errors in some of the paths. 6. Verification of the approach As it is evident the consistency checks for and IMD document can be computationally very expensive due to the alternative paths that can occur because of internal and external interaction. In the following we attempt a worst case analysis of the approach. Though several heuristic methods and specific features may improve the overall cost. 6.1 Theoretical verification Let t be the number of tuples in the scenario, and let the worst case assumption that each tuple has a start event of the type ANY(k, e 1.. e m ). In the following we evaluate the time complexity of the steps of the methodology: Step 1:Transformation of tuples to cnfs and evaluation of the cnf As we mentioned before we assume the worst case in which every tuple has a start event of the form: ANY(k, e 1.. e m ). This means that for each tuple we will have m different cnfs produced. Then checking the integrity of a scenario tuple k implies: 39

40 - carrying out the checks that are related to the constraints of Table 2. The former step means that for each pair of actions appearing in the cnf we have to verify the constraints of Table 2 which is of linear complexity (O(n)), where n is the number of nodes. - carrying out the consistency check of the cnfs according to [Dechter et al. 91]. m * t k Assuming t is the number of scenario tuples in the scenario, different cnfs have to be verified, in the worst case where start event is of the form ANY(k, e 1.. e m ). As it is known from [Dechter et al. 91] the complexity of this task is O(n 3 ) where n is m the number of nodes in the cn. Thus the overall worst case complexity is to( n 3 ) k Step 2: Finding the different event paths This problem can be reduced to all the possible event sequences that may occur. In the general case we should consider all the permutations of the start events. So assuming a number of t tuples we have t start events and therefore all the possible permutations are t!. Of course this can be optimized to a great extent since not all the sequences of events are allowed and also the StartApp event is always the first in each permutation. Thus the result of this stage is the set of legal permutations which in the general case is much smaller that t!. Step 3&4 : Composition & evaluation of the cns Assuming the legal permutations of events, as result of the previous step, we carry out the construction of the alternative cns using the following algorithm: 1. For each of the legal paths cn= 2. for each event ei cnl = cnfs of the tuples started by ei cn = cn cnl add necessary arcs(between events and actors with the same name) evaluate cn - state consistency constraints (qualitative) action sequence constraints (quantitative) consistency check (qualitative/quantitative) 40

41 The complexity of the above algorithm is based on the number of alternative paths (and hence cns) which in the worst case is O(t!), Since the number of different alternative cnfs because of the complex start events is: m Ο k t and the number of nodes in the cn which is n, then the overall worst case complexity is: t m Ο t! n k 3 The complexity formula becomes: O((t-1)!m t n 3 ) dominated by m t. Typical values for the parameters are the ones of the example presented in the paper: t=4, m=3, k=1 and n=9. Then the cost of exhaustive checking for this would be: steps in a memory structure. Memory in modern computers is cheap and we can expect the process to end in reasonable time. Assuming that memory seeking & read/write operations take time of the order of nanosecond (10-9 sec) the above example would take approximately 0, sec. to complete. Though this case would be very rare as the vast majority of the paths in the scenario are pruned during the initial steps of the methodology: the vast majority of the paths are not legal and heuristic methods are used to reject them in early stages of the methodology, so that the approach is computationally feasible. The points in the methodology where considerable pruning is taking place follow: first step. The cnfs are build and a first consistency check is done (see Section 4.1.3). Warnings are raised when constraints presented in Table 1 and Table 2 are not fulfilled. In such a case, the user can decide if the cnf must be considered further or not. At this stage of the methodology, cnfs can already be considered as not valid and therefore pruned from further processing. second step: alternative execution paths are build. In the formula showing the time complexity of this step of the methodology, we consider that in the worst case, there are t! paths, with t corresponding to the number of tuple in the scenario. But most of the times, the scenario has been designed for a given number of predefined execution paths and do not allow all the 41

42 possible permutations of the tuples (for instance, in the example given in Fig. 5, only 2 paths are found instead of 4!). Therefore, t! is a value which is never reached. At the end of the second step of the methodology, several paths can already be considered as not valid if they do not start and end in the Idle State and therefore pruned for the forthcoming steps. third step: cnfs are composed with according to the paths considered as valid, and a consistency check is done (see Section 4.3.2). Warnings are raised when an object is not explicitly teminated in a given path. Paths are again checked with regards to constraints of Table 1 and Table 2, raising errors if they are not fulfilled. At this step of the methodology, cnfs can already be considered as not valid and therefore pruned for the final step, dealing with the temporal consistency check. An issue related to the user interface is that only cns and paths (with their corresponding cnfs) raising errors and/or warnings are shown to the user. Thus the user is not overwhelmed by the multitude of alternative paths that are produced in memory. 6.2 Implementation The above verification was also illustrated by an implementation. The methodology we have analyzed in the previous sections has been used for the implementation of tool which can be used to check IMD scenarios. For the implementation of the methodology we have developed an object-oriented model, which was implemented in C++ under Windows 98/NT. An IMD scenario file is imported to the tool. It is a simple ASCII file which has the format presented in Appendix A. The tool implements the approach in four steps. During the execution of each step the application detects both temporal errors and warnings. The results of each step are presented to the user. The first step aims at translating the IMD scenario script into CNFs that are kept in appropriate main memory resident structures. During translation the application detects syntactic errors. The contents of the scenario are graphically presented to the user (see Fig. 12). The second step aims at finding all the possible paths that 42

43 can be followed as a result of interaction. The third step composes the CNFs that have resulted from the first step according to the sequences defined by the different paths (resulting from the second step) into constraint networks and carries out again a first check for warnings. The fourth step checks the temporal integrity of the resulting cns and reports on warnings and errors. Figure 12. The User interface of the IMD Integrity checking tool. After the execution of the last step a minimal network for each alternative constraint network is constructed while the network is labeled as consistent or inconsistent. The user can then perform changes in the IMD scenario in order to improve its temporal integrity. There are some limitations though. In the present implementation of step 2 (Paths Construction) author intervention is required, as the algorithms for finding the execution plans do not cover the case of the composite event TIMES(n,e 1 ) in scenario tuples. For instance in the example presented in Fig. 12 there are only 2 paths that result from step 2. The paths can be visualized as it appears in Fig. 13, while errors and warnings appear in the dialog box in Fig

44 Figure 13 Visualization of a valid path. Figure 14. Reporting errors and warnings 7. Related Work There have been research efforts on the issue of scenario verification and integrity. In [Courtiat and De Oliveira 96], a synchronization model for the formal description of multimedia documents is presented, while in [Buchanan and Zellweger 95] an approach for automatic generation of consistent presentation schedules is presented. In [Courtiat and De Oliveira 96], the user formalization is automatically translated into a RT-LOTOS formal specification, allowing a verification of a multimedia document aiming to identify potential temporal inconsistencies. Multimedia documents are described through a hierarchical model, and incomplete timing is allowed. The user-interaction can also be represented and a media object can be expressed as one logical unit without avoiding however to refer to parts of it. A set of synchronization patterns are given, and formal semantics, in addition to a verification technique, are provided. In the proposed hierarchical model, start and end events are mandatory, while external and internal events are optional. The synchronization is done through the names of the events, or through explicit scripting. A presentation library provides monolithic and stream media objects. A constraint library is also provided. Constraints deal with temporal equalities (simultaneous events, precedence of a given event on another one, within a given temporal interval). Constraints also deal with temporal inequalities as precedence of an event to another, within an 44