Verifying Cyber-Physical Systems by Combining Software Model Checking with Hybrid Systems Reachability

Transcription

1 Verifying Cyber-Physial Systems by Combining Software Model Cheking with Hybrid Systems Reahability ABSTRACT Stanley Bak Cyber-physial systems (CPS) span the ommuniation, omputation and ontrol domains. Creating a single, omplete, and detailed model of a CPS is not only diffiult, but, in terms of verifiation, probably not useful; urrent verifiation algorithms are likely intratable for suh allenompassing models. However, speifi CPS domains have speialized formal reasoning methods that an suessfully analyze ertain aspets of the integrated system. To prove overall system orretness, however, are must be taken to ensure the interfaes of the proofs are onsistent and leave no gaps, whih an be diffiult sine they may use different model types and desribe different aspets of the CPS. This work proposes a bridge between two important verifiation methods, software model heking and hybrid systems reahability. A ontrat automaton (CA) expresses both () the restritions on the interations between the appliation and the ontroller, and () the desired system invariants. A sound assume-guarantee style ompositional proof rule deomposes the verifiation into two parts one verifies the appliation against the CA using software model heking, and another verifies the ontroller against the CA using hybrid systems reahability analysis. In this way, the proposed method avoids state-spae explosion due to the omposition of disrete (appliation) and ontinuous (ontroller) behavior, and an leverage verifiation tools speialized for eah domain. The power of the approah is demonstrated by verifying ollision avoidane using models of a distributed group of ommuniating quadopters, where the provided models are software ode and ontinuous -d quadopter dynamis. CCS Conepts Software and its engineering Formal software verifiation; This material is based upon work funded and supported by the Department of Defense under Contrat No. FA87-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded researh and development enter. [Distribution Statement A] This material has been approved for publi release and unlimited distribution. Please see Copyright notie for non-us Government use and distribution. DM (SEI); 88ABW , 06 JUN 06 (AFRL). Publiation rights liensed to ACM. ACM aknowledges that this ontribution was authored or o-authored by an employee, ontrator or affiliate of the United States government. As suh, the Government retains a nonexlusive, royalty-free right to publish or reprodue this artile, or to allow others to do so, for Government purposes only. EMSOFT 6, Otober , Pittsburgh, PA, USA 06 Copyright held by the owner/author(s). Publiation rights liensed to ACM. ACM ISBN /6/0... $5.00 DOI: Keywords Sagar Chaki haki@sei.mu.edu verifiation; yber-physial systems; ompositionality; assume-guarantee; hybrid systems; software model heking.. INTRODUCTION A yber-physial system (CPS) onsists of a tight oupling between software and the physial world. CPSs play a ruial role in many aspets of our day-to-day lives, ranging from thermostats, ars and airplanes, to medial devies, nulear power plants and eletri grids. Sine many CPSs are safety-ritial appliations, it is important to assure their orret behavior to the maximum extent possible. Formal verifiation provides a high level of onfidene in a system s operation, and is therefore a desirable assurane approah. However, salable formal verifiation of CPSs is an open hallenge, and the topi of this paper. A general CPS may onsist of a distributed set of agents in a shared physial environment. Communiation is performed over a network, whih may not neessarily be reliable. The agents are eah implemented using C-language soure ode that is run periodially by a real-time sheduler. The goal of this work is enable the verifiation of high-level properties, whih deal with the physial world and in relation to multiple agents. With this goal in mind, the speifi ontribution is a deomposition of one part of the larger verifiation proess. In partiular, the proposed method enables formal reasoning between the software ode on a single agent and the physial environment with whih it interats. Combined with other verifiation approahes, we show how this deomposition enables end-to-end reasoning about highlevel system properties. We onsider a CPS agent onsisting of two layers an appliation A and a ontroller C. The single-agent system S, is a omposition of these two, S = A C. This omposition is performed along the analysis boundaries, where A is analyzed using software model heking and C is analyzed with hybrid systems reahability tools. Note that in our approah, the ontroller model C onsists of not just the low-level ontroller, but also the ontinuous plant dynamis. The appliation A and ontroller C exeute in parallel and ommuniate via shared variables. The appliation is available as soure ode that alls a speifi set of funtions (API) to aess (read/write) the shared variables, while the plant/ontroller model is represented by a hybrid automaton whih interats with the environment based on the shared variables. In order to enable end-to-end reasoning, it beomes neessary to verify yber-physial properties, whih are true for the ombined appliation and ontroller system, but not neessarily for the individual parts. We want to verify that the system satisfies some yber-physial safety property Φ, i.e.,

2 A C = Φ. There are two hallenges to doing this diretly. First, there are no algorithms to verify a omposition of soure ode with hybrid automata. Seond, even if we developed suh an algorithm, using it would likely suffer from state-spae explosion for any pratial systems, as it would require us to onstrut A C. We propose an approah that sidesteps these two problems. Our key intuition is that A C = Φ typially holds beause A and C interat in restrited ways, and eah assumes a speifi behavioral pattern that the other guarantees. In partiular, A alls the API funtions in a welldefined sequene with parameters that respet ertain preonditions, while C maintains invariants involving the physial system state in relation with the shared variables. We leverage this intuition by using a ontrat automaton (CA) M to apture the restrited interation between A and C, as well as the target safety property Φ. Further the CA leads to an assume-guarantee style proof rule that enables ompositional verifiation: A M C M A C M where X Y informally means that X refines Y, i.e., behaviors of X an onform to the behavior of Y. We define the premises and onlusion of the proof rule formally, and prove its soundness. We also desribe proedures for disharging the premises using domain-speifi tools. Speifially, A M is verified with a software model heker, while C M is verified using a hybrid systems reahability tool. Finally, we should how the onlusion of the proof rules implies that the invariants of M are transferred to the system A C, thereby verifying that A C satisfies the target yber-physial property Φ. Our approah not only avoids omposing A and C (thus ameliorating state-spae explosion) but also uses software verifiation and hybrid verifiation tools synergistially. Disharging the two premises is still omplex, but they build on software model heking and hybrid systems analysis tools, where a lot of progress is being made. One a yber-physial property Φ is proven about individual agents, other verifiation methods an use Φ to reason about high-level system properties dealing with the distributed interations. To the best of our knowledge, this is the first end-to-end formal verifiation of high-level properties of a distributed yberphysial system, that inludes both the appliation software and the ontroller, using a sound ombination of software model heking and hybrid systems reahability analysis. The rest of the paper is organized as follows. Setion first presents a running example of an end-to-end verifiation problem dealing with distributed ollision avoidane for quadopters. Then, Setion 3 introdues ontrat automata, and the ompositional proof rule, and applies this to the quadopter system. Next, Setion 4 shows how the premises of our proof rule an be disharged using software and hybrid systems verifiation tools. In Setion 5, we use the developed method to prove yber-physial properties, and uses these properties to omplete the high-level, distributed ollision avoidane proof. Finally, we disuss related work in Setion 6, followed by a onlusion.. QUADCOPTER CPS EXAMPLE We demonstrate our approah by proving end-to-end ollision avoidane, in terms of physial distanes, between a Current setpoint spur = (0,0) 0 Cell Ids y Positions Position pos = (0,0) Next setpoint spnxt = (5,0) x Figure : Example quadopter. group of ommuniating quadopters moving in a ontinuous -d spae. Eah quadopter ontains a path-planning appliation, written in C, that ommuniates with other quadopters and updates its own setpoints. The evolution of the quadopters physial aspets is governed by differential equations, whih are not oupled between different quadopters (although they move in a shared environment). A low-level ontroller periodially atuates eah system based on the urrent setpoint, aording to some real-time sheduling poliy like Rate Monotoni [36]. Communiation between quadopters is unreliable, i.e., message delivery is not guaranteed. We assume a finite -d spae with a fixed number of quadopters. We also have ontrol over the initial states. The desired high-level property is that no two quadopters ever ollide, i.e., assuming a given quadopter radius, the distane between any two quadopters is always greater than HELI_RADIUS. The setpoint-generating appliation needs a strategy to prevent ollisions. In our ase, the appliation software generates setpoints based on a -dimensional grid whih overs the physial spae. Eah ell is a 5 5 square, and is represented by a unique pair of integers, whih we refer to as the grid id. The left-bottom-most ell has id (0, 0), and the integer ids inrease moving to the right or up, as in a traditional oordinate axis. While the appliation software generates the next setpoint, a ontroller is responsible for atually moving the quadopter. The setpoints for eah diretion are a 5-multiple of a ell id (i.e., they orrespond to the enters of ells). The appliation and ontroller ommuniate via two shared variables: (i) spur = (spur x, spur y ): the urrent setpoint; and (ii) spnxt = (spnxt x, spnxt y ): the next setpoint. In addition, the ontroller model maintains a variable pos = (pos x, pos y ) to denote the position of the enter of the quadopter w.r.t. the global oordinate system. For example, in Figure, we have spur = pos = (0, 0) and spnxt = (5, 0), meaning that the enter of the quadopter oinides with the enter of ell (0, 0). Given two oordinates = (x, y) and = (x, y ) we write to mean (x x, y y ), to mean ( x, y ) and to mean (x x y y ). Suppose that the quadopter is hovering over its urrent setpoint. One the appliation has omputed the next setpoint, it moves the quadopter to this setpoint as follows: (Step ) The appliation alls the funtion update setpoint(x, y) where the arguments ontains the value of the next setpoint; this funtion sets the value of spnxt to (x, y), whih triggers the ontroller to move to spnxt. Funtion update setpoint(x, y) returns a void value.

3 (Step ) The appliation then repeatedly polls the state of the system by alling the funtion has arrived(), until it returns true. This funtion returns true only if: pos spnxt (0., 0.). In addition, if has arrived() returns true, it updates the value of spur to be spnxt. Corret interation between the appliation and the ontroller, whih we will show is suffiient to prove yberphysial properties about the ombined system, requires several onditions: (C) The appliation always alls update setpoint(x, y), with arguments that satisfy the ondition (x, y) spur = (5, 0) (x, y) spur = (0, 5). (C) One the appliation alls update setpoint(x, y), it an keep alling has arrived() until it gets a return value of true; one has arrived() returns true, the appliation an only then start to all update setpoint(x, y) again. (C3) When the quadopter is hovering (i.e., spnxt = spur), the ontroller must maintain the following invariant: Φ hover pos spur (.5,.5). (C4) When the quadopter is moving (i.e., spnxt spur = (5, 0) spnxt spur = (0, 5)), the ontroller must maintain the following invariant: Φ move min(spur x, spnxt x ).5 pos x max(spur x, spnxt x ) +.5 min(spur y, spnxt y ).5 pos y max(spur y, spnxt y ) +.5 Note that onditions C C restrit the sequene of funtion alls that an be made by the appliation, and the arguments that an be passed, while onditions C3 C4 restrit the behavior of the ontroller. In the next setion, we will see how a ontrat automaton an be used to both speify and verify suh onditions formally. 3. CONTRACT AUTOMATON We assume a omputational model where the appliation and ontroller exeute in parallel, and ommuniate via three types of shared variables. These shared variables fall in three ategories: (i) Cyber variables V C : these are written by the appliation only, during funtion alls; (ii) Parameter variables V P ar : these are used as parameters of funtions alled by the appliation to interat with the ontroller; and (iii) Physial variables V P : these are modified by the ontroller only. We write V to denote the set of all variables, i.e., V = V C V P ar V P. All variables are typed. We use real (R) and Boolean (B) variables. For brevity, we use the symbol for a type to also denote the set of elements of that type. Thus, R is also the set of all real numbers. Funtions an also return void ( ) values. In our example from Figure, we have V P = {pos : (R, R), V C = {spur : (R, R), spnxt : (R, R), and V P ar = {x : R, y : R. Expressions. Let D = R B be the set of all non-void values (reals, true, and false). Given a set of variable V V, we write Expr(V ) to denote set of expressions onstruted from V D, using numeri operators (+,,, /, et.), relational operators (<,, >,, et.), and logial operators (,,, et.). spnxt = spur Φ hover f: update_setpoint x, y req: (x, y) spur = 5,0 x, y spur = (0,5) grd: true A: spnxt (x, y) rv: hover spnxt spur = 5,0 spnxt spur = 0,5 Φ move f: has_arrived( ) req: true grd: pos spnxt > (0.,0.) A: rv: false f: has_arrived( ) req: true grd: pos spnxt (0.,0.) A: spur spnxt rv: true wait Figure : Example ontrat automaton. Funtions and Funtion Calls. A funtion is a triple (fn, p, rt) where fn is the funtion name, p V P ar is a list of its parameters, and rt is its return type. The set of all funtions via whih the appliation interats with the ontroller is denoted Fun. Indeed, for our purposes, the semantis of the appliation is a set of exeution traes, where eah trae is a sequene of funtion alls. A funtion all is a triple (f, a, rv) where f = (fn, p, rt) is a funtion, a : p D maps eah parameter to an argument of appropriate type, and rv rt is a return value of appropriate type. Example. In our example from Figure, we have Fun = {f, f where: (i) f = (update setpoint, x, y, ); and (ii) f = (has arrived,, B). Some possible funtion alls are (f,,, ), (f,, false), et. Definition (Appliation). An appliation is defined by a C-language program that makes alls to Fun. Definition (Controller). A ontroller is defined by a hybrid automaton over the variables V P V C. Assignments. An assignment is a pair (lhs, rhs) where lhs V is the left-hand side and rhs Expr(V) is the righthand side. The set of all assignments is Asgn. Definition 3 (Contrat Automaton). Formally, a ontrat automaton (CA) is a 5-tuple (S, I, T, Inv, L) where: S is a finite set of loations; I S is the initial loation; T S S is a transition relation; Inv : S Expr(V P V C) maps eah loation to an expression over the physial and yber variables; informally, Inv(l) is the invariant that a orret ontroller should maintain when the system is in loation l; L : T Fun Expr Expr Asgn (D { ) labels eah transition with information about the funtion all from the appliation that triggers the transition, a guard under whih the transition ours, a sequene of assignments that the transition exeutes,

4 and the return value of the triggering funtion all that the transition results in. Formally, if L(l, l ) = (f, req, grd, U, rv), then it means: The transition from l to l is triggered by a all to funtion f = (fn, p, rt) by the appliation. Any suh all must satisfy the ondition req, whih is an expression over V C p. One the transition is triggered, it an only our if ondition grd, whih is an expression over V P V C p, holds. The key differene between req and grd is that while every all to f by A must satisfy req, it does not have to satisfy grd. If the transition ours, it exeutes the assignments in U and then the all to f returns with value rv. Note that the labeling of a transition provides a semanti desription of the orret implementation of f. Indeed, f must implement the funtion: if (grd) then {U; return rv; We will use this intuition for the verifiation steps presented in the following setions. Example. Figure shows the ontrat automaton M for the quadopter system desribed in Setion. The automaton has two loations hover and wait. The initial loation is hover. Loations are labeled with orresponding invariants, and transitions are labeled with details about the funtion alls that trigger them. Note how M enfores the onditions C C4 from Setion. Speifially, onditions C C are enfored by the possible transitions and the funtion alls labeling them, while onditions C3 C4 are enfored by the invariants labeling the loations. 3. Contrat Automaton Semantis To define the semantis of a ontrat automaton, we have to first define states, and how expressions are evaluated. A state σ : V D is a partial assignment of variables to values. The domain of σ is denoted Dom(σ). We write σ σ to denote the state obtained by merging σ and σ with disjoint domains, i.e., if Dom(σ ) Dom(σ ) =, then: (σ σ )(v) = σ i(v) v Dom(σ i), i {, Given a set of variables V, the set of all states σ suh that Dom(σ) = V is denoted Σ(V ), i.e., Σ(V ) = {σ : V D Dom(σ) = V Given a state σ and a set of variables V Dom(σ), the projetion of σ on V, denoted σ V, is the state suh that: Dom(σ V ) = V v V (σ V )(v) = σ(v) Given a state σ, and an expression e, we write [e, σ]] to denote the value obtained by evaluating e under σ in the natural way. For example, if σ(v ) = 5 and σ(v ) = 3, then [[v v, σ]] =, and [[v v > 3, σ]] = false. We write σ = e to mean [[e, σ]] = true, and σ = e to mean [[e, σ]] = false. Trajetory. Given two states σ and σ suh that (σ V C) = (σ V C), a trajetory from σ to σ is the sequene of states enountered as a finite amount of time elapses, due to the ontinuous dynamis and the low-level ontroller. This is a trajetory in the hybrid automaton sense, whih inludes intervals of ontinuous evolution and disrete jumps. It is an infinite sequene of states starting with σ and ending with σ that does not modify the yber variables, ( σ σ V C) = (σ V C). Given expression e, we write = e to mean σ σ = e. Contrat Automaton Transition. Let M = (S, I, T, Inv, L) be a ontrat automaton. Its semantis is given by a state transition system, where eah state is a pair (l, σ) suh that l S and σ Σ(V P V C). There are two types of transitions appliation-triggered and ontroller triggered. An appliation-triggered transition is of the form (l, σ) (l, σ ) suh that: (i) (l, l ) T ; (ii) (σ V P ) = (σ V P ); note that this means an appliation-triggered transition does not alter the values of physial variables; (iii) σ = Inv(l) σ = Inv(l ); and (iv) L(l, l ) = (f, req, grd, U, rv) and = (f, a, rv) suh that: σ a = req grd; and {σ au{σ a, i.e., state σ a is obtained from σ a by exeuting the assignments in U; note that we need a sine it may be read (but not updated) by U. The set of all appliation-triggered transitions is denoted δ(m) A. A ontroller-triggered transition is of the form (l, σ) (l, σ ) suh that: (i) is a trajetory from σ to σ ; (ii) (σ V C) = (σ V C); and (iii) = Inv(l); note this means that the invariant of l is maintained at all intermediate states as M transitions from σ to σ. A ontroller-triggered transition does not alter the loation of the ontrat automaton. The set of all ontroller-triggered transitions is denoted δ(m) C. Definition 4 (Contrat Automaton Semantis). An exeution of M is an alternating sequene of ontrollertriggered and appliation-triggered transitions: (l, σ ) (l, σ ) (l, σ )... (l n, σ n ) n (l n, σ n) suh that: l = I i [, n ] (l i, σ i) i (l i, σ i) δ(m) C i [, n ] (l i, σ i) i (l i+, σ i+) δ(m) A The semantis of a ontrat automaton M, denoted [[M]], is the set of all its exeutions. 3. Refinement Our broad goal is to show that, if an appliation A and a ontroller C both refine a ontrat automaton M, then the system omposed of A and C refines M as well. In this setion, we present this formally. We begin with the semantis of an appliation. Definition 5 (Appliation Semantis). For our purposes, an appliation is a blak-box that makes alls to funtions in Fun. Thus, the semantis of A, denoted [[A]] is a set of exeutions, where eah exeution π is a sequene of states and funtion alls, i.e., π = σ n σ... σ n σ n suh that eah σ i maps yber variables to values, i.e., i Dom(σ i) = V C. Definition 6 (Controller Semantis). Sine the ontroller C is defined by a hybrid automaton, its semantis is given by a set of exeutions over V C V P, and an initial

5 state Init C Expr(V C V P ). An exeution of C is a sequene σ σ, σ σ n,..., σ n σ n suh that: (i) σ = Init C; (ii) i [, n ] (σ i V C) = (σ i V C); (iii) i [, n ], i is a (hybrid) trajetory from σ i to σ i+; and (iv) i [, n ] σ i V P = σ i+ V P. Intuitively, eah trajetory represents evolution of C without interferene from A, and the possible jump from one trajetory to the next is aused by a funtion all. The semantis of C, denoted [[C]], is the set of all its exeutions. Definition 7 (System Semantis). The system S = A C is an asynhronous interleaving of the appliation and the ontroller where funtion alls by the appliation interleave with the evolution by the ontroller. The semantis of S, denoted [[S]], is given by a set of exeutions where eah exeution is a sequene of the form: σ σ n σ... σ n σ n n σ n suh that eah transition σ i i σ i represents ontinuous evolution by the ontroller, and eah σ i i σi+ represents a funtion all by the appliation. In other words: (SS) i Dom(σ i) = Dom(σ i) = V C V P (SS) i (σ i V C = σ i V C) (σ i V P = σ i+ V P ) (SS3) σ σ, σ σ n,..., σ n σ n [[C]] (SS4) σ V C σ V n C... σ n V C [[A]] Definition 8 (Appliation Refinement). A refines M, denoted A M, if every exeution of A, that maintains the invariants in eah mode of the ontrat automaton M, orresponds to some exeution of M. Formally: A M σ n σ... σ n σ n [[A]] l = I, l,..., l n σ,..., σ n σ σ = Inv(I) i [, n ] σ i σ i = Inv(l i) σ i σ i = Inv(l i) = l n (l n, σ n σ n ) n (l n, σ n σ n ) δ(m) A Note that eah l i is a loation of M and eah σ i Σ(V P ) maps V P to values. Definition 9 (Controller Refinement). C refines M, denoted C M, if every trajetory in C, that obeys the transitions (ordering and pre/post onditions) from M, orresponds to some exeution of M. Formally: C M σ σ, σ σ n,..., σ n σ n [[C]] = Inv(I) l = I, l,..., l n i [, n ] i (l i, σ i) i (l i+, σ i+) δ(m) A = n = Inv(l n ) This onsists of heking that the ontroller s reahable set of states, under any appliation A whih satisfies the disrete transition onditions in the ontrat automaton, does not violate the mode invariants in the ontrat automaton. Definition 0 (System Refinement). S = A C refines M, denoted S M, if every exeution of S orresponds to an exeution of M. Formally: S M σ σ n σ... σ n σ n n σ n [[S]] l, l 3,..., l n (I, σ ) (I, σ ) (l, σ )... (l n, σ n ) n (l n, σ n ) n (l n, σ n) [[M]] 3.3 Cyber-Physial Properties A ontrat automaton s power is in proving yber-physial properties. These are properties whih are true not solely on the basis of the appliation software, or the ontroller, but instead require both to satisfy ertain properties (expressed olletively in the ontrat automaton). The CPS properties are expressed as relations over the yber and physial variables. For the ontrat automaton from Figure, the orresponding yber-physial property Φ is: (Φ hover spnxt = spur) (Φ move ( spnxt spur = (5, 0) spnxt spur = (0, 5))) It follows from Definition 4 that all reahable states of M satisfy its invariant. In other words: Proposition (Invariant Satisfation). (l, σ ) (l, σ ) (l, σ )... (l n, σ n ) n (l n, σ n ) n (l n, σ n) [[M]] i [, n] σ i = Inv(M) i [, n ] i = Inv(M) The main power of Definition 0 is that it implies that if S M, then Inv(M) also holds on S (where Inv(M) is defined as the disjuntion of invariants over all loations in M given in definition 3). Formally: Proposition (Invariant Preservation). S M = σ σ n σ... σ n σ n n σ n [[S]] i [, n] σ i = Inv(M) i [, n ] i = Inv(M) However, heking S M diretly is omplex beause S and M ombine disrete behavior by the appliation with ontinuous behavior by the ontroller. In the next setion, we show how S M an be heked ompositionally using two separate verifiation steps one for A M and another for C M, and how eah of these an be ahieved using domain-speifi verifiation tools. 3.4 Compositional Refinement Chek We now present our main theorem in the form of an assume-guarantee style proof rule. Theorem (Compositional Refinement). A M C M A C M Proof. Let S = A C, A M, C M, and π [[S]] be any exeution of S. Let: π = σ σ n σ... σ n σ n n σ n The degenerate ase of an exeution where the appliation never exeutes is taken are of as part of the base ase below.

6 For i [, n], let us write σ A,i to mean σ i V C. By ondition SS4 in Definition 7 we know that: σ A, n σa, σa,3... σ A,n σ n [[A]] From ondition SS3 in Definition 7, we know that: σ σ, σ σ n,..., σ n σ n [[C]] For i [, n ], define σ i = σ i V P. From ondition SS in Definition 7, we know that i [, n ] σ i = σ i+ V P. We will now show by indution that l,..., l n suh that (I, σ ) (I, σ ) (l, σ )... (l n, σ n ) n (l n, σ n ) n (l n, σ n) [[M]]. Then, our result follows diretly from Definition 0. Base Case: From Definition 9, we know that = Inv(I). Hene (I, σ ) (I, σ ) δ(m) C. Also note that σ = σ A, σ, σ = Inv(I), and σ = σ A, σ. Hene from Definition 8, we have l (I, σ ) (l, σ ) δ(m) A. From Definition 4, we have (I, σ ) (I, σ ) (l, σ ) [[M]]. Indutive Step: Suppose l,..., l m suh that (I, σ ) (I, σ ) (l, σ )... (l m, σ m ) m (l m, σ m ) m (l m, σ m) [[M]]. Using the indutive hypothesis, and Definition 9, we know that m = Inv(l m). Hene (l m, σ m) m (l m, σ m) δ(m) C. Again note that σ m = σ A,m σ m and σ m+ = σ A,m+ σ m. Hene from the indutive hypothesis and Definition 8, we have l m+ (l m, σ m) m (l m+, σ m+) δ(m) A. From Definition 4, this means (I, σ ) (I, σ ) (l, σ )... (l m, σ m) m (l m, σ m) m (l m+, σ m+) [[M]]. This ompletes the proof. Note that our proof rule is not omplete. Consider a ontroller C that fails to maintain the invariant Inv(l ) when loation l is reahed via funtion all. Suppose it is omposed with an appliation A that never alls, i.e., the appliation prevents the ontroller from reahing the bad state. In this ase, the onlusion of our rule A C M holds, but the premise C M does not. 4. VERIFYING PROOF-RULE PREMISES In this setion, we illustrate how to disharge the two premises of the proof rule given in Theorem, i.e, A M and C M. 4. Cheking Appliation Refinement We assume that the appliation A is a C-language program with alls to Fun. To hek A M, we onstrut stub-funtions for eah f Fun that hek the onditions in Definition 8. We then verify A along with the stub-definitions of Fun using an off-the-shelf software model heker. Our stub-funtions for Fun are non-deterministi. This is neessary sine the onditions in Definition 8 involve quantifiers. More speifially, we assume a software model heker that supports three features: (i) non-deterministi value *; (ii) assume a funtion that bloks all exeutions that invoke it with a false argument, typially used to model the environment under whih a speifi part of a program is exeuted; and (iii) assert a funtion that aborts all exeutions that invoke it with a false argument, typially used to detet the violation of safety properties. All these features are supported by most state-of-the-art software model hekers. For example, the bounded model enum Lo {hover, wait; Lo lo = hover; void update_setpoint(double x, double y) { pos = *; //-- assign non-deterministi value if (lo == hover) { assume(inv_hover); assert(req_hover_wait); spnxt = (x,y); assert(inv_wait); lo = wait; return; assert(0); _Bool has_arrived() { if (lo == wait) { pos = *; //-- non-deterministi hoie between //-- two outgoing transitions from wait if (*) { assume(inv_wait); assume( pos - spnxt > (0.,0.)); assert(inv_wait); lo = wait; return 0; else { assume(inv_wait); assume( pos - spnxt <= (0.,0.)); spur = spnxt; assert(inv_hover); lo = hover; return ; assert(0); Figure 3: Stub definitions for our example ontrat automaton; INV_x denotes invariant Inv(x); REQ_a_b denotes the req omponent of label L(a, b). The * is a non-deterministi hoie. void A() { for (int n=;;++n) { update_setpoint(n,0); while(!has_arrived()); void A() { for (int n=;;++n) { update_setpoint(n,0); while(has_arrived()); Figure 4: Two example appliations where initially spur = spnxt = (0,0). A() refines our example ontrat automaton; A() does not. heker bm [7] supports non-determinism via return values of undefined funtions, assume via a all to the funtion CPROVER_assume, as well as assert. Consider a ontrat automaton M = (S, I, T, Inv, L). The body of the stub funtion for eah f fun is generated as follows: (a) Introdue a global variable lo to trak the urrent state of M; lo is initialized to I. (b) For eah transition (l, l ) T with L(l, l ) = (f, req, grd, U, rv) generate ode that: (i) is exeuted only if lo = l; (ii) assigns non-deterministi values to V P ; (iii) assume-s Inv(l); (iv) assert-s ondition req; (v) assume-s ondition grd; (vi) exeutes assignments in U; (vii) asserts Inv(l ); (viii) updates lo to l ; and (ix) return-s rv. Example. Figure 3 shows the stub funtions for update_setpoint and has_arrived from our example ontrat automaton in Figure. We omit statements that have no effet (e.g., assert-ing or assume-ing true). Note that the assert(0) at the end of eah funtion ensures that the funtion is never alled when the ontrat automaton is in an inappropriate state. Also, sine there are two transition from state wait labeled by arrived, they are both allowed nondeterministially. The following theorem expresses the orretness of our

7 proedure. Theorem (Appliation Refinement Chek). The C-language program A together with the stub definitions of funtions in Fun onstruted as above has no exeutions that violate an assertion if and only if A M. Proof. (Sketh) Consider any σ σ σ 3... σ n n σn+ [[A]]. It an be shown that there exists a sequene of loations l,..., l n+ that satisfy ondition AR of Definition 8 if and only if the C-language program A together with the stub definitions of Fun exeutes a sequene of funtion alls,..., n suh that for i [, n] the value of lo when i is alled is l i, and the final value of lo is l n+. Example. Figure 4 shows two possible example appliations (note the real quadopter ode we use is signifiantly more omplex). A() refines our example M and the program obtained by ombining it with the stub definitions in Figure 3 does not violate any assertions. A() does not refine our example M and the program obtained by ombining it with the stub definitions in Figure 3 violates an assertion when it first alls update_setpoint(5,0), then alls has_arrived() whih returns false, and then alls update_setpoint(0,0). The appliation ode for eah quadopter was written in a domain-speifi language, alled dmpl [3], for programming distributed real-time systems, whih inludes a C-language ode generator. This feature was used to generate the C- language soure for the appliation A. The stub definitions for funtions update_setpoint_x(), has_arrived_x(), update_setpoint_y() and has_arrived_y() were reated manually from M as shown in Figure 3. They were also written in dmpl, and then onverted automatially to C soure ode. The ombined appliation and stub funtions, onsisting of about 700 LOC, were then verified using bm. Sine bm is a bounded model heker, and our appliation does not terminate, bm annot verify properties over (logially) unbounded program exeutions by itself. Therefore, we manually reated loopinvariants and verified them to be indutive using bm, thus enabling us to prove unbounded properties. Essentially, to prove that I is an invariant of a loop with body B, we verify the following program with bm HAVOC(); CPROVER_assume(I); B; assert(i); where HAVOC() assigns all relevant variables non-deterministi values. Note that the semantis of C is untimed and purely logial, and is therefore appropriate for modeling appliationtriggered transitions. Using a laptop with a quad-ore.9 GHz CPU and 6 GB of RAM, the hek took about 3.5 seonds. These invariants were also strong enough to imply all the assertions in the ode. This proves A M. 4. Cheking Controller Refinement We assume that the physial system and low-level ontroller C are modeled together as a hybrid automaton [9]. To hek C M, we onstrut a hybrid automaton H M using M suh that the omposed hybrid automaton C H M reahes a forbidden error state if C M. We then use an off-the-shelf hybrid system reahability analysis tool to verify that the forbidden states are not reahable in C H M. In order to do this we need a hybrid automaton model heker whih supports: (i) forbidden state heking; (ii) transitions with may-semantis; and (iii) automaton omposition. Figure 5: Converted hybrid automaton extrated from ontrat automaton. These features are generally supported by hybrid systems model hekers like spaeex [8] or flow* [5]. Automaton omposition an be performed by an external tool [4], if not supported natively by the reahability tool. For -d position dynamis, we onsidered a simple, doubleintegrator system, where the position s derivative ẋ = v, the veloity s derivative v = a, and the aeleration a is the ontrol input. We use a proportional-derivative (PD) low-level ontroller, with a proportional gain of 0 and a derivative gain of 3. Although hybrid systems reahability an handle more ompliated dynamis and ontrollers, the fous of this work on the interfae between the software and a hybrid automaton model. Thus, salability and auray of hybrid systems reahability analysis beomes an orthogonal problem. The H M derived from the ontrat automaton M is given in Figure 5. The proess of reating this automaton onsists of first diretly extrating the invariants and appliation guarantees from the original ontat automaton. Next, the model is onverted into a form amenable to analysis by a reahability tool, whih onsists of things like onverting disjuntions in guards to multiple transitions, using ompound onditions instead of min/max funtions, and eliminating irular stutter transitions whih do not affet analysis. These steps ould be automated in a model transformation framework [4]. The last step involves reasoning about model symmetry in order to failitate detetion of fixpoints within reahability analysis. If the x diretion, for example, was unbounded, then the reahable set of states would be infinite, and reahability using flow-pipe onstrution would not omplete. We take advantage of dynamis symmetry in the x and y diretions in order to redue the analysis to a single dimension, and furthermore, reenter the system to 0 whenever the ontroller settles near a new setpoint (the transition with the has_arrived label the H M automaton in Figure 5 has its reset assignment hanged from spur := spnext to x := x - spnext && spnext := 0 && spur := 0). This symmetry redution step needs to be proven orret, for example by using reahability redution transformations [5]. The quadopter low-level PD ontroller performs high-

8 Position x Figure 6: The SpaeEx model of the ontinuous approximation of sampled quadopter dynamis. The modes speify invariants and ODEs, while transitions have guards and instantaneous reset assignments. Veloity v (a) Position x Veloity v (b) Figure 7: (a) Time-bounded reahability of the omposed hybrid automaton, without symmetry redution; (b) Reahability omputation with symmetry redution, whih reahes a fixpoint. frequeny sampling and atuation, subjet to a real-time sheduler suh as Rate Monotoni or Earliest-Deadline First [36], whih guarantees one exeution per period, but is not stritly periodi. Suh a system an be abstrated for reahability analysis by a ontinuous one with an extra noise term in the dynamis, to aount the effets of any possible jitter in the shedule [6]. We use the ontinuous approximation of the ontroller in our hybrid automaton model, shown in Figure 6. A hybrid systems reahability tool an be used to analyze the hybrid automaton model onsisting of the omposition of the extrated automaton in Figure 5 and the ontroller/plant given in Figure 6. Without symmetry redution, a time-bounded analysis an be performed. The projetion of the reah set in terms of x position and veloity, omputed with Flow, is shown in Figure 7(a). After hanging the reset upon reahing the setpoint, as desribed above, a fixpoint in the reahable states an be deteted. On a.3 GHz quad-ore CPU laptop with 6 GB of memory, SpaeEx took about 33 seonds, 7 of whih was the reahability omputation. A -d projetion of the reah set output by the tool is given in Figure 7(b). Neither of these models reah the error mode in the extrated ontrat automaton from Figure 5, whih means that the ontrat s invariants are always met for the given plant/ontroller ombination, and therefore C M. An example of a plant/ontroller whih does not refine M would be one with an unstable ontroller. Suh as system would not have a response in the neessary bounds given in the invariants, and therefore reahability analysis would detet that the error mode is reahable. 5. PROVING HIGH-LEVEL PROPERTIES The ontrat automaton method enables the proving of yber-physial properties, whih deal with individual quadopters. The yber-physial property Φ onsists of the disjuntion of the invariants of eah of the modes of the ontrat automaton M, shown before in Figure. By Theorem, we have proven Φ holds for our quadopter system, by heking A M (done in Setion 4.) and C M (done in Setion 4.). Now, we go beyond proofs of properties of individual quadopters. We illustrate one way to use the CPS property Φ with additional formal verifiation tehniques in order to perform end-to-end reasoning about ollision avoidane between multiple quadopters. In partiular, we want to show ollision avoidane in a group of quadopters in a finite, shared spae. Speifially, we onsider a system onsisting of 0 quadopters moving on a d area (i.e., 0 0 ells). As mentioned before, the quadopter logi was programmed in dmpl. In addition to allowing C-language ode generation, dmpl also allows use of the synhronous model of omputation as a primitive in algorithm design. The ode generated from dmpl uses a barrier-based protool [], built on top of the madara [5] middleware, to implement this synhronous model of omputation. Also, dmpl s semantis takes are of paket loss and out-of-order arrival in the ommuniation layer. The ode generated from dmpl uses message retransmission and madara s paket reordering to remedy these situations. Our system exeutes a synhronous distributed ollision avoidane protool. Eah quadopter maintains a ell variable ellur orresponding to the urrent setpoint, and a ell variable ellnext orresponding to the destination setpoint. Eah ell is treated as a shared resoure, and a quadopter always loks a ell by ommuniating with the others before moving into it. The synhronous model of omputation is used to implement this distributed loking. We refer to the 0 quadopters as N 0, N,..., N 9. Eah quadopter has its own opy of yber and physial variables. For any suh variable x V C V P, we use x[i] to denote the opy of x for quadopter N i. Thus, for example, spur x [] is the x oordinate of the urrent setpoint of N and pos y [3] is the y oordinate of the urrent position of N 3. To prove ollision avoidane, one property we need is that the ells defined by ellur[i] and ellnext[j] are always mutually disjoint for distint quadopters, i.e., 0 i < j < 0 ellur[i] ellur[j] ellur[i] ellnext[j] ellnext[i] ellur[j] ellnext[i] ellnext[j] Note that this means essentially proving the orretness of the distributed loking algorithm. A seond property to hek is that for every quadopter, the setpoints are 5 times the orresponding integer ell ids, 0 i < 0 5 ellur[i] = (spur x [i], spur y [i]) 5 ellnext[i] = (spnxt x [i], spnxt y [i]) The verifiation step for these two properties leverages the synhronous model of omputation provided by dmpl. The ollision avoidane logi for all 0 quadopters is ombined into a single C-language program using the sequentialization tehnique [], where omputation proeeds in rounds based on the guarantees provided by the madara middleware. The ombined program onsists of about 7.5 KLOC,

9 Potential Error Software bug modifies setpoint twie in a row Software bug hanges setpoint by both x and y Controller s gains are too high ausing quadopter to overshoot into neighboring ell Controller logi unstable Real-time period of low-level ontroller too low has_arrived ondition too aggressive Barrier synhronization inorretly used in ommuniation protool Software does not reason about loss of ommuniation Buffer distanes in ells too small Heliopters too large for a given grid size Detetion SW SW HY HY HY HY DIST DIST SMT SMT Table : A list of possible design and implementation errors, and where our approah would detet them. The detetion loations are Software Model Cheking (SW), Hybrid Systems Reahability (HY), Distributed System Sequentialization (DIST), and High-Level SMT Proof (SMT). about 0 times the size of the single-quadopter appliation refinement hek, whih is then verified using bm. On same.9 GHz laptop that was used for the appliation refinement hek, verifiation requires about 900 seonds. Given these three properties (the yber-physial property from the ontrat automaton Φ and the two properties proven using sequentialization of the distributed system), we an now prove global ollision avoidane. The three properties were formally written using SMT syntax, and as well as an additional assertion whih enodes the ondition under whih a ollision ours (the positions are within twie the heliopter radius). This ondition in SMT syntax is: (<= (abs (- (pos i) (pos j))) (*.0 HELI_RADIUS)) Here, i and j are the x positions of two non-idential quadopters (the hek for y positions is similar). The satisfiability of these ombined properties was then heked using Z3 [], taking a fration of a seond. If HELI_RADIUS <.0, the SMT solver returned unsat, indiating that no onfiguration is possible where all the properties are true and a ollision is ourring. If HELI_RADIUS.0, then the SMT solver an produe ounter-examples demonstrating a ollision may be possible. For example, a possible ounterexample has one quadopter moving along the x diretion from ell 0 to ell, and another quadopter moving from ell 3 to ell. In this ase, first quadopter may be at position 6.5, while the seond is at position 8.5 (reall they are permitted to deviate from the setpoints by up to.5 units). In this ase, the quadopters are exatly HELI_RADIUS apart (when HELI_RADIUS =.0). To the best of our knowledge, this is the first formal verifiation of a distributed yber-physial system that inludes both the appliation software and the ontroller, using a sound ombination of software model heking and hybrid reahability analysis. The proof inludes end-to-end formal reasoning without gaps between analysis approahes, exept for syntati translations of properties, whih ould be automated (this translation would then ideally be proven orret). This makes it apable of athing a large variety of design and implementation mistakes. An outline of possible system errors, and where they would be deteted using the proposed approah, is provided in Table. 6. RELATED WORK Software Model Cheking. Our work is omplementary to, and leverages, verifiation tehniques [3] for sequential C-language programs [7]. Sequentialization has been used for onurrent program verifiation. However, most of this work is targeted toward multi-threaded software [34, 0] or real-time software [] exeuting on a single proessor, not distributed appliations. There has also been work on verifying distributed algorithms [3], while our goal is to verify distributed yber-physial systems where eah node has both disrete appliations and hybrid omponents. Hybrid-systems verifiation targets systems modeled using hybrid automata [], whih are best suited for modeling physial aspets of CPS with simpler disrete behaviors. Hybrid automata onsist of, roughly, finite state mahines ombined with differential equations within eah mode. Various hybrid automata model hekers exist depending on the omplexity of the differential equations. Tools for omputing reahability exist for timed automata [4], linear hybrid automata [8], and systems with general, nonlinear dynamis [5]. Other analysis methods for hybrid systems inlude falsifiation [3, ], where the goal is to searh for onrete inputs that lead to a property-violating trae. Composition Verifiation. Assume-guarantee reasoning was proposed in the ontext of distributed programs [33], networks of proesses [37], and program verifiation [39]. L* has been used [9, 4] to learn assumptions automatially. Compositional verifiation tehniques have also been explored for model heking [8], probabilisti system verifiation [], omponent-based reasoning with reals[6, 4], and hybrid systems [9, 8]. Within hybrid systems, analysis tratability an be improved by analyzing loal omponents, and then reasoning separately about their omposition [30, 3, 7]. However, these approahes assume systems with semantially uniform omponents (e.g., finite state automata), while we handle systems with disrete and dense hybrid omponents. Nuzzo et al. onstrut ontrats aross different domains, suh as Linear Temporal Logi and Signal Temporal Logi [38], but for multi-layer ontroller synthesis. Cross-Domain Reasoning. Some researh explores reasoning aross domains by abstrating the system from one domain into the other, where all the reasoning is performed. For example, ontinuous systems ontrolled by periodi software ontrollers are analyzed by onverting the ontinuous dynamis to equivalent software ode whih advanes physial variables aording to the solutions of the differential equations, whih may not always be available [4]. The system is then analyzed using off-the-shelf software verifiers, whih may not sale over long time horizons. Alternatively, the ontinuous dynamis is abstrated by maneuver automata [6], whih are finite state mahines with timing information, desribing both trim onditions and transitions between them. Suh models an be used to synthesize distributed ontrol strategies using SMT solvers [40]. Combined models with imperative semantis for programs and differential equations has also been proposed [0], but their formal analysis remains diffiult. Symboli exeution of C software has been used to generate ounter-examples for hybrid systems that explore all exeution paths [43]. The StarL framework [35] ontains primitives, speifiations, and Java ode, that an be omposed and reasoned manually with the PVS theorem prover. However, the ode itself is not proven to onform to the formal PVS speifiations.

10 7. CONCLUSION We presented a method to verify end-to-end safety properties of distributed CPSs. The ruial step was proving yber-physial properties, whih required reasoning over a ombined software system and a hybrid automaton model of the low-level ontroller and plant. We used a ontrat automaton (CA) to formally desribe the orret behavior of the appliation (in terms of legal sequene of API funtion alls and their pre-post-onditions and return values) and the ontroller (in terms of invariants maintained by its ontinuous dynamis). A sound assume-guarantee style proof rule was used to deompose the verifiation into two parts one that verifies the appliation against the CA using software model heking, and another that verifies the ontroller against the CA using hybrid systems reahability analysis. The approah avoids the omposition of disrete (appliation) and ontinuous (ontroller) behavior, ameliorating state-spae explosion. It also permits the use of domain-speifi (software and hybrid automata) speialized verifiation tools. The subsequent domain-speifi analysis is simpler than the original ombined CPS analysis. We used our approah to verify physial ollision avoidane between a group of ommuniating quadopters in a -d spae. Our end-to-end proof is entirely performed using formal verifiation tools, exept for syntati translations of properties along the tool boundaries, whih ould be automated. 8. REFERENCES [] R. Alur, C. Couroubetis, N. Halbwahs, T. A. Henzinger, P.-H. Ho, X. Niollin, A. Olivero, J. Sifakis, and S. Yovine. The algorithmi analysis of hybrid systems. Theoretial Computer Siene, 38, 995. [] Y. Annpureddy, C. Liu, G. Fainekos, and S. Sankaranarayanan. S-taliro: A tool for temporal logi falsifiation for hybrid systems. Springer, 0. [3] L. Aştefănoaei, S. Bensalem, and M. Bozga. A ompositional approah to the verifiation of hybrid systems. In Theory and Pratie of Formal Methods. Springer, 06. [4] S. Bak, S. Bogomolov, and T. T. Johnson. HyST: A soure transformation and translation tool for hybrid automaton models. In Pro. of HSCC, 05. [5] S. Bak, Z. Huang, F. A. T. Abad, and M. Caamo. Safety and progress for distributed yber-physial systems with unreliable ommuniation. ACM Trans. Embed. Comp. Sys., 4(4), 05. [6] S. Bak and T. T. Johnson. Periodially-sheduled ontroller analysis using hybrid systems reahability and ontinuization. In Pro. of RTSS, 05. [7] T. Ball and S. K. Rajamani. Automatially Validating Temporal Safety Properties of Interfaes. In Pro. of SPIN, 00. [8] L. Benvenuti, A. Ferrari, L. Mangerua, E. Mazzi, R. Passerone, and C. Sofronis. A ontrat-based formalism for the speifiation of heterogeneous systems. In Speifiation, Verifiation and Design Languages, 008. FDL 008. Forum on, pages 4 47, Sept 008. [9] S. Bogomolov, G. Frehse, M. Greitshus, R. Grosu, C. S. Pasareanu, A. Podelski, and T. Strump. Assume-Guarantee Abstration Refinement Meets Hybrid Systems. In Pro. of HVC, 04. [0] O. Bouissou. From ontrol-ommand synhronous programs to hybrid automata. In Pro. of ADHS, 0. [] S. Chaki and J. Edmondson. Model-Driven Verifying Compilation of Synhronous Distributed Appliations. In Pro. of MODELS, 04. [] S. Chaki, A. Gurfinkel, and O. Strihman. Time-Bounded Analysis of Real-Time Systems. In Pro. of FMCAD, 0. [3] S. Chaki and D. Kyle. DMPL: Programming and verifying distributed mixed-synhrony and mixed-ritial software. Tehnial Report CMU/SEI-06-TR-005, 06. resoures.sei.mu.edu/library/asset-view.fm?assetid= [4] S. Chaki and N. Sinha. Assume-Guarantee Reasoning for Deadlok. In Pro. of FMCAD, 006. [5] X. Chen, E. Ábrahám, and S. Sankaranarayanan. Flow*: An analyzer for non-linear hybrid systems. In Pro. of CAV, 03. [6] A. Cimatti and S. Tonetta. Contrats-refinement proof system for omponent-based embedded systems. Si. Comput. Program., 97: , 05. [7] E. Clarke, D. Kroening, and F. Lerda. A Tool for Cheking ANSI-C Programs. In TACAS, 004. [8] E. Clarke, D. Long, and K. MMillan. Compositional model heking. In Pro. of LICS, 989. [9] J. M. Cobleigh, D. Giannakopoulou, and C. S. Păsăreanu. Learning Assumptions for Compositional Verifiation. In Pro. of TACAS, 003. [0] L. Cordeiro and B. Fisher. Verifying multi-threaded software using smt-based ontext-bounded model heking. In Pro. of ICSE. Assoiation for Computing Mahinery, 0. [] L. de Alfaro, T. A. Henzinger, and R. Jhala. Compositional Methods for Probabilisti Systems. In Pro. of CONCUR, 00. [] L. M. de Moura and N. Bjørner. Z3: An Effiient SMT Solver. In Pro. of TACAS, 008. [3] A. Donzé. Breah, a toolbox for verifiation and parameter synthesis of hybrid systems. In Pro. of CAV, 00. [4] P. S. Duggirala and M. Viswanathan. Analyzing real time linear ontrol systems using software verifiation. In Pro. of RTSS, 05. [5] J. R. Edmondson and A. S. Gokhale. Design of a Salable Reasoning Engine for Distributed, Real-Time and Embedded Systems. In Pro. of KSEM, 0. [6] E. Frazzoli, M. A. Dahleh, and E. Feron. Maneuver-based motion planning for nonlinear systems with symmetries. Robotis, IEEE Transations on, (6), 005. [7] G. Frehse. Compositional verifiation of hybrid systems with disrete interation using simulation relations. In Pro. of CACSD, 004. [8] G. Frehse, C. L. Guerni, A. Donzé, S. Cotton, R. Ray, O. Lebeltel, R. Ripado, A. Girard, T. Dang, and O. Maler. SpaeEx: Salable Verifiation of Hybrid Systems. In Pro. of CAV, 0. [9] T. A. Henzinger. The Theory of Hybrid Automata. In Pro. of LICS, 996. [30] Z. Huang and S. Mitra. Proofs from simulations and modular annotations. In Pro. of HSCC, 04. [3] R. Jhala and R. Majumdar. Software model heking. ACM Computing Surveys (CSUR), 4(4), 009. [3] A. John, I. Konnov, U. Shmid, H. Veith, and J. Widder. Parameterized model heking of fault-tolerant distributed algorithms by abstration. In Pro. of FMCAD, 03. [33] C. B. Jones. Speifiation and Design of (Parallel) Programs. In Pro. of 9th IFIP World Congress, 983. [34] A. Lal and T. W. Reps. Reduing Conurrent Analysis Under a Context Bound to Sequential Analysis. In Pro. of CAV, 008. [35] Y. Lin and S. Mitra. Starl: Towards a unified framework for programming, simulating and verifying distributed roboti systems. In Pro. of LCTES, 05. [36] C. Liu and J. Layland. Sheduling algorithms for multiprogramming in a hard-real-time environment. JACM, 0(), 973. [37] J. Misra and K. M. Chandy. Proofs of Networks of Proesses. TSE, 7(4), 98. [38] P. Nuzzo, H. Xu, N. Ozay, J. B. Finn, A. L. Sangiovanni-Vinentelli, R. M. Murray, A. DonzÃl, and S. A. Seshia. A ontrat-based methodology for airraft eletri power system design. IEEE Aess, : 5, 04. [39] A. Pnueli. In Transition from Global to Modular Temporal Reasoning About Programs. Logis and Models of Conurrent Systems, 3, 985. [40] I. Saha, R. Ramaithitima, V. Kumar, G. J. Pappas, and S. A. Seshia. Automated omposition of motion primitives for multi-robot systems from safe ltl speifiations. In Pro. of IROS, 04. [4] A. University and U. University. Uppaal - a tool suite for verifiation of real-time systems [4] M. W. Whalen, A. Gaek, D. Cofer, A. Murugesan, M. P. E. Heimdahl, and S. Rayadurgam. Your what is my how: Iteration and hierarhy in system design. IEEE Software, 30():54 60, Marh 03. [43] A. Zutshi, S. Sankaranarayanan, J. Deshmukh, and X. Jin. Symboli-numeri reahability analysis of losed-loop ontrol software. In Pro. of HSCC, 06.