Distributed Real-Time and Embedded Systems

Size: px

Start display at page:

Download "Distributed Real-Time and Embedded Systems"

Shanna Price
6 years ago
Views:

Context Deadlock voidance for Goal: Formalization of Middleware Services Distributed Real-Time

r V e n k i t a U n i v e r s i t y a t N o v 7, 6 S t pplication Middleware J o i n t w o r k w

t o n pplication Real-time Event Channel Event Filtering Deadlock avoidance Event Correlation OS

Deadlock is a classical problem in Computer Science Deadlock is the situation in which resources

Conditions:.Mutual exclusion.hold and wait.no preemption 4.

1 Context Deadlock voidance for Goal: Formalization of Middleware Services Distributed Real-Time and Embedded Systems pplication H e n n y S i p m a S t a n f o r d U n i v e r s i t y C e s a r V e n k i t a U n i v e r s i t y a t N o v 7, 6 S t pplication Middleware J o i n t w o r k w i t h S a n c h e z, z o h a r m a n n a, S u b r a m o n i a n, C h r i s G i l l Wa s h i n g t o n pplication Real-time Event Channel Event Filtering Deadlock avoidance Event Correlation OS L o u i s OS OS OS4 Scheduler OS5 OS6 Washington University, November 7, 6 Motivation Deadlock Deadlock is a classical problem in Computer Science Deadlock is the situation in which resources have been allocated to various tasks in such a way that none of the tasks can continue. Conditions:.Mutual exclusion.hold and wait.no preemption 4.Circular wait Habermann, 969 Classical example: Dining philosophers Washington University, November 7, 6 4 Washington University, November 7, 6 4

2 Solutions: Deadlock Detection Deadlock Prevention pproach: pproach! Statically break the circular wait Make one of the components release its resources and go back to an earlier state Disadvantages: 4 Sacrifices concurrency and reduces resource utilization Problem: Common in databases, but not practical in embedded systems Burden on the programmer 5 Washington University, November 7, 6 Washington University, November 7, Deadlock avoidance problem space Solutions: Deadlock voidance CENTRLIZED pproach:? DISTRIBUTED Unsolvable Dynamically assign resources on request based on availability and other processes needs Unsolvable Max utilization Impractical [Dijkstra, 65] [Singhal, 95] Call sequences [delfaro, 5] FMS [Sanchez++,5,6] Dijkstra s Banker s algorithm 7 Washington University, November 7, Washington University, November 7, 6 8

Solutions objective method centralized distributed prevention detection " " " " Banker s algorithm: maximum # resources avoidance Flexible manufacturing systems: # resources + order of request 9

3 Solutions objective method centralized distributed prevention detection " " " " Banker s algorithm: maximum # resources avoidance Flexible manufacturing systems: # resources + order of request 9 Design a deadlock avoidance protocol that! is local to each component (does not require any communication between components) impractical in general [Singhal95]! makes use of static information (call graphs)! allows maximum concurrency! can be implemented efficiently practical with more a priori knowledge of process structure: Call Graphs [Sanchez++ 5,6,6,6] Washington University, November 7, 6 Washington University, November 7, 6 9 outline distributed real-time embedded system! Introduction! Deadlock avoidance protocols! Liveness! Distributed priority inheritance! Related Work! Conclusions n n4 B n distributed components, B, C, D, E n C E n5 D n B Sequence of calls Washington University, November 7, 6 n n C n4 n5 E Washington University, November 7, 6

4 distributed real-time embedded system example of deadlock ssumptions: Two components with two threads each! synchronous distributed system! Limited resources! Remote method invocations! WaitOnConnection policy! rbitrary number of processes spawned! ll processes terminate Two call graphs B n n B m B Resources are threads or execution contexts Washington University, November 7, 6 4 Distributed system model system! Introduction! Deadlock avoidance protocols! Liveness! Distributed priority inheritance! Related Work! Conclusions Each component maintains local variables including T: (constant) total number of threads t: number of threads currently available! G: {G,G,...}, a set of acyclic call graphs m B n B n 4 outline S: <R,G>! R: {,B,...}, a set of components. Washington University, November 7, 6 n C n4 n5 E 5 Washington University, November 7, Washington University, November 7, 6 6

5 Deadlock avoidance protocol annotated call graph Protocol schema: (executed for every incoming method call) when Enabledn(V) do Inn(V,V ) n:: f() m B n B entry section n n! Global call graph: G = G C n5 E n4 method invocation Outn(V,V ) exit section annotation represents the minimum number of threads that should be reserved for other processes (executing the same or other call graph) : update to local variables if thread is granted Outn(V,V ) : update to local variables when thread is released 7... = < V,E> G! nnotation # : V $!, mapping nodes to natural numbers EnablednV) : condition evaluated over local variables of if true: a thread can be granted Inn(V,V ) Washington University, November 7, 6 8 Washington University, November 7, protocol basic-p n:: protocol basic-p when Enabledn(V) do Inn(V,V ) entry section f() method invocation Outn(V,V ) when t > #(n) do t++ n:: : t++ f() method invocation t-- exit section EnablednV) : t > #(n) Inn(V,V ) entry section exit section Question: What conditions must # satisfy to ensure deadlock freedom? t: number of currently available threads Outn(V,V ) : t- 9 Washington University, November 7, 6 9 Washington University, November 7, 6

6 annotated call graph annotation theorem nswer: # must have no cyclic dependencies Example: n n B m B m when t > #(n) do t++ Node v directly depends on node w if n::! v makes a method call to w, or! v, w run in the same component and Basic-P f() t-- #(v) % #(w) Node v depends on node w if Theorem Let # be an annotation of the global call graph. If Basic-P is used in all components to decide allocation of threads, and # is acyclic, then the system is deadlock free.! there is a path from v to w that includes at least one! is acyclic if! no node depends on itself Washington University, November 7, 6 Washington University, November 7, 6 annotation theorem is tight annotation theorem is tight: example n C! nnotation Theorem provides a sufficient condition for deadlock freedom! Question: Is it also a necessary condition? m Theorem Let # be an annotation of the global call graph. If Basic-P is used in all components to decide allocation of threads, and! is not acyclic, then, given sufficient resources, a deadlock is always reachable. C n n B m B No deadlocks are reachable with But, deadlocks are reachable with TC= TC = Increasing the total number of threads may turn a deadlock-free system into a system that is not deadlock-free Checking whether a system with a cyclic annotation is deadlock-free is NP-complete Washington University, November 7, 6 4 Washington University, November 7, 6 4

7 how to compute an acyclic annotation?! Simplest solution: height in the graph 4 n D n n C n B n4 How to compute an acyclic annotation?! Not a solution: local height in the graph n5 E n n B m B T = 4 TB = TC = TD = 5 TE = Minimum number of threads required:! Need something in between Wastes resources! Washington University, November 7, How to compute an acyclic annotation?! Introduction! Deadlock avoidance protocols " Liveness! Distributed priority inheritance! Related Work! Conclusions Fix some reverse topological order on the nodes For each node n in the order compute the set of nodes S reachable via ($ &)* set #(n) = + #(m) where m S and m resides in the same component as n! No unique minimal solution B B 7 6 outline! Polynomial-time algorithm B Washington University, November 7, 6 B Washington University, November 7, Washington University, November 7, 6 8

8 Basic-P: shortcomings Basic-P: shortcomings Basic-P may cause starvation: Basic-P may cause starvation: G G G n n B m B B G continuous stream of processes executing G will stop processes executing G forever 9 n n B m B B continuous stream of processes executing G will stop processes executing G forever Washington University, November 7, 6 Washington University, November 7, 6 9 Liveness under Fair scheduler Essential invariant For deadlock avoidance it is sufficient to maintain the invariant: {... p... } llocation Manager for every component, for every i: p Scheduler t[i] ' T - i t[i]: number of active processes in with annotation i or greater llocation manager determines set of processes for which taking the resource is safe " Maintaining this invariant precisely requires T storage space Scheduler selects the process that gets the resource " has only one variable; it maintains the invariant by always assuming the worst case Objective: Design protocol for allocation manager that provides liveness provided the scheduler is fair " Question: can we maintain it precisely? What do we gain? Washington University, November 7, 6 Washington University, November 7, 6

9 versus when ((n) do allocate(n) n:: when t > #(n) do t++ f() f() deallocate(n) t-- with ((n): i>#(n). t[i] ' T - i # allocate(n): i'#(n). t[i]++ deallocate(n): i'#(n). t[i]-- i'#(n). t[i] +' T - i t[] t[] t[] t[] t Washington University, November 7, 6 Washington University, November 7, 6 4 versus 4 versus t[] t[] t[] t[] t[] t {} {} t[] t[] t {} 5 t[] {} Washington University, November 7, Washington University, November 7, 6 6

10 versus versus t[] t[] t[] t[] t[] t t[] t[] t[] t {,} {,} {,} {,} Washington University, November 7, Washington University, November 7, 6 7 versus! 8 allows more concurrency than Theorem (Deadlock avoidance): If protocol is used to determine whether thread allocation is safe and # is acyclic then no deadlock is reachable. t[] t[] t[] t[] Theorem (Liveness) If protocol is used to select a process for thread allocation then every waiting process is eventually selected t {,,} {,} 9 Washington University, November 7, Washington University, November 7, 6 4

11 versus versus n n B m B n n B m B t[] t[] t[] t Washington University, November 7, 6 4 t[] t {m} {m} Washington University, November 7, versus versus n 4 n n B m B n n! t[] t[] {m} 4 m B t {m} n B t[] n t[] t {n,m} {m} Washington University, November 7, Washington University, November 7, 6 44

12 versus versus n n n B m B! n n n B m B t[] t[] t[] t {n,m} {m} {m,m} Washington University, November 7, 6 45 t {n,m} t[] Washington University, November 7, versus versus leaves n 46 n n B m B leaves n t[] 47 m B t[] t {n,m} n B t[] n t[] t {m,m} {n} {m} Washington University, November 7, Washington University, November 7, 6 48

13 versus versus n n n B m B n t[] t[] t[] {n} {m} {n} Washington University, November 7, 6 49 m B t n B t[] n t {m} - {m,m} - {m} - {m,m}... 5 Washington University, November 7, Basic-P vs Live-P: concurrency when ((n) do allocate(n) verage maximum legal annotation, for T = and varying load Maximum Legal nnotation n:: 8 when t > #(n) do t++ f() f() deallocate(n) 6 t-- with 4 ((n): i>#(n). t[i] ' T - i # allocate(n): i'#(n). t[i]++ deallocate(n): i'#(n). t[i]-- i'#(n). t[i] +' T - i Load 5 Washington University, November 7, Washington University, November 7, 6 5

14 Live-P: implementation! Basic-P : Experiments () one instruction per allocation Time for 6 operation pairs, for uniform α and L = 9 rray 8 Complete RedBlack 7 data structure time space array O(T) O(T) O(log T) O(T) binary tree Time(ms)! Live-P : red-black tree O(log L) O(L) T: total number of threads in the component L: total number of processes executing in the component Total number of resources T Washington University, November 7, 6 5 Washington University, November 7, Experiments () 54 versus Time for 6 operation pairs, for uniform α and L = T 4 9 Time(ms) rray 8 Complete RedBlack 7! t[] t[] t[] t[] t Total number of resources T 55 {,,} {,} Washington University, November 7, Washington University, November 7, 6 56

versus reachable state space allocation sequences allocation sequences t[] t[] t[] t[] {,,} -- t {,,} 57 -- -- -- -- -- state {,,} state space -- Washington University, November 7, 6 58 Washington

15 versus reachable state space allocation sequences allocation sequences t[] t[] t[] t[] {,,} -- t {,,} state {,,} state space -- Washington University, November 7, 6 58 Washington University, November 7, 6 57 intermediate protocols 58 mixing Protocols allocation sequences Theorem: Every allocation protocol that! is enabled whenever is enabled, and! is disabled whenever is disabled k- state space guarantees freedom from deadlock k- k- for k =..T : " k storage space " guarantees liveness up to annotation k- 59 Washington University, November 7, Washington University, November 7, 6 6

16 varying levels of concurrency outline! Introduction! Deadlock avoidance protocols " Liveness! Distributed priority inheritance! Related Work! Conclusions 6 Washington University, November 7, 6 Washington University, November 7, motivation 6 Priority inheritance! It is common in DRE systems to assign priorities to processes to ensure " Priority inheritance requires a decrease in annotation that more critical tasks meet their deadlines! Schedulers then give preference to higher-priority tasks! Possibility of priority inversion! Priority inheritance protocols help alleviate priority inversions! Distributed priority inheritance hard to achieve (n,) (m,) B (r,) (m,) B (r,) B (r,) B (r,) (r,) B " Deadlock avoidance is preserved, because new state is reachable by 6 Washington University, November 7, Washington University, November 7, 6 64

17 outline Related work Classical deadlock avoidance:! Introduction! Deadlock avoidance protocols " Liveness! Distributed priority inheritance " Related Work! Conclusions! E.W. Dijkstra, Cooperating sequential processes, EWD-, 965.!.N. Habermann, Prevention of system deadlocks, CCM, 969.! J.W. Havender, voiding deadlocks in multi-tasking systems, IBM Systems Journal, 968.! R.C. Holt, Some deadlock properties of computing systems, CM Computer Surveys, 97.! E.C. Coffman, M.J. Elphick, and. Shoshani, System deadlocks, Computing Surveys, Washington University, November 7, 6 66 Washington University, November 7, 6 65 Related work Related work Flexible Manufacturing Systems: Middleware and distributed systems:! S..Reveliotics, M.. Lawley, P.M. Ferreira, Polynomial complexity! D.C. Schmidt, M. Stal, H. Rohnert, F. Bushmann, Pattern-oriented deadlock avoidance policies for sequential RS, IEEE Transactions on utomatic Control, 997.! K.Y. Xing, B.S. Hu, H.X. Chen, Deadlock avoidance policy for petri net modeling of FMSs with shared resources, IEEE Transactions on utomatic Control, 996.! M.P. Fanti, B. Maione, S. Mascolo, B. Turchiano, Event-based feedback control for deadlock avoidance in flexible production systems, IEEE Transactions on Robotics and utomation, 997.! M.V. Iordache, J.G. Moody, P.J. ntsaklis, Synthesis of deadlock prevention supervisors using petri nets, IEEE Transactions on Robotics and utomation,.! J. Ezpeleta, F. Tricas, F. Garcia-Valles, J.M. Colom, Banker s solution for deadlock avoidance in FMS with flexible routing and multiresource states, IEEE Transactions on Robotics and utomation, software architecture: Patterns for concurrent and networked objects, Volume,.! V. Subramonian and C.D. Gill, generative programming framework for adaptive middleware, HICSS, 4.! D.C. Schmidt, S. Munfee, S. Flores-Gaitan,.S. Gokhale, lleviating priority inversion and non-determinism in real-time CORB ORB core architectures, RTS, 998.! L. de lfaro, V. Raman, M. Faella, R. Majumdar, Code aware resource management, EMSOFT, 5.! F. Muller, Priority inheritance and ceilings for distributed mutual exclusion, RTSS, 999. Washington University, November 7, Washington University, November 7, 6 68

18 outline Conclusions! Introduction! Deadlock avoidance protocols " Liveness! Distributed priority inheritance! Related Work " Conclusions "... : a spectrum of deadlock avoidance protocols that! require no communication between components! make use of static information of the processes! allow varying levels of concurrency! provide varying levels of liveness! vary in complexity of implementation and space required! support priority inheritance " Proofs of correctness based on parameterized transition systems " Experimentation in CE in progress " Partial results for recursive call graphs 69 Washington University, November 7, 6 7 Washington University, November 7, 6 69 papers 7 future work! C. Sanchez, H.B. Sipma, Z. Manna, C.D. Gill, V. Subramonian, Thread allocation protocols for distributed real-time and embedded systems, FORTE, 5.! C. Sanchez, H.B. Sipma, Z. Manna, V. Subramonian, C.D. Gill, On efficient distributed deadlock avoidance for real-time and embedded systems, IPDPS, 6.! C. Sanchez, H.B. Sipma, Z. Manna, C.D. Gill, Efficient distributed deadlock avoidance with liveness guarantees, EMSOFT, 6.! V. Subramonian, C.D. Gill, C. Sanchez, H.B. Sipma, Reusable models for timing and liveness analysis of middleware for distributed real-time embedded systems, EMSOFT 6.! C. Sanchez, H.B. Sipma, C.D. Gill, Z. Manna, Distributed priority inheritance for real-time and embedded systems, OPODIS, 6.! C. Sanchez, H.B. Sipma, Z. Manna, Reachable state spaces of distributed deadlock avoidance protocols, submitted. 7! Formal characterization and measurement of levels of concurrency allowed by the different protocols! Process placement to obtain optimal concurrency! Incorporation of more information on the call graph to increase concurrency! Static analysis methods to obtain call graphs! Mixed WaitOnConnection - WaitOnReactor allocation policy! Integration with scheduling Washington University, November 7, Washington University, November 7, 6 7

On Efficient Deadlock Avoidance for Distributive Recursive Processes

On Efficient Deadlock Avoidance for Distributive Recursive Processes César Sánchez, Henny B. Sipma, and Zohar Manna Computer Science Department Stanford University, Stanford, CA 94305, USA {cesar,sipma,zm}@cs.stanford.edu