ccopyright by Aamod Arvind Sane 1998

Transcription

1 TECHNIQUES FOR DEVELOPING CORRECT, FAST, AND ROBUST IMPLEMENTATIONS OF DISTRIBUTED PROTOCOLS BY AAMOD ARVIND SANE THESIS Submitted in partial fulllment of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate College of the University of Illinois at Urbana-Champaign, 1998 Urbana, Illinois

2 ccopyright by Aamod Arvind Sane 1998

3 TECHNIQUES FOR DEVELOPING CORRECT, FAST, AND ROBUST IMPLEMENTATIONS OF DISTRIBUTED PROTOCOLS Aamod Arvind Sane, Ph.D. Department of Computer Science University of Illinois at Urbana-Champaign, 1998 Roy H. Campbell, Advisor A distributed system must satisfy three requirements: it should correctly implement process interactions to realize desired behavior, it should exhibit satisfactory performance, and it should have a robust software architecture that accommodates changing requirements. This thesis presents research that addresses each of these concerns. The thesis presents new techniques for designing protocols that coordinate process interactions. The specication technique allows designers to design protocols by topdown renement. Renement steps divide the original protocol into sub-protocols that have smaller state spaces than the original protocol. Therefore, the divided protocols can be automatically veried without encountering state-space explosion. The complete protocol is synthesized by composing the divided protocols. The thesis also shows how protocols can be tailored for improved performance. A new technique for designing high-performance distributed shared memory consistency protocols is presented. The technique optimizes consistency protocols by using information about previous memory accesses to anticipate future communication. Such anticipation allows communication to overlap with computation, resulting in improved application performance. iii

4 Finally, the thesis presents a software architecture for implementing systems with interacting distributed objects. The architecture allows systems to be incrementally extended with new objects and new operations, including operations over objects on remote systems. This is achieved using design patterns, and a novel scheme for incremental construction of state machines. The architecture was used to build a virtual memory system that is smoothly extended to support distributed shared memory. iv

5 TABLE OF CONTENTS Chapter 1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Contributions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Thesis Outline : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 4 2 A Protocol Design Technique : : : : : : : : : : : : : : : : : : : : : : : : : : : Goal : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : The Problem : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Our Solution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Background and Related Work : : : : : : : : : : : : : : : : : : : : : : : : Verication Systems : : : : : : : : : : : : : : : : : : : : : : : : : High-Level Service Specication : : : : : : : : : : : : : : : : : : : Synthesis Methods : : : : : : : : : : : : : : : : : : : : : : : : : : Our Approach : : : : : : : : : : : : : : : : : : : : : : : : : : : : : The Synthesis Method : : : : : : : : : : : : : : : : : : : : : : : : : : : : Synthesis : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Process and System : : : : : : : : : : : : : : : : : : : : : : : : : : Automata : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Automata and Processes : : : : : : : : : : : : : : : : : : : : : : : Protocols : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Protocol Synthesis : : : : : : : : : : : : : : : : : : : : : : : : : : Specifying Coordination : : : : : : : : : : : : : : : : : : : : : : : : : : : Constraint-Rule Specications : : : : : : : : : : : : : : : : : : : : Action-Rule Specications : : : : : : : : : : : : : : : : : : : : : : Observation-Rule Specications : : : : : : : : : : : : : : : : : : : Proving Implementation : : : : : : : : : : : : : : : : : : : : : : : Implementing Constraints, Actions, and Observations : : : : : : : : : : : Synthesizing Constraint Rules : : : : : : : : : : : : : : : : : : : : Synthesizing Action Rules : : : : : : : : : : : : : : : : : : : : : : Observations via Memory and Messages : : : : : : : : : : : : : : Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 45 v

6 3 Distributed Shared Memory Consistency Protocols : : : : : : : : : : : : : : : Goal : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : The Problem : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Our Solution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Background and Related Work : : : : : : : : : : : : : : : : : : : : : : : : Sequential Consistency : : : : : : : : : : : : : : : : : : : : : : : : Beyond Sequential Consistency : : : : : : : : : : : : : : : : : : : Synchronization in Distributed Shared Memory : : : : : : : : : : Our Approach : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Coordinated Memory : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Adaptive Barriers : : : : : : : : : : : : : : : : : : : : : : : : : : : Other Adaptive Constructs : : : : : : : : : : : : : : : : : : : : : Designing Consistency Protocols : : : : : : : : : : : : : : : : : : : : : : : Consistency Specication : : : : : : : : : : : : : : : : : : : : : : : Adaptive Barrier : : : : : : : : : : : : : : : : : : : : : : : : : : : Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Implementation and Performance : : : : : : : : : : : : : : : : : : : : : : Experimental Platform : : : : : : : : : : : : : : : : : : : : : : : : Applications : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 77 4 A Software Architecture : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Goal : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : The Problem : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Our Solution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Background and Related Work : : : : : : : : : : : : : : : : : : : : : : : : Basic Objects : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Interactions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Operations : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Why the New Architecture : : : : : : : : : : : : : : : : : : : : : : : : : : Examples : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Why Change is not Easy : : : : : : : : : : : : : : : : : : : : : : : What Needs to be Redesigned : : : : : : : : : : : : : : : : : : : : : : : : Data Structures and Synchronization : : : : : : : : : : : : : : : : Interactions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : A Solution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Architecture of the Virtual Memory System : : : : : : : : : : : : : : : : Exporting Functionality : : : : : : : : : : : : : : : : : : : : : : : Organizing the Internals : : : : : : : : : : : : : : : : : : : : : : : Concurrency Control : : : : : : : : : : : : : : : : : : : : : : : : : Operations Using Object-Oriented State Machines : : : : : : : : : Implementing Remote Interactions : : : : : : : : : : : : : : : : : Dynamic Page Distribution : : : : : : : : : : : : : : : : : : : : : 108 vi

7 4.6 Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Conclusion : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Future Research : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 114 Bibliography : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 115 vii

8 Chapter 1 Introduction This thesis presents techniques for the design and implementation of protocols that coordinate the actions of concurrent processes in a distributed system. The design of novel memory consistency protocols for a distributed shared memory system illustrates the application of these techniques. The system is implemented using a new software architecture for designing object-oriented systems with concurrent and distributed operations. Protocols are dicult to design because systems of interacting concurrent processes exhibit a large number of behaviors. Therefore, computer-aided methods are used for protocol design. Currently, such methods can be classied into either verication methods or synthesis methods. Verication methods let users model the protocols in a suitable language, and check that model obeys desired properties by exhaustive search ofthe system state space. But the detailed, low-level models often result in very large state spaces. The search is made tractable by exploiting patterns in the state space to reduce the states actually examined. Even so, many practical protocols remain beyond the reach of exhaustive search. Synthesis methods avoid building complex low-level models. Instead, they translate high-level specications to low-level implementations. But these methods often require manual proofs, or are useful only in restricted cases such as peerto-peer communication protocols. Ideally, wewould like a design method that combines the clarity of high-level specications of synthesis methods with the automated checking characteristic of verication methods. 1

9 In this thesis, we develop such a design method. We introduce an approach for dividing the task of protocol design into several steps. The division produces protocols that have small state spaces either because they are abstract or because they implement parts of the original protocol. Therefore, their correctness can be easily established using verication tools. We then show how to implement the divided protocols so that the complete protocol can be synthesized by combining the divided protocols. We have applied the synthesis method to guide the implementation of new distributed shared memory consistency protocols. A distributed shared memory (DSM) system simulates shared memory over networked computers. DSM systems allow programs designed for shared memory multiprocessors to be used over networked computers. DSM systems use local memories of the networked computers as caches for the simulated shared memory. Just like shared memory multiprocessors, caches in DSM systems replicate the shared data for eciency, but then require protocols to ensure that the replicas remain consistent. In this thesis, we develop consistency protocols that allow DSM systems to operate eciently over wide-area networks characterized by high-latency high-bandwidth interconnections. A protocol that performs well over a wide-area network must be able to utilize the bandwidth to overcome latency. Our protocols gain their eciency using information about process synchronization and past memory access patterns to predict future requests from other processes. This technique reduces the time processes spend waiting for data to arrive. When computations are regular, this anticipatory communication overlaps communication and computation, giving good speedups for distributed shared memory programs over wide-area networks. Protocol implementations derived by our method are state machines that dene protocol behavior. However, the programmer is still left to manage a myriad details of the implementation environment. In our case, the protocol implementation has to be a part of a virtual memory system that supports distributed shared memory. 2

10 In this thesis, we present asoftware architecture for building object-oriented systems that have many concurrent operations on groups of objects. The architecture allows the system to be incrementally extended with new objects and new operations. It smoothly implements interactions between objects on remote systems. In the course of designing the architecture, we have discovered several design patterns, and a new technique for constructing state machines incrementally using an object-oriented approach. The architecture is used to build a virtual memory system. The resulting system is exible: beginning with simple virtual memory facilities, we extended it with facilities like distributed shared memory in an orderly manner. 1.1 Contributions This thesis makes the following contributions: A method for designing process coordination protocols based on { A family of notations to express protocols at dierent levels of abstractions. { A set of transformations to rene protocols from one level to the next. { Application of the method to design memory consistency protocols. Distributed shared memory consistency protocols that { Improve over the performance of existing protocols { Perform well over either wide-area and local-area networks. A software architecture for object systems with concurrent operations on groups of objects. The architecture is based on: { Object-oriented state machines that facilitate construction of state machines by inheritance, composition and other object-oriented techniques. { Design patterns that simplify concurrency control, remote interactions, and resource management. 3

11 1.2 Thesis Outline In Chapter 2 we present our method for synthesizing distributed shared memory protocols. We begin with a review of background and related work and identify our contribution. Next, we chapter present the basic theory and discusses the notations we use at dierent levels of abstraction. After that we present a set of transformations for synthesizing the protocol implementation from a specication. We then show howinterpret the implementation as a shared memory or message passing program. In Chapter 3, we develop our consistency protocols. We present the evolution of consistency protocols, and highlight our approach. Then we explain and formally specify our protocols and comment on the implementation. We conclude this part with performance results. In Chapter 4, we present our new software architecture. We use the design of a virtual memory as the primary example. First we explain the usual architecture of virtual memory systems, motivating the basic objects and operations. Then we critique it by considering the impact of changes, and motivate the new architecture. The architecture is discussed subsequently. In Chapter 5, we review the contributions and identify problems for future research. 4

12 Chapter 2 A Protocol Design Technique In this chapter, we present a new technique for designing nite state process coordination protocols. We begin by presenting the problem and our solution in brief. Then we examine the background research in detail, and contrast our solution with it. The rest of the chapter presents the formal details. 2.1 Goal The Problem Protocols that describe the behavior of systems with concurrent interacting components are dicult to design, because such systems exhibit a large variety of behaviors. Ahu- man designer may overlook undesirable interactions in the system, leading to errors such as deadlock. So an automated method for synthesizing such systems is highly desirable. There are two types of approaches for computer-aided protocol design, verication methods and synthesis methods. Verication methods help debug previously designed protocols by exhaustive search of state spaces, while synthesis methods start with protocol specications and translate them to low-level protocol implementations. We used these methods in our research for designing distributed shared memory consistency protocols. These protocols describe systems that have avery large number 5

13 of states. Therefore, we could only verify simplied versions of the protocols. We also attempted to use synthesis methods. But methods that had tool support are designed for synthesizing peer-to-peer communication protocols, or OSI protocol stacks. Synthesis by hand, based on specications with algebraic or logical languages that could describe multi-party protocols, requires manual proofs of the specication. Such proofs were practical only for simple versions of the protocol. Thus, while verication methods are applicable to a wide class of protocols, they are limited by the need to describe protocols in detail, as well as the limitations of exhaustive search. On the other hand, synthesis methods provide abstract protocol description languages, but the methods require manual correctness proofs. Also, the abstract descriptions can be more dicult to produce than low-level descriptions based on communicating automata. This experience suggested the need for a design method that could combine the desirable attributes of both verication and synthesis methods Our Solution We introduce an approach for dividing the task of protocol design into several steps. At each step, the protocol is simple enough that exhaustive search is tractable, so that verication tools can be used to establish correctness. The simplication is achieved using notations that support abstraction and decomposition. We introduce a family of notations that are all communicating automata, except that the communication is expressed at dierent degrees of abstraction. We chose notations based on communicating automata because communicating automata are a familiar model used in popular verication tools. In the rst step, a designer uses specications that express process coordination as abstract predicates that suppress details of communication and control. Whole system verication is done at this step. In the second step, the designer produces implementations of the predicates in a notation that expresses control but not the details of communication. Here we verify each predicate implementation separately. The design method requires the implementations to have certain properties 6

14 that permit composition so that the composite system does not have tobeveried again. In the third step, the predicate implementations are translated to protocols that express details of communication media. Again, these translations obey conditions that allow safe composition. Composing the translations terminates the protocol synthesis. We expand on these ideas in the following. Step 1 In our method, the initial design is specied by automata that describe only the desired coordination between automata, without saying how it is implemented. Thus, the initial models are extensional, abstract, and relatively simple. The notation we use formalizes a common way to describe process coordination. For example, mutual exclusion between two processes is often described as follows: \when one process is in its critical region, the other should not be in its critical region." Here, we divide the execution of a process into regions, and express coordination as a predicate on the regions of interacting processes. Our rst notation describes processes as automata, and coordination as predicates on their states. The notation suppresses details of how processes control each other to implement the predicates, as well as details of communication, and leads to models with small state spaces. We use verication tools to verify deadlock freedom, liveness, and other system properties. Step 2 The next step in a design is to show how to implement the predicates. Most distributed systems support communication mechanisms like message passing that allow one process to control another process unidirectionally. Bidirectional control is achieved by some form of request-response interaction. Our next model describes how a process controls another by unidirectional actions. At this level, we do not model elements like message queues. For example, consider a predicate over two process that says: \when one process enters region x, the other should enter y". This might be used to describe the process opening a TCP connection. One implementation might be: \First, the target waits for the connector request. Then the connector sends its request and waits for the reply". We formalize such a description with a notation where transitions in one process 7

15 may enable or disable transitions in another process. We use verication tools to ensure that such an implementations correctly implements a predicate. The original, abstract protocol may have several predicates. We establish conditions to ensure that implementations of the predicates can be composed without loss of correctness. Thus, the abstract protocol can be translated to the lower-level protocol by translating each predicate separately. Step 3 In the nal step, we use a notation that models communication media. The idea is that one process may \observe" the current state of another process and use the information to choose its transitions. An observation is easily implemented in a message passing system: a process can request the current state of another process, and wait for the response. Similarly, in a shared (or distributed) memory system, one process may observe the state of another process by reading shared (distributed) variables. Observation protocols are used to implement process control: a transition of process P that requires process Q to be in a certain state is disabled when Q is in a dierent state. We use verication tools to ensure that an observation protocol correctly implements process control, and hence the second-level protocols. Again, we establish conditions to ensure that implementations can be composed safely. These conditions carry over from the second-level protocols. Thus, the abstract protocol can be translated to observation-based protocols, and hence to shared memory and message passing programs Summary Our approach has several advantages. The notations based on communicating automata are familiar, and allow us to use popular protocol verication tools based on communicating automata. The design approach allows us to simplify protocols, rst by abstraction, and then by decomposition, so that the state space presented to a verier is smaller. The abstract predicates on processes can be implemented in various ways, and dierent im- 8

16 plementations can coexist in a synthesized protocol. Also, we can use known algorithms to implement the predicates, as long as we ensure that the composition conditions hold. In this thesis, we develop the formal basis for the approach. We have used it to design the consistency protocols for distributed shared memory, described in Chapter Background and Related Work We describe some of the previous research on protocol design. We then relate our approach to this work Verication Systems Verication systems (also called model checkers) such as SPIN [Hol91], SMV [McM92], and Mur' [Dil96] are designed to verify the correctness of nite state distributed protocols. Each verication system provides a language for a precise and understandable mathematical model of the system. For instance, SPIN uses communicating nite state machines [BZ83]. Another formal language allows the user to specify correctness predicates; the systems mentioned above use variants of temporal logic [MP91]. Temporal logic allows the user to express notions such as \if process P takes action a, Q will eventually respond with action b". The systems include algorithms that examine the complete state space of a system and verify that the state graph satises the correctness criteria [Hol91]. These systems have two drawbacks. First, the modeling languages are at a fairly low level (messages in SPIN, shared memory in SMV), so that constructing the models is tedious and error prone. Second, exhaustive exploration of the system state space can be intractable: this is the state explosion problem. Recent research has concentrated on developing techniques for checking correctness without exhaustive analysis, as well as methods for managing large state spaces in limited memory. Partial-order methods [WG93] attempt to eliminate states that arise from modeling concurrency as interleaving. If the state transitions of two processes are independent, 9

17 then in an execution of the system, the transitions may be permuted without aecting correctness of the execution. Therefore, instead of examining all permutations, we can examine the state space for an arbitrary permutation of independent transitions and still check correctness. Moreover, dependencies among the state transitions of a nite-state system can be approximated by examining the source code. Using such dependency information, partial order methods guide the search over a limited part of the system state space. Symbolic model checking [McM92] uses binary decision diagrams (BDDs) to represent the state space. The symbolic representation allows compact storage of a large state space. Algorithms that search the state space to verify correctness can be changed to operate directly on the BDD representation. BDDs work best for digital circuitry that has many replicated components. Traditional state exploration may outperform BDDs for distributed protocols [Hu95]. Fair reachability [GH85, GY84, LM95] methods force state transitions of processes in a distributed system and explore the resulting state space. This space is smaller than the state space generated when some processes do not take steps. The smaller space is sucient tocheck some properties like deadlocks. Abstraction [Lon93, PD97] Abstract Interpretation [Lon93], and Composition [Lon93] based approaches are developed to present model checkers with simpler systems to verify. These approaches use user-dened equivalence relations [Pon95] induction over replicated components [McM92], symmetry [Ip96], language containment [Kur94] and similar approaches to eliminate irrelevant states in a system. In methods based on abstraction, it is enough to check the abstract model to ensure that a property that holds in the abstracted system is really true of the actual system. Methods based on composition and simulation use theorems that show how to decompose system properties of interest when verifying components or simulations. All these methods require human intervention. Methods for managing a large number of states in limited memory include techniques such assupertrace [Hol91] and hash compaction [WL93]. These methods use hash tables to remember whether a state has been reached in the exhaustive search. The hash table 10

18 only stores an approximate description of a state, so that there is a small probability that one state is mistaken for another. Thus, the exhaustive search omits some states, and some system errors will not be detected. On the other hand, many more states can be stored in the same amount of memory, so that approximate search is applicable to larger systems. State space caching methods [GHP92] use memory as a cache, trading verication time for memory. The verication systems have a signicant drawback: the use of low-level models rst introduces irrelevant system states, and then techniques like partial-order methods attempt to extract abstract system description High-Level Service Specication Other research such as path expressions [Cam74] and logic of knowledge [HM90] has concentrated on high-level notations for describing protocols. Designers often informally describe relationships between the processes in distributed systems in terms of what one process \knows" about another process. For instance, in the description of TCP [Pos81] we nd: \An established connection is said to be halfopen if one of the TCPs has closed or aborted the connection at its end without the knowledge of the other,...". Thelogic of knowledge formalizes this notion of knowledge so that programs may include knowledge statements directly without referring to the method for gaining and losing knowledge. Such knowledge protocols are abstract and easy to specify [HZ87]. Some results [CM86] hint atways to implement gain and loss of knowledge. But so far reducing knowledge specications to actual programs has proven dicult [FHMV95]. Path expressions [Cam74] are a well-known and easy to use notation for specifying process coordination. Path expressions are regular expressions that describe the sequences of process activities in a distributed system. Campbell [Cam76] investigates several variants of path expressions. For some restrictive types of path expressions, it can be proven a priori that problems like deadlock do not exist, and there are known 11

19 algorithms to translate such path expressions to low-level P and V operations. But more expressive notations may be dicult to understand and implement [Hol91]. While verication systems use temporal logic to specify correctness predicates, there have been attempts to use it to specify systems. Temporal logic has been used as a programming language [Gab87]. However, descriptions based purely on temporal logic have proven dicult to understand in practice [Lam94] Synthesis Methods Synthesis methods translate a high-level specication to a low level language like communicating nite state machines or CSP. Tableau based methods [MW84] translate specications in temporal logic to languages like CSP or Buchi automata. The synthesis method produces a model for the formula as an existence proof. Tableaus were developed as proof techniques for mathematical logic. A tableau is a systematic way of decomposing a logical formula into subformulae until we reach elementary formulae. The truth of elementary formulae can be easily veried, and the tableau structure ensures that we verify enough elementary formulae to guarantee the truth of the original formula. When applied to temporal logic, the tableau can be interpreted as an automaton [MW84]. The automaton is then regarded as a centralized synchronizer for all processes that interleaves their actions so that the temporal formulae hold on the resulting sequence of actions. But such a centralized solution is undesirable in practice. Also, as noted above, descriptions based purely on temporal logic have proven unwieldy. Finite State methods are used in synthesizing communication protocols. They begin with a description of all desirable interactions in the system to be designed, and decompose them into communicating nite state machines. But these methods are often limited in various ways and appear to be too inexible for use in practice [PS91]. The approach of specifying desirable interactions seems applicable only to small systems [PS91] and decomposition is a dicult problem [PS91]. 12

20 In a related method [BZ83], the user starts with a dummy initial state for each process in the system to be synthesized. The user then species message transmissions for each process, and the synthesis software deduces the corresponding message receptions. The software traces all possible states where a reception may occur and updates the receiver state machine. After each update, the system warns the user if there are states without any messages in transit and none to be transmitted. Such states correspond to deadlock situations. Conformance to the service specication is not guaranteed by the method, although verication methods can be use after the synthesis is complete. Translation methods [KHvB92] translate specications in notations like LOTOS [BvdLV95] to message exchanges. The specications dene an ordering of operations, and the translation methods produce state machines that generate the sequences. These are suitable where service specication can be done as sequences of operations. But specications are often done in other styles [VSvSB91] Our Approach Our work was inspired by research on the logic of knowledge. This research showed that notions like \a process knows" were enough to express many interesting protocols succinctly. The treatment by Chandy and Mishra [CM86] reduced the logical operators to an algebraic form. Path expressions [Cam74] and LOTOS [BvdLV95] were earlier examples of the use of conjunction and disjunction predicates. We combined these ideas with the observation that the operators could be regarded as an abstract form of communication between communicating automata. This combination leads to succinct specications that can be checked by verication tools developed for communicating automata. The next question was how to describe the implementations of constraints without modeling the peculiarities of communication media. This would allow us to model control ow without the extraneous system states introduced by communication media. The model we use here is similar to LOTOS and Path expression operators that permit 13

21 specifying orders of execution. The novelty isinshowing that the implementations can be composed in way that they do not interfere with one another. The nal model is similar to the usual communicating nite state machines with single element queues. The dierence is that we communicate the current state rather than unstructured values. This makes it easy to translate the protocols to either shared memory or message passing with optimizations. Our approach gives a design technique that allows designers to simplify protocols by decomposition and abstraction. Since our development, we have found that the LO- TOSPHERE [BvdLV95] project has informally described the idea of design styles that mirror our own. They observe that experience shows that early specications are best described in Constraint-oriented style, while later designs in a State-oriented style. Our design method can be seen as a formalization of this observation. This unexpected similarity between our development and LOTOS research has strengthened our belief in the utility of the method. 2.3 The Synthesis Method In this section, we present the formal details of the synthesis method. We introduce our models of process, distributed system, and show howwe use automata to denote processes. Then we discuss our three notations for describing protocols. The rst notation represents communication using abstract operations. The next two notations rene these operators so that they can be implemented using shared memory and message passing programs. For each notation, we show howtoprove that one protocol implements another. Then we describe some implementations for the abstract communication operators, and show how the implementations can be expressed using shared memory and message passing. 14

22 2.3.1 Synthesis Let Beh be a set of desired behaviors, such as a set of sequences of events. Let L s be a specication language and L i an implementation language that specify desired subsets of Beh. Let the meaning of a specication be given by the function [[:]] s : L s! 2 Beh, while the meaning of an implementation by [[:]] i : L i! 2 Beh. Then we dene the problem of synthesis as follows. Denition 1 A synthesis method is a total function S : L s! L i such that given a specication, S() denotes the same behaviors as, [[S()]] i =[[]] s. A classic example of synthesis is the construction of a nite state machine that recognizes a set of ASCII words!, given a regular expression that species!. Here, L s is the language of regular expressions, L i the description of automata, and Beh is the set of all ASCII words. The synthesis method inductively translates the regular expression into a nite state machine. A protocol is a set of rules that describes how processes in a distributed system interact. For example, a le transfer protocol is a set of rules followed by processes on two machines in order to transfer les from one machine to another. A protocol synthesis method takes a description of the externally visible behavior b for a set of processes, and produces programs that the processes must execute in order to implement behavior b. We dene processes and distributed systems as sequences of abstract events. The events represent activities such as memory accesses or message transmission. Protocols are specied using automata. The behavior of each process is specied with an automaton, and the the joint behavior of a distributed system by the product of these automata. The state transitions in an automaton that represents a process may depend on the transitions of automata that represent other processes. The rules that govern this dependence constitute a model of process communication. Our synthesis method begins with a high-level specication with an abstract form of communication rules. Through a series of intermediate steps, the high-level specication is translated to programs that 15

23 use shared memory or message passing for communication. In the following, we make these ideas more precise Process and System Denition 2 A process P is a pair (E p ;R p ) where E p is a nite set of events and R p is a set of runs, a set innite sequences over E p. The events of two processes are disjoint: for every pair of processes P and Q, E p \E q = ;. Denition 3 A distributed system P is a pair (E; R) where E is S P 2PE P, and R a set of system runs, a set of sequences over E such that for every run, 2R, and every process P, the projection P is a run of P, P 2 R P. The runs of processes are specied using automata, and the runs of a distributed system as the product of the automata of the constituent processes. 16

24 2.3.3 Automata Denition 4 A nite state automaton is a tuple of the form (; ; ; ; ), where is a non-empty nite alphabet, a nonempty nite set of states, a transition relation, S S, a nonempty set of starting states and a nonempty set of nal states. Also, all states of are reachable, i.e., for all s 2, there is a sequence s 0 ;:::;s n where s n = s; s 0 2, and for 0 i<n,(s i ;a;s i+1 ) 2 for some a 2. Let, be the set of transitions f(s; s 0 )g such that for some letter a 2, (s; a; s 0 ) 2. A trace t of automaton A on a word w = a 0 :::a n,1 in is a sequence of states s 0 ;:::;s n where s 0 2 and for every s i ;s i+1 in t, (s i ;a i ;s i+1 ) 2. Note that a trace can also be thought of as a sequence of transitions, (s 0 ;s 1 ); (s 1 ;s 2 );:::. States and transitions that constitute a trace are said to occur in that trace. The automaton A accepts aword w if the last state s n is a nal state, s n 2. The set of all words accepted by an automaton is the language L of the automaton. We are interested in automata that accept innite words. Let w = a 0 a 1 ::: be an innite word and t be a trace s 0 ;s 1 ;::: over w. Let the limit of a trace t be the set of states that appear innitely often in t, lim(t) =fs j s = s i innitely ofteng. Then A accepts w if there is a state s 2 that appears innitely often in t, lim(t)\ 6= ;. This condition, Buchi acceptance, denes Buchi automata. The language of the automaton is is set of innite words L! that is accepted by the automaton. Since we specify distributed systems as products of automata, we use a slightly different presentation of the acceptance condition called generalized acceptance [GW94]. Let F = ff 1 ;:::;F k g;f i 2 ;k 0 be a set of sets of accepting states. Then the automaton A accepts w if for every F i, lim(t)\f i 6= ;. If there is only one F i, then the condition is the same as Buchi acceptance. Here, A is intended to be the product of of automata A i ;:::;A k. The condition ensures that accepted sequences are those where every automaton goes through its accept state innitely often. Thus we enforce fairness by requiring that every automaton must make progress. 17

25 Denition 5 A generalized Buchi automaton is a nite state automaton that accepts innite words under the generalized acceptance condition. Henceforth, we will assume that every automaton is equipped with a generalized acceptance condition Automata and Processes We use automata to denote processes. Intuitively, wewant either automata states or automata transitions to represent nite sequences of process events. For example, when describing mutual exclusion, we refer to sequences of critical and non-critical states. but when describing serial communication, we might use send and receive transitions. Technically, weachieve this by dening the alphabet to be a set isomorphic to either the set of states or the set of transitions,. In the rst case, accepted words dene acceptable sequences of states. In the second case, accepted words dene acceptable sequence of transitions. Correspondence between runs and words is established though a semantic mapping from letters to process events. Each letter corresponds to a set of nite sequences of process events. This mapping is inductively extended to map words (sequences over the alphabet) to runs (sequences over events). Let A be an automaton and P a process. Let [[]] be a function, [[]] : A! 2 E p, such that for every pair of two distinct letters a; b 2 A,[[a]]\[[b]] =;. The semantics is dened as follows: Denition 6 Automaton A =(; ; ; ; ) is said to denote process p =(E;R) if there exists a (nondeterministic) function [[]] : A! 2 E p extended to the words accepted by A as follows: [[]] = [[aw]] =[[a]][[w]] for letter a and word w. 18

26 If is isomorphic to, events of P are represented by the states of A. If is isomorphic to,, events of P are represented by transitions of A Protocols Protocols describe the behavior of distributed systems. A protocol is dened by automata products with restrictions. In a protocol, individual automata denote process behavior, the product denotes system behavior, and the restrictions on the product model communication. We rst dene the notion of automata products without restriction. Denition 7 Given A =( A ; A; A ; A ; A ) with acceptance condition F A, and B = ( B ; B; B ; B ; B ) with acceptance condition F B, and disjoint alphabets and states, the free product A B =(; ; ; ; ) is dened as follows: =( A B )[ A [ B = A B = A B = A B Given s =(s A ;s B );t =(t A ;t B );s;t 2, and a =(a A ;a B );a 2, (s; a; t) 2 if (s A ;a A ;t A ) 2 A and (s B ;a B ;t B ) 2 B. Given s = (s A ;s B );t = (t A ;s B );s;t 2, and a = a A ;a 2, (s; a; t) 2 if (s A ;a A ;t A ) 2 A. Symmetrically for B. F = S F ff i 2FA i Bg[ S f F i 2FB A F i g With this denition, A or B may have individual or joint transitions in AB. A state s A is said to occur in a trace of the product AB if it occurs in some product state (s A ;s B ). Similarly, a transition (s A ;t A ) occurs in a product trace if it occurs either individually 19

27 or jointly in a product transition. The acceptance condition for innite words requires that a word be accepted by A B if both A and B pass innitely often through their accepting states. In the free product, A and B execute their state transitions independently. But if A and B communicate, the product cannot have all possible transitions. This observation motivates the following denition of a protocol. Denition 8 A protocol p with automata A and B is the free product A B and a set of restrictions (a subset of all transitions) Cthat describes an automaton such that p =, and all the other sets are dened by the states reachable from via the transitions in,c. Note that although a protocol is an automaton, we prefer to specify it as a free product with restrictions. This allows us to reason separately about the structure of component processes (represented by the free product) and communication (represented by the restriction). Dierent ways of specifying the automata and C are used at dierent stages of the synthesis method Protocol Synthesis Having dened protocols, we can now dene protocol synthesis. First we dene the notion of protocols that serve as specication and implementation. This is based on the standard notion of language substitution [HU79]. A substitution is a mapping from an alphabet to subsets of 0. The mapping is used to transform words in a language L() over to words in a language L( 0 )over 0. We use substitution to transform a specication into an implementation. Let l 2 2 be a a set of nite words over some alphabet. Let (l) be the letters of used in the words of l. The languages l; m over are distinct if they use dierent letters, (l)\(m) =;. Denition 9 A protocol q implements a protocol p (conversely, p species q) 20

28 Every automaton of p corresponds to exactly one automaton of q. There is a nondeterministic renement function : p! 2 q such that for every pair of distinct letters a; b 2 p, the languages (a) and (b) are distinct, and the words accepted by q are just the rened words accepted by p (extending to words by induction). Let P be an automaton of p and Q be the automaton of q that corresponds to P. Then there is a nondeterministic function a : P! 2 Q that maps letters of P to distinct languages, and the words accepted by Q are just the rened accepted words of P. This denition captures the ideas that a protocol implementation is composed from automata with terminating executions, adds detail to the specication, and every step of the specication is rened in a distinct way. Note also that the relation between specication and implementation is dened in terms of alphabets. Since words can describe either sequences of state or transitions, an implementation may rene either states or transitions. Finally, notice that the relationship of implementation to specication is dened in terms of relationships between the components, the automata and their states. Thus, we reason can about the (innite) words via straightforward induction. We maynow dene protocol synthesis simply as follows: Denition 10 A protocol synthesis method is a function that given a specication protocol produces its implementation protocol. By comparing Denition 6 and Denition 9, it is clear that we can always choose denotations such that implementations that conform to Denition 9 preserve behavior. Indeed, in practice we rene a protocol several times until the events of the process of interest 21

29 have a one-to-one relationship to transitions of the protocol; the nal step is a ordinary shared memory or message passing program. 2.4 Specifying Coordination Automata in a protocol aect each others' state transitions. The rules that describe the eects model interprocess communication. We use a variety of rules to specify protocols. The most extensional, abstract protocol specications are given by Constraint rules. Constraint-rule specications are implemented by Action-rule specications. Action rules include more details of communication. In turn, Action-rule specications are implemented by Observation-rule specications. Observation-rule specications are intensional; they are suciently detailed so that they can be easily translated to shared memory or message passing programs. System properties like absence of deadlock and reachability are veried once and for all at the most abstract level for Constraint-rule specications. The subsequent syntheses preserve these properties. A property is dened as a set of words over the alphabet of interest [Alp86]. A word w has a property if w 2. A language L i preserves properties of language L s if for every property s of L s, there is a unique property i of L i, and if a word w s has property s, then the corresponding word w i does not have any property disjoint from i. Lemma 1 An implementation preserves properties of its specication. Proof. From Denition 9, by induction, every word accepted by an implementation protocol corresponds to exactly one word accepted by the specication protocol. Therefore, a property s of the a specication maps to a a unique property i. Furthermore, if i is a property disjoint from i, and w s 2 s isaword accepted by the specication with a corresponding word w i of the implementation, then w i 62 i (and w i 2 i ). Thus, the implementation preserves properties. 22

30 Action-rule specications are derived from Constraint-rule specications by translating constraint rules to action rules, and Observation-rule specications are derived from Action-rule specications by translating action rules to observation rules. We show that each translation leads to automata that are implementations of the corresponding specication Constraint-Rule Specications Constraint-rule specications express essential coordination among processes. A specication describes desirable sequences of process behavior as succinctly as possible. Constraintrule specications are extensional: they describe the eects of coordination, but not the details of how processes implement coordination or properties of communication media. Constraint-rule specications are designed so that techniques for specifying and verifying protocol properties are easily applicable. For the purposes of this thesis, two types of constraints suce. One constraint requires that processes synchronize their behavior, and the other species that behaviors be disjoint. In the following, we dene Constraint-rule specications, and show a simple example, the dining philosophers. We discuss the advantages and disadvantages of this style of specication. Then we explain how protocol properties can be checked at this level using verication algorithms Denitions Let P =(E p ;R p ) be a process, and Ep be a partition of the set of events E p into regions. This denition formalizes intuitive notions like the \critical region" used to describe parts of the run of a process. Dene a bijection [[]] :! E p. Let A =(; ; ; ; ), where = and a word corresponds to the sequence of states of the trace on that word. A is said to be a region automaton if it denotes P using [[]] according to Denition 6. Every state of a region automaton denotes a distinct region of that process. Constraint-rule specications are protocols that use region automata. 23