A Fail-Aware Datagram Service

Transcription

1 A Fail-Aware Datagra Service Christof Fetzer and Flaviu Cristian htt:// Abstract In distributed real-tie systes it is often useful for a rocess Ô to know that another rocess Õ will not use a certain iece of inforation that Ô has sent to Õ beyond a certain deadline. If Ô can learn about the occurrence of the deadline by sily easuring the assage of tie on its own local clock, we call this kind of inforation exchange counication by tie. We show that counication by tie is ossible in systes where there exists no a riori known uer bound on the transission delay of essages and where clocks are not synchronized. It is sufficient if one can coute an a osteriori uer bound on the transission delay of a essage Ñ, i.e. at the tie when Ñ is delivered. We show how one can coute an a osteriori uer bound on the one-way essage transission delay of a essage even if the clocks of the sender and receiver rocess are not synchronized. We use this ethod to design a fail-aware datagra service. This service suorts counication by tie by delivering all essages whose couted one-way transission delays are saller than a given bound as fast and all other essages as slow. We secify the roerties of this service and rovide an efficient ileentation for it. To illustrate how this service suorts counication by tie, we sketch a leader election rotocol that guarantees the existence of at ost one leader at any real-tie and we show how it allows the detection of out-of-date sensor inforation in rocess control alications. 1 Introduction Synchronous systes are characterized by certain counication [3]: any two non-crashed rocesses can send each other essages and it is guaranteed that each of these essages is delivered to the destination rocess within soe a riori known axiu tie ÑÜ. This allows rocesses of synchronous systes to counicate by easuring the assage of tie on their local clocks. This aer aeared in IEE Proc.-Softw., Vol. 146, No. 2, Aril 1999, Christof Fetzer is now with AT&T Labs Research. Flaviu Cristian died after a long and courageous battle against cancer in Aril For exale, consider that a correct rocess Ô broadcasts every tie units an alive -essage: when a rocess Õ has been waiting for an alive-essage fro Ô for ore than ÑÜ tie units, Õ knows that Ô has crashed. Indeed, if Ô were alive, it would send its alive -essage every tie units and the certain counication guarantee would ensure that Õ would receive a new alive -essage fro Ô within ÑÜ tie units of the recetion of the revious alive -essage fro Ô. In ost ractical distributed systes, counication is not certain, e.g. one has to consider that soe essages are delivered late or not at all. Even if one uses a distributed real-tie syste with redundant counication channels, one cannot necessarily assue that counication is certain. There always exists a non-zero robability È that soe essage even when essages are sent via redundant channels is delivered late or is droed. For safety critical systes designed assuing certain counication, a violation of the certain counication roerty ight result in a safety failure. Hence, such a safety critical syste ight only be safe with a robability of at ost ½ È. If ½ È is too sall, one cannot assue that counication is certain. See [19] for a discussion of this coverage roble of syste assutions. In systes characterized by uncertain counication, it is not obvious that rocesses can counicate by easuring the assage of tie on their local clocks, since if a rocess sends a essage to another rocess, the underlying counication syste does not guarantee delivery within a bounded tie. We show that counication by tie is also ossible in systes characterized by uncertain counication. One iortant echanis for counication by tie is the tie locking echanis (see Figure 1 ; [12]): a rocess Ô guarantees not to change the value of soe variable Î for at least, say, ÐØ tie units after Ô has sent a essage Ñ to another rocess Õ. In other words, Ô locks Î for ÐØ tie units. Process Õ knows at the recetion of Ñ that Ô will not change Î for ÐØ Ø Ñµ tie units, where Ø Ñµ is the transission delay of Ñ. An iortant roerty is that Ô can change Î after ÐØ tie units (Ô can easure this with its local clock) without having to wait for a essage fro Õ. We will describe in Section 5 how one can use tie locking in an election algorith to ake sure that at any tie there is at ost one leader. Note that, to use tie locking, a rocess has to be able to calculate an a osteriori uer bound on the one-way transission delay of 1

2 essages. In this aer, we describe a fail-aware datagra service that coutes such a bound for rocesses. t td() V = value ="V=value for lt" t+lt knows: does not know V knows: V=value knows: does not know V s < lt-td() u tie Figure 1: Tie locking echanis: rocess Ô roises at real-tie Ø not to change variable Î for at least ÐØ tie units. Process Õ learns at the recetion of Ñ the value of Î for the next ÐØ Ø Ñµ tie units. Process Ô learns by using its local clock to easure ÐØ tie units that it is authorized to change the value of Î. Let us sketch how tie locking can be used to switch a fail-safe syste to a safe ode when unasked counication failures occur. Consider a distributed syste that controls a railway-crossing. This syste is safe as long as the gates are closed whenever a train is in the crossing. Hence, by lowering the gates this syste can be switched to its safe ode. Such a switch to a safe ode ight be necessary to guarantee the safety of the syste if counication with reote sensors (that are detecting the arrival of trains) is lost. Consider that there are two rocesses Ô and Õ. Process Ô deterines whether the syste has to be switched to a safe ode, e.g. Õ deterines whether sufficient sensor data is available to kee the gates oen. The task of rocess Õ is to switch the syste to a safe ode on behalf of Õ, i.e. it closes the gates. If Õ becoes disconnected fro Ô, Õ has to assue that it is not safe anyore to kee the gates oen. Using tie locking we can achieve that as follows (see Figure 2). Process Ô sends eriodically essages to Õ saying that the syste is safe for the next ÐØ tie units. Inforally, these safe-essages say don t switch the syste to a safe ode for the next ÐØ tie units. If Ô does not succeed to let Õ constantly know that the syste is safe (e.g. due to a droed essage), Õ will switch the syste to a safe ode. safe lt safe lt safe lt knows syste is safe safe lt safe switches syste to safe ode tie Figure 2: Alication of tie locking: rocess Ô lets Õ know if the syste is safe. As soon as Õ does not know that the syste is safe, it switches the syste to a safe ode. The tie locking echanis uses an a osteriori uer bound on the transission delay of essages to ake sure that a receiver does not use inforation after the inforation has exired. We shall exlain this in the context of the railway-crossing exale. To ensure that the syste stays safe, Õ has to be able to deterine the age of a essage. Consider the scenario deicted in Figure 3. At first, the syste is safe and Õ knows that due to Ô s safe-essages. However, after soe tie Ô stos sending safe-essages because the syste is not guaranteed to be safe anyore. Process Õ has in this case to switch the syste to a safe ode. Process Ô ight even send switch -essages to let Õ know that Õ has to switch the syste to a safe ode. Note these switch-essages are actually redundant. Now, consider that Õ does not receive the switch essages (they are droed by the network) and instead, Õ receives soe very slow safe-essages. If Õ would not deterine the transission delay of a essage, Õ ight wrongly deterine that the syste is still safe and kee the gates u. However, since the tie locking echanis uses an uer bound on the transission delay of essages, Õ ignores all slow safe-essages. knows: syste safe safe safe safe dro lt knows: syste safe switch safe lt knows switch is needed switch dro lt switch slow sg tie Figure 3: Detecting slow essages: to ensure the safety of the syste, rocess Õ has to be able to detect slow essages. Otherwise, due to the arrival of a slow safeessage, Õ ight assue that the syste is safe in cases where Õ is suosed to switch the syste to a safe ode. In this aer, we describe a fail-aware datagra service that calculates an uer bound on the one-way transission delay of every essage it delivers. While any clients will use this uer bound, soe ight just use a ore abstract classification of the transission delay of essages: the service classifies delivered essages as either slow or fast. We showed in [9] that classifying essages as either slow or fast can be used to dynaically adat tie-outs, i.e. rocesses do not have to use fixed tie-outs instead, the syste can adat the tie-outs on the fly. The fail-aware datagra service is a basic distributed counication service that we have successfully used in the design and ileentation of several other distributed services such as a clock synchronization service [8], a local leader election service for artitionable systes [15] and a node ebershi service [11] (see Figure 4). Others have used this fail-aware datagra service to build a fully autoated train control syste [7]. We roose a secification for this service and a rotocol that ileents it efficiently using local, unsynchronized hardware clocks: a hardware clock roceeds with about the sae seed as real-tie but the difference between the values of two hardware clocks can be arbitrary. Hence, also the difference between a hardware clock and realtie can be arbitrary. To illustrate how the fail-aware datagra service suorts 2

3 counication by tie, we describe a sile leader election rotocol. In articular, we show how the fail-aware datagra service can be used to ensure that at any oint in real-tie there exists at ost one leader. We also deonstrate the use of a fail-aware datagra service in the context of rocess control alications: we show how to detect out-of-date sensor inforation using the fail-aware datagra service. 2 Related Work Many synchronous distributed services are secified by using safety roerties, i.e. roerties that should always hold. To ileent a safety roerty, it is often necessary that one has certain counication. Fail-awareness [10, 12] is a general ethod for extending the safety roerties of a fault-tolerant synchronous service by an excetion indicator so that the new, extended service becoes ileentable in distributed systes with uncertain counication but with access to local hardware clocks. The idea is that the indicator tells a server and its clients whether soe safety roerty currently holds or if it ight be violated because the syste has suffered too any failures. fail-aware broadcast fail-aware ebershi fail-aware clock synchronization local leader election fail-aware clock reading indeendent assessent fail-aware datagra tied asynchronous syste deends uon Figure 4: Fail-aware services that deend uon the failaware datagra service. More concretely, a fail-aware distributed service has to rovide for each safety roerty Ë an indicator Á Ë such that whenever Á Ë is true, Ë holds. Therefore, if Ë does not hold at soe oint Ø, indicator Ë has to be false at Ø. A fail-aware service is reuired to set its indicators to true, whenever the failure freuency is below an a riori secified bound. This reuireent akes sure that a failaware service can indicate a violation of a safety roerty only if there is a good reason for that, i.e. the occurrence of too any failures. The indicators of a service allow a client to see whether the safety roerties the client deends uon are currently valid. An indicator ight indicate a violation of a safety roerty because soe rocess suffers a erforance failure (i.e. the rocess has issed a deadline), a rocess has crashed, a essage has been droed (i.e. has suffered an oission failure), or a essage has suffered a erforance failure (i.e. its transission delay is greater than soe given threshold Æ). We give a ore detailed descrition of the failure odel in Section 4. A fail-aware service Ë that deends uon a lower level service ÄË ight also indicate a violation of one or ore of its safety roerties if ÄË indicates that it cannot guarantee all its safety roerties. In other words, when a service cannot ask the safety roerty violation of ÄË (i.e. Ë cannot guarantee its own safety roerties anyore), Ë has to indicate this to its client. A lower level failure that cannot be asked will therefore switch off the indicators of all safety roerties it ight violate. Undetected erforance failures ight not only delay the execution of a client rocess, they ight cause a containation roble: late essage deliveries and slow rocesses ight corrut the state of rocesses and such rocesses with corruted state ight send essages that corrut the state of other rocesses. For exale, recall the scenario of Figure 3 where undetected essage erforance failures can lead to an unsafe syste. Failawareness allows the clients to learn that the safety roerties of lower level services ight be violated. Hence, clients learn that they cannot deend uon these safety roerties anyore. In articular, fail-awareness suorts the containent of erforance and crash failures and the violation of safety roerties at each level of abstraction, so that at no level of abstraction state containation occurs. In other words, fail-awareness allows the detection of failures before they can corrut the syste state. For concreteness, consider a synchronous datagra service that is characterized by the following roerty (T): the transission delay of any essage delivered by the service is at ost tie units. Since in asynchronous systes there exists no uer bound on the transission delay of essages, essages ight be delivered after tie units. For asynchronous systes we extend roerty (T) by introducing a binary indicator, which can take the values slow or fast, to ensure the following failawareness roerty: the transission delay of any essage delivered as fast is at ost tie units. Note that one can actually use this indicator to enforce the original safety roerty (T): if one dros all slow essages, one can ake sure that no essage with a transission delay of ore than tie units is delivered. However, soeties it is advantageous that rocesses also receive slow essages. Hence, we classify essages as slow or fast and leave it to the clients whether they dro slow essages. The fail-aware datagra rotocol we roose is ileentable on to of a conventional datagra service, like UDP [18]. The calculation of the uer bound on the transission delay of a essage relies on the ain idea which underlies robabilistic reote clock reading [1], naely that by easuring round-tri essage delays one can calculate uer bounds on one-way transission delays. 3

4 Each essage that the fail-aware datagra service delivers is classified as either slow or fast. Such a classification of a essage can also be erfored with the hel of internally synchronized clocks. Even though internal clock synchronization can be achieved in asynchronous systes by robabilistic ethods [1, 10], the solution roosed in this aer (which does not use synchronized clocks) achieves a better recision since we only need to have airwise synchronized clocks in the sense that any two connected rocesses know aroxiately the distance between their own two hardware clocks. The fail-aware datagra service also allows the correct classification of essages sent between rocesses in different network artitions although the clocks of these rocesses ight be out of sync. Note that unless in each network artition there exists an access to a reference tie rovider, such as a GPS receiver [6], one cannot guarantee that the deviation between all clocks is bounded. We actually use the fail-aware datagra service in a fail-aware clock synchronization algorith to read reote clocks [13]. The fail-aware datagra service is the foundation of all other fail-aware services we have designed and ileented (see Figure 4; [13]). It has been vital in the design of all theses services because they all need to reject slow essages. We use the fail-aware datagra service in artitionable systes to rovide the abstraction of clean artitions in the sense that, even when slow essages fro other artitions arrive, higher level rotocols see non overlaing logical artitions without inter-artition counication (see [12]). A fail-aware datagra service is used in a fully autoated train control syste [7] to hel to detect slow essages that could cause the syste to becoe unsafe. 3 Fail-Aware Datagras We address the roble of how to detect late essages by roosing a fail-aware datagra service that delivers a essage as either slow or fast. Processes can use this service to send each other essages. In the following we call such essages fail-aware essages (abbreviated by fa-essages ). Say a rocess Ô sends an fa-essage Ñ at real-tie to soe rocess Õ (Figure 5). At the delivery of Ñ at soe real-tie Ø, the fail-aware datagra service calculates an uer bound (denoted by ٠ѵ) on the transission delay of Ñ (denoted by Ø Ñµ), i.e. ٠ѵ Ø Ñµ Ø. If the uer bound is at ost, i.e. ٠ѵ, the service classifies Ñ as fast. Otherwise, Ñ is classified as slow. See Section 7 for a colete secification of the fail-aware datagra service. The fail-aware datagra service allows a receiver to detect when the transission delay of an fa-essage was greater than soe given threshold. It is then u to the receiver to handle such a slow fa-essage in a anner that akes sense at its level of abstraction [2]. For exale, 0 fa-send(,) fast slow uer bound Figure 5: On the delivery of an fa-essage Ñ, the failaware datagra service calculates an uer bound on the transission delay of Ñ. If this uer bound is at ost, Ñ is classified as fast and otherwise, it is classified as slow. the receiver ay reject all slow fa-essages to ensure at its level of abstraction an oission failure seantics for its counication service. Each client can choose an aroriate constant deending on the client s reuireents. If a client is indeendent of the actual choice of, the syste ight adat dynaically to iniize the nuber of slow fa-essages (see [9] for why and where this is useful). For silicity and since a dynaic adatation of is orthogonal to the issues discussed in this aer, we assue in the following that is fixed. The intuition behind calculating the uer bound ٠ѵ on the transission delay Ø Ñµ of a fa-essage Ñ is as follows (see Figure 6). One selects another essage Ò that Ô has received fro Õ before Ô sends Ñ to Õ. Process Õ ight have received Ò several seconds before. In articular, Ô does not have to receive a essage fro Õ for every fa-essage it sends to Õ. One essage every coule seconds is tyically sufficient. Assue for a oent that the hardware clocks roceed with the sae seed as realtie. In this case we can use Õ s hardware clock to easure the tie between the recetion of Ñ and the sending of Ò ( in Figure 6) and we can use Ô s hardware clock to easure the tie between the sending of Ñ and the recetion of Ò ( ). Since the transission delay of Ò lus Ñ is obviously µ µ (if clocks roceed with the sae seed as real-tie), we can bound the transission delay of Ñ by µ µ. We will show how to irove the uer bound 1) when knowing a iniu transission delay for Ò and 2) by selecting Ò aroriately in case we have ultile essages to choose fro. 4 Tied Asynchronous Syste Model The tied asynchronous syste odel [4] is an abstraction of the roerties of ost distributed systes encountered in ractice, built out of a set of workstations con- 4

5 n={a,... } A B td() < (D-A)-(C-B) A,B,C,D: local tie stas C ={A,B,C,..} Figure 6: To calculate an uer bound on the transission delay of Ñ, one uses the tie stas of soe other aybe even several seconds ago received essage Ò. nected by a LAN or WAN. The tied odel akes very few assutions about a syste and hence, alost all ractical distributed systes can be described as tied asynchronous systes. Since it akes such weak assutions, any solution to a roble in the tied odel can be used to solve the sae roble in a ractical distributed syste. The tied odel is however sufficiently strong to solve any ractically relevant robles, such as clock synchronization, highly available leadershi, ebershi, atoic broadcast and availability anageent [4]. The tied odel describes a distributed syste as a finite set of rocesses È linked by an asynchronous datagra service. The datagra service rovides riitives to transit unicast and broadcast essages. A one-way tie-out delay Æ is defined for the transission delays of essages: although there is no guarantee that a essage will be delivered within Æ tie units, this one-way tieout is chosen so as to ake the likelihood of a essage being delivered within Æ tieouts suitably high [2]. We say that a rocess receives a essage Ñ in a tiely anner iff the transission delay of Ñ is at ost Æ. When the transission delay of Ñ is greater than Æ, we say that Ñ has suffered a erforance failure or that Ñ is late [2]. We assue that there exists a constant Æ ÑÒ that denotes the iniu essage transission delay: any essage sent between two reote rocesses has a transission delay of at least Æ ÑÒ tie units. By reote we ean that the essage is sent via a network. We will see later that the correctness of the uer bound calculated by the fail-aware datagra service deends on the correctness of Æ ÑÒ, i.e. if Æ ÑÒ is chosen too big, the calculated bounds ight be too sall. The safest choice is to assue that Æ ÑÒ ¼. However, we will see that the calculated uer bounds are getting tighter for Æ ÑÒ ¼. Another safe choice is to coute a lower bound for each essage Ñ sent (or, received) by a couter Ô as follows: if is the axiu network bandwidth of Ô, Þ Ñµ the size of Ñ in bits, the transission delay of Ñ is at least Þ Ñµ. D The asynchronous datagra service has an oission/erforance failure seantics [2]: it can dro a essage or it can fail to deliver a essage in a tiely anner, but the robability that it delivers corruted essages is negligible. Broadcast essages allow asyetric erforance/oission failures: a rocess ight receive a broadcast essage Ñ in a tiely anner, while another rocess ight receive Ñ late or not at all. The asynchronous datagra service satisfies the following reuireents: Validity: when a rocess Ô receives a essage Ñ fro Õ at soe tie Ø, then indeed there exists soe earlier tie Ø such that Õ has sent Ñ to Ô at. No-dulication: a rocess receives a essage Ñ at ost once, i.e. when essage Ñ is delivered to rocess Õ at tie, then there exists no other tie Ø such that the datagra service delivers Ñ to Õ at Ø too. The rocess anageent service defines a scheduling tie-out delay, eaning that a rocess is likely to react to any trigger event within tie units (see [4]). If Ô takes ore than tie units to react to a trigger event, it suffers a erforance failure. We say that Ô is tiely in an interval Ø iff at no oint in Ø Ô is crashed and Ô does not suffer any erforance failure in Ø. We assue that rocesses have crash/erforance failure seantics [2]: they can only suffer crash and erforance failures. Processes can recover fro crashes. Two rocesses are said to be connected [5] in Ø iff they are tiely in Ø and each essage sent between the in Ø Æ is delivered in a tiely anner (see Figure 7). We denote that Ô and Õ are connected in Ø by using the redicate connected(,,s,t). s δ realtie δ δ n 1 1 n 2 t- δ 2 Figure 7: Tiely rocesses Ô, Õ are connected in Ø iff all essages sent between the in Ø Æ are delivered within Æ tie units. Processes have access to local hardware clocks with a bounded drift rate. Correct hardware clocks dislay strictly onotonically increasing values. We denote the local hardware clock of a rocess Ô by À Ô. For silicity, we assue in this aer that we can neglect the granularity of a hardware clock: e.g. a clock has a resolution of ½ or saller. Hardware clocks are roceeding within a linear enveloe of real-tie: the drift rate of a correct hardware clock À Ô is bounded by an a riori given constant so that, for any interval Ø : t 5

6 Ø µ ½ µ À Ô Øµ À Ô µ Ø µ ½ µ. An iortant assution is that the hardware clock of any non-crashed rocess is correct. Inforally, we reuire that we can neglect the robability that the drift rate of a hardware clock is not within. Whether soe failure robability is negligible deends on the stochastic reuireents of an alication [2, 19]. For non-critical alications, the use of a sile counter connected to a uartz oscillator ight rovide a sufficiently close aroxiation of a crash failure seantics, i.e. one can neglect the robability that any clock failure excet clock crash failures occur. For safety critical alications, such an ileentation ight not be sufficient. However, one can use ultile oscillators and counters to ake sure that the robability of any clock failure excet a clock crash failure becoes negligible [14]. For silicity, we assue that a hardware clock does not recover fro a crash. Hardware clocks do not have to be synchronized: the deviation À Ô Øµ À Õ Øµ between correct hardware clocks À Ô and À Õ is not assued to be bounded. For ost uartz clocks available in odern couters, the axiu hardware clock drift rate is in the order of ½¼ to ½¼. Since is such a sall uantity, in what follows we neglect ters in the order of ¾ or higher. In articular, we will euate ½ µ ½ ½ µ and ½ µ ½ ½ µ. When a rocess easures the length of an interval Ø by Ì Ë À Ô Øµ À Ô µ, the error of this easureent is within Ì Ëµ Ì Ëµ since Ì Ëµ ½ µ Ø Ì Ëµ ½ µ. 5 Leader Election Exale In this section, we show how the fail-aware datagra service can be used to ileent a leader election rotocol. For silicity, we only describe a fragent of a leader election rotocol, the full rotocol being described in [15]. The leader roble can be secified using a safety roerty (S) and a tieliness roerty (T): (S) at any real-tie there exists at ost one leader, and (T) if there exists a ajority of rocesses connected in a real-tie interval Ä, then there exists a rocess Ô such that Ô is leader for soe tie Ø ¾ Ä. In this aer, we only address the issue of how to guarantee that at any tie there exists at ost one leader. All essages sent by the election rotocol are fail-aware essages, i.e. are sent with a fail-aware datagra service. A rocess Ô that wants to becoe leader broadcasts an election -essage to let the other rocesses know of its intention to becoe leader. After Ô has broadcasted an election -essage, Ô ust collect a ajority of fast suort -essages to becoe leader. A rocess Õ that suorts Ô s election by sending a suort-essage to Ô, guarantees (using its local hardware clock) not to send a suort-essage to another rocess for the next, say, Ñ Ø (axiu suort tie) real-tie units (Figure 8). Thus, when rocess Ô receives a fast suort-essage fro Õ, it knows that Õ will suort Ô s election for at least the next Ñ Ø tie units. Process Ô can use its local hardware clock to guarantee that it stos using Õ s suort essage before it exires : when Ô has received the fast suortessage Ñ at local tie Ì, Ô uses Ñ only as long as the following suort condition is true: ÙÔÔÓÖØ Øµ À Ô Øµ Ì Ñ Ø µ ½ µ. The factor of ½ µ takes care of the drift rate of Ô s hardware clock, which is within. Since Ô knows that a ajority of rocesses suort it, and no rocess in that ajority can suort a rocess other than Ô for a known aount of tie. It follows that Ô is the only leader that can exist for as long as Ô s suort redicate evaluates to true for a ajority of the rocesses. election T suort st leader st realtie Figure 8: A rocess Õ transitting a suort-essage guarantees not to send any other suort-essage for at least Ñ Ø tie units. One of the ain robles in the design of a leader election roble is that in soe runs the rocesses cannot correctly decide if the current leader has crashed or is just very slow. An interesting feature of the sketched leader election rotocol is that actually rocesses do not have to know if the current leader has crashed or is just slow. Process Õ knows that Ô will not use Õ s suort for ore than Ñ Ø- real-tie units. Hence, it can send a suort essage to another rocess after waiting for at least Ñ Ø(1+) local clock tie units. In articular, when Õ would susect that Ô has crashed, it could suort (after Ñ Ø(1+) local clock tie units) another rocess without knowing for sure that Ô has crashed. This is an exale of rocesses Ô and Õ counicating by tie: Ô knows that Õ will suort Ô s election for at least the next Ñ Ø tie units and Õ knows by waiting for Ñ Ø tie units that Ô will not use Õ s suort anyore. Note that if Ô would not reject slow suort essages, it ight be able to becoe leader at a tie when the other rocesses already suorted the election of another ro- 6

7 cess. Hence, the detection of slow essages is iortant to enforce that there is at ost one leader at a tie. Note that if there were ore than one leader, this ight lead to the containation of services deending on one uniue leader. In other words, soe undetected slow essages ight lead to serious containation robles in higher services. Undetected slow essages ight not only lead to containation robles, they ight also cause the invalidation of liveness roerties. For exale, recall that a rocess Õ only suorts the election of a rocess Ô when Õ receives Ô s election-essage Ñ as a fast essage (Figure 8). If Õ would not check that Ñ is fast, Õ could suort the election of a rocess Ô that already gave u to becoe leader a long tie ago. Since Õ has to wait for Ñ Ø tie units before it can suort another rocess, rocesses have to avoid suorting slow election-essages. In articular, even when a ajority Å of rocesses could counicate in a tiely anner with each other, slow essages sent by rocesses outside of Å could revent the election of a leader in Å. By discarding slow essages, the fail-aware datagra service enables the rocesses in Å to elect a leader in a bounded aount of tie even when they receive slow election essages fro outside of Å. In any of our rotocols, the fail-aware datagra service hels to guarantee tieliness reuireents even when slow essages arrive. 6 Process Control Exale The fail-aware datagra service can even be used to detect out-of-date inforation when essages are forwarded by one or ore interediate rocesses (Figure 9). Let us assue that rocess Ô schedules to read its hysical sensor every clock tie units and sends the reading in a fail-aware essage Ò to the controller rocess Õ (see seudo-code in Figure 31 and Section 8.4 for further exlanations of the seudo-code). The controller has to ake sure that the inforation contained in essage Ò is u-to-date, that is, Ò is at ost tie units old. An ileentation roble that arises is that it is ossible for Ô to read its sensor at clock tie, be reeted by the oerating syste for soe tie, and then be resued again to send essage Ò at clock tie (Figure 10). Process Õ has to ake sure that the real-tie duration µ is at ost instead of only detecting the case that the duration µ is at ost. We solve this roble by virtually extending the transission tie of Ñ to µ: we allow rocesses that use the fail-aware datagra service to set the send tie sta of essages. Process Ô will thus set the send tie sta of Ò to, instead of using the default send tie sta. A siilar roble can occur at rocess Ö: Ö has to ake sure that the coands sent by Õ are u-to-date in the sense that a coand received in a essage Ñ was generated no ore than tie units ago. The solution to Sensor Controller Actuator r Figure 9: Process Ô reads eriodically a local sensor, transits the reading to Õ, who rocesses this inforation and transits the result to Ö. this roble is the sae as above: Õ uses its receive tie sta of Ò as send tie sta of Ñ. Hence, erforance failures that ight occur between and can be detected by Ö because Ñ is delivered as a slow essage when Ñ s extended transission delay was greater than. Process Ö can set a tier to ake sure that it gets at least every Å clock tie units a fast coand essage: when a tie-out event occurs, it has to switch to a failsafe ode. For exale, when Ö would control the gates of a rail-way crossing, Ö would be ileented by a hard real-tie thread (or, a thread and soe sile hardware in case one does not have hard real-tie threads), and Ö would lower the gates whenever it has to switch to failsafe ode because it has not received a fast coand essage for ore than Å tie units. In case one cannot use hard real-tie threads, we described in [13] how a sile hardware circuit allows one to solve this roble even when Ö does not rovide any real-tie guarantees. r A B C D Ε n a b c d e clock tie realtie Figure 10: Process Õ has to detect when duration and Ö has to detect when. By using and instead of the default values and as send tie stas, the fail-aware datagra service allows Õ and Ö to detect these erforance failures. 7 Service Secification In this section, we give a secification of the fail-aware datagra service. To do this, we first introduce the riitives that this service rovides to its clients. 7

8 7.1 Priitives The fail-aware datagra service rovides the following riitives to transit unicast and broadcast essages: fa-send(,): sends an unicast essage Ñ to rocess Õ, fa-broadcast(): broadcasts Ñ to all rocesses including the sender of Ñ, and fa-deliver(,,flag): is an ucall initiated by the failaware datagra service to deliver essage Ñ sent by rocess Ô. The value of the flag can be either slow or fast. The classification of essages is based on the uer bound calculated at the recetion of a essage and the value of. The relationshi between the transission delay of a essage and its classification is hence as follows: any essage Ñ that is delivered as a fast essage exerienced an actual transission delay of at ost (see Figure 11), and any essage that exerienced a transission delay of ore than tie units is delivered as a slow essage (see Figure 12). s fa-send(,) fa-deliver(,, fast) t s+ realtie Figure 11: If Ñ is delivered as a fast essage, the receiver knows that Ñ s transission delay Ø µ is at ost. A broadcast essage Ñ sent by a rocess Ô ight be delivered as slow by soe rocesses and as fast by other rocesses since the transission delay of Ñ can vary for different rocesses. Other rocesses ight not deliver Ñ at all because of an oission failure in the underlying datagra service. So far, we have not given a reuireent that deterines when the service has to deliver a essage as fast. Ideally, we would like any essage with a transission delay of at ost to be delivered as fast and any essage with a transission delay of ore than to be delivered as slow. The roble with this aroach is that one cannot calculate a tight uer bound on one-way transission delays (e.g. due to the natural drift of hardware clocks). s fa-send(,) > s+ Figure 12: If Ñ s transission delay Ø, it is delivered as a slow essage. fa-deliver(,, slow) t realtie µ is ore than To circuvent this iossibility, one could cheat in the following way. On the recetion of a essage Ñ, one deterines an uer bound on the transission delay of Ñ. If the uer bound indicates that Ñ s transission delay is at ost, Ñ is delivered as fast. Otherwise, Ñ s delivery is delayed until its transission delay is definitely ore than. While this additional delay of slow essages is useful in soe situations, we do not address this issue further since 1) soe alications want to receive also slow essages as soon as ossible, and 2) it is easy to rovide such an additional delay on alication level. In this aer we want 1) to allow the fail-aware datagra service to be able to deliver essages as soon as they arrive, 2) to exclude trivial ileentations that behave correctly by delaying all essages by and then delivering the as slow. Hence, when secifying the service: 1) we allow a bounded uncertainty in the classification of essages 2) we tie the secification of fast to the bound Æ given by the tied odel to enforce that tiely essages ust be classified as fast (see Figure 13). Let us consider that a fail-aware datagra service sends each essage Ñ (i.e. an fa-essage) as a essage of the underlying asynchronous datagra service (a d-essage). Again, one would ideally like a tiely d-essage (with transission delay at ost Æ) to be delivered as a fast fa-essage, and a late d-essage (i.e. its transission delay is greater than Æ) to be delivered as a slow fa-essage. As entioned before, the roble with this goal is that a rocess cannot calculate a tight uer bound on the transission delay of every essage Ñ it receives. However, we show later that the error of the calculated uer bound for a essage Ñ sent between connected rocesses can be bounded by a sall constant ÑÜ ¼. We relax the above ideal goal: we reuire that any tiely essage between two connected rocesses be delivered as a fast essage and any essage that takes ore than Æ ÑÜ tie units be delivered as slow. In what follows we assue Æ ÑÜ holds, where the error constant ÑÜ is defined in Section 8.3. We have to 8

9 allow the fail-aware datagra service to deliver essages that exerience a transission delay within Æ as either slow or fast according to the calculated uer bound (see Figure 13). r o s fa-broadcast() fast slow or fast s+ δ s+ slow realtie Figure 13: Broadcast essage Ñ can be delivered as fast by Õ and as slow by Ó. 7.2 Secification To silify our secification, we assue that each failaware datagra essage is uniue. For exale, in the roosed rotocol a sender adds its id and the current value of its local hardware clock to each essage it sends. Since correct clocks are strictly onotonic for successive readings, all essages are uniue. We use the following redicates to denote fail-aware datagra service related events: fa-deliver Ø Ô (,s,flag): essage Ñ, the identification of the sender of Ñ, and the indicator Ð is delivered at real-tie Ø to rocess Ô. fa-send Ø Ô(,): rocess Ô sends the fa-essage Ñ to rocess Õ at real-tie Ø. fa-broadcast Ø Ô(): Ô broadcasts the fa-essage Ñ at real-tie Ø. Since a rocess cannot calculate a tight uer bound on the transission delay of a essage, we derive a relaxed tieliness reuireent for our service. If rocesses Ô and Õ becoe connected at tie and stay connected until tie Ø, all essages sent between Ô and Õ within Ø are tiely. Since the rocesses need round-tri essage airs to calculate an uer bound on one-way essage delays, at the start of the tie interval Ø, Ô and Õ ight not have any essage air that can rovide the with a good uer bound. Hence, we allow that, for soe bounded tie after becoing connected, two rocesses ay erroneously deliver soe tiely essages as slow. We therefore only reuire that all tiely essages sent between Ô and Õ during an interval Ø be delivered as fast (instead of reuiring that all tiely essages sent between Ô and Õ during interval Ø be delivered as fast ). Note that even though a rocess is soeties allowed to deliver a tiely essage as slow, a fail-aware datagra service is never eritted to deliver a essage with a transission delay greater than as fast. We use a notation siilar to that of [17] to secify the failaware datagra service. Each occurrence of variable denotes a distinct unnaed, universally uantified variable siilar to those used in Prolog. The roerties that a fail-aware datagra service should satisfy are defined as follows: Validity: Only if a rocess Õ has sent a essage Ñ to a rocess Ô, essage Ñ and the identity of Õ will be delivered to Ô. Ô Õ Ñ Ø: fa-deliver Ø Ô (,, ) µ Ø: fa-send Õ (,) fa-broadcast Õ (). Non-dulication: Messages are delivered at ost once. Ô Ñ Ø: fa-deliver Ø Ô (,, ) fa-deliver Ô (,, ) µ Ø. Fail-awareness: A fast essage has a transission delay of at ost. Ô Õ Ñ Ø: fa-deliver Ø Ô (,,fast) µ Ø- : fa-send Õ (,) fa-broadcast Õ (). Tieliness: When two rocesses Ô and Õ are connected in interval, and Õ transits a unicast or broadcast essage Ñ to Ô at tie, then Ñ is delivered as a fast essage to Ô by. Ô Õ Ñ: connected(,,s-,s+ ) fa-send Õ(,) fa-broadcast Õ (,) µ Ø fa-deliver Ø Ô (,,fast). 7.3 Discussion Note that our tieliness reuireent does not rescribe that rocesses send eriodically essages to each other. However, to ileent the tieliness reuireent, a rotocol ight have to ake sure that rocesses send each other essages. The inforation that a rocess Ô has about rocess Õ s clock ages with tie, hence they ust exchange essages at least every, say, tie units. This does not ean though that the fail-aware datagra rotocol itself has to eriodically exchange such essages. If the essage traffic between Ô and Õ is sufficiently rich, no eriodic essages are ever sent on behalf of the rotocol. Our secification does not reuire that an fa-essage carried by a late d-essage be actually delivered to the destination rocess. Tyically, a fail-aware datagra service will deliver each fa-essage Ñ unless the underlying 9

10 datagra dros the d-essage carrying Ñ. In our secification we do not want to exclude service ileentations that dro soe late essages to shed the load during high load situations. Another reason for not reuiring that all late fa-essages be delivered is that UDP does not guarantee that a essage is delivered at ost once [18]: to enforce at ost once delivery seantics, the inforation that a essage has already been delivered has to be ket for only tie units when the service dros all essages with a transission delay greater than. 8 Protocol In this section, we roose a rotocol for the fail-aware datagra service. The validity and non-dulication reuireents of the fail-aware datagra service are already satisfied by the underlying asynchronous datagra service (assution of the tied asynchronous syste odel). We therefore restrict ourselves to describe the ileentation of the fail-awareness and the tieliness reuireents. The roosed rotocol sends all fa-essages as d-essages to which it adds soe rotocol-secific inforation. The fail-awareness reuireent is ileented using an aroach siilar to that of robabilistic reote clock reading [1]: a rocess Ô coutes for each d-essage Ñ it receives an uer bound on the transission tie by easuring on its local clock the duration of an asynchronous datagra round-tri which includes Ñ. 8.1 Uer Bound To ileent the fail-aware datagra service, a rocess Õ that receives a essage Ñ has to calculates an uer bound ٠ѵ on the transission delay Ø Ñµ of Ñ (see Figure 14). c td() = d-c d e() 0 realtie c+ub() Figure 14: Process Õ calculates on the recetion of a essage Ñ an uer bound ٠ѵ on the transission delay Ø Ñµ. For local essages an uer bound can be couted in the following way: when a rocess Ô sends a essage Ñ to itself with send tie sta and receive tie sta, the transission delay of Ñ can be bounded by (see Figure 15), Ø Ñµ µ ½ µ (1) since the drift rate of Ô s hardware clock is not saller than. This aroach of bounding the transission delay is not alicable for two reote rocesses Ô and Õ because hardware clocks are not synchronized. C =H(c) c (D-C)(1+ ρ) D =H(d) clock tie realtie d Figure 15: The transission delay Ø Ñµ of a local essage Ñ can be bounded using the send and receive tie stas À Ô µ and À Ô µ. To calculate an uer bound on the transission delay Ø Ñµ of a reote essage Ñ, the receiver Õ needs to know the send and receive tie stas of a round-tri essage air Ò Ñµ (see Figure 16). The transission delay Ø Ñµ of Ñ can be exressed as the length of the round-tri µ inus Ò s transission delay Ø Òµ µ inus the duration µ between the recetion of Ò at tie and the tie at which Ô sends Ñ: Ø Ñµ µ µ µ (2) Since the drift rate of hardware clocks is bounded by, a hardware clock easures the length Ð of a real-tie interval with an error that is within Ð Ð. The send and receive tie stas of essages Ò and Ñ are À Õ µ, À Ô µ, and À Ô µ, À Õ µ, resectively. Using these tie stas, we can derive the following bounds for the length of the intervals and (see Figure 16): µ ½ µ µ ½ µ (3) µ ½ µ µ ½ µ (4) The iniu transission delay of any reote essage is at least Æ ÑÒ. Hence, the length of interval is at least Æ ÑÒ : Ø Òµ Æ ÑÒ (5) An uer bound for the transission delay Ø Ñµ can be derived by using an uer bound for the length of the interval and lower bounds for the intervals and (see Figure 16): Ø Ñµ Õ¾ µ µ µ (6) µ ½ µ Æ ÑÒ µ ½ µ (7) 10

11 A a δ in n =H(a) B=H(b) b (C-B)(1- ρ) C (D-A)(1+ ρ) c d D realtie realtie clock tie Figure 16: The transission delay Ø Ñµ is bounded by calculating an uer bound for the length of and lower bounds for the length of and. We define the uer bound Ù Ò Ñµ calculated by Õ on Ø Ñµ using essage Ò as follows: Ù Ò Ñµ Õ µ ½ µ Æ ÑÒ µ ½ µ (8) To be able to calculate that uer bound, each rocess Ô stores for any other rocess Õ the send and receive tie sta of soe essage Ò that Ô has received fro Õ (Figure 17). Each rocess Ô aintains two arrays ËÌ Ô and ÊÌ Ô such that entry ËÌ Ô Õµ contains the send tie sta of Ò and ÊÌ Ô Õµ the receive tie sta of Ò. The tie stas are deterined by the local hardware clocks of the rocesses: when rocess Õ has sent Ò at real-tie and Ô has received Ò at real-tie, then n B=RT () A=ST () ËÌ Ô Õµ À Õ µ (9) ÊÌ Ô Õµ À Ô µ (10) C=ST (),ST (),RT (),ST () D clock tie Figure 17: Each rocess Ô aintains an array of send tie stas ËÌ Ô and receive tie stas ÊÌ Ô. ËÌ Ô Õµ and ÊÌ Ô Õµ contain the send and receive tie stas of soe essage Ò that Ô has received fro Õ. ËÌ Ô Ôµ contains the send tie sta of the ost recent essage that Ô has sent. Each rocess Ô that wants to send an unicast essage Ñ to soe rocess Õ sends together with Ñ the send and receive tie sta of soe essage Ò that Ô has received fro Õ and the send tie sta of Ñ (see Figure 17). Process Õ can then use the tie stas of the round-tri consisting of essages Ò and Ñ to calculate an uer bound on the transission delay of Ñ. When rocess Ô broadcasts a essage Ñ, it includes arrays ËÌ Ô and ÊÌ Ô to allow all rocess to extract the tie-stas of a round-tri essage air (see Figure 18). To reduce the inforation iggy-backed on broadcast essages, our ileentation of a fail-aware datagra service only iggy-backs the two arrays on heler -essages (see below). All other broadcast essages only iggy-back the send tie sta. This reduction in essage size however reuires that each rocess Õ aintains an additional array that stores two additional tie-stas ( and ) for each reote rocess Ô. ST () r RT () n nr ST (r) RT (r) ST (),ST, RT D D r clock tie Figure 18: Broadcast essages contain the two arrays ËÌ Ô and ÊÌ Ô of the sender Ô to allow all receivers to extract tie stas of round-tris: rocess Õ uses the tie stas of air Ò Õ Ñµ while rocess Ö uses the tie stas of Ò Ö Ñµ. 8.2 Udate of Tie Stas Let us assue that rocess Õ has stored the receive and send tie stas ¼ ËÌ Õ Ôµ and ¼ ÊÌ Õ Ôµ of soe essage Ñ ¼ sent to Õ by Ô (Figure 19). When Õ receives a essage Ñ with tie stas and fro Ô, it udates its entries ËÌ Õ Ôµ and ÊÌ Õ Ôµ. The silest solution would be to just store the tie stas fro the ost recently received essage Ñ. However, this solution could increase the calculated uer bound for any essage Ò that Õ will send to Ô with resect to the uer bound deterined by the tie stas of Ñ ¼. We can uantify the difference between the two uer bounds Ù Ñ Òµ and Ù Ñ ¼ Òµ in the following way: Ù Ñ Òµ Ù Ñ ¼ Òµ Õ µ ½ µ µ ½ µ ¼ µ ½ µ ¼ µ ½ µ µ ½ µ µ ½ µ ¼ µ ½ µ 11

12 ¼ µ ½ µ ¼ µ ½ µ ¼ µ ½ µ (11) Process Õ will thus relace the tie stas of Ñ ¼ by the tie stas of Ñ iff Ñ will guarantee a lower uer bound for any future essage Ò that Õ will send to Ô. Hence, Õ will erfor this relaceent iff Ù Ñ Òµ Ù Ñ ¼ Òµ which is euivalent to C Ù Ñ Òµ Ù Ñ ¼ Òµ Õ½½ ¼ µ ½ µ ¼ µ ½ µ (12) D C D E n F clock tie Figure 19: Process Õ deterines at the recetion of a new essage Ñ fro Ô if the tie stas of Ñ could rovide tighter uer bounds for essages that Õ will send to Ô than the ones of Ñ ¼. µ ½ Õ µ Ø Òµ µ ½ Ô µ µ Õ µ µ Ô µ Ø Òµ Æ ÑÒ µ (15) This euation shows that the error can grow u to ties the distance between the recetion of Ò at clock tie and the transission of Ñ at clock tie (see Figure 16). Hence, to reduce the error ade in calculating an uer bound on the transission delays of essages, we have to ake sure that each rocess Ô gets a eriodic essage fro each connected rocess Õ. We can ensure that by reuiring that each tiely rocess Õ broadcasts at least every, say, clock tie units a essage (see Figure 20). Note that tyically clients of a fail-aware datagra service, such as a grou ebershi or clock synchronization service do eriodic broadcasts anyway to all other rocesses. Thus, if clients send broadcast essages with a freuency of at least ½, the fail-aware datagra service is silent, in the sense that, it only transits essages it was reuested to send by its clients. τ τ τ 8.3 Error of Uer Bound We define the error Ò Ñµ of the uer bound Ù Ò Ñµ on the essage transission delay Ø Ñµ as follows (see Figure 14): Ò Ñµ Ù Ò Ñµ Ø Ò Ñµ (13) Since Ù Ò Ñµ is an uer bound on Ø Ñµ, Ò Ñµ is never negative, i.e. Ò Ñµ ¼. To silify the derivation of an uer bound for Ò Ñµ, let us assue that the drift Ô of rocess Ô s hardware clock À Ô is constant (for the tie interval we consider), i.e. Ô ¾ and let the drift Õ ¾ of À Õ also be constant. The transission delay Ø Ñµ can therefore be exressed as follows (see Figure 16): Ø Ñµ Õ¾ µ µ µ µ ½ Õ µ Ø Òµ µ ½ Ô µ (14) Ò Ñµ Õ½ ½ Ù Ò Ñµ Ø Ò Ñµ µ ½ µ Æ ÑÒ µ ½ µ r T0 T1 T2 T3 clock tie Figure 20: A tiely rocess Õ sends at least every clock tie units a broadcast essage to ake sure that all connected rocesses can udate their tie stas for Õ. To calculate an uer bound on Ò Ñµ for connected rocesses, let us consider two rocesses Ô and Õ that are connected, i.e. the transission delay of any essage sent between Ô and Õ is at ost Æ and the two rocesses send each other essages at least every clock tie units. Let Ò ½ denote the essage associated with the tie stas ËÌ Ô Õµ and ÊÌ Ô Õµ and let Ñ be a essage that Ô sends to Õ at tie (see Figure 21). Furtherore, let Ò ¾ be the next essage fro Õ to Ô after Ò ½. Assue Ò ¾ arrives just after Ô has sent Ñ: for silicity, let us assue Ò ¾ arrives at tie. Let Ü denote the real-tie duration between the send ties of essages Ò ½ and Ò ¾ (see Figure 21). Since Õ is tiely and sends essages at least every clock tie units, Ü is bounded by: ¼ Ü ½ µ (16) 12

13 a b δ in x τ(1+ρ) τ(1+ρ)+δ δ in B n 1 n 2 A δ c C realtie Figure 21: A rocess Ô receives at least every ½ µ Æ Æ ÑÒ real-tie units a essage fro each connected rocess Õ and the first essage Ô receives after becoing connected arrives within ½ µ Æ real-tie units. We assue that Ñ is sent just before Ò ¾ arrives. The ties,, and can be exressed relative to using the transission delays of the essages Ò ½, Ò ¾, and Ñ and the duration Ü: δ Ø Ò ½ µ (17) Ü Ø Ò ¾ µ (18) Ü Ø Ò ¾ µ Ø Ñµ (19) Note that the transission delays of essages Ò ½, Ò ¾, and Ñ are bounded by Æ because we assued that Ô and Õ are connected: Ø Ò ½ µ Ø Ò ¾ µ Ø Ñµ ¾ Æ ÑÒ Æ (20) Since the axiu drift rate of hardware clocks is very sall, the following roerties hold D ½ µ ¾ ½ (21) ½ µ ¾ ¼ (22) The drift rate of Ô s and Õ s hardware clock is bounded by, Ô Õ ¾ (23) Given the above assutions about Ô and Õ, the axiu error Ò Ñµ is bounded by, Ò Ñµ Õ½ µ Õ µ µ Ô µ Ø Ò ½ µ Æ ÑÒ µ Õ¾ Õ½ Õ¾¾ ½½½ µ¾ µ¾ Ø Ò ½ µ Æ ÑÒ µ ½ Õ µ¾ µ ½ Ô µ¾ Ø Ò ½ µ Æ ÑÒ µ¾ µ¾ Ø Ò ½ µ Æ ÑÒ Ü Ø Ò ¾ µ Ø Ñµ µ¾ Ü Ø Ò ¾ µ Ø Ò ½ µµµ¾ d ½¾½¾¼ Ø Ò ½ µ Æ ÑÒ Ü Ø Ò ¾ µ ¾Ø ѵ ½ ¾µØ Ò ½ µ Æ ÑÒ ½ µ Æ ½ ¾µÆ Æ ÑÒ Õ¾¾ ½ µæ Æ ÑÒ (24) Hence, we can set the axiu error ÑÜ of the uer bound on the transission delay of a essage sent between two connected rocesses to ÑÜ Õ¾ ½ µæ Æ ÑÒ (25) The rotocol classifies all essages with a calculated uer bound of at ost as fast and all other essages as slow. To guarantee the Tieliness reuireent, we have to define such that Æ ÑÜ, that is: Õ¾ ¾ µæ Æ ÑÒ (26) This ensures that the calculated uer bound for Ñ is at ost : Ù Ò Ñµ Õ½ Ø Ñµ Ò Ñµ Õ¾ Æ ÑÜ Õ¾ (27) Because it takes at ost ½ µ Æ tie units until rocess Ô receives a essage fro soe connected rocess Õ (see Figure 21), the tieliness reuireent is satisfied for constant ½ µ Æ (28) The fail-awareness reuireent is satisfied because (1) a essage Ñ is classified as slow whenever the calculated bound Ù Ò Ñµ is ore than, and (2) Ù Ò Ñµ is always greater than the actual transission delay Ø Ñµ. 8.4 Pseudo-Code The seudo-code of the rotocol described above is given in Figures 29 and 30. The asynchronous datagra service assued by the tied asynchronous syste odel rovides oerations to send, to broadcast, and to deliver essages: we denote these oerations by the functions send and broadcast, and the ucall event deliver. The syste rovides alar clocks that allow each rocess Ô to set alars: oeration SetAlar(T) tries to wake u Ô within of Ô s hardware clock dislaying value Ì. Process Ô is never awakened before À Ô Øµ Ì but Ô can suffer a erforance failure, that is, Ô can be awaken when À Ô Øµ Ì. The rocess anageent service generates an event ÙÔØÏ ÍÔ Ì µ to infor Ô of the 13

14 alar Ì of the alar clock ÙÔØ. An alar clock stores only the last alar tie, i.e. setting a new alar Í before the syste infored Ô of a revious alar Ì will cancel the revious alar of the alar clock. The oerations fa-send and fa-broadcast allow clients to ass the send tie sta as an otional arguent (see also Section 6): when no tie sta is secified, the current value of the hardware clock (denoted by function À µ) is used as send tie sta. The transission delay of a essage Ñ is the duration between the real-tie when the sender deterines the send tie sta of Ñ and the realtie when the receiver rocess reads its hardware clock to deterine the receive tie sta of Ñ. Note that our rotocol naturally tolerates essage oission failures. To see why, consider Figure 16 with essages Ñ and Ò. If the d-essage Ñ is lost, Õ does not have to deliver it, and hence does not have to coute an uer bound for its transission delay. If the d-essage Ò is lost, Ô and Õ were not connected in the real-tie interval [a,d]. If then the tieliness reuireent does not aly for Ñ, so the fail-aware datagra service does not have to deliver Ñ as a fast essage. If then Õ should have sent another essage Ò to Ô after tie such that, if Ô and Õ are connected in [ ], then Ò is delivered before tie and its tie stas can be used to coute an uer bound within for Ñ. are broadcasted at least every clock tie units by the service contain the colete send and receive tie sta arrays ËÌ and ÊÌ. Furtherore, the heler-essages are sent together with the higher level ebershi essages (whenever a ebershi service is activated) to reduce the nuber of essages transitted. Another issue we address in our ileentation are asyetric disconnection failures: a rocess Ô can send essages but it cannot receive essages 1. When Ô has never received a essage fro Õ, Õ will coute an infinite uer bound for all essages it receives fro Õ because there exists no round-tri that allows the coutation of an uer bound. Our ileentation uses an exiration tie after which the tie stas in ËÌ Ô Õ and ÊÌ Ô Õ have to be relaced by the tie stas of a new essages Ñ even when Ñ does not guarantee better bounds (see Section 8.2). Process Õ that receives a essage fro Ô delivers essages as slow when the iggybacked tie stas and are exired. In other words, in a bounded aount of tie after rocess Ô failed to receive essages fro soe rocess Õ, Õ will deliver all essages fro Ô as slow, allowing its clients to reject these essages and to transfor the one-way connection into a syetric disconnection. Lack of transitivity for the connected relation is another source of instability that we resolve at the fail-aware ebershi abstraction level [12]. 8.5 Ileentation Our ileentation of a fail-aware datagra service on a cluster of SUN workstations rovides soe additional functionality that is useful for soe clients. In articular, the receiver of a essage Ñ can get the calculated uer bound of Ñ as well as the send and receive tie stas of Ñ. Returning the calculated uer bound instead of sily returning the fast or slow Boolean, allows each client to select its own (it is reasonable to exect that different clients of the fail-aware datagra service use different s). The ileentation increases the lower bound Æ ÑÒ of essage transission delays with increasing essage sizes (see Section 9). This allows the rotocol to calculate a better uer bound for the essage transission delays of long essages. Note that the service can set Æ ÑÒ to zero whenever no better lower bound is known. In articular, that allows an installation of a fail-aware datagra service on various latfors without reliinary erforance easureents. However, to calculate a better uer bound, one has to rovide a better lower bound for the transission delays of essages. Our ileentation iniizes the length of essages by iggy-backing only one to three tie-stas on a essage sent by a client: a broadcast essage contains only the send-tie sta, while an unicast essages contains three tie stas,, (see Figure 17 for the labeling of the tie stas). Only heler-essages which 9 Perforance We easured the erforance of our rotocol on the network of SUN IPX workstations linked by a 10Mbs Ethernet at our Deendable Systes Laboratory at UCSD. In the first easureent we deterine the round-tri essage transission ties for four different essage sizes (Figure 22): a rocess Õ sends eriodically a essage Ò to another rocess Ô and Ô relies with a essage Ñ of the sae size as Ò. The easureents shown in Figure 22 deterine the round-tri tie µ µ for four different essage sizes, where are the tie stas of Ò and Ñ as given in Figure 17. The iniu round-tri tie increases with the size of the essages sent (see Figure 23). Our ileentation of a fail-aware datagra service adats the lower bound Æ ÑÒ of the transission of essages to the size of the essages transitted. The increase of the iniu round-tri tie for a essage size above 1472Bytes is due to the fact that UDP has to slit these essages into two Ethernet ackets. The transission tie of broadcast essages is slightly greater than that of unicast essages (Figure 24). The 1 This error actually occurs in soe networks and can be the source of ajor instabilities in higher level rotocols. 14

15 no. round-tris K.5K 1K 1.4K µ s Figure 22: Measured round-tri ties based on 20,000 round-tris for four different essage sizes. Each roundtri consists of two essages of the sae size. in. round-tri delay 4s 3.5s 3s 2.5s 2s broadcast sg round-tris unicast sg round-tris.25kb.5kb 7.5kB 1kB 1.4kB essage size Figure 24: The easured iniu round-tri ties of broadcast essages are greater than the ones of unicast essages. 4.5s eas. unicast round-tri in used unicast round-ti in avg. unicast round-tri delay 4s eas. broadcast round-tri in used broadcast round-tri in in. round-tri delay 3.5s 2.5s round-tri delay 3.5s 3s 2.5s 2s 1.5s.25kB.75kB 1.2kB essage size 1.75kB Figure 23: The easured iniu and average roundtri ties increase with the essage size. Our ileentation increases Æ ÑÒ linearly with the essage size. This grah shows ¾Æ ÑÒ since we lotted round tris..25kb.5kb 7.5kB 1kB 1.4kB essage size Figure 25: The easured iniu round-tri ties of broadcast essages increase with the essage size. Our ileentation uses a slightly greater Æ ÑÒ for broadcast essages than for unicast essages. This figure shows ¾Æ ÑÒ for each essage size. only difference to the above easureents is that the essages are sent as UDP broadcasts instead of UDP unicasts. The increase in transission tie is consistent with the easureents erfored by others [16]. To exclude that this increase is due to the rocessing of the local looback essages, we also easured the round-tri ties when the relies to a broadcast essage by the reote rocess Ô are artificially delayed, i.e. we increased tie. However, that delay did not show any iroveent on the round-tri ties µ µ. To calculate better uer bounds for broadcast essages, our ileentation uses different lower bounds for broadcast and unicast essages (Figure 25). The next easureent shows an uer bound on the error of the uer bounds calculated for unicast essages (Figure 26). We used two rocesses that sent each other ing-ong essages. The figure shows the difference between the calculated uer bound and the iniu essage transission delay Æ ÑÒ. In this easureent, we used the uer bounds of 100,000 unicast essages each with a length of 248 bytes. We also erfored easureents of the uer bounds calculated by the fail-aware datagra service used by a local leadershi service [15]. This easureent involved one rocess Ô eriodically broadcasting essages and another rocess Õ is iediately relying to Ô with a unicast essage. Figure 27 shows the distribution of the uer bounds calculated by Ô for Õ s relies. The next easureent involved six rocesses. We include this easureent to illustrate that the counica- 15

16 no. essages µ s Figure 26: Measured axiu error (in ) of the calculated uer bounds for about 100,000 unicast essages. no. essages µ s Figure 27: Couted uer bounds (in ) for unicast essages based on 100,000 relies. tion attern of an alication can (of course) affect the essage transission delays. This easureent is also based on the local leader election rotocol: one rocess Ô is eriodically broadcasting essages and five other rocesses are relying iediately to Ô. After receiving a rely, Ô first rocesses this rely before it receives the next rely. Hence, the transission delays of a rely essage increases by the rocessing tie of the receding rely essages. This is the reason for the the five eaks in the grah (see Figure 28). no. essages s Figure 28: Couted uer bounds for unicasts based on 500,000 relies. 10 Conclusion One roble in designing rotocols for distributed systes is that undetected slow essages can invalidate safety roerties. Thus, it is iortant to detect slow essages. For exale, a slow essage Ñ fro a sensor has to be rejected since the inforation of Ñ ight be out-ofdate. In this aer, we described a fail-aware datagra service that allows the detection of slow essages by tagging each essage it delivers as either fast or slow. Any essage that is delivered as fast has a transission delay of at ost tie units and hence, any essage with a transission delay of ore than tie units is delivered as slow. Our ileentation of a fail-aware datagra service calculates an uer bound on the transission delay of each essage it delivers. This ileentation iggy-backs at ost three tie-stas on each essage it transits for a client. To suort fa-broadcast essages, a server Ô has to ake sure that it eriodically transits two tie stas (,, see Figure 17) to each connected rocess Õ. In tyical alications the service iggy-backs that inforation on essages sent by higher level services, such as ebershi, and thus, does not have to send any additional essages. Effectively, the service establishes a airwise clock synchronization between connected rocesses. This is achievable even in systes that can artition. We showed how a fail-aware datagra service can be used by rocesses to counicate by tie. The failaware datagra service is the foundation of several other fail-aware services that we have designed (e.g [13]), such as fail-aware clock synchronization, fail-aware ebershi and fail-aware atoic broadcast. The datagra service has been roven useful in the design of safety critical distributed systes[7]. 16

17 References [1] F. Cristian. Probabilistic clock synchronization. Distributed Couting, 3: , [2] F. Cristian. Understanding fault-tolerant distributed systes. Counications of ACM, 34(2):56 78, Feb [3] F. Cristian. Synchronous and asynchronous grou counication. Counications of the ACM, 39(4):88 97, Ar [4] F. Cristian and C. Fetzer. The tied asynchronous distributed syste odel. IEEE Transactions on Parallel and Distributed Systes, ages , Jun [5] F. Cristian and F. Schuck. Agreeing on rocessorgrou ebershi in asynchronous distributed systes. Technical Reort CSE95-428, Det of Couter Science and Engineering, University of California, San Diego, La Jolla, CA, [6] P. H. Dana. Global ositioning syste (gs) tie disseination for real-tie alications. Real-Tie Systes Journal, 12(1), [7] D. Essae, J. Arlat, and D. Powell. Padre: A rotocol for asyetric dulex redundancy. In Proceedings of the Seventh IFIP International Working Conference on Deendable Couting for Critical Alications, San Jose, USA, Jan [8] C. Fetzer. Fail-aware clock synchronization. Technical Reort Tie-Services, Dagstuhl-Seinar-Reort; 138, Mar htt:// [9] C. Fetzer. The essage classification odel. In Proceedings of the 17th ACM Syosiu on Princiles of Distributed Couting, Puerto Vallarta, Mexico, June htt:// [10] C. Fetzer and F. Cristian. Fail-awareness in tied asynchronous systes. In Proceedings of the 15th ACM Syosiu on Princiles of Distributed Couting, ages a, Philadelhia, May htt:// [11] C. Fetzer and F. Cristian. A fail-aware ebershi service. In Proceedings of the 16th Syosiu on Reliable Distributed Systes, ages , Oct htt:// [12] C. Fetzer and F. Cristian. Fail-awareness: An aroach to construct fail-safe alications. In Proceedings of the 27th Annual International Syosiu on Fault-Tolerant Couting, Seattle, Jun htt:// [13] C. Fetzer and F. Cristian. Fortress: A syste to suort fail-aware real-tie alications. In IEEE Worksho on Middleware for Distributed Real-Tie Systes and Services, San Francisco, Dec htt:// [14] C. Fetzer and F. Cristian. Building faulttolerant hardware clocks. In Proceedings of the Seventh IFIP International Working Conference on Deendable Couting for Critical Alications, ages 59 78, San Jose, USA, Jan htt:// [15] C. Fetzer and F. Cristian. A highly available local leader service. IEEE Transactions on Software Engineering, ages , Set.-Oct [16] O. Heranns and M. Schuba. Perforance investigations of the i ulticast architecture. Couter Networks and ISDN Systes, 28: , [17] L. Laort. How to write a long forula. Technical Reort SRC119, Syste Research Center of Digital Euient, Palo Alto, Ca, Dec [18] J. Postel. User datagra rotocol. Technical Reort RFC768, USC / Inforation Sciences Institute, [19] D. Powell. Failure ode assutions and assution coverage. In Proceedings of the 22nd International Syosiu on Fault-Tolerant Couting Systes, ages ,

18 iort tye sg; % essage tye const yid : P; % id of executing rocess const Æ ÑÒ : tie; % iniu transission delay const : tie; % transission tieout delay const : tie; % broadcasts scheduled every const : tie; % ax. drift rate of hardware clocks const : tie; % scheduling tieout delay function H() : tie; % hardware clock var ST: P tie; % send tie stas RT: P tie; % receive tie stas udate: alar clock; % udate alar clock sg ( FA Broadcast, sg, P tie, P tie); ( FA Unicast, sg, tie, tie, tie); ( FA Heler, tie); function calcdelay(: P, A: tie, B: tie, C: tie, D: tie) : tie begin if yid then return (D C)(1+); else return (D A)(1+) (C B)(1 ) Æ ÑÒ ; end rocedure udatestate(: P, C: tie, D: tie) begin C ST(); D RT(); diff (D D )(1 ) (C C )(1+); if (diff 0) then ST() C; RT() D; endif end; Figure 29: Part 1 of the seudo-code for a fail-aware datagra service. 18

19 task FailAwareDatagra exort fa send( : sg, receiver : P, otional sendtiesta : tie); fa broadcast( : sg, otional sendtiesta : tie); iort fa deliver( : sg, sender : P, fast : boolean, receivetiesta : tie); begin udate.setalar(h()); for all ¾ P begin ST() ½; RT() 0; end loo select event when fa send(,, sendtieh()): ST(yid) sendtie; send( FA Unicast,, ST(), RT(), ST(yid)); end select end loo end when fa broadcast(,, sendtieh()): ST(yid) sendtie; broadcast( FA Broadcast,, ST, RT); udate.setalar(sendtie+ ); when deliver( FA Broadcast, ST, RT) fro A ST(yid); B RT(yid); C ST(); D H(); delay calcdelay(,a,b,c,d); udatestate(,c,d); fa deliver(,, delay,d); when deliver( FA Unicast, A, B, C) fro D H(); delay calcdelay(,a,b,c,d); udatestate(,c,d); fa deliver(,, delay,d); when deliver( FA Heler, C) fro D H(); udatestate(,c,d); when udate.wakeu(t): ST(yid) H(); broadcast( FA Heler, ST(yid)); udate.setalar(st(yid)+ ); Figure 30: Part 2 of the seudo-code for a fail-aware datagra service. % events issued by client % event sent to client % broadcast heler essage iediately 19

20 iort tye SensorInfo; % sensor data tye Coand; % coand for actuator const : tie; % scheduling tieout delay const : tie; % sensor reading eriod const Å : tie; % axiu tie between coands const : P; % id of controller rocess const r : P; % id of actuator rocess function H() : tie; % local hardware clock function ReadSensor() : SendorInfo; % read value fro sensor function ProcessData(info : SensorInfo) : Coand; % create coand rocedure ExecuteCoand(cd : Coand); % execute coand cd rocedure FailSafeMode(); % switch to fail safe ode task Sensor var NextRead: alar clock; NextRead.SetAlar(H()+ ); loo select event when NextRead.WakeU(T): now H(); fa send(, ReadSensor(), now); NextRead.SetAlar(now+ ); end select end loo end task Controller loo select event when fa deliver(sensor data, sender, fast, C): if (fast) then fa send(r, ProcessData(sensor data), C); end select end loo end task Actuator var waitcoand : HardRTAlarClock; % reuires hard real tie scheduling! waitcoand.setalar(h()+å); loo select event when fa deliver(coand, sender, fast, E): if (fast) then ExecuteCoand(coand); waitcoand.setalar(e+å); when waitcoand.wakeu(t): FailSafeMode(); end select end loo end Figure 31: Pseudo-code of sensor exale. 20

21 11 Aendix Sy. Sec. Meaning 3 real-tie stas of a roundtri essage air 3 clock tie stas of a roundtri essage air È 4 finite set of rocesses Æ 4 one-way tie-out delay 3 threshold used to classify essages Æ ÑÒ 4 iniu essage transission delay ÑÜ 8.3, E. 25 axiu error of uer bound for a essage sent between two connected rocesses Ò Ñµ 8.3, E. 13 error of uer bound Ù Ò Ñµ fast 3 a osteriori uer bound of a fast essage is at ost fa-essage 3 fail-aware essage, i.e. essage sent by a fail-aware datagra service 7.2 ax. initialization tie after which two connected rocesses have to deliver fast essages À Ô 4 hardware clock of rocess Ô Ñ Ò essages Ô,Õ,Ö rocesses 4 axiu drift rate of a hardware clock Ø Ù Ú real-tie values Ë Ì Í Î clock tie values 4 scheduling tie-out delay slow 3 a osteriori uer bound of a slow essage is greater than ËÌ Ô, ÊÌ Ô 8.3 array of send and receive tie stas aintained by Ô 8.3 iniu broadcast interval of a rocess Ø Ñµ 3 transission delay of essage Ñ Ù Ñµ 3 uer bound on the transission delay Ø Ñµ Ù Ò Ñµ 8.1, E. 8 uer bound of Ñ calculated using essage Ò Biograhies Christof Fetzer received his diloa in couter science fro the University of Kaiserlautern, Gerany (12/92) and his Ph.D. fro UC San Diego (3/97). He is currently a research scientist at UC San Diego and he will join AT&T Labs in He received a two-year scholarshi fro the DAAD and two best student aer awards. Dr. Fetzer has ublished over 25 research aers in the field of distributed systes. Flaviu Cristian is Professor of Couter Science and Engineering at the University of California, San Diego. He received his PhD fro the University of Grenoble, France, in After carrying out research in oerating systes and rograing ethodology in France and working on the secification, design, and verification of fault-tolerant software in England, he joined IBM Research in While at IBM, he worked in the area of faulttolerant distributed systes and rotocols. After joining UCSD in 1991, he founded the Deendable Systes Laboratory where he and his collaborators design and build suort services for roviding high availability in distributed systes. Dr.Cristian has ublished over 100 aers in international journals and conferences in the field of deendable systes. 21