Integrating External and Internal Clock Synchronization *

Transcription

1 appeared n: Real-Tme Systems, Volume 12, Number 2, (1997) Integratng External and Internal Clock Synchronzaton * ork Department of Computer Scence & Engneerng Unversty of Calforna, San Dego La Jolla, CA 92093! 0114 Edtor: Ulrch Schmd fetzer@chrstof.org Abstract. e address the problem of how to ntegrate fault-tolerant external and nternal clock synchronzaton. In ths paper we propose a new externalnternal clock synchronzaton algorthm whch provdes both external and nternal clock synchronzaton for as long as a majorty of the reference tme servers (servers wth access to reference tme) stay correct. hen half or more of the reference tme servers are faulty, the algorthm degrades to a fault-tolerant nternal clock synchronzaton algorthm. e prove that at least 2" +1 reference tme servers are necessary for achevng external clock synchronzaton when up to " reference tme servers can suffer arbtrary falures, thus the proposed algorthm provdes maxmum fault-tolerance. In ths paper we also derve lower bounds for the best maxmum external devaton achevable n standard mode and the best drft rate achevable n degraded mode. Our algorthm s optmal wth respect to these two bounds: (1) the maxmum external devaton s optmal n standard mode, and (2) the drft rate of the clocks s optmal n standard and degraded mode. Keywords: External Clock Synchronzaton, Internal Clock Synchronzaton, Lower Bounds, Optmal External Clock Synchronzaton 1. Introducton Synchronzed clocks are crucal for many fault-tolerant real-tme systems. e assume the exstence of a global tme base called real-tme. Reference tme s a granular representaton of real-tme and s typcally provded by a standard source of tme such as NIST. Clocks can be externally or nternally synchronzed [1]. A clock s externally synchronzed f at any pont n real-tme the dstance between ts value and reference tme s bounded by an a pror gven constant called maxmum external devaton. A set of clocks s nternally synchronzed f at any pont n real-tme the dstance between the values of two correct clocks n the set s bounded by an a pror gven constant called the maxmum nternal devaton and each clock runs wthn a lnear envelope of real tme. Externally synchronzed clocks are always nternally synchronzed by two tmes the maxmum external devaton, but nternally synchronzed clocks are not always externally synchronzed. The systems we consder n ths paper consst of a set of tme servers whch provde tme nformaton to local clents. e assume that each tme server has access to a local hardware clock. Hardware clocks can be vewed as a counter whch s ncremented by the * An earler verson of ths paper was publshed n the Proceedngs of the 15th Internatonal Conference on Dstrbuted Systems, May, Ths work was partally supported by grants from the Ar Force Offce of Scentfc Research, the German Academc Exchange Servce (DAAD), the Powell Foundaton, Sun Mcrosystems, and the Mcroelectroncs Innovaton and Computer Research Opportuntes n Calforna.

2 #$% & ' (*)' +-,.* (3,8. tcks of an oscllator. Because of mprecsons of the oscllator and temperature changes, hardware clocks drft apart from reference tme. The rate of ths drft s typcally very small. However, even when the hardware clocks would be ntally synchronzed wth reference tme, eventually the dstance between a hardware clock and reference tme can be greater than any a pror gven constant. To acheve external synchronzaton, at least one tme server must have access to reference tme. Such servers can use GPS or rado recevers to receve the tme sgnal broadcasts by a standard source of tme. A tme server wth a local access to reference tme s called a reference tme server, the other servers are non-reference tme servers. The falure model assumed n ths paper allows that tme servers suffer arbtrary falures. To llustrate the effects of arbtrary falures, let us assume that the correct non-reference tme servers have to be externally synchronzed wthn, say, one mnute and nternally synchronzed wthn 2 mnutes. A reference tme server 9 whch has suffered an arbtrary falure could tell two non-reference tme servers : and ; at reference tme <= :>> that the tme s <? :=> and <7@ :?>, respectvely. If : and ; would just set ther clocks to ther own approxmaton of 9 s clock (.e. <? :=> and <7@ :?>, respectvely), they would be nether externally nor nternally synchronzed. In general, arbtrary falures of reference tme servers can not only dsturb the external synchronzaton of non-reference tme servers, they can also perturb ther nternal synchronzaton. Snce the cost of the hardware components needed to access external reference tme sgnals s n general not neglgble, one typcally tres to mnmze the number of reference tme servers needed n a gven system. Our frst goal s therefore to nvestgate the ssue of how many reference tme servers are necessary to provde external synchronzaton of non-reference tme servers when up to A reference tme servers can suffer arbtrary falures. Note that the hardware clocks of non-reference tme servers can be used to detect some falures of reference tme servers. For example, f : reads the externally synchronzed clock of reference tme server 9 when : s hardware clock shows <? :>> and <= :>> and : s approxmatons are <? :>> and <? :=>, respectvely, then : knows that 9 s faulty (under the assumpton that : tself s correct) and can reject 9 s clock value. hen all falures of reference tme servers would be detectable, one obvously needs only ACBD< reference tme servers to mask A detectable reference tme server falures. However, snce hardware clocks can drft apart from real-tme, not all falures of reference tme servers are detectable. e prove that even when processes use ther hardware clocks to detect falures of reference tme servers, to provde external synchronzaton when up to A reference tme servers can suffer arbtrary falures, a total of?aebe< reference tme servers s necessary and suffcent. A related queston that we nvestgate s how closely can one bound the drft rate of the clocks of non-reference tme servers when half or more of the reference tme servers can suffer arbtrary falures. hen we requre that non-reference tme servers have to be externally synchronzed as long as a majorty of the reference tme servers stay correct and the maxmum drft rate of hardware clocks s F, we can show the followng result: the best maxmum drft rate of non-reference tme servers achevable when half or more of the reference tme servers have suffered arbtrary falures s?f.

3 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. #$ V In most fault-tolerant real-tme systems, we are aware of, nternal synchronzaton s requred for the correct operaton of the system, whle external synchronzaton s manly used to gve users convenent reference tme nformaton. For example, n the Advanced Automaton System, the correctness of automated recovery from component falures depends on correct nternal clock synchronzaton [2] but the system can contnue to functon n the absence of external clock synchronzaton n a degraded way. Our second goal s to ntegrate external and nternal clock synchronzaton: we requre that an externalnternal clock synchronzaton algorthm mantans externally and nternally synchronzed clocks for as long as at least a majorty of reference tme servers reman correct, and degrades to nternal clock synchronzaton when half or more of the reference tme servers become faulty. The two man motvatons for ntegratng nternal and external clock synchronzaton are as follows: 1. The correctness of some applcatons must be guaranteed ndependently of the correctness or avalablty of reference tme sgnals. For example, when the correctness of an applcaton only depends on nternal clock synchronzaton, externalnternal clock synchronzaton guarantees the applcaton s correctness even when there are no rado tme sgnals or these sgnals are producedforged by an adversary. 2. Reducton of cost by mnmzng the number of reference tme servers. For example, a practcal system could contan only one reference tme server. Ths server would be a sngle pont of falure whch could dsturb the correctness of the complete system. An ntegrated externalnternal clock synchronzaton algorthm wll guarantee that the clocks of correct non-reference tme servers wll stay externally synchronzed for as long as the reference tme server stays correct, and wll be nternally synchronzed even when the reference tme server fals. An externalnternal tme servce s n one of the followng modes: standard, degraded, or ntegraton mode. e ntroduce these modes to specfy the requrements and propertes of the tme servce. However, only a global observer can correctly determne the current mode of a tme servce because ths requres to decde how many reference tme servers are currently faulty. Tme servers cannot always correctly do ths decson (see Secton 5.2) and thus, cannot always correctly determne the current mode. e assume that the system starts n standard mode: (1) all clocks of correct non-reference tme servers are externally and nternally synchronzed, and (2) a majorty of the reference tme servers are correct. In case half or more of the reference tme servers become faulty, the servce transts to degraded mode. hen faled reference tme servers are repared and the correct reference tme servers form a majorty agan, the servce s n ntegraton mode. Durng ntegraton mode the external synchronzaton of correct non-reference tme servers s reestablshed n case t was lost durng degraded mode. The servce transts to standard mode, when the external synchronzaton s reestablshed.

4 #$ X & ' (*)' +-,.* (3,8. In ths paper we also address the queston of how close an external clock synchronzaton algorthm can synchronze a non-reference tme server to reference tme. e show that the best maxmum external devaton achevable s the sum of the followng terms: the maxmum error made by a correct reference tme server n approxmatng reference tme, the maxmum error readng a remote clock, and an addtonal error term whch takes care about the drft of the hardware clock of a correct non-reference tme server between two successve adjustments of ts externally synchronzed clock. The optmal maxmum external devaton s of partcular nterest because external clock synchronzaton algorthms typcally bound the maxmum nternal devaton by twce the maxmum external devaton. hen connectng a reference tme sgnal recever va a standard nterface (lke a seral nterface) to avod addtonal hardware and operatng system support, the external devaton can become qute sgnfcant. For example, n our Dependable Systems Laboratory, a reference tme server can make an error of more than 10 ms approxmatng reference tme because the used GPS recever guarantees no better approxmaton when accessed va ts seral nterface. Usng a smple external clock synchronzaton algorthm would mean that the clocks of non-reference tme servers can be up to 20 ms apart. Our thrd goal s therefore to show how one can bound the maxmum nternal devaton ndependently of the maxmum external devaton. The proposed externalnternal clock synchronzaton algorthm acheves a tght nternal synchronzaton even when the reference tme servers are only loosely synchronzed to reference tme. Other propertes of the proposed algorthm are as follows: n standard mode the maxmum external devaton and the maxmum drft rate of the clocks s optmal, and n degraded mode the maxmum drft rate of the clocks s optmal (about twce the maxmum drft rate of hardware clocks). 2. Related ork Integraton of external and nternal clock synchronzaton has recently been addressed by several authors [11, 12, 5, 9, 15]. The paper of Schmd [11, 12] addresses the ntegraton of external and nternal clock synchronzaton n the context of real-tme systems. Kopetz et al. [9] addressed the problem of how to ntegrate external and nternal clock synchronzaton n the context of synchronzng multcluster real-tme systems. Ther paper only consders one reference tme server. [15] proposes an external synchronzaton scheme that degrades to nternal synchronzaton n case reference tme servers become unavalable. The scheme swtches agan to external synchronzaton as soon as the reference tme servers become avalable agan. Ths approach does not ft completely wth our requrements for externalnternal clock synchronzaton: the swtch from nternal to

5 _ 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. #$ Y external clock synchronzaton can volate our requrement that the drft rates of externallynternally synchronzed clocks are always bounded by a small constant (see Secton 4.3). 3. System Model e consder dstrbuted systems consstng of computer nodes connected by a communcaton network. The communcaton servce allows the processes that run on the nodes to communcate wth each other by exchangng messages. Each node runs ether a reference tme server or a non-reference tme server process (see Fgure 1). The reference tme servers have local access to reference tme,.e. they can approxmate reference tme wthout communcatng wth other nodes. The goal of reference tme servers s to provde access to reference tme for the non-reference tme servers: each reference tme server mantans a reference clock (Secton 3.2) whch can be read by non-reference tme servers. Each tme server has a local hardware clock (Secton 3.3). Non-reference tme servers have no local mean to approxmate reference tme and have to read remote reference clocks to do that. The goal of non-reference tme servers s to provde an externallynternally synchronzed vrtual clock (Secton 3.4) for local clents. e denote the set of tme servers by Z, the set of reference tme servers by Z[, and the set of non-reference tme servers by Z]\. Let ^ `ba Z a, ^c[ `da Ze[ a, and ^c\ `ba Ze\ a denote the number of tme servers, reference tme servers, and non-reference tme servers, respectvely. e assume that each reference and each non-reference tme server has a rank whch s unque amongst the reference and non-reference tme servers, respectvely. e denote the rank of a tme server by functon 9fgh and can formally express these constrans by: 9Oj-Z [ : 9fghHk69lejnm> oqprpos^ [2t <u 9vow9xyjzZ [ : 9fghHk69vl ` 9fghHk69qxl {}9v ` 9x :JjzZ \ : 9fghHk~:lj m>o7ppro ^ \ t <u :os;ƒj Z \ : 9fghHk :4l ` 9fghHk ;l {ˆ: ` ; e sometmes use the term process or server to refer to a tme server Reference Tme e assume that the granularty of real-tme s nfntesmal, and we denote wth zš the set of all real-tme values. Let us also assume that wth each event one can assocate ts pont of occurrence n real-tme ŒŽ k l [10]. Reference tme s a granular representaton of (see real-tme. It s defned by a sequence of tcks Œ 9, jcm> o7<o? o7pprpru and Œ 9q j zš Fgure 2). Tck Œ 9 represents the pont n real-tme when reference tme Œ 9 starts. More precsely, reference tme satsfes the followng requrements: 1. the dstance between two successve tcks s exactly N, where constant s the granularty of reference tme: ŒŽ k6œ 9 l t ŒŽ k6œ 9 l ` o

6 #$ & ' (*)' +-,.* (3,8. GPS-Recever GPS-Recever Reference Server... Reference Server node node LAN Clent Non-reference server Clent node... Clent Non-reference server Clent node Fgure 1. Typcal archtecture of an externalnternal tme servce. 2. real-tme and reference tme are n perfect synchrony: Œ 9q ` ŒŽ k6œ 9q lo 3. for any and any pont n real-tme Œ such that Œ 9 š Œœ Œ 9 the value of reference tme at real-tme Œ s Œ 9. The granularty of reference tme s neglgble for the systems we consder. In what follows, we wll therefore not dstngush between reference tme and real-tme and sometmes use term real-tme nstead of term reference tme. t 3 reference tme t 2 t 1 t 0 t 0 t 1 t 2 t 3 real-tme Fgure 2. Reference tme represents real-tme by a sequence of tcks žÿ * y ẅ «ª s.

7 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. #$ 3.2. Reference Clocks Reference tme servers have a local access to reference tme whch allows them to approxmate reference tme at any pont n tme wthn some a pror gven error. To abstract from mplementaton detals of ths access, we assume that each reference tme server 9 mantans a reference clock ±z whch represents 9 s current approxmaton of reference tme. The reference clock of a correct reference tme server s wthn a constant nterval of real-tme (see Fgure 3). e represent the reference clock of a reference tme server 9 by a functon ±² from real-tme to clock-tme: ±² ³ zš {dµ*š. Term ±² kgœwl denotes the value of 9 s reference clock at real-tme Œ. A reference clock ±- s correct at tme Œ f the devaton between real-tme and ±z s bounded by an a pror gven constant : a ±² kgœwl t Œ a š p The drft rate of a correct reference clock ±z s zero, because for any two ponts and n real-tme such that ¹ the followng property holds: kg t l t? š ±² kg Hl t ±² k l š kg t lbº?. Reference clocks can be mplemented n varous ways. For example, n our Dependable Systems Laboratory two workstatons have access to a GPS-recever va a seral nterface and a thread n each reference tme server : perodcally reads the current reference tme provded by the GPS-recever to determne the dfference» between reference tme and : s hardware clock. The current value of : s reference clock s defned by the sum of the current value of : s hardware clock plus the last computed dfference value». In our system the maxmum devaton between a reference clock and reference tme s more than 10 mllseconds. Implementng reference clocks n the operatng system kernel or n hardware could reduce the devaton to a few mcroseconds. T3 clock-tme t+ Xr t- T2 T 1 T0 real-tme t 0 t 1 t 2 t 3 Fgure 3. Process s reference clock ¼e½ s wthn ¾ of real-tme.

8 v v # NÀ & ' (*)' +-,.* (3, Hardware Clocks Each tme server has access to a local hardware clock. A hardware clock typcally conssts of an oscllator and a countng regster that s ncremented by the tcks of the oscllator. A hardware clock has a certan granularty Á : a tck of the oscllator ncrements the value of the hardware clock by Á. Correct hardware clocks are therefore monotoncally ncreasng. Because of mpressons of the oscllator, temperature changes, and agng, a hardware clock can drft apart from real-tme. The drft of a hardware clock s however wthn a narrow envelope of real-tme (see Fgure 4). e denote wth µ*š the set of clock values,.e. µ*š contans all values a hardware clock can dsplay. e represent process : s hardware clock by a functon ÂÄ from real-tme to clock-tme:  zš {}µhš. Term  kgœwl therefore denotes the value dsplayed by the counter of : s hardware clock at real-tme Œ. A hardware clock  s correct at tme f for all tmes and Œ such that š Œ š,  measures the passage of tme durng real-tme nterval Å owœ Æ wth an error of at most F4kGŒ t l B Á, where F s the maxmum hardware clock drft rate specfed by the clock manufacturer: (1-F )(t-s)-g š  (t)-â k l š (1+F )(t-s)+g. For most quartz clocks avalable n modern computers, the constant F s of the order of <q> Ç È to <> ÇÉ. Snce F s such a small quantty, we gnore terms F for?. For example, we equate kž<êbºf l Ç wth kž< t FNl and kž< t FNl Ç wth kž<ebºfnl. In what follows, we wll assume that the granularty of hardware clocks s neglgble. T3 clock-tme dt -1 = +ρ dt dt -1 = 0 dt H p T2 T 1 dt -1 = ρ dt T0 t 0 t 1 t 2 t 3 real-tme Fgure 4. The drft rate of process Ë s hardware clock Ì Í s wthn [-Î,+Î ].

9 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. # 4# 3.4. Vrtual Clocks Clock synchronzaton algorthms do n general not synchronze the local hardware clocks of nodes. Instead, they ntroduce the concept of vrtual clocks whch are mantaned by tme servers (see Fgure 5). Our goal wll be to ensure that the vrtual clocks of correct non-reference servers are externallynternally synchronzed. Clents can read the vrtual clock of ther local tme server to access reference tme (standard mode) or nternal tme (degraded and ntegraton mode). Snce we assume that only non-reference tme servers have local clents, only non-reference tme servers mantan vrtual clocks. In what follows we wll use the term correct vrtual clock to denote the vrtual clock of a correct nonreference tme server. The vrtual clock of tme server :SjJZ Ï s represented by a mappng Ð from real-tme to clock-tme: Ð Ñ zš {}µhš. Term Ð k6œwl therefore denotes the value of : s vrtual clock at real-tme Œ. In the protocols we propose, the value Ð k6œwl s determned by addng to the local hardware clock  k6œwl a perodcally updated adjustment value. e denote the ponts n realtme when a process : updates ts adjustments value (that s, adjusts ts vrtual clock) by an nfnte, strctly ncreasng sequence kgœ Ò l. The value of : s adjustment value between two successve adjustments that occur at tmes Œ Ò and ŒsÒ s denoted by Ó³Ò : for any tme Œj Å ŒwÒ owœwò l the value of : s adjustment value at tme Œ s ÓLÒ. e say that process : s n round h at tme Œ ff Œ Ò š Œz ÔŒwÒ. e assume that for any tme ŒȨŒŽÕ there exsts a h such that ŒsÒ š Œ] ÖŒwÒ. Process : s vrtual clock at Œ s defned by, Ð k6œwl ` Ó Ò B  kgœwl. To adjust ts clock, a process has to read remote vrtual and reference clocks. In partcular, n the protocols we propose a process : wants to read the remote vrtual clocks wth respect to the adjustment values of ts current round h. e therefore use the well known concept of round clocks. Process : s round clock ÐcÒ for round h s defned by: ŒØ ÐyÒ kgœwl ` ÓÒ BÙÂO k6œwl. Note that round clocks are monotoncally ncreasng because the hardware clocks are monotoncally ncreasng and the adjustment value ÓyÒ of round clock ÐƒÒ s constant. A vrtual clock Ð s not necessarly monotonc because decreasng : s adjustment value at some tme Œ Ò from Ó Ò to Ó Ò ÚÓ Ò decreases the value of Ð at tme Œ Ò Remote Clock Readngs In the proposed clock synchronzaton protocols each non-reference tme server estmates at the end of round h the values dsplayed by all round clocks of round h and the values of the reference clocks. Ths approxmaton s performed by a remote clock readng method. The readng method allows each tme server : to estmate the value dsplayed by a remote round clock Ð ÛÒ or a reference clock ±² when : s own current round clock Ð Ò dsplays some gven tme Ü.

10 # $ & ' (*)' +-,.* (3,8. Vrtual Clock + Adjustment Value Hardware Clock Fgure 5. Example: process Ë s vrtual clock ÝÍ s defned as the sum of the current value of Ë s hardware clock and a perodcally determned adjustment value. Let term µò Û kgüœo :l denote : s approxmaton of ; s round clock ÐOÒ Û at the tme when : s round clock Ð Ò shows Ü. e assume that the remote clock readng error provded by the readng method s bounded by an a pror gven constant Þ, to be called the maxmum clock readng error. hen the remode clock readng method s nvoked by process: durng round h and : requests to read ; s round clock ÐOÒ Û at a tme when : s current round clock ÐyÒ shows Ü, we requre that for any tme Œ such that ÐOÒ k6œwl ` Ü and : and ; are correct at Œ : a ÐyÒ Û kgœwl t µò Û kgü]o6:l a š The remote clock readng method can also estmate remote reference clocks. e denote wth ßÒ kgüœo :l the approxmaton of reference clock ± by a non-reference tme server : when : s current round clock Ð Ò dsplays Ü. e assume that the readng error for remote reference clocks s bounded by the same constant Þ. hen process : nvokes the method at tme durng ts current round h and : requests to read reference clock ± at a tme : s current round clock ÐƒÒ shows Ü, we requre that for any tme Œ such that ÐOÒ kgœwl ` Ü and : and 9 are correct n nterval Å owœ Æ : a ±² kgœwl t ßÒ k6üœo6:4l a š Þ]p The executon of the remote clock readng method takes tme. e assume that any remote clock readng performed by a correct process : termnates n at most àl» clocktme unts: when : nvokes the method at real-tme to approxmate a reference clock ± (or a remote round clock ÐƒÒ Û ) for a tme when : s round clock ÐƒÒ shows Ü, then the clock readng method returns ts approxmaton of ±z (or ÐyÒ Û ) at some tme, and : s hardware clock progressed by at most àl» between tmes and :  kg Hl t  k l š àl». Thus, à³»skž<lbef l real-tme unts are suffcent to read any remote round or reference clock. To be able to bound the clock readng error by a constant, the tme Ü at whch the remote clock s approxmated has to be constraned: when a process : at real-tme approxmates Þ]p

11 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. # the value that ± (or ÐyÒ Û ) shows at real-tme Œ, the readng error contans a part that s proportonal to the dstance between Œ and because (1) : does not know the the actual drft rates of  (and Â Û ), and (2)  and ± can drft apart by up to F a Œ t a between Œ and, whle (3)  and Â Û can drft apart by up to?f a Œ t a between tmes Œ and. In other terms, we have to constran the dstance between Œ and to be able to bound the readng error by an a pror gven constant. Therefore, we assume that ÐyÒ has not yet shown tme Ü when : nvokes the method at tme : ÐcÒ k«le ÙÜ, and ÐyÒ shows Ü n at most àl» clock-tme unts after : has nvoked the method: ÐcÒ k l*b à³» š Ü. Readng a local hardware, reference, or vrtual clock can be done wth a much smaller error than readng a remote clock. e therefore assume that a process can read a local clock wth a neglgble readng error Falure Model e assume that hardware clocks, reference clocks, and processes can experence arbtrary falures. A correct tme server has a correct hardware clock, and a correct reference tme server has a correct reference clock. For smplcty, we assume that a faled non-reference tme server : does not recover: when : s faulty at real-tme, then : s faulty for all tmes Œ. Thus, we do not explan the rentegraton of recovered non-reference tme servers. However, we address ssues related to the repar of reference tme servers. Our falure model allows faulty reference tme servers to recover. In Secton 8 we descrbe how the external synchronzaton of correct vrtual clocks can be reestablshed dung ntegraton mode n case t was lost durng degraded mode. In ths paper we do not dscuss clock readng falures whch occur when a correct tme server : estmates a remote vrtual or reference clock of a correct tme server ; wth an error greater than Þ. Methods for makng synchronzaton algorthms tolerant to remote clock readng falures are proposed n [3]. 4. Requrements In ths secton we wll defne nternal, external, and externalnternal clock synchronzaton more precsely Internal Clock Synchronzaton The frst objectve of an nternal clock synchronzaton algorthm s to bound the devaton between any two correct vrtual clocks by a constant á to be called the maxmum nternal devaton.

12 # % & ' (*)' +-,.* (3,8. Requrement (bounded nternal devaton): for any two non-reference tme servers : and ; that are correct at real-tme Œ : a Ð kgœwl t Ð Û k6œwl a š á. The second goal s to bound the drft rate of vrtual clocks by a constant Fâã 1. To account for the granularty and the adjustments of vrtual clocks, a constant» s ntroduced. Constant» s called the maxmum dscontnuty of vrtual clocks. Requrement (bounded drft rate): for any non-reference tme server : that s correct at tme Œ and for all tmes š Œ : (1-F â )(t-s)-dš C (t)-c (s)š (1+F â )(t-s)+»2p An nternal clock synchronzaton algorthm s correct when t satsfes both of the above requrements. The two requrements of nternal clock synchronzaton allow that a vrtual clock s non-monotonc,.e. there can exst a correct process : and two ponts n real-tme and ŒÑäå such that Ð k«lääåð kgœwl. The clock synchronzaton protocols we propose wll actually show such a non-monotonc behavor for certan runs. However, the protocols can be extended to provde monotonc clock behavor usng contnuous clock amortzaton wthout ncreasng the maxmum nternal devaton nor the maxmum external devaton [13]. The dea s that any adjustment of a vrtual clock can be spread over some tme nterval, that s, effectvely the speed of a vrtual clock s temporarly slowed down or ncreased. Contnuous clock amortzaton does therefore not have to adjust a vrtual clock backwards External Clock Synchronzaton A correct external clock synchronzaton algorthm has to bound the devaton between a correct vrtual clock and real-tme by an a pror gven constant æ, to be called the maxmum external devaton. Requrement (bounded external devaton): for any tme Œ and any non-reference tme server : that s correct at Œ, a Ð k6œwl t Œ a š æp Lke the drft rate of a reference clock, the drft rate of an externally synchronzed clock s zero ExternalInternal Clock Synchronzaton Externalnternal clock synchronzaton requres that n standard mode all correct vrtual clocks are externally and nternally synchronzed, n degraded and ntegraton mode all correct vrtual clocks are nternally synchronzed, and external synchronzaton s eventually reestablshed durng ntegraton mode (unless half or more of the reference tme servers become faulty agan).

13 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. # V To defne the requrements of externalnternal clock synchronzaton more precsely, we frst gve a precse defnton of the dfferent modes of an externalnternal tme servce. The defnton of degraded mode s based on the fact that a non-reference tme server has to read a majorty of reference clocks to be able to bound the external devaton of ts vrtual clock (see Secton 5.2). To guarantee that a correct process succeeds to read a majorty of correct reference clocks, t s actually not suffcent to requre that at any pont n tme there exsts a majorty of correct reference tme servers. To explan ths, consder the followng. hen the remote clock readng method s nvoked at some pont n tme to read reference clock ±, the method can take up to àl»¹kw<]búfnl tme unts to return ts approxmaton of ± for some pont n tme Œ š BÚà³»SkŽ<ÊBºF l. The approxmaton returned s only guaranteed to be wthn çb-þ of real-tme Œ when 9 s correct for all tmes between and Œ (see Secton 3.5). Thus, we have to requre that a majorty of reference tme servers are correct n nterval Å Qow ƒb à³»skž<èbnf l Æ to ensure that a correct process that nvokes the remote clock readng method at (to read all reference clocks) succeeds to read a majorty of correct reference clocks wth an error of at most Þ. Knowng that, n the followng we use expresson that a majorty of reference tme servers are correct at Œ to say that there exsts a majorty é of reference tme servers such that each 92jÚé s correct n nterval Å Œ t àl»¹kž<øb FNlêowŒ Æ. The servce s n degraded mode at tme Œ ff there exsts no majorty of reference tme servers that are correct at Œ. The servce s n ntegraton mode at Œ ff a majorty of reference tme servers are correct at Œ, the servce swtched from degraded mode to ntegraton mode at some tme ƒ ÚŒ : the servce s n degraded mode at and n ntegraton mode durng nterval k owœwl, and not all correct vrtual clocks are externally synchronzed at tme Œ. The servce s n standard mode at Œ ff a majorty of reference tme servers are correct at Œ, the servce swtched from degraded mode or ntegraton mode to standard mode at Œ : the servce s n degraded mode or ntegraton mode n some non- some tme š empty nterval Å Qo l, n standard mode durng nterval Å owœwl, and all correct vrtual clocks were externally synchronzed by æ t F9ëeìí at tme. e assume that an externalnternal tme servce starts ts executon at some pont n real-tme. For smplcty, we assume that the servce s n standard mode at tme and that ntally all correct vrtual clocks are externally and nternally synchronzed. A correct externalnternal clock synchronzaton protocol has to satsfy requrements (bounded nternal devaton), (bounded drft rate) and the followng two requrements: Requrement (bounded externalnternal devaton): for any tme Œ such that the servce s n standard mode at tme Œ and any non-reference tme server : whch s correct at Œ : a Ð k6œwl t Œ a š æp

14 # X & ' (*)' +-,.* (3,8. e also requre that the external clock synchronzaton s eventually reestablshed durng ntegraton mode unless the servce transts to degraded mode agan: Requrement (ntegraton requrement): for any tme Œ such that the servce s n ntegraton mode at Œ, there exsts a tme îäeœ such that the system s n standard mode or degraded mode at. 5. Redundancy Requrements for External Clock Synchronzaton In ths secton we derve a lower and an upper bound for the number of reference tme servers needed to provde externally synchronzed clocks n case that up to A reference tme servers can suffer arbtrary falures. Usng the lower bound, we derve also a lower bound on the best maxmum drft rate of vrtual clocks achevable by an externalnternal clock synchronzaton algorthm n degraded mode. In systems wthout access to reference tme and wthout message authentcaton, at least 3A +1 tme servers are needed to guarantee nternal clock synchronzaton despte A arbtrary tme server falures [6]. e show below that 2A +1 reference tme servers are needed and suffcent to guarantee external synchronzaton of non-reference tme servers n case that up to A reference tme servers can suffer arbtrary falures. Snce external clock synchronzaton mples nternal clock synchronzaton, we have to explan why external clock synchronzaton needs less redundancy. To do ths, let us attempt to apply an external clock synchronzaton algorthm to synchronze the vrtual clocks of non-reference tme servers nternally: we assgn a set à of 2A +1 non-reference tme servers the role of reference servers. The set ï of remanng non-reference tme servers synchronze ther clocks to the servers n à usng an external clock synchronzaton algorthm. Obvously, ths scheme are them- only provdes nternal synchronzaton of the clocks n ï when the clocks n à selves nternally synchronzed. Achevng ths nternal synchronzaton of the servers n à has to be provded by another algorthm,.e. an nternal clock synchronzaton algorthm. However, the clocks n à cannot be nternally synchronzed because A of the 2A +1 servers n à can fal [6]. hen we replace the servers n à by reference tme servers, these servers receve external tme sgnals whch allow them to synchronze ther clocks externally and therefore also nternally. The 2A +1 lower bound result for external clock synchronzaton depends on the assumpton that reference tme servers can suffer arbtrary falures. In partcular, t depends on the fact that not all arbtrary falures of reference tme servers are detectable. By detectable we mean that whenever a non-reference tme server : tres to read a reference clock ±, then : can decde f 9 s correct or faulty. For example, n synchronous systems crash falures are detectable and n case only crash falures can occur, all reference tme server falures are detectable. Only A +1 reference tme servers are needed to mask A detectable reference tme server falures because a non-reference tme server : can acheve external clock synchronzaton n the followng way: 1. : reads at least every, say, 9 ëeìí tme unts all reference clocks, 2. : rejects all readngs from reference tme servers whch are detectably faulty, and

15 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. # Y 3. : synchronzes ts clock to one of the remanng, that s, correct reference clock readngs. out of A +1 reference tme servers suffer detectable falures, : succeeds to read the clock of a correct reference tme server 9 such that : wll not reject 9 s readng and synchronze ts clocks to ts approxmaton of 9 s reference clock. Thus, : s clock s externally synchronzed. Our lower bound proof wll use the fact that hardware clocks can drft apart from realtme and that ths drft permts faulty reference tme servers to drft apart from real-tme (wthn certan bounds) wthout beng detectable by non-reference tme servers. Ths proof allows us to show that n degraded mode the maxmum drft rate of correct vrtual clocks s at least twce the maxmum hardware clock drft rate. The upper bound proof s based on a smple external clock synchronzaton protocol whch acheves external synchronzaton as long as a majorty of reference tme servers stay correct. hen at most A 5.1. Maxmum Replcaton e propose an external clock synchronzaton algorthm ðoðyï to show that 2A +1 reference tme servers are suffcent to guarantee external clock synchronzaton under the assumpton that up to A reference tme servers can suffer arbtrary falures. Our second goal s to show an upper bound for the maxmum external devaton achevable by an external clock synchronzaton algorthm. Let us assume that any correct vrtual clock ÐÊ s adjusted at least every 9 ëeìí tme unts. Algorthm ðoðyï acheves a maxmum external devaton of æ š EBÙÞ B F9 ëêìêí, where accounts for the maxmum external devaton of reference clocks, Þ for the maxmum error : makes n readng correct reference clocks, and F9 ëeìí for the maxmum drft of : s hardware clock between two adjustments. ncorrect readngs wthn +Λ of real-tme ncorrect readngs +Λ t +Λ X[0]... X[l]... X[F]... X[o]... X[N R -1] F+1 > F Fgure 6. The " +1 smallest and " +1 greatest readngs of reference clocks contan the readng of at least one correct reference clock (denoted by ¼cñ ò ó and ¼Ññ ôsó, respectvely). The " +1th smallest clock readng (¼cñ "Qó ) s therefore wthn ¾2õÄö of real-tme.

16 _ ` _ ` # & ' (*)' +-,.* (3,8. The ntuton behnd ðcðyï s as follows (see Fgure 6). Each correct non-reference tme server : reads at least every, say, 9ëÊìêí tme unts all reference clocks. Server : sorts these readngs n ncreasng order and stores them n an array X[0::^ [ -1]. e assume that at most A reference tme servers are faulty and that there are at least A B < correct reference tme servers,.e. ^c[ø ù?acbc<. Therefore, amongst the A +1 smallest readngs there exsts at least one readng that belongs to a correct reference clock. In partcular, there exsts an ndex ú such that > š ú š A and entry ±ÙÅ ú Æ belongs to a correct reference clock. Smlarly, we can dentfy a readng of a correct reference clock amongst the A +1 greatest clock readngs and therefore fnd an ndex such that ^Ä[ t A t < š š ^O[ t < and entry ±ÙÅ Æ belongs to a correct reference clock. Snce we assume that ^²[Ô ˆ?ACBC<, entry ±ÙÅ A³Æ s between ±ÙÅ ú Æ and ±ÙÅ Æ,.e. ú š A š and because array ± s sorted, ±ÙÅ ú Æ š ±ºÅ A³Æ š ±ºÅ Æ. Because ±ÙÅ úgæ belongs to a correct reference clock, ±ÙÅ úgæ s at most BJÞ smaller than Œ (see Fgure 6). Smlarly, because ±ÙÅ Æ belongs to a correct reference clock, ±ºÅ Æ s at most DBÙÞ greater than Œ. Snce entry ±ºÅ A³Æ s between ±ÙÅ ú Æ and ±ÙÅ Æ, t s at most ÖBÙÞ apart from real-tme. A pseudo-code descrpton of algorthm ðoðyï s gven n Fgure 7. Ths algorthm s executed by all non-reference tme servers. e descrbe the executon of algorthm ðcðyï has to externally synchronze by a correct non-reference tme server :. Algorthm ðoðyï : s vrtual clock Ð whch s represented by functon Ð (lne 2). The external functon  k l returns the current value of : s hardware clock. The value of clock Ð at tme Œ s the sum of : s hardware clock at Œ and the value of : s adjustment varable Ó at Œ. Process : adjusts ts vrtual clock n an endless loop (lnes 4-8): 1. : reads all reference clocks (lne 6) for some pont Œ n real-tme when ts current round clock wll show Ü (lne 5), 2. : sorts the clock readngs n ncreasng order and stores them n array ± (lne 7), and 3. : adjusts ts vrtual clock by the dfference between the value : s new round clock should show at Œ (whch s ±ÙÅ A³Æ ) and the value : s current round clock shows at Œ (whch s Ü ). The approxmaton of a reference clock ; s performed by the external functon ß. e assume that all executon tmes except that of the remote clock readng method (whch can take up to à³» clock-tme unts) are neglgble. The executon tme of the loop body (lnes 5-8) s therefore at most àl» clock-tme unts because the approxmaton of the reference clocks s done n parallel. Moreover, : s clock Ð shows a value of at most Ü when startng the executon of lne (8). The adjustment value Ó s updated at least every 9qëeìí àl»¹kž<øbùf l real-tme unts. The maxmum external devaton of vrtual clock Ð s therefore at most æ ûbeþúb F9 ëeìí because : s hardware clock drfts apart from real-tme by at most F9 ëeìí between two adjustments. THEOREM 1 For any tme Œ and any non-reference tme server : that s correct at Œ and executes algorthm ðoðyï : a Ð kgœwl t Œ a š æ.

17 v v 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. # (1) varable Tme A = 0, X[0::^ [ -1], T; (2) functon Tme C() m return H() + A; u (3) functon vod synchronzer () m (4) forever m (5) T = C() + RD; (6) parallel m ;ƒj-ze[ ±ÙÅ 9fgh*k6;l Æ ` ß Û k6ülèu (7) sort(x); (8) Ó ` ÓºB k6±ùå A³Æ t ܳl2uÚu Fgure 7. External clock synchronzaton algorthm. Proof: To analyze algorthm ðcðyï, we descrbe the executon of ðoðyï by a correct nonreference tme server : by fve nfnte sequences: sequence k«ò l conssts of the ponts n real-tme when: adjusts ts clock,.e. the ponts n real-tme when : updates adjustment varable Ó n lne (8), sequence k Ó³Ò content of varable Ó sequence kgü³ò l : entry Ó³Ò denotes the new content of varable Ó s changed at tme Ò from Ó³Ò Ç to Ó³Ò, l : entry ÜÒ Ç denotes the content of varable Ü at tme Ò at tme Ò,.e. the sequence k6œwò l conssts of the ponts n real-tme for whch: approxmates the reference clocks, and sequence kg± Ò l : entry ± Ò denotes the content of array ± at tme Ò. by For any round hzäú> and any tme Œ such that Ò š Œ Ò,, : s vrtual clock s defned Ð k6œwl ` Ó³Ò B  kgœwl. Process : approxmates the reference clocks at the end of round h (.e. at tme Œ Ò wth respect to ts adjustment value ÓLÒ of round h. The relaton between ÜLÒ and ŒwÒ therefore, Readng a remote clock takes at most àl» ÓÒ B ÂO kgœwò l ` ÜÒ ) s. clock-tme unts. Snce we neglect all other t Ò s bounded by, executon tmes, for any round h the dstance Ò >Ñ Ò t Ò š àl»¹kw<øbîf l ` 9qëeìí p By defnton of Ü (lne 5), the maxmum dstance between Ò and ŒsÒ s bounded by, > š ŒwÒ t Ò š 9 ëeìí p The error : makes readng a correct reference clock s bounded by Þ : for any reference tme server 9 such that 9 s correct at ŒsÒ, when entry of the sorted array ±çò contans : s

18 š Ò v ` v v v v v Ò #% À & ' (*)' +-,.* (3,8. approxmaton of ± for tme ŒwÒ, then a ± Ò Å «Æ t ±² k6œ Ò l a š Þ. Snce at most A reference tme servers can suffer falures, the array ± Ò has at most A entres that belong to faulty reference servers. Snce each of the two sub-arrays ±ÙÅ >² NA³Æ and ±ÙÅ Aü r *^c[ t <êæ has at least AÖBý< entres (^c[ý å?aebý< ), there exst two ndces úêjúm>o7pprposañu and jºmaêoqprpos^c[ t <u such that the entres ± Ò Å úgæ and ± Å Æ contan the approxmatons of two correct reference tme servers. Because a correct reference clock s at most apart from real-tme and a correct tme server succeeds to read the reference clock of a correct reference tme server wth an error of at most Þ, the followng propertes are vald: These two propertes mply that Snce the entres of array ± Thus, by transtvty: Ths s equvalently to: By defnton of Ó a ±nò Å ú Æ t ŒwÒ a š EBÙÞ, a ± Ò Å Æ t ŒwÒ a š ÖB Þ. ŒwÒ t t Þ š ±nò Å ú Æ, ± Ò Å Æ š ŒwÒ Bº EBÙÞ. are sorted n ncreasng order and ú š A š : ±nò Å ú Æ š ±nò Å A³Æ š ±nò Å Æ. at tme ŒsÒ ÓÒ ` Usng the fact that Ü³Ò Ç ` ÓÒ Ç ÓÒ ÓÒ Ç By smplfcaton, Œ Ò t t Þ š ± Ò Å A³Æ š a ±nò Å A³Æ t ŒwÒ a š ÖBÙÞ. (lne 8), the followng property holds: Œ Ò Bº ÖBÙÞ. Ó³Ò Ç B k6±nò Å A³Æ t Ü³Ò Ç l. B  kgœwò l (by defnton of Ð ), we get B kg±nò Å A³Æ t ÓÒ Ç t  k6œwò lwl. Ó Ò ` ± Ò Å A³Æ t  k6œ Ò l. By nsertng that property n the defnton of Ð, Ð k6œwl ` k6±nò Å A³Æ t  kgœwò lslbùâ k6œwl. Snce the drft rate of : s hardware clock s bounded by F, and Œ s bounded by Ò š Ò and a Œ t ŒwÒ a š 9qëeìí, we can derve: Œ a Ð k6œwl t Œ a ` a kg± Ò Å A³Æ t  kgœwò lbùâ k6œwlwl t Œ a ` a kg± Å A³Æ -Œ Ò lb k ÂO k6œwl -ƒk6Œ Ò lwl t š ÖB ÞçBîF a Œ t ŒwÒ a ÖB ÞçBîF9qëeìí ` æ. kgœ -Œ Ò l a þ

19 ÿ 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. #%*# 5.2. Mnmum Redundancy e wll show n the lower bound theorem that at least 2A +1 reference tme servers are needed to mask up to A arbtrary falures of reference tme servers. Snce we are nterested n external synchronzaton of non-reference tme servers, we assume that there exsts at least one non-reference tme server,.e. Zœ\ `. The ntuton behnd ths theorem s that a non-reference tme server : cannot always correctly decde whch reference tme servers are correct and whch are faulty. In partcular, when a faulty reference tme server 9 drfts apart from real-tme by at most?f, : cannot necessarly detect that 9 s faulty because : s hardware clock can also drft apart from real-tme by up to F (see Fgure 8). hen half of the reference tme servers would be faulty, n a scenaro lke the one depcted n Fgure 8, : has to be able to decde whch reference clocks are correct to bound the external devaton: : has eventually to decde f t adjusts ts clock towards ± v or ± x. T3 T2 clock-tme dt dt -1 = dr + ρ X r1 H p dt dt -1 = X r2 dr T 1 dt dt -1 = dr ρ T0 t 0 t 1 t 2 t 3 real-tme Fgure 8. Process Ë cannot correctly decde f reference clock ¼ ½ or ¼ ½ (or both) are faulty, because Ë does not know the drft rate of ts hardware clock: (1) when!*î, ¼ ½ s faulty, (2) when çî, ¼ ½ s faulty, (3) when ²Î, ¼ ½ and ¼ ½ are faulty. The proof of the lower bound theorem s done by contradcton. hen less than 2A +1 reference tme servers are avalable, for any synchronzaton algorthm that clams to acheve external synchronzaton t s possble to fnd two for non-reference tme servers ndstngushable runs n whch the set of reference tme servers s parttoned nto two sets and wth each contanng at most A reference tme servers. In the frst run contans all correct reference tme servers and contans all faulty ones, whle n the second run the stuaton s reversed. The runs can be chosen such that all reference clocks n each set show at any tme the same value, but there s a small drft between two reference clocks that belong to the dfferent sets and. For example, let the drft rate 9 of all hardware clocks n the system be the same, let the drft rate of the reference clocks n be 9 t F, and let the drft rate of the reference clocks n be 9ÊB F. Let 9 be defned by 9 ` BF n the frst run,.e. the reference tme servers n are correct and the ones n are faulty.

20 #%$ & ' (*)' +-,.* (3,8. In the second run let 9 =-F, that s, the reference tme servers n are faulty and the ones n are correct. The devaton between the reference clocks of and wll eventually be greater than any gven bound, thus each non-reference tme server : has eventually to decde to whch of the two sets t should synchronze ts clock. e show that ths decson cannot always be done correctly: f the decson s to synchronze towards and the algorthm actually experences the second run,.e. the decson s wrong. Smlarly, f the decson s towards and the algorthm experences the frst run, the decson s equally wrong. Thus, less than 2A +1 reference clocks are not enough for achevng external clock synchronzaton despte up to A faulty reference clocks. THEOREM 2 (LOER BOUND THEOREM) There s no correct external clock synchronzaton algorthm capable of maskng A arbtrary falures of reference tme servers f the total number of reference tme servers s not greater than 2A. Proof: To prove ths theorem, we ntroduce a formal model whch s a varant of our system and falure model. Ths formal model makes stronger assumptons than the ones used n the systemfalure model, e.g. all remote clock readng errors are assumed to be zero. External clock synchronzaton n ths model becomes therefore smpler to accomplsh. e prove that the lower bound theorem s vald n ths formal model: at least?a B < reference tme servers are needed to provde external clock synchronzaton n case up to A reference tme servers can suffer arbtrary falures. Snce the formal model uses stronger assumptons, any run that satsfes the requrements of the formal model also satsfes the requrements of the systemfalure model. In other terms, the set Ayà of vald runs of the formal model s a subset of the set à of vald runs of the systemfalure model. Therefore, any external clock synchronzaton algorthm that would acheve external clock synchronzaton for less than?a B < reference tme servers n any run of à has also to acheve external synchronzaton for any run of Ayà. Snce we show that ths s not even possble for the runs n Ayà, there exsts no algorthm that can acheve external clock synchronzaton n the assumed systemfalure model. Formal Model The formal model conssts of two components (see Fgure 9): the envronment whch determnes the behavor of the reference and hardware clocks, and the algorthm that determnes the values of the vrtual clocks. The behavor of the envronment s descrbed by a set of nfnte state sequences (e-runs), e.g. these e-runs determne for each pont n reference tme the values of the reference and hardware clocks. A clock synchronzaton algorthm s represented by a set of nfnte state sequences (c-runs): they descrbe for any vald e-run, all possble reactons of the algorthm. The ntuton behnd the lower bound proof s that we show how to select for

21 3G.H(H'I8+4, (43G.IJ3G.H(H' +H.*,KL,.*0J'NM(H' +H.*,KO1KPQ1 RS5TQ.1U*+P8.H36), (436P. #% each algorthm an e-run such that one of the reactons of the algorthm one c-run that matches the e-run does not bound the external devaton of some vrtual clock. A run s an nfnte sequence of states. A state s a set of bndngs that assgns each varable of the algorthm or envronment exactly one value. Any varable of a run à s ether an nput, an output, or a hdden varable: the value of an nput varable s externally determned, e.g. the nput varable of an algorthm s determned by the envronment and cannot be changed by the algorthm, the value of an output varable s nternally determned but can be read externally, e.g. the envronment defnes the values of reference clocks and an algorthm can read but not modfy theses values, and the value of a hdden varable s nternally determned and cannot be read externally, e.g. the envronment defnes real-tme as a hdden varable whch cannot be read by an algorthm. Runs descrbng the behavor of the envronment (e-runs) have the followng varables (see Fgure 9): 9 ± 9 : : for each reference server an output varable that contans the current value of s reference clock, for each tme server an output varable ÂÑ that contans the current value of s hardware clock, a hdden varable g* that represents the pont n real-tme when the current state has occurred, and for each tme server : a hdden varable 7 99 that s true ff : s correct n the current state. A run descrbng the behavor of a clock synchronzaton algorthm (c-run) has the followng varables: for each reference server 9 an nput varable ±z that contans the current value of 9 s reference clock, for each tme server : an nput varable  that contans the current value of the hardware clock of tme server :, and for each non-reference tme server ; an output varable Ð Û that contans the current value of ; s vrtual clock. The nput varables of a clock synchronzaton algorthm are the output varables of the envronment. The fact that the algorthm has drect access to the output varables of the envronment mples that (n the formal model) an algorthm can read reference and hardware clocks wthout any error.