Integration of high accurate Clock Synchronization into Ethernet-based Distributed Systems

Transcription

1 Integration of high accurate Clock Synchronization into Ethernet-based Distributed Systems Martin Horauer, Roland Höller Abstract Emerging networks destined to handle voice, data and Internet Protocol (IP) traffic benefit from a reliable, high accurate distributed time service. In most cases, the end-to-end communication path requires tuning and continuous maintenance to achieve acceptable quality for service integration. Several sources of latency, as they are protocol processing, system transfer rates and frame forwarding mechanisms within a local network communication path are beneath notice or remain undiscovered. Assessing these parameters as well as simultaneous triggering of events and synchronous data acquisition at several nodes is impossible without tight clock synchronization. Recently, micro-segmentation by using switched Ethernet technology has become popular to handle all traffic in enterprize networks. Providing distributed services like clock synchronization for these networks is mandatory. In this paper we discuss several engineering aspects associated with the development of a high-accuracy fault-tolerant external clock synchronization module for distributed systems. The presented solution aims at an unrivaled precision and accuracy in the range below 100ns, while using solely existing communications channels like Fast-Ethernet. It is applicable for packet-oriented data networks, inserts time information into data packets at the interface between the physical layer transceiver and the network controller upon packet transmission and reception respectively. Local time is supplied by a high-resolution rate-adjustable adder-based clock, which also contains hardware support easing interval-based external clock synchronization, like maintaining time and accuracy intervals and interfaces to GPS receivers. Keywords Clock synchronization, Distributed Systems, Global Positioning System (GPS), Universal Time-Coordinated (UTC), Application- Specific Integrated Circuit (ASIC), Adder-based clock, Location-based Services, Voice-over-IP (VoIP), Fast-Ethernet. I. INTRODUCTION A distributed system is a collection of autonomous computers linked together as computer network and supported by software that enables the collection to operate as an integrated facility. They allow the sharing of information and resources over a wide geographic spread and they are usually superior to traditional centralized systems in terms of sharing, cost, growth and autonomy. In contrary there are still some short-comings and weaknesses with existing implementations. Due to the distributed nature, these systems have to cope with unreliable and insecure communications and independent failures. These problems aggravate as soon as the system is operating critical real-time applications such as aerospace systems, life support systems, nuclear power plants, drive-by-wire systems or computer-integrated manufacturing systems. Common to all these applications is the demand for maximum reliability and high performance of the controlling hard and software, since a single failure in these applications can Martin Horauer is with the Technikum Vienna, Höchstädtplatz 3, A-1200 Vienna, Martin.Horauer@technikum-wien.at, and with the Department of Computer Technology, Vienna University of Technology, Gußhausstr , A-1040 Vienna, Roland Höller is with the Technikum Vienna, Höchstädtplatz 3, A-1200 Vienna and with the Department of Industrial Electronics and Material Science, Vienna University of Technology, Gußhausstr , A-1040 Vienna, Roland.Hoeller@tuwien.ac.at, lead to failure of the whole system. In addition to the aforementioned, an increasing number of distributed applications, such as process-control applications, transaction processing applications, or communication protocols rely on autonomous computers that need to cooperate for initiation of actions or recording of events. Therefore causal ordering is often required, a means that can be provided with the help of synchronized clocks so that every computer node has up to a certain precision the same view of time. Synchronizing an ensemble of distributed clocks comes in two flavors: Internal clock synchronization aims to keep the deviation between all clocks bounded, i.e., if C i (t) and C j (t) denote two fault-free clocks within a system, the worst case precision π satisfies C i (t) C j (t) π t t 0. External clock synchronization relates to the problem that clocks are required to follow an external reference like Universal Time Coordinated (UTC), the only legal standard of time. The maximum deviation towards this reference time is called accuracy α, formally C i (t) t α t t 0. Reaching both goals jointly turns out to be a non-trivial problem, since a certain tradeoff seems to be involved, as in [1]. When synchronized clocks are at hand, the performance of a distributed system can be improved by reducing communication traffic, see [10] for some practical uses. In general, for most systems a synchronization tightness in the range of several ms is sufficient, a requirement that can be fulfilled with pure software based clock synchronization mechanisms, see [12]. Other services require their clocks to be synchronized in the range down to several ns. For example the location-based services in near future communications systems depend on tight clock synchronization. In 1997 the U.S. Federal Communications Commission introduced a mandate to enforce all mobile telephone networks to provide mobile locations for emergency services. This mandate requires that cellular, personal communications services and specialized mobile radio service providers deploy a means of automatically locating emergency callers within 125m in 67% of all measurements by October 31, The most promising method, Time Difference of Arrival (TDOA), measures the relative arrival time of the signal from the mobile at three base stations. Precise synchronization of the base stations will be required, since any small synchronization error directly relates to the accuracy of the caller location. Another application that benefits from high accurate synchronization is fault detection and location in power systems. Here tight clock synchronization can be used to provide means for online fault detection and an estimation of the fault location (see [8]). Reliable provision of electrical power is of utmost importance for our daily human life. This, in turn, requires redundant, reliable power system components that can be easily maintained and exchanged, see [11]. Disregard of these criterions can

2 have catastrophic consequences as they were experienced after recent power outages in the US, where a power outage during the months July till August 1996 affected millions of end customers in the south west of the United States and in northern Mexiko. As well as in New-Zealand, where in February 1998 large districts of Auckland experienced a major power outage for several weeks after four central power cables broke down. Further applications and services can benefit from tighter clock synchronization as well (e.g. Quality of Service for Voice over IP systems). Others in computer science, like multimedia or distributed computing, will eventually emerge when this new technology becomes available. With the knowledge and experience gained along with the research project SynUTC i, we are legitimately convinced, that it is possible to provide clock synchronization with a precision and accuracy in the range of several ns to demanding applications solely via the exchange of messages over existing packet oriented networks. Our patent pending architecture yields a tremendous improvement in the order of several magnitudes over existing realizations from ms to ns and will eventually turn out as a new enabling technology. The remainder of this paper is structured as follows: Section 2 gives an overview on the diversity of existing approaches and summarizes all requirements that need to be considered when addressing high accurate clock synchronization. In Section 3 we describe our new architecture and illustrate some implementation related issues. Next we briefly present the required software and render a mechanism to evaluate our implementation. Finally, we point out some future developments that can be based on the presented research. II. CLOCK SYNCHRONIZATION IN DISTRIBUTED SYSTEMS Clock synchronization is presently available for different application domains with varying levels of precision and accuracy. Purely software-based solutions run on commercial off-the-shelf networking hardware, providing precision down to 10ms if implementation related issues are optimized. For the widely used Network Time Protocol a worst-case precision of 20ms was observed under some realistic conditions, see [20]. By adding a small amount of hardware support, the precision can be enhanced to some µs. These hybrid approaches rely on a dedicated clock chip that provides facilities for clock correction and allows to reduce the inherent error made in reading remote clocks. Existing realizations of the MARS, the TTP and the SynUTC project show that a precision and accuracy in the µs range is feasible, see [6], [7] and [18] for details. With the advent of GPS technology, external clock synchronization with an accuracy below 100ns and below became worldwide available. GPS usually exhibits very reliable operation, but has some limitations as well. E.g. every node s GPS receiver needs a roof-top antenna and they are exposed to rather large delays until correct time is provided after power-up or when the i The SynUTC-project received support from the Austrian Science Foundation (FWF) grant P10244-ÖMA, the OeNB Jubiläumsfonds-Projekt 6454, the BMfMV research contract Zl /2-iV/B/9/96, the Gesellschaft für Mikroelektronik (GMe), and the START programme Y41-MAT. It is a cooperative effort of Oregano Systems and the departments of Automation, Computer Technology and Industrial Electronics and Material Science. See for further information. contact to the orbital GPS system becomes degraded. A longterm evaluation of several off-the-shelf GPS timing receivers [4] revealed spurious 1pps omissions and errors due to certain receiver technologies. Furthermore the dependability upon a single system seems questionable for critical real-time implementations. Hence GPS is great when complemented with other technologies, see [1] or [21]. The latter sprays external time obtained from GPS satellites into broadcast-type LANs with a software-based approach with accuracies in the 10µs range. Even smaller precisions can be achieved if a separate fully connected clocking network is used [13]. In this article we do not further consider such pure hardware based solutions because of their limited use and practicality for large scale distributed systems, both in terms of the number of nodes and the distance to be bridged. From here onwards we consider systems that are connected by some kind of message passing system solely. A clock synchronization algorithm is executed at every node and periodically initiates the exchange of clock synchronization messages. With the help of a transmit and a receive timestamp, the receiving node can estimate the value of the sender s clock. Remote clock estimates from all other nodes in the distributed system are fed into the clock synchronization algorithm, which computes a correction value for the local clock with a convergence function, see [19] or [1] for some examples. In addition, some nodes in the distributed system require access to an external time-source, e.g. with a suitable GPS receiver, to allow for external clock synchronization in a similar way. The correction is succinctly applied to the local clock to correct its state and rate respectively. Enforcing these adjustments deserves special attention, since several applications dictate specific properties. In particular, timestamps obtained from local clocks need to be monotonic and free of discontinuities of predefined extent. A technique called linear continuous amortization, as hosted in the UTCSU ASIC, see [16], can be used to carry out state adjustments. Several results given in [1] and [15] as well as our previous gained experimental results [17] led to the identification of several items that need to be addressed when trying to optimize the precision and accuracy for both external and internal clock synchronization in distributed applications: 1. The remote clock reading error has to be minimized. 2. The local oscillator that paces the hardware clock shall only exhibit a very small oscillator drift and provide good stability. 3. The local clock should be fine grained and should allow for both state and rate adjustments. 4. A tight coupling of external time sources is mandatory for external clock synchronization. The remote clock reading error, the dominating factor in packet oriented networks, can be reduced as soon as appropriate timestamp facilities are placed next to the physical layer of a network connection, see [3]. Taking the impurities of the local oscillator into consideration, one could either employ a very stable ovenized oscillator (e.g. an OCXO) or implement an according clock rate synchronization mechanism. Due to board space, power and cost requirements the latter solution is usually preferable although it comes at the expense of a small compu-

3 tational overhead. A fine granular clock design, that provides state and rate adjustments, respectively, would either require a clock running at a very high frequency or some other sophisticated mechanism, e.g. an adder-based clock. Finally, coupling of external time sources usually requires some hardware support as well. III. HIGHLY ACCURATE CLOCK SYNCHRONIZATION This section presents an un-rivaled hardware support for tight clock synchronization over packet networks in the range below 100ns. This new architecture exploits several short-comings found in other implementations and focuses towards an industrial realization. In particular we identify three items that need to be addressed: 1. Development of an ASIC supporting clock synchronization, containing: An adder-based local clock with fine granularity and a mechanism for linear continuous amortization. A mechanism to maintain a bound on an external time reference. An interface to an external time source, e.g. GPS. Timestamping capabilities. A standardized interface that is located next to the physical layer of a particular network technology. A programming interface via clock synchronization packets solely. 2. Development of a network interface hosting: An off-the-shelf media access controller. The previously mentioned ASIC supporting clock synchronization. A separate physical layer device. 3. Add-on support for store and forward devices in an end-toend communications path (e.g. switches). Ethernet technology will be chosen as the communication platform for our prototypes, since it has recently gained widespread use in enterprize wide networks and will eventually emerge in other areas as well (e.g. Gigabit Ethernet Technology is increasingly used in the backbone area as well and Fast Ethernet gains widespread use in the field of automation and control systems ii ). A. The Clock ASIC The local clock forms the centrepiece for a hardware support in a distributed system. We propose a digital clock consisting of an oscillator driving an adder instead of a simple counter. Employing an adder gives the freedom to add a particular amount (clock step) to the clock register at every pulse. A rate change can be achieved by varying this amount, which goes in effect almost instantly and holds up linearity. A straightforward binary coding scheme, closely following the NTP-time format, can be employed to maintain local time. To satisfy requirements for a clock synchronization tightness in the range of several ns, the clock should be arbitrarily state adjustable for initialization purposes and rate adjustable not coarser than s/s to avoid ii The High-Speed Ethernet (HSE) Program of the Fieldbus Foundation establishes Fast Ethernet in the fieldbus area and the Industrial Automation Open Network Alliance (IAONA) tries to establish Ethernet technology in the field of automation as well. influences due to clock granularity. Dealing with the accuracy requirement means that local time has to follow an external time source as close as possible. Unfortunately, UTC is neither directly observable nor permanently accessible, so only a range can be determined where UTC currently lies. More specifically, in our setting we propose an accuracy interval A(t) capturing the reference time in the sense that t A(t) t t 0. Supporting this interval-based approach requires to maintain an upper accuracy α + (t) and a lower accuracy α (t) meant relative to the local clock C(t), such that A(t) = [C(t) α (t), C(t) + α + (t)]. Accuracies are kept time-dependent to enlarge them properly in order to account for maximum oscillator drifts, hence sustaining the inclusion of the external reference. This process of linear deterioration would go on perpetually, however, periodic re-synchronization executed by a clock synchronization algorithm aims to shrink A(t), by exploiting knowledge of the external time reference provided by suitable receivers, e.g. GPS-receivers. An in-depth discussion of several details about this particular mechanism can be found in [19]. Both time and accuracy information should be maintained by an according clock synchronization algorithm, executing on a host CPU, and performing corrections via continuous amortization, see Fig. 1. Instead of adjusting the clock state instantaneously, we achieve the same effect by modifying the clock rate for a specific amount of time, that is controlled by the clock synchronization algorithm. The latter also maintains the accuracy intervals, that in contrast are allowed to change instantaneously, since they are kept relative to the local clock. T=C(t) Clock Time t α + (t) Clock Time α - (t) Accuracy Intervals Fig. 1. Continuous Amortization Reference Time Real Time Interfacing to an external time source and provision of timestamping capabilities can all be facilitated by sampling the local time and accuracy into dedicated registers. In particular this should be done either with every rising edge of the one-pulse-

4 per-second signal provided by GPS timing receivers, or when a clock synchronization packet is decoded. The timestamps are then subsequently mapped into the clock synchronization packets that are periodically sent and received respectively. This mechanism should be implemented by dedicated logic within the clock ASIC, which recognizes clock synchronization packets by their unique Ethernet type field as they pass the MII interface. To allow reliable operation with off-the-shelf Ethernet technology, the standard Media Independent (MII), that interconnects the Medium Access Layer of any Ethernet, Fast- Ethernet or Gigabit-Ethernet network controller with an according device, should be employed. Similar to the timestamping mechanism, where timestamps of the local clock are mapped into the MII data stream, programming information to/from the clock ASIC can be transported within the remaining payload of the clock synchronization packets. The great benefit of this mechanism is that no direct register access via an additional external interface is required to program the clock ASIC. Reading/setting of the clock registers can be done at a higher protocol level, thus facilitating re-use of clock synchronization mechanisms under varying operating systems without a need to redo the whole driver implementation. Figure 2 shows a block-diagram of the clock ASIC. The local clock and the accuracy intervals are maintained within the Local Time Unit (LTU) and the Accuracy Unit (ACU) respectively. A GPS module (GPU) provides interfaces for coupling an external time reference to the distributed system, the Application Unit (APU) hosts features to support timestamping of application events and generation of time-out signals and finally the Timestamp Unit (TSU) provides all required timestamping mechanisms as well as a programming interface via the MII bus. Next to maintaining local clock and accuracy information, timestamping is of utmost importance. In the MARS and the SynUTC projects timestamps are inserted transparently into the outgoing clock synchronization, when the network controller reads the assembled packet from the local memory for transmission, see [6] and [18] respectively. Measurements revealed that the speed of the local bus and network controller inherent FIFO s are restricting tighter clock synchronization. Therefore we propose to place the timestamping mechanism into the clock ASIC in situ between the medium access and the physical layer. B. Network Card The Peripheral Component Interconnect (PCI) bus has gained widespread use in recent years with the success of personal computers. A common implementation, a 32-bit 33 MHz PCI bus under a contrived (single long burst) condition is theoretically capable of 1.06 Gb/s data transfer. Thus, when PCI is used, the peripheral bus is not likely to be a performance limiting factor for end stations that have 10 Mb/s or 100 Mb/s interfaces. The similarity with Compact-PCI, an industrial variant, will ease migration to an industrial embedded platform when required. Hence PCI is a promising candidate for the prototype implementation of a suitable network interface card as illustrated in Figure 3. Network Controller Network Card PCI Bus Fig. 3. Block-diagram of the Network Card LTU ACU SSU GPU GPS The clock ASIC is placed into the Media-Independent- that is used to connect Ethernet controller devices with a suitable physical layer device. Adopting this standardized interface allows to make re-use of almost every Ethernet controller, hence making this approach suitable for a wide range of existing standard technologies. The clock ASIC is transparent to network traffic except for dedicated clock synchronization packets that are identified by a unique type field. When a clock synchronization packet is detected timestamps are inserted into the packet. In addition some engineering issues concerning the Ethernet checksum and the allowable latency of the signals need to be addressed, see [3]. MII APU TSU Fig. 2. Block-diagram of the Clock ASIC Application MII C. Hardware Add-On for Switches Heretofore, we assumed a shared media network without additional interconnection devices. Recently, micro-segmentation by using Ethernet switches has become more popular to handle all traffic in enterprize networks. The principle function of the switch is to bridge frames between ports by achieving the maximum throughput. The information needed for switching is the Medium Access (MAC) destination address, which is located at

5 the beginning of the ethernet frame. Basically, switches can be classified by two major categories of handling a frame, named: Cut Trough mode, where the switch tries to start the retransmission as soon as it has received the necessary elements. Store and Forward mode, which requires reception of the entire frame before deciding to retransmit it. Store and forward switches are the preferred ones, since they allow for packet filtering in particular they filter runt packets and improve by this way the network load and provide additional value added services. Considering transmission delay variations, the impact of cut through type switches may be negligible but the variations caused by store and forward switches deteriorate the remote clock reading error, hence prohibiting tight clock synchronization. To alleviate the deterioration caused by switches a mechanism that measures the variable duration a packet remains on a switch on-the-fly is required. To circumvent these restrictions a hardware add-on for switching technology iii as depicted in Figure 4 can be employed. This module when added in front of a switch, measures the duration it takes for a clock synchronization packet from receipt at one particular switch port until retransmission via another switch port. Up/Down-Links Fig. 4. Clock Synchronization Hardware Add-On for Switches The clock ASIC on the receiving port inserts an additional timestamp t sw r into the payload of the clock synchronization packet at a predefined offset from the packet header. Next the clock ASIC situated at the port that is used to output the packet draws a second timestamp t sw t and computes the difference t sw t t sw r and succinctly overwrites the timestamp t sw r by this difference. One additional timestamp field shall be used to accumulate all intermediate delays when a chain of switches is used. Thus the clock ASIC on the outgoing port needs to update this field as well. iii This mechanism can be equally well included into the switch hardware, although this requires the design of a new switch. Up/Down-Links To accomplish all these mechanisms several clock ASIC s should be operated in lock step from one clocking device. Finally additional mechanisms are required to periodically check the synchrony of the clocks, otherwise one clock ASIC out of sync would go undetected. Several engineering issues need to be addressed to re-synchronize and re-integrate a faulty clock ASIC. IV. CONCLUSIONS AND FUTURE WORK The presented architecture facilitates external clock synchronization even in Ethernet-based distributed systems. The new Media Independent -based timestamping method will allow to improve the achievable precision and accuracy of shared-media based clock synchronization mechanisms by the order of several magnitudes by drastically reducing the remote clock reading error. In order to reduce this parameter dedicated hardware support at the network interface card as well as at every network element, that is in between an end-to-end communications path, is required. The presented solution requires only small/no modification of existing hard and software, in fact the required mechanisms are merely added to existing implementations. We are presently planning a thorough evaluation with our prototype implementation, where we need to assess the variability of all impaired parameters, most notably the clock reading error and the clock drift, for a suitable setup. Next we are planning a comprehensive evaluation of the clock precision and accuracy. This can be accomplished by programming every clock to issue a trigger signal at predetermined clock states. Relating these signals to each other and to external reference clocks allows to assess the internal clock precision π and the external clock accuracy α. First measurements conducted with our prototype implementation let us suggest that a worst-case precision and accuracy of about 50ns in Fast-Ethernet networks seems feasible. A successful implementation of this proposed architecture will lead to new applications and give direction for further research topics being implemented on top of this distributed clock synchronization mechanism. The following research areas are some examples that could be further exploited: Multimedia: Tight clock synchronization facilitates better assessment of Quality of Service measures in an end-to-end communications path. This in turn helps to shape the datagram streams of Voice-over-IP or Video-on-Demand multimedia applications to the varying end-to-end communications situation. Network Protocols: Synchronized clocks can be used to improve the performance of network protocols and distributed algorithms. They make it possible to replace communication with local computation. Instead of node p asking another node q whether some property holds, it can deduce the answer based on some information about q from the past together with the current time on p s clock, see [10]. Mobile location-based services [14], or on-line fault detection and location services in power systems [8] are further examples that rely on tight clock synchronization. Furthermore, many others will emerge when our proposed enabling technology becomes available.

6 ACKNOWLEDGMENTS The authors would like to acknowledge the suggestions and vital support from Ulrich Schmid, Nikolaus Kerö, Gerhard Cadek and the rest of our SynUTC project team. REFERENCES [1] C. Fetzer, F. Cristian, Integrating External and Internal Clock Synchronization, Journal of Real-Time Systems, May 1997, No. 3, Vol. 12 (2), pp [2] M. Horauer, Hardware Support for Clock Synchronization in Distributed Systems, Supplement of the 2001 International Conference on Dependable Systems and Networks, pp. A.10-A.13, Göteborg, July [3] M. Horauer, N. Kerö, U. Schmid, A network interface for highly accurate clock synchronization, Proceedings Austrochip 2000, Graz - Austria, October 2000, pp [4] D. Höchtl, U. Schmid, Long-Term Evaluation of GPS Timing Receiver Failures, Proceedings of the 29th IEEE Precise Time and Time Interval Systems and Application Meeting (PTTI 97), Dec. 1997, pp [5] M. Horauer, U. Schmid, K. Schossmaier, NTI: A Network Time M-Module for High-Accuracy Clock Synchronization, Proceedings of the 6th International Workshop on Parallel and Distributed Real-Time Systems (WPDRTS), Orlando Florida USA, April [6] H. Kopetz, A. Damm, Ch. Koza, M. Mulazzani, W. Schwabl, Ch. Senft, R. Zainlinger, Distributed Fault-Tolerant Real-Time Systems: The MARS Approach, IEEE Micro, Feb. 1989, pp [7] H. Kopetz, A. Krüger, D. Millinger, A. Schedl, A Synchronization Strategy for a Time-Triggered Multicluster Real-Time System, Proceedings Reliable Distributed Systems (RDS 95), Sep [8] N. Kerö, U. Schmid, M. Horauer, Verfahren für die Synchronisation von Computeruhren in Netzwerken, Department of Automation, TU Vienna 183/1-105, March 2000, Austrian Gebrauchsmuster GM 153/2000. (in German) [9] H. Kopetz, W. Ochsenreiter, Clock Synchronization in Distributed Real- Time Systems, IEEE Transactions on Computers, 1987, pp [10] B. Liskov, Practical uses of synchronized clocks in distributed systems, Distributed Computing, 1993, pp. 211 ff. [11] C. Liu, J. Jung, G.T. Heydt, V. Vittal, A.G. Phadke, The Strategic Power Infrastructure Defense (SPID) System, IEEE Control Systems Magazine, Aug. 2000, pp [12] D.L. Mills, Internet time synchronization: The network time protocol, IEEE Transactions on Communications, Oct. 1991, vol. 39, No. 10, pp [13] P. Ramanathan, K.G. Shin, R.W. Butler, Fault-Tolerant Clock Synchronization in Distributed Systems, IEEE Computer, Oct. 1990, pp. 33 ff. [14] J.H. Reed, J. Krizman, B.U. Woerner, T.S. Rappaport, An Overview of the Challenges and Progress in Meeting the E911 Requirement for Location Service, IEEE Communications Magazine, pages 34 37, April [15] U. Schmid, Orthogonal Accuracy Clock Synchronization, Chicago Journal of Theoretical Computer Science 2000(3), 2000, pp IEEE Computer, 1990, Vol. 23 (10), pp [16] K. Schossmaier, U. Schmid, M. Horauer, D. Loy, Specification and Implementation of the Universal Time Coordinated Synchronization Unit (UTCSU), Journal of Real-Time Systems, May 1997, No. 3, Vol. 12 (1), pp [17] U. Schmid, M. Horauer, N. Kerö, How to Distribute GPS-Time over COTS-based LANs, 31st Annual Precise Time and Time Interval (PTTI) Systems and Applications Meeting, Dana Point - California, December [18] U. Schmid, J. Klasek, Th. Mandl, H. Nachtnebel, G.R. Cadek, N. Kerö, A Network Time M-Module for Distributing GPS-time over LANs, Journal of Real-Time Systems, Jan. 2000, No. 1, Vol. 18, pp [19] U. Schmid, K. Schossmaier, Interval-based clock synchronization Journal of Real-Time Systems, May 1997, No. 3, Vol. 12 (2), pp [20] G.D. Troxel, Time Surveying: Clock Synchronization over Packet Networks, PhD thesis, Massachusetts Insitute of Technology, Departement of Electrical Engineering and Computer Sciene, [21] P. Veríssimo, L. Rodrigues, A. Casimiro, Cesiumspray: a precise and accurate global clock service for large-scale systems, Journal of Real-Time Systems, May 1997, No. 3, Vol. 12 (1), pp