Delivering Sub-Microsecond Accurate Time to Linux Applications Around the World

Transcription

1 Delivering Sub-Microsecond Accurate Time to Linux Applications Around the World Time Where It s Needed 1 Time Offsets, Delays And Delay Variations No Way Around Them!1 The Four Sources Of Sub-Microsecond Time Error2 Network Path Delay Related Errors2 Linux Stack Delay Related Errors3 Linux Clock Call Delay Errors3 Clock Instability Errors4 Solving The Sources Of Time Synchronization Error5 PTP Hardware/Software Combinations 5 Why Hardware Based PTP Slaves Deliver The Best Timing5 Solving Network Path Delay Related Errors5 More Packets For Better Filtering5 OCXO Oscillator Provides A Stable Time Base5 Hardware Time Stamping Improves Time Accuracy6 Validating Synchronization Over The Network6 Push Time To Memory To Improve Speed And Accuracy6 Verifying High Availability DMA Time Writes7 Verifying DMA Time Stamp Accuracy7 Best Possible Time To Utc At The Application Layer8 Best Possible Time Around The World9 Conclusion, PTP Clock Cards Deliver The Best Possible Time Accuracy9 Accurate Time Where It s Needed While the science of timekeeping has fascinated many over the years, what counts most today is delivering accurate time to where it s needed If a person needs the time they may look at their phone to see an accurate time that is distributed through the mobile network If an application program running on a computer needs the time it may execute a command that accesses the system clock That application is a consumer of the time and not a clock Ultimately, it is the accuracy and precision of the time delivered to the application program that matters most The more accurate and precise the time requirement is, the harder it is to deliver it This whitepaper explores the challenges in delivering sub-microsecond accurate time to a Linux program, and it will present IEEE 1588 Precise Time Protocol (PTP) solutions and test data showing how to overcome those challenges Time Offsets, Delays and Delay Variations No Way Around Them! Our objective is to deliver accurate and precise time over a packet based network to a computer, then on to the application programs that need it Timing packets on a network can be delayed, thereby impairing the ability to accurately transfer time from one clock to another There are factors that cause the delays change, so there will also be variations in the amount of delay, making adjusting for the delay more difficult In fact, these delays are the primary source of time transfer error In time synchronization we rely on accurately measuring timing packet transit delays to adjust the clocks to compute the correct time As the need for more accurate time at a program increases, the impact that variable packet delays have on the ability to correct a clock increases as well To make matters worse, the quality of a clock also effects how well you can synchronize the time Understanding the problems and how to compensate for them greatly helps in delivering accurate and precise time to clocks and ultimately to an application program The difference between the times on two clocks is known as the offset In timekeeping we strive to keep the offset below a particular value so that we can assign an accuracy value to the time on the clock Generally in synchronization schemes one clock is more accurate than the other so the offset is tracked relative to the more accurate clock The process of setting one clock to another is a matter of exchanging timing packets, computing the offset between a master clock and a slave clock, and making an adjustment to the slave clock (called clock steering when done periodically) Page 1 of 9

2 The time the packets take to travel from the master clock to the slave is called delay, or path delay Figure 1 shows the PTP packet exchanges and Equation 1 is the fundamental offset equation the slave will use to compute the time offset and correct the time Slave Time Offset= (t 2 -t 1 )+(t 3 -t 4 ) 2 Equation 1: Determination of clock error at the slave derived from PTP message departure and arrival timestamps t 1 Master Time Timestamps known by slave Sync Slave Time t 2 t 1, t 2 The Four Sources of Sub-Microsecond Time Error There are four main sources of error in delivering time to an application program over a network from a master clock Asymmetric time transfer delays between the master clock and the program on the slave computer requesting the time accounts for three of them The fourth is clock instability at the slave computer (assuming the master is the more stable time source) With sub-microsecond accurate time required at an application program on a slave computer, oscillator drift and even a slight temperature change caused by the slave computer fan can throw the time off by several microseconds Network Path Delay Related Errors In packet based networks, timing packets are exchanged between the master clock and the slave clock for the purpose of computing the time offset at the slave If the packet exchanges were instantaneous there would be no delay and the offset could be computed perfectly If the packet exchange delay on the master-to-slave path and slave-to-master paths were identical; the offset could be easily and precisely computed since the delays would cancel each other mathematically The case where path delay is the same both ways between master and slave is called symmetric delay, and time transfer over packet networks most often assumes symmetric delay t 4 Delay_Req Delay_Resp t 3 t 1, t 2, t 3 t 1, t 2, t 3, t 4 Figure 1: Sequence diagram showing the timing event packet exchange between a PTP master clock and a PTP slave clock Master Sync Messages Sent on Time Network Delay Time Offset Between Clocks Sync Messages Arrive with Varible Delays Slave Figure 2: Timing messages sent from the master to the slave (and vice versa) can experience variable delays caused by switches in the network and application delays leading to timing errors at the slave clock Page 2 of 9

3 Unfortunately, path delay is different between master-to-slave and slaveto-master, and this difference is called asymmetric delay Furthermore, not only is the delay different along the two paths, but the delay difference can also vary with each packet sent As illustrated in Figure 2, Asymmetric path delays are the main problem in accurate and precise time synchronization using packet based techniques over a network Data Traffic Timing Traffic Switch Figure 3: Packet queuing inside the switch where data and timing packets must be forwarded to the same port can introduce Packet Delay Variation (PDV) leading to timing errors In a simplified network, asymmetric delay adds a time offset error equal to one half the difference in delay times For example, if it took one second for the timing packet to go from the master to the slave and three seconds on the reverse path, the offset computed by the slave would be incorrect by one second Apply this to computer hardware that is time stamping packets being exchanged every second between a master and a slave with the master having 50 nanosecond timestamp accuracy (That is, a PTP grandmaster with accuracy to UTC or Coordinated Universal Time such as a master clock with a GPS reference) These timing packets transit a LAN containing switches (or worse, routers) that may add asymmetric path delay in the 10s of microseconds, which result in similar size time offset computation errors Queuing and delays inside network switches are generally the culprit in packet delays Switches are often store-andforward devices Packets received on one port are stored temporarily while the device figures out which forwarding port(s) to send them to We are now at the heart of one of the four sources of time transfer error Asymmetric delay introduced by switches and routers in the path between the master and the slave is caused by packet queuing Received packets are placed in a queue (ie buffer) for the forwarding port until they can be sent out This happens extremely fast unless the forwarding port is busy receiving traffic from multiple receive ports, in which case there will be a delay in forwarding the packets The time the packet is delayed is called the residence time See Figure 3 Due to the variable queuing delays inside the switch the timing packets exchanged between the master and slave are delayed and the length of the delay is variable This variable delay is called Packet Delay Variation (PDV) PDV refers to the arrival time jitter for timing packets that are sent from the master or slave to the other but have variable arrival times at the destination In the world of microsecond and sub-microsecond time synchronization over packet networks, PDV makes it very difficult to accurately compute precise offsets and adjust the slave clock Network time protocols, such as NTP and PTP can compensate for packet delay, but only if the delay is the same in each direction This is why the error is referred to as delay asymmetry error Average Network Delay= (t 2 -t 1 )+(t 4 -t 3 ) 2 Equation 2: The average of the forward and reverse path delays is calculated from the time stamps of the PTP sync and delay request message departure and arrival times Linux Stack Delay Related Errors As timing packets exit the network they enter the Linux stack of either the master or slave computer While a computer may be very fast at delivering timing packets quickly to the clock steering program, any delivery time variation in microseconds directly adds microseconds of time synchronization error This is because it likely introduces an asymmetric path delay Note that delays delivering a received packet to the timing program are just as critical as delays in sending timing packets out of the computer Differences in these stack transit times directly affect how accurately and precisely the time can be corrected, particularly in computers targeting sub-microsecond synchronization In other words, the Linux stack is another place that delay asymmetry can occur Linux Clock Call Delay Errors Recall that the fundamental objective is delivering accurate and precise time to a computer program that requires it To get that time a Linux program may execute the clock_gettime command to obtain the time stamp from the computer clock Unfortunately, the amount of time to get that time stamp can be variable and the amount of delay is unknown Sometimes the Linux kernel can respond almost immediately with the system time, and sometimes the kernel is executing a task which cannot be interrupted Consequently the time stamp may be relatively old for the purposes of trying to deliver submicrosecond accurate time stamps to the application Page 3 of 9

4 Consider this simple test In this case the duration of a clock_gettime command was measured using the time stamp counter (TSC) The TSC was read, the clock_gettime command executed, and the TSC read again The TSC values were differenced, the result saved and the operation repeated The results are presented in histogram form below in Chart 1 Generally it took 3991 TSC counter ticks to get the time on this computer It is apparent that quite a few of the time reads were delayed and the delays are variable, the worst was over 700,000 TSC cycles The problem is not the delay itself, but rather in not knowing what the delay was If the delay were uniform it could be accounted for, but the clock_ gettime command has unknown variable delays that are not possible to ascertain when sub-microsecond time stamping is required Consequently you never know the age of the time stamp Time-Stamp Counter (TSC) representative of a certain number of CPU counter cycles Max TSC Duration Min 2736 TSC Duration Average 3991 TSC Duration Std Dev 3906 TSC Duration Chart 1: Histogram of the length of time it takes for the Linux kernel to respond to a clock_gettime function call The time is measured in counts of the processor s Time Stamp Counter Clock Instability Errors The fourth source of time error is the one that is not delay related, it s clock frequency related Simply put, a clock is an oscillator and a counter In an old fashioned mechanical pendulum clock the pendulum is the oscillator and the clock face/dial is the counter A simple clock mechanism counts 60 swings of the pendulum and advances the minute hand 1 minute (or 6 degrees of rotation) In a common computer clock the oscillator is made of a quartz crystal and vibrates at a known approximate frequency The vibration cycles are measured and translated into ticks per second from which the clock operates frequency of the vibration changes, which further changes the speed of the clock This means the time will speed up or slow down depending on temperature Consider the data below in Chart 2 In this case a software program is working to correct the time on a server However, periodically (every minutes) the fan in the server turns on and changes the internal temperature including that of the quartz oscillator inside the server used to keep the time With a temperature change comes an oscillator frequency change and a subsequent time change The data shows how the clock steering software is making adjustments to compensate for the periodically changing time Seen is more than 2 microseconds of error on a server not under any sort of load Tests showed that on a server under load the time variations were worse due to the increased temperature variations While quartz is a very inexpensive and convenient oscillator element for computers, the stability of the frequency of vibration of quartz is subject to a number of factors, not the least of which is frequency drift and temperature change Drift is easy to understand The oscillator frequency is not perfect, can change over its lifetime, and can add microseconds of error per day to the time Thus a computer clock typically runs fast or slow and needs continuous adjusting A bigger problem is that if the temperature changes slightly, the Chart 2: Time error due to oscillator frequency changes caused a fan turning on and off Page 4 of 9

5 Solving the Sources of Time Synchronization Error Solving all of the sources of time synchronization error described above requires a combination of items working together And though the four problems are most easily understood by looking at each individually, the solution is best understood by looking at a group of individual components that make up the whole Fundamental to accurate time synchronization between servers is an end-to-end timing solution that is accurate and reliable It starts with an appropriate master clock Ideally that clock is referenced to an accurate and stable time source such as GPS, supports PTP hardware time stamping and elevated PTP packet exchange rates Next is an appropriate slave solution The most effective and accurate PTP slave solution will have these three elements: hardware time stamping, tuned PTP packet prefiltering and clock steering algorithms, and an improved oscillator These are essential elements needed to achieve the best possible time transfer accuracy under typical network conditions PTP Hardware/Software Combinations The authors of the IEEE 1588 Precise Time Protocol understood the sources of timing error and architected the protocol so that software and hardware can work together to improve the accurate transfer of time PTP addresses several fundamental problems related to packet delay variation and clock drift What PTP lacks however is any uniform timing packet pre-filtering or clock steering algorithms to steer the local clock/oscillator The result is that time synchronization performance can vary widely from PTP vendor to PTP vendor as timing algorithms vary in effectiveness in different network environments PTP testing standards exist in some industries for specific applications and not others, so it is important to understand and even measure time transfer performance in your intended network configuration and application Why Hardware Based PTP Slaves Deliver the Best Timing Generally speaking, accurate time is desired by programs operating at the application layer of a host Linux server A clock card operating as PTP slave connects to the PCIe bus of the typical server and can act as an excellent clock to deliver the needed time stamps to the applications These cards support hardware time stamping, should have advanced PTP packet pre-filtering and clock steering algorithms, should have an improved oscillator, and should have an effective way to deliver precise time over the PCIe bus to the Linux applications that need it The PTP clock card used in the following solution is the Symmetricom SyncPoint PCIe-1000 shown in Figure 4 below Solving Network Path Delay Related Errors The combination of good packet filtering and clock steering algorithms used in conjunction with a very stable oven controlled crystal oscillator (OCXO) and hardware based PTP packet time stamping provides excellent results More Packets for Better Filtering Increasing the PTP packet exchange rate and employing effective packet filtering helps to identify and eliminate delayed PTP packets that can skew clock steering algorithms The higher the PTP packet rate the more data that is available to the filtering algorithms to identify on-time packets and reject delayed packets This provides the clock steering algorithm the best information from which to steer the oscillator OCXO Oscillator Provides a Stable Time Base Using a better oscillator such as an Oven Controlled Crystal Oscillator, or OCXO, provides two key advantages OCXOs are very stable and largely temperature insensitive They house the quartz crystal in an elevated temperature oven that is well insulated, literally in an oven If the outside temperature fluctuates the internal temperature can be adjusted to compensate The result is a very thermally stable oscillator well suited First, PTP accommodates a rapid exchange of timing packets over a standardized network between the master and the slaves This can provide more timing information over a shorter period time to enable vendor written time stamp filtering and clock steering algorithms to ignore delayed packets and compute mathematically better adjustments to the local clock/oscillator Second, PTP was designed to be hardware time stamping friendly as well as accommodate nanosecond time stamp resolution Figure 4: High Accuracy SyncPoint PCIe-1000 PTP Clock Card Page 5 of 9

6 to network operations The other advantage is that because the OCXO is so the frequency adjusts at a slower rate Once the oscillator frequency has been corrected by the steering algorithm, frequency adjustments are responded to slowly over time and any short term errant perturbations from the clock steering algorithm or temperature changes have minimal effect on the time Think of these advantages as a very large ship (the OCXO) moving in the right direction (frequency) Gusts of wind (PDV) and small waves (temperature changes) will have very little impact on the speed and direction of the ship SyncServer S350 IEEE 1588 PTP Network Switches and Traffic Impair Time Transfer Accuracy PCIe PPS Output to Grandmaster to Check Card Accuracy High Accuracy SyncPoint PCIe-1000 PTP Clock Card Figure 5: 1PPS loopback example to measure SyncPoint PCIe-1000 accuracy to Grandmaster Hardware Time Stamping Improves Time Accuracy PTP hardware time stamping in a server completely eliminates the operating system stack delays in both directions (packet ingress/egress) On ingress, PTP packets are time stamped while still in networking hardware, before they are sent to the operating system This time stamping happens as the packets exit the network and before they transit the operating systems stack Similarly when a PTP packet is sent, it is time stamped as the last operation before it leaves the host computer and enters the network As a result, any variable delay the packet encounters, large or small, in transiting the operating system is eliminated Validating Synchronization Over The Network Validating the SyncPoint PCIe-1000 card time accuracy, based on time transferred over the network, requires comparing the time on the SyncPoint card to that of the master it is synchronizing to Obtaining this measurement is very straightforward The SyncPoint card outputs a once per second (1PPS) pulse that corresponds to the full integer second (no fractional seconds) This 1PPS is measured by the grandmaster and the time transfer accuracy from the master over the network to the SyncPoint card is easily determined Figure 5 below shows this test setup Chart 3: Typical time synchronization accuracy master to slave Chart 3 above shows how well the SyncPoint PCIe-1000 card can synchronize to a PTP grandmaster over an Ethernet network In a benchmark test over an 8 hour period the PTP card was reliably accurate within 18 nanoseconds Push Time to Memory to Improve Speed and Accuracy Once we have delivered accurate time over the network, the next step is to move time from the SyncPoint card to an application program A PTP clock card connects to the PCIe bus Traditionally time is requested from the card by a program This in effect latches a time register on the card and the value is returned to the requesting program This exchange over the bus takes time and can be subject to delays similar to the clock_gettime command, or even worse (these delays are explored further below) The solution as shown in figure 6 is to push time to a DMA memory location so that a program can read that location frequently with little delay Writing time to DMA memory is an effective way to provide highly available and accurate time to an application Time can be expressed as an integer number of nanoseconds and delivered as a single 64 bit number This is useful in time differencing operations if time deltas are of most importance and the difference operation needs to be done quickly Similarly the time can be delivered in major and minor time representing the time in seconds and fractional seconds The challenge in either case is verifying the time accuracy and understanding how the time stamps advance Page 6 of 9

7 Verifying High Availability DMA Time Writes It is convenient that the attribute to be measured is speed in writing the time to memory Time writes are not continuous, but discreet and limited by the speed of the electronics and operating system By reading the DMA time at a very high rate it is possible to observe the time being updated in memory For a perfect clock, the expected result is a periodic advancement in time over an equivalent duration of time Or, simply put, if the measurements were plotted there would be a uniform stair step of N nanoseconds every N nanoseconds Chart 4 below shows that in this test case of rapidly reading the time in memory the time advanced 560 nanoseconds every 560 nanoseconds This uniform advancement in time means that an application on this server can read the DMA time well in excess of 1 million times per second and get an advancing UTC referenced time stamp every time Verifying DMA Time Stamp Accuracy High availability time also needs to be accurate to be of use Generally, to measure time accuracy a more accurate clock is needed In this case it is necessary to determine the time accuracy at the application with just one clock, the SyncPoint PTP clock card The time accuracy of the PTP clock in this instance has been previously determined to be accurate to the master to 18 nanoseconds Since the attribute we are measuring is time, a relatively simple technique will be used to verify the accuracy of time in DMA memory Aside from reading time from DMA memory, time can also be directly read from the SyncPoint PTP card time registers directly over the PCIe bus This technique is shown in Figure 7 Since time should always be advancing in an absolute sense, a time read from DMA followed by a time read over the bus followed by a second DMA time read should all be sequential in time SyncServer S350 PTP Grandmaster SyncServer Grandmaster Time Register PTP Timing Over Ethernet 18 Nanoseconds Compute Server Hardware SyncPoint Clock Card Time Register Push Time Linux OS Figure 6: The stages of time transfer from the PTP Grandmaster time register to a software application running on Linux Chart 4: Time value in the DMA memory address is updated every 560 nanoseconds Application Time in DMA Memory Page 7 of 9

8 The basic relationship that describes this advancement of time is as follows when the time is read alternately from each source: DMA time 1 < Bus Card Time < DMA time 2 If this process is repeated at a high rate it is possible to begin to converge on the accuracy of the DMA time relative to the accurate time register on the hardware clock The above relationship states that if DMA Time 1 is subtracted from the Bus Card Time it should be a positive number Similarly if the Bus Card Time is subtracted from DMA Time 2 it should also be positive This consecutive reading of the times was performed and the differencing time stamps are presented in Chart 5 The values are all positive so we know the relationship proved correct and that the timestamps are synchronized to within some accuracy tolerance (discussed in next section) For this discussion it is assumed that the time required to read the time stamp from DMA memory is very small (sub 100 nanoseconds) Best Possible Time At The Application Layer The measurement technique used in the test described above includes the time required to latch and read the time, thus it is necessary to remove that elapsed processing time to arrive at a comparison of the DMA time stamps compared to the Bus Card register time This estimate involves more robust statistics because the process time is not normally distributed, but has a minimum duration and only gets longer depending on the hardware and software involved with this particular test configuration and computer Deeper analysis of 30,000 measurements favorably reveals the DMA time aligns to the Bus Card Time to less than 290 nanoseconds 50% of the time, and to less than 490 nanoseconds 99% of the time SyncServer S350 PTP Grandmaster SyncServer Grandmaster Time Register PTP Timing Over Ethernet 18 Nanoseconds Compute Server Hardware SyncPoint Clock Card Time Register Push Time Linux OS Pull Time Application Time in DMA Memory Compare Time Read Over PCIe Bus Figure 7: Time is read both from the DMA memory location and the directly from the SyncPoint clock card time register, across the PCIe bus Tbus-Tdma1 Tdma2-Tbus Chart 5: The Difference in time when reading the DMA time value first and then reading the SyncPoint time register over the PCIe bus is labeled Tbus-Tdma1 The difference when reading time across the bus and then from DMA is labeled Tdma2-Tbus Page 8 of 9

9 But what about the other 1%? The other 1% is due to delays in the test setup such as latching or in particular reading the time from the bus card For the computer used in this test there were instances, as can be seen in Chart 5 the red data, where the offset was larger This is due to delays in getting the time to the test program from the bus card Path delays between the time register on the card, the PCIe bus, or the operating system itself delayed the transfer of the time In this 30,000 sample test case there were several instances the time stamp transfer was delayed over a microsecond and in two cases more the 10 microseconds This also can be seen in Chart 5 The problem is that these path delays in the test system were random and of unknowable duration In mission critical systems a delay in a getting a time stamp has two serious side effects, one is the system was held waiting and the second is the duration is unknown so the age/accuracy of the time stamp is also unknown It is for this reason that pushing the time to DMA memory circumvents this time transfer delay behavior seen in this test set up Best Possible Time Around the World The key take away here is that this 490 nanosecond or better timestamp is referenced to the world s extremely accurate Coordinated Universal Time (UTC) Add to that the 18 nanosecond offset of the SyncPoint card to the master and the 50 nanosecond offset of the master to UTC (GPS referenced master clock) and you have time stamps at the application layer better than 560 nanoseconds to UTC Thus, server applications in one city or country can be very closely time synchronized to server applications in another city or country Precise Time Transfer Problem Timing packets delayed by the network Timing packets delayed by the Linux OS stack Clock instability reduces time accuracy Software calls for time randomly delayed by the Linux OS The other important factor is that this is a characterization of a general purpose computer A faster server can move time more quickly to memory and further improve the time accuracy at the application layer The time accuracy in this particular test could be further improved by simply subtracting half of 560 (280) from the DMA time stamp This would provide time to ± 280 nanoseconds relative to the SyncPoint card and be well below microsecond time accuracy This is reasonable only because we know the time is always advancing Solution These three technologies used together: Hardware based PTP time stamping at the master and slave clocks Optimized timing packet pre-filtering and steering algorithms Stable OCXO oscillator insensitive to errant time adjustments caused by PDV Hardware time stamping eliminates the detrimental effect of Linux stack delays Use of a more stable OCXO oscillator that is also less temperature sensitive Write time directly to DMA memory so Linux applications have immediate access to the correct time Conclusion, PTP Clock Cards Deliver the Best Possible Time Accuracy It is the combination of the end end-toend timing solution integrating hardware and software that delivers the best possible time to a Linux application As demonstrated here, a PTP clock card obtaining the time from a reliable source and pushing that time to DMA memory provides the most accurate UTC referenced time to Linux applications The integration of hardware and software overcomes the four main sources of time transfer error when moving time over a network and delivering it to the application The use of a dedicated PTP clock card configured with PTP hardware time stamping, an OCXO oscillator and DMA time writes improves time accuracy and availability for Linux applications and reduces the primary sources of time error in the transferring and delivering of that time to the applications 2300 Orchard Parkway San Jose, California tel: fax: wwwsymmetricomcom 2012 Symmetricom Symmetricom and the Symmetricom logo are registered trademarks of Symmetricom, Inc All specifications subject to change without notice WP/AccurateTimeForLinuxApplications/101612