Solarflare (Legacy) PTP User Guide

Solarflare (Legacy) PTP Copyright 2012 SOLARFLARE Communications, Inc. All rights reserved. The software and hardware as applicable (the "Product") described in this document, and this document, are protected by copyright laws, patents and other intellectual property laws and international treaties. The Product described in this document is provided pursuant to a license agreement, evaluation agreement and/or non-disclosure agreement. The Product may be used only in accordance with the terms of such agreement. The software as applicable may be copied only in accordance with the terms of such agreement. The furnishing of this document to you does not give you any rights or licenses, express or implied, by estoppel or otherwise, with respect to any such Product, or any copyrights, patents or other intellectual property rights covering such Product, and this document does not contain or represent any commitment of any kind on the part of SOLARFLARE Communications, Inc. or its affiliates. The only warranties granted by SOLARFLARE Communications, Inc. or its affiliates in connection with the Product described in this document are those expressly set forth in the license agreement, evaluation agreement and/or non-disclosure agreement pursuant to which the Product is provided. EXCEPT AS EXPRESSLY SET FORTH IN SUCH AGREEMENT, NEITHER SOLARFLARE COMMUNICATIONS, INC. NOR ITS AFFILIATES MAKE ANY REPRESENTATIONS OR WARRANTIES OF ANY KIND (EXPRESS OR IMPLIED) REGARDING THE PRODUCT OR THIS DOCUMENTATION AND HEREBY DISCLAIM ALL IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT, AND ANY WARRANTIES THAT MAY ARISE FROM COURSE OF DEALING, COURSE OF PERFORMANCE OR USAGE OF TRADE. Unless otherwise expressly set forth in such agreement, to the extent allowed by applicable law (a) in no event shall SOLARFLARE Communications, Inc. or its affiliates have any liability under any legal theory for any loss of revenues or profits, loss of use or data, or business interruptions, or for any indirect, special, incidental or consequential damages, even if advised of the possibility of such damages; and (b) the total liability of SOLARFLARE Communications, Inc. or its affiliates arising from or relating to such agreement or the use of this document shall not exceed the amount received by SOLARFLARE Communications, Inc. or its affiliates for that copy of the Product or this document which is the subject of such liability. The Product is not intended for use in medical, life saving, life sustaining, critical control or safety systems, or in nuclear facility applications. SF-107094-CD Issue 6 Issue 6 Solarflare Communications 2012 i

NOTE: This user guide is applicable to the Solarflare (Legacy) PTP implementation (ptpd2). Solarflare s latest PTP software is Solarflare Enhanced PTP (sfptpd) - for more details please refer to the Solarflare Enhanced PTP (SF-109110-CD) available from https:// support.solarflare.com/. Solarflare recommend that users transition to Solarflare Enhanced PTP (sfptpd) to take advantage of the new features not available in ptpd2 and to benefit from future Solarflare PTP development. Future Solarflare adapters will only be supported by Solarflare Enhanced PTP (sfptpd). Issue 6 Solarflare Communications 2012 ii

Table of Contents Chapter 1: Introduction..................................................... 1 1.1 Purpose............................................................. 1 1.2 Definitions, Acronyms and Abbreviations.................................. 1 Chapter 2: Overview....................................................... 2 2.1 Solarflare PTP Network Adapters......................................... 2 2.2 Software support...................................................... 4 2.3 PTP stack............................................................ 5 Chapter 3: Installation...................................................... 8 3.1 System Requirements.................................................. 8 Chapter 4: Using PTPd..................................................... 13 4.1 Download PTPd...................................................... 13 4.2 Run PTPd........................................................... 13 4.3 PTPd modes......................................................... 14 4.4 PTP Statistics........................................................ 15 4.5 PTPd in Operation.................................................... 15 4.6 Accuracy of the SFN5322F SFN6322F under Network Load.................. 18 4.7 Loss of PTP Link Network Connection.................................... 19 Chapter 5: Benchmark Tests................................................ 20 5.1 Test Configuration.................................................... 20 5.2 Solarflare timecheck test application.................................... 21 5.3 OpenOnload......................................................... 22 5.4 Install - Test Components.............................................. 22 5.5 Benchmark test procedures............................................ 24 Chapter 6: Pulse Per Second (1PPS).......................................... 27 6.1 Asymmetric Networks................................................. 27 6.2 1PPS Measurement Procedure......................................... 28 6.3 1PPS in Practice...................................................... 29 Chapter 7: Known Issues and Limitations..................................... 31 Appendix A: ptpd2 options................................................. 32 Appendix B: ptpd2 Delay/Offset Data........................................ 36 Appendix C: Statistical Data................................................ 38 Issue 6 Solarflare Communications 2012 ii

Chapter 1: Introduction 1.1 Purpose This document describes software support of Solarflare's SFN5322F and SFN6322F Server Adapters. These Solarflare SFP+ Time Synchronization Server Adapters support hardware time stamps of PTP packets and can be deployed in networks where there is a requirement to support the IEEE 1588 Precision Time Protocol. The document describes installation procedures for the network adapter and software components that can be used to test the time synchronization capabilities of the SFN5322F and SFN6322F. 1.2 Definitions, Acronyms and Abbreviations PTP Precision Time Protocol PTPd PTP daemon - implementation of IEEE-1588-2008 (PTP version 2) NTP PPM 1PPS Network Time Protocol Parts Per Million 1 Pulse Per Second NOTE: This user guide is applicable to the Solarflare (Legacy) PTP implementation (ptpd2). Solarflare s latest PTP software is Solarflare Enhanced PTP (sfptpd) - for more details please refer to the Solarflare Enhanced PTP (SF-109110-CD) available from https:// support.solarflare.com/. Solarflare recommend that users transition to Solarflare Enhanced PTP (sfptpd) to take advantage of the new features not available in ptpd2 and to benefit from future Solarflare PTP development. Future Solarflare adapters will only be supported by Solarflare Enhanced PTP (sfptpd). Issue 6 Solarflare Communications 2012 1

Chapter 2: Overview 2.1 Solarflare PTP Network Adapters The Solarflare SFN5322F and SFN6322F are dual port SFP+ 10G Ethernet network server adapters that can generate hardware timestamps for PTP packets in support of a network precision time protocol deployment, and in accordance with the IEEE 1588-2008 specifications. With hardware precision and performance, the PTP adapters facilitate PTP slave servers to accurately synchronize internal clocks to a network master clock, or serve as the master clock source. Figure 1: The Adapter Time Stamping Components The SFN5322F and SFN6322F are based on, and retain all hardware and software features and functionality of Solarflare's established SFN5122F and SFN6122F network adapters respectively. In addition, the adapters contain a dedicated time stamping unit which is driven from a high precision oscillator. Receipt or transmission of a PTP formatted packet triggers the generation of an accurate hardware timestamp which is passed by the SFC9020 to the network device driver. The adapter also allows a PTP stack running on the attached server to discipline the adapter's oscillator (both absolute time and clock rate). Note that packet hardware time stamping is only supported on a single port of the network adapter which is the port closest to the PCIe connector. The adapter timestamp function is compliant with the IEEE 1588-2008 (PTP version 2) specifications, and can function as either an 'ordinary' clock or 'master' clock in the network. Issue 6 Solarflare Communications 2012 2

SFN5322F Dual-Port 10GbE SFP+ Adapter The SFN5322F is based on the SFN5122F Dual Port SFP+ adapter with additional components for hardware timestamping of PTP packets. Figure 2: The SFN5322F Adapter Benchmark latency measurements recorded with the SFN5322F adapter are (RTT/2) UDP 2.9us, TCP 3.1us. SFN6322F Dual-Port 10GbE SFP+ Adapter The SFN6322F is based on the SFN6122F Dual Port SFP+ adapter with additional components for hardware timestamping of PTP packets. The SFN6322F also features a 1PPS input that can be used to calibrate the PTP offset and an extremely accurate 1PPS output timing signal aligned to the adapter s Stratum 3 clock. The SFN6322F combines precision time synchronisation with ultra-low latency 10G Ethernet. Figure 3: The SFN6322F Adapter Benchmark latency measurements recorded with the SFN6322F adapter are (RTT/2) UDP 2.2us, TCP 2.4us. Issue 6 Solarflare Communications 2012 3

Time Synchronization Features Ability to maintain synchronization of the system clock typically within 200ns offset from a network master clock. The accuracy obtained is dependent on machine and operating system but can be within 50ns offset from a PTP master clock. Stratum 3 compliant oscillator; Oscillator drift < 0.37 PPM per day; < 4.6PPM over 20 years. Ability to capture a hardware timestamp as selected frames enter/leave the Ethernet MAC. Time stamping for packets formatted according to IEEE 1588-2008 (PTP version 2). Hardware timestamps exposed to Linux via the standard SO_TIMESTAMPING socket API on kernels 2.6.30 later. IOCTL support for time stamping on older kernels, from 2.6.18, that predate SO_TIMESTAMPING. Ability to discipline the network adapter s high precision oscillator in response to PTP timing information. 2.2 Software support Firmware This section describes the software support for the time synchronization Server Adapters. Please refer to chapter 3 Installation on page 8 for instructions on identifying and updating the adapter firmware version. Network driver A Solarflare v3.2 network device driver is required to support the time synchronization features of the adapter. The driver is distributed as both a standalone source RPM or with the OpenOnload distribution from version 201109-u1 onwards. See Installation on page 8 for more details. The v3.2 network driver exposes a number of features of the adapter including: Hardware time stamping of PTP formatted packets. For kernels prior to 2.6.30 there is no standard interface for supporting hardware packet time stamping and the driver exposes this functionality via an IOCTL interface. Starting in Linux kernels 2.6.30, support for hardware time stamping of network packets on TX and RX is formalized and integrated via a new socket option SO_TIMESTAMPING. Details of this interface can be found at http://lxr.linux.no/linux/documentation/networking/ timestamping.txt. Before an application can receive timestamps on a socket it must first issue the IOCTL SIOCSHWTSTAMP to register both the types of packets it wants to receive timestamps on and the format of timestamps it wishes to receive. The SFN5322F SFN6322F hardware time stamping is limited to PTP formatted packets. Access to the control of the precision oscillator on the SFN5322F SFN6322F. The SFN5322F SFN6322F contain a precision clock accurate to < 1 PPM per year. The driver allows the absolute time and frequency of this clock to be controlled via an IOCTL interface. Issue 6 Solarflare Communications 2012 4

2.3 PTP stack PTP slave This interface is used by the Solarflare supplied PTPd stack to run the PTP protocol using the precision oscillator on the adapter. Solarflare are actively monitoring the work to add PTP Hardware Clocks (PHCs) into the Linux kernel (first integrated in Linux kernel version 3.0) which would provide a vendor independent kernel API for controlling the adapter's clock. There are a number of PTP stack implementations which can be used with the adapter. However, it should be noted that many of the available PTP stacks do not support hardware timestamps. PTPd is a freely available PTP implementation available from http://ptpd.sourceforge.net/. It is BSD/ ISC-style licensed and runs on most 32-bit and 64-bit UNIX OSs including Linux. The latest version is PTPd 1.1.0 which supports PTP version 1 (IEEE-1588-2002) and PTPd 2.1.0 release which supports PTP version 2 (IEEE-1588-2008). Although there are outstanding patches to PTPd to support hardware time stamping, these are yet to be integrated into the main code base. Also, none of the outstanding patches support running PTPd on the adapter and synchronizing system time directly to this clock. Therefore, as a convenience to customers, Solarflare have enhanced PTPd 2.1.0 (for PTP version 2) to take advantage of all the hardware features of SFN5322F SFN6322F adapters. Solarflare intends to continue to enhance PTPd for use in high precision environments. Solarflare have currently only verified the operation of the SFN5322F and SFN6322F adapters with this supplied PTPd stack. To obtain the highest accuracy when operating as a PTP slave, Solarflare's supplied PTPd stack uses a two stage approach to synchronize the server s system time. PTP runs directly between the upstream master clock and the high precision clock on the SFN5322F SFN6322F adapter. Hardware timestamps of PTPv2 packets allow PTPd to accurately synchronize and discipline the frequency of the high precision oscillator on the adapter. A second clock servo runs between the adapter s clock and the system clock to synchronize and discipline the server's system clock. Figure 4: PTP Slave PTPd output, as shown in the following extract, shows both the adapter s time offset from the master and the offset of the system clock from the adapter's clock. Issue 6 Solarflare Communications 2012 5

[master->nic] 2011-09-30 10:10:01.86712, offset: 45ns, [nic->system] offset: -121ns Applications running on the server can call standard POSIX/Linux system time calls such as clock_gettime() to obtain an accurate time reading. Most modern Linux kernels implement these calls entirely at user space (using a VDSO) and provide an accurate time access API with low CPU overhead. On latest CPU generation machines (Xeon CPU E3130 @ 3.20GHz, Solarflare have measured the execution time of calls to clock_gettime() as 61ns when making a syscall (kernel.vsyscall64=0), and 24ns when configured to use a VDSO (kernel.vsyscall64=1). NOTE: Although it would seem useful for applications to have the ability to read the time directly from the clock on the SFN5322F and SFN6322F adapters, this would require hardware reads across the PCIe bus. PCIe reads are slow and such a call would take in the order of a microsecond to complete and would therefore introduce significant errors in the time obtained. Loss of PTP Link Network Connection If the network connection to the master clock is lost at any point, Solarflare s PTPd continues to discipline the system clock using the adapter s high precision oscillator. This prevents the system clock drifting beyond the accuracy of the high precision oscillator on the adapter. Once network connectivity to the PTP master is restored, PTPd will once again start to discipline the SFN5322F SFN6322F adapter s high precision oscillator to correct any drift that has occurred. Refer to Loss of PTP Link Network Connection on page 19 for further details. PTPd Master Modes The Solarflare adapter can function as a master clock in two modes as identified below. Non-NTP Master Mode An undisciplined system clock on a server is not an ideal clock source for a PTPd master. Therefore when operating as a PTP master, by default, Solarflare's supplied PTPd stack uses the adapter's high precision oscillator as the time source used by PTP. When PTPd starts as a master, the system clock time is read once and programmed into the SFN5322F SFN6322F adapter so it has a coherent start time. The adapter clock then free runs using its Stratum 3 compliant oscillator with PTPd allowing a number of PTPd slaves to synchronize to this time source. The system time on the server acting as the master is continuously adjusted and disciplined by the adapter clock. The time published via PTPd is the adapter clock time. Figure 5: PTP Master - Non NTP Mode Issue 6 Solarflare Communications 2012 6

NOTE: This scheme gives a highly stable clock to be used as a PTPd master. However, the time programmed into the SFN5322F SFN6322F when PTPd starts will likely have a small offset from UTC. Also no attempt is made to keep the high precision oscillator on the adapter synchronized to UTC. Therefore, over time it would be expected that adapter time will deviate from UTC. NTP Master Mode In NTP mode it is assumed that NTP is being used to synchronize the system time on the server running the PTP master. Therefore, the master server must have access to an NTP server, usually via a LAN, and the NTP service must be enabled on the master server. A clock servo runs between the system clock and the adapter clock to synchronize and discipline the adapter s clock once per second. PTP between master and slave ensures that the slave adapter clock is synchronized to the master adapter clock. The slave system clock is disciplined by the slave adapter clock. The time published via PTPd is the system clock time. Figure 6: PTP Master - NTP Mode NOTE: Unlike the non-ntp master mode, the time source for PTP is the less stable system clock. However, as the system clock is being disciplined by NTP the PTP time published with PTPd stays in sync with UTC. Typically, NTP clients slew the system clock by a large amount when they routinely update. The result is that it can take a period of time for PTP slaves to re-converge on the master after such adjustments are made by NTP. For this reason Solarflare recommend either: - selecting an NTP client that takes care to make small and gradual changes to system time, or - deploying a third party dedicated PTP master, most of which can discipline their clock from an external NTP server. Issue 6 Solarflare Communications 2012 7

Chapter 3: Installation 3.1 System Requirements This section identifies all components required to deploy the SFN5322F or SFN6322F adapter for PTP operation. Linux 32bit or 64 bit kernel. SFN5322F or SFN6322F network server adapter. Solarflare supplied PTPd stack with support for hardware time stamping. Solarflare v3.2 network driver. This driver can be installed from one of two packages: - A source driver RPM or - OpenOnload distributions starting from 201109-u1 release. NOTE: OpenOnload is not used to support the PTP time stamping operation of the adapter. Hardware time stamping of PTP packets is handled between the network device driver and firmware running on the adapter. OpenOnload need only be installed by customers who want to accelerate networking applications with OpenOnload OR use the Solarflare 'timecheck' utility with OpenOnload as a way to validate the functionality of the time synhronization adapter. Adapter Driver and Firmware Versions PTP operation and hardware timestamping require a Solarflare network adapter driver v3.2 or later and adapter firmware v3.2.0.6039 or later. To check the Solarflare adapter driver and firmware use the ethtool command. # ethtool -i eth4 driver: sfc version: 3.2.0.6099 firmware-version: 3.2.0.6100 bus-info: 0000:07:00.0 If updated driver or firmware are required these can be downloaded from https:// support.solarflare.com/. Issue 6 Solarflare Communications 2012 8

Step 1: Server Pre-Install Setup Tickless Kernel - nohz A feature of modern operating systems is that they use a "tickless" kernel which aims to reduce power consumption during kernel idle periods. This is achieved by stopping the regular timer tick on CPU cores which are idle. However, experiments at Solarflare have proven that PTPd produces improved and more consistent results when the kernel always receives periodic timer ticks. PTPd relies on the ability to accurately change the speed of the system clock by very small and precise amounts. The Linux kernel implements this adjustment to system clock rate with integer arithmetic, minimizing the error term to the target clock rate in every timer tick. However, when the timer tick doesn t run, the error in tracking to the requested clock rate increases, and the system time diverges from the clock rate requested. When the system wakes from idle, the timer tick runs and the kernel corrects for the error term. Whether the kernel operates in a tickless mode is configured by the nohz boot time option with the majority of Linux distributions defaulting to a tickless kernel. To achieve the highest accuracy with PTP, Solarflare suggest configuring the kernel to receive timer ticks even when the system is idle. This can be achieved by adding "nohz=off" to the kernel boot parameters in the /boot/ grub/grub.conf file. IPTables Users must ensure that no rule exists in iptables that will prevent PTP packets from reaching the slave PTPd process. Step 2: Install the Network Adapter Complete instructions for the deployment and installation of the network adapter can be found in the Solarflare Server Adapter SF-103837-CD. Step 3: Install Network Driver An v3.2 or newer Solarflare network device driver is required to enable the hardware time synchronization function of the SFN5322F and SFN6322F network adapters. The driver can be downloaded and installed from a source RPM or as part of an OpenOnload installation. Download and Install OpenOnload OpenOnload distributions starting from September 2011 update1 include a v3.2 network driver which supports the time synchronization functionality of the adapter. 1 Download the openonload-201109-u1.tgz file from http://www.openonload.org/ 2 Copy the file to a directory on the target machine e.g. /tmp and as root execute the following commands: tar -zxvf openonload-201109-u1.tgz Issue 6 Solarflare Communications 2012 9

This will unpack the tar file and, within the current directory, create a sub-directory called openonload-201109-u1 which contains the scripts directory from which subsequent commands should be invoked. 3 The following command will build and install OpenOnload and required drivers in the system../onload_install 4 Following successful installation the following 3 lines are output by the onload_install process are: Onload_install: Build complete Onload_install: Installing OpenOnload. Onload_install: Install complete. 5 If Solarflare drivers were already installed on the server use the following command to reload the latest driver from the OpenOnload distribution: onload_tool reload 6 To confirm OpenOnload installation and that the kernel module component is loaded use the onload -v command: onload -v OpenOnload 201109-u1 Copyright 2006-2011 Solarflare Communications, 2002-2005 Level 5 Networks Built: Dec 22 2011 13:20:00 (release) Kernel module: 201109-u1 usage: onload [options] <command> <command-args> options: --profile=<profile> -- comma sep list of config profile(s) --force-profiles -- profile settings override environment --no-app-handler -- do not use app-specific settings --app=<app-name> -- identify application to run under onload --version -- print version information -v -- verbose -h --help -- this help message Download and install the driver source RPM 1 Download the driver source RPM zip file SF-103848-LS from https://support.solarflare.com/. 2 Copy the zipfile to a directory on the target machine e.g. /tmp, unzip it and, as root, execute the following commands: rpmbuild --rebuild sfc-3.2.0.6040-1.src.rpm Issue 6 Solarflare Communications 2012 10

Building the binary RPM from the source RPM will output a 'Wrote ' line identifying the location of the binary file - as in the following example (actual output depends on the kernel the driver is built on): Wrote: /root/rpmbuild/rpms/x86_64/kernel-module-sfc-rhel6-2.6.32-131.0.15.el6.x86_64-3.2.0.6040-1.x86_64.rpm 3 Install the driver (using example rpm location from above): rpm -ivh /root/rpmbuild/rpms/x86_64/kernel-module-sfc-rhel6-2.6.32-131.0.15.el6.x86_64-3.2.0.6040-1.x86_64.rpm Preparing... ########################################### [100%] 1:kernel-module-sfc-RHEL6########################################### [100%] 4 Load the network adapter driver. modprobe sfc For more details of how to install a Solarflare source RPM install refer to the Solarflare Server Adapter SF-103837-CD. Step 4: Update Adapter Firmware NOTE: The Solarflare Utilities RPM for Linux contains a boot ROM utility (sfboot) and a flash firmware update utility (sfupdate). The RPM package is available as a 32bit binary and 64bit binary: SF-107183-LS is a 32bit binary SF-107601-LS is a 64bit binary The network adapter firmware must be version 3.2.0.6039 or later to support PTP. 1 Download the sfutilities package from https://support.solarflare.com/. 2 Unzip the file to reveal the binary RPM 3 Install the RPM e.g. # rpm -Uvh sfutils-<version>.rpm 4 Identify the current firmware version on the adapter. # sfupdate 5 Replace the adapter firmware with the version in this sfupdate. # sfupdate --write Full instructions on using sfupdate can be found in the Solarflare Network Adapter SF- 103837-CD. Issue 6 Solarflare Communications 2012 11

Step 5: Connect the SFN5322F SFN6322F Timestamping Port Hardware time stamping is only functional on the port closest to the PCIe connector. Figure 7: Identify Timestamping Port Use ethtool to identify the hardware timestamping port: ethtool -i ethn driver: sfc version: 3.2.0.6040 firmware-version: 3.2.0.6039 bus-info: 0000:07:00.0 From the PCI bus-info a zero function value (last digit) identifies the timestamping port on the network adapter. It can be useful to use ethtool to identify the timestamping port on the back panel by 'blinking' the port LED for a specified number of seconds: ethtool -p ethn 10 Issue 6 Solarflare Communications 2012 12

Chapter 4: Using PTPd 4.1 Download PTPd As a convenience to customers Solarflare have adapted a version of PTPd to work with the SFN5322F and SFN6322F adapters. The ptpd2 daemon is not compatible with IEEE- 1588-2002 (PTP version 1). NOTE: The Solarflare PTPd RPM package is available as a 32bit binary and 64bit binary: SF-107595-LS is a 32bit binary SF-107596-LS is a 64bit binary 1 Download the required PTPd package from https://support.solarflare.com/. 2 Unpack the compressed file using the tar command e.g. tar -zxvf SF-107595-LS-8.tgz 4.2 Run PTPd NOTE: PTPd can take 30 minutes or more to stabilize the times on the slave machines. To view all PTPd command line options run the following command../ptpd2 -H Ensure that the timestamping port of the adapter is configured with an IP address suitable for the network configuration. Stop the ntpd service. This is essential on a PTP slave. NTP can be enabled on the PTP master, but this can cause PTPd to take longer to stabilize times. To start the ptpd2 process as the PTP master (command line mode - without NTP)./ptpd2 -W -b <ethn> -d -c -S To start the ptpd2 process as the PTP master (command line mode - with NTP)./ptpd2 -G -b <ethn> -d -c -S To start the ptpd2 process as a PTP master (software timestamping). This mode is compatible with non-ptp adapters from Solarflare which do not benefit from the time synchronization features of the PTP adapters../ptpd2 -W -b <ethn> -Xsystem -d -c -S Issue 6 Solarflare Communications 2012 13

OR To start the ptpd2 process as a PTP slave (command line mode)./ptpd2 -g -b <ethn> -d -c -S To start the ptpd2 process as a PTP slave (software timestamping). This mode is compatible with non-ptp adapters from Solarflare which do not benefit from the time synchronization features of the PTP adapters../ptpd2 -g -b <ethn> -Xsystem -d -c -S Where N is the identifier of the timestamping port on the adapter. 4.3 PTPd modes By default, Solarflare s PTPd selects the optimal mode to run in based on whether it is started as a PTP master or a PTP slave (see section PTP stack on page 5 for more details). However, PTPd supports a number of modes to control how it obtains time stamps and which clocks it synchronizes. If desired these modes, summarized in the table below, can be manually selected on the command line. Command line Option Clock adjusted by PTP Table 1: PTP Mode H/W timestamp of PTP packets Requires/ uses NIC clock skewing system System No No 1 linux_hw System Yes No 2 both Both system and adapter Notes Yes Yes 3 nic Adapter Yes Yes 4 Table Notes: 1 Only mode provided by PTPd. 2 Time used and controlled by PTP is the system time. Network adapter is used to time stamp PTP packets. On kernels 2.6.30 onwards, this mode uses SO_TIMESTAMPING interface - the timestamp returned as system time with conversion from NIC time performed by driver. On kernels that predate SO_TIMESTAMPING, PTPd uses an ioctl() to obtain the same result. 3 PTP used to control network adapter time - the main time seen by PTP stack is the network adapter's time. Offsets between system and adapter time are provided by the network driver. System time is slaved to adapter time by feeding these offset into another instance of the clock servo in the PTP stack. This is the default mode for Solarflare PTPd when configured to run as a PTP master or slave. Issue 6 Solarflare Communications 2012 14

4 PTP uses and controls the time within the network adapter. Timestamps are obtained from the SO_TIMESTAMPING socket interface. 4.4 PTP Statistics Solarflare PTPd produces offset and delay data using the -d and -D command line options. Refer to Appendix B: ptpd2 Delay/Offset Data on page 36 for details. 4.5 PTPd in Operation The following graph represents the output from a PTP slave server and shows the offset of the slave hardware clock as it converges on the master clock Figure 8: PTPd in Operation Issue 6 Solarflare Communications 2012 15

Figure 9 below shows the offset (as reported by PTPd) of the precision oscillator on the SFN5322F from the PTPd master after PTPd has run for sufficient time to stabilize. Figure 9: Time offset master Figure 10 below shows the corresponding offset (as reported by PTPd) of the system time from the precision oscillator on the SFN5322F during the same period Figure 10: Time offset NIC to system Issue 6 Solarflare Communications 2012 16

Figure 11, shows the difference in time between two slaves (as measured by TS-Associates AppTap timing card) as PTPd pulls the two slaves into synchronization. Each slave is within approximately 100 nanoseconds of the master and therefore the difference between the two slaves is at most 200 nanoseconds. Figure 11: Time Difference between two slaves Issue 6 Solarflare Communications 2012 17

4.6 Accuracy of the SFN5322F SFN6322F under Network Load To obtain the highest accuracy the PTP protocol requires a network with constant latency. Standards such as PTP boundary clock and PTP transparent clock allow network switches to be PTP aware and measure latencies to allow the PTP end points to compensate for any variance in switching times for PTP packets. However, even with standard non-ptp aware switches, the two stage PTP synchronization approach used by the adapter can provide good accuracy under significant network load. Solarflare has demonstrated slave to master offsets within 200ns on a lightly loaded network. However, even under bursty conditions of up to 50% 10G line rate, the SFN5322F SFN6322F demonstrated slave to master offsets of within 500ns. When the bursty condition cleared, the slave to master offsets returned to within 200ns. Figure 12 shows SFN5322F PTP accuracy when used in an environment with bursty network load of up to 50% line rate. The test employed the SFN5322F adapters as master and slave configured via a Cisco Nexus 5000 series switch. Figure 12: PTPd Under Load Issue 6 Solarflare Communications 2012 18

4.7 Loss of PTP Link Network Connection If the network connection to the master clock is lost at any point, Solarflare s PTPd continues to discipline the system clock using the adapter s high precision oscillator. This prevents the system clock drifting beyond the accuracy of the high precision oscillator on the adapter. Once network connectivity to the PTP master is restored, PTPd will once again start to discipline the adapter s high precision oscillator to correct any drift that has occurred. Figure 11 below demonstrates the effect on the slave adapter of losing the network connection to the master server. The disconnect can be seen occurring just after 2.24 on the master to NIC trace. The precision oscillator continues to discipline the system clock during the outage. When the network connection is restored the adapter hardware clock continues to track to the master clock. Figure 13: Effect of Master Disconnect Issue 6 Solarflare Communications 2012 19

Chapter 5: Benchmark Tests This section describes a simple test configuration and procedure to verify PTP operation using the SFN5322F SFN6322F time synchronization server adapter. It should be noted that this procedure cannot replicate the accurate measurements that can be attained from a hardware based measurement solution (see the example output in Figures 6-9). However, this software solution, benefiting from the low latency and low jitter of OpenOnload, can provide an approximation of the timing accuracy possible. As with any precision timing measurements extreme care needs to be taken to ensure that measurement errors do not dominate the results. 5.1 Test Configuration The following diagram shows the equipment configuration required to verify PTP operation using the Solarflare supplied "timecheck" application. Figure 14: Test Configuration If the PTP master is being supplied via a server containing an SFN5322F SFN6322F adapter, then the PTP master and "timecheck" control operations can be co-located on the same server. Test Events Sequence The "timecheck" test application runs on the control and multiple slave servers. Its aim is to allow users to determine the difference in system time between multiple slave machines. It does this by a simple packet exchange between the control and slave machines. Its operation is as follows: 1 Each second a message is multicast by the timecheck control to all slave servers. 2 This message should reach all slaves (Slave#1 and Slave#2) at the same time. 3 The message is received at the timecheck application running on the slaves via the socket API. a) The timecheck application immediately retrieves the local system time using clock_gettime(clock_realtime) b) The timecheck application on each slave machine returns the system time recorded in (a) to the timecheck control (via TCP). Issue 6 Solarflare Communications 2012 20

4 The timecheck control application calculates and displays the time offset between the time the control message was initially sent and the time it was received by each slave server. If each slave server receives the message at the same time then the difference in the times reported in step 4 monitors the time difference between the 2 slaves. Although the multicast message should reach all machines at a very similar time (step 2) there can be considerable variance in when the message is received via the recv() socket API by the timecheck application on the slaves (step 3). For timecheck to provide meaningful results the difference in time between the calls to obtain system time at the slaves (step 3a) is required to be very small or this variance will be larger than the time difference being measured. The variance in step 3 can be minimized by running the timecheck application on the slaves using OpenOnload. OpenOnload can be configured to "spin" (busy-wait) on the slave machines at usermode waiting for the message to arrive. This minimizes the jitter in the time the recv() socket API call returns and therefore minimizes the time difference of step 3a on the multiple slave machines. Fixing the timecheck application to run on an available CPU core and isolating (or shielding) this CPU core from processing interrupts further minimizes the time difference at step 3a on the multiple slave machines. It should be noted that significant timing errors will be displayed by timecheck from any events that cause jitter in step 3 above. For example, If OpenOnload is not set correctly to busy-wait spinning, or set to a spin time which is too short, the received multicast message will suffer jitter from the associated interrupt and rescheduling of the application to unblock. If the timecheck application is not affinitized to run on a single free CPU then any rescheduling of timecheck to run on a different CPU core will cause significant errors. If the CPU core that timecheck is running on is servicing interrupts then this disruption can also cause significant timing errors. NOTE: All the above are examples of measurement error and are not errors in the time synchronization between the PTP slaves. Extreme care is needed to ensure that measurement errors do not dominate the results returned by 'timecheck' or any other measurement technique. 5.2 Solarflare timecheck test application The Solarflare supplied timecheck application is a command line utility which can be used to test the hardware time stamp function of the SFN5322F SFN6322F adapter. Multicast messages are sent to address 225.1.2.3 through port 41234 and responses received at port 41235 by default. These values can be configured using the timecheck command line options. The timecheck application should be run on control and up to 10 slave servers. Issue 6 Solarflare Communications 2012 21

5.3 OpenOnload OpenOnload is the Solarflare accelerated network middleware that achieves low latency and minimum jitter for time-sensitive TCP and UDP applications. OpenOnload is an implementation of TCP and UDP over IP which is dynamically linked into the user's application address space and is granted direct (but safe) access to the network directly from the application without involvement of the operating system. This technique is known as 'kernel bypass'. No changes are required to the user application. An application running with OpenOnload can spin (busy-wait) for packets to arrive at its socket receive queue. When a packet does arrive it can be instantly processed by the application. This is in contrast to an interrupt driven model where an application calls into its socket to retrieve any received packets, finding none present the application process will 'sleep' and will have to be 'woken' by an interrupt when a packet does finally arrive. Applications accelerated by OpenOnload have lower latency and display less jitter or variance because they are subject to fewer scheduling holdups and interrupts than an application that relies purely on kernel resources. 5.4 Install - Test Components Download and Install OpenOnload and the Network Adapter Driver OpenOnload distributions from the September-2011 update1 distribution includes the v3.2 network driver supporting the time synchronization functionality of the SFN5322F SFN6322F adapter. 1 Download the openonload-201109-u1.tgz file from http://www.openonload.org/ 2 Copy the file to a directory on the target machine e.g. /tmp and as root execute the following commands: tar -zxvf openonload-201109-u1.tgz This will unpack the tar file and, within the current directory, create a sub-directory called openonload-201109-u1 which contains the scripts directory from which subsequent commands should be invoked. 3 The following command will build and install OpenOnload and required drivers in the system../onload_install 4 Following successful installation the following 3 lines are output by the onload_install process are: Onload_install: Build complete Onload_install: Installing OpenOnload. Onload_install: Install complete. Issue 6 Solarflare Communications 2012 22

5 If Solarflare drivers were already installed on the server use the following command to reload the latest driver from the OpenOnload distribution: onload_tool reload 6 To confirm OpenOnload installation and that the kernel module component is loaded use the onload -v command: onload -v OpenOnload 201109-u1 Copyright 2006-2011 Solarflare Communications, 2002-2005 Level 5 Networks Built: Dec 22 2011 13:20:00 (release) Kernel module: 201109-u1 usage: onload [options] <command> <command-args> options: --profile=<profile> -- comma sep list of config profile(s) --force-profiles -- profile settings override environment --no-app-handler -- do not use app-specific settings --app=<app-name> -- identify application to run under onload --version -- print version information -v -- verbose -h --help -- this help message Download and Build the timecheck test application To download and build the timecheck application 1 Download the timecheck package SF-106824-LS from https://support.solarflare.com/. 2 Copy the compressed tar file to a server and unpack using the tar command: tar -zxvf SF-106824-LS-3.tgz 3 This will create a subdirectory called timecheck containing the src directory. Change to the src directory and run the make command to build the timecheck application. On all hosts - Master and Slave Servers Server running Linux RHEL5 or 6 (with regular or MRG 2.0 kernel). SFN5322F or SFN6322F network adapter. OpenOnload September-2011 update1 or later Onload release. PTPd running. 'timecheck' application running. Issue 6 Solarflare Communications 2012 23

5.5 Benchmark test procedures Timecheck control server Configure a route to ensure that multicast traffic is received at the correct interface ip route add 225.1.2.0/24 dev ethn Run the timecheck application../timecheck -c -l 1000 <address> Where address is the IP address of the interface through which control will receive responses from the timecheck slave servers. Slave clock server Configure a route to ensure that multicast traffic is received at the correct interface ip route add 225.1.2.0/24 dev ethn Improved results can be achieved if the spinning time (busy-wait) period is extended beyond the multicast receive interval (1 second) and if the CPU processing the PTP messages is isolated from normal kernel balancing and scheduling algorithms. To isolate cpu 1 set the isolcpus parameter in the /boot/grub/grub.conf file, i.e. isolcpus=1. Set the maximum time OpenOnload will spin waiting on the recv() socket call made by timecheck using the EF_POLL_USEC environment variable. export EF_POLL_USEC=10000000 Run timecheck application (with OpenOnload). onload taskset -c 1./timecheck <address> Where address is the IP address of the interface over which multicast messages are received and responses returned. The slave process exits when the timecheck control completes its iterations. Timecheck Usage To view all timecheck command line options run the following command../timecheck -? Usage: timecheck -c -s -p port -a address -l loops -d delay -u -r replyport -o {interface_ip} -c [--control] control -s [--slave] slave Issue 6 Solarflare Communications 2012 24

-p port [--port] network port for control send -a address [--address] network address must be multicast -l loops [--loops] number of iterations (control) -d delay [--delay] delay between iterations (control, seconds) -r replyport [--replyport] network port for control receive -u [--unsorted] don't sort output by IP address -o [--oneline] one client per line (only applicable for sorted output) interface_ip IP address over which multicast to be transferred Timecheck Output The following is an example of the timecheck output displayed on a terminal attached to the control server. (Columns are numbered for ease of explanation). 172.16.129.26 172.16.129.27 1 2 3 4 5 6 67 C 1317736824.140829466 : 172.16.129.26-0.166259544 172.16.129.27-0.165619343 68 C 1317736826.140891758 : 172.16.129.26-0.166695508 172.16.129.27-0.166059467 69 C 1317736828.140956131 : 172.16.129.26-0.167125354 172.16.129.27-0.166500100 70 C 1317736830.141017713 : 172.16.129.26-0.167555910 172.16.129.27-0.166941247 71 C 1317736832.141077431 : 172.16.129.26-0.167988353 172.16.129.27-0.167383992 72 C 1317736834.141136857 : 172.16.129.26-0.167826270 172.16.129.27-0.168420275 73 C 1317736836.141195717 : 172.16.129.26-0.168852345 172.16.129.27-0.168268672 74 C 1317736838.141254831 : 172.16.129.26-0.169285233 172.16.129.27-0.168712029 75 C 1317736840.141314272 : 172.16.129.26-0.169716563 172.16.129.27-0.169153899 76 C 1317736842.141336779 : 172.16.129.26-0.170148606 172.16.129.27-0.169596441 77 C 1317736844.141395522 : 172.16.129.26-0.170581035 172.16.129.27-0.170039336 Before time stamps are displayed the timecheck control lists all endpoints added as clients (slaves). On a line of output slave servers are listed in IP address order. 1. Sequence number of the response 2. time (seconds) the request was sent from the master 3. IP address of first slave server 4. Offset of first slave clock from master clock 5. IP address of second slave server 6. Offset of second slave clock from master clock From the sample timecheck output above it can be observed that the offset values calculated by the master for each of the two slaves (columns 4 and 6) increase with each iteration. This happens because the system clock on the master server is not being disciplined (the PTP master is running from the high precision oscillator on the adapter). The point of interest in the output above, is the time difference between the two slaves. The two slaves servers remain in extreme close proximity to each other - and this happens because both system and NIC clocks on the slave servers are being disciplined. Issue 6 Solarflare Communications 2012 25

Example timecheck output This following graph is generated from running timecheck with OpenOnload on 2 slaves as PTPd pulls the two slaves into synchronization with the PTP master. Timecheck is affinitized at each slave (using taskset) to run on a single core with CPU shielding isolating these CPU cores from operating system interruptions. NOTE: The accuracy of these measurements are limited and will show timing differences larger than really exist i.e. due to measurement limitations. Data outliers are expected and can be due to SMIs or other operating system interrupts that cannot be shielded e.g. timer interrupts Figure 15: Time Difference - Two Slaves Issue 6 Solarflare Communications 2012 26

Chapter 6: Pulse Per Second (1PPS) 6.1 Asymmetric Networks Asymmetric networks present a particular problem when attempting to account for network latency during PTP offset calculations between master and slave servers. PTP assumes symmetry in the network and the PTP protocol is not able to detect asymmetry in the network paths between master and slave. Asymmetry can be present for a number of reasons including the store and forward delay in switches serving asymmetric networks as illustrated in Figure 16. Figure 16: Asymmetric Network The result is that PTP offsets between master and slave will converge, but will be wrong by a constant offset from the master equal to half the asymmetry, for example Transmit time master to slave: 5us Transmit time slave to master: 1us One way delay: (5+1)/2 = 3us So 3us is added to the time offsets received from the master clock. 1PPS will display a mean offset value of 2us (5-3) Actual asymmetry should be double this observed value i.e. 4us. Measuring and Adjusting for Asymmetric Latency The Solarflare SFN6322F adapter supports 1PPS input/output interfaces to allow asymmetry in the network to be measured. On a dedicated wire connection between master 1PPS output and slave 1PPS input, the master emits a single pulse every second. The leading edge of each pulse denotes the exact start of a one second period. When the leading edge of a pulse is detected by the slave adapter, firmware on the adapter is able to calculate the offset from its own start of second period. The observed mean 1PPS offset value (doubled) is applied to PTPd which uses this value to compensate for asymmetry in the network. This calibration is only required once when configuring the network and need only be performed on one slave server in each network segment which share a common network path to the PTP master. Issue 6 Solarflare Communications 2012 27

There is no need for a permanent 1PPS connection to the Solarflare adapter. Refer to 1PPS in Practice on page 29. 6.2 1PPS Measurement Procedure 1. PTPd should be running between master and slave servers, but does not need to be fully stabilized before the 1PPS value can be applied. 2. The master 1PPS output should be connected to a single slave 1PPS input. 3. On the slave server, for a short period e.g. 10-30 minutes, observe the 1PPS mean offset value from the ptp_mc_pps_off_mean file to identify the mean offset value. Refer to Appendix C: Statistical Data on page 38 for instructions on reading the 1PPS statistical data files. 4. On all slaves on the same network segment, configure PTPd with knowledge of the mean 1PPS offset. e.g if ptp_mc_pps_off_mean = 3400 - stop the PTPd process - restart the PTPd process adding the -l command line option and the ptp_mc_pps_off_mean value (doubled) i.e. -l6800,0. The offset is always input as a positive value. 5. Continue to observe the 1PPS compensated mean offsets. 6. Repeat steps 3-5 adding the 1PPS mean offset (doubled) value each time to the last applied value until the observed 1PPS mean value is reduced to its lowest possible value. Example: first period observed 1PPS mean offset: -3400 ptpd2 command line option applied value: -l6800,0 second period observed 1PPS mean offset: 40 ptpd2 command line option applied value: -l6880,0 Issue 6 Solarflare Communications 2012 28