White Rabbit for Time Transfer

Transcription

1 White Rabbit for Time Transfer Erik van der Bij Hardware and Timing Section CERN 5 June 2014 Technology and Instrumentation in Particle Physics TIPP 2014 Amsterdam Erik van der Bij White Rabbit for Time Transfer 1/40

2 Outline 1 Introduction 2 Motivation 3 Time Distribution 4 Data Distribution 5 System Components 6 Applications 7 Openness 8 Summary Erik van der Bij White Rabbit for Time Transfer 2/40

3 What is White Rabbit? Main features of a White Rabbit network: transparent, high-accuracy synchronization low-latency, deterministic data delivery designed for high reliability Accelerator s control and timing International collaboration Based on well-known technologies Open Hardware and Open Software Erik van der Bij White Rabbit for Time Transfer 3/40

4 White Rabbit enhanced Ethernet Few thousands nodes Fiber medium Up to 10 km fiber links Bandwidth: 1 Gbps WR Switch: 18 ports Non-WR Devices Ethernet features (VLAN) & protocols (SNMP) Erik van der Bij White Rabbit for Time Transfer 4/40

5 Introduction Motivation Time Distribution Data Distribution System Components Applications Openness White Rabbit enhanced Ethernet Two separate services (enhancements to Ethernet) provided by WR: Synchronization: accuracy better than 1 ns precision (tens of ps sdev skew max) Deterministic, reliable and low-latency Control Data delivery Erik van der Bij White Rabbit for Time Transfer 5/40 Summary

6 White Rabbit applications Existing applications: CERN Neutrinos to Gran Sasso Erik van der Bij White Rabbit for Time Transfer 6/40

7 White Rabbit applications Existing applications: CERN Neutrinos to Gran Sasso Future applications: CERN and GSI Erik van der Bij White Rabbit for Time Transfer 6/40

8 White Rabbit applications Existing applications: CERN Neutrinos to Gran Sasso Future applications: CERN and GSI The Large High Altitude Air Shower Observatory (China) Erik van der Bij White Rabbit for Time Transfer 6/40

9 White Rabbit applications Existing applications: CERN Neutrinos to Gran Sasso Future applications: CERN and GSI The Large High Altitude Air Shower Observatory (China) Potential applications: Cherenkov Telescope Array Erik van der Bij White Rabbit for Time Transfer 6/40

10 White Rabbit applications Existing applications: CERN Neutrinos to Gran Sasso Future applications: CERN and GSI The Large High Altitude Air Shower Observatory (China) Potential applications: Cherenkov Telescope Array European deep-sea research infrastructure (KM3NET) Long distance Time Transfer Erik van der Bij White Rabbit for Time Transfer 6/40

11 Origin of White Rabbit Project Renovation of CERN s General Machine Timing (GMT) GMT is great but...: Manual calibration (with CS clock) RS-422, 500kbps Unidirectional communication Separate network required Custom design, complicated maintenance White Rabbit is meant to solve these problems Erik van der Bij White Rabbit for Time Transfer 7/40

12 Origin of White Rabbit Project Renovation of CERN s General Machine Timing (GMT) GMT is great but...: Manual calibration (with CS clock) RS-422, 500kbps Unidirectional communication Separate network required Custom design, complicated maintenance White Rabbit is meant to solve these problems Erik van der Bij White Rabbit for Time Transfer 7/40

13 Introduction Motivation Time Distribution Data Distribution System Components Applications Openness Time distribution in White Rabbit High accuracy/precision synchronization Deterministic, reliable and low-latency Control Data delivery Erik van der Bij White Rabbit for Time Transfer 8/40 Summary

14 Time Distribution in White Rabbit Network Synchronization with sub-ns accuracy and tens of ps precision Combination of Precision Time Protocol (PTP) synchronization Synchronous Ethernet (SyncE) syntonization Digital Dual-Mixer Time Difference (DDMTD) phase detection Erik van der Bij White Rabbit for Time Transfer 9/40

15 Precision Time Protocol (IEEE1588) Packet-based synchronization protocol Synchronizes local clock with the master clock Link delay evaluated by measuring and exchanging packets tx/rx timestamps Erik van der Bij White Rabbit for Time Transfer 10/40

16 WR Precision Time Protocol Extension Master time Announce Slave time Relative Delay Coefficient: t 1 t 4 Sync Follow_Up Delay_Req Delay_Resp t2 t 3 (t4 -t)-(t 1 3 -t) 2 µ= 2 µ WR Link Delay Model Fixed Delays: rxs txs rxm txm delay = µ + asymmetry ms offset = t - (t + ) ms 2 1 delay ms Solution for Ethernet over a Single-mode Optical Fiber αµ + α asymmetry = txm + rxs 2 + α Erik van der Bij White Rabbit for Time Transfer 11/40

17 Synchronous Ethernet (SyncE) Common clock for the entire network All network devices use the same physical layer clock Clock is encoded in the Ethernet carrier and recovered by the receiver chip (PHY). Erik van der Bij White Rabbit for Time Transfer 12/40

18 DDMTD: Phase tracking PTP limitation: timestamping granularity Solution: take advantage of SyncE and measure phase shift Erik van der Bij White Rabbit for Time Transfer 13/40

19 WR PTP Ongoing standardization effort to include WR into PTPv3 Extension to PTP (IEEE1588) defined as PTP Profile Compatible with standard PTP gear Erik van der Bij White Rabbit for Time Transfer 14/40

20 WR PTP Ongoing standardization effort to include WR into PTPv3 Extension to PTP (IEEE1588) defined as PTP Profile Compatible with standard PTP gear According to ISPCS Plug Fest results White Rabbit is the most accurate PTP implementation in the world! Erik van der Bij White Rabbit for Time Transfer 14/40

21 WR time transfer performance: lab tests Erik van der Bij White Rabbit for Time Transfer 15/40

22 WR time transfer performance: lab tests Erik van der Bij White Rabbit for Time Transfer 15/40

23 WR time transfer performance: temperature tests Erik van der Bij White Rabbit for Time Transfer 16/40

24 Introduction Motivation Time Distribution Data Distribution System Components Applications Openness Data distribution in White Rabbit High accuracy/precision synchronization Deterministic, reliable and low-latency Control Data delivery Erik van der Bij White Rabbit for Time Transfer 17/40 Summary

25 Control Data Two types of data: Control Data (High Priority, HP) Standard Data (Best Effort) Characteristics of Control Data Sent in Control Messages Sent by Data Master(s) Broadcast (one-to-alot) Deterministic and low latency Reliable delivery Erik van der Bij White Rabbit for Time Transfer 18/40

26 Data Redundancy Re-transmission of Control Data not possible Forward Error Correction additional transparent layer: One Control Message encoded into 4 Ethernet frames, Recovery of Control Message from any 2 frames FEC can prevent data loss due to: bit error network reconfiguration Erik van der Bij White Rabbit for Time Transfer 19/40

27 Introduction Motivation Time Distribution Data Distribution System Components Applications Openness Maximum Latency and Redundancy Upper bound latency by design: < 10us for high-priority data On fail of link, reconfigures network automatically while loosing maximum 2 frames Erik van der Bij White Rabbit for Time Transfer 20/40 Summary

28 White Rabbit Network Components Erik van der Bij White Rabbit for Time Transfer 21/40

29 White Rabbit Switch (V3) Central element of WR network Original design optimized for timing, designed from scratch BASE-BX10 ports Capable of driving 10 km of SM fiber Open design (H/W and S/W) Erik van der Bij White Rabbit for Time Transfer 22/40

30 WR Node: WR PTP Core Erik van der Bij White Rabbit for Time Transfer 23/40

31 WR Node: WR PTP Core Erik van der Bij White Rabbit for Time Transfer 23/40

32 WR Node: SPEC board FMC-based Hardware Kit: Carrier boards in PCI-Express, VME, PXIe All carriers are equipped with a White Rabbit port FMCs (FPGA Mezzanine Cards) with ADCs, DACs, TDCs, fine delays, digital I/O Erik van der Bij White Rabbit for Time Transfer 24/40

33 WR Node: SPEC board - PCI Express Erik van der Bij White Rabbit for Time Transfer 25/40

34 WR Node: SVEC board - VME Erik van der Bij White Rabbit for Time Transfer 26/40

35 WR Node: SPEXI board - PXI Express Erik van der Bij White Rabbit for Time Transfer 27/40

36 White Rabbit applications Control and timing system Field bus recommended at CERN Time Transfer RF distribution Distributed oscilloscope... Erik van der Bij White Rabbit for Time Transfer 28/40

37 WR at CERN Erik van der Bij White Rabbit for Time Transfer 29/40

38 Ethernet Clock distribution a.k.a. Distributed DDS Distributed Direct Digital Synthesis Replaces dozens of cables with a single fiber. Works over big distances without degrading signal quality. Can provide various clocks (TTC, RF, bunch clock) with a single, standardized link. May send many atomic clocks over the same fibre? In both directions? Erik van der Bij White Rabbit for Time Transfer 30/40

39 Distributed oscilloscope Common clock in the entire network: no skew between ADCs. Ability to sample with different clocks via Distributed DDS. External triggers can be time tagged with a TDC and used to reconstruct the original time base in the operator s PC. Erik van der Bij White Rabbit for Time Transfer 31/40

40 NL: Time Distribution over 80 km with amplifiers WR link VSL Delft NIKHEF Amsterdam Field trial in SURFnet dark telecom fiber (2 80 km) Collaboration VU-SURFnet-VSL- NIKHEF Aim: sub-ns time transfer of UTC(VSL): Cs clocks + satellite link to TAI Equipment calibrated by VU- TU/e-NIKHEF Needed for optical frequency measurements at VU! Future: useful for VLBI, LOFAR? Erik van der Bij White Rabbit for Time Transfer 32/40

41 NL: Time Distribution over 80 km with amplifiers VSL Delft link VSL-TU Delft SURFnet link NIKHEF Rx WR node 1 Tx Atomic clock Delft WDM time & freq reference UTC(VSL) λ measurement time & freq comparison Rx WR node 2 Tx WDM WDM Leiden λ 2 SOA SOA λ 1 λ 2 SOA SOA WDM λ 1 = 1471 nm λ 2 = 1491 nm WDM λ 1 WDM WDM time & freq reference WDM Amsterdam Tx WR node 3 Rx Atomic clock time & freq comparison Tx WR node 4 Rx Erik van der Bij White Rabbit for Time Transfer 32/40

42 FI: Time Distribution over 1000 km with amplifiers Espoo Kajaani WR time link 1000 km Map: Jani Myyry, CSC 100x longer than WR-spec of 10km! IT Center for Science Ltd Uplink + downlink at THz in FUNET Coriant DWDM network operated by CSC. Not bidir-optics as in WR-spec! GrandMaster node locked to UTC(MIKE) at Espoo Slave node at Kajaani outputs PPS WR-link compared against GPS-PPP solution using Dicom GTR50/51 receivers at both ends of link. Result: <2 ns error over 60 days Erik van der Bij EFTF White 2014 Rabbit June for 25th: Time Time Transfer Transfer 33/40 in a Wide Area

43 Open Hardware Open Hardware advantages: Get a design just the way we want it Peer review Design re-use Healthier relationships with companies Open Hardware Repository ( CERN Open Hardware License (OHL) Erik van der Bij White Rabbit for Time Transfer 34/40

44 Open Hardware Repository Erik van der Bij White Rabbit for Time Transfer 35/40

45 CERN Open Hardware License ohwr.org/cernohl Provides a solid legal basis Open Software licences not usable (GNU, GPL,... ) Developed by Knowledge and Technology Transfer Group Defines conditions of using & modifying licenced material Same principles as Open Software Anyone can see the source (design documentation) Anyone is free to study, modify and share Any modification and distribution under same licence Persistence makes everyone profit from improvements Erik van der Bij White Rabbit for Time Transfer 36/40

46 Open Hardware works ohwr.org Projects 86 active projects 53 initiated by CERN groups, 33 by other institutes Types of designs 75 hardware designs (of which 44 FMC projects) 31 re-usable IP blocks (some included in HW projects) General tools Production test environment (Python based) Image processing Wishbone slave generator, crossbar data from June 2014 Erik van der Bij White Rabbit for Time Transfer 37/40

47 Dispelling the commercial vs open myth Erik van der Bij White Rabbit for Time Transfer 38/40

48 Summary White Rabbit better than 1 ns accuracy, tens of ps precision Tested up to 10 km (and even 1000 km :-) 2000 nodes WR components available from industry PCIe, PXIe, VME, Switch Fully open design: Hardware, Gateware, Software Large community in various fields Proven results Project still under development, open for improvements Ready to have fun with! ohwr.org/projects/white-rabbit Erik van der Bij White Rabbit for Time Transfer 39/40

49 Questions and answers ohwr.org/projects/white-rabbit Erik van der Bij White Rabbit for Time Transfer 40/40