WR Project Status 1/ 1 The White Rabbit Project Technical introduction and status report T. W lostowski BE-CO Hardware and Timing section CERN November 11, 2010
WR Project Status 2/ 1 Introduction Outline
WR Project Status 3/ 1 Introduction Origin of the name Oh dear! Oh dear! I shall be too late! The White Rabbit in charge of real time
WR Project Status 4/ 1 Introduction Development model Developed in the frame of ACCOR project Open source design done in collaboration with industry Commercial production and support
WR Project Status 5/ 1 Introduction Why White Rabbit? General Machine Timing limitations Low speed (500 kbps) Lack of bidirectional communication Complicated maintenance
WR Project Status 6/ 1 Introduction What is White Rabbit?
WR Project Status 7/ 1 Introduction What is White Rabbit? An extension to Ethernet which provides: Synchronous mode (Sync-E) - common clock for physical layer in entire network, allowing for precise time and frequency transfer. Deterministic routing latency - a guarantee that packet transmission delay between two stations will never exceed a certain boundary.
WR Project Status 8/ 1 Introduction Design goals Scalability Up to 2000 nodes. Range 10 km fiber links. Precision 1 ns time synchronization accuracy, 20 ps jitter.
WR Project Status 9/ 1 Technology overview Outline
WR Project Status 10/ 1 Technology overview Technologies used in White Rabbit Sub-nanosecond synchonization in WR is achieved by using the following three technologies together: Precision Time Protocol (IEEE1588) Synchronous Ethernet DMTD phase tracking
WR Project Status 11/ 1 Technology overview Network topology
WR Project Status 12/ 1 Technology overview Precision Time Protocol (IEEE1588) PTP Protocol (IEEE1588) PTP Synchronizes local clock with the master clock by measuring and compensating the delay introduced by the link. Packet timestamping Link delay is measured by exchanging packets with precise hardware transmit/receipt timestamps
WR Project Status 13/ 1 Technology overview Precision Time Protocol (IEEE1588) PTP Protocol (IEEE1588) Master time scale t 1 t 4 Slave time scale t 2 t 3 Having values of t 1...t 4, slave can: calculate one-way link delay: δ ms = (t 4 t 1 ) (t 3 t 2 ) 2 syntonize its clock rate with the master by tracking the value of t 2 t 1 compute clock offset: offset = t 2 t 1 + δ ms
WR Project Status 14/ 1 Technology overview Precision Time Protocol (IEEE1588) Disadvantages of traditional PTP All nodes have free-running oscillators Frequency drift has to be continously compensated, causing lots of network traffic That doesn t go well with determinism...
WR Project Status 15/ 1 Technology overview Synchronous Ethernet Synchronous Ethernet Common clock for the entire network All network nodes use the same physical layer clock, generated by the System Timing Master Clock is encoded in the Ethernet carrier and recovered by the receiver chip (PHY). PTP is used only for compensating clock offset. Having the same clock frequency everywhere enables phase detector technology as the means of measuring time.
WR Project Status 16/ 1 Technology overview Synchronous Ethernet Synchronous Ethernet
WR Project Status 17/ 1 Technology overview Phase tracking Phase tracking Plain PTP PTP alone is not enough if we want very good accuracy, because of the granularity of the timestamps. Solution Measure the phase shift between transmit and receive clock on the master side, taking the advantage of Synchronous Ethernet.
WR Project Status 18/ 1 Technology overview Phase tracking Phase tracking Master reference clock m DMTD phase detector recovered clock Transmitter Receiver with CDR PLL Link (up to 10 km single mode fiber) clock loopback Slave Receiver with CDR PLL Transmitter uncompensated clock DMTD-based phase shifting PLL recovered in-phase clock Monitor phase of bounced-back clock continuously Phase-locked loop in the slave follows the phase changes measured by the master
WR Project Status 19/ 1 Technology overview White Rabbit Switch White Rabbit Switch Central element of WR network Fully custom design, designed from scratch 10 1000Base-LX ports, capable of driving 10 km of SM fiber 200 ps synchronization accuracy
WR Project Status 20/ 1 Technology overview White Rabbit Switch White Rabbit Switch Designed in microtca MCH (Management Carrier Hub) format. Multi-PCB design: base board with main big FPGA and CPU and Clocking Mezzanine, which handles the timing. Can work in standalone mode (without a microtca crate) via mini-backplane.
32 8 32 WR Project Status 21/ 1 Technology overview White Rabbit Switch Switch block diagram - main part Front Panel External management port Ext PPS in SMA Ext PPS out SMA 64 MB SDRAM 256 MB r/w NAND Flash (main partition) 8 MB r/o DataFlash (failsafe partition) 10/100Mbps PHY + magnetics To FPGA From FPGA EBI0 EBI1 AT91SAM9263 CPU GPIO SPI0 GPIO/SSP0 MII UART SPI1 GPIO GPIO 32-bit bus with IRQs/DMA (from clocking mezzanine) DMTDCLK_MAIN CMI interface CPU interface internal network controller JTAG Config Clock Mezzanine Inteface (CMI) Main FPGA Altera EP3C120 Switch Management interface (SMI) 512 kb ZBT SSRAM WR switch 2x uplink PHYs (to clocking mezzanine) 8x downlink PHYs to backplane I/F section Serial console Level translator 8x SMI links LEDs From CPU From FPGA REFCLK_MAIN (from clocking mezzanine) 1:10 LVTTL fanout FPGA_REFCLK REFCLK_DPx (for downlink PHYs) ZBT_CLK resets main/failsafe flash select Sensors Atmega128 watchdog & IPMB hub IPMB-A,B (2x) IPMB-L (8x) utca-specific System FPGA handles all packet processing CPU implements PTP stack and management functions (SNMP, Spanning Tree)
WR Project Status 22/ 1 Applications Outline
WR Project Status 23/ 1 Applications Possible applications of White Rabbit Large-scale data acquisition systems Precise time tagging Clock & trigger distribution Robust event delivery
WR Project Status 24/ 1 Applications WR in Co-HT s Hardware Kit WR in Co-HT s Hardware Kit ADC FMCs DAC Time-to-Digtal Fine delay Carrier board FMC connector Memory FPGA WR port Power Co-HT FMC-based Hardware Kit: FMCs (FPGA Mezzanine Cards) with ADCs, DACs, TDCs, fine delays, digital I/O Carrier boards in PCI-Express, VME and utca formats All carriers are equipped with a White Rabbit port
WR Project Status 25/ 1 Applications WR in Co-HT s Hardware Kit Ethernet Clock distribution a.k.a. Distributed DDS time & frequency reference Encoder time & frequency reference Receiver input clock DDS synthesizer Phase detector DDS control words digital filter DDS control WR Network DDS control DDS synthesizer in-phase reconstructed clock Distributed Direct Digital Synthesis Replaces dozens of cables with a single fiber Works over big distances without degrading signal quality Can provide various clocks (TTC, RF, bunch clock) with a single, standarized link
WR Project Status 26/ 1 Applications WR in Co-HT s Hardware Kit Distributed oscilloscope Front-end cards Analog signals ADC WR Network Master Operator's PC Triggers TDC WR Switch Sampling clocks DDS WR Switch WR Switch Common clock in the entire network: no skew between ADCs Ability to sample with different clocks via Distributed DDS External triggers can be time tagged with a TDC and used to reconstruct the original time base in the Operator s PC.
WR Project Status 27/ 1 Planning Outline
WR Project Status 28/ 1 Planning Current status WR Switch development status Switch hardware Working and debugged V2 hardware prototype Tested on 10-km fiber links Interoperates with standard Ethernet gear Switch software Done the Hardware Abstraction Layer and PTP daemon Sub-nanosecond accuracy over PTP has been achieved Verified interoperability with other PTP devices on ISPCS 2010 Plug Fest
WR Project Status 29/ 1 Planning Current status Already achieved... According to ISPCS Plug Fest results...... White Rabbit is the most accurate PTP implementation in the world!
WR Project Status 30/ 1 Planning Development plans for Q4 2010 and 2011 CERN - White Rabbit developers T. Wlostowski HDL development Further improvements in the synchronization performance Co-design of V3 Switch hardware with the industrial partners M. Lipinski Development of fault-proof event delivery protocol PTP stack support HDL development
WR Project Status 31/ 1 Planning Development plans for Q4 2010 and 2011 External contributors GSI Darmstadt Fieldbus protocol development, working on reliability HDL development Alessandro Rubini (GNUDD) Main software developer Device drivers Embedded Linux support
WR Project Status 32/ 1 Planning Development plans for Q4 2010 and 2011 Integrasys Development of switch management software IEEE802.x compatibility testing Seven Solutions Co-design of V3 switch hardware (technical spec available mid-november) Prototype production and testing Elproma (not confirmed yet) Commercial WR OEM modules and time servers
WR Project Status 33/ 1 Planning Development plans for Q4 2010 and 2011 Foreseen milestones WR Switch Full basic functionality of HDL and software: mid-december 2010 V3 prototype: Q3 2011 Commercial product: Q1 2012 WR Ecosystem FMC Carriers: VME version prototype done, PCIe available at the end of November WR timing node: Q2 2012 Mezzanines: Full set of cards available Q4 2011
WR Project Status 34/ 1 Planning Development plans for Q4 2010 and 2011 Summary A data link fulfilling all our needs in synchronization and determinism. A successful collaboration including institutes and companies. Full system available at the middle of 2012