A Unified DAQ Interconnection Network with Precise Time Synchronization

Transcription

1 A Unified DAQ Interconnection Network with Precise Time Synchronization Frank Lemke, David Slogsnat, Niels Burkhardt, Ulrich Bruening Abstract This paper focuses on the interconnection network used as a part of the Data Acquisition System of the Compressed Baryonic Matter experiment at the Facility for Antiproton and Ion Research in Darmstadt, Germany. This experiment will have special demands on the Data Acquisition System like limited space for hardware, radiation tolerance, flexibility for different types of network traffic and support for synchronization mechanisms. The specialty of the CBM network is that it uses only a single bidirectional fiber link for all network abilities and providing a deterministic latency message for precise time synchronisation. This led to the development of a new network and protocol. Index Terms Communication networks, protocols, synchronization, jitter T I. INTRODUCTION HIS paper describes a part of the Data Acquisition (DAQ) system of the Compressed Baryonic Matter (CBM) experiment at the Facility for Antiproton and Ion Research (FAIR) at GSI Darmstadt [1]. The FAIR [2] organization works together with the GSI [3] for constructing and running the planed FAIR facility. This facility will extend the existing GSI Linear Accelerator and the GSI Synchrotron. Main goal of the project is to provide highest beam intensities, high quality beams, high beam energy, high beam power, and enable efficient parallel usage. The planned extension consists of two high-energy superconducting synchrotrons built on top of each other in a subterranean tunnel, also of five collector, cooler, and storage rings and numerous new detectors serving five fields of physics [4]. The accelerator complex will deliver intense ( ) pulsed uranium beams at 2.7 GeV/u for U 28+, intense ( ) pulsed proton beams at 29 GeV, high-energy ion beams of maximum energies around 45 GeV/u for Ne 10+ and close to 35 GeV/u for fully stripped U 92+. The storage Rings, equipped with beam cooling facilities, internal targets, and inring experiments, will provide beams at energies 0.74 GeV/u for U 92+ and for antiproton beams at energies of 3 GeV up to a Manuscript received May 23, 2009; revised January 18, Frank Lemke, David Slogsnat, Niels Burkhardt and Ulrich Bruening are with the Computer Architecture Group at the University of Heidelberg, B6 26, Mannheim, Germany. (phone: ; fax: ; {frank.lemke, david.slogsnat, niels.burkhardt, ulrich.bruening}@ziti.uni-heidelberg.de). Fig. 1. GSI Darmstadt (GSI blue left, FAIR red right) [4] maximum energy of 14 GeV. The FAIR extension is shown in Fig. 1. The CBM experiment will investigate heavy-ion collisions in fixed target geometry, key observables are detection of open charm and light and heavy vector mesons in the dielectron and di-muon channel [1]. The main detectors are the Silicon Tracking System (STS), the Ring Imaging Cherenkov detector (RICH), Transition Radiation Detectors (TRD), the Muon Chamber/absorber system (MUCH), Resistive Plate Chambers (RPC), the Electromagnetic Calorimeter (ECAL), and the Projectile Spectator Detector (PSD) [5]. These CBM detectors have special demands on the DAQ System. Selftriggered front-end electronic (FEE) modules are used to collect data from the detectors. The final event selection will be done after event building in a processor farm. The data flow of the detector will be up to 1TB/s. Special synchronization mechanisms are needed for synchronization of the detectors. Radiation tolerance must be provided by the frontend part of the network. The network protocol must efficiently support four types of communication: clock distribution, time synchronization, control messages, and data streams. Space and budget are limited. The latter led to the usage of commercial off-the-shelf (COTS) parts in particular Field Programmable Gate Arrays (FPGAs). Also we use only a single bidirectional link for the four types of communication, which is implemented as a pair of fibers. This solution is efficient in providing all required features. The fiber solution also overcomes the length restriction and the problem of different voltage level at the detectors. Consideration of all these facts led to the solution for the unified interconnection network with precise time synchronization. This system, which will be used for the CBM DAQ system, is described in the following. In Chapter II the CBM network and DAQ structure is described. Then chapter III, State of the Art, gives an

2 overview of other implementations. Detailed information about the CBM network and protocol are shown in chapter IV. A description of the specific synchronization mechanism follows in chapter V. The prototype system is outlined in chapter VI. Finally a conclusion is given in chapter VII. II. CBM NETWORK AND DAQ STRUCTURE The CBM network is a hierarchical structured network which connects three different kinds of boards: the Readout Controller Board () located close to the FEE, the Data Combiner Board (DCB) in the inner network part and the Active Buffer Board (ABB) at the backend of the network [6]. The main experiment data flow is unidirectional from the detector frontend electronics to the cluster farm at the backend. Control transactions are mostly initiated from the backend and control data is exchanged in both directions. Administration packets are transmitted only on link level and are not routed through the network. A structural overview with indicated message streams is shown in Fig. 2. FEE Frontend Data flow DCB Control Optical Link Deterministic Latency Messages Fig. 2. CBM network structure with message streams ABB Backend The is responsible for interfacing to the network links on one side and connecting to various FEEs of different kinds of detectors. This is not done directly, but by connecting to front-end boards (), which carry the FEE. These s can be of different types. In the current version they are assembled with a varying amount of n-xyter ASIC [7] chips. The n-xyter chips are self triggered and data driven. The completely initializes and controls all its connected s. The measured data from the n-xyters is always collected by a and subsequently send through the links of the network. The network consists of an arbitrary number of hierarchical levels of DCBs. These boards are used for bundling small data packets to larger ones and combining the data. The current version enables the connection of up to 5 s. For later development steps a first filtering of data could be implemented in the DCBs. The DCB further transmits the data to an ABB. This ABB [8] represents the interface to the commodity DAQ cluster farm. During processing the data, an ABB sorts it considering the temporal and special information. Immediately afterwards it is written into the specified local buffers. Multiple of these -DCB- ABB trees are used in parallel for one detector system. The Data Acquisition Backbone Core (DABC) [9] is used as a general purpose software framework for the implementation of the DAQ. III. STATE OF THE ART Networks used for time synchronization must support the broadcast communication in order to distribute the central clock event to all endpoints. All these event messages must arrive at the endpoints after a fixed travelling time. A straightforward solution to this requirement is to use a dedicated network just for time synchronization and make all paths to the endpoints of equal length. Trying to synchronize endpoints in a packet oriented general purpose network like a LAN is difficult due to the variable latencies for message transports. Switching of network paths is the main cause for the variable latencies. Special protocols have been developed to synchronize endpoints as close as possible, e.g. the network time protocol (NTP) [10] which allows time syncs in the range of some microseconds. The precise time protocol (PTP) [11] allows a more precise synchronization. The CBM network requires time synchronization with a resolution in the range of hundreds of picoseconds and this can only be achieved by special hardware support. In a fused network, various traffic classes can interfere with each other and the goal to achieve a deterministic latency for the time synchronization messages is difficult. The solution is to provide a dedicated hardware path through all switching nodes with fixed latency. The arbitrated output path of the switch must have a mechanism for inserting a synchronization message without variable timing. This can be achieved with the Priority Insertion Method where the synchronization event message is inserted in the outgoing stream immediately when it arrives at the send port. A similar mechanism is part of the HyperTransport protocol [12], but not with the intention to provide latency guarantees. Using the fixed latency path to the endpoint and return the event in the same way to the clock source allows to measure the distance to the endpoint and thus allows having variable but fixed distances from clock source to each endpoint. This method is similar to the Cristian algorithm [13]. The LHC experiment at CERN [14] is comparable to FAIR both in complexity as well as in the requirements. In LHC, a dedicated timing, trigger and control (TTC) network is used to distribute the clock and to broadcast trigger and control information over unidirectional fibre links. The topology is a tree, in which 1:32 optical tree couplers manifold the signals. At the leafs of the tree, a VCXO in a PLL reduces RMS jitter of the MHz clock to 7 ps and further distributes the clock electrically. To compensate time-of-flight delays in the detector as well as skew in the TTC network itself, programmable deskew taps with a minimum granularity of 104 ps are used. Data packets are broadcast using a timedivision-multiplexing arbitration.

3 IV. CBM NETWORK AND PROTOCOL The CBM network protocol takes into account that three different functions are integrated into the network. The detector data transport, the detector control and the timing synchronization use the same optical links and lead to three specific traffic classes. The link layer protocol has been specifically designed to support this feature. Furthermore, each traffic class can be handled with a different fault tolerance level. An important aspect of the proposed network protocol is the single error correction, double error detection (SECDED) ability over 8b/10b-coded serial links. In the 8b/10b protocol, single bit errors that occur on the link may cause an error multiplication, generating up to 8 bit errors in the same code word [15]. Plain forward-error correction in the 8 bit domain thus is overly expensive. Previously it has been proposed to insert ECC data between 10b characters. This method is most efficient for larger data blocks. But even for 96 bit ECC-controlled blocks, the bandwidth overhead that is introduced is 17% [16]. Every control character in our protocol has the size of a physical transfer digit (phit), consisting of 16 data bits plus 2 control bits. To avoid naming confusions, these control characters are named control phits in our protocol. The entire set of link-layer control phits, which is for example used to frame packets, is fault tolerant with SECDED capabilities. Optimized packet structures achieve a high link utilization of about 90%. The payload of Data Transport Messages and Detector Control Messages is protected by a for error detection. An automatic hardware retransmission mechanism guarantees the reliability of Detector Control Messages. Time synchronization and various special event signalling are provided by Deterministic Latency Messages. The deterministic latency for these messages is achieved by priority request insertion. A. Traffic Classes Three different traffic classes are provided in order to fit the requirements of the specific message types in the network. Each traffic class is represented by a virtual channel to make these classes independent of each other. This enables quality of service features. In particular, the classes have different priorities to access the physical link. The following traffic classes are supported: Deterministic Latency Messages (DLM) Data Transport Messages (DTM) Detector Control Messages (DCM) The link layer has a speciality build in for the time synchronization of large networks, the Deterministic Latency Messages (DLM). This type of message has a fixed length with a packet size of only one phit. They are more a special control character than a message because they carry no real payload. Instead they carry information that is directly encoded in the control characters. DLMs, as their name indicates, must always have a deterministic latency in the entire network. Therefore one of the most important features is priority request insertion, this guarantees an insertion of DLMs at any given time into the link even during data or control packets are send. Also, latency within the FPGA, and in particular within the link ports, must be constant and known beforehand. Therefore, DLMs take a dedicated data path within the FPGA that provides these features. Also the network physical layer has to provide a deterministic latency. This must be ensured at the initial restart of the system, but as well at any time later if components should be reinitialized or reset. The accumulated round trip time for DLMs must always have the same total number of clock cycles in the network. Therefore the used FPGAs have to provide a serializer/deserializer (SerDes) module with the possibility to achieve a deterministic latency. The used Xilinx FPGAs do support this only with a special configuration and an additional algorithm to guarantee the right latency setup. With such an implementation, DLMs are an ideal vehicle for synchronization or time critical service purposes message signalling. For the traffic class of Data Transport Messages (DTM) the most important ability is high bandwidth for streaming the detector data. This requirement is reached by the usage of an optimized packet structure. With a payload of up to 64 bytes, and an overhead of 6 bytes, this leads to a maximum link utilization for data of data/(data+framing+crc) = 64/70 = % (64/70*8/10 = % with 8b/10b coding). Data Transport Messages can have any fault tolerance level, but currently data correction is not implemented, only error detection and the extraction of faulty packets is implemented. Erroneous packets become marked and continue their way through the network, this enables the analysis of problems within the network. In case error correction should be needed in the future, a link-level retransmission as it exists for DCMs may be added. In contrast to DTMs, DCMs as the third message class need a high level of fault tolerance. Therefore they are protected with s and automatically retransmitted in case of an error. Service packets like acknowledgements or idles are handled by the link port and are not visible to the core module. They are also coded as special characters to provide fault tolerance. The credit based flow control for DCMs is handled internally in the link port modules, while DTMs are using a stopcontinue protocol. DLMs are not subject to this, as the feature of deterministic latency does not allow for backward flow control. The receiver of a DLM must always be able to process it. B. Packet Format Phits of 16 data bits plus 2 control bits are the smallest data unit in the network. For the serial transmission on the fiber, each phit is translated into two 10b code words. A phit may either be data payload of a packet, or a control character. Packet structures are shown in Fig. 3. DTM and DCM packets may have a payload of 8 to 64 bytes. Every packet is framed with start and end phits and a 16 bit. Although theoretically, no framing of a packet would be required, the start-of-a-packet and the end-of-packet characters provide

4 additional security at little cost and also avoid carrying a length field in the header which must be interpreted to detect the end of a packet. The first phit of a packet is a start-of-packet () control character for DTMs and start-of-special-control (SOSC) for DCMs. End-of-packet () control characters mark the end. To distinguish virtual channels, there are different start characters for the different virtual channels. DLMs are somehow different from this concept, as they do not contain payload information. Instead, they consist of one control phit. Bits SOSC Fig. 3. Packet formats 8 64 Bytes Bit Control 8 64 Bytes SYNC DLM SYNC SYNC INIT INIT ACK IDLE C. Control Characters Besides start and end characters, a link protocol must support DLMs, credits and acknowledgements. Additionally, idle, retransmission and management characters exist. As the number of 8b/10b K characters is by far not sufficient to encode these control characters, a combination of K and D characters is used [15]. The first character of a control phit is one of K.28.7, K.27.7, K.28.3, which have a hamming distance HD>=3. The second character is one of a set of D characters with HD>=4. There is no closed formula to generate such a set of D characters. Instead, the set has been found by a brute-force search through the set of D characters. Decoding with error correction and detection are thus performed by a lookup-table based approach. K and D characters are combined in such a way that all single bit errors in a phit can be corrected and all double bit errors can be detected, thus providing SECDED capabilities [17]. D. Error Correction & Detection Single bit errors in control phits can be corrected using the SECDED mechanisms described above. Assuming that there are no hardware defects, double bit errors within the same control phit are extremely rare. Loss of data in such a case is acceptable. Nevertheless, it is important to detect such an error in order to prevent inconsistent system states. In the case of detected double or multi bit errors in control characters the link will thus be reinitialized. Besides this protection of the protocol s control information, another important topic is how payload information is protected. For both the payloads of DTMs and DCMs, the receiver can detect errors by checking the packets. It is acceptable to have data loss for DTMs, these will simple be discarded if an error has been detected. For DCMs, such a data loss is not acceptable. DCMs are protected by a retransmission-based protocol. The granularity of retransmission is a packet. All packets that are transmitted over the link are also copied to a retransmission buffer on the sender side. If the receiving part of the receiver side positively acknowledges the reception of a packet, the buffer space can be freed. If the acknowledgement is a negative acknowledgement, the sending part will instead initiate a retransmission of all packets in the retransmission buffer. As the retransmission occurs on link-level and the latency of link transmission and processing is low, a hardware-implemented retransmission FIFO has a low FPGA resources usage. E. Link Initialization The goal of the link initialisation is to ensure a proper link start-up after system power up, reset, or cable hot plug. The link initialisation checks the availability of the link cable and tries to establish a bidirectional link channel. During link initialization, each link port starts sending a predefined pattern. This allows the Xilinx multi-gigabit transceivers (MGT) on both sides of the link to synchronize on the stream, to recover the clock and to properly de-serialize the incoming data. If the MGTs can recover a stable clock, this is the first indication that there is actually an active sender on the other side of the link. The link port will then continue to send the pattern, and will also check the incoming data for correctness. After a defined period of receiving correct initialization data, a link port will try to do a handshake with the other side. If the other side did not detect an error either, the handshake succeeds and the link switches to operation mode. Otherwise, initialization is restarted again. All CBM network modules are implemented in Verilog HDL. Their control interface is optimized for adaption in all parts of the network and delivers easy to use valid-stop synchronization for all traffic classes except for DLMs. All low level link information, which is not needed by the upper layers, is intentionally non transparent through the interface. V. SYNCHRONIZATION MECHANISM The most important ability to enable synchronization over fiber links is to guarantee a deterministic latency over the links. Therefore the FPGA s serial transceivers have to be configured in a special way for always starting with the same basic latency after link initialization. The control logic implementation must also allow a deterministic insertion of synchronization signals or characters into the link. For Xilinx FPGAs, we use the Multi-Gigabit Transceivers (MGTs), as well as the GTP transceivers. Different control logic parts of the MGTs and GTPs have to be configured to guarantee a reproducible deterministic behaviour. The basic configuration is derived from the MGT user guide [18] and is adapted with the following steps. The wrapper generated by the RocketIO Wizard had to be modified to have access to the

5 barrel shifter position signal RXLOSSOFSYNC of the alignment block to read out the deserialized data. The fabric interface had to be chosen with the width of 32 bit to guarantee deterministic behaviour at the interface. This is required because the internal MGT width is 32 bit and multiplexing must be avoided. Also the derived parallel clock generated by the clock data recovery (CDR) can lead to different barrel shifter positions. To obtain deterministic latency the CDR can be reset for altering the barrel shifter position of alignment until the predefined value is reached. This is achieved by restarting the low level initialization of the GT11_INIT_RX module. As the phase relation between the various local clock oscillators is not fixed, the resulting barrel shifter position varies with each reset and the right alignment will always be achieved after a number of resets. Through this configuration method a reproducible and deterministic latency is achieved after link initialization when the same bit file is used for all FPGAs. Another very important fact which has to be considered is the usage of an identical clock in all parts of the network. Therefore the clock must be derived from a single oscillator and the system must provide a SerDes bit precise deterministic latency. This is needed to achieve the same deterministic latency after restarting the network or parts of it. Therefore at least the send clocks of the transmitters in the network must be identical and the deterministic logic parts need to be source synchronous. A system clock to ensure this could be provided to all parts by a separate clock distribution net or through a clock recovery for reusing the input clock in the transmit path. We decided against the overhead of a separate network, but for a clock recovery mechanism. The clock recovery mechanism possible within the Xilinx FPGAs of the CBM network delivers a peak-to-peak jitter between 80ps and 100ps. This jitter is not precise enough to be used as SerDes reference clock, because it does not fulfil the MGT specification [19] maximum value of 40ps. The solution is to use an additional jitter cleaner device to clean the received clock for CDR and feed it back into the FPGA as reference clock for the transmit path. Fig. 4 shows the clock distribution scheme used to spread the clock throughout the planned system. From one arbitrary but fixed ABB the source clock is provided to all others. Thereby most logic parts run with a recovered clock. To keep costs as low as possible, the idea was to use only COTS parts for the jitter cleaning. It should not only be usable for jitter cleaning, but also as a stand-alone clock source. A standard mezzanine connector is used to plug it onto system boards. This connector delivers the default recovered clock from the FPGA and all configuration signals for the IC at the extension board. The optional clock source is a Silicon Labs SI570, a via I2C configurable oscillator with low jitter performance. FEE FEE Fig. 4. Clock distribution using recovered jitter cleaned clock for transmission Due to our bad experiences regarding jitter quality of clock switching within the FPGA, we use a separate clock switching IC. A Texas Instruments CDCLVD110 is used to select between the onboard clock and the recovered clock. The chosen clock is then cleaned or configured within the National Semiconductor LMK03000 chip. The clock jitter of the cleaned clock for peak-to-peak is below 40ps, which is enough for usage as reference clock for the MGTs and GTPs. The magnitude of the measured RMS jitter is below 10ps. Fig. 5 shows a jitter measurement with a cleaned clock of the jitter cleaner board. The jitter cleaned clock has been used as transmit clock within a test setup at our lab and passed all requirement of the MGTs. Fig. 5. Jitter cleaned clock DCB DCB Recovered clock ABB ABB Clock distribution The cleaned recovered clock together with the user-defined MGT/GPT configurations for achieving a fix initialization state guarantees the deterministic behaviour of the link. Therefore the latency for each link is measured during link initialization and DLMs are periodically sent to synchronize an epoch marker. As every other control phit, DLMs are coded with a 1 bit forward error correction. After initialization of the epoch counter in the s, arriving DLMs are only used to check that the detector is still synchronous. After the Clock distribution Clock source

6 loss of several DLMs it is assumed that an error occurred and a resynchronization starts. Thus even a complete DLM could get lost in the network without influencing the functionality. Additionally there are DLMs available for user defined special event signalling with deterministic latency. This could be a trigger mechanism for certain detector readout features or a special control implementation. In order to guarantee the deterministic latency of DLMs, another requirement is a structurally separated hardware implementation with a fixed latency within the FPGAs. In addition the insertion into the link message stream with fixed latency is necessary. This could happen within a deterministic time multiplexing scheme or with a guaranteed latency for an insertion at any time. The later solution is more flexible and the priority request insertion was chosen for the implementation. This enables the possible insertions shown in Fig. 6, between two packets, within message parts or even right into the data payload of a packet. As DLMs are infrequent events, we were able to simplify hardware structures by allowing a maximum of 1 DLM within a packet. This leads to a minimum interval size between DLMs of maximum message size/hardware processing speed = 70 bytes / 2 bytes per clock cycle = 35 clock cycles, which every sender of DLMs has to observe. DLM hotplug support for the optical links. Communication with DCMs works without errors. With the means of error injection error detection and retransmission was tested. All in all, the functionality of all components could be verified. Test stimulus for DTMs was generated in dedicated test pattern generators in the s. The theoretical peak bandwidth utilization for our protocol is 8b/10b coding loss*maximum data payload/maximum data packet size*link speed = 8/10*62/68*312.5 MB/s = MB/s, this is 96.87% of the maximal possible bandwidth of 250 MB/s with 8b/10b coding on a 2.5Gb/s link. Measurements for data bandwidth utilization show that we reach MB/s, which is about 92.72% of the possible bandwidth. Due to current bandwidth estimations, the optical links are designed as a 2.5 Gb/s link, but if necessary they could be adapted to 5Gb/s or 6.24Gb/s. This has already been evaluated with test implementations in our hardware labs. In numerous lab tests with our generated bitfiles including all implementation features described in this paper we always reach the same deterministic latency when using identical data file versions. This means that components may be reset in the system without the need to measure the link latency again, which improves the ease of use of the system. Bits 8 64 Bytes 8 64 Bytes DLM DLM 8 64 Bytes Fig. 6. DLM insertion examples VI. THE PROTOTYPE SETUPS The used prototype build-ups consist of DCB V1.3, a successor of the HTX-Board [20], Avnet ADSAES-XLX- V5LXT as ABB and V2 boards, which are also called SysCore Boards [21]. The preferred combinations for our tests were ABB - 2s and ABB - DCB - 2, as this is sufficient to prove that our concepts are working in reality. The proposed clock recovery mechanism was used in all parts of the system to clock the main logic except for the source system. Jitter cleaning has only been used on DCBs, which was adequate to prove deterministic latency into detector direction when fixed length optical cables were used. This was also sufficient to demonstrate jitter cleaner functionality and performance. In the future jitter cleaning will be available in all parts. This enables automatic measurements of deterministic latencies by ping-pong messages and thereby variable but fixed latencies in the systems. The tests showed that the automatic low level and high level initialization of optical links works reliable, including Fig. 7. Time difference measurement at two receiving s The deterministic latency of links has been proven in several long time tests of up to 168 hours without errors. At random times, we measured peak-to-peak clock jitter, which was below 40 ps. MGTs did not lose their link locks during the long time tests, while they are very sensitive to jitter. Synchronization accuracy between two s is by construction one bit clock cycle with the used link speed of 2.5 GHz, i.e. 400ps or better. This means that the difference between two different s is never greater than this, and also that the jitter between clock source and is never greater. We measured the phase difference in practice. Fig. 7 shows the 250MHz clocked signals of two boards that signal the reception of a sent DLM. As the cables of the measurement probes had the same length, the pulses occur at the same time. We observed a variation of up to 576ps. It can be assumed that this larger number is due to the delay of the

7 CMOS outputs of the FPGA that we used for measuring. A measurement using low-voltage differential signaling (LVDS) would deliver more precise data. This has not yet been performed due to the lack of differential measurement equipment. Also, the measured clock variation is within the requirements. VII. CONCLUSIONS Our experiences with networks in the area of cluster computing led us to the development of this next generation DAQ network. We presented the protocol and implementations of a fully functional prototype. The network combines three different functions onto the same network fabric, while past solutions often used different networks for these features. Our unified solution is expected to save costs and manpower in particular during the build-up phase. Deterministic Latency Messaging together with lowjitter and synchronous clock recovery allow for precise time synchronization in this self-triggered detector system. With basic measurements, we could prove that the synchronization accuracy is below 576ps, and expect the real accuracy to be lower than 400ps. The network protocol is scalable, as an example additional virtual channels may be added in the future, or link speeds may be increased. Also, error correction for data packets may be added. This allows for reuse in future detector systems with minimal changes. At the same time, an effective link bandwidth of up to 73% is reached with 8/10b coding. Further improvements can be achieved by using scrambling as a promising method. Self-synchronous scramblers do not multiplicate errors. They also could overcome the 20% bandwidth overhead that 8b/10b coding introduces. However, scrambling is worse when it comes to DC-balance and maximum running lengths. Without full control and knowledge over the FPGA serializers, such an approach is currently a game of luck. The new concept of the network has been implemented, verified by simulation and tested in system prototypes. The functionality and usability of all developed concepts is proven in hardware during numerous lab tests. Also, other groups that participate in this project successfully tested our design. The next step in development will be the construction of a larger demonstrator. Exact measurements of the minimal achievable clock jitter for logic within s using the jitter cleaned clock have to be done. Also a setup for long time usage during everyday s lab tests in one of the lab detector test systems at GSI in Darmstadt will be installed. REFERENCES [1] H. H. Gutbrod, I. Augustin, H. Eickhoff, K-D. Gross, W. F. Henning, D. Kraemer, and G. Walter, FAIR Baseline Technical Report Executive Summary, Gesellschaft fuer Schwerionenforschung mbh (GSI), Member of the Hermann- von- Helmholtz- Association of National Research Centres (HGF), Sept [2] The FAIR website, [Online]. Available: [3] The GSI website, [Online]. Available: [4] D. Cleary, A Lab to Get the Measure of Matter, Science, Vol. 318, pp , Nov. 2, [5] The CBM website, [Online]. Available: [6] N. Abel, F. Lemke, and W. Gao, Design and implementation of a hierarchical DAQ network, presented at the Deutsche Physikalische Gesellschaft e.v. Fruehjahrstagung, March 10, [7] C. J. Schmidt, U. Trunk, H. K. Soltveit, A. Brogna, S. Buzzetti, W. Dabrowski, T. Fiutowski, G. Modzel, K. Solvag, R. Szcygieland, and P. Wiacek, Test results on the n-xyter ASIC, a self triggered, sparcifying readout ASIC, presented at the Topical Workshop on Electronics for Particle Physics 2007, Prague, Czech Republic, TWEPP- 07, Sept. 3-7, [8] W. Gao, A. Kugel, A. Wurz, G. Marcus, and R. Maenner, Active Buffer Design for Event Building in CBM Experiment, presented at the 16th IEEE NPSS Real Time Conference RT09, Beijing, China, May [9] J. Adamczewski-Musch, H. G. Essel, N. Kurz, and S. Linev, First Release of Data Acquisition Backbone Core (DABC), presented at the 16th IEEE NPSS Real Time Conference RT09, Beijing, China, May [10] D. L. Mills, Network Time Protocol Version 4 Reference and Implementation Guide, NTP Working Group, Technical Report , University of Delaware, June [11] J. C. Eidson and J. Tengdin, IEEE-1588 Standard for a precision clock synchronization protocol for networked measurement and control systems and applications to the power industry, presented at Distributech 2003, Las Vegas, USA, Feb [12] The HyperTransport 3.1 specification, HyperTransport Consortium, August 18, [13] F. Cristian, Probabilistic clock synchronization, in Distributed Computing, Springer Berlin / Heidelberg, June [14] B. Taylor, Timing Distribution at the LHC, presented at the 8th Workshop on Electronics for LHC Experiments, Colmar, France, Sept. 9-13, [15] D. Slogsnat, Tightly-Coupled and Fault-Tolerant Communication in Parallel Systems, Dissertation, Universitaet Mannheim, Aug. 4, [16] D. McMahon, A. Kirby, B. Schofield, and K. Springer, Data and forward error control coding techniques for digital signals, US Patent Number , [17] S. Lin and D. Costello, Error Control Coding,, 2. ed., international ed., Pearson/Prentice Hall, Upper Saddle River, NJ, [18] Virtex-4 RocketIO Multi-Gigabit Transceiver, UG076 (v4.1), Xilinx Corporation, San Jose, USA. [19] Virtex-4 FPGA Data Sheet: DC and Switching Characteristics, DS302 (v3.4), Xilinx Corporation, San Jose, USA. [20] H. Fröning, M. Nüssle, D. Slogsnat, H. Litz, and U. Brüning, HTX- Board: A Rapid Prototyping Station, presented at the 3rd annual FPGAworld Conference, Stockholm, Sweden, Nov. 16, [21] N. Abel, J. Adamczewski-Musch, H.G. Essel, U. Kebschull, S.Linev, and S. Mueller-Klieser, Software development for CBM readout controller board, in CBM Progress Report 2008, CBM Collaboration, Darmstadt, Germany, March 2, ACKNOWLEDGMENT We gratefully acknowledge the following person for his outstanding support: Walter F.J. Mueller, GSI Helmholzzentrum fuer Schwerionenforschung GmbH. We also want to thank Sven Schenk from the Computer Architecture Group for bringing his experiences with Xilinx and Altera serializers into this project.