NUCLEAIRE EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH CERN LIBRARIES, GENEVA CERN-ECP

Transcription

1 DK) ORGANISATION EUROPEAN POUR LA RECHERCHE NUCLEAIRE EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH CERN/ECP September 1996 CERN LIBRARIES, GENEVA CERN-ECP Testing HIPPI Switch Configurations for Event Building Applications Arie Van Praag, Ralf Spiwoks, Robert van der Vlugt, Abstract Current plans for Atlas Event Building are centered around using high performance switching technology composed of either one switch or a network of switches. The bandwidth on this level is calculated to be from 1 to 10 GBytes/s with a channel speed to the processor farm ranging from 10 to 100 MBytes/s. The High Performance Parallel Interface (l HPPI ), a well established technology with cheap interfaces and fast switches and a data rate of 100 MBytes/s, is a serious candidate to be used in the event building. Extensive tests have been performed with Data Sources running at 40 MBytes/s. However few tests have been done with HIPPI switches, near the 100 1\/[Bytes/s HIPPI specification. The following series of tests will give a more detailed view of switch behavior in an event building function. Up to the limits of available material, measurements will be done with a single and a double switch configuration. Presented at the SOZOPOL-96 workshop on Relativistic Nuclear Physics, Sozopol, Bulgaria, 30 September - 6 October 1996 OCR Output

2 analysis. The Atlas architecture [1] uses three levels (LVL], mum Latency RATE [Hz] LVL2 and LVL3) as shown in DATA RATE 10-1 o Fig: l. At LVLI, special purpose pipeline processors act on reduced LV'-1 memory ~2 usec granularity data, obtained from a 10*-10 selected subset of the detectors, GB/s to define the so called "Regions derandomizing Of Interest" (ROI). The LVL2 memory trigger uses full granularity and Mux Mux uux muitipiexers full precision data from most of digital buffer the detectors, but only examines LVL2 memories ~1 10 msec the data identified as a ROI by LVL3 Abstract Testing HIPPI Switch Configurations for Event Building applications Arie Van Praag, Ralf Spiwoks, Robert van der Vlugt, CERN Current plans for Atlas Event Building are centered around using high performance switching technology composed of either one switch or a network of switches. The bandwidth on this level is calculated to be from l to 10 GBytes/s with a channel speed to the processor farm ranging from 10 to 100 MBytes/s. The High Performance Parallel Interface (HIPPI ), a well established technology with cheap interfaces and fast switches and a data rate of 100 MBytes/s, is a serious candidate to be used in the event building. Extensive tests have been performed with Data Sources running at 40 MBytes/s. However few tests have been done with HIPPI switches, near the 100 MBytes/s I-HPPI specification. The following series of tests will give a more detailed view of switch behavior in an event building function. Up to the limits of available material, measurements will be done with a single and a double switch configuration. Introduction The coming LHC (Large Hadron Collider) is planned to produce a bunchcrossing in the detectors every 25 nsec. corresponding to a frequency of 40 MHz. The generated data rates reach values in the order of TBytes/s. As only a fraction of the obtained data will be of interest, filters with several levels of triggers are foreseen, so that only the selected data will be stored for nea 0ur/eveurnuitninc lswitch INTF I I 'NTF I IINTF Ilnterlace 1 O' MB/s LVL1. From LVL2 the data of a single event goes via many parallel channels to LVL3, where the full data is examined, selected and recorded for further Zr $= r analysis. The Atlas data " ~ Ma/s acquisition architecture uses Data Storage large data switches to build the event in a single processor of the Fig: 1. The atlas DAQ architecture LVL3 processor farm. To fulf`1ll the requirements for this part of the data processing a number of promising technologies are now appearing on the market. One of those which is available and fullfills all the conditions, including for the OCR Output

3 switches, is HIPPI. [2.3]. Thcsc switchcs cxist in sizcs up to 32 X 32 channels, with 128 X 128 channels announced for the near future. Switch Latency is generally under 1 usec. The bandwidth is 100 MBytes/s for each channel. Tests with HIPPI for event building have been done with data generators of up to 40 MfBytes/s using 3 channels and a switch [4]. Switch behavior at higher speeds will be verified in these series of tests, including the combination of two coupled switches. The obtained results will be compared with simulated data. Test Methods and Materials The test set up is a simplified application of the event builder (Fig: 2.) The limits are determined by the size of the available switches and by the number of HIPPI data sources that are capable to generate data rates between respected The block size of the data that represents the sub-event must _ c giiisl _ I.-1 HSI 80 MBytes/s and 100 MBytes/s. The standard HIPPI M "? TW ; uq ; s, flow control is at E*h m * indifferent the same time "6g " j S Fig: 2. Test Configuration be selectable from 0.5 KBytes to l MByte. The sub-event size will be incremented in a sequence. As sources Slate 2 will be used, which is a preloaded memory with output sequencer. [5][6], and for the Destinations Neddi's ( Never EnDing Destination Interface ) and HIPPI test equipment. Slate as Data Source From a size of 1 MByte sub-events down to 5 KBytes, the Slate data rate is higher than 90 MBytes/s. Sub-events with sizes between 5 KBytes and 2 KBytes use Short Bursts. This introduces a start-up overhead for each Request cycle that slows down the data rate. The sub-events Event Data into Switch Switch Output Sorted Events VME p _ mmo, gl I-lgl +6 Slam Dm g m Destinations or Neaamana one VME P SS sized from 1 KByte to 0.5 KBytes are fixed data blocks. The only limiting factor is the start-up overhead. A synchronous master-slave connection, where one master starts all other slaves in parallel by a hardware connection, is used to start the data stream. Fig: 3. Slate Data Stream The data set for each Slate is a string emulating sub-events equal to the number of Destinations (Fig:3). Each of the Destinations is addressed using the HIPPI "Logical addressing mode", the "camp on" option is set to avoid time-outs. OCR Output

4 The destinations The Destinations that replace the inputs of a processor farm should be able to sustain a data rate close to 100 MBytes/s. At least one Destination must be able to test data integrity. Test points for measurements need to be present on a number of Destinations. For those without data checking NEDDI's can be used. These devices are able to handle the HIPPI handshake; the HIPPI data lines however are not connected. The complete device is build in a standard HIPPI connector. For the control of data integrity HIPPI test equipment can be used (Table 1). The latter have test points and data verification. Both are capable to support a 100 MBytes/s data rate. In a later stage a VMEbus processor or PCI bus workstation [8] will be connected to at least one of the channels. Switches The HIPPI switches are standard commercially available products made by two different manufacturers. The maximum configurations needed for these tests are 6 X 6 channels and two times 5 X 5 channels. Plug-in modules can be exchanged to adapt to these configurations. The data rate per channel is 100 MBytes/s and switch latency is shorter than 1 usec. An overview of the equipment available is given in Table l. One type of switch has the possibility to program up to 8 routes for the same Logical Address, wich, is very usefull to couple two or more switches. Source Destination Switches Tvpe No I Tvpe No I Type Size Multichannel Slate 2 6 I NEDDI 4 I Avaika 8X 8 Yes CERN HIPPI Testbox 2 I Avaika 3 X 3 Yes Avaika (IOSC) Tester 1 I Essential 4 X 4 No Motorola VME Processor: Processors Digital Alpha 500 Maverick + PCI-HIPPI 90 MBytes/s. [8] Table 1 max. throughput 40 MBvtes/s. Test Configurations From the many possibilities to test switches some are of particular interest. A single l ' li 1 l %II I5Il'1 I "T"tI " fi "Ifl I" I"F I" I fii'i%ii"i I OCR Output Fig. 4: Test Configurations

5

6

7 . n more OCR OutputOCR Outputby diffcrcnt mcasurcmcnts outside CERN [l 1][l2]. The noted effect is less pronounced with the Avaika switch than with Essential switch. It can be concluded that the latter one has a slightly longer latency. Event building with Single Switches It is shown that with a single switch, event building can be done at speeds as high as 92 MBytes/s/channel. The throughput with 6 Slates as sub fr c 2 gz t= event generators has data-rates over 540 MBytes/s. In both cases = - =i. ~ E F1.. F g: 9. Simulation Swatch data rate versus And with double switches connected in parallel, the HIPPI equipment behaves as it would be expected: event building data rates have been observed of up to 89 MBytes/s/channel which represents a total bandwidth of almost 540 MBytes/s. It has to be said that these values are flattered by the ratio of data generators available to the number of used cross connections. Future Evaluations 1,.. _-.t:.,,`», ~,e 4,.2.E.: EE }_,. -..».:i» i»...=- l --- = z -*` -- -:= Measured Switch data rate gi the limit is not HIPPI but the Slate data generators. As the correspondence with the t computer models is satisfactory (Fig: 9.) further complex simulations are foreseen to define precisely the detector architecture at the LVL3 level. In the case that 2 switches are These tests show that HIPPI is successful in event building at very high data rates; however more experiments need to be done to show that it can easily be adapted in large High Energy Physics detectors. Fast HIPPI interfaces for the PCI bus exist, the same modules with a PMC form factor become available. It is important to continue these tests in a VMEbus environment where one or more channels drive sub-events into a switch using PMC HIPPI sources. After the switch PMC HIPPI Destinations deliver the built event into a host memory. If the speed of the PCI bus in the VMEbus units is insufficient, an interim solution is possible with the same interfaces as PCI modules and a workstation as host. Acknowledgment The authors wish to thank Gigalabs (formerly Avaika Networks Corporation) and Essential Communications for the switches they made available to do this tests. Thanks go to Robert McLaren of CERN for the many useful discussions during the preparation of these tests. Thanks also to all those who have made their Slate modules available to us for these tests. OCR Output

8

9 OCR Output 50 us/div.2 msldiv Event 1 Event 1 Event 2 Event 2 Event 3 Event 5 OP 1: A single Round, One Switch 3X3, 5 KByte OP 2: Start-up Cycle, One Switch, SX5, 10 KByte 2 ms/div 20 us/div Event 1 Event 1 Event 2 Event 2 Event 5 Event 6 OP 3: Accumulated Slate overheads produce gaps, One Switch, 5X5, 50 KByte OP 4: Two Switches BXB, 1 Interconnect, 1 KByte.2 ms/div 50 us/div Event 1 Event 1 Event 2 Event 2 Event 6 Event 6 OP 5:A Single Round, Two Switches, SX6, 2 Interconnects, 10KByte, 6X6 OP 6: Start-up Cycle, Two Switches, BXB, 2 Interconnects, 1 KByte 50 us/div 20 us/div Event 1 Event 1 Event 2 Event 2 Event 6 Event 6 OP 7: Data Stream, Two Switches GX6, OP 8: Very Short Sub Event, Two 2 Interconnects, 10 Kbyte Switches, BXB, 2 Interconnects, 64 Bytes OCR Output

10

11

12

13