A DAQ Architecture for Agata experiment

Transcription

1 1 A DAQ Architecture for Agata experiment Gaetano Maron INFN Laboratori Nazionali di Legnaro

2 On-line Computing Requirements Outline 2 Event Builder Technologies for the On-line System Run Control and Slow Control Agata Demonstrator Project Organization Off-line Infrastructure

3 Agata Global On-line Computing Requirements Front-end electronic and pre-processing Gbps (1 Gbps x 200) Max 10 Gbps (50 Mbps x 200) Pulse Shape Analysis Event Builder 1.5 x 10 6 SI95(?) (present algorithm) 5 x 10 3 SI95 10 Gbps 1 Gbps SI95 = SpecInt 95 1 SI95 = 10 Cern Unit = 40 MIPS Tracking Storage 3 x 10 5 SI95 (no GLT) 3 x 10 4 SI95 30 khz) GLT = Global Level Trigger

4 Input to the DAQ System 4 PREAM FADC FADC DSP DSP DSP DSP FPGA Mux PC Detector i=1-200 PSA Farm 1 Gbps x MSI95 (now) 50 Mbps x 200

5 Farming 5 Computational nodes Data Servers Commodity PC farms + linux are a well established technology Rack mountable cases allow to fit hundreds PCs in few racks (with blade form factor) about 16 ksi95 per rack in 2004 about 50 ksi95 per rack in 2007 About 150 ksi95 per rack in 2010 Gateways

6 Agata on-line system 6 Front-End F1 F2 F3 F Gbps PSA Farm R1 R2 R3 R200 Event Builder Builder Network 10 Gbps HPCC builder Tracking Farm Data Servers B1 B2 B20 ds1 ds2 ds3 ds4 Storage (1000 TB) 10 Gbps 1 Gbps 1 Gbps 100 Mbps > 1 Gbps

7 Event Building: simple case 7 Clock R1 R2 R3 R200 PSA Farms Time Slot 1 Time Slot 2 Time Slot 3 T01 T02 T03 T04 T05 T06 T07 T08 T09 T10 T11 T12 T01 T02 T05 T06 T07 T09 T01 T02 T04 T05 T06 T09 T10 T12 T01 T03 T07 T08 T09 T12 T01 T03 T07 T09 T12 Builder Network (10 Gbps) BU1 BU2 BUn Builder Units Where n could range (now) from 10 to 15 according to the efficiency of the event builder algorithm and to the communication protocol used In the final configuration (~ 2005) we could image To have a single 10 Gbps output link and a single BU

8 Time Slot Assignment 8 Det i MUX PSA FARM Ev # Ev # Ev # Ev Ev Ev # Ev Ev Ev # # Ev Ev Ev # # Buf # Buf Ev Ev # Buf Ev Ev # Ev Ev Buf # TTC assigns EV # (time stamp) MUX buffers events according to a given rule and then define the Time Slot MUX assigns a buffer # to this collection of events MUX distributes the buffers to PSA farm according to their buffer # EB # Detector Slice PSA farm shrink down the incoming buffers, a further buffering is then needed PSA assigns a EB (Event Builder) # to the new buffers PSA distributes the buffer to Event Builder farm according to such number. Slice 1 Slice 2 Slice 200 All this is synchronous for all the detectors Event Merging in the EB Farm is then feasible EB FARM

9 Agata Event Builder: some more requirements Clck R1 R2 R3 R200 Readout Farms 9 Time Slot 1 Time Slot 2 Time Slot 3 T01 T02 T03 T04 T05 T06 T07 T08 T09 T10 T11 T12 T01 T02 T05 T06 T07 T09 T01 T02 T04 T05 T06 T09 T10 T12 T01 T03 T07 T08 T09 T12 T01 T03 T07 T09 T12 Builder Network (10 Gbps) BU1 BU2 BU3 Builder Units - delayed coincidence can span more time slots. - fragments of the same events are in different BUs.

10 HPCC for Event Builder 10 - High speed links (> 2 Gbps) - low latency switch - fast inter processor comm - low latency message passing Builder Network High Performance Computing and Communication (HPCC) System

11 Technologies for Agata On-line System 11 On-line Software Networking Trends Event Builder CPU Trends Building blocks for the Agata Farms Storage Systems

12 On-line Software: a common framework Environment for data acquisition applications communication over multiple network technologies concurrently e.g. input on Myrinet, output on TCP/IP over Ethernet, Infiniband, etc. configuration (parametrization) and control protocol and bookkeeping of information cross-platform deployment write once, use on every supported platform (Unix, RTOS) high-level provision of system services memory management, synchronized queues, tasks built-in efficiency enablers zero-copy and buffer loaning schemes usable by everyone Aim at creating interoperable systems PSA farm applications, event builder, tracking commonly managed CMS 12

13 Local networking is not an issue. Already now Ethernet fits the future needs of the NP experiments link speed max 1 Gbps switch aggregate bdw O(100) Gbps O(100) Gbit ports per switch O(1000) FastEthernet per switch Networking Trends 256 Gbps 192 Gbps 128 Gbps 64 Gbps Myrinet Infiniband 10 GbEth GigaEthernet Aggregate bandwidth within a single switch 13 If HCCP is requested (e.g. Agata builder farm) options are Myrinet, Infiniband = 4 x Myrinet, 10 Gbit Eth 10 µsec latency time Myrinet one way latency MB/s Myrinet throughput 2003

14 Event Builder and Switch Technologies 14

15 CMS EVB Demonstrator 32x32 15 CMS

16 Myrinet EVB (with Barrel Shifter) 16 CMS

17 Raw GbEth EVB CMS 17

18 GbEth full Standard TCP/IP 18 CMS CPU Load 100 %

19 TCP/IP CPU Off-loading - iscsi 19 Internet SCSI (iscsi) is a standard protocol for encapsulating SCSI command into TCP/IP packets and enabling I/O block data transport over IP networks iscsi adapters combines NIC and HBA functions. Application Layer Driver Layer Network Interface Card Storage HBA FC Storage IP Server FC Server IP Server iscsi Adapter File Block Block Block Block 1. take the data in block form 2. handle the segmentation and processing with TCP/IP processing engine 3. send IP packets across the IP network I Processor IP Packets Link Layer IP Packets on Ethernet FC Packets IP Packets on Ethernet Intel GE 1000 T IP Storage Adapter

20 Comments on the Agata Event Builder 20 Agata Event Builder is not an issue (also now). CMS experiment has already shown the ability to work with an order of magnitudine better that the Agata requirements Agata could work, also in the prototype, fully on standard TCP/IP Agata could require an hpcc based Event Builder. Technologies already exist for that, but never applied to event builder problems. Should not be a big issue, but requires time to develop, debug, test, etc. Myrinet (now) 10 GEthernet on desktops (now, but expensive) Infiniband (soon) All this does not mean Agata Event Builder is an easy task: We have a 2 event builder chain (single detector level, global level) 3 different numbers to be assigned (event number, buffer number, event builder number). All of them have to be synchronized, etc. etc. All this means Agata Event Builder is a feasible task, also now; providing the proper resources (human in particular)

21 Processors and Storage trends SI95 Now 250 GB Now 2007 CPU = 250 SI CPU = 700 SI95 Year disk = 1 TByte

22 Building Blocks for the Agata Farms 22 ~ 1700 euro now Year SI95 x Box 1 U CPU Box with 2 processors 40 Boxes x Rack Configurations Detector 15 Detectors 200 Detectors Farms Type Nr. Boxes Nr. Racks Nr. Boxes Nr. Racks Nr. Boxes Nr. Racks PSA Farm Builder Farm / /4 Track. Farm No GLT Track, Farm GLT / /2

23 Blade Based Farms ~30000 euro now 1 Blade Box with 2 processors 14 Boxes x crate (7 U) 6 Blade crates x rack = 108 Boxes Power = 16 KW x Rack ~ 2150 euro now backplane Gbps backplane SW2 SW1 Configurations Farms Type Nr. Blades Detector Nr. Racks Nr. Blades Detectors Nr. Racks Nr. Blades Detectors Nr. Racks 2 x 4 x 1 Gbps uplinks PSA Farm Builder Farm 35-1/ < crate < crate Track. Farm No GLT / Track, Farm GLT < crate 20 1/5

24 On-line Storage On-line Storage needs 1-2 weeks experiments Max 100 TByte / experiment (no GBT) Max 1000 TByte/year disk = 4 Tbyte Storage Agata System: 250 disks (+ 250 for mirroring) 24 Archiving O (1000) TB per year can not be handled as normal flat files Not only physics data stored run conditions Calibration Correlation between physics data, calibration and run condition are important for off-line analysis Data Base technology already plays an important role in physics data archiving (Babar, LHC experiments, etc.). Agata can exploit their experience and development

25 Storage Technologies Trends Application Servers Data Servers GEth/iSCSI 25 Infiniband SAN enabled disk array gateway Commodity Storage Area Network share all the farms nodes. Technololgies interested for us are: - iscsi over Giga (or 10 Giga) Ethernet - Infiniband Full integration between the SAN and the farm is realized if a Cluster File System is used. Example of Cluster File System are: - LUSTRE ( - STORAGE TANK (IBM soon off. released)

26 Example of a iscsi SAN available today 26 Application Servers GEth/iSCSI Data Servers Host adapters: - Intel GE 1000 T - Adaptec ASA LSI ecc. 2 x GE LSI logic imegaraid 1 SATA iscsi Controller RAID SATA Controller 16 = ~ 5 Tbyte x controller SATA = Serial ATA

27 Data Archiving: Advance Parallel Server 27 Input Load balancing switch Data Servers Low latency Interconnection (e.g. HPCC) Internet Intranet Shared Data Caching Storage Area Network Scalability Example: Oracle Real Application Cluster

28 Run Control and Slow control 28 Slow Control Front-end electronic and pre-processing Run Control and Monitor System Pulse Shape Analysis Event Builder Tracking Storage

29 Run Control and Slow Control Technological Trends 29 Virtual Counting Room Take a shift from distance Tele-presence (web cam, videoconference and chatting, etc.) Web based technologies SOAP Web Services and Grid Services (Open Grid Service Architecture) Data Base Demonstrators in operation at CMS test beams facilities

30 RCMS present demonstrators 30 Java MySQL Or Oracle Java TomCat Containers Or Grid Services SOAP CMS

31 Slow Control Trends 31 Ethernet every where Agata could be fully controlled by Ethernet connections, including the front end electronics This lead to have an homogeneous network avoiding the use of bridges between busses, software drivers to peform the bridging, etc. Tini system Embedded web server and embedded java virtual machines on the electronics Embedded Java should guarantee an homogeneous development environment, portability, etc. Xilink Virtex II Pro

32 Agata Demonstrator ( ) 32 Front-End F1 F2 F3 F15 PSA Farm P1 P2 P3 P15 14 Blade Centers Event Builder HPCC builder Builder Network B1 B2 15 x 2 Eth Switch 2 Dual processor Servers + Myrinet Tracking Farm T1 T2 3 full Blade Centers no GLT 4 blades with GLT Data Servers SAN Storage Area network = iscsi iscsi Disk Array + SATA disks TByte 1 Gbps 100 Mbps

33 Project Break Down 33 F1 F2 F3 F15 3. Slow Control P1 P2 P3 P15 Builder Network B1 B2 T1 T2 SAN 2. Run Control & Monitoring Main Data Flow

34 Human Resources Agata has requirements very closed to the LHC experiments. We should exploit the LHC experiments technologies in particular for: Data Acquisition (framework, readout, event builder, etc.) Run Control (framework for distributed system, basic GUI, etc.) Slow Control Data Storage (storage techniques, DBs, etc.) In the following table it is assumed to adopt CMS (a LHC experiment) technologies for most of the above items. The red numbers indicates the FTEs required to build the Agata applications over the CMS Frameworks Slow Control Framework Development and maintenance (fte x year) Agata s Applications (fte x year) 34 Run Control & Monitoring Main Data Flow (FE read out, event builders, storage, etc. 4 FTE (LNL + Cern) 5 FTE (Cern) 2 FTE 1 FTE (LNL) + 3 FTE

35 Resources Distribution 35 We could organize 3 development centers Main Data Flow: Legnaro Run Control: XX Slow Control: YY Human resources should be concentrated in the related dev.centers Legnaro: 4 FTE (1 from LNL, 3 from ext) XX: 2 YY:?

36 Conclusions(I) No fundamental technological issues for the final Agata on-line system: 36 The experiment requirements and the present understanding of the PSA algorithm fit with a final (2010) moderate size ( O(1000) machines) on-line system. Only a 3 times improvement in the PSA calculations lead to a system much more manageable (3 racks). Both network and event builder issues already fit with the today available technologies. Storage requirements (1000 TByte) fit with the evolution of the storage technologies. Demonstrator Same architecture of the final system, only scaled down to the foreseen number of detector.

37 Conclusions (II) 37 Strong collaboration with LHC experiments and in particular with CMS is suggested, as we can benefits of: Their developments Support (partially) Experience and advice

38 38 OFF-LINE INFRASTRUCTURE A GRID APPROACH

39 > = 10 Gbps links T1 T1 T1 T0 T1 T1 LHC Off-line Infrastructure T1 T1 T1 T2s CERN Data Production Center: - on-line system - online storage ( ~ PBytes) - central archive Regional Computing Facilities (Country bounded): - computing power for analysis - on-line storage - local archive Local (in the Country) Computing Facilities: - Dep. Farm Analysis 39 Agata could exploit the LHC off-line infrastructure and in particular the Regional Computing (RC) centers Agata Data Production Center should be linked properly to such regional centers typical experiment will take about 1 day to copy the entire data set no tape copy; tape only for backup New model for off-line analysis: Grid approach

40 World Wide Farming: the GRID GRID is an emerging model of large distributed computing. Main aims are: Transparent access to multi-petabyte distributed data bases Easy to plug in Hidden complexity of the infrastructure GRID tools can help significantly the computing of the experiments promoting the integration between data centers and the regional computing facilities GRID technology fits the Agata experiment off-line requests 40 FermiLab/USA T1 T1 Brookhaven/USA T1 LYON/FR T0 T1 RAL/UK T1 T1 CNAF/IT KARLSRUHE/GE LHC Computing Grid (LCG) Infrastructure

41 Local job submission Wkld Mngm - find free resources -submit jobs to grid-farms Resource Broker DataGRID Data Mover EU has funded the DataGrid project. The main activities of the project have been: Workload Management Services for grid scheduling and resource management Data Management High speed data mover and replica Monitoring Services Global states and error monitoring Local Farm Management tools for new automated system management techniques of large computing farms 41 Local Farm Management Based on Globus (Argonne) toolkit Evolution in EU FP6: EGEE

42 Off-line Agata GRID: a possible scenario Data are produced by a given experiment in the Agata Data Taken Center 42 The experiment data set is temporary stored in the Center and optionally (if computing power is available) pre-processed with the aim to reduce their size. Original data set could be saved in the Center The (pre-processed) data set is then moved to the Regional Centers. (It could be moved directly to the final destination (lab, dep., etc.) if enough storage is available) Massive data processing (like sorting, matrix production, etc.) can be performed at the level of Regional Centers. Grid helps to provide the location of such data sets, indicating the computing power available and their location, etc. All the data sets are grid reachable and then available for processing to the whole collaboration. Outcomes are still grid reachable and then available for further analysis Final analysis can be performed exploiting all the Grid computing power available.

43 User Interface User Interface User Interface On-line Agata Grid? Take control from a distance Supporting Services Virtual Control Room Virtual Control Room Telprs Svc DAQ Svc TRG Svc Slow Svc GRID Domain Farm Services Instr. Instr. Farm 43 Raw data for on-line monitoring, Special exp line, Processing Farm extension? Video Conf. & Chat Service Data Repres. Service Storage Services Instr. Instr. DBs Tn Farms On/Off-line DBs e.g. Calibrations

44 Agata GRID: conclusions 44 GRID is becoming a serious and robust infrastructure, well funded and supported (see EGEE EU project) All the LHC experiments (see LCG project: will use it to integrate both the computing power and storage capability provided by the Regional Centers and by the production center (CERN) into a cooperative environment. Agata could join this infrastructure providing (or taken an agreement with) the needed computing power and storage to the Regional Centers. Agata, adopting the GRID middleware as proposed by LCG, has to take care of the impact on the development. This will be mainly on: On/Off-line Data Bases of the experiment and the way to access them Local farms (department level) Production Center Farm (if any) On-line software for what is concerning the DB access GRID technology could also be used to have a real time control of the experiment (control the experiment from a distance)