experiment E. Pasqualucci INFN, Sez. of University of Rome `Tor Vergata', Via della Ricerca Scientica 1, Rome, Italy

Transcription

1 Using SNMP implementing Data Flow and Run Controls in the KLOE experiment E. Pasqualucci INFN, Sez. of University of Rome `Tor Vergata', Via della Ricerca Scientica, 0033 Rome, Italy M. L. Ferrer, W. Grandegger, E. Pace INFN, Laboratori Nazionali di Frascati, Via Enrico Fermi 40, Frascati, Italy A. Doria INFN, Sez. of University of Napoli, Via d'oltremare Pad. 20, 8025 Napoli, Italy Event builder systems in high rate physics experiments are going to be congured as computer networks built around switches (FDDI, ATM, FCS, HIPPI). Data coming from dierent sectors of the apparatus go into the switch in such a way that all the data related to the same event are concentrated in the same processor. A central processor that is responsible of the data ow control manages the assignment between events and processors. The standard TCP/IP is frequently used to transfer data, while the standard SNMP is used to manage the network devices. The KLOE experiment intent to extend the use of the SNMP protocol in its DAQ system. Private extensions of the SNMP standard MIB are used to implement the network related and feed-back functionalities of the data ow protocol. Similar extensions are used to implement the command distribution of the Run Control system. The KLOE event builder overview The KLOE experiment is currently under development and is scheduled to run in the fall of 997 at the new -factory DANE in Frascati, Italy. The KLOE Data acquisition system 2;3;4, requiring a sustained data throughput of 50 MB/s, uses custom readout devices to concentrate data related to a given trigger number and coming from a sector of the apparatus, into a VME readout manager board 5. The trigger rate is expected to be 0 khz and 0 dierent chains of readout are foreseen. Data related to the same trigger number need to be concentrated in a processor where the building of the complete event is done. The data acquisition system uses a FDDI GIGAswitch from Digital to connect the 0 readout chains to a farm of 50 processors. A commercial VME-CPU board with FDDI interface is in charge of grouping sub-events corresponding to a given set of triggers and sending these packets ( 50kB) to a dened destination in the farm. The TCP/IP protocol is used to manage multiple connections between VME crates and farm processors. A sub-farm is dened corresponding to the processors connected to the same switch port. One of them, the net-processor, is dedicated to receive, build and distribute complete events using shared memory mechanisms.

2 Due to the bridge capabilities and the input and output buering in the GI- GAswitch, a dierent number of ports can be dedicated to the readout chains and to the farms. Network devices in the event builder system of KLOE, e.g. GIGAswitch 6 and Ethernet-FDDI bridges, are congured and controlled using the Simple Network Management Protocol (SNMP). The SNMP software running on these devices is supplied by the vendors. Computer communication interfaces and hosts 7 can also be controlled using the SNMP daemons and agents that by default are included in the kernels of many operating systems in the market.. The Data Flow Control (DFC) A central processor in the event builder system, DFC, is responsible for the control of the event ow, optimizing the distribution of trac between dierent ports of the switch, and avoiding the overloading of sub-farm elements. The DFC retrieves the network map and produces a table of `enabled' Internet destination addresses, ordered in such a way that packets of consecutive sub-events are sent through dierent switch ports. This table resides in memory shared with all the VME CPUs sending data to the farm. A net-processor can ask to be `disabled' or `enabled' by sending to the DFC `input queue almost full' or `queue available' messages. A specialized protocol assures the use of the correct table as a function of the maximum trigger number contained in the packet..2 The Run Control (RC) The Run Control system distributes commands to tasks running in the processors and collaborating to the DAQ work. Commands like Init activity, Start Run,..., are sent and some acknowledgements about the completion of the command execution from the remote tasks need to be received before the trigger system is activated (or after it is deactivated). Variables are dened in each node to maintain the history of the Run Control commands execution. The DFC and RC functionalities listed above are implemented in KLOE using the SNMP protocol. 2 The Simple Network Management Protocol, SNMP SNMP was proposed as a standard by the Internet Engineering Task Force (IETF) and is widely used to manage network devices, to obtain and/or set informations about network conguration, performance, faults, accounting and security. SNMP version is dened in [RFC 55], [RFC 57] and [RFC 22], while version 2 is dened in [RFC ]. Managed objects are dened in a Management Information Base (MIB) by working groups within IETF, but extensions or proprietary MIBs are developed by vendors of hardware or software if their new products require them. SNMP is implemented using a reliable UDP through a well dened UDP port. Every managed device activates at boot time a daemon that is able to understand 2

3 KLOE KLOE_Machine KLOE_Tools KLOE_MachineType KLOE_MachineQFull KLOE_Processes... KLOE_MachineEnableFlag KLOE_ProcControl _LastCommand _Status KLOE_ProcStatus _LastCommandStatus _CommandStatus _Command _CommandStatus.i _Command.i KLOE_ProcName KLOE_ProcName.i KLOE_ProcStatus.i Figure : The KLOE MIB subtree SNMP requests, and to obtain or execute by dedicated agents the information or action required. TRAPs are also dened, for instance in case of crash of a communicating interface, that are converted by the agent into interrupts to be sent through a dierent UDP port to a list of specied hosts in the network. Well documented public domain software exists, in particular Carnegie Mellon University (CMU) distributes a very nice portable SNMP daemon (including agent functionality) and a set of SNMP utilities: get, set, walking in the MIB tree, daemon for SNMP traps, that have been used in the implementation described in this paper. 3 The KLOE-SNMP libraries Based on the CMU software a set of routines have been written to obtain the network map and to allow to send commands and retrieve acknowledgements by the RC. Every processor in the DAQ system activates a SNMP daemon-agent that implements the MIB sub-tree in g.. Variables are dened to manage the dierent machine types, RC commands, KLOE debugging tools and to perform process control. In particular, the KLOE MachineType and KLOE MachineEnableFlag variables indicate the role of the processor in the DAQ system (VME-CPU, net-processor,...) and the ability to be included in the map at the beginning of the run. 3. Network mapping The network map is implemented as a list of structures, each of them containing information about a port of the switch and a list of nodes connected to that port including their characteristics (addresses, functionalities,...). Local tables in the GIGAswitch contain lists of the MAC addresses that are seen through every port. Moreover, an ARP-server is needed in the DAQ network to obtain correspondences between MAC and Internet addresses. Therefore initial trac need to be originated through the switch to ll GIGAswitch and ARP-server tables. The sequence of SNMP get operations implemented in the initialization routine to retrieve the network map is given in the following. the ARP server is interrogated to obtain 3

4 KLOE Command status codes: Success 0 Idle - Not Allowed KLOE _LastCommand _Status Active _LastCommandStatus _CommandStatus _Command Init_Activity Init_Run Start_Run Stop_Run Before Init run _LastCommand _Status 2 Initialized _LastCommandStatus _CommandStatus _Command Init_Activity Init_Run Start_Run Stop_Run After Init run Figure 2: The KLOE RC subtree before and after the Init run command { the FDDI interface number { the nodes seen through this interface { the mapping between their MAC and Internet addresses. the GIGAswitch is interrogated to realize the number of the port where each processor is seen all the nodes are then interrogated to extract the table containing those that will receive events from the DAQ, reading the variables KLOE MachineType and KLOE MachineEnableFlag. This list is used to initialize the DFC table; the use of SNMP traps to implement the `input queue almost full' and `queue available' messages are under study. 3.2 The Run Control protocol Fig. 2 presents the MIB variables related to the execution of a Run Control command e.g. the Init run. To execute a remote command, the RC processor sets the KLOE RC LastCommand variable. Available commands are dened in the KLOE RC Command table. A rst-level acknowledgement (`command received') is implemented in the SNMP protocol itself. As a consequence, the agent submits the command to the processes listed in the local table and sets the KLOE RC LastCommandStatus variable to the appropriate error or `executing' code. Local variables will be modied after completion of the local execution, e.g. KLOE RC Status will present the actual general status of the Run Control (in this case is `Initialized'), KLOE RC CommandStatus will present the status of each command (`success', `failure', `idle', `not allowed') and KLOE RC LastCommandStatus will present the error or success code related to the last executed command. In such a way the KLOE RC subtree maps in each KLOE node the RC status as a state machine. The RC obtain the second-level acknowledgement (`command executed') by getting the KLOE RC LastCommandStatus variable. 4

5 4 Conclusions The SNMP version framework, and in particular the libraries and agents implemented by CMU, have been used with success for the KLOE data acquisition system, and in particular the DFC protocol and the command distribution mechanism of the Run Control system. No special SNMP security system, as dened in the version 2, is necessary now, but it's included in the CMU software and could be easily implemented. Proprietary MIBs have been dened. KLOE private agents have been used, including both public and KLOE private MIBs. These agents can replace the SNMP daemon included in the operating system, e.g. on DEC and HP nodes, but further work is in progress to extend these original daemons to include the KLOE agent. In this way every network management capability is retained in the system. References. The KLOE collaboration, KLOE, a general purpose detector for DANE, LNF-92/09 (992) The KLOE collaboration, The KLOE detector, Technical Proposal, LNF- 93/002 (993) 2. The KLOE collaboration, The KLOE Data Acquisition System, LNF-95/04 (995) 3. H. Beker, Data transfer mechanisms for the KLOE Data acquisition System, presented in sect. C.3 in this conference. 4. M. L. Ferrer, The KLOE DAQ system and DAQ issues for experiments in the year 2000, plenary talk in this conference 5. A. Aloisio, Level DAQ system for KLOE, presented in sect. C. in this conference. 6. Digital Equipment Corporation, GIGAswitch system Manager's Guide and Release Notes (version 2.), Part Numbers: EK-GGMGA-MG.A0 and AA- PZT9C-TE. 7. S. Waldbusser - Network Working Group, RFC 27, Remote Network Monitoring Management Information Base Carnegie Mellon University, Nov.99 5