Monitoring the Usage of the ZEUS Analysis Grid

Transcription

1 Monitoring the Usage of the ZEUS Analysis Grid Stefanos Leontsinis September 9, 2006 Summer Student Programme 2006 DESY Hamburg Supervisor Dr. Hartmut Stadie National Technical University Of Athens Abstract The Grid has been already used very successfully for Monte Carlo production for ZEUS experiment. The next natural step is the submission of user's analysis jobs to the Grid. In the last two moths 3184 jobs, in total, were successfully submitted to various sites of the grid with a total runtime of 1162 days and 45637MB of output size. To give the users feedback, a exible monitoring system is needed. The implementation of the display part of this system is described in this note. I developed a set of PHP scripts that display all the necessary information about the submitted jobs, allows the users to check the status of jobs submitted to ZEUS Grid, and shows informations about the eciency of the sites that the jobs are submitted to. 1

2 Contents 1 The Grid The Grid Structure Job Submission on the Grid Running ZEUS Applications on the Grid ZEUS Grid Toolkit Layout of the Integrated Monte Carlo Production ZEUS Analysis on the Grid (ZAG) Monitoring the Job Submissions on the ZEUS Grid Summary of the Job Submissions on ZEUS Grid Statistics of the Grid Use Conclusions 11 5 Acknowledgments 12 2

3 1 The Grid The idea of the Grid goes back to the 90's and it was based on the principal of the electric current grid. This grid supplies electric energy to its users with a fairly simple and standardised way. Every user is able to use the grid by plugging in a device which is supplied with a specic voltage and has a standardised plug. An equivalent model is used for the computers. At this moment, millions of computers and storage devices are connected with the Internet, worldwide. The only thing required now, is an infrastructure and a standardised interface which will provide transparent access to this computing power and storage space, in a homogeneous (uniform) way. The Grid provides a service for the public use of this computing power and storage space on the Internet. The Grid moves further than the usual connection among computers and aims at the transformation of the world computer web into a great computational tool. Actually, the Grid diers from the World Wide Web as it provides the users, not with only information, but with processing power and storage space as well. The supremacy of the Grid can be shown in its applications. When a user wishes to run an application, the Grid locates the best available place for the execution of this application and it executes it at this point without any necessary actions from the user himself. The Grid can easily facilitate, in great extent, the process of great amounts of information coming from dierent computers. Again, it locates, the best available information source, without any action required by the user, and executes the respective process. Additionally, this analysis can be done in collaboration with partners around the world, as the Grid connects every user as if they were working in a local intranet. An impressive function of the Grid is that the user does not need to know which and where the computing sources needed for the task are located. The only thing required is for the Grid to provide the computing power and the storage space through a standardized interface The framework (structure) of the Grid usually consists of levels which serve a specic task. In general, higher levels focus on the user, where lower levels focus on the computers and networks. The middle software level provide the tools for the participation of the dierent elements (servers, storage elements, networks) in the uniform Grid environment. The middle software represents the Grid intelligence that unites the remaining elements and is characterized as the Grid brain. At the base of the whole substructure the network that ensures the connection of the Grid resources is situated. Over the network one can nd the level of the resources that consists of the resources that are the components of Grid as computers, storage elements, electronic data catalogs, even detectors and telescopes that can be connected directly to the network. The highest level of the Grid substructure is the level of applications that incorporates all the dierent user applications (physics, engineering, economics, etc), ports and development tools that support these applications. This is the level with which the Grid users cooperate with each other. 1.1 The Grid Structure Grid sections are the following: Resource Broker: This unit receives the commands of the users and examines the information catalogs for the appropriate resources. BDII: It is the information catalog that collects the information that is relevant to the available resources. The 3

4 catalog is very probable to be in the same machine with the Resource Broker. Replica Manager: It is used for the coordination of the le copying to the testbed during their transfer from one storage element to another. Something like this, is useful for data redundancy, but also for the data transfer closer to the machines that will realize the process. Replica Catalogue: Collects information for the data copies. A logical le can be matched with one or more natural les which are themselves copies of the same data. Therefore, the logical le can be related to one or more natural les names. The list containing the copied les can located together with the same engine as the Replica Manager. Computer Element: Computer Element refers to the unit which collects the requirements for a specic task. Following, the Computer Element delivers these requirements to the Worker Nodes (WN), which afterwards will realize the task itself. The Computer Element provides an interface on the local waiting systems cluster. One computational element is able to control one or more working knots. Additionally, one working knot can be set up in the same matching with the computational element. Worker Node (WN): The Worker Node is the engine which will process the input data. Storage Element (SE): The Storage Element is the engine which will provide storage room to the testbed. also provides with a uniform interface for dierent/individual storage systems. It User Interface (UI): It is the engine which allows the users' access to the testbed 1.2 Job Submission on the Grid A job submission can be described in seven steps divided in two cycles. The rst one involves the registration procedure and the last one is responsible for the actual job submission. The registration procedure consists of three steps and has to be done only the rst time one enters the Grid and then it only has to be renewed once a year. The rst step is to join a Virtual Organization (VO). Virtual Organizations are distributed communities. High Energy Physics (e.g. LHC VO), Earth Observation (e.g. EO VO) and Biology (e.g. Biomed VO) are examples of such communities, consisting of several institutions and individuals sharing the same interests and the same scientic goals. They greatly benet from putting together their computing resources, data and scientic instruments. Joining a Virtual Organization will provide access to the Grid facility. In order to prove your identity on the Grid Facility, you need a certicate from one of the Certication Authorities. The request for a certicate from a Certication Authority can be made by lling in and submitting an online form (e.g. The Certication Authority will check your identity, issue the certicate and give it back to you via the Web. Once you have your certicate you need to install it in the user interface (the machine you will use to access the Grid Facility). With this step the registration procedure ends and the actual job submission takes place. Every time you start a session on the Grid, you need to create temporary credentials. This is necessary to avoid exposing your certicate to an insecure network. Proxy credentials have default expiration time of 12 hours. During 4

5 this period you can work on the testbed. To run a job on the LCG/EGEE Grid Facility, you have to describe it in the Job Description Language (JDL). JDL species job characteristics such as the application to use, the input data, the required resources, etc. Once you have the jdl le for your job, you can submit it to the Resource Broker. At the same time, the Logging & Book-keeping service logs the job as submitted. Based on the information given in the JDL le, the Resource Broker queries the Information Service and the Replica Catalog to check resources. The Replica Catalog and the Information Service hold information on the current status of all the sites. The Resource Broker uses this info to match the job to a suitable Computing Element. During this phase, the job is in WAITING status. The Resource Broker makes its choice. It has found a suitable Computing Element and the Storage Element with the necessary data. It informs the logging & Book-keeping service of its decision.the Resource Broker submits the job to the selected Computing Element service of its decision. The Computing Element gets any necessary data from the Storage Element. The Job is eventually executed in the chosen Computing Element. During all this process, you can check the status of your job by contacting the Logging & Book-keeping Service.The execution of the job has completed on the Computing Element. The Computing Element transfers the output to the Resource Broker. You can now retrieve your Output from the Resource Broker. When nished, book-keeping information is purged. 2 Running ZEUS Applications on the Grid Around 70% of ZEUS Monte Carlo production is done on the ZEUS Grid. The past year more than more than 240 million jobs were submitted in various sites of the grid with a total of more than 14TB of output size. The Monte Carlo production package for ZEUS is an application built-in on top of ZEUS Grid Toolkit [1]. 2.1 ZEUS Grid Toolkit The ZEUS Grid toolkit is the basic toolkit for the implementation of the new production system. It is written in object-oriented Perl and consists of a set of classes for basic data structures, job submission, data transfer and output logging and validation. The parts of the toolkit that use Grid client tools directly are encapsulated, because a variety of client tools to access Grid services exist on dierent sites and new projects are being developed. However, the main concepts for data handling and job submission do not change. Consequently, the ZEUS Grid toolkit implements the Strategy pattern and adds an additional layer of abstraction to the usage of Grid client tools. Abstract interfaces are dened for data handling and job operations, and the correct middleware implementation is chosen at run time based on a conguration le and the installed middleware packages. In addition, the encapsulation enables us to x known deciencies of the Grid tools. In earlier versions of the LCG middleware, failed or never-ending data transfers were the main cause for job failures. In the ZEUS Grid toolkit, all data transfer commands are, therefore, run with a timeout and are retried automatically a congurable number of times in case of failures. Furthermore, the size of the le and a checksum are used to validate the le after every transfer to insure the integrity of the data. These measures have been found to reduce the failure rate considerably. 5

6 2.2 Layout of the Integrated Monte Carlo Production To include Grid resources in the Monte Carlo production system completely transparently for the users, the existing interfaces for the submission of a Monte Carlo request and querying its state are reused. As can be seen in Fig. 1, a central scheduler distributes incoming Monte Carlo requests either to the traditional Monte Carlo sites or to a gateway to the Grid resources. This setup allows us to preserve the resources of the HERA-1 production system and to reuse most of the existing scheduler. One node set up as a LCG user interface acts as a bridge between the production system and the Grid world. Cron jobs process the incoming requests and keep track of the individual Grid jobs. A database is used to store the state of the Monte Carlo requests and their associated Grid jobs. All the code is written in object-oriented Perl and submit jobs. Since the ZEUS Grid toolkit is able to simultaneously support dierent middleware projects, we were able to establish submission to a non-lcg site, which belongs to the University of Wisconsin in the USA and runs the Grid2003 middleware. Jobs are submitted directly to this site using an implementation of our job submission interface based on the Globus toolkit [2]. For any new request, a cron job translates this request into a set of Grid jobs and copies the input le to the storage element at DESY. The LCG jobs are submitted using the resource broker at DESY. Every Grid job processes between 1000 and 2000 events which corresponds to a run time of around 3h. The status of the jobs is updated regularly and failed jobs are automatically resubmitted. When a Grid job has nished, its output sandbox is retrieved and the log le is checked for error conditions. The result of this check is stored in an additional database table. If the job has passed the check, the Monte Carlo output le is transferred from the DESY storage element to the nal tape storage tool. As both systems use the dcache mass storage subsystem [8,9], this is a very fast operation. When all Grid jobs for one request have nished, the gateway returns the request to the production system with all necessary bookkeeping les and an archive containing all log les of the individual jobs. 6

7 Figure 1: Fig. 1. Layout of the integrated Monte Carlo production When a Grid job starts running on a worker node, it rst copies two archives containing all the run scripts for the Monte Carlo production and the calibration constants and the three executables belonging to the requested Monte Carlo version from a Grid storage element to the local disk. This avoids the requirement of pre-installed software. As each Grid job only processes part of the input le, the events needed for this job are then extracted from the input le. The executables are run consecutively and their log les are checked after each run. If no errors are found, the output le is nally copied to the DESY storage element. The execution of data handling operations and all commands are logged to the standard output. The same is done with errors that occurred during the data validation or were spotted in the log les. This output is analyzed by the production system after the job output was retrieved. A crucial problem for diagnostics on the Grid is the fact that a job waiting for data for a long time or hitting an endless loop on an executable is generally killed by the local batch system and delivers no output. If this happens, one does not obtain any information about the cause of the failure. Therefore, it is important to guarantee that the job nishes within the queue time limit to identify these problems. This is achieved with ZEUS Grid toolkit by imposing a timeout on all commands executed on the worker node. 2.3 ZEUS Analysis on the Grid (ZAG) The next step for ZEUS Grid is to extend the usage of Grid resources to user`s analysis jobs. This is much more dicult as it is not a central production and the users might submit many dierent jobs with dierent requirements. A job submission framework has been developed based on the experiences gained from the Monte Carlo production. This system is now in a testing phase. In the last two moths 3184 jobs, in total, were successfully submitted to various sites of the grid with a total runtime of 1162 days and 45637MB of output size. To give the users feedback, a exible monitoring system is needed. 7

8 3 Monitoring the Job Submissions on the ZEUS Grid Every job submitted to the ZEUS Analysis on the Grid (ZAG) system is also led to a MySQL database. So, in order to check the status of a job you either type the command edg-job-status or go to the MySQL database and type select user, state from anajobs where user="<name of user>";. The status of a ZAG job can be Ready (state=1), Submitted (=2), Cleared (=3), Done (=4) or Completed (=5). The database also contains information about the site the job is submitted to. You can check the site that the job is submitted to with the command select user, site from submissions where user="<name of user>"; 3.1 Summary of the Job Submissions on ZEUS Grid In order to be easier to check the status of the submitted jobs on the ZEUS Grid, I built a page where all the users can be informed about the status of their jobs. The page contains a simple drop-down menu which includes all the users from the analysis job database using MySQL and an Update button which can be used to refresh the page and update the status of the jobs. In the menu there is also an all option, which includes every job from every user. Figure 2: Monitoring the user's jobs 3.2 Statistics of the Grid Use The ZAG database also contains information about the size, the runtime and the result (whether it was successful or not) of the jobs of each user. Using these data I made another page that monitors the total completed jobs, the Grid jobs success rate, the runtime and the Outputsize for every user. This page also gives you the option to select a specic period of time that you want to extract the information from. 8

9 Figure 3: Stats of the ZEUS Grid use The third page displays informations about the sites that the jobs are submitted to. It also features the option to select a specic period of time that you want to extract the informations from. If the success rate of a site, in the chosen period, is under 50% the percentage is shown in grey background. Each site has a link to another page, which contains information about the error that failed the job. Figure 4: Statistics of Sites the ZEUS Grid use 9

10 Figure 5: Error messages of the selected site 10

11 4 Conclusions The ZEUS Grid Analysis database contains valuable data about the job submissions to the ZEUS Analysis on the Grid (ZAG) system. These data include information about the size, the runtime and the result of the jobs that each user submits. The presented display system oers a helpful and easy way to gain information about job submissions on the ZEUS Grid. They, in general, replace any eort to type MySQL or grid commands and thus users do not have to be Grid experts to get information about their jobs. The system also provides information about the past use of ZEUS Analysis Grid. 11

12 5 Acknowledgments I would like to thank my supervisor, Hartmut Stadie, for helping me and and for the all the time he spent for solving my problems. I also would like to thank my parents, Gregory and Christina, and my brother Stamatis. You really made my work easier... 12

13 References [1] H. Stadie et al.: Monte Carlo mass production fot the ZEUS experiment on the Grid, Nuclear Instruments and Methods in Physics Research A 559 (2006) [2] I. Foster, C, Kesselmann, Globus: a metacomputing infrastructure toolkit, Int. J. Supercomput. Appl. 11 (2) (1997) 115. [3] P. Fuhrmann, dcache, LCG storage element and enhanced use cases, in: Proceedings of the International Conference on Computing in High Energy Physics, 2004 [4] dcache: 13