Usage statistics and usage patterns on the NorduGrid: Analyzing the logging information collected on one of the largest production Grids of the world

Transcription

1 Usage statistics and usage patterns on the NorduGrid: Analyzing the logging information collected on one of the largest production Grids of the world Pajchel, K.; Eerola, Paula; Konya, Balazs; Smirnova, Oxana; Erkarslan, U.; Ould-Saada, F.; Ellert, M.; Renner Hansen, J.; Langgard Nielsen, J.; Wäänänen, A.; Ekelöf, T.; Konstantinov, A. Published in: CERN Link to publication Citation for published version (APA: Pajchel, K., Eerola, P., Konya, B., Smirnova, O., Erkarslan, U., Ould-Saada, F.,... Konstantinov, A. (005. Usage statistics and usage patterns on the NorduGrid: Analyzing the logging information collected on one of the largest production Grids of the world. In A. Aimar, J. Harvey, & N. Knoors (Eds., CERN (Vol., pp CERN. General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. Users may download and print one copy of any publication from the public portal for the purpose of private study or research. You may not further distribute the material or use it for any profit-making activity or commercial gain You may freely distribute the URL identifying the publication in the public portal Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. L UNDUNI VERS I TY PO Box7 00L und Download date: 0. Dec. 08

2 USAGE STATISTICS AND USAGE PATTERNS ON THE NORDUGRID ANALYZING THE LOGGING INFORMATION COLLECTED ON ONE OF THE LARGEST PRODUCTION GRIDS IN THE WORLD. K. Pajchel *, Dr. U. Erkarslan, Prof. F. Ould Saada, University of Oslo, Norway Dr. M. Ellert, CERN, Geneva, Switzerland Prof. P. Eerola, Dr. B. Kónya, Dr. O. Smirnova, Lund University, Sweden Prof. J. R. Hansen, Dr. J. L. Nielsen, Dr. A. Wäänänen, Niels Bohr Institute, Copenhagen, Denmark Prof. T. Ekelöf, Uppsala University, Sweden Dr. A. Konstantinov, Vilnius University, Lithuania and University of Oslo, Norway Abstract The Nordic Grid facility (NorduGrid [] came into operation during summer 00 when the Scandinavian ATLAS HEP group started to use the Grid for the ATLAS Data Challenges (DC and was thus the first Grid ever contributing to an ATLAS production. Since then, the Grid facility has been in continuous 4/7 operation. NorduGrid is being used by a growing set of active users from various scientific areas including physics, chemistry, biology and informatics. It has given major contributions to the ATLAS Data Challenge [] and the ongoing Data Challenge. The increasing number of resources has made NorduGrid one of the largest production Grids in the world, continuously running on more than 0 sites more than 000 CPUs. The resources range from small test clusters at academic institutions to large farms at several supercomputer centers and the NorduGrid software runs on clusters with very different Linux distributions. This presentation gives a short overview of the design and implementation of the NorduGrid middleware, logging and monitoring facilities. It will be followed by a description of a typical job on NorduGrid and the information about its parameters which are monitored online and persistified in the logging service. MIDDLEWARE The NorduGrid middleware (or Advanced Resource Connector, ARC [] is an open source software solution distributed under the GPL license, enabling productionquality computational and data Grids. It is a light weight Grid solution, designed to support a dynamic, heterogeneous Grid facility, spanning different computing resources and user communities. Emphasis is put on scalability, stability, reliability, robustness, and performance of the middleware. These goals of the architecture of the ARC imply concrete decentralization and functional independence of the components. NorduGrid provides innovative solutions essential for a production quality middleware: the Grid Manager, gridftpd (the ARC/NorduGrid GridFTP server, the information model and providers (NorduGrid schema, User Interface and broker (a "personal" broker integrated into the user interface, extended Resource Specification Language (xrsl, logging service, and the monitoring system. Most of these services are provided through the security layer of the GSI. The middleware builds upon standard open source solutions like the OpenLDAP, OpenSSL, SASL and Globus Toolkit (GT libraries. ARC main components are: Grid services running on the resources: the Grid Manager, the gridftpd and the information service. Indexing services for the resources and data. Client making intelligent use of the distributed information and data available on the Grid. Distribution The middleware is available as source and binary in tarballs and RPMS. * katarina.pajchel@fys.uio.no

3 nordugrid cluster: a cluster, nordugrid queue: a queue at the cluster, accessible by the ARC users, nordugrid job: a Grid job, associated with a queue, nordugrid authuser: a user, authorized to submit jobs to a given queue It also displays information about storage facilities, authorised users, and groups of users (Virtual Organisations, using the corresponding objects. Figure : NorduGrid's ARC middleware components. LOGGING A Logger service is one of the Web services implemented by NorduGrid and based on gsoap and Globus IO API. It provides a frontend to the underlying MySQL database to store and retrieve information about computing resources usage (jobs. This database can be accessed through a graphical Web interface implemented using PHP4, JavaScript and JPGraph based on GD library. The main goals for the Logger: Provide information about the development and usage of NorduGrid over time. Provide statistics for different clusters, time intervals, applications and users. A SCHEMATIC EXAMPLE OF A TYPICAL NORDUGRID JOB (& (executable="myprog" (inputfiles= ("myprog" " ("myinputfile" "gsiftp:// (outputfiles= ("myoutput" "gsiftp:// (disk=000 (notify="e myname@mydomain.org" Once a proxy is obtained, jobs can be submitted to NorduGrid using simple commands and the user can keep track of the job through the whole process. Submitting: > ngsub f myjob.xrsl The user can now follow the job on the Monitor Check the status of the job: >ngstat [options] [jobs] Retrieve the results of a finished job: >ngget [options] [jobs] Figure : Content of the Logger database. The columns shown are temporary. The database is being extended. THE GRID MONITOR The Grid Monitor allows the user and system administrators alike to monitor and check in an intuitive way the status of the ARC Grid facility and the information stored in the Information System. The structure of the Monitor is built upon the following objects: The job usage record gets logged in the logger database by the Grid Manager and can later be displayed using the Web based Logger. GRAPHICAL INTERFACES Nordugrid offers intuitive access to the system data through two different Web based graphical interfaces. The users can easily follow their jobs and obtain information about the performance of the system and job statistics.

4 Grid Monitor The Monitor is the starting point for browsing the system information. It shows the current state of the system. Grid jobs are not kept infinitely in the information system. Those whose lifetime has expired are removed from clusters and thus they disappear from the monitor too. All the elements in the Monitor are links to the information about the different objects in the system. NGLogger The NGLogger is a graphical interface to the underlying MySQL database. The logger service is the persistent database for job usage records. It provides information which is complementary to the Monitor, showing the history of the NorduGrid usage. The queries are based on different basic approaches to the database like cluster, time period, application or user. Figure : Cluster alias Cluster load showing both Grid and non Grid jobs Queuing jobs Figure 4: Example of cluster information Attributes: dump of all the attributes of the nordugridcluster object. Queues at a given cluster. Figure 5: NGLogger The user can get access to information about his or her usage of NorduGrid through a search form. Search form. Number of jobs. Time from submission of the job until it is finished.

5 CONCLUSION NorduGrid offers a user friendly monitoring service showing the present status of the system as well as a complementary tool, the logger service, which provides the history of the activity on the Grid. Both services are web based and offer graphical interfaces to the usage data. REFERENCES [] NorduGrid live: Grid Monitor and NGLogger [] Eerola, P; Ekelof, T; Ellert, M; Hansen, J R; Hellman, S; Konstantinov, A; Konya, B; Myklebust, T; Nielsen, J L; Ould Saada, F; Smirnova, O; Wäänänen, A, Atlas Data Challenge on NorduGrid [arxiv:physics/0060] [] The NorduGrid/ARC User Guide, Available: time period dataset cluster Figure 6: Data Challenge on NorduGrid Parameters to be set by the user: Figure 7: Number of jobs recorded in the logger. Over jobs were run successfully in September 004. Notice that it is not the Grid, but the experimental logger service which was down in March and April. The early information may be incomplete.