HPC Metrics in OSCAR based on Ganglia

Transcription

1 HPC Metrics in OSCAR based on Ganglia Google Summer of Code 2006 Report Babu Sundaram, Department of Computer Science, University of Houston Mentor: Erich Focht, Open Source Cluster Application Resources August 21, Project Overview and Goals The Open Source Cluster Application Resources (OSCAR) toolkit allows easy and flexible installation and maintenance of high performance computing (HPC) clusters. Often, large clusters, managed by distributed resource management (DRM) systems, are used to solve hard problems via HPC. In such settings, it is a non-trivial challenge to obtain varied, dynamic status information pertaining to the user jobs and the nodes (such as network, CPU, memory and storage). System administrators need this to monitor cluster status, locate, reason and solve any performance problems/bottlenecks and obtain detailed understanding of the cluster usage. Users need this to know the execution characteristics of their jobs and identify any load imbalances in communication or computation. Ganglia is a scalable, robust and ecient implementation of a distributed monitoring system for HPC clusters and is currently a supported package in OSCAR. However, this provides only the basic load metrics from the client nodes, but lacks in detailed job and resource information. It is a critical, open problem to enrich and extend this information by adding job-related and other custom HPC metrics and provide mechanisms in OSCAR to graphically interpret this information in a way useful to users, admins and developers. This project s goals were extending the Ganglia-OSCAR package and adding an infrastructure to obtain custom HPC metrics from a OSCAR cluster. Specific tasks included enabling support for gexec and authd services in Ganglia. Also, Job Monarch (a set of tools to monitor and archive job information) support was added to the Ganglia-OSCAR package to handle job-related status information from DRMs. Additionally, Job Monarch package was extended to support Sun Grid Engine DRM. Sufficient code has been added to allow using DRMAA bindings in the near future. The metric feeds from Job Monarch were handled as Ganglia XML streams for archiving and monitoring and can be interpreted graphically via the Ganglia interface, which was extended to support job- and user-specic views. 1

2 2 Details of the work The core of the summer work consisted of adding gexec and associated support in Ganglia and extending its interface using Job Monarch tools. Major tasks involved were as below: Addition of gexec tool to the existing OSCAR-Ganglia package This task involved creation of SRPM, binary rpms for libe, authd and gexec. There were some issues in compiling gexec with the latest version of Ganglia toolkit. So, I added some patches and modified the Makefile of gexec to allow correct building and installation. The correct versions of libe, authd were also identified so gexec can be successfully installed and configured. Once the authd rpm is installed, the SSL keys (/etc/auth priv.pem, /etc/auth pub.pem) are generated so password-less authentication can be done between cluster nodes prior to doing gexec. These keys were made part of the image by post rpm nochroot so they are automatically located at the correct locations on deployment at the client nodes via OSCAR. Creation of Job Monarch package for OSCAR for job monitoring Job Monarch is an addon for the Ganglia monitoring system and plugs into an existing Ganglia setup. jobmond is the job monitoring daemon that gathers PBS/Torque/SGE batch statistics on jobs/nodes and submits them into Ganglia s XML stream. jobarchived is the Job Archiving Daemon. It listens to Ganglia s XML stream and archives the job and node statistics. It stores the job statistics in a Postgres SQL database and the node statistics in RRD files. In order to enhance the graphical support in Ganglia to monitor user jobs that are under the control of batch queuing systems, a package for Job Monarch was created and added to the OSCAR framework. This package requires Ganglia web frontend to be pre-installed. The Job Monarch package modifies the default templates in Ganglia to custom ones so job information can be displayed on the frontend. The fields on the webpages are populated by querying and parsing the information obtained by the interfaces with local batch queuing system on the cluster. Given the current status, Job Monarch works out-of-the-box with Torque/PBS and SGE queuing systems. With this package, sysadmins/users can view jobwise node statistics and user-specific resource usage. Adding support for SGE and DRMAA in Job Monarch As part of the summer work, Job Monarch was extended to support SGE queuing system. Basically, it does a qstat -xml and parses it to produce a dictionary of job information with jobids as keys. It ensures that only new jobs are added to the existing information, old jobs are removed if they have completed or aborted, and updated if there is a status change (say, from pending to running ), otherwise only the reported timestamp is changed to indicate that this is the most recent query. (A job running on multiple nodes has the nodes elements containing the names of the nodes delimited by ; ). Then, it broadcasts this information via gmetric in a format understandable to Job Monarch web frontend. The default page for monitoring this information can be found at on the master node. One can see the exact gmetrics that get reported by choosing a specific node or the job list by clicking the job list boxes that appear below that name. The sequence of boxes indicate the nodes and those nodes with running jobs are marked with the alphabet J. 2

3 Creation of DRMAA-Python package for OSCAR DRMAA is a specification to provide the high level functionality necessary for an application to consign a job to a DRM system including common operations on jobs like termination or suspension. Developers can use this to easily and uniformly interface their applications across various DRMs. A package for DRMAA s python bindings (available as part of SGE project) was created for the OSCAR framework. By installing this package, one could get a python API interface that they can import to create DRMAA compliant applications. With the current status, this binding allows exploiting the DRMAA C implementation of SGE. As a later effort, the Job Monarch package will be ported to utilize this interface when certain issues are sorted out within SGE s DRMAA implementation. More information on this is available at bug.cgi?id= Weekly Work Breakdown A detailed weekly work log can be found below: Week 1: May 24th - May 31st - Completed Ganglia compilation with gexec - Building libe, authd, gexec - There were some problems getting the correct versions of the above that work correctly with latest Ganglia 3.0.x - Identified the correct versions of the components above for building - Wrote correct spec files for libe(0.3.0), authd(0.2.2) and gexec(0.3.6) - Successfully built the RPMs and SRPMs on FC4-i386 - There were some portions with gexec implementation that were using old Ganglia 2.x Week 2: Jun 1st - 7th - Implemented patches to gexec so it built correctly with Ganglia 3.0.x - Modified the paths to header files - Added the requirement for ganglia-devel and libe or later - Added the linking to expat library - Created the updated spec file for gexec - Got SVN access to OSCAR repository; Created hpcmetrics dir for the SoC code - Completed a test bed setup in UH using FC5 on i386 with OSCAR 5.0 from trunk - Rebuilt all the RPMs for FC5 3

4 Week 3: Jun 8th - 14th - Made changes to Ganglia s spec file - to allow gexec support - enable-gexec as part of configure phase in ganglia build - Tested the modified Ganglia package on OSCAR cluster on Master node - Brushed up on my Python knowledge to start work with Job Monarch - Read up on DRMAA, obtained some familiarity with DRMAA-Python implementation Week 4: Jun 15th -21st - Built DRMAA Python on FC5-i386 with SGE s C bindings as the DRM - Created DRMAA python spec file for building RPMs; Requires DRMAA - Modified SGE-OSCAR package spec so it provides DRMAA that is required by DRMAA- Python - Created RPMs, SRPM for DRMAA-Python Preliminary tests to monitor SGE jobs via DRMAA API - OSCAR Package for DRMAA-Python - Renamed authd RPMS to gexec-authd to avoid conflict with RFC 1413 identd daemon (Also called authd). Otherwise, the identd daemon RPM was installed instead of authd prior to gexec Week 5: Jun 22nd - 28th - Changes to Job Monarch implementation were requested from Ramon Bastiaans - An if test is added to check whether to use pbs interface or DRMAA s - Support was added to express the Interface needed as part of Monarch s config file - New BATCH API option introduced in jobmond.conf; When set to DRMAA it will use the Python binding (onto SGE s C binding) - some unexpected delays this week - A few servers were compromised by intrusions at my departmental resources - DRMAA API issues - Could submit jobs to OSCAR-SGE; But wait() call on SGE jobs fails due to ValueError Week 6: Jun 29th - Jul 5th - Clarified for ValueError with SGE developers; issue exists; will be fixed in SGE6.0u9 - Note: SGE s current DRMAA implementation cannot handle jobs outside the current session - For the summer work, it was decided to use qstat of SGE directly and not via DRMAA - DRMAA-Monarch integration completed; Note: But this code is not being used currently for job information 4

5 Weeks 7 & 8: Jul 6th - 19th - Parsing code for SGE was changed; DataGatherer now does qstat -xml with no -ext option - The SGE information is prepared so as to obtain the reporting style as PBS/Torque - A new handler case was added to correctly parse the time information in XML returned by older qstat (in SGE6.0u2 and such) - Troubles with test OSCAR cluster; the clients cannot connect to Master node via network boot; tftp times out Week 9 & 10: Jul 20th - Aug 2nd - Refined SGE-to-Monarch interface so no explicit temporary files were used to store qstat queries (used os.popen call) - The implementation of SGEGatherer was changed to produce a dictionary of dictionaries of job info in order to reuse Monarch s PBS frontend code - To accommodate the above, compilegmetricval() and submitjobdata() functions were added to SGEGatherer class - For jobs that were already in R(unning) state when jobmond is started, the submit time will be shown as start of epoch Week 11: Aug 3rd - 9th - Testing Job Monarch frontend; It was not producing the graphics; httpd error logs indicated mbstring() absence as a fatal error - Issues with php-mbstring was resolved after discussions with Ramon - New libtoga.php was obtained that works correctly with php versions 5.0 and above - getepochformat() function was added to SGEGatherer so as to convert qstat s time report into epoch format - submission time and start time are now correctly displayed on the Monarch s wen frontend - Got the Job Monarch frontend working on a non-oscar cluster Week 12: Aug 10th - 16th - First version of Job Monarch package was completed - Test OSCAR cluster was successfully installed with FC5 on i386 - Job Monarch package tested successfully on the OSCAR cluster - Test job (touchmem) from Erich was submitted as to let it run for a long time nonintrusively - Snapshots and update sent to OSCAR-devel Week 13: Aug 17th - 21st - Documentation of the work - Added the key generation support to gexec-authd rpms - Modified post rpm nochroot to include the keys as part of the image for clients - Refined the gexec and authd package and tested them 5

6 - Changed SARA logos in Job Monarch s package to OSCAR banners as local images 4 Documentation, testing and licensing The last week was spent on documenting the work in Summer including a weekly breakdown of the tasks done. Also, I installed the Job Monarch package with Ganglia and SGE on a test OSCAR cluster and successfully tested it. The code developed as part of this summer work is licensed under GNU public license. All the code developed as part of this project is available via the OSCAR s SVN repository. The root for the various sub-tasks and the related code is located at Additional repository for the Monarch package is also hosted in the Computer Science department of UH and is available at babu/oscar/jobmonarch. References [1] Open Source Cluster Application Resources (OSCAR) Project, [2] Sun Grid Engine Open Source Software, [3] TORQUE Resource Manager, [4] Google Summer of Code, [5] OSCAR SoC SVN repository, /hpcmetrics [6] DRMAA Specification Home Page, [7] Job Monarch Tools and Addons for Ganglia Toolkit, 5 Acknowledgments I sincerely thank Erich Focht for being patient with me and mentoring my work through the summer. Also, my thanks to Ramon Bastiaans and Bernard Li for their valuable suggestions and feedback. Further, this work would never have been accomplished without the help of various developers in the OSCAR and SGE community. Also, I thank the CS department at UH for providing me with the resources to develop and test the code that was produced as part of this summer work. 6