Parallel Computing in EGI

Transcription

1 Parallel Computing in EGI V. Šipková, M. Dobrucký, and P. Slížik Ústav informatiky, Slovenská akadémia vied Bratislava, Dúbravská cesta 9 {Viera.Sipkova, Miroslav.Dobrucky, Peter.Slizik}@ savba.sk Abstract. EGI.eu is a foundation established in February 2010 to create and maintain a pan-european Grid Infrastructure (EGI) so as to guarantee the long-term availability of a generic e-infrastructure for all European research communities and their international collaborators. Its work builds on previous EU-funded grid projects: LHC, DataGrid, EGEE, a.o. EGI does not develop the software deployed in the grid infrastructure, all upgrades and new programs are produced with external technology providers. Concerning the compute area, the major highlights of the first release of the EGI middleware - EMI1, are improvements in the CREAM service, extensions in the JDL language, and the support for the user defined fine-grained mapping of processes to physical resources. This makes possible for grid applications to employ various parallel programming models. This work presents the overview of the current job management facilities provided by the EGI middleware components. 1 Introduction EGI.eu is a foundation established in February 2010 to create and maintain a pan-european Grid Infrastructure (EGI) [1], in collaboration with National Grid Initiatives (NGIs) and European International Research Organizations (EIROs), so as to guarantee the long-term development, availability and sustainability of grid services and e-infrastructure for all European research communities and their international partners. Its work builds on previous EU-funded projects which raised this goal from the initial concept of a scalable, federated, distributed computing system. The distributed computing grid was originally conceived in 1999 to analyze the experimental data produced by the particle accelerator Large Hadron Collider (LHC) at CERN (European Organization for Nuclear Research). The research and development of grid technologies started in January 2001 within the European Data Grid [2] project which proved the successful application of the grid in research fields of the high energy physics, earth observation and bioinformatics. Upon its completion in 2004, a new project, called Enabling Grid for E-sciencE (EGEE) [3], took over the grid s further development. EGEE allowed researchers the access to computing resources on demand, from anywhere in the world and at any time of the day. By April 2010 when the last project phase

2 was finished, there were about 13 million jobs per month running on the EGEE infrastructure, hosted by a network of 300 computer centers worldwide. A special role in EGI is covered by the 4-year project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe (EGI-InSPIRE) [4], which mission is to coordinate the transition from EGEE to EGI and to provide support for the current and emerging user communities. When starting in May 2010, EGI-InSPIRE has gathered 51 beneficiaries, 141 partners, 39 National Grid Initiatives within Europe and 47 countries around the world. EGI-InSPIRE is also supporting the integration of new distributed computing infrastructures such as clouds, supercomputing networks and desktop grids. EGI itself does not develop the software deployed in the grid infrastructure, all upgrades and new programs are produced with the independent technology providers. Up to now, EGI.eu has formalized and contracted agreements with the following external partners: European Middleware Initiative (EMI) [5] - the primary goal of EMI is to deliver a consolidated set of grid middleware components (as part of the Unified Middleware Distribution, UMD) to extend the interoperability and integration with emerging computing models, and to strengthen the reliability, stability and manageability of the middleware services. Initiative for Globus in Europe (IGE) [6] - the IGE project aims to be a comprehensive service provider for the European e-infrastructure regarding the development, customization, provisioning, support, and maintenance of components of the grid middleware Globus. Simple API for Grid Applications (SAGA) [7] - SAGA is a programming abstraction that offers the basic functionality required to build distributed applications, tools and frameworks, so as to be independent of the details of the underlying middleware systems and infrastructure. StratusLab [8] - the StratusLab project was set up to develop a complete, open-source cloud distribution that allows grid and non-grid resource centers to offer and to exploit an infrastructure as a Service cloud. It focuses on enhancing distributed computing infrastructures such as EGI with virtualisation and cloud technologies. The grid world has a lot of specialized jargon, some of acronyms frequently used in this paper are explained below: CE CREAM JDL MPI OpenMP SE VOMS WN WMProxy WMS Computing Element Computing Resource Execution And Management Job Description Language Message Passing Interface Open Multi-Processing Storage Element Virtual Organization Membership Service Worker Node Workload Manager Proxy Workload Management System

3 2 EGI Middleware A grid middleware is a specific software product, placed between the infrastructure and user applications, which enables sharing the heterogeneous grid resources. Resources include commodity or HPC clusters, disk storage, various instruments, data archives or digital libraries, and software packages. For the first period of its existence, EGI has operated using the grid middleware glite3.2 [9]. During the 6 years of EGEE, glite components have been progressively improved and made increasingly robust and efficient to satisfy the requirements of a large variety of research communities. In May 2011 EMI delivered the first release of the grid middleware: EMI1 Kebnekaise [10], which features the first complete and consolidated set of components from glite [9], ARC [11], dcache [12] and UNICORE [13]. The reference platform for EMI1 is the Scientific Linux 5/64bit. EMI1 is focussing primarily on laying the foundations for the distribution, increasing the level of integration among the original middleware stacks, improving the compatibility with mainstream operating systems guidelines, and extending the compliance with existing standards. EMI 1 has introduced a number of changes and new functionalities in areas of the Security, Computing, Data and Infrastructure. Security - includes middleware services and components that enable and enforce the grid security model allowing the safe sharing of resources on a large scale. The major highlights comprise: the replacement of GSI with SSL in security components, most notably VOMS; the REST-based interface for obtaining X.509 attribute certificates in VOMS Admin service; and the initial integration of ARGUS authorization with middleware services. Computing - includes middleware services and corresponding client components involved in the processing and management of user requests concerning the execution of a computational task. The major highlights comprise: the full support for the CLUSTER service in CREAM; the initial support for GLUE 2 in all CEs; the integration of ARGUS authorization in CREAM; and the initial implementation of common MPI methods across the different compute services. In MPI it includes the support for the user defined fine-grained mapping of processes to physical resources, the basic support for OpenMP, new command line options for MPI-Start, and the support for SLURM& Condor schedulers. Data - includes middleware services and corresponding client components involved in the processing and management of user requests concerning storage management, data access and data replication. The major highlights comprise: the adoption of pnfs4.1 and WebDAV standards in dcache; and the preview version of a messaging-based SE File Catalog synchronization service (SEMsg). Infrastructure - includes middleware services and components that offer a common information and management functionality to deployed grid services. They involve the Information System and Service Registry, the grid

4 messaging infrastructure, the service monitoring and management, Logging and Bookkeeping services, and the accounting functionality. The major highlights are: the ARC CE, UNICORE Web services, CREAM service, dcache, and improvements in the Disk Pool Manager (DPM) via the experimental adoption of the GLUE 2 information model standard. The following sections address the glite component of the EGI middleware. 3 Job Management Services Job management services in the grid are concerned with the acceptance, scheduling, monitoring, and management of remote computations, called jobs. A job consists of a computation, and optionally, file transfer and management operations related to the computation. In grid terminology, a Computing Element (CE) represents a set of computational resources localized at a site (i.e. a cluster). CE consists of: a Grid Gate (GG) acting as a generic interface to the cluster, a Local Resource Management System (LRMS), called also batch system, the Cluster itself: a collection of Worker Nodes (WNs) where jobs are run. The central role of GG is accepting jobs and dispatching them for execution on WNs via the LRMS. The GG implementation in glite3.2 is the CREAM based CE [14] - a lightweight service, which is responsible for performing all the job management operations. CREAM accepts job submissions and other requests through the Workload Management System (WMS) [15], or through a generic client (e.g. an end-user willing to directly submit jobs to a CREAM-CE). WMS is a software service of the glite which takes care of distributing and managing tasks across computing and storage resources available in the grid. As a global grid resource broker, WMS forms a reliable and efficient entry point to high-end services on the grid presenting a common front-end for user job submissions. The main service providing access to the WMS is the WMProxy. Both services the CREAM and WMProxy implement similar functionalities and expose a web services interface that the user can interact with by means of the Command Line Interface. The client commands enable to perform the following operations: delegation of proxy credentials to speed up subsequent operations; renewal of delegations (i.e. proxy of submitted jobs) match-making - to display the list of CEs which match the JDL requirements submission of jobs for execution monitoring the status of submitted jobs cancellation of jobs at any point in their life cycle suspension/resumption of running jobs retrieval of the job execution logging and the output of finished jobs getting information about jobs, WMS/CREAM services, and CEs The following tables summarize the most relevant client commands of WMProxy and CREAM services.

5 WMProxy Client Commands glite-wms-delegate-proxy glite-wms-job-submit glite-wms-job-status glite-wms-job-cancel glite-wms-job-output glite-wms-job-perusal glite-wms-job-logging-info glite-wms-job-list-match glite-wms-job-info CREAM Client Commands glite-ce-delegate-proxy glite-ce-job-submit glite-ce-job-status glite-ce-job-cancel glite-ce-job-output glite-ce-job-suspend glite-ce-job-resume glite-ce-job-purge glite-ce-proxy-renew glite-ce-job-list glite-ce-service-info 4 Description of Grid Jobs Jobs to be submitted to the grid must be described using the Job Description Language (JDL) [16]. JDL is a flexible, high-level language based on the Condor Classified Advertisements which enables to describe jobs and aggregates of jobs with arbitrary dependency relations, and to express any requirements and constraints on the CE, WNs, SE and installed software. In general, JDL attributes hold request-specific information which designate in some way actions that have to be performed. As the JDL is an extensible language, the user is allowed to use whatever attribute for the description of a job request, however, only a certain subset of attributes is accepted. Some of the attributes are mandatory, without specifying them the service cannot handle the request. A typical example of a JDL file for a simple job looks like as follows: Type = "Job"; JobType = "Normal"; Executable = "start-mytest.sh"; Arguments = "mytest.exe myinput.dat myoutput.dat"; StdOutput = "stdout.txt"; StdError = "stderr.txt"; InputSandbox = {"start-mytest.sh","mytest.exe","myinput.dat"}; OutputSandbox = {"stdout.txt","stderr.txt","myoutput.dat"}; The Type attribute represents the type of the request, and JobType defines the type of the job. The Executable attribute specifies the name of the executable/command to be carried out, and Arguments specifies its input arguments. StdOutput/StdError assign the names to standard streams(output/error) of the job. The Shell-script start-mytest.sh is supposed to invoke the actual

6 executable mytest.exe with arguments myinput.dat and myoutput.dat. Files which should be transfered between the client User Interface machine or external data SE and the compute resource before/after the job execution must be identified by InputSandbox/OutputSandbox attributes. Comparing the sets of JDL attributes supported by the WMS [17] and CREAM service [18], there is rather big difference between them. For instance, unlike CREAM which supports at the moment only simple job requests, the WMS can handle also compound job structures: Type = "Job"; # a simple job Type = "DAG"; # a DAG of dependent jobs Type = "Collection"; # a set of independent jobs A Directed Acyclic Graph (DAG) is defined as a set of jobs where the input/output/execution of one of more jobs may depend on one or more other jobs. A Collection is defined as multiple independent jobs with a common description. WMS currently supports two job types: JobType = "Normal"; # a simple batch-job JobType = "Parametric"; # a job with parametric attributes where the Parametric job represents multiple jobs with one parameterized description. CREAM allows for the time being only the Normal job type. A special class is constituted by parallel MPI and multi-threaded OpenMP tasks. These are included into the type Normal, but they need special handling. 4.1 MPI Jobs The MPI (Message Passing Interface) [19] has become a standard for programming distributed memory systems. In case of an MPI job, the JDL file must contain the attribute CpuNumber (an integer greater than 1) which defines the number of CPUs to be allocated. The Requirements attribute may be used to force the selection of such sites which have the necessary hardware configuration and MPI software support. The MPI application itself need to be initialized by means of an executable script, for instance, through the invocation of MPI-Start [20]. The MPI-Start is an abstraction layer that offers a unique interface to the grid middleware to start MPI programs with various execution environment implementations. It has already become a fixed part of the EMI1 middleware. Using plugins it supports different combinations of execution environments (Open MPI, MPICH, MPICH2, LAM-MPI, PACX-MPI) and batch schedulers (PBS/Torque, SGE, LSF). Besides the capability to start jobs, MPI-Start provides a hooks framework which makes possible customizing the MPI-Start behavior, the simple file distribution for sites without a shared file system, and for user applications to perform any pre-processing (e.g. program compilation, data fetching) and/or post-processing (e.g. storage of application results, clean-up) actions. MPI-Start can be controlled via environment variables or command line switches. The following sample shows a JDL example for an MPI job:

7 Type = "Job"; JobType = "Normal"; CpuNumber = 8; Executable = "start-mytest_mpi.sh"; Arguments = "8 mytest_mpi.exe";... Requirements = other.gluehostarchitectureplatformtype == "x86_64" && other.glueceinfototalcpus >= 8 && Member("MPI-START", other.gluehostapplicationsoftwareruntimeenvironment) && Member("OPENMPI", other.gluehostapplicationsoftwareruntimeenvironment); The Requirements attribute will be used during the matchmaking process for selecting those CEs having a 64bit architecture with at least 8 CPUs, and MPI- Start with Open MPI software installed. 4.2 OpenMP Jobs The OpenMP (Open Multi-Processing) - is an Application Programming Interface [21] that provides a parallel programming model for shared memory architectures ranging from the standard desktop computers to supercomputers. For the time being the submission of parallel OpenMP jobs is possible only via the CREAM clients. As the CREAM has become an integrated part of the EMI execution service, the CREAM JDL is constantly varying and extending in order to better address new requirements and scenarios. For example, an 4-threaded OpenMP job may be described in the following way: Type = "Job"; JobType = "Normal"; HostNumber = 1; WholeNodes = True; SMPGranularity = 4; Executable = "mytest_omp.exe"; Arguments = "4"; Environment = {"OMP_NUM_THREADS=4"};... Requirements = other.gluehostarchitecturesmpsize >= SMPGranularity; The HostNumber attribute defines the number of nodes the user wishes to obtain for his job. The SMPGranularity specifies the number of cores that any host involved in the allocation has to dedicate to the job, it corresponds to the number of slave-threads which will be forked off by the master-thread. The WholeNodes is a boolean variable which indicates whether whole nodes should be used exclusively or not.

8 5 Conclusion Current computing systems allow applying of many parallel solutions at the same time to achieve the maximum performance and efficiency gains. The EGEE project was concentrated first of all on providing a computing infrastructure and on running distributed applications, the support for parallel applications was rather poor. In July 2011, EMI1 Kebnekaise has been released by EGI as the first version of the UMD 1.0.0, which delivers a significant number of new features. The support for new CREAM JDL attributes and the fine-grained mapping of MPI processes and OpenMP threads to physical resources extend the variety of applications which may be grid-enabled, and improve the resource utilization as well. Acknowledgements. This work was partially supported by the following projects: EGI-InSPIRE RI , VEGA No. 2/0211/09, and ASFEU OPVV CRISIS ITMS References 1. EGI - European Grid Infrastructure EDG - European Data Grid EGEE - Enabling Grid for E-sciencE EGI-InSpire - Integrated Sustainable Pan-European Infrastructure for Researchers in Europe EMI - European Middleware Initiative IGE - Initiative for Globus in Europe SAGA - Simple API for Grid Applications StratusLab project S. Burke, et al.: glite3.2 User Guide, April 19, EMI 1 Kebnekaise ARC - Advanced Resource Connector dcache/srm - Storage Middleware System UNICORE - Uniform Interface to Computing Resources EGEE User s Guide - CREAM Service, January 31, EMI User s Guide - WMS Command Line Interface, June 30, Fabrizio Pacini: Job Description Language HowTo, December 17, http: // 2-Document.pdf 17. Fabrizio Pacini: Job Description Language Attributes Specification (for the glite WMS), March 11, CREAM Job Description Language Attributes Specification, February 28, MPI - Message Passing Interface Standard MPI-Start and MPI-Utils, EMI Document 1.0.1, Jun 14, OpenMP - Open Multi-Processing, API Specification for Parallel Programming.