glite Grid Services Overview

Transcription

1 The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) glite Grid Services Overview Antonio Calanducci INFN Catania Joint GISELA/EPIKH School for Grid Site Administrators Valparaiso, Chile, 22th-26th November

2 What is a Grid? o Collection of geographically distributed heterogeneous resources. o Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources Valparaiso, School for Grid Site SysAdmin,

3 Grid Systems & Applications... Aims to Integrate Virtualize Manage RESOURCEs and SERVICEs acros different VOs o VO dynamic collection of multiple organizations providing coordinated resource sharing. o The glite middleware is the set of software packages able to do this. Valparaiso, School for Grid Site SysAdmin,

4 Grid User Valparaiso, School for Grid Site SysAdmin,

5 Grid overview Members of the grid can dynamically be organized into multiple virtual organizations. Each of these VOs may have different policies and administrative control. All the VOs are part of a large grid and can share resources accordingly to several policies. The resources shared among VOs may be data, special hardware, processing capabilities, Software and licenses. Members of a grid can be part of multiple VOs at the same time. Valparaiso, School for Grid Site SysAdmin,

6 Job life cycle Valparaiso, School for Grid Site SysAdmin,

7 What is glite? o The middleware for grid computing developed by EGEE. o Provides a framework for building grid applications tapping into the power of distributed computing and storage resources across the internet. o glite follows a Service Oriented Architecture (SOA). Ø Facilitate the interoperability among Grid services. Ø Allow easier compliance with upcoming standards. Ø The services work together in a concerted way but can also be deployed and used independently allowing their exploitation in different contexts. o Lightweight (existing) services. Ø Easily and quickly deployable. o Open source license. o The future : EMI (European Middleware Initiative) Valparaiso, School for Grid Site SysAdmin,

8 grid Terminology o Computational Resource o Physical machines on which users wants to run their programs and store (or access) data files, without worrying about their physical location o Job o A computational task (a binary application or script) that a user wants to run on the Grid, to retrieve results later o Job Submission o It is the action of delegating the application to the Grid middleware for its execution. Valparaiso, School for Grid Site SysAdmin,

9 glite Main components services o The User Interface (UI) is the user entry point. o The Workload Management System (WMS) is a set of services which are in charge of finding the suitable resources for the submitted jobs and schedule the jobs. o The Logging and bookkeeping service (LB) keeps track of user job execution in terms of statuses: Ready, Scheduled, Waiting, Running, Done. o The Computing element (CE) is the computational resource, the entry point to a cluster or PCs handled by a job queue management system; in particular: TORQUE/PBS, LSF, CONDOR, SGE Valparaiso, School for Grid Site SysAdmin,

10 glite Main components services o The Worker Nodes (WNs) are the machines where jobs are really executed. o The Information System and Monitoring (BDII) maintain data related to available grid resources and their health statuses, in order to optimize resource exploitation o The Virtual Organization Management service (VOMS) is the way glite improves the management of authentication and authorization to the Grid resources. The VOMS allows to their own members to define different access rights to VO s resources. o The Storage element (SE) and the File catalogue (LFC) allow to manage Grid files and offer a mechanism to locate them easily for users and jobs. Valparaiso, School for Grid Site SysAdmin,

11 glite Middleware Services Valparaiso, School for Grid Site SysAdmin,

12 glite Grid access Two possibilities: APIs or CLI Built on top of GridPortals and GUIs Valparaiso, School for Grid Site SysAdmin,

13 glite Security o Authentication based on X.509 PKI infrastructure Ø Certificate Authorities (CA) issue (long lived) certificates identifying individuals (much like a passport). Ø Certification Authorities (CA) can generate service certificates which identify Grid services or hosts in the whole Grid. Ø In order to reduce vulnerability, on the Grid user identification is done by using (short lived-12 hours) proxies of their certificates. o The use of Proxy certificates allows the following: Ø Delegation: in which a remote service acts on behalf of the user as long as the delegation remains valid. It is used to securely transfer user s credentials to a service. This happens by making signed copies of the original proxy. (Single Sign On).( VOMS Ø Add additional info (Add VO specific information provided by Ø Possible to store a long term proxy on a secure server (MyProxy). Valparaiso, School for Grid Site SysAdmin,

14 glite Security: AutH/AhtZ o o Authentication Ø The user receive a certificate from a CA. Ø He/She connects to the UI (typically via SSH). Ø He/She Creates the proxy (single sign on) All grid services will use this proxy to identify the user. Authorization Ø The user has to subscribe to a VO (VOMS). Ø The VO establishes the user rights. Ø In any Grid service it will be verified if the user belongs to the VO and assigns the proper access rights to the user. Valparaiso, School for Grid Site SysAdmin,

15 VOMS o Virtual Organization Membership Service is a service that keeps track of the members of a VO. o Support MyProxy (stored proxies). o VOMS Grants users authorization to access the resource at VO level. Valparaiso, School for Grid Site SysAdmin,

16 VOMS oit extends the proxy certificates with info on VO membership, groups, roles(e.g. administrator, software engineer, student) & capabilities. Ø Each VO has a database containing group membership, roles and capabilities information for each user. Ø User contacts VOMS server requesting his authorization information. Ø If request is legitimate, Server sends signed authorization info to the client. Ø Client includes them in a proxy certificate. Valparaiso, School for Grid Site SysAdmin,

17 Information System and Monitoring o What? Ø System to collect information on the state of resources. o Why? Ø To discover resources of the grid and their nature. Ø To check for health status of resources. Ø To provide data in order to manage the workload more efficiently. o How? Ø Monitoring and publishing fresh data on the state of resources. o Who? Ø User searching specific resources for their activity. Ø Workload Management System. Ø Other monitoring system Valparaiso, School for Grid Site SysAdmin,

18 Information System and Monitoring o The IS architecture used in glite is Berkeley Database Information Index (BDII): Ø Stores information at VO level. o Site GIIS (Grid Index Information Server): Ø Stores information at site level. o GRIS(Grid Resource Information Server): Ø Stores information at resource level. Valparaiso, School for Grid Site SysAdmin,

19 Information System and Monitoring Valparaiso, School for Grid Site SysAdmin,

20 Workload Management o WMS (Workload Management System) is a set of middleware components which responsible of distribution and management of jobs across Grid resources. o Two core components of WMS: Ø WM(Workload Manager): accepts and satisfy requests for job management. (Matchmaking) is the process of assigning the best available resource. Ø LB (Logging & Bookkeeping): keeps track of job execution in term of events: (Submitted, Running, Done,...) Valparaiso, School for Grid Site SysAdmin,

21 Computing Element o Service that represents the computing resource that is responsible of to manage the queue of jobs to execute. o CE responsible to collect accounting info. o The CE may be used by a Generic Client: an end-user interacting directly with the Computing Element, or by the Workload Manager, which submits a given job to an appropriate CE found by the matchmaking process. o Submission models : Ø PUSH (Eager Scheduling) (jobs pushed to CE) Valparaiso, School for Grid Site SysAdmin,

22 Computing Element o Job management through the WMS provides many benefits compared to direct job submission to the CE: The WMS manages multiple CEs, and is able to forward jobs to the one which better satisfies a set of requirements, which can be specified as part of the job description. The WMS can be instructed to handle job failures: if a job aborts due to problems related to the execution host, the WMS can automatically resubmit it to a different CE. The WMS provides a global job tracking facility using the LB service. The WMS supports complex job types (job collections, job with dependencies) which can not be handled directly by the CEs. Valparaiso, School for Grid Site SysAdmin,

23 Storage Element o The Storage Element is the service which allows a user or an application to store data for future retrieval. o To define a storage element, we need to know: Ø Storage Resource Manager (SRM). Ø Storage Resource Types. Ø Transfer Protocol. Valparaiso, School for Grid Site SysAdmin,

24 Storage Element o Storage Resource Manager (SRM): Ø It is a middleware interface application that makes standard data management operations between SEs of different resource type transparent to user. Ø These data management operations include: Ø File transfer. Ø Space reservation. Ø Renaming of files. Ø File directory creation. Valparaiso, School for Grid Site SysAdmin,

25 Storage Element o Storage Resource Types: 1. For relatively small SEs: Ø Disk-based storage implementation is employed together with disk pool manager (DPM), as the storage resource manager. 2. For bigger SEs: Ø The mass storage system (MSS) is implemented with CASTOR (CERN Advanced STORage Manager) as the storage resource manager. o For hybrids between disk pool storage and MSS, we have dcache as the storage resource manager. o Transfer Protocol: Ø Ø Ø To transfer of files in and out of the SE. Globus GridFTP mandatory. Others if available (https, ftp, etc). Valparaiso, School for Grid Site SysAdmin,

26 LFC File Catalog o keeping track of the location of files. o Organize files in a logical way so that a file will be accessible from anywhere. o The identification of files on the storage elements is done through the use of different identifiers: Ø Logical File Name (LFN) An alias created by a user to refer to some item of data, e.g. lfn:/ grid/gilda/ /run2/track1 Ø Globally Unique Identifier (GUID) A non-human-readable unique identifier for an item of data, e.g. guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6 ( (PFN) Ø Storage URL (SURL) (or Physical File Name The location of an actual piece of data on a storage system, e.g. srm://grid009.ct.infn.it/dpm/ct.infn.it/gilda/output10_1 ( TURL ) Ø Transport URL Contains access protocol and physical port information that allow for the physical retrieval of the file, done by SRM e.g. rfio:// lxshare0209.cern.ch//data/alice/ntuples.dat Valparaiso, School for Grid Site SysAdmin,

27 LFC File Catalog o while GUIDs and LFNs are used for the identification of files, storage URLs and transport URLs provide the necessary information to access and retrieve the files concerned o Resolves logical filenames (LFN) to physical location of files (URL understood by SRM) and storage elements. Valparaiso, School for Grid Site SysAdmin,

28 Metadata o o o o o AMGA Arda Metadata Grid Application Metadata is data about data. On the Grid: information about files Ø Ø Describe files. Locate files based on their contents through queries Metadata can describe any grid entity/object Ø ex: JobIDs - add logging information to your jobs Monitoring of running applications: Ø ex: ongoing results from running jobs can be published on the metadata server Simplified DB access on the grid Ø Grid applications that needs structured data can model their data schemas as metadata. Valparaiso, School for Grid Site SysAdmin,

29 Questions Valparaiso, School for Grid Site SysAdmin,