The EU DataGrid Fabric Management

Size: px

Start display at page:

Download "The EU DataGrid Fabric Management"

Helen Anderson
5 years ago
Views:

1 The EU DataGrid Fabric Management The European DataGrid Project Team DataGrid is a project funded by the European Union Grid Tutorial 4/3/2004 n 1

2 EDG Tutorial Overview Workload Management Services Data Management Services Networking Information Service Fabric Management Grid Tutorial 4/3/2004 Fabric Management n 2

3 Contents Definition Fabric Management architecture overview User job management overview Present status (R2.0) LCFG LCAS + edg_gatekeeper Fmon (Fabric MONitoring) RMS ( Management System) Grid Tutorial 4/3/2004 Fabric Management n 3

4 What is Fabric Management Definitions: Cluster (or Farm): A collection of computers on a network that can function as a single computing resource through the use of tools which hide the underlying physical structure. Fabric: "A complete set of computing resources (processors, memory, storage) operated in a coordinated fashion with a uniform set of management tools". Functionality: Enterprise system administration scalable to ~10K nodes Provision for running grid jobs Provision for running local jobs Grid Tutorial 4/3/2004 Fabric Management n 4

5 Architecture logical overview Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt Fabric Gridification Local User Management Monitoring & Fault Tolerance Farm A (LSF) Farm B (PBS) Grid Data Storage Configuration Management (Mass storage, Disk pools) Installation & Node Mgmt Grid Tutorial 4/3/2004 Fabric Management n 5

6 Architecture logical overview Interface Interface between between Gridwide Gridwide services services local local fabric; fabric; Provides Provides local local Grid User authentication, authentication, authorization authorization mapping mapping of of grid grid credentials. credentials. Data Mgmt Broker Fabric Gridification Grid Info Services Fabric mgt subsystems Other services Local User Management Monitoring & Fault Tolerance Farm A (LSF) Farm B (PBS) Grid Data Storage Configuration Management (Mass storage, Disk pools) Installation & Node Mgmt Grid Tutorial 4/3/2004 Fabric Management n 6

7 Architecture logical overview Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt provides provides transparent transparent access (WP2) access (both (both job job admin) admin) to to different different cluster cluster batch batch systems; systems; enhanced enhanced capabilities capabilities (extended (extended Local User scheduling scheduling policies, policies, advanced advanced reservation, reservation, local local accounting). accounting). Grid Data Storage (Mass storage, Disk pools) Fabric Gridification Management Farm A (LSF) Installation & Node Mgmt Farm B (PBS) Monitoring & Fault Tolerance Configuration Management Grid Tutorial 4/3/2004 Fabric Management n 7

8 Architecture logical overview Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt Fabric Gridification Local User provides provides the the tools tools to to install install manage manage all all software Grid Data software running running on on the the fabric Storage fabric nodes; nodes; (WP5) Agent Agent to to install, install, upgrade, upgrade, remove (Mass storage, remove configure configure software Disk pools) software packages packages on on the the nodes. nodes. bootstrap bootstrap services services software software repositories. repositories. Management Farm A (LSF) Installation & Node Mgmt Farm B (PBS) Monitoring & Fault Tolerance Configuration Management Grid Tutorial 4/3/2004 Fabric Management n 8

9 Architecture logical overview Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt Local User provides provides a Fabric a central central storage storage Gridification management management of of all all fabric fabric configuration configuration information; information; central central DB DB set set of of protocols protocols APIs APIs to to store Management store retrieve retrieve information. information. Monitoring & Fault Tolerance Farm A (LSF) Farm B (PBS) Grid Data Storage Configuration Management (Mass storage, Disk pools) Installation & Node Mgmt Grid Tutorial 4/3/2004 Fabric Management n 9

10 Architecture logical overview Grid User Data Mgmt Broker Fabric Gridification Grid Info Services provides provides the the tools tools for for gathering gathering monitoring monitoring information information on on fabric fabric nodes; nodes; central WP4 subsystems central measurement measurement repository repository stores stores all all monitoring Other Wps monitoring information; information; fault fault tolerance tolerance correlation correlation engines engines detect detect failures failures trigger trigger recovery recovery actions. actions. Local User Management Monitoring & Fault Tolerance Farm A (LSF) Farm B (PBS) Grid Data Storage Configuration Management (Mass storage, Disk pools) Installation & Node Mgmt Grid Tutorial 4/3/2004 Fabric Management n 10

11 User job management (Grid local) Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt Fabric Gridification Local User Management Monitoring Farm A (LSF) Farm B (PBS) Grid Data Storage (Mass storage, Disk pools) Grid Tutorial 4/3/2004 Fabric Management n 11

12 User job management (Grid local) Submit Submit job job Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt Fabric Gridification Local User Management Monitoring Farm A (LSF) Farm B (PBS) Grid Data Storage (Mass storage, Disk pools) Grid Tutorial 4/3/2004 Fabric Management n 12

13 User job management (Grid local) Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt Fabric Gridification publish publish resource resource accounting accounting information information Management Monitoring Local User Farm A (LSF) Farm B (PBS) Grid Data Storage (Mass storage, Disk pools) Grid Tutorial 4/3/2004 Fabric Management n 13

14 User job management (Grid local) Grid User Broker Grid Info Services Fabric mgt subsystems Other services Optimized Optimized selection selection of of site site Data Mgmt Fabric Gridification Local User Management Monitoring Farm A (LSF) Farm B (PBS) Grid Data Storage (Mass storage, Disk pools) Grid Tutorial 4/3/2004 Fabric Management n 14

15 User job management (Grid local) Grid User Broker Grid Info Services Fabric mgt subsystems Other services Authorize Authorize Map Map grid grid local local credentials credentialsdata Mgmt Fabric Gridification Local User Management Monitoring Farm A (LSF) Farm B (PBS) Grid Data Storage (Mass storage, Disk pools) Grid Tutorial 4/3/2004 Fabric Management n 15

16 User job management (Grid local) Grid User Broker Grid Info Services Fabric mgt subsystems Other services Data Mgmt Select Select an an optimal optimal (WP2) batch batch queue queue submit submit Return Return job job status status output output Local User Fabric Gridification Management Monitoring Farm A (LSF) Farm B (PBS) Grid Data Storage (Mass storage, Disk pools) Grid Tutorial 4/3/2004 Fabric Management n 16

17 Fabric Release 2.0 Installation configuration: LCFG (Local ConFiGuration system) Gridification: LCAS + edg_gatekeeper Monitoring FMON Management RMS Grid Tutorial 4/3/2004 Fabric Management n 17

18 LCFG (Local ConFiGuration system) LCFG is originally developed by the Computer Science Department of Edinburgh University Widely used fabric management tool, whose purpose is to hle automated installation configuration in a very diverse evolving environment Basic features: automatic installation of O.S. installation/upgrade/removal of all (rpmbased) software packages centralized configuration management of machines extendible to configure manage EDG middleware custom application software LCFGng: updated version of LCFG, customized to EDG needs. This is the version used at EDG 2.0 Grid Tutorial 4/3/2004 Fabric Management n 18

19 LCFGng system architecture Config files LCFGng Config Files Make XML Profile HTTP Web Server XML Profile +inet.services telnet login ftp +inet.services telnet login ftp +inet.allow telnet login ftp sshd +inet.allow telnet login ftp sshd +inet.allow_telnet ALLOWED_NETWORKS +inet.allow_telnet Read ALLOWED_NETWORKS Load +inet.allow_login ALLOWED_NETWORKS +inet.allow_login Profile rdxprof ALLOWED_NETWORKS Profile ldxprof +inet.allow_ftp ALLOWED_NETWORKS +inet.allow_ftp ALLOWED_NETWORKS +inet.allow_sshd ALL +inet.allow_sshd ALL Profile Generic +inet.daemon_sshd yes +inet.daemon_sshd yes Object Component auth.users mickey +auth.users mickey +auth.userhome_mickey Local cache /home/mickey +auth.userhome_mickey /home/mickey +auth.usershell_mickey /bin/tcsh +auth.usershell_mickey /bin/tcsh LCFG Objects Server Client nodes Abstract configuration parameters for all nodes stored in a central repository A collection of agents read configuration parameters either generate traditional config files or directly manipulate various services Grid Tutorial 4/3/2004 Fabric Management n 19

20 LCFGng system architecture LCFGng Config Files Make XML Profile Server Abstract configuration parameters for all nodes stored in a central repository HTTP Web Server XML Profile Read Profile rdxprof XML profiles Load Profile ldxprof <inet> <inet> <allow cfg:template="allow_$ tag_$ daemon_$"> <allow cfg:template="allow_$ Profile Generic tag_$ daemon_$"> <allow_record cfg:name="telnet"> <allow_record cfg:name="telnet"> Object Component <allow> , </allow> <allow> , </allow> </allow_record> </allow_record> Local cache </auth> LCFG Objects </auth> <user_record cfg:name="mickey"> <user_record cfg:name="mickey"> <userhome>/home/mickeymousehome</userhome> <userhome>/home/mickeymousehome</userhome> Client nodes <usershell>/bin/tcsh</usershell> <usershell>/bin/tcsh</usershell> </user_record> </user_record> A collection of agents read configuration parameters either generate traditional config files or directly manipulate various services Grid Tutorial 4/3/2004 Fabric Management n 20

21 LCFGng system architecture LCFGng Config Files HTTP Read Profile rdxprof Load Profile ldxprof Make XML Profile Web Server XML Profile Profile Generic Profile Object Component Local cache inet auth LCFG Objects Server Client nodes Abstract configuration parameters for all nodes stored in a central repository A collection of agents read configuration parameters either generate traditional config files or directly manipulate various services Grid Tutorial 4/3/2004 Fabric Management n 21

22 LCFGng system architecture LCFGng Config Files HTTP Read Profile rdxprof Load Profile ldxprof /etc/shadow /etc/shadow Make XML /etc/group Profile /etc/group Server Web Server XML Profile /etc/passwd /etc/passwd mickey:x:999:20::/home/mickey:/bin/tcsh mickey:x:999:20::/home/mickey:/bin/tcsh Profile Generic Profile Object Component Local cache inet auth LCFG Objects Client nodes Abstract configuration parameters for all nodes stored in a central repository A collection of agents read configuration parameters either generate traditional config files or directly manipulate various services Grid Tutorial 4/3/2004 Fabric Management n 22

23 LCFGng system architecture LCFGng Config Files HTTP Read Profile rdxprof Load Profile ldxprof Make XML Profile Web Server XML Profile Profile Generic Profile Object Component Local cache inet auth LCFG Objects Server Client nodes Abstract configuration parameters for all nodes stored in a central repository A collection of agents read configuration parameters either generate traditional config files or directly manipulate various services Grid Tutorial 4/3/2004 Fabric Management n 23

24 LCFGng: what s a component? It's a simple Perl script Each component provides a Configure() method invoked on startup or when configuration changes A simple typical component behavior: Started when notified of a configuration change Loads its configuration (locally cached) Configures the appropriate services, by translating config parameters into a traditional config file reloading a service if necessary (e.g. restarting a init.d service). LCFGng provides components to manage the configuration of services of a machine: inet, auth, nfs, cron,... Admins/mw developers can build new custom components to configure manage their own applications Grid Tutorial 4/3/2004 Fabric Management n 24

25 LCFGng component template #!/usr/bin/perlw package = qw(lcfg::component); use strict; use LCFG::Component; use LCFG::Config; # EDG specific sub Configure($$@) { my ($self,$res) my $config=lcfg::config>new($res); my $age=$config>getvalue('/persons/john/age'); $self>fail( "too young") unless ($age>18); } new LCFG::MyComp() > Dispatch(); Grid Tutorial 4/3/2004 Fabric Management n 25

26 LCFGng component template (II) Configure() method: # Do the configuration sub Configure($$@) { } my ($self,$res) my $config=lcfg::config>new($res); # do whatever is necessary to reconfigure your service Grid Tutorial 4/3/2004 Fabric Management n 26

27 LCFGng Component Example (I) Edglcfgsyslog configures /etc/syslog.conf.def add_$ Additions add_$ s defined on server: syslog.additions monitoring kernel syslog.add_monitoring /var/obj/tmp/monitor.fifo syslog.add_kernel local3.* kern.* /var/log/kernel.log Grid Tutorial 4/3/2004 Fabric Management n 27

28 LCFGng Component Example (II) Component: Configure() method. First part: template generation sub { my my $config=lcfg::config>new($res); my $syslogconf='/etc/syslog.conf'; my $status = LCFG::Template::Substitute ( '/usr/lib/lcfg/conf/syslog/template', '/var/obj/conf/syslog/config', 0, $res ); unless (defined($status)) { $self>logmessage($@); $self>fail( "failed to create config file (see logfile)"); } Grid Tutorial 4/3/2004 Fabric Management n 28

29 LCFGng Component Example(III) Component: Configure() method. Second part: add missing resources #extra values from 'additions', using NVA API my $additions=$config>getelement('/additions'); while ($additions>isnextelement()) { my $add=$additions>getnextelement(); my $add_el=$config>getelement('/additions/'.$add.'/add'); push(@add_array,$add_el>getvalue()); } if (scalar (@add_array)) { open (CFG, '>>/var/obj/conf/syslog/config') $self>fail('cannot open config file (see logfile)'); print CFG join("\n",@add_array)."\n"; close (CFG) $self>fail('cannot close config file (see logfile)'); } Grid Tutorial 4/3/2004 Fabric Management n 29

30 LCFGng Component Example(IV) Component: Configure() method. Third part: copy config file, restart service # copy over to definitive location if files are different if (system("/usr/bin/cmp s $syslogconf var/obj/conf/syslog/config")){ $self>logmessage("updating config file"); system('cp f '.$syslogconf.' '.$syslogconf.'.old'); system('cp f /var/obj/conf/syslog/config '.$syslogconf) && $self>fail("copying template to $syslogconf: ". $?); } # restart service if (system('/sbin/service syslog restart')) { $self>fail('init.d syslog restart failed: '. $?); } return 1; #OK } Grid Tutorial 4/3/2004 Fabric Management n 30

31 Software distribution with LCFGng It is done with an LCFGng tool called updaterpms. The stard system packaging format is used: rpm for Linux It is managed via an LCFGng component. Functionality: 1. Compares the packages currently installed on the local node with the packages listed in the configuration 2. Computes the necessary install/deinstall/upgrade operations 3. Orders the transaction operations taking into account rpm dependency information 4. Invokes the packager (RPM) with the right operation transaction set Packager functionality: 1. Read operations (transactions) 2. Downloads new packages from repository 3. Executes the operations (installs/removes/upgrades) Grid Tutorial 4/3/2004 Fabric Management n 31

32 Gridification Architecture External to fabric Grid Scheduler (WP1) Other WP4 fabric non gridification mgt component Gridification component Internal to fabric FabNAT Globus Gatekeeper request in JDL In VOMS signed, established security context RMS Job repository ComputingElement SE StorageElement farms Policy LCMAPS uid/gid other tokens Credential Rep. LCAS plug ins static list wallclocktime quota check resource use use Configuration Mgmt, Installation Mgmt Policy (Configuration Mgmt) FLIDS Policy (Configuration Mgmt) Grid Tutorial 4/3/2004 Fabric Management n 32

33 LCAS 1.0 (EDG release 1.2) The Local Centre Authorization Service (LCAS) hles authorization requests to the local computing fabric. Authorization plugin framework, library. The authorization decision of the LCAS is based upon the users' proxy certificate the job specification in RSL (JDL) format. The certificate RSL are passed to (plugin) authorization modules, which grant or deny the access to the fabric. Three stard authorization modules are provided by default: lcas_userallow.mod, checks if user is allowed on the fabric (currently the gridmap file is checked). lcas_userban.mod, checks if user should be banned from the fabric. lcas_timeslots.mod, checks if fabric is open at this time of the day for datagrid jobs. Grid Tutorial 4/3/2004 Fabric Management n 33

34 Authentication control flow EDG gatekeeper LCAS accept policy C=IT/O=INFN /L=CNAF /CN=Pinco Palla /CN=proxy Original Gatekeeper TLS auth assist_gridmap Jobmanager* VOMS pseudo cert GSI AuthN LCAS authz call out LCMAPS open, learn, &run: return legacy uid Job Manager fork+exec args, submit script allowed timeslot banned Grid Tutorial 4/3/2004 Fabric Management n 34

35 Fabric Monitoring architecture Monitored nodes Measurement Repository (MR) Database Sensor Sensor Sensor Monitoring Sensor Agent (MSA) Consumer Local Consumer Cache Consumer Consumer Global Consumer Grid Tutorial 4/3/2004 Fabric Management n 35

36 Monitored nodes Sensors measure Metrics mainly locally: CPU utilization, network throughput, daemons status, etc. MSA Monitoring Sensor Agent: collects sensors data, stores them in a local cache sends them to the central repository triggers measurements according to the configured schedule. Local Cache: allows local consumers to have fast access to monitoring information. Data can be collected even if the node istemporally isolated from central repository. Outgoing data consist of a value associated to a metric identifier (what), a timestamp (when), a target identifier (where), an agent identifier (who). Grid Tutorial 4/3/2004 Fabric Management n 36

37 Measurement Repository Receives samples from all the nodes Stores data: Plain text DB Oracle DB Follows data up to subscribers Answers queries information Consumers can subscribe to metrics receive notification when new samples are available Grid Tutorial 4/3/2004 Fabric Management n 37

38 RMS Management System Grid Tutorial 4/3/2004 Fabric Management n 38

39 RMS functionality The Management System (RMS) is responsible for the management of local user jobs, grid user jobs maintenance jobs within a site, that is build of one to many clusters. It enhances capabilities of the Globus Toolkit job management that submits jobs to a specific resource. The enhancements enabled by the RMS include: advanced scheduling mechanisms, e.g. backfill, fair share support for advance reservations support for maintenance tasks, e.g. node on/off. Support to the EDG grid accounting infrastructure Grid Tutorial 4/3/2004 Fabric Management n 39

40 Summary Fabric management components deployed on release 2.0: Installation Configuration management functionality: LCFG (Local ConFiGuration system): hles automated installation, configuration management of machines. Gridification functionality: LCAS (Local Centre Authorization Service) + edg_gatekeeper: hle authorization requests to the local computing fabric. Fabric monitoring functionality: FMON: provides a client (MSA) running sensors on each node to monitor, a central server to collect data. Management functionality: RMS: management of local user jobs, grid user jobs maintenance jobs within a site, built from one to many clusters Grid Tutorial 4/3/2004 Fabric Management n 40

Grid Authentication and Authorisation Issues. Ákos Frohner at CERN

Grid Authentication and Authorisation Issues Ákos Frohner at CERN Overview Setting the scene: requirements Old style authorisation: DN based gridmap-files Overview of the EDG components VO user management: