YAIM Overview. Bruce Becker Meraka Institute. Co-ordination & Harmonisation of Advanced e-infrastructures for Research and Education Data Sharing

Transcription

1 Co-ordination & Harmonisation of Advanced e-infrastructures for Research and Education Data Sharing Research Infrastructures Grant Agreement n YAIM Overview Bruce Becker Meraka Institute

2 Outline What is YAIM? History, status, prospects, design What is it used for? How is it organised? How should we use it? Practical examples

3 Short history European DataGrid project ( ) WP4 Fabric Management Mandate to to deliver a computing frabric comprised of all the necessary tools to manage a centre providing grid services on clusters of thousands of nodes Produced LCFG (Local ConFiGuration system) Evolved into LFCng (next generation, client-server architecture) subsequently Quattor community effort, open source software EGEE projects (I, II, III ) Adopted YAIM Yet Another Installation Manageer YAIM Ain't an Installation Manager

4 YAIM Present and Future YAIM in EMI : All module the responsibility of product teams yaim-core/clients : mainly bug fixes EMI reccomendation : Adopt standard configuration mechanisms suggested by the operating systems supported YAIM post EMI : Up to product teams. WLCG TEG Operations and Tools final report : middleware configuration should be improved and not based on one single configuration management tool

5 YAIM Design YAIM Implements a configuration method for glite software Aims to be simple configuration method that can be used to set up simple grid sites, but can also easily be adapted to meet the needs of larger sites Main features Modular structure Bash syntax Node type configuration facility Well-defined directory structure Basic configuration files

6 YAIM modular structure Yaim-core (glite-yaim-core) Common functions and definitions One yaim module per node type/service Specfic functions and configuration files 21 modules (bold are used in SAGrid) dpm-yaim glite-apel-yaim glite-ce-yaim-cream-ce glite-lb-yaim glite-px-myproxy-yaim glite-yaim-bdii glite-yaim-clients glite-yaim-cluster glite-yaim-ge-utils glite-yaim-lsf-utils glite-yaim-mpi glite-yaim-torque-client glite-yaim-torque-server glite-yaim-torque-utils glite-yaim-wms lfc-yaim yaim-argus_server yaim-fts yaim-glexec-wn yaim-storm yaim-voms

7 YAIM Directory structure /bin /opt/glite/yaim /examples /site-info /vo.d /functions /local /services /libexec /utils /nodes /log /pre /node-info.d /post /etc

8 YAIM Directory structure In /opt/glite/yaim/ the following get created by default: bin/yaim/ - the main binary defaults/ - config files with defaults shipped with the yaim core and module rpms etc/ - scripts for setting grid environment in the shell functions/ - functions executed during the configuration functions/local adapted/improved/local version of the same functions log/ location of yaim's log file libexec/ main secipts used by yaim node-info.d/ - function list definition files for the service

9 YAIM Directory Structure The following directories can/must be used for advanced configuration they should be created by the sitee-admin functions/pre functions to beb executed before the main one functions/post functions to be executed after the main one <path>/siteinfo/vo.d/ - VO (DNS-like VO's) specific info <path>/siteinfo/nodes host-specific config

10 YAIM config examples/defaults examples/ - example config files distributed with yaim edgusers.conf details of the UNIX users needed to run service-specific daemons and processes [groups,users].conf UNIX [group,users] details for pool accounts configuration See [groups,users].conf.readme wn-list.conf list of WN FQDN in this site See wn.list.readme siteinfo/site-info.def main configuration file siteinfo/services/[emi,glite]-<service> - service-specific settings Defaults/ - Variables with meaningful default values site-info.pre & site-info.post provided by yaim-core <node-type>.pre & <node-type>.post provided by service-pecific yaim modules

11 Using YAIM to configure services Files are sourced in the following order: /opt/glite/yaim/defaults/site-info.pre /opt/glite/yaim/defaults/[emi,glite]-node-type.pre <siteinfo_dir>/site-info.def <siteinfo_dir>/services/[emi,glite]-node-type <siteinfo_dir>/nodes/machine.domain /opt/glite/yaim/defaults/site-info.post /opt/glite/yaim/defaults/[emi,glite]-node-type.post <siteinfo_dir>/vo.d/vo_name /opt/glite/yaim/node-info.d/[emi,glite]-node-type Then the following functions are executed : /opt/glite/yaim/functions/pre/myfinc/ /optglite/yaim/functions/local/my_function OR /opt/glite/yaim/functions/my_func /opt/glite/yaim/functions/post/my_function

12 Example tasks with YAIM See options with yaim -h Check what you can configure: /opt/glite/yaim/bin/yaim -a -s <siteinfo_dir>/site-info.def Configure service(s): /opt/glite/yaim/bin/yaim -c -s <siteinfo_dir>/site-info.def -n emi_dpm_mysql Run a function (avoid a full reconfiguration of the entire serivce): /opt/glite/yaim/bin/yaim -r -s <siteinfo_dir>/site-info.def -n emi_dpm_mysql -f config_mkgridmap Verify your site-info.def /opt/glite/yaim/bin/yaim -v -s <siteinfo_dir>/site-infod.ef -n emi_dpm_mysql Create an rpm package from the site-info.def : /opt/glite/yaim/bin/yaim -p -s <siteinfo_dir>/site-info.def

13 Hints for more fulfilling usage There are two best-practices for managing configuration settings : Method A: Create only one site-info.def, user.conf and group.conf for all nodes in the site Put all the custom variables in nodes/ or services/ directory, distinguish them by profile installed or hostname Export the configuration directory via NFS to all hosts at the site Method B: On each host, create a separate site-info.def, user.conf, group.conf file and put it in some top-level (non-shared) directory e.g. /root Remember to save the files in the case of scratching of hosts or re-installation

14 Example : configure a site BDII In /opt/glite/yaim/etc/meraka/site-info.def : SITE_NAME=ZA-MERAKA SITE_DESC="Meraka Institute Grid site (SAGrid)" SITE_SUPPORT_ ="bbecker@csir.co.za" SITE_SECURITY_ ="rmooi@csir.co.za" SITE_LOC="Pretoria, South Africa" SITE_WEB=" SITE_ ="bbecker@csir.co.za" SITE_LAT= SITE_LONG= SITE_OTHER_GRID="NGI_ZA" In /opt/glite/yaim/etc/meraka/services/glite-bdii: These variables are relevant in the context of the whole site These variables are relevant only in the context of the particular service BDII_REGIONS="WMS CE" BDII_WMS_URL="ldap://wms.c4.csir.co.za:2170/mds-vo-name=resource,o=grid" BDII_CE_URL=" ldap://ce01-node001.c4.csir.co.za:2170/mds-vo-name=resource,o=grid" Configure the service: /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/meraka/site-info.def -n BDII_site

15 Advanced usage example: add a new VO ########## # sagrid # ########## VO_SAGRID_SW_DIR=$VO_SW_DIR/sagrid VO_SAGRID_DEFAULT_SE=$SE_HOST VO_SAGRID_STORAGE_DIR=$CLASSIC_STORAGE_DIR/sagrid VO_SAGRID_VOMS_SERVERS="vomss://voms.sagrid.ac.za:8443/voms/sagrid?/s agrid/" VO_SAGRID_VOMSES="'sagrid' 'voms.virt.sagrid.ac.za' '15001' /C=IT/O=INFN/OU=Host/L=ZA-UFS/CN=voms.virt.sagrid.ac.za sagrid VO_SAGRID_VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'" QUEUES and LFC should contain short VO names: QUEUES=... sagrid LFC_CENTRAL= sagrid <QUEUE_NAME>_GROUP_ENABLE variable should be named after the short VO name in CAPITAL LETTERS SAGRID_GROUP_ENABLE= sagrid

16 Add the relevant users and groups Check the roles add relevant users : 27109:sgmsagrid001:27100,2700:sgmsagrid,sagrid:sagrid:sgm: 27209:sgmsagrid002:27100,2700:sgmsagrid,sagrid:sagrid:sgm: 27309:sgmsagrid003:27100,2700:sgmsagrid,sagrid:sagrid:sgm: 2701:sagrid001:2700:sagrid:sagrid:: 2702:sagrid002:2700:sagrid:sagrid:: 2719:sagrid019:2700:sagrid:sagrid:: 2720:sagrid020:2700:sagrid:sagrid:: Ensure that the UIDs and GIDs are available! Add the relevant groups to groups.conf : "/sagrid/role=softwaremanager":::sgm: "/sagrid/grelc/das/*":sagrid::: "/sagrid/sagrid":::: "/sagrid":::: Configure site services with yaim

17 References Full YAIM400 reference: Guide400 Variable reference: fo_configuration_variables Thanks to Cristina Aiftimiei for original slides : EGI Technical Forum lay.py?sessionid=64&contribid=318&confid=