Co-ordination & Harmonisation of Advanced e-infrastructures for Research and Education Data Sharing Research Infrastructures Grant Agreement n. 306819 YAIM Overview Bruce Becker Meraka Institute
Outline What is YAIM? History, status, prospects, design What is it used for? How is it organised? How should we use it? Practical examples 22.3.2013 2
Short history European DataGrid project (2001 2003) WP4 Fabric Management Mandate to to deliver a computing frabric comprised of all the necessary tools to manage a centre providing grid services on clusters of thousands of nodes Produced LCFG (Local ConFiGuration system) Evolved into LFCng (next generation, client-server architecture) subsequently Quattor community effort, open source software EGEE projects (I, II, III - 2004-2010) Adopted YAIM Yet Another Installation Manageer YAIM Ain't an Installation Manager 22.3.2013 3
YAIM Present and Future YAIM in EMI : All module the responsibility of product teams yaim-core/clients : mainly bug fixes EMI reccomendation : Adopt standard configuration mechanisms suggested by the operating systems supported YAIM post EMI : Up to product teams. WLCG TEG Operations and Tools final report : middleware configuration should be improved and not based on one single configuration management tool 22.3.2013 4
YAIM Design YAIM Implements a configuration method for glite software Aims to be simple configuration method that can be used to set up simple grid sites, but can also easily be adapted to meet the needs of larger sites Main features Modular structure Bash syntax Node type configuration facility Well-defined directory structure Basic configuration files 22.3.2013 5
YAIM modular structure Yaim-core (glite-yaim-core) Common functions and definitions One yaim module per node type/service Specfic functions and configuration files 21 modules (bold are used in SAGrid) dpm-yaim-4.2.7-1 glite-apel-yaim-1.0.2-4 glite-ce-yaim-cream-ce-4.3.0-4 glite-lb-yaim-4.4.3-3 glite-px-myproxy-yaim-4.1.7-1 glite-yaim-bdii-4.3.11-1 glite-yaim-clients-5.0.1-2 glite-yaim-cluster-2.1.0-4 glite-yaim-ge-utils-4.3.0-4 glite-yaim-lsf-utils-5.0.0-2 glite-yaim-mpi-1.1.11-1 glite-yaim-torque-client-5.1.0-1 glite-yaim-torque-server-5.1.0-1 glite-yaim-torque-utils-5.1.0-2 glite-yaim-wms-4.1.5-1 lfc-yaim-4.2.4-1 yaim-argus_server-1.5.1-1 yaim-fts-4.1.14-1 yaim-glexec-wn-2.3.2-1 yaim-storm-4.2.2-1 yaim-voms-1.1.1-1 22.3.2013 6
YAIM Directory structure /bin /opt/glite/yaim /examples /site-info /vo.d /functions /local /services /libexec /utils /nodes /log /pre /node-info.d /post /etc 22.3.2013 7
YAIM Directory structure In /opt/glite/yaim/ the following get created by default: bin/yaim/ - the main binary defaults/ - config files with defaults shipped with the yaim core and module rpms etc/ - scripts for setting grid environment in the shell functions/ - functions executed during the configuration functions/local adapted/improved/local version of the same functions log/ location of yaim's log file libexec/ main secipts used by yaim node-info.d/ - function list definition files for the service 22.3.2013 8
YAIM Directory Structure The following directories can/must be used for advanced configuration they should be created by the sitee-admin functions/pre functions to beb executed before the main one functions/post functions to be executed after the main one <path>/siteinfo/vo.d/ - VO (DNS-like VO's) specific info <path>/siteinfo/nodes host-specific config 22.3.2013 9
YAIM config examples/defaults examples/ - example config files distributed with yaim edgusers.conf details of the UNIX users needed to run service-specific daemons and processes [groups,users].conf UNIX [group,users] details for pool accounts configuration See [groups,users].conf.readme wn-list.conf list of WN FQDN in this site See wn.list.readme siteinfo/site-info.def main configuration file siteinfo/services/[emi,glite]-<service> - service-specific settings Defaults/ - Variables with meaningful default values site-info.pre & site-info.post provided by yaim-core <node-type>.pre & <node-type>.post provided by service-pecific yaim modules 22.3.2013 10
Using YAIM to configure services Files are sourced in the following order: /opt/glite/yaim/defaults/site-info.pre /opt/glite/yaim/defaults/[emi,glite]-node-type.pre <siteinfo_dir>/site-info.def <siteinfo_dir>/services/[emi,glite]-node-type <siteinfo_dir>/nodes/machine.domain /opt/glite/yaim/defaults/site-info.post /opt/glite/yaim/defaults/[emi,glite]-node-type.post <siteinfo_dir>/vo.d/vo_name /opt/glite/yaim/node-info.d/[emi,glite]-node-type Then the following functions are executed : /opt/glite/yaim/functions/pre/myfinc/ /optglite/yaim/functions/local/my_function OR /opt/glite/yaim/functions/my_func /opt/glite/yaim/functions/post/my_function 22.3.2013 11
Example tasks with YAIM See options with yaim -h Check what you can configure: /opt/glite/yaim/bin/yaim -a -s <siteinfo_dir>/site-info.def Configure service(s): /opt/glite/yaim/bin/yaim -c -s <siteinfo_dir>/site-info.def -n emi_dpm_mysql Run a function (avoid a full reconfiguration of the entire serivce): /opt/glite/yaim/bin/yaim -r -s <siteinfo_dir>/site-info.def -n emi_dpm_mysql -f config_mkgridmap Verify your site-info.def /opt/glite/yaim/bin/yaim -v -s <siteinfo_dir>/site-infod.ef -n emi_dpm_mysql Create an rpm package from the site-info.def : /opt/glite/yaim/bin/yaim -p -s <siteinfo_dir>/site-info.def 22.3.2013 12
Hints for more fulfilling usage There are two best-practices for managing configuration settings : Method A: Create only one site-info.def, user.conf and group.conf for all nodes in the site Put all the custom variables in nodes/ or services/ directory, distinguish them by profile installed or hostname Export the configuration directory via NFS to all hosts at the site Method B: On each host, create a separate site-info.def, user.conf, group.conf file and put it in some top-level (non-shared) directory e.g. /root Remember to save the files in the case of scratching of hosts or re-installation 22.3.2013 13
Example : configure a site BDII In /opt/glite/yaim/etc/meraka/site-info.def : SITE_NAME=ZA-MERAKA SITE_DESC="Meraka Institute Grid site (SAGrid)" SITE_SUPPORT_EMAIL="bbecker@csir.co.za" SITE_SECURITY_EMAIL="rmooi@csir.co.za" SITE_LOC="Pretoria, South Africa" SITE_WEB="http://www.meraka.co.za" SITE_EMAIL="bbecker@csir.co.za" SITE_LAT=-25.75582 SITE_LONG=28.27864 SITE_OTHER_GRID="NGI_ZA" In /opt/glite/yaim/etc/meraka/services/glite-bdii: These variables are relevant in the context of the whole site These variables are relevant only in the context of the particular service BDII_REGIONS="WMS CE" BDII_WMS_URL="ldap://wms.c4.csir.co.za:2170/mds-vo-name=resource,o=grid" BDII_CE_URL=" ldap://ce01-node001.c4.csir.co.za:2170/mds-vo-name=resource,o=grid" Configure the service: /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/meraka/site-info.def -n BDII_site 22.3.2013 14
Advanced usage example: add a new VO ########## # sagrid # ########## VO_SAGRID_SW_DIR=$VO_SW_DIR/sagrid VO_SAGRID_DEFAULT_SE=$SE_HOST VO_SAGRID_STORAGE_DIR=$CLASSIC_STORAGE_DIR/sagrid VO_SAGRID_VOMS_SERVERS="vomss://voms.sagrid.ac.za:8443/voms/sagrid?/s agrid/" VO_SAGRID_VOMSES="'sagrid' 'voms.virt.sagrid.ac.za' '15001' /C=IT/O=INFN/OU=Host/L=ZA-UFS/CN=voms.virt.sagrid.ac.za sagrid VO_SAGRID_VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'" QUEUES and LFC should contain short VO names: QUEUES=... sagrid LFC_CENTRAL= sagrid <QUEUE_NAME>_GROUP_ENABLE variable should be named after the short VO name in CAPITAL LETTERS SAGRID_GROUP_ENABLE= sagrid 22.3.2013 15
Add the relevant users and groups Check the roles add relevant users : 27109:sgmsagrid001:27100,2700:sgmsagrid,sagrid:sagrid:sgm: 27209:sgmsagrid002:27100,2700:sgmsagrid,sagrid:sagrid:sgm: 27309:sgmsagrid003:27100,2700:sgmsagrid,sagrid:sagrid:sgm: 2701:sagrid001:2700:sagrid:sagrid:: 2702:sagrid002:2700:sagrid:sagrid:: 2719:sagrid019:2700:sagrid:sagrid:: 2720:sagrid020:2700:sagrid:sagrid:: Ensure that the UIDs and GIDs are available! Add the relevant groups to groups.conf : "/sagrid/role=softwaremanager":::sgm: "/sagrid/grelc/das/*":sagrid::: "/sagrid/sagrid":::: "/sagrid":::: Configure site services with yaim. 22.3.2013 16
References Full YAIM400 reference: https://twiki.cern.ch/twiki/bin/view/lcg/yaim Guide400 Variable reference: https://twiki.cern.ch/twiki/bin/view/lcg/site-in fo_configuration_variables Thanks to Cristina Aiftimiei for original slides : http://www.pd.infn.it/~aiftim/ EGI Technical Forum 2012 - https://indico.egi.eu/indico/contributiondisp lay.py?sessionid=64&contribid=318&confid=1019 22.3.2013 17