OAR batch scheduler and scheduling on Grid'5000

Transcription

1 OAR batch scheduler and scheduling on Grid'5000 Olivier Richard (UJF/INRIA) joint work with Nicolas Capit, Georges Da Costa, Yiannis Georgiou, Guillaume Huard, Cyrille Martin, Gregory Mounié and Pierre Neyron

2 O u t l i n e Oar Introduction Principles / Design Features Oar on Grid5000 Deployment Support Simple Grid Coallocator (OarGrid) DAS3-G5K Different Interconnecting Ways Conclusion

3 O a r : I n t r o d u c t i o n / h i s t o r y Developed after use of OpenPBS/PBSPro on Icluster1 (225 PC cluster (rank 385 TOP500/juin01 )) [ ] Need of a scalable, robust and flexible batch scheduler with short development cycle [ ] OAR v 1.6 is running on production clusters. (more than 3 million exectued jobs) Next version: [2007-] OAR v 2

4 G e n e r a l o r g a n i z a t i o n o f B a t c h S c h e d u l e r s Batch schedulers can be divide in 2 main parts Job mangement Resource (node) management Job Management Resource Management

5 D e s i g n ( 1 / 3 ) : B a s e d o n h i g h l e v e l s o f t w a r e c o m p o n e n t s Relational database engine mysql or postgresql Scripting Language Perl Parallel Launcher (for large clusters)[optional] Taktuk Others Sudo: security approach SSH: remote access CPUSET (Linux): procesus confinement on CPU and memory banks [optional]

6 D e s i g n ( 2 / 3 ) : M o d u l a r i t y & w o r k d i s t r i b u t i o n OAR is separated in different modules: jobs submissions handling scheduling jobs execution nodes monitoring a central automaton orchestrate the work accept notifications from client programs execute modules when needed execute periodic tasks

7 D e s i g n ( 3 / 3 ) : O p e n e d a r c h i t e c t u r e no fixed interface, oar modules just have to conform to data organization in the database semantics of fields in the database state diagram for jobs and nodes development of a module is not limited by any interface every internal data is exposed when accessing the database just require a language able to make sql queries enable short development cycle

8 O A R s c h e d u l i n g a l g o r i t h m first-fit which bluids an internal Gantt chart of jobs natively provide backfilling easy visualization of scheduling predictions advanced scheduling algorithm using a more elaborate knapsack algorithm to schedule groups of jobs [v 1.6] better global result support for moldable job simple fairsharing algorithm [v 2.0] based on cost function which takes in account project and user resource consumptions

9 M a i n F e a t u r e s Admission rules Flexible resources schema * Hierarchical resources * Multipe resources * Moldable job support * cpuset support* Matching of resources Hold and resume jobs Multi schedulers support Multi queues with priority Best effort queues (for exploiting idle resources) Enhanced submission expression * Checkpointing support * ssh as remote execution protocols (Taktuk for large cluster) Dynamic insertion/deletion of compute node First Fit Scheduler with matching resource Advance Reservation No Daemon on compute nodes Environnement of Demand support (KaDeploy integration) Check compute nodes before launching Activity visualization tools (GanttChart) * in version 2.0

10 OAR on Grid'5000

11 N e e d s f o r G r i d ' One batch scheduler per cluster Mainly Support and control deployment operation Allow grid experiments in simple way

12 D e p l o y m e n t s u p p o r t What is deployment (provisionning) install all software stack on compute node operation available for all users Why deployment : for deep experiments: Applications Data Middleware Librairies OS like OS modification evaluation, network emulation.. to evalute complete grid or cluster solution (like globus, or other) in simple way

13 D e p l o y m e n t e x a m p l e Goal : replay workload traces on real multicluster system Applications Data Deploy Middleware Librairies OS A cluster environment Servers OAR/Kadeploy NFS/LDAP... SERVER SGE/NFS/Login... login Compute Nodes G5K's Cluster CLUSTER

14 D e p l o y m e n t S u p p o r t a special queue tagged deploy in prologue script: set rights to deploy for allocated nodes for interactive job user is logged on the node which has kadeploy commands (it is not a compute node!) in epilogue script: unset rights to deploy reboot nodes on reference (production) environment OAR does not require specific daemon on compute nodes, this simply deployment support

15 O A R G R I D Main scheduling objective for G5K : coallocation for simultaneous jobs start. A very simple grid extension submit an advance reservation on each selected clusters for c in #clusters ssh c oarsub -r now end_for if a reservation is rejected => 2 modes 1) default: stop submissions and delete accepted reservation(s) 2) forced: continue to submit other reservations a database keeps all information (job_grid id, list of job id for each cluster submission)

16 S o m e D A S 3 - G 5 K i n t e r c o n n e c t i o n w a y s

17 D A S 3 - G 5 K i n t e r c o n n e c t i o n w a y s Hypothesis: IP connectivity works and resource sharing policy is defined Different ways (no exhaustive) DAS3 -> G5K Only provide accounts for DAS3 users Deploy DAS3 environment on G5K Use Globus Gatekeeper on G5K G5K -> DAS3 Only provide accounts for G5K users Use Globus Gatekeeper on G5K Deploying Capability on DAS3?

18 D A S 3 - G 5 K Deploying DAS3 environment on G5K nodes Needs a DAS3 environment a DAS3 global user advantages: simple short to do it disad.: DAS3 environment G5K's Cluster DAS3 = Grid'5000 experiment

19 D A S 3 - G 5 K Globus Gatekeeper on G5K Needs Certificat for GSI/WS-Security Meta-scheduler: Koala? GRAM adaptor for oar Advantages: + Globus GateKeeper More secure Grid standard G5K's Cluster Opportunity for other interconnections Disasd. More complex (for admins and users)

20 D A S 3 - G 5 K Deployment on DAS3? hard reboot and console access (IPMI standard) integration SGE / Kadeploy? difficult? Advantage deep experiments Disadvanage a new tool for DAS3's users and admins

21 C o n c l u s i o n OAR is a mature resource management system for cluster simple to adapt, to extend, to control kadeploy support advance reservation for simplify grid experiment (oargrid) OARv2 (Available in first quater 2007) [beta11 avail. now] DAS3-G5K interconnections Multiple complementary ways (no-exhaustive) DAS3 environment on G5K Add Globus Gatekepeer on G5K clusters Deploy on DAS3