Single System Image OS for Clusters: Kerrighed Approach

Size: px

Start display at page:

Download "Single System Image OS for Clusters: Kerrighed Approach"

Preston Walton
5 years ago
Views:

1 Single System Image OS for Clusters: Kerrighed Approach Christine Morin IRISA/INRIA PARIS project-team October 7th,

Clusters for Scientific Computing Clusters are now recognized as a reasonable platform for scientific computing For applications which are not worth executing on expensive

2 Clusters for Scientific Computing Clusters are now recognized as a reasonable platform for scientific computing For applications which are not worth executing on expensive machines As departmental computation servers Small to medium size clusters are going to be common place computer architectures Up to 64 nodes October 7th, 2003 Linux Clusters Workshop 2

3 Scientific Computing Programming Paradigms October 7th, 2003 Linux Clusters Workshop 3

4 Application Execution on SMP (1/3) October 7th, 2003 Linux Clusters Workshop 4

5 Application Execution on SMP (2/3) October 7th, 2003 Linux Clusters Workshop 5

6 Application Execution on SMP (3/3) October 7th, 2003 Linux Clusters Workshop 6

7 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 7

8 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 8

9 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 9

10 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 10

11 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 11

12 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 12

13 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 13

14 Application Execution on Clusters October 7th, 2003 Linux Clusters Workshop 14

15 Single System Image Virtual SMP Same interface as a traditional OS for an SMP machine Same vision for all applications Efficiency Properties of a SSI OS Resource distribution transparency Intra- and inter- application resource sharing High availability October 7th, 2003 Linux Clusters Workshop 15

16 Kerrighed Combining high performance, high availability and ease of programming Global resource management Processor, memory, disk Integrated resource management Dynamic resource management To deal with configuration changes Small clusters < 100 nodes Extension of the standard OS running on each node Linux based prototype October 7th, 2003 Linux Clusters Workshop 16

17 Kerrighed Features Configurable global scheduler for global process management KerNet for global data stream management Containers for global memory management Checkpointing Portable high performance communication system Kerrighed provides a full Pthread interface on a cluster October 7th, 2003 Linux Clusters Workshop 17

18 Configurable Global Scheduler Design goals It should be possible to implement any traditional placement or load balancing strategy Development and integration of global scheduling policies should be easy Development environment Modular architecture Dynamic configuration of the global scheduler Without stopping the system and the applications Not only configurable but adaptive global scheduler Efficient process management mechanisms With minimal modifications to the OS kernel No modification to the local OS scheduler October 7th, 2003 Linux Clusters Workshop 18

19 Modular Global Scheduler Global Scheduling Manager Global Scheduling Manager Local Analyzers Local Analyzers Monitors Monitors Standard OS Standard OS October 7th, 2003 Linux Clusters Workshop 19

20 Configuration All components are configured with XML files All components can be hot-loaded and hot-removed XML XSL Code Generation Probe or Analyzer or Scheduler (kernel module) Loader (kernel module) Load Remove Load/remove October 7th, 2003 Linux Clusters Workshop 20

21 Development Environment October 7th, 2003 Linux Clusters Workshop 21

22 Process Management Mechanisms Global Scheduler (Application Management) Global Scheduler (Application Management) Process Duplication Process Checkpt Process Migration Process Duplication Process Checkpt Process Migration Ghost Process Mngt Ghost Process Mngt Memory Disk Network Network Disk Memory October 7th, 2003 Linux Clusters Workshop 22

23 Global Data Streams Message communication Pipe, FIFO, Unix sockets Inet sockets (UDP, TCP) Char devices KerNet Efficient migration of processes using message communication inside the cluster No modification to applications Dynamic streams & KerNet sockets October 7th, 2003 Linux Clusters Workshop 23

24 Communication Architecture MPI (MPICH, ) Applications Inet Sockets Unix Sockets Pipes/FIFO Char Devices KerNet High Performance Communication System (Kernel Level Interface) Network (Infiniband, Myrinet, Gigabit Ethernet, ) October 7th, 2003 Linux Clusters Workshop 24

25 KerNet Create Attach N1 N2 N3 N1 Attach N3 October 7th, 2003 Linux Clusters Workshop 25

26 KerNet Wake-up N1 N2 N2 N3? N1 Suspend N3 October 7th, 2003 Linux Clusters Workshop 26

27 Global Memory Management Different services Shared virtual memory Remote paging Cooperative file cache A unique concept: the container Software object to store and share data cluster wide Global management of physical memory Memory segments and files are associated to containers October 7th, 2003 Linux Clusters Workshop 27

28 Data Sharing Kerrighed implements a kernel level DSM based on containers Sequential consistency, page granularity The complete address space of a process is shared including the stack of each of its threads Stack Container 1 Stack Text Container 2 Text Data Container 3 Data Mapped file Container 4 Mapped file Thread 1 Thread 2 October 7th, 2003 Linux Clusters Workshop 28

29 Integration of Containers in a Standard OS Host Operating System VM Manager File System Host Operating System VM Manager File System Linker Linker Linker Linker Container Linker Linker Linker Linker Memory Manager Disk Manager Memory Manager Disk Manager Memory Disk Memory Disk October 7th, 2003 Linux Clusters Workshop 29

30 Kerrighed Prototype Extension of Linux kernel 9 modules ( lines) Process & load balancing (Aragorn, lines) Application Krgthread Application User Space Containers (Gandalf, lines) Synchronization (Elrond, 8000 lines) Ghosts (Nazgul, 3500 lines) Communication (Gimli, Gloin, lines) KerNet (4700 lines)/legolas(2500 lines) Tools (Iluvatar, 4000 lines) Limited patch to the kernel (300 lines) LibKrgthread (2000 lines) Aragorn (scheduling) Gandalf (container) Elrond (synchronization) Nazgul (ghost) Kernet/Legolas Gimli Gloin Network Iluvatar Linux Kernel Kernel Space 140 M/M since 1999 October 7th, 2003 Linux Clusters Workshop 30

31 Conclusion Kerrighed: first Linux based cluster OS providing the illusion of a virtual SMP Full Pthread support MPI OpenMP, multithreading Configurable adaptive global scheduler for process placement and migration Transparent checkpointing Kerrighed V0.72 available as an open source software under the GPL licence ( 100 downloads since mid-november 2002 October 7th, 2003 Linux Clusters Workshop 31

32 Perspectives: Research Directions High performance I/O Exploitation of cluster standard disks High availability Transparent cluster reconfigurations after node addition, eviction or failure Grid-aware OS for cluster federations P2P infrastructure with clusters as nodes Large scale data sharing Resource allocation Scalable checkpointing algorithms Security October 7th, 2003 Linux Clusters Workshop 32

Perspectives: Technology Transfer Kerrighed research prototype (2000-2003) CRECO EDF

33 Perspectives: Technology Transfer Kerrighed research prototype ( ) CRECO EDF/INRIA CIFRE Ph.D. grant (Geoffroy Vallée) Industrial Post-Doc (Renaud Lottiaux) Experimentations with first industrial applications provided by EDF HRM1D, CATHARE, Cyrano 3, Aster Kerrighed robustness and full set of functionalities ( ) COCA PEA funded by DGA Partnership with CGEY and ONERA-CERT 2 full time engineers (Renaud Lottiaux, David Margery) Experimentations with industrial applications Ligase, Gorf3D, Mixsar, RTI HLA October 7th, 2003 Linux Clusters Workshop 33

34 Perspectives: Technology Transfer Next Step: Kerrighed durability Including Kerrighed in a Linux Distribution for high performance computing (OSCAR, )? Kerrighed development consortium? Transfering Kerrighed to a company involved in cluster construction and software development? October 7th, 2003 Linux Clusters Workshop 34

Kerrighed is registered as a community trademark. http://www.kerrighed.

35 Kerrighed is registered as a community trademark. kerrighed.users@irisa.fr October 7th, 2003 Linux Clusters Workshop 35

36 Kerrighed Team Faculty Christine Morin (DR, INRIA) PhD students Geoffroy Vallée (CIFRE-EDF) Pascal Gallard (INRIA) Gaël Utard (INRIA) Louis Rilling (ENS-Cachan) Engineers Renaud Lottiaux (INRIA) David Margery (INRIA) Former member Ramamurthy Badrinath (IIT Kharagpur, India) May 2002 April 2003 October 7th, 2003 Linux Clusters Workshop 36

Kerrighed: A SSI Cluster OS Running OpenMP

Kerrighed: A SSI Cluster OS Running OpenMP EWOMP 2003 David Margery, Geoffroy Vallée, Renaud Lottiaux, Christine Morin, Jean-Yves Berthou IRISA/INRIA PARIS project-team EDF R&D 1 Introduction OpenMP only