An introduction to checkpointing. for scientifc applications

Size: px

Start display at page:

Download "An introduction to checkpointing. for scientifc applications"

Basil McDonald
5 years ago
Views:

1 UCL/CISM An introduction to checkpointing for scientifc applications November 2016 CISM/CÉCI training session

2 What is checkpointing?

3 Without checkpointing: $./count 1 2 3^C $./count With checkpointing: $./count 1 2 3^C $./count 4 5 6

4 Without checkpointing: With checkpointing: $./countcheckpointing: $./count 'saving' a computation 3^C 3^C $./count $./count so that it can be resumed later 1 4 (rather than started again)

5 Why do we need checkpointing?

6 Goals of checkpointing in HPC: 1. Fit in time constraints 2. Debugging, monitoring 3. Cope with NODE_FAILs 4. Gang scheduling and preemption

7 The idea: Save the program state Values in variables Open fles... Position in the code Signal or event... every time a checkpoint is encountered and restart from there upon (un)planned stop rather than bootstrap again from scratch starting loops at iteration 0 creating tmp fles...

8 The key questions... Transparency Transparency for for developer developer Portability Portability to to other other systems systems Size Size of of state state to to save save Checkpointing Checkpointing overhead overhead Do I need to write a lot of additional code? Can I stop on one system and restart on another? How many GB of disk does it require? How many FLOPs lost to ensure checkpointing?

9 Who's in charge of all that? Transparency for developer Portability to other systems Size of state to save Checkpointing overhead the application itself a library the compiler a run-time the OS the hardware

10 Today's agenda: How to make your program checkpoint-able -> concepts and examples -> recipes (design patterns) and signals -> Slurm -> parallel checkpointing

11 So you can play On hmem: ~dfr/checkpointing.tgz

12 1 Making a program checkpoint-able by saving its state every iteration and looking for a state fle on startup.

13 Without checkpointing: $./count 1 2 3^C $./count With checkpointing: $./count 1 2 3^C $./count 4 5 6

14 Without checkpointing: $./count 1 2 3^C $./count $./count 1 2 3^C $./count 4 5 6

15 The general recipe 1. Look for a state fle (name can be hardcoded, or, better, passed as parameter) 2. If found, then restore state (initialize all variables with content of the fle state) Else, bootstrap (create initial state) 3. Periodically save the state In the previous example : The state is just an integer Periodically means at each iteration

16 So you can play 1. Translate 'count' in your favorite language 2. Adapt it to enable checkpointing

17 Python recipe

18 R recipe

19 Octave recipe

20 Fortran recipe

21 C recipe

22 Java recipe

23 2 Using UNIX signals to reduce overhead : do not save the state at each iteration -- wait for the signal.

24 UNIX processes can receive 'signals' from the user, the OS, or another process

25 UNIX processes can receive 'signals' from the user, the OS, or another process ^C ^D kill -9 kill ^Z fg, bg

26 UNIX processes can receive 'signals' from the user, the OS, or another process e.g.

27 UNIX processes can receive 'signals' from the user, the OS, or another process e.g.

28 UNIX processes can receive 'signals' with an associated default action

29 UNIX processes can receive 'signals' and handle ('trap') them

30 The general recipe 1. Register a signal handler (a function that will modify a global variable when recieving a signal) 2. Test the value of the global variable periodically (At a moment when the state is consistent an easy to recreate) 3. If the value indicates so, save state to disk (and optionally gracefully stop) In the previous example : The state is just an integer Periodically means at each iteration

31 So you can play Adapt your program to handle signals

32 Useful links C: Fortran: Python: Octave: R: Java:

33 Previous C recipe

34 C signal recipe

35 C signal recipe

36 Fortan signal recipe

37 Fortan signal recipe

38 Java signal recipe

39 Python signal recipe

40 Octave signal recipe

41 R signal recipe

42 3 Use Slurm signaling abilities to manage checkpoint-able software in Slurm scripts on the clusters.

43 scancel is used to send signals to jobs

44 Example: use scancel --signal USR1 $SLURM_JOB_ID to force state dump for reviewing/debugging Python signal recipe

45 --signal to have Slurm send signals automatically before the end of the allocation

46 Set non-zero return code when stopping because of a received signal Fortran signal recipe

47 Then you can have your job re-queued automatically

48 Set a non-zero exit code C: exit(1) Fortran: stop 1 Octave: exit( 1 ) R: quit( status=1 ) Python: sys.exit( 1 ) Java: System.exit( 1 )

49 Or chain the jobs...

50 Using a signal-based watchdog to re-queue the job just before it is killed

51 4 Parallel programs are better checkpointed after a global synchronization.

52 In the fork-join model, checkpoint after a join and before a fork Checkpoint here Easily ensure state consistency Allows restarting with a different number of threads

56 5 Use programs and libraries that enable other programs with checkpoint/restart capabilities.

57 Such program needs to: 1. Access the process' memory (the c/r program forks itself as the process, or uses a kernel module) 2. Access the processor state at any moment (it uses signals to interrupt the process and provoke storage of the registers on the stack) 3. Track the state changing actions (fork, exec, system, etc.) (wrap standard library functions with LD_PRELOAD'ed custom functions) 4. Inject checkpointing code in the program (LD_PRELOAD a library with signal handlers)

58 Such program needs to: 1. Access the process' memory (the c/r program forks itself as the process, or uses a kernel module) 2. Access the processor state at any moment (it uses signals to interrupt the process and provoke storage of the registers on the stack) 3. Track the state changing actions (fork, exec, system, etc.) (wrap standard library functions with LD_PRELOAD'ed custom functions) 4. Inject checkpointing code in the program (LD_PRELOAD a library with signal handlers)

59 LD_PRELOAD magic

60 LD_PRELOAD magic

62 UCL/CISM Summary, Wrap-up and Conclusions. October 2014 CISM/CÉCI training session

63 Never again force your users to click 'Discard'...

64 Make initializations conditional Save minimal reconstructable state periodically Save full workspace upon signal Checkpoint after a synchronization

65 So you can play Adapt your own program for checkpoint/restart what is the minimal reconstructible state? what file format for the checkpoint? what frequence/what signal? what start strategy: look for checkpoint file vs command-line parameter, etc.? what initalization should I modify? what files should I re-open?

An introduction to checkpointing. for scientific applications

damien.francois@uclouvain.be UCL/CISM - FNRS/CÉCI An introduction to checkpointing for scientific applications November 2013 CISM/CÉCI training session What is checkpointing? Without checkpointing: $./count