Implemen'ng Asynchronous Checkpoint/Restart for CnC. Nick Vrvilo, Vivek Sarkar Rice University Kath Knobe, Frank Schlimbach Intel September 24, 2013

Transcription

1 Implemen'ng Asynchronous Checkpoint/Restart for CnC Nick Vrvilo, Vivek Sarkar Rice University Kath Knobe, Frank Schlimbach Intel September 24,

2 MoCvaCng CnC Checkpoint/Restart (C/R) what s good for simplifying parallelism is good for simplifying resilience * Pushing CnC for extreme- scale programming Resilience is a must for large- scale applicacons CnC has many interescng properces: ExecuCon graphs ExecuCon froncers Single- assignment data Hierarchy Side- effect- free computacon steps How best to approach C/R for CnC? * Inter- Agency Workshop on HPC Resilience at Extreme Scale

3 Current C/R SoluCon Obstacles Problems Requires global coordinacon Bursty communicacon pa]ern Saving non- essencal data Explicit checkpoint in code Periodic global checkpoints Symptoms SynchronizaCon overhead Poor network uclizacon Large checkpoint files Invasive to programmer User required to set frequency 3

4 OpportuniCes for OpCmizaCon * CnC Features ComputaCon Graphs Single- assignment data Monotonically growing state Discrete computacon steps Side- effect- free step code Poten'al Proper'es Transparent Asynchronous ConCnuous Resumable * Alex Nelson (HP) idencfied some of these traits for a TStreams C/R prototype in

5 Implementa'on for CnC on Habanero C 5

6 C/R for CnC on Habanero- C Why Habanero- C (HC)? Rice implementacon access to source code AnCcipaCon for distributed implementacon CnC- HC and control tags Control tags are conceptual in CnC- HC Steps prescribed directly 6

7 Transparent to the Programmer HC- CnC runcme manages: Item colleccon put/get operacons Step prescripcon Step teardown RunCme encapsulates all C/R operacons C/R transparency E.g. CNC_PUT operacon also handles checkpoincng the produced data item 7

8 ConCnuously CheckpoinCng Stream CnC state changes into checkpoint Mirrors CnC state of data items and steps Checkpoint grows with execucon froncer (XF) Mirroring the leading edge of XF Trailing edge of XF handled separately Place hooks in CnC- HC funccons that alter the execucon froncer CNC_PUT CNC_PRESCRIBE Step teardown (support code) 8

9 Asynchrony Don t stop/stall computacon to checkpoint Enqueue state changes for checkpoincng Dedicated thread handles checkpoint wricng (parallels dedicated worker in some Rice runcmes) Single- assignment data guarantees queued data won t change before checkpointed 9

10 Resuming from a Checkpoint Implement C/R at CnC step granularity Monotonic state state is always sane CnC Restart Skip inical environment operacons Re- prescribe live steps Re- put live data Restart process is order- agnoscc Graph dependencies manage execucon order Could restore in parallel 10

11 Checkpoint s Trailing Edge Checkpoint state s trailing edge must be computed independently of execucon state Computa'on: Start StepA Put ItemX Finish StepA Checkpoint: Start StepA Finish StepA Put ItemX 11

12 State of CnC- HC C/R ImplementaCon Only supports scalar integer data (need user- defined serializacon funccons otherwise) Only supports integer- tuple tags/keys Writes checkpoint data to disk Separate process to maintain checkpoint s XF Manual restart amer failure 12

13 ApplicaCon AssumpCons Seeding from environment is checkpointed Get counts for data items Extra C/R dependency E.g. an item only dies amer it s checkpointed Get counts one extra get Observing CnC contracts Produces / consumes / tag funccons 13

14 Example App: Pascal s Triangle 14

15 nc k with Pascal s Triangle N K

16 Graph Spec for n C k App CollecCons Tags: Steps: Data: triangle [int] <int row, int col> edgetag <int row, int col> innertag <int row, int col> edgestep <int row, int col> innerstep <int row, int col> 16

17 Graph Spec for n C k App PrescripCon RelaCons edgetag <r, c> edgestep <r, c> innertag <r, c> innerstep <r, c> 17

18 Graph Spec for n C k App PrescripCon RelaCons edgetag <r+1, c> edgestep <r, c> edgetag <r+1, c+1> innertag <r+1, c> innerstep <r, c> innertag <r, c> 18

19 Graph Spec for n C k App PrescripCon RelaCons N K edgestep <r, c> edgetag <r+1, c> edgetag <r+1, c+1> innertag <r+1, c> LeN Right innerstep <r, c> innertag <r, c> 19

20 Graph Spec for n C k App Input/Output RelaCons edgestep <r, c> triangle <r- 1,c- 1> triangle <r- 1,c> innerstep <r, c> triangle <r,c> innerstep <r, c> triangle <r,c> 20

21 C/R with Pascal s Triangle 21

22 nc k C/R Example Staging Checkpoint 22

23 nc k C/R Example edgestep<0,0>: put(tri,<0,0>,1) pres(edgestep,<1,0>) pres(edgestep,<1,1>) Staging Checkpoint es<0,0> 23

24 nc k C/R Example edgestep<0,0>:» put(tri,<0,0>,1) pres(edgestep,<1,0>) pres(edgestep,<1,1>) Staging es<0,0> { puts: tri<0,0>[1] } Checkpoint es<0,0> 24

25 nc k C/R Example edgestep<0,0>: put(tri,<0,0>,1)» pres(edgestep,<1,0>) pres(edgestep,<1,1>) Staging es<0,0> { puts: tri<0,0>[1] pres: es<1,0> } Checkpoint es<0,0> 25

26 nc k C/R Example edgestep<0,0>: put(tri,<0,0>,1) pres(edgestep,<1,0>)» pres(edgestep,<1,1>) Staging es<0,0> { puts: tri<0,0>[1] pres: es<1,0> es<1,1> } Checkpoint es<0,0> 26

27 nc k C/R Example edgestep<0,0>:» put(tri,<0,0>,1) pres(edgestep,<1,0>) pres(edgestep,<1,1>) Staging es<0,0> { puts: tri<0,0>[1] count:1 pres: es<1,0> es<1,1> count:2 gets: } Checkpoint es<0,0> 27

28 nc k C/R Example edgestep<0,0>:» put(tri,<0,0>,1) pres(edgestep,<1,0>) pres(edgestep,<1,1>) Staging es<0,0> { puts: tri<0,0>[1] count:1 pres: es<1,0> es<1,1> count:2 gets: } Checkpoint es<0,0> 28

29 nc k C/R Example Staging Checkpoint tri<0,0>[1] es<1,0> es<1,1> 29

30 nc k C/R Example edgestep<1,0>: put(tri,<1,0>,1) pres(edgestep,<2,0>) Staging Checkpoint tri<0,0>[1] es<1,0> es<1,1> 30

31 nc k C/R Example edgestep<1,0>:» put(tri,<1,0>,1) pres(edgestep,<2,0>) Staging es<1,0> { puts: tri<1,0>[1] } Checkpoint tri<0,0>[1] es<1,0> es<1,1> 31

32 nc k C/R Example edgestep<1,0>: put(tri,<1,0>,1)» pres(edgestep,<2,0>) Staging es<1,0> { puts: tri<1,0>[1] pres: es<2,0> } Checkpoint tri<0,0>[1] es<1,0> es<1,1> 32

33 nc k C/R Example edgestep<1,0>:» put(tri,<1,0>,1) pres(edgestep,<2,0>) Staging es<1,0> { puts: tri<1,0>[1] count:1 pres: es<2,0> count:1 gets: } Checkpoint tri<0,0>[1] es<1,0> es<1,1> 33

34 nc k C/R Example edgestep<1,0>:» put(tri,<1,0>,1) pres(edgestep,<2,0>) Staging es<1,0> { puts: tri<1,0>[1] count:1 pres: es<2,0> count:1 gets: } Checkpoint tri<0,0>[1] es<1,0> es<1,1> 34

35 nc k C/R Example Staging Checkpoint tri<0,0>[1] es<1,1> tri<1,0>[1] es<2,0> 35

36 nc k C/R Example edgestep<1,1>: put(tri,<1,1>,1) pres(innerstep,<2,1>) pres(edgestep,<2,2>) Staging Checkpoint tri<0,0>[1] es<1,1> tri<1,0>[1] es<2,0> 36

37 nc k C/R Example edgestep<1,1>:» put(tri,<1,1>,1) pres(innerstep,<2,1>) pres(edgestep,<2,2>) Staging es<1,1> { puts: tri<1.1>[1] } Checkpoint tri<0,0>[1] es<1,1> tri<1,0>[1] es<2,0> 37

38 nc k C/R Example edgestep<1,1>:» put(tri,<1,1>,1) pres(innerstep,<2,1>) pres(edgestep,<2,2>) CRASH! Staging es<1,1> { puts: tri<1.1>[1] } Checkpoint tri<0,0>[1] es<1,1> tri<1,0>[1] es<2,0> 38

39 nc k C/R Example Restart: 1. Throw out Staging 2. Re- add items/steps from checkpoint 3. ConCnue as normal Staging es<1,1> { puts: tri<1.1>[1] } Checkpoint tri<0,0>[1] es<1,1> tri<1,0>[1] es<2,0> 39

40 nc k C/R Example innerstep<2,1>:» put(tri,<2,1>,2) pres(innerstep,<3,1>) Staging is<2,1> { puts: tri<2,1>[1] count:1 pres: is<3,1> count:1 gets: tri<1,0>[1] tri<1,1>[1] } Checkpoint tri<1,0>[1;gc=1] tri<1,1>[1;gc=2] is<2,1> es<2,2> 40

41 nc k C/R Example Staging Checkpoint tri<1,1>[1;gc=1] es<2,2> tri<2,1>[1] is<3,1> 41

42 Conclusion 42

43 Summary Working prototype of C/R on CnC- HC Write- once data + monotonic state more asynchrony SCll have many more opcons to explore Transparent Con'nuous Asynchronous Resumable 43

44 Future Work More benchmarks and empirical data Granularity of checkpoint data Distributed CnC C/R Checkpoint/ConCnue IntegraCon with hierarchy in CnC MulC- level checkpoints Transparent Checkpoint/Restart CnC offline debugger 44