An Evaluation of Checkpoint Recovery For Massively Multiplayer Online Games

Transcription

1 An Evaluation of Checkpoint Recovery For Massively Multiplayer Online Games Marcos Vaz Salles Tuan Cao Benjamin Sowell Alan Demers Johannes Gehrke Christoph Koch Walker White

2 MMO Games Simulate long-lived, interactive virtual worlds Persist across user sessions

3 MMO Games Simulate long-lived, interactive virtual worlds Can existing DBMS main-memory checkpointing techniques help to achieve low cost durability? Persist across user sessions

4 Our Contributions Show how to adapt consistent checkpointing techniques Provide a thorough simulation model and evaluate six different recovery strategies

5 Our Contributions Show how to adapt consistent checkpointing techniques Provide a thorough simulation model and evaluate six different recovery strategies Why simulation? Ability to evaluate different types of hardware

6 Talk Outline Introduction to MMOs The game state The game loop Consistent Checkpointing Techniques Requirements Performance Criteria Performance Model Experiments Experimental Setup Model Validation Results Conclusion

7 The Game State Tables containing game objects (e.g., attributes of game characters) Must be kept in main memory

8 The Game State

9 The Game Loop Update Receive player input Process player actions Handle updates 30 ticks/sec Draw The game state is consistent at the end of every tick. 7

10 The Game Loop 30 ticks/sec Update Draw Receive player input Process player actions Handle updates Perform checkpointing 33ms The game state is consistent at the end of every tick. 7

11 Requirements Small overhead Entire checkpointing process must fit into the game simulation Uniform overhead Low latency, no hiccup in the game No data loss Recover to the point of the crash

12 Requirements Small overhead Uniform overhead No data loss

13 Requirements Performance Criteria Small overhead Average Overhead Time Uniform overhead Overhead Distribution No data loss Checkpointing Time Recovery Time

14 Checkpointing Techniques All Objects Dirty Objects Log Based Double Backup Eager Copy Copy On Update

15 Checkpointing Techniques All Objects Log Eager Copy Dirty Objects Double Backups Copy On Update

16 Checkpointing Techniques All Objects Log Eager Copy Dirty Objects Double Backups Copy On Update

17 Checkpointing Techniques All Objects Log Eager Copy Dirty Objects Double Backups Copy On Update Dirty Objects

18 Checkpointing Techniques All Objects Log Based Eager Copy Dirty Objects Double Backup Copy On Update Log-Based

19 Checkpointing Techniques All Objects Log Based Eager Copy Dirty Objects Double Backup Copy On Update Log-Based

20 Checkpointing Techniques All Objects Log Based Eager Copy Dirty Objects Double Backup Copy On Update Double Backup

21 Checkpointing Techniques All Objects Log Based Eager Copy Dirty Objects Double Backup Copy On Update Double Backup Containing a consistent copy of the game state

22 Checkpointing Techniques All Objects Log Based Eager Copy Dirty Objects Double Backup Copy On Update Double Backup Containing a consistent copy of the game state

23 Checkpointing Techniques All Objects Log Based Eager Copy Dirty Objects Double Backup Copy On Update Double Backup Containing a consistent copy of the game state

24 Checkpointing Techniques All Objects Log Eager Copy Dirty Objects Double Backups Copy On Update Update Eager Copy Draw Update Draw Pause the simulation loop Copy objects to memory Creates a hiccup in the game 15

25 Checkpointing Techniques All Objects Log Eager Copy Dirty Objects Double Backups Copy On Update Update Copy On Update Draw Update Intercept each update Copy object to memory before updating it Draw Overhead is spread over ticks 16

26 Checkpointing Algorithms All Objects Dirty Objects Eager Copy Copy On Update Double Backup Log Naive-Snapshot X X X Copy-On-Update Dribble-And- Atomic-Copy- Dirty-Objects X X X X X X Partial-Redo X X X Copy-On-Update X X X Copy-On- Update-Partial- Redo X X X

27 Checkpointing Algorithms All Objects Dirty Objects Eager Copy Copy On Update Double Backup Log Naive-Snapshot X X X Copy-On-Update Dribble-And- Atomic-Copy- Dirty-Objects X X X X X X Partial-Redo X X X Copy-On-Update X X X Copy-On- Update-Partial- Redo X X X

28 Checkpointing Algorithms Naive-Snapshot Partial-Redo Copy-On-Update All Objects Dirty Objects All Objects Dirty Objects All Objects Dirty Objects Log Double Backup Log Double Backup Log Double Backup Eager Copy COU Eager Copy COU Eager Copy COU

29 All Objects Dirty Objects Naive-Snapshot Log Eager Copy Double Backup COU Update Draw Update Draw

30 All Objects Dirty Objects Naive-Snapshot Log Eager Copy Double Backup COU Update Draw Update Draw

31 All Objects Dirty Objects Partial-Redo Log Eager Copy Double Backup COU Update Draw Update Draw

32 All Objects Dirty Objects Partial-Redo Log Eager Copy Double Backup COU Update Draw Update Draw

33 All Objects Dirty Objects Copy-On-Update Log Eager Copy Double Backup COU Update Draw Copy object to main memory before updating it Update Draw

34 All Objects Dirty Objects Copy-On-Update Log Eager Copy Double Backup COU Update Draw Copy object to main memory before updating it Update Draw

35 All Objects Dirty Objects Copy-On-Update Log Eager Copy Double Backup COU Update Draw Copy object to main memory before updating it Update Draw

36 All Objects Dirty Objects Copy-On-Update Log Eager Copy Double Backup COU Update Draw Copy object to main memory before updating it Update Draw

37 Performance Model Components In main-memory copy time Disk write time Update handler overhead Recovery Time

38 Performance Model Components In main-memory copy time Disk write time Update handler overhead Recovery Time T memcopy (k) = k S obj B mem T disk-write (k) = k S obj B disk T overhead = O bit + O lock + T memcopy T recovery = T restore + T replay T restore (Partial-Redos) = (k C + n) S obj B disk T restore (The other algorithms) = n S obj B disk

39 Update Handler Overhead T overhead = O bit + O lock + T memcopy O bit O lock T memcopy Overhead to test a dirty bit The cost to lock an object Cost of a memory copy of an atomic object T memcopy = S obj B mem S obj B mem Size of an object Memory bandwidth

40 Experimental Validation Implemented Naive-Snapshot and Copy-On-Update in C++ Running on our high-end server Intel Core 2 Quad 2.4 Ghz 4MB cache 4GB RAM Ubuntu (kernel ) 80GB 7200rpm SATA drive with 8MB cache Game ticks operate at 30Hz

41 Overhead Time

42 Checkpointing Time

43 Recovery Time

44 Experiments Scaling with update rates Prototype game

45 Data Set 1: Synthetic Trace Generate updates according to a Zipf distribution Vary the number of updates per tick from 1k to 256k updates per tick

46 Data Set 2: Prototype Game Trace We simulated a medieval battle of the type common in many MMOs Knights, archers and healers, divided into two teams The objective is to defeat as many enemies as possible Refer to

47 Simulation Parameters Memory Bandwidth Memory Latency Lock Overhead Bit test/set Overhead Disk Bandwidth B mem O mem O lock O bit B disk From our micro-benchmarks running on Intel Core 2 Quad 2.4 Ghz 4MB cache 4GB RAM Ubuntu (kernel ) 80GB 7200rpm SATA drive with 8MB cache

48 Performance Criteria Average Overhead Time Overhead Distribution Recovery Time Checkpointing Time

49 Scaling on Number Of Updates Per Tick (Overhead Time) Avg. Overhead Time [sec], logscale Naive Snapshot Partial Redo Copy-on-Update # Updates per Tick, logscale

50 Scaling on Number Of Updates Per Tick (Overhead Time) Avg. Overhead Time [sec], logscale Naive Snapshot Partial Redo Copy-on-Update # Updates per Tick, logscale

51 Latency Analysis (8k updates per tick) Tick Length [sec], logscale Latency Limit Naive Snapshot Partial Redo Copy-on-Update Tick #

52 Scaling on Number Of Updates Per Tick (Overhead Time) Avg. Overhead Time [sec], logscale Naive Snapshot Partial Redo Copy-on-Update # Updates per Tick, logscale

53 Scaling on Number Of Updates Per Tick (Overhead Time) Avg. Overhead Time [sec], logscale Naive Snapshot Partial Redo Copy-on-Update # Updates per Tick, logscale

54 Latency Analysis (256k updates per tick) Tick Length [sec], logscale Latency Limit Naive Snapshot Partial Redo Copy-on-Update Tick #

55 Scaling on Number Of Updates Per Tick (Checkpointing Time) Avg. Time to Checkpoint [sec], logscale Naive Snapshot Partial-Redo Copy-on-Update # Updates per Tick, logscale

56 Scaling on Number Of Updates Per Tick (Recovery Time) Est. Recovery Time [sec], logscale Naive Snapshot Partial Redo Copy-on-Update # Updates per Tick, logscale

57 Experiments with Prototype Game Server (Overhead Time)

58 Experiments with Prototype Game Server (Checkpointing Time)

59 Experiments with Prototype Game Server (Recovery Time)

60 Conclusion Methods based on double backup recover much faster than methods checkpointing to logs For moderate update rates, the best method in terms of both latency and recovery time is Copy-On-Update In case of extremely high update rates, we recommend Naive-Snapshot Available to download at :

61 References K. Salem and H. Garcia-Molina. Checkpointing Memory-Resident Databases. In Proc. ICDE, W. White, A. Demers, C. Koch, J. Gehrke, and R. Rajagopalan. Scaling Games to Epic Proportions. In Proc. SIGMOD, G. Bronevetsky, R. Fernandes, D. Marques, K. Pingali, and P. Stodghill. Recent Advances in Checkpoint/Recovery Systems. In Proc. IPDPS, D. J. DeWitt, R. Katz, F. Olken, L. Shapiro, M. Stonebraker, and D. Wood. Implementation Techniques for Main Memory DatabaseSystems. In Proc. SIGMOD, D. Rosenkrantz. Dynamic Database Dumping. In Proc. SIGMOD, Richard Fujimoto. Parallel and Distributed Simulation Systems. John Wiley & Sons, 2000.

62 Conclusion Methods based on double backup recover much faster than methods checkpointing to logs For moderate update rates, the best method in terms of both latency and recovery time is Copy-On-Update In case of extremely high update rates, we recommend Naive-Snapshot Available to download at :

63 SSD - Scaling on Number Of Updates Per Tick (Overhead Time) Avg. Overhead Time [sec], logscale Naive Snapshot Dribble and Copy on Update Atomic Copy Dirty Objects Partial Redo Copy-on-Update Copy-on-Update-Partial-Redo # Updates per Tick, logscale

64 SSD - Latency Analysis (64k updates per tick) Tick Length [sec], logscale Latency Limit Naive Snapshot Dribble and Copy on Update Atomic Copy Dirty Objects Partial Redo Copy-on-Update Copy-on-Update-Partial-Redo Tick #

65 SSD - Scaling on Number Of Updates Per Tick (Checkpointing Time) Avg. Time to Checkpoint [sec], logscale Naive Snapshot Dribble and Copy on Update Atomic-Copy-Dirty-Objects Partial-Redo Copy-on-Update Copy-on-Update-Partial-Redo # Updates per Tick, logscale

66 SSD - Scaling on Number Of Updates Per Tick (Recovery Time) Est. Recovery Time [sec], logscale Naive Snapshot Dribble and Copy on Update Atomic Copy Dirty Objects Partial Redo Copy-on-Update Copy-on-Update-Partial-Redo # Updates per Tick, logscale