Accelerating Multiprocessor Simulation with a Memory Timestamp Record

Transcription

1 Aelerating Multiproessor Simulation with a Memory Timestamp Reord Kenneth Barr Heidi Pan Mihael Zhang Krste Asanovi Marh, 5 Massahusetts Institute of Tehnology

2 Intelligent sampling gives est speed-auray tradeoff for uniproessors (Yi, HPCA `5) Single sample Fastforward + single sample Fastforward + Warmup + sample detailed ISA only detailed ISA only measure d e t a i l e d ignored ignored ignored Seletive Sampling (SimPoints) Statistial Sampling Statistial sampling w/ Fast Funtional Warming (SMARTS, FFW) ISA+µarh Memory Timestamp Reord ISA+MTR Update Reonstrut ahes Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

3 Snapshots amortize fast-forwarding, ut require slow warming or ind to a partiular µarh ISA only snapshots: Slow due to warmup, ut allows any µarh ISA+µarh snapshots: Fast (less warmup), ut tied to µarh MTR snapshots: Fast, NOT tied to µarh, supports multiproessors Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

4 Multiproessors simulation is espeially slow More ores More state/omplexity Long, omplex simulations CPU CPU CPUn $ $ $ Memory Diretory Full system, threaded apps More variaility More simulation CPUs Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 4 time

5 For full-system simulations of ommerial workloads, sutle variation matters! (Alameldeen and Wood, ) CPU 4 Time =.5 All produe same result, eah has different runtime DRAM refresh Hard disk arrangement delays DMA Inoming paket interrupts appliation Loking order reversed Time =.8 Time =. Proesses migrate Is our new gizmo a suess? Maye OS just ordered threads differently! Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 5

6 What is the Memory Timestamp Reord (MTR)? MTR is astrat piture of an multiproessor s oherene state CPUn- Last Writetime Last Writer N- Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 6

7 What is the Memory Timestamp Reord (MTR)? MTR is astrat piture of an multiproessor s oherene state Allow quik update during fast forwarding Fill in onrete ahes and diretory prior to sampling CPUn- Last Writetime Last Writer CPU CPU CPUn $ $ $ N- Memory Diretory Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 7

8 MTR example: update MTR ontains one entry per memory lok; loality keeps it sparse. New aess times overwrite old (self-ompressing) MTR: a d e CPUn- Memory Trae: Time 4 Last Writetime CPU Last Writer Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 8

9 MTR example: update MTR ontains one entry per memory lok; loality keeps it sparse. New aess times overwrite old (self-ompressing) MTR: a d e CPUn- Memory Trae: Time 4 Last Writetime Read a CPU Last Writer Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 9

10 MTR example: update MTR ontains one entry per memory lok; loality keeps it sparse. New aess times overwrite old (self-ompressing) MTR: a d e CPUn- Memory Trae: Time 4 Last Writetime Read a Read e CPU Last Writer Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

11 MTR example: update MTR ontains one entry per memory lok; loality keeps it sparse. New aess times overwrite old (self-ompressing) MTR: a d e CPUn- Memory Trae: Time 4 Last Writetime Read a Read e Read CPU Last Writer Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

12 MTR example: update MTR ontains one entry per memory lok; loality keeps it sparse. New aess times overwrite old (self-ompressing) MTR: a d e CPUn- Memory Trae: Time 4 Last Writetime Read a Read e Read Read CPU Last Writer Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

13 MTR example: update MTR ontains one entry per memory lok; loality keeps it sparse. New aess times overwrite old (self-ompressing) MTR: a d e CPUn- 4 Memory Trae: Time 4 Last Writetime Read a Read e Read Read CPU CPU Write Last Writer Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

14 . Coalese: determining orret ahe tags MTR: CPUn- Last Writetime Last Writer Cahe: Way Way Set Set Set Set Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 4

15 . Coalese: determining orret ahe tags MTR: CPUn- Last Writetime Last Writer Cahe: Way Way Set Set Set Set Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 5

16 . Coalese: determining orret ahe tags MTR: CPUn- Last Writetime Last Writer Cahe: Way Way Set Set Set Set Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 6

17 MTR example: oalese Choose organization One set, two ways Coalese Determine whih loks map to same set Only ways most reent timestamps are present. Chek validity later. CPUn- Last Writetime Last Writer a 4 CPU d e Way Way Way Way Set Set Set Set Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 7

18 MTR example: oalese Choose organization One set, two ways Coalese Determine whih loks map to same set Only ways most reent timestamps are present. Chek validity later. CPUn- Last Writetime Last Writer a 4 CPU d e What are the ontents of ahe? Way Way Set Set Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 8

19 MTR example: oalese Choose organization One set, two ways Coalese Determine whih loks map to same set Only ways most reent timestamps are present. Chek validity later. CPUn- Last Writetime Last Writer a 4 CPU d e What are the ontents of ahe? Way Way Set Set a Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 9

20 MTR example: oalese Choose organization One set, two ways Coalese Determine whih loks map to same set Only ways most reent timestamps are present. Chek validity later. CPUn- Last Writetime Last Writer a 4 CPU d e What are the ontents of ahe? Way Way Set Set a Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

21 MTR example: oalese Choose organization One set, two ways Coalese Determine whih loks map to same set Only ways most reent timestamps are present. Chek validity later. CPUn- Last Writetime Last Writer a 4 CPU d e What are the ontents of ahe? Way Way Set a Set Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

22 MTR example: oalese Choose organization One set, two ways Coalese Determine whih loks map to same set Only ways most reent timestamps are present. Chek validity later. CPUn- Last Writetime Last Writer a 4 CPU d e What are the ontents of ahe? Way Way Set e Set Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

23 MTR example: oalese Choose organization One set, two ways Coalese Determine whih loks map to same set Only ways most reent timestamps are present. Chek validity later. CPUn- Last Writetime Last Writer a 4 CPU d e What are the ontents of ahe? CPU? Way Way Way Way Set e Set Set Set 4 Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

24 Fixup: determine orret status its Way Way Set Set Set Set Cahe Way Way Set Set Set Set Cahe Way Way Set Set Set Set Cahe n- Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 4

25 MTR example: fixup Reads prior to a write are invalid, valid writes are dirty, et CPUn- Last Writetime Last Writer a 4 CPU d e Whih ahe has the most reent opy of? Way Way Way Way Set e Set Set Set 4 invalid Valid, dirty Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 5

26 MTR example: diretory reonstrution CPUn- Last Writetime Last Writer a 4 CPU d e a d e State S M S I S Sharers CPU Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 6

27 The MTR supports many popular organizations and protools Snoopy or diretory-ased Multilevel ahes Inlusive Exlusive Time-ased replaement poliy Strit LRU Cahe deay Invalidate, Update, Update-Invalidate MSI, MESI, MOESI Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 7

28 Evits annot e reorded in the MTR, ut many an e inferred MTR: address CPU Writetime Writer n+k n CASE A: writes reads = dirty CASE B: writes writes eviting reads = lean n Time n+k Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 8

29 Evaluation / Results

30 Detailed, full-system, exeution-driven, x86, SMP simulation CPU N- CPU CPU Stall Detailed Memory System Magi Memory Main Memory Bohs Memory Timestamp Reord SMP Bohs provides: devies (allowing an OS), x86 Deoding and Exeution, Magi Memory Detailed Mode Enale Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

31 Detailed, full-system, exeution-driven, x86, SMP simulation CPU N- CPU CPU Stall Detailed Memory System Magi Memory Main Memory Bohs Memory Timestamp Reord SMP Bohs provides: devies (allowing an OS), x86 Deoding and Exeution, Magi Memory Detailed Mode Enale Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

32 Reonstruting with the MTR CPU N- CPU CPU Stall Detailed Memory System Magi Memory Main Memory Bohs Memory Timestamp Reord Memory Timestamp Reord: allows swithing etween funtional fast-fwd and detailed simulation Detailed Mode Enale Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

33 Our detailed memory model an stall a proessor s exeution ased on timing models CPU N- CPU CPU Stall Detailed Memory System Magi Memory Main Memory Bohs Memory Timestamp Reord Detailed Memory System provides: ahe oherene, network, DRAM timing, stall signal Detailed Mode Enale Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5.

34 Benhmarks! NASA Advaned Superomputing Parallel Benhmarks: sientifi (omp. fluid dynamis) OpenMP (loop iterations in parallel) Fortran OS enhmarks denh: (Sama) several lients making file-entri system alls Apahe: several lients hammer we server (via loopak interfae) Cilk: hekers: AI searh plies in parallel uses spawn/syn primitives (dynami thread reation/sheduling) Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 4

35 We ompare MTR to full detailed and a naïve implementation of SMP fast forwarding. Baseline : full detailed simulation (overnight) Baseline : naïve funtional fast forwarding (FFW) Funtional simulation of ISA Cahe / diretory state kept aurate Tag heks, replaement poliy enfored Diretory onsulted and updated On miss/oherene miss, invalidate outstanding opies Omits network messages, queues, latenies (present in detailed mode) Hypothesis Both FFW and MTR should e aurate and fast MTR should e faster than FFW To e useful, FFW and MTR must answer questions in the same way as a detailed model, ut faster Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 5

36 Full-system experiments must respet system variation, or risk inorret predition! Methodology Every k yles hoose vitim proessor Vitim will run 5% slower to emulate variation Note: variation has MUCH larger effet during fast mode Bar shows the median of eight runs, with tiks for min and max. Eah run is a valid result! Ideal: range for fast runs should e within range of (all possile) detailed. Can t draw onlusions aout disrepany until we understand distriution denh Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 6 Cahe miss rate (%) : : :

37 Repliating detailed -mode stats less ruial than aurate answers to design questions Assume oserved = atual With respet to reply message types, the MSI vs. MESI hange is dramati. All fast-fwd ars move with the detailed ar. Movement eyond range of detailed runs Disover evits to more losely math detailed run Or, tune vitim/slowdown (no amig. resolution) (no amig. resolution) writeak rep Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 7

38 Runtime (normalized to FFW :) MTR averages up to.45x faster than FFW FFW (mg) : : : MTR (mg) : : : Detailed Simulation Detailed Warming Fast to detailed Fast Forward MTR spends less time in fast forward MTR does less work in ommon ase Time saved in fast forward time less than MTR transition ost MTR has ostlier transition, ut Reonstrution sales with touhed lines not total aesses Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 8

39 Conlusion: Memory Timestamp Reord provides fast, aurate, and flexile SMP simulation MTR.45X faster than funtional warming 7.7X faster than our detailed simulator Eliminates need to regenerate snapshots Answers arhitetural questions similar to detailed simulation Future work Simultaneous multiple-onfiguration simulation MTR ompression for disk snapshots Parallelized update and reonstrution Barr, Pan, Zhang, and Asanović. ISPASS. Marh, 5. 9