Atomic Coherence: Leveraging Nanophotonics to Build Race-Free Cache Coherence Protocols. Dana Vantrease, Mikko Lipasti, Nathan Binkert

Transcription

1 Atomic Coherence: Leveraging Nanophotonics to Build Race-Free Cache Coherence Protocols Dana Vantrease, Mikko Lipasti, Nathan Binkert 1

2 Executive Summary Problem: Cache coherence races make protocols complicated Solution: Atomic Coherence framework to protect in-flight coherence requests from encountering races Evaluation: Nanophotonics implementation (for speed) 2

3 Outline Substrate Implementations (9 slides) Results (4 slides) Coherence Complexity (7 slides) Atomic Coherence (3 slides) 3

4 Cache Coherence Cache Coherence: Goal: Maintain a proper ordering of reads/writes to an address How: Cache coherence protocol Core 0 $CACHE$ A=0xCAFE Core 1 $CACHE$ A=0xCAFE A=0xCAFE 4

5 Example: MSI (SGI-Origin-like, directory, invalidate) Stable States 5

6 Example: MSI (SGI-Origin-like, directory, invalidate) Stable States Transitory States 6

7 Example: MSI (SGI-Origin-like, directory, invalidate) Stable States Transitory States Races ( unexpected events from conflicting requests) 7

8 Cache Coherence Complexity L2 MOETSI Transitions (broadcast snooping bus) [Lepak Thesis, 03] 8

9 Cache Coherence Verification Headache Papers: So Many States, So Little Time: Verifying Memory Coherence in the Cray X1 Intel Core 2 Duo Errata: AI39. Cache Data Access Request from One Core Hitting a Modified Line in the L1 Data Cache of the Other Core May Cause Unpredictable System Behavior Formal Methods: e.g. Leslie Lamport s TLA+ (4 people, 2 man years for EV6) 9

10 Cache Coherence Address the problem: Verification RACES Headache Papers: So Many States, So Little Time: Verifying Memory Coherence in the Cray X1 Intel Core 2 Duo Errata: AI39. Cache Data Access Request from One Core Hitting a Modified Line in the L1 Data Cache of the Other Core May Cause Unpredictable System Behavior Formal Methods: e.g. Leslie Lamport s TLA+ (4 people, 2 man years for EV6) 10

11 Addressing Races Address the problem: RACES Races: Are common transitions in FSM, but are rare in practice Were easier to detect/resolve back in the day than they are today Blocking Bus: $ $ $ Mesh: $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ (no physical ordering point) 11

12 Outline Results Coherence Complexity Substrate Implementations Atomic Coherence 12

13 Atomic Coherence: Simplicity Goal Traditional: Atomic Coherence: How are races dealt with? 13

14 14 Atomic Coherence: Race Resolution One coherence request per address active at a time How: Mutex system (ideally mutex-per-cacheline) 2 1 Seize Mutex[Addr] Perform Coherence on Addr Release Mutex[Addr]

15 Atomic & Coherence Substrates Atomic Substrate (Races dealt with here) S I Coherence Substrate (No races here) M Separate Substrates: + Decoupled + Simple(r) - Longer miss path - Requires resources 15

16 Outline Results Coherence Complexity Substrate Implementations Atomic Coherence 16

17 Substrate Implementations Aggressive (w/ nanophotonics b/c we want to minimize overhead) Aggressive (w/ extra features b/c it s easy w/o races) + + S + M I Atomic Substrate Coherence Substrate 17

18 Substrate Implementations Aggressive (w/ nanophotonics b/c we want to minimize overhead) Silicon Nanophotonics (Target Year ~2017) - Mutex = pulse of light - Mutexes continually circulate - Mutex Free/Taken = Light/Darkness Atomic Substrate 18

19 Aggressive Atomic Substrate $ $ $ λ0 λ1 λ λ0 λ1 λ63 Detectors $ ( ) $ $ $ $ $ $ $ $ Dateline Injectors $ $ $ $ = 1 mutex = 2 cm =1 5 GHz 4mutex λ = λ 4waveguides waveguide mutexes (1024 < # cache lines, so employ hash) (All 1024 mutexes pass by a node every 4 cycles) 2 cm 19

20 Photonic Operation Idle: Active: λ0 λ1 λ λ0 λ1 λ63 Detectors Injectors λ0 λ1 λ63 Detectors Injectors λ0 λ1 λ λ0 λ1 λ63 $ Injectors Detectors... λ0 λ1 λ Ring Resonator (like a λ Magnet) ~3.5 μm [HP] 20

21 Substrate Implementations Aggressive (b/c we want to minimize overhead) Aggressive (b/c it s easy w/o races) + + S + M I Atomic Substrate Coherence Substrate 21

22 Extension: Coherence Substrate It is easy to add protocol enhancements in the absence of races Proposal: Maintain shared sourcing states (O/F). F $ $ F State: (like a clean O state) I S S S S S $ $ $ $ vs $ I No F State: (evicted or not supported) I S $ S S S S $ $ $ $ Directory Directory 22

23 ShiftF F I I S F S $ $ $ $ Evict ShiftF Directory Fail ShiftF Atomic: { Evict-F Shift-F } Proposal: Maintain shared sourcing states (O/F). Method: Upon O/F block eviction, try to shift O/F state to sharer 23

24 Substrate Implementations Aggressive (b/c we want to minimize overhead) Aggressive (b/c its easy w/o races) + + S + M I Atomic Substrate Coherence Substrate 24

25 Outline Results Coherence Complexity Substrate Implementations Atomic Coherence 25

26 λ0 λ1 λ63 Experimental Setup λ0 λ1 λ $ $ $ $ $ $ $ $ $ $ $ $ Dateline $ $ $ $ Studies: Complexity Performance System Cores Interconnect Atomic Substrate Mutexes Atomicity Coherence 128, single-threaded, in-order, 5GHz Optical data & control (64 wg) 1024 (4 wg) 4 cycles revolution Guaranteed through L2 control permissions MOEFSI, directory, invalidate 26

27 Complexity Results (L2 transitions) # Unique L2 Transitions Static Analysis Non-Atomic Atomic MESI MOEFSI MOEFSI Atomic +4-9 = -5 MOEFSI Atomic + Shift F Dynamic Analysis # L2 Transitions: traditional MESI > atomic MOEFSI Atomic protocols are fully exercised by our random tester. 27

28 Performance Results Converting MOEFSI to atomic causes a slight slowdown * Relative to nonatomic MOEFSI ShiftF (more than) overcomes the slowdown Mutex circulation and hashing conflicts are contributors to slowdown 28

29 Atomic Coherence Conclusion Atomic Protocols are simpler protocols Atomic Protocols can have high performance Aggressive atomic substrate w/ nanophotonics Aggressive coherence substrate w/ ShiftF 29

30 Thanks! 30

31 Experimental Setup Component System Cores Interconnect L1 Caches L2 Caches Directory Memory Atomic Substrate Detail Mutexes 1024 Waveguides Rev. Period Coherence 128, single-threaded, in-order Optical (Corona bw scaled by ¼) 64 wg, < 4 cycles, 2 cycles/msg 64, 32 KB, 4-way 16, 2 MB, 16-way 16, 24-way 80 ns, 4 cycles/msg 4 wg < 4 cycles MOEFSI, directory, inv WORKLOADS Scientific/Commercial: 16 threads * 8 instances L1-Miss Trace-Driven (GEMS) 31