PVCoherence. Zhang, Bringham, Erickson, & Sorin. Amlan Nayak & Jay Zhang 1

Transcription

1 PVCoherence Zhang, Bringham, Erickson, & Sorin Amlan Nayak & Jay Zhang 1

2 (16) OM Overview M Motivation Background Parametric Verification Design Guidelines PV-MOESI vs OP-MOESI Results Conclusion E MI MI2 SD I 2

3 Issue with Coherence Protocols Difficult to automatically verify for many core systems Better performance Complex protocols Difficult to formally verify 3

4 GOAL Architect arbitrarily large flat protocols such that they can be verified using a mostly-automated methodology 4

5 (16) OM Overview M Motivation Background Parametric Verification Design Guidelines PV-MOESI vs OP-MOESI Results Conclusion E MI MI2 SD I 5

6 Coherence Protocols Two Primary Categories: Snooping & Directory-based (ex. MSI, MESI, MOESI, MESIF) 6

7 L2 $ Directory Controller Network on Chip Core 0 Core 1 Core 2 Core 3 7

8 State Space Exploration Formally verifying a coherence protocol Theorem Proving 8

9 State Space Exploration (with Murphi) 9

10 GOAL Architect arbitrarily large flat protocols such that they can be verified using a mostly-automated methodology 10

11 GOAL Architect arbitrarily large flat protocols such that they can be verified using a mostly-automated methodology 11

12 Model Protocol in Murphi Check invariants 12

13 Model Protocol in Murphi Check invariants 13

14 Murphi Processor Node State Definition Input messages Cache line State Output transition/messages 14

15 MOESI protocol Processor Cache Controller I 15

16 MOESI protocol Processor Cache Controller I 16

17 MOESI protocol Processor Cache Controller ISD I 17

18 MOESI protocol Processor Cache Controller ISD I 18

19 MOESI protocol Processor Cache Controller M OMA OMAC O MIA MA OIA E MI3 MAD SIA S MI2 IIA ISS ISD ISI EI I IEI SEI 19

20 Model Protocol in Murphi Check invariants 20

21 Check invariants 1. Permission Invariant: Single-Writer, Multiple-Reader 2. Data Invariant: Read returns value of last write 21

22 Check invariants 1. Permission Invariant: Single-Writer, Multiple-Reader 2. Data Invariant: Read returns value of last write 22

23 Cache Line ST Core 0 Core 1 Core 2 Core 3 SINGLE Writer 23

24 Cache Line LD LD LD Core 0 Core 1 Core 2 Core 3 SHARER SHARER SHARER Multiple Reader 24

25 Check invariants 1. Permission Invariant: Single-Writer, Multiple-Reader 2. Data Invariant: Read returns value of last write 25

26 (16) OM Overview M Motivation Background Parametric Verification Design Guidelines PV-MOESI vs OP-MOESI Results Conclusion E MI MI2 SD I 26

27 PARAMETRIC VERIFICATION (PV) Treat number of nodes as a parameter (using Abster tool) Prove properties of design agnostic to parameter size This process scales to many nodes and is almost fully automatic 27

28 Simple-PV Process Flow How to design a readily verifiable Coherence Protocol 28

29 Create model w/ small # nodes 1. Make Param. Model (Abster) Fail NOT VERIFIABLE! Success 2. Model Check (using Murphi) Success Verification Success Fail Fix Bug Yes Counter Example Bug in Protocol? NOT VERIFIABLE! State Space Explosion No 3. Refine Model (Manual) Yes Refinable? No Simple-PV Process Flow 29

30 Create model w/ small # nodes 1. Make Param. Model (Abster) Fail Create model NOTw/ small # VERIFIABLE! nodes Success 2. Model Check (using Murphi) Success Verification Success Fail Fix Bug Yes Counter Example Bug in Protocol? NOT VERIFIABLE! State Space Explosion No 3. Refine Model (Manual) Yes Refinable? No Simple-PV Process Flow 30

31 Automatically Create Parametric Model Create parametric model from non-parametric model N concrete nodes -> 2 concrete nodes + Other Node (N-2) Abster automatic abstraction tool Abster over-approximates the behavior of the N-2 nodes If Abster fails, modify protocol until it is compatible 31

32 To Memory L2 $ N-2 parameterized nodes Directory Controller Network on Chip Core 0 Core 1 Core 2 Core 3 32

33 Create model w/ small # nodes 1. Make Param. Model (Abster) Fail Create model NOTw/ small # VERIFIABLE! nodes Success 2. Model Check (using Murphi) Success Verification Success Fail Fix Bug Yes Counter Example Bug in Protocol? NOT VERIFIABLE! State Space Explosion No 3. Refine Model (Manual) Yes Refinable? No Simple-PV Process Flow 33

34 Automatically Model Check the model (MURPHI) 34

35 Create model w/ small # nodes 1. Make Param. Model (Abster) Fail Create model NOTw/ small # VERIFIABLE! nodes Success 2. Model Check (using Murphi) Success Verification Success Fail Fix Bug Yes Counter Example Bug in Protocol? NOT VERIFIABLE! State Space Explosion No 3. Refine Model (Manual) Yes Refinable? No Simple-PV Process Flow 35

36 Manually Refine the Model Over-approximation leads to spurious invariant violations Must modify behavior of Other node KEY: Add constraint and check their validity Add invariant (lemma) must be true for non-abstracted model Check on the concrete nodes 36

37 System Architecture for PVCoherence 37

38 Memory Inclusive L2 $ Inclusive Directory $ Network on Chip Core 0 Core 1 Core 2 38

39 $ LD Miss ST Miss GetS/GetE GetM CORE Eviction PutM PutO PutE 39

40 Network on Chip VC0 VC1 VC2 Requests Responses Forwards 40

41 Objective HUMAN EFFORT AUTOMATION during protocol design 41

42 (16) OM Overview M Motivation Background Parametric Verification Design Guidelines PV-MOESI vs OP-MOESI Results Conclusion E MI MI2 SD I 42

43 Guidelines for Simple-PV compliance 43

44 #1: Identical Nodes #2: No variables must depend on number of Nodes #3: No ordering over list/queue sized by number of nodes #4: Should not parameterize buffers/queues in more than 1-dim. 44

45 Identical Nodes Memory Inclusive L2 $ Inclusive Directory $ Network on Chip Core 0 Core 1 Core 2 45

46 Heterogeneous Nodes Memory Inclusive L2 $ Inclusive Directory $ Network on Chip Core 0 Core 1 Core 2 Core 0 Core 0 Core 1 Core 1 Core 2 Core 2 46

47 #1: Identical Nodes #2: No variables must depend on number of Nodes #3: No ordering over list/queue sized by number of nodes #4: Should not parameterize buffers/queues in more than 1-dim. 47

48 N-1 Nodes to track (N not known) Core 1 Core 0 Core 0 TAG STATE DATA Sharer Count Core 0 Cache Line 48

49 N-1 Nodes to track (N not known) Core 1 Core 0 Core 0 TAG STATE DATA Sharer Count Core 0 Cannot compare with parameterized value or carry out math operations 49

50 N-1 Nodes to track (N not known) Core 1 Core 0 Core 0 TAG STATE DATA Sharer Count Sharer Set Core 0 Replace with sharer set (bit vector) 50

51 #1: Identical Nodes #2: No variables must depend on number of Nodes #3: No ordering over list/queue sized by number of nodes #4: Should not parameterize buffers/queues in more than 1-dim. 51

52 Inclusive L2 $ Inclusive Directory $ Network on Chip Req3 Req2 Req1 Req0 Independent FIFO Request Queue Is permitted by this method Core 0 Core 1 Core 2 52

53 Inclusive L2 $ Inclusive Directory $ Shared Request Queue FIFO UNORDERED Network on Chip Req2 Req1 Req0 Core 0 Core 1 Core 0 Core 0 53

54 #1: Identical Nodes #2: No variables must depend on number of Nodes #3: No ordering over list/queue sized by number of nodes #4: Should not parameterize buffers/queues in more than 1-dim. 54

55 NO Buffer[N][N] Practical limitation Ack Ack(s) Core 0 Core 1 Core 0 Core 0 55

56 L2 (Directory) collects all invalidation Acks Memory Inclusive L2 $ Inclusive Directory $ Network on Chip GetM Ack Ack Core 0 Core 1 Core 0 Core 0 56

57 L2 Sends aggregate Ack to Core 0 Memory Inclusive L2 $ Inclusive Directory $ Network on Chip GetM Ack + data Core 0 Core 1 Core 0 Core 0 57

58 (16) OM Overview M Motivation Background Parametric Verification Design Guidelines PV-MOESI vs OP-MOESI Results Conclusion E MI MI2 SD I 58

59 Optimizations (OP-MOESI) 59

60 Optimization Compatible with SimplePV? Impact Adding Exclusive State Y NO IMPACT Adding Owned state Y Add 2 lemmas Adding Self-Upgrade Y Add lemma Adding Silent Evictions Y Add lemma Removing the completion messages for GetS when data response comes from L2$ Y Add lemma Removing the completion messages for GetM N -- 60

61 To ensure successful abstraction by Abster For GetM OP-MOESI to PV-MOESI Replace response counter with sharer set Let L2 collect Acks and send aggregated Ack to requesting L1 Remove point-to-point ordering in all VCs and avoid races by adding extra messages or transient states but without blocking L1 61

62 OP-MOESI to PV-MOESI To ensure successful verification by Murphi Problem: multiple in flight GetM requests without ordering Solution: L2 blocks other subsequent requests whenever it receives a GetM request until it receives a Completion message from the requesting L1 Impact: performance decrease due to blocking at L2 62

63 Evaluation: OP-MOESI vs. PV-MOESI 63

64 Runtime 64

65 Network Traffic Overhead 65

66 Performance Scalability 66

67 Storage Overhead TAG STATE DATA SHARER SET SHARER COUNTER TAG STATE DATA SHARER L2$ SET NO CHANGE Overhead is generally negligible 67

68 CONCLUSION (16) Design of parametrically verifiable coherence protocols is possible given that the guidelines introduced here are adhered to M There is no significant performance drawbacks or storage overheads MI Automation is key E MI2 68 I

69 Debate Is it necessary for a protocol to be parametrically verifiable? There are a few design corners that are cut to make such PV-compliant protocols work. Is this worth it? 69