Verifying Cache Coherence in ACL2. Ben Selfridge Oracle / UT Austin

Transcription

1 erifying Cache Coherence in ACL Ben Selfridge Oracle / UT Austin

2

3 Goals of this talk Define cache coherence Present a cache coherence protocol designed Present an ACL proof that the protocol is safe (whatever that means) Discuss how we might use ACL to verify more complicated protocols s it worth it? (Why not just use a model checker?) s it possible? (nductive invariants are hard )

4 Goals of this talk Define cache coherence Present a cache coherence protocol designed Present an ACL proof that the protocol is safe (whatever that means) Discuss how we might use ACL to verify more complicated protocols s it worth it? (Why not just use a model checker?) s it possible? (nductive invariants are hard )

5 What are caches? Small, quick-access memory on a chip Used for repeated accesses to the same locations To read/write, the processor must obtain the line from memory, copy it to cache When it s done writing, the cache line is copied back to main memory (at some point)

6 Cache Coherence With + processors, this gets complicated We can allow multiple processors to read simultaneously But write simultaneously? Hmm

7 Cache Coherence 0 0

8 Cache Coherence 3 0

9 Cache Coherence 3 4

10 Cache Coherence 3 4 BAD!!! Both caches believe they have up-to-date copies of the memory location, but they see different values!

11 Cache Coherence To prevent this from happening, add state to each cache line (invalid, read-only, read-write) Two in read-only? Allowed Two in read-write? Not Allowed One in read-only, one in read-write? Not Allowed These guarantees are commonly called cache coherence.

12 Cache Coherence Protocols To ensure coherence, cache coherence protocols are used to manage the state across caches Cores send messages to communicate f want a cache line, must request it f hold a cache line, must send my data back at some point Network properties vary widely between protocols Protocols must be designed ERY CAREFULLY to maintain coherence

13 Example Protocol: designed a simple cache coherence protocol called. Cache lines can be in one of two states: = valid (read/write) = invalid There is no read-only state!

14 Example Protocol: There are n caches along with an additional agent, the directory, residing in the main memory The directory keeps track of who currently owns each cache line (either a cache, or the main memory) For the remainder of this talk, assume there is only one cache line that can be shared between memory and caches. (This avoids confusion.)

15 Caveat: state is interpreted differently The ectory has two states, and in state means has the data, and no one else does in state means does not have the data; either someone else has it in state, or it s currently in transit Remember this, otherwise you ll get confused

16 Transition Tables load/ store Cache Controller evict Fwd-Get Put-Ack Send Get to / D LLEGAL LLEGAL LLEGAL LLEGAL D stall stall Copy to cache / stall perform load/ store Send Put to / A LLEGAL Send to Req / LLEGAL A stall stall LLEGAL Send to Req / A -/ A stall stall LLEGAL LLEGAL -/

17 Transition Tables load/ store Cache Controller evict Fwd-Get Put-Ack Send Get to / D LLEGAL LLEGAL LLEGAL LLEGAL stable states D stall stall Copy to cache / stall perform load/ store Send Put to / A LLEGAL Send to Req / LLEGAL A stall stall LLEGAL Send to Req / A -/ A stall stall LLEGAL LLEGAL -/

18 Transition Tables load/ store Cache Controller evict Fwd-Get Put-Ack Send Get to / D LLEGAL LLEGAL LLEGAL LLEGAL D stall stall Copy to cache / stall transient states perform load/ store Send Put to / A LLEGAL A stall stall LLEGAL Send to Req / Send to Req / A LLEGAL -/ A stall stall LLEGAL LLEGAL -/

19 Transition Tables ectory Controller Get Put (from owner) Put (from non-owner) (Cache line in main mem only) Send to Req, set Owner to Req / LLEGAL Send Put-Ack to Req / - (some cache has cache line in state ) Send Fwd-Get to Owner, set Owner to Req / Copy data to memory, send Put- Ack to Owner, clear Owner / Send Put-Ack to Req / -

20 Example Protocol: Before we go any further, let s take a look at how this protocol works in practice.

21 Get transaction

22 Get transaction Cache wishes to obtain the cache line. is in state, indicating no other cache currently has the data.

23 Get transaction

24 Get transaction D Get

25 Get transaction D Get

26 Get transaction D Get

27 Get transaction

28 Get transaction Cache has successfully obtained the cache line.

29 Get transaction

30 Get transaction

31 Get transaction Cache wants to obtain the cache line, and Cache already has it.

32 Get transaction D Get

33 Get transaction Fwd-Get D Get

34 Get transaction Fwd-Get D Get

35 Get transaction Fwd-Get D Get

36 Get transaction

37 Get transaction Cache has successfully obtained the cache line.

38 Put transaction

39 Put transaction Cache has the cache line, and wishes to evict (transition to ).

40 Put transaction A Put Cache sends a Put to, but does not evict the cache line yet.

41 Put transaction A Put Put-Ack When Cache receives Put-Ack from, it is safe to evict the cache line.

42 Put transaction A Put Put-Ack When Cache receives Put-Ack from, it is safe to evict the cache line.

43 Put transaction Cache has successfully evicted, and now owns the data.

44 Put/Get race What if Cache Puts, and Cache Gets at the same time?

45 A Put/Get race Put

46 A Put/Get race Put Get D

47 A Put/Get race Put??? Get D Which message arrives first?

48 A Put/Get race Put Get D Suppose the Put from Cache arrives first. (We will explore the other case in a moment.)

49 A Put/Get race Put Put-Ack Get D

50 A Put/Get race Put Put-Ack Get D

51 Put/Get race A Put Put-Ack Get D

52 Put/Get race A Put Put-Ack Get D

53 Put/Get race A Put Put-Ack Get D Since received the Put first, there is no need for the two Caches to communicate directly.

54 Put/Get race

55 Put/Get race Cache has evicted successfully, and Cache has obtained the cache line successfully.

56 Put/Get race A Put??? Get D

57 Put/Get race A Put Get D Now, suppose received the Get first.

58 Put/Get race A Put Fwd-Get Get D First, forwards the Get request to Cache, since Cache is still the owner.

59 Put/Get race A Put Fwd-Get Put-Ack Get D Then, receives Cache s Put. Since Cache is no longer the owner, simply responds with a Put-Ack, and throws out the incoming.

60 Put/Get race A Put Fwd-Get??? Put-Ack Get D Which message arrives first?

61 Put/Get race A Put Fwd-Get Put-Ack Get D Suppose the Fwd-Get arrives first.

62 Put/Get race A A Put Fwd-Get Put-Ack Get D Because Cache hasn t evicted yet, he still has the data. He sends it along to Cache and evicts (although he still awaits a Put-Ack from ).

63 Put/Get race A A Put Fwd-Get Put-Ack Get D Cache receives the Put-Ack, and transitions to.

64 Put/Get race A A Put Fwd-Get Put-Ack Get D Cache receives the, and transitions to.

65 Put/Get race

66 Put/Get race Cache has upgraded successfully. Cache s attempt to evict was effectively aborted since the Get request was serviced before the Put request arrived.

67 Put/Get race A Put Fwd-Get??? Put-Ack Get D

68 Put/Get race A Put Fwd-Get Put-Ack Get D Now, suppose Cache receives the Put-Ack first.

69 Put/Get race A Put Fwd-Get Put-Ack Get D Cache, having received Put-Ack, evicts the cache line. (Um. where s the data?)

70 Put/Get race A Put Fwd-Get Put-Ack BAD!!!! Get D Then, Cache receives a Fwd-Get. He can t forward the data, because he already evicted!

71 Put/Get race A Put Fwd-Get Put-Ack BAD!!!! Get D Solution: use the same channel for Fwd-Get and Put-Ack, and require point-to-point ordering.

72 Put/Get race Lesson: A cache coherence protocol may seem relatively simple, but concurrency and data races can lead to some odd behavior. We need to be ERY careful when designing these protocols in order to ensure bad things don t happen.

73 Correctness of We wish to demonstrate that the cache coherence protocol is correct. For our protocol, this means that no two caches can have the cache line in state simultaneously. We believe we have designed our protocol well, but it s actually deceptively complicated We used ACL to construct an invariant-style proof of this property

74 Proof strategy. Let P = property we want to prove is an invariant. Let Props = {P } 3. for each P i in Props: A. Try to prove: Props(m) -> P i (step m) B. For each failed subgoal, create new P j that lets us prove that subgoal, and add it to Props until A is proved by ACL C. Repeat until we have proved everything in Props is preserved by step 4. We have shown Props(m) -> Props(step m). Since P is in Props, we have shown that if we start from a state where Props(m) is true, then we can run the protocol as long as we want, and P will always be true.

75 Next time ll present the correctness proof in some detail. ll talk about some ideas ve had for using ACL both to design AND verify complex, scary cache protocols.

76 Proving correctness n industry, model checkers are usually used to verify coherence for these protocols. For complicated protocols, model checkers can fail to terminate fast enough. Even if you manage to get model checker to finish the proof, you may need to make so many simplifications in the encoding that the proof won t actually convince too many people. m interested how a theorem prover like ACL can be used to aid in these verification efforts.

77 Other stuff

78 Correctness of : nitial At first, we started the proof by specifying correctness as no two caches are in state We then asked ACL to prove that this property was preserved by all transitions Each failed subgoal suggested a new property we needed to assume The hope: at some point, these invariants will become closed Attempt

79 Correctness of : nitial Attempt discovered after a lengthy, time-consuming proof attempt, that had ended needing to assume an invariant that couldn t prove couldn t prove it because the property blew up - in order to prove it, needed to assume something more complicated, and then to prove the more complicated property, needed to assume something even MORE complicated, etc. still can t figure out exactly where went wrong; if anyone has any intuition, or is interested enough to discuss it, let me know

80 needed to prove that this could never happen: t s clear why - both Cache and are in. The thinks is the owner.

81 needed to prove that this could never happen: Let s backtrack and see how this could have happened

82 needed to prove that this could never happen: Let s backtrack and see how this could have happened

83 needed to prove that this could never happen: 3 Let s backtrack and see how this could have happened

84 needed to prove that this could never happen: D 3 Let s backtrack and see how this could have happened

85 needed to prove that this could never happen: D Fwd-Get 3 Let s backtrack and see how this could have happened

86 needed to prove that this could never happen: Fwd-Get D 3 4 Let s backtrack and see how this could have happened

87 needed to prove that this could never happen: Fwd-Get D 3 4 Let s backtrack and see how this could have happened