A Cache Coherence Protocol to Implement Sequential Consistency. Memory Consistency in SMPs

Transcription

1 6.823, L20--1 A Cache Coherence rotocol to Ipleent Sequential Consistency Laboratory for Coputer Science M.I.T. Meory Consistency in SMs CU-1 CU , L20--2 A 100 cache-1 A 100 cache-2 CU-Meory bus A 100 eory Suppose CU-1 updates A to 200. write-back policy: eory and cache-2 have stale values write-through policy: cache-2 has a stale value Do these stale values atter? What is the view of shared eory for prograing? Sequential Consistency, Relaxed eory odels age 1

2 Caches & Sequential Consistency prog T1 ST X, 1 ST Y,11 prog T2 LD Y, R1 ST Y, R1 LD X, R2 ST X,R2 cache-1 X= 1 Y=11 cache-2 Y = Y = X = X = eory X = 0 Y =10 X = Y = Scenario 1 T1 is executed T2 is executed cache-1 writes back X & Y cache-2 writes back X & Y assue a write-back cache prog T1 ST X, 1 ST Y,11 prog T2 LD Y, R1 ST Y, R1 LD X, R2 ST X,R2 cache-1 X= 1 Y=11 cache-2 Y = Y = X = X = Scenario 2 T1 is executed eory X = 0 Y =10 X = Y = 6.823, L20--3 cache-1 writes back Y T2 reads Y T2 writes Y T2 reads X T2 writes X cache-1 writes back X cache-2 writes back X & Y Write-through Caches & Sequential Consistency 6.823, L20--4 prog T1 ST X, 1 ST Y,11 prog T2 LD Y, R1 ST Y, R1 LD X, R2 ST X,R2 cache-1 X= 0 Y=10 cache-2 Y = Y = X = 0 X = eory X = 0 Y =10 X = Y = Write-through caches don t preserve sequential consistency either; consider scenario: T1 is executed T2 is executed age 2

3 6.823, L20--5 Maintaining Sequential Consistency Multiple copies of a location in various caches can cause SC to breakdown. Hardware support is required such that only one processor at a tie has write perission for a location no processor can load a stale copy of the location after a write cache coherence protocols A Syste with Multiple Caches 6.823, L20--6 L2 L2 Interconnect Modern systes often have hierarchical caches Each cache has exactly one parent but can have zero or ore children Only a parent and its children can counicate directly Inclusion property is aintained between a parent and its children, i.e., a L i a L i+1 age 3

4 Cache Coherence rotocols for SC 6.823, L20--7 write request: the address is invalidated or updated in all other caches before the write is perfored read request: if a dirty copy is found in soe cache, a write-back is perfored before the eory is read We will focus on Invalidation protocols State Encoding 6.823, L20--8 (Sh, R(ε)) (Sh, R(ε)) a 1.1 a 1.2 a 1.3 (Sh, R(1.3.1)) Interconnect 1 a (Ex, R( )) Each address in a cache keeps two types of state info sibling info: does any of y siblings have a copy -ExvsSh children info: has this address been passed on to any of y children - W(id) vs R(dir) where dir = id 1... id n directory of children R(ε) eans uncached age 4

5 Cache State Iplications 6.823, L20--9 Sh cache s siblings and decedents can only have Sh copies Ex each ancestor of the cache ust be in Ex either all children can have Sh copies or one child can have an Ex copy Once a parent gives an Ex copy to a child, the parent s data is considered stale A processor cannot overwrite data in Sh state in Initial State 6.823, L L2 L2 Interconnect All M s (caches) except the outerost M (hoe) are epty (ε) Initialize each cell of the outerost eory to be Cell(a,-,(Ex,R(ε))) age 5

6 Load & Store rules 6.823, L pb pb id Load rule <id, Cell(a,v,(cs,R(ε))) >, <t,load(a)>;pb, pb <id, Cell(a,v,(cs,R(ε))) >, pb, pb <t,v> Store rule <id, Cell(a,-,(Ex,R(ε))) >, <t,store(a,v)>;pb, pb <id, Cell(a,v,(Ex,R(ε))) >, pb, pb <t,ack> 6.823, L Data ropagation Between Caches Child k Child k arent arent Caching rules R-caching rule W-caching rule De-caching rules Write-back rule Invalidate rule age 6

7 Caching Rules: arent to Child 6.823, L id k k id R-caching rule <id, Cell(a,v,(cs,R(dir))) >, <id k, k > if id k dir <id, Cell(a,v,(cs,R(id k dir))) >, <id k, Cell(a,v,(Sh,R(ε))) k > W-caching rule <id, Cell(a,v,(Ex,R(ε))) >, <id k, k > <id, Cell(a,v,(Ex,W(id k ))) >, <id k, Cell(a,v,(Ex,R(ε))) k > De-caching Rules: Child to arent 6.823, L id k k id Writeback rule < id, Cell(a,-,(Ex,W(id k ))) >, <id k, Cell(a,v,(Ex,R(dir))) k > < id, Cell(a,v,(Ex,R(id k ))) >, <id k, Cell(a,v,(Sh,R(dir))) k > Invalidate rule < id, Cell(a,v,(cs,R(id k dir))) >, <id k, Cell(a,v,(Sh,R(ε))) k > < id, Cell(a,v,(cs,R(dir))) >, <id k, k > age 7

8 Local Rules 6.823, L id k k id Soe rules require observing and changing the state of two caches siultaneously (atoically), e.g., Writeback rule < id, Cell(a,-,(Ex,W(id k ))) >, <id k, Cell(a,v,(Ex,R(dir))) k > < id, Cell(a,v,(Ex,R(id k ))) >, <id k, Cell(a,v,(Sh,R(dir))) k > Usually not possible, especially if the caches are separated by a network DSM and Messages 6.823, L id k out interconnect id j in id rovide each M with in and out queue: < id,, in, out > FIFO essages passing between each (src,dest) pair Introduce request and reply essages: Msg(id src,id dest,cd,h/l,a,v) Low priority (L) sg cannot block high priority (H) sg age 8

9 Making the Rules Local 6.823, L Each rule is replaced by two rules: one for the sender and one for the receiver For exaple, the Writeback rule splits into Child s action < id k, Cell(a,v,(Ex,R(dir))) k, in k, out k > <id k, Cell(a,v,(Sh,R(dir))) k, in k, out k ;sg(id k,id,wbrep,h,a,v) > where id = parent(id k ) arent s action < id, Cell(a,-,(Ex,W(id k ))), sg(id k,id,wbrep,h,a,v);in, out > < id, Cell(a,v,(Ex,R(id k ))), in, out > Cache State Transitions 6.823, L Inv InvRep FlushRep ExRep ShRep Sh UpgradeRep Ex optiizations WbRep What causes a state transition? or When should a rule be applied? age 9

10 When to Apply a Rule 6.823, L id k id L2 < a, v, (Sh,R(dir)) > Consider the following rules: Load rule <id, Cell(a,v, (cs,r(ε))), in, out>, <t,load(a)>;pb, pb <id, Cell(a,v, (cs,r(ε))), in, out>, pb, pb <t,v> R-caching rule for Sender (arent) <id, Cell(a,v,(cs,R(dir))), in, out > if id k dir <id, Cell(a,v,(cs,R(id k dir))), in, out;sg(id,id k,shrep,h,a,v) > Suppose a is not in but is present in L2 in the Sh state. What should be done when executes Load(a)? Issuing Requests 6.823, L id k id L2 < a, v, (Sh,R(dir)) > If a is not in, send a request (ShReq) to L2, and set the cache state to be transient (Cacheending) <id,, in, out>, <t,load(a)>;pb, pb if a c <id, Cell(a,-,Cacheending), in, out;sg(id,parent(id),shreq,l,a)>, <t,load(a)>;pb, pb The load instruction reains suspended age 10

11 6.823, L rotocol X2: A rotocol for a syste with two eory levels ( + M) Xiaowei Shen Siplified states: Cache state: Meory state: Sh and Ex R[dir] and W[id] Load Rules 6.823, L Load-hit rule <id, Cell(a,v,Sh) c, in, out>, <t,load(a)>;pb, pb <id, Cell(a,v,Sh) c, in, out>, pb, pb <t,v> <id, Cell(a,v,Ex) c, in, out>, <t,load(a)>;pb, pb <id, Cell(a,v,Ex) c, in, out>, pb, pb <t,v> Load-iss rule <id, c, in, out>, <t,load(a)>;pb, pb if a c <id, Cell(a,-,Cacheending) c, in, out;sg(id,hoe,shreq,l,a)>, <t,load(a)>;pb, pb age 11

12 Store-hit rule Store Rules 6.823, L <id, Cell(a,-,Ex) c, in, out>, <t,store(a,v)>;pb, pb <id, Cell(a,v,Ex) c, in, out>, pb, pb <t,ack> Store-iss rule <id, c, in, out>, <t,store(a,v)>;pb, pb if a c <id, Cell(a,-,Cacheending) c, in, out;sg(id,hoe,exreq,l,a)>, <t,store(a,v)>;pb, pb <id, Cell(a,-,Sh) c, in, out>, <t,store(a,v)>;pb, pb <id, Cell(a,-,Cacheending) c, in, out;sg(id,hoe,exreq,l,a)>, <t,store(a,v)>;pb, pb 6.823, L rocessing ShReq Messages (at Hoe) Uncached or Outstanding Shared Copies <Cell(a,v,R(dir)), sg(id,hoe,shreq,l,a);in, out> if id dir <Cell(a,v,R(id dir)), in, out;sg(hoe,id,shrep,h,a,v)> Outstanding Exclusive Copy <Cell(a,v,W(id )), sg(id,hoe,shreq,l,a);in, out> if id id <Cell(a,v,T W (id )), sg(id,hoe,shreq,l,a);in, out;sg(hoe,id,wbreq,h,a)> age 12

13 6.823, L rocessing ExReq Messages (at Hoe) Uncached <Cell(a,v,R(ε)), sg(id,hoe,exreq,l,a);in, out> <Cell(a,v,W(id)), in, out;sg(hoe,id,exrep,h,a,v)> Outstanding Shared Copies <Cell(a,v,R(dir)), sg(id,hoe,exreq,l,a);in, out> if dir ε <Cell(a,v,T R (dir)), sg(id,hoe,exreq,l,a);in, out;ulticast(hoe,dir-{id},invreq,h,a)> Outstanding Exclusive Copy <Cell(a,v,W(id )), sg(id,hoe,exreq,l,a);in, out> if id id <Cell(a,v,T W (id )), sg(id,hoe,exreq,l,a);in, out;sg(hoe,id,flushreq,h,a)> 6.823, L rocessing Reply Messages (at cache) ShRep <id, Cell(a,-,Cacheending) c, sg(hoe,id,shrep,h,a,v);in, out> <id, Cell(a,v,Sh) c, in, out> ExRep <id, Cell(a,-,Cacheending) c, sg(hoe,id,exrep,h,a,v);in, out> <id, Cell(a,v,Ex) c, in, out> age 13

14 6.823, L rocessing Request Messages (at cache) WbReq <id, Cell(a,v,Ex) c, sg(hoe,id,wbreq,h,a);in, out> <id, Cell(a,v,Sh) c, in, out;sg(id,hoe,wbrep,h,a,v)> FlushReq <id, Cell(a,v,Ex) c, sg(hoe,id,flushreq,h,a);in, out> <id, c, in, out;sg(id,hoe,flushrep,h,a,v)> InvReq <id, Cell(a,v,Sh) c, sg(hoe,id,invreq,h,a);in, out> <id, c, in, out;sg(id,hoe,invrep,h,a)> <id, Cell(a,v,Cacheending) c, sg(hoe,id,invreq,h,a);in, out> <id, Cell(a,v,Cacheending) c, in, out;sg(id,hoe,invrep,h,a)> 6.823, L rocessing Reply Messages (at hoe) WbRep <Cell(a,-,T W (id)), sg(id,hoe,wbrep,h,a,v);in, out> <Cell(a,v,R(id)), in, out> FlushRep <Cell(a,-,T W (id)), sg(id,hoe,flushrep,h,a,v);in, out> <Cell(a,v,R(ε)), in, out> InvRep <Cell(a,v,T R (id dir)), sg(id,hoe,invrep,h,a);in, out> if dir ε <Cell(a,v,T R (dir)), in, out> <Cell(a,v,T R (id)), sg(id,hoe,invrep,h,a);in, out> <Cell(a,v,R(ε)), in, out> age 14