Spin Locks and Contention. Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Transcription

1 Spin Locks and Contention Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

2 MIMD Architectures memory Shared Bus Memory Contention Communication Contention Communication Latency Distributed 2

3 Today: Revisit Mutual Exclusion Think of performance, not just correctness and progress Begin to understand how performance depends on our software properly utilizing the multiprocessor machine s hardware And get to know a collection of locking algorithms 3 (1)

4 What Should you do if you can t get a lock? Keep trying spin or busy-wait Good if delays are short Give up the processor Good if delays are long Always good on uniprocessor 4 (1)

5 What Should you do if you can t get a lock? Keep trying spin or busy-wait Good if delays are short Give up the processor Good if delays are long Always good on uniprocessor our focus 5

6 Basic Spin-Lock CS. spin lock critical section Resets lock upon exit 6

7 Basic Spin-Lock lock introduces sequential bottleneck CS. spin lock critical section Resets lock upon exit 7

8 Basic Spin-Lock lock suffers from contention CS. spin lock critical section Resets lock upon exit 8

9 Basic Spin-Lock lock suffers from contention CS. spin lock critical section Resets lock upon exit Notice: these are distinct phenomena 9

10 Basic Spin-Lock lock suffers from contention. spin lock CS critical section Resets lock upon exit Seq Bottleneck no parallelism 10

11 Basic Spin-Lock lock suffers from contention CS. spin lock critical section Resets lock upon exit Contention??? 11

12 Contenção Refere-se à disputa pelo acesso exclusivo a um recurso compartilhado. Alta/baixa Granularidade do recurso 12

13 Review: Test-and-Set Boolean value Test-and-set (TAS) Swap true with current value Return value tells if prior value was true or false Can reset just by writing false TAS aka getandset 13

14 Review: Test-and-Set public class AtomicBoolean { boolean value; public synchronized boolean getandset(boolean newvalue) { boolean prior = value; value = newvalue; return prior; } } 14(5)

15 Review: Test-and-Set public class AtomicBoolean { boolean value; public synchronized boolean getandset(boolean newvalue) { boolean prior = value; value = newvalue; return prior; } } Package java.util.concurrent.atomic 15

16 Review: Test-and-Set public class AtomicBoolean { boolean value; public synchronized boolean getandset(boolean newvalue) { boolean prior = value; value = newvalue; return prior; } } Swap old and new values 16

17 Review: Test-and-Set AtomicBoolean lock = new AtomicBoolean(false) boolean prior = lock.getandset(true) 17

18 Review: Test-and-Set AtomicBoolean lock = new AtomicBoolean(false) boolean prior = lock.getandset(true) Swapping in true is called test-and-set or TAS 18(5)

19 Test-and-Set Locks Locking Lock is free: value is false Lock is taken: value is true Acquire lock by calling TAS If result is false, you win If result is true, you lose Release lock by writing false 19

20 Test-and-set Lock class TASlock { AtomicBoolean state = new AtomicBoolean(false); void lock() { while (state.getandset(true)) {} } void unlock() { state.set(false); }} 20

21 Test-and-set Lock class TASlock { AtomicBoolean state = new AtomicBoolean(false); void lock() { while (state.getandset(true)) {} } void unlock() { state.set(false); }} Lock state is AtomicBoolean 21

22 Test-and-set Lock class TASlock { AtomicBoolean state = new AtomicBoolean(false); void lock() { while (state.getandset(true)) {} } void unlock() { state.set(false); }} Keep trying until lock acquired 22

23 Test-and-set Lock class TASlock { AtomicBoolean state = new AtomicBoolean(false); Release lock by resetting state to false void lock() { while (state.getandset(true)) {} } void unlock() { state.set(false); }} 23

24 Space Complexity TAS spin-lock has small footprint N thread spin-lock uses O(1) space As opposed to O(n) Peterson/Bakery How did we overcome the Ω(n) lower bound? We used a RMW operation 24

25 Performance Experiment n threads Increment shared counter 1 million times How long should it take? How long does it take? 25

26 time Graph no speedup because of sequential bottleneck ideal threads 26

27 Mystery #1 TAS lock time threads Ideal What is going on? 27 (1)

28 Test-and-Test-and-Set Locks Lurking stage Wait until lock looks free Spin while read returns true (lock taken) Pouncing state As soon as lock looks available Read returns false (lock free) Call TAS to acquire lock If TAS loses, back to lurking 28

29 Test-and-test-and-set Lock class TTASlock { AtomicBoolean state = new AtomicBoolean(false); void lock() { while (true) { while (state.get()) {} if (!state.getandset(true)) return; } } 29

30 Test-and-test-and-set Lock class TTASlock { AtomicBoolean state = new AtomicBoolean(false); void lock() { while (true) { while (state.get()) {} if (!state.getandset(true)) return; } } Wait until lock looks free 30

31 Test-and-test-and-set Lock class TTASlock { AtomicBoolean state = new AtomicBoolean(false); void lock() { while (true) { while (state.get()) {} if (!state.getandset(true)) return; } } Then try to acquire it 31

32 Mystery #2 TAS lock time TTAS lock Ideal threads 32

33 Mystery Both TAS and TTAS Do the same thing (in our model) Except that TTAS performs much better than TAS Neither approaches ideal 33

34 Opinion Our memory abstraction is broken TAS & TTAS methods Are provably the same (in our model) Except they aren t (in field tests) Need a more detailed model 34

35 Bus-Based Architectures cache cache Bus cache memory 35

36 Bus-Based Architectures Random access memory (10s of cycles) cache cache Bus cache memory 36

37 Bus-Based Architectures Shared Bus broadcast medium One broadcaster at a time Processors and memory all snoop cache cache Bus cache memory 37

38 Per-Processor Caches Small Fast: 1 or 2 cycles Address & state information Bus-Based Architectures cache cache Bus cache memory 38

39 Jargon Watch Cache hit I found what I wanted in my cache Good Thing 39

40 Jargon Watch Cache hit I found what I wanted in my cache Good Thing Cache miss I had to shlep all the way to memory for that data Bad Thing 40

41 Cave Canem This model is still a simplification But not in any essential way Illustrates basic principles Will discuss complexities later 41

42 Processor Issues Load Request cache cache Bus cache memory data 42

43 Processor Issues Load Request Gimme data cache cache Bus cache memory data 43

44 Memory Responds cache cache cache Got your data right here Bus memory data Bus 44

45 Processor Issues Load Request Gimme data data cache Bus cache memory data 45

46 Processor Issues Load Request Gimme data data cache Bus cache memory data 46

47 Processor Issues Load Request I got data data cache Bus cache memory data 47

48 Other Processor Responds I got data data cache cache Bus Bus memory data 48

49 Other Processor Responds data cache cache Bus Bus memory data 49

50 Modify Cached Data data data Bus cache memory data 50 (1)

51 Modify Cached Data data data Bus cache memory data 51 (1)

52 Modify Cached Data data data Bus cache memory data 52

53 Modify Cached Data data data Bus cache What s up with the other copies? memory data 53

54 Cache Coherence We have lots of copies of data Original copy in memory Cached copies at processors Some processor modifies its own copy What do we do with the others? How to avoid confusion? 54

55 Write-Back Caches Accumulate changes in cache Write back when needed Need the cache for something else Another processor wants it On first modification Invalidate other entries Requires non-trivial protocol 55

56 Write-Back Caches Cache entry has three states Invalid: contains raw seething bits Valid: I can read but I can t write Dirty: Data has been modified Intercept other load requests Write back to memory before using cache 56

57 Invalidate data data Bus cache memory data 57

58 Invalidate Mine, all mine! data data Bus cache memory data 58

59 Invalidate Uh,oh cache data data Bus cache memory data 59

60 Invalidate Other caches lose read permission cache data Bus cache memory data 60

61 Invalidate Other caches lose read permission cache data Bus cache This cache acquires write permission memory data 61

62 Invalidate Memory provides data only if not present in any cache, so no need to change it now (expensive) cache data Bus cache memory data 62(2)

63 Another Processor Asks for Data cache data Bus cache memory data 63(2)

64 Owner Responds Here it is! cache data Bus cache memory data 64(2)

65 End of the Day data data Bus cache memory data Reading OK, no writing 65 (1)

66 Mutual Exclusion What do we want to optimize? Bus bandwidth used by spinning threads Release/Acquire latency Acquire latency for idle lock 66

67 Simple TASLock TAS invalidates cache lines Spinners Miss in cache Go to bus Thread wants to release lock delayed behind spinners 67

68 Test-and-test-and-set Wait until lock looks free Spin on local cache No bus use while lock busy Problem: when lock is released Invalidation storm 68

69 Local Spinning while Lock is Busy busy busy Bus busy memory busy 69

70 On Release invalid invalid Bus free memory free 70

71 Everyone misses, rereads On Release miss invalid miss invalid Bus free memory free 71 (1)

72 Everyone tries TAS On Release TAS( ) invalid TAS( ) invalid Bus free memory free 72 (1)

73 Problems Everyone misses Reads satisfied sequentially Everyone does TAS Invalidates others caches Eventually quiesces after lock acquired How long does this take? 73

74 Measuring Quiescence Time X = time of ops that don t use the bus Y = time of ops that cause intensive bus traffic P 1 P 2 P n In critical section, run ops X then ops Y. As long as Quiescence time is less than X, no drop in performance. By gradually varying X, can determine the exact time to quiesce. 74

75 Quiescence Time time Increses linearly with the number of processors for bus architecture threads 75

76 Mystery Explained TAS lock time threads TTAS lock Ideal Better than TAS but still not as good as ideal 76

77 Solution: Introduce Delay If the lock looks free But I fail to get it There must be lots of contention Better to back off than to collide again time r 2 d r 1 d d spin lock 77

78 Dynamic Example: Exponential Backoff time 4d 2d d spin lock If I fail to get lock wait random duration before retry Each subsequent failure doubles expected wait 78

79 Exponential Backoff Lock public class Backoff implements lock { public void lock() { int delay = MIN_DELAY; while (true) { while (state.get()) {} if (!lock.getandset(true)) return; sleep(random() % delay); if (delay < MAX_DELAY) delay = 2 * delay; }}} 79

80 Exponential Backoff Lock public class Backoff implements lock { public void lock() { int delay = MIN_DELAY; while (true) { while (state.get()) {} if (!lock.getandset(true)) return; sleep(random() % delay); if (delay < MAX_DELAY) delay = 2 * delay; }}} Fix minimum delay 80

81 Exponential Backoff Lock public class Backoff implements lock { public void lock() { int delay = MIN_DELAY; while (true) { while (state.get()) {} if (!lock.getandset(true)) return; sleep(random() % delay); if (delay < MAX_DELAY) delay = 2 * delay; }}} Wait until lock looks free 81

82 Exponential Backoff Lock public class Backoff implements lock { public void lock() { int delay = MIN_DELAY; while (true) { while (state.get()) {} if (!lock.getandset(true)) return; sleep(random() % delay); if (delay < MAX_DELAY) delay = 2 * delay; }}} If we win, return 82

83 Exponential Backoff Lock public class Backoff implements lock { public void lock() { int delay = MIN_DELAY; while (true) { while (state.get()) {} if (!lock.getandset(true)) return; sleep(random() % delay); if (delay < MAX_DELAY) delay = 2 * delay; }}} Back off for random duration 83

84 Exponential Backoff Lock public class Backoff implements lock { public void lock() { int delay = MIN_DELAY; while (true) { while (state.get()) {} if (!lock.getandset(true)) return; sleep(random() % delay); if (delay < MAX_DELAY) delay = 2 * delay; }}} Double max delay, within reason 84

85 Spin-Waiting Overhead TTAS Lock time Backoff lock threads 85

86 Backoff: Other Issues Good Easy to implement Beats TTAS lock Bad Must choose parameters carefully Not portable across platforms 86

87 Idea Avoid useless invalidations By keeping a queue of threads Each thread Notifies next in line Without bothering the others 87

88 Anderson Queue Lock next idle flags T F F F F F F F 88

89 Anderson Queue Lock next acquiring getandincrement flags T F F F F F F F 89

90 Anderson Queue Lock next acquiring getandincrement flags T F F F F F F F 90

91 Anderson Queue Lock next acquired Mine! flags T F F F F F F F 91

92 Anderson Queue Lock next acquired acquiring flags T F F F F F F F 92

93 Anderson Queue Lock next acquired acquiring flags getandincrement T F F F F F F F 93

94 Anderson Queue Lock next acquired acquiring flags getandincrement T F F F F F F F 94

95 Anderson Queue Lock next acquired acquiring flags T F F F F F F F 95

96 Anderson Queue Lock next released acquired flags T T F F F F F F 96

97 Anderson Queue Lock next released acquired flags Yow! T T F F F F F F 97

98 Anderson Queue Lock class ALock implements Lock { boolean[] flags={true,false,,false}; AtomicInteger next = new AtomicInteger(0); int[] slot = new int[n]; 98

99 Anderson Queue Lock class ALock implements Lock { boolean[] flags={true,false,,false}; AtomicInteger next = new AtomicInteger(0); int[] slot = new int[n]; One flag per thread 99

100 Anderson Queue Lock class ALock implements Lock { boolean[] flags={true,false,,false}; AtomicInteger next = new AtomicInteger(0); Next flag to use 100

101 Anderson Queue Lock class ALock implements Lock { boolean[] flags={true,false,,false}; AtomicInteger next = new AtomicInteger(0); ThreadLocal<Integer> myslot; Thread-local variable 101

102 Thread-local variable (Java) Cada thread tem a sua própria copia de cada variável, inicializada independentemente. Métodos get() e set() 102

103 Anderson Queue Lock public lock() { int slot = next.getandincrement(); myslot.set(slot); while (!flags[slot % n]) {}; //(fila circular) flags[slot % n] = false; //Aqui? } public unlock() { flags[(myslot.get()+1) % n] = true; } 103

104 Anderson Queue Lock public lock() { int myslot = next.getandincrement(); while (!flags[myslot % n]) {}; flags[myslot % n] = false; } public unlock() { flags[(myslot+1) % n] = true; } Take next slot 104

105 Anderson Queue Lock public lock() { int myslot = next.getandincrement(); while (!flags[myslot % n]) {}; flags[myslot % n] = false; } public unlock() { flags[(myslot+1) % n] = true; } Spin until told to go 105

106 Anderson Queue Lock public lock() { int slot[i]=next.getandincrement(); while (!flags[slot[i] % n]) {}; flags[slot[i] % n] = false; } public unlock() { flags[slot[i]+1 % n] = true; } Prepare slot for re-use 106

107 Anderson Queue Lock public lock() { Tell next thread to go int myslot = next.getandincrement(); while (!flags[myslot % n]) {}; flags[myslot % n] = false; } public unlock() { flags[(myslot+1) % n] = true; } 107

108 Performance TTAS queue Shorter handover than backoff Curve is practically flat Scalable performance FIFO fairness false sharing? 108

109 Anderson Queue Lock Good First truly scalable lock Simple, easy to implement Bad Space hog One bit per thread Unknown number of threads? Small number of actual contenders? 109

110 CLH Lock FIFO order Small, constant-size overhead per thread 110

111 Initially idle tail false 111

112 Initially idle tail false Queue tail 112

113 Initially idle tail false Lock is free 113

114 Initially idle tail false 114

115 Purple Wants the Lock acquiring tail false 115

116 Purple Wants the Lock acquiring tail false true 116

117 Purple Wants the Lock acquiring Swap tail false true 117

118 Purple Has the Lock acquired tail false true 118

119 Red Wants the Lock acquired acquiring tail false true true 119

120 Red Wants the Lock acquired acquiring tail false Swap true true 120

121 Red Wants the Lock acquired acquiring tail false true true 121

122 Red Wants the Lock acquired acquiring tail false true true 122

123 Red Wants the Lock acquired acquiring tail false true true Implicitely Linked list 123

124 Red Wants the Lock acquired acquiring tail false true true 124

125 Red Wants the Lock acquired acquiring tail false true true true Actually, it spins on cached copy 125

126 Purple Releases release acquiring false Bingo! tail false false true 126

127 Purple Releases released acquired tail true 127

128 Space Usage Let L = number of locks N = number of threads ALock O(LN) CLH lock O(L+N) 128

129 CLH Queue Lock class Qnode { AtomicBoolean locked = new AtomicBoolean(true); } 129

130 CLH Queue Lock class Qnode { AtomicBoolean locked = new AtomicBoolean(true); } Not released yet 130

131 CLH Queue Lock class CLHLock implements Lock { AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode = new Qnode(); public void lock() { Qnode pred = tail.getandset(mynode); while (pred.locked) {} }} 131(3)

132 CLH Queue Lock class CLHLock implements Lock { AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode = new Qnode(); public void lock() { Qnode pred = tail.getandset(mynode); while (pred.locked) {} }} Tail of the queue 132(3)

133 CLH Queue Lock class CLHLock implements Lock { AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode = new Qnode(); public void lock() { Qnode pred = tail.getandset(mynode); while (pred.locked) {} }} Thread-local Qnode 133(3)

134 CLH Queue Lock class CLHLock implements Lock { AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode = new Qnode(); public void lock() { Qnode pred = queue.getandset(mynode); while (pred.locked) {} }} Swap in my node 134(3)

135 CLH Queue Lock class CLHLock implements Lock { AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode = new Qnode(); public void lock() { Qnode pred = queue.getandset(mynode); while (pred.locked) {} }} Spin until predecessor releases lock 135(3)

136 CLH Queue Lock Class CLHLock implements Lock { public void unlock() { mynode.locked.set(false); mynode = pred; } } 136(3)

137 CLH Queue Lock Class CLHLock implements Lock { public void unlock() { mynode.locked.set(false); mynode = pred; } } Notify successor 137(3)

138 CLH Queue Lock Class CLHLock implements Lock { } public void unlock() { } mynode.locked.set(false); mynode = pred; Recycle predecessor s node 138(3)

139 CLH Lock Good Lock release affects predecessor only Small, constant-sized space Bad Doesn t work for uncached NUMA architectures 139

140 NUMA Architecturs Acronym: Non-Uniform Memory Architecture Illusion: Flat shared memory Truth: No caches (sometimes) Some memory regions faster than others 140

141 NUMA Machines Spinning on local memory is fast 141

142 NUMA Machines Spinning on remote memory is slow 142

143 CLH Lock Each thread spin s on predecessor s memory Could be far away 143

144 Abortable Locks What if you want to give up waiting for a lock? For example Timeout Database transaction aborted by user 144

145 Back-off Lock Aborting is trivial Just return from lock() call Extra benefit: No cleaning up Wait-free Immediate return 145

146 Queue Locks Can t just quit Thread in line behind will starve Need a graceful way out 146

147 Queue Locks spinning spinning spinning true true true 147

148 Queue Locks locked spinning spinning false true true 148

149 Queue Locks locked spinning false true 149

150 Queue Locks locked false 150

151 Queue Locks spinning spinning spinning true true true 151

152 Queue Locks spinning spinning true true true 152

153 Queue Locks locked spinning false true true 153

154 Queue Locks spinning false true 154

155 Queue Locks pwned false true 155

156 Abortable CLH Lock When a thread gives up Removing node in a wait-free way is hard Idea: let successor deal with it. 156

157 Initially idle Pointer to predecessor (or null) tail A 157

158 Initially idle Distinguished available node means lock is free tail A 158

159 acquiring Acquiring tail A 159

160 acquiring Acquiring Null predecessor means lock not adquired or was released A 160

161 acquiring Acquiring Swap A 161

162 acquiring Acquiring A 162

163 Acquired locked Pointer to AVAILABLE means lock is free. A 163

164 Normal Case locked spinning spinning Null means lock is not free & request not aborted 164

165 One Thread Aborts locked Timed out spinning 165

166 Successor Notices locked Timed out spinning Non-Null means predecessor aborted 166

167 Recycle Predecessor s Node locked spinning 167

168 Spin on Earlier Node locked spinning 168

169 Spin on Earlier Node released spinning A The lock is now mine 169

170 Time-out Lock public class TOLock implements Lock { static Qnode AVAILABLE = new Qnode(); AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode; 170

171 Time-out Lock public class TOLock implements Lock { static Qnode AVAILABLE = new Qnode(); AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode; Distinguished node to signify free lock 171

172 Time-out Lock public class TOLock implements Lock { static Qnode AVAILABLE = new Qnode(); AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode; Tail of the queue 172

173 Time-out Lock public class TOLock implements Lock { static Qnode AVAILABLE = new Qnode(); AtomicReference<Qnode> tail; ThreadLocal<Qnode> mynode; Remember my node 173

174 Time-out Lock public boolean lock(long timeout) { Qnode qnode = new Qnode(); mynode.set(qnode); qnode.prev = null; Qnode mypred = tail.getandset(qnode); if (mypred== null mypred.prev == AVAILABLE) { return true; } 174

175 Time-out Lock public boolean lock(long timeout) { Qnode qnode = new Qnode(); mynode.set(qnode); qnode.prev = null; Qnode mypred = tail.getandset(qnode); if (mypred == null mypred.prev == AVAILABLE) { return true; } Create & initialize node 175

176 Time-out Lock public boolean lock(long timeout) { Qnode qnode = new Qnode(); mynode.set(qnode); qnode.prev = null; Qnode mypred = tail.getandset(qnode); if (mypred == null mypred.prev == AVAILABLE) { return true; } Swap with tail 176

177 Time-out Lock public boolean lock(long timeout) { Qnode qnode = new Qnode(); mynode.set(qnode); qnode.prev = null; Qnode mypred = tail.getandset(qnode); if (mypred == null }... mypred.prev == AVAILABLE) { return true; If predecessor absent or released, we are done 177

178 Time-out Lock long start = now(); while (now()- start < timeout) { Qnode predpred = mypred.prev; if (predpred == AVAILABLE) { return true; } else if (predpred!= null) { mypred = predpred; } } 178

179 Time-out Lock long start = now(); while (now()- start < timeout) { Qnode predpred = mypred.prev; if (predpred == AVAILABLE) { return true; } else if (predpred!= null) { mypred = predpred; } } Keep trying for a while 179

180 Time-out Lock long start = now(); while (now()- start < timeout) { } Qnode predpred = mypred.prev; if (predpred == AVAILABLE) { return true; } else if (predpred!= null) { } mypred = predpred; Spin on predecessor s prev field 180

181 Time-out Lock long start = now(); while (now()- start < timeout) { Qnode predpred = mypred.prev; if (predpred == AVAILABLE) { return true; } else if (predpred!= null) { mypred = predpred; } } Predecessor released lock 181

182 Time-out Lock long start = now(); while (now()- start < timeout) { } Qnode predpred = mypred.prev; if (predpred == AVAILABLE) { return true; } else if (predpred!= null) { } mypred = predpred; Predecessor aborted, advance one 182

183 Time-out Lock if (!tail.compareandset(qnode, mypred)) qnode.prev = mypred; return false; } } 183

184 Time-out Locks if (!tail.compareandset(qnode, mypred)) qnode.prev = mypred; return false; } } Do I have a successor? If CAS fails: I do have a successor, tell it about mypred 184

185 Time-out Locks if (!tail.compareandset(qnode, mypred)) qnode.prev = mypred; return false; } } If CAS succeeds: no successor, simply return false 185

186 Time-Out Unlock public void unlock() { Qnode qnode = mynode.get(); if (!tail.compareandset(qnode, null)) qnode.prev = AVAILABLE; } 186

187 Time-out Unlock public void unlock() { Qnode qnode = mynode.get(); if (!tail.compareandset(qnode, null)) qnode.prev = AVAILABLE; } If CAS failed: exists successor, notify successor it can enter 187

188 Timing-out Lock public void unlock() { Qnode qnode = mynode.get(); if (!tail.compareandset(qnode, null)) qnode.prev = AVAILABLE; } CAS successful: set tail to null, no clean up since no successor waiting 188

189 One Lock To Rule Them All? TTAS+Backoff, CLH, MCS, ToLock Each better than others in some way There is no one solution Lock we pick really depends on: the application the hardware which properties are important 189

190 This work is licensed under a Creative Commons Attribution- ShareAlike 2.5 License. You are free: to Share to copy, distribute and transmit the work to Remix to adapt the work Under the following conditions: Attribution. You must attribute the work to The Art of Multiprocessor Programming (but not in any way that suggests that the authors endorse you or your use of the work). Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights. 190