10/28/11. DD2451 Overview. DD2451 Overview. FDD 3008 Overview. Course Material

Transcription

1 /8/ DD45 Parallel and Distributed omputing --- FDD38 Distributed lgorithms Lecture Mads Dam utumn/winter Lectures double lectures Twice a week until beg. December Focus on theory and principles ut look at some systems too lso some programming issues DD45 Overview Exercises Exercises given for each lecture Discussion at end of each lecture 3 homework sets to be handed in individually Homework rules and grading see the course page We may use peer review Deadlines apply DD45 Overview FDD 38 Overview Paper presentations hance to present research papers See presentation sessions in schedule on course web Optional, but counts towards final grade Select paper you want to present Some advice on course web Let Mads know by mail at the latest Nov! Other matters: ourse committee/kursnämnd ny volunteers? KTH social any interest? nything else? Introductory postgraduate level Some additional requirements Paper presentation obligatory Final report 8- pages Preferably: pplication to PhD project Topic selection quite free Generally: Herlihy s book at bit too informal for pg level More precision and formality expected in handins, reports, presentations nyway the material gets more advanced as we move along ourse Material Main textbook: Herlihy and Shavit Focus on shared memory model Used a lot in first lectures Later more attention to message-passing and distribution Other textbooks used later on Other material (papers, course notes) to be announced on course page Slides: Much material due to Maurice Herlihy, rown, and Roger Wattenhofer, ETH, Zurich, thanks! What Is Parallel and Distributed omputing bout? How to distribute a computing task between several processors/ threads/nodes/cores? f (x, y)? 5! When? How? Performance? Models? Failures? Security? Scale? What is possible? What is not possible? orrectness? Fault tolerance?

2 /8/ What Is Parallel and Distributed omputing bout? What Is Parallel and Distributed omputing bout? How to distribute a computing task between several processors/ threads/nodes/cores? How to distribute a computing task between several processors/ threads/nodes/cores? When? How? Performance? Models? Security? Scale? What is possible? What is not possible? orrectness? Fault tolerance? When? How? Performance? Models? Security? Scale? What is possible? What is not possible? orrectness? Fault tolerance? What Is Parallel and Distributed omputing bout? What Is Parallel and Distributed omputing bout? How to distribute a computing task between several processors/ threads/nodes/cores? How to distribute a computing task between several processors/ threads/nodes/cores? When? How? Performance? Models? Security? Scale? What is possible? What is not possible? orrectness? Fault tolerance? When? How? Performance? Models? Security? Scale? What is possible? What is not possible? orrectness? Fault tolerance? Does Scale Matter? Why Is It Important? It s all about scale! Small scale: Small numbers of nodes, fixed interconnects, shared memory, little latency, fixed architectures, rare failures, mostly static Large scale: ig numbers, system organization important, heterogenous architectures, message passing, big latency, frequent failures, more dynamic Ultimately: Read: The Free Lunch Is Over, Herb Suer

3 /8/ High performance High reliability High availability Why Is It Important? oncurrency is indispensable ut parallelizalon and distribulon is tricky! ourse Overview First small scale Mutual exclusion Linearizability, sequenlal consistency, the Java memory model onsensus, impossibility results, consensus numbers, registers This part mainly uses Herlihy s book So let s get started ourse Overview, II Then large scale Fault tolerance yzanlne fault tolerance onsistency models, weak consistency PP systems, DHT s Mostly slides and handouts Then some addilonal topics among: Leader eleclon, spanning trees, vertex colouring, and/or Spin locks, concurrent data structures Events Events a,b: : read(x = true) the event of thread reading variable x and ge\ng the result true : write(y = ) do. of wrilng to y Note: y might have been already doesn t maer Events: re instantaneous, -me(a) = Lme at which event a occurs (Not the same as atomic, not the same as a program statement) synchrony: No two events occur at the same Lme a b a -me Events, II Thread, Process Examples: Reading a variable WriLng a variable Outpu\ng a value Inpu\ng a value Receiving a signal alling a method Returning from a method Returning normally from a method Returning exceplonally from a method (etc) depending on which abstraclons we assume thread / process is a state machine aka transilon system aka transilon graph aka (lots of other things) a a c Need to dislnguish between events and event occurrences Events are not necessarily observable c b b d 3

4 /8/ States Traces Model dependent Threads: Object states assignments of values to fields Thread states do. to local variables Program counters (Maybe more, caches, load state, network state, etc) Network state Wire state uffer states onfiguralon state IP numbers Lots and lots and lots thread produces a trace = sequence of events tr = a a a n a a a n Precedence, happens- before (Lamport): a - > b : Same as -me(a) < -me(b) Processor state: Registers Memory aches, pipelines onfiguralon The precedence relalon is a total order Irreflexive: not(a - > a) nlsymmetric: If a - > b then not(b - > a) TransiLve: If a - > b and b - > c then a - > c Total: For all a b, either a - > b or b - > a (a,b) Intervals = the interval between a and b = the set of Lme point from -me(a) to -me(b) (endpoints included) (a,b) - > (c,d) = (a,b) precedes (c,d) = b - > c = (a,b) ends before (c,d) starts (a,b), (c,d) concurrent, if not (a,b) - > (c,d) a b c d a c b d rilcal SecLons public class ounter { private long value; public long getndincrement() { temp = value; value = temp + ; return temp; Say: ssign globally unique sequence numbers What if two threads call getndincrement in parallel? rilcal SecLons, II Locks public interface Lock { Tread : read(value=) write(temp=) read(temp=) write(value=) public void lock(); public void unlock(); Tread : read(value=) public class ounter { private long value; write(value=) Lock.lock(): Entering crilcal seclon Lock.unlock(): ExiLng public long getndincrement() { temp = value; value = temp + ; return temp; 4

5 /8/ Using Locks public class ounter { private long value; private Lock lock; public long getndincrement() { lock.lock(); try { int temp = value; value = value + ; finally { lock.unlock(); return temp; ProperLes Mutual exclusion: S i : The i:th interval during which thread is in its crilcal seclon For any,, i, j, S i - > S j or S j - > S i bsence of deadlock: If some thread calls lock() and never returns (i.e. it has acquired the lock) then other threads acquire and release the lock infinitely onen No starvalon: Each call to lock() eventually returns Threads, First empt class LockOne implements Lock { private boolean[] flag = new boolean[]; while (flag[otherthread]) { Threads, First empt Lemma: Mutual exclusion holds Proof: Suppose flag[thisthread] = false S S Notes: LockOne deadlocks flag array should be declared volalle, why? class LockOne implements Lock { private boolean[] flag = new boolean[]; while (flag[otherthread]) { flag[thisthread] = false Threads, First empt Threads, First empt S S write (flag[] = true) read (flag[] = false) S write (flag[] = true) read (flag[] = false) S write (flag[] = true) (Otherwise would have len c.s. already) class LockOne implements Lock { private boolean[] flag = new boolean[]; while (flag[otherthread]) { flag[thisthread] = false class LockOne implements Lock { private boolean[] flag = new boolean[]; while (flag[otherthread]) { flag[thisthread] = false 5

6 /8/ Threads, First empt Threads, First empt S write (flag[] = true) S read (flag[] = false) read (flag[] = false) (by the code) write (flag[] = true) S write (flag[] = true) S read (flag[] = false) read (flag[] = false) (but flag[] was never wrien to! QED) write (flag[] = true) class LockOne implements Lock { private boolean[] flag = new boolean[]; while (flag[otherthread]) { flag[thisthread] = false class LockOne implements Lock { private boolean[] flag = new boolean[]; while (flag[otherthread]) { flag[thisthread] = false Threads, Second empt LockTwo, Mutual Exclusion public class LockTwo implements Lock { private int victim; while (victim == thisthread) {; write (viclm = ) read (viclm = ) S (by the code) S Notes: victim = : defers to LockTwo also deadlocks public class LockTwo implements Lock { private int victim; while (victim == thisthread) {; LockTwo, Mutual Exclusion LockTwo, Mutual Exclusion S S write (viclm = ) read (viclm = ) write (viclm = ) (since viclm must be wrien) S write (viclm = ) S read (viclm = ) write (viclm = ) read (viclm = ) (must be placed somewhere ) public class LockTwo implements Lock { private int victim; while (victim == thisthread) {; public class LockTwo implements Lock { private int victim; while (victim == thisthread) {; 6

7 /8/ LockTwo, Mutual Exclusion S write (viclm = ) S read (viclm = ) write (viclm = ) read (viclm = ) (but impossible) Peterson s lgorithm while (flag[otherthread] && victim == thisthread) {; flag[thisthread] = false; public class LockTwo implements Lock { private int victim; while (victim == thisthread) {; Notes: Flag your interest and step back if other thread also wants a go Peterson s lgorithm, Mutual Exclusion Peterson s lgorithm, Mutual Exclusion S S write (flag[] = true) S write (flag[] = true) S write (viclm = ) read (viclm) write (viclm = ) read (viclm) read (flag[]) write b (flag[] = true) read (flag[]) Don t yet know which value write (viclm = ) (Else goes first) while (flag[otherthread] && victim == thisthread) {; flag[thisthread] = false; while (flag[otherthread] && victim == thisthread) {; flag[thisthread] = false; Peterson s lgorithm, Mutual Exclusion Peterson s lgorithm, Mutual Exclusion S S write (flag[] = true) S write (flag[] = true) S write (viclm = ) read (viclm=) (since was last to assign to viclm) write (viclm = ) read (viclm=) write b (flag[] = true) read (flag[]) write b (flag[] = true) read (flag[]=false) (since entered S) write (viclm = ) write (viclm = ) while (flag[otherthread] && victim == thisthread) {; flag[thisthread] = false; while (flag[otherthread] && victim == thisthread) {; flag[thisthread] = false; 7

8 /8/ Peterson s lgorithm, Mutual Exclusion Peterson s lgorithm, No StarvaLon S write (flag[] = true) write (viclm = ) read (viclm=) write b (flag[] = true) read (flag[]=false) (Impossible, qed) write (viclm = ) S Suppose thread spins from Lme t onwards Then viclm = and flag[] = true infinitely onen (i.o.) Since flag[] = true i.o.: Either enters and exits S i.o. ut then viclm = almost always ( almost always = always, from some Lme t onwards) Else spins in the lock() loop ut then viclm = almost always, a contradiclon while (flag[otherthread] && victim == thisthread) {; flag[thisthread] = false; while (flag[otherthread] && victim == thisthread) {; flag[thisthread] = false; Lamport s akery lgorithm Lamport s akery lgorithm Used to declare interest, as earlier class akery implements Lock { boolean[] flag; Smallest non- zero label Label[] label; gets to go public akery (int n) { flag = new boolean[n]; label = new Label[n]; for (int i = ; i < n; i++) { flag[i] = false; label[i] = ; I want the cs class akery implements Lock { flag[i] = true; label[i] = max(label[],,label[n-])+; while (exists k flag[k] && (label[i],i) > (label[k],k)) {; Pick a new Lcket, later than any already in use Doorway Wait unll the new Lcket is minimal Uses lexicographic order to break symmetry Lamport s akery lgorithm class akery implements Lock { flag[i] = false; The akery lgorithm, orrectness Lemma: The akery algorithm is deadlock free Proof: Eventually a wailng thread will hold the least (label, thread) pair. QED Lemma: The akery algorithm is first- come- first- served Proof: First through the doorway gets the smallest Lcket. QED Lemma: The akery algorithm salsfies mutual exclusion Proof: heck the textbook. 8

9 /8/ Timestamps akery algorithm: Timestamp: (label, thread) pair Timestamp domain unbounded for correctness Two operalons needed: scan: Sample the label array label: Produce a new label, strict upper bound of scanned labels Possible to do this using only bounded data? ounded Timestamps Possible to build bounded Lmestamp system which Is wait- free i.e. no locks For mullple threads (cf. Dolev- Shavit- SIM- 97) Here only sequenlal solulon Not wait- free So useless for the akery algorithm SLll of interest Two Thread ounded Precedence Graph Three Thread ounded Precedence Graph? : has precedence over : has precedence over ut is not transilve! Eh? Want as well? What if wants precedence? Note that precedence here is not transilve! Three Thread ounded Precedence Graph Three Thread ounded Precedence Graph Edge from all vertex in cycle to any vertex in cycle etc moves 9

10 /8/ Three Thread ounded Precedence Graph n Thread ounded Precedence Graph moves nd so on n thread graph size is 3 n- (!) So Is the akery lgorithm the Way to Go? The akery lgorithm is Succinct, Elegant, and Fair. Q: So why isn t it praclcal? : Well, you have to read n dislnct variables Shared Memory Shared read/write memory localons are called registers (for historical reasons) ome in different flavors MulL- Reader- Single- Writer (flag[]) MulL- Reader- MulL- Writer (victim[]) Less intereslng for now: SRMW and SRSW rt of Multiprocessor 57 rt of Multiprocessor 58 n MRSW Registers re Needed Theorem: t least n MRSW (mull- reader/single- writer) registers are needed to solve deadlock- free mutual exclusion. n registers like flag[] This is a lower bound result generally tough Proof Technique ssume that an algorithm for a lock object with < n MRSW registers exists The lock objects salsfies mutual exclusion, no deadlock, no starvalon Derive a contradiclon I.e., show a bad execulon that violates properles must exist The lock object must work correctly for all client programs So choose one with n threads which infinitely onen tries to enter and leave the crilcal seclon To be starvalon free whenever all threads in idle state - outside c.s. and outside the lock - each thread must be able to enter c.s. ON ITS OWN

11 /8/ Proof: n MRSW Registers re Needed Each thread must write to some register before entering c.s. Run and from idle state unll about to write to their registers can visit the cs Nothing has been wrien yet or can write and proceed to enter cs as well They have no way of telling whether or not is in cs Upper ound akery algorithm uses n MRSW registers So the bound is (prey) Lght ut what if we use MRMW registers? Like victim[]? write write S S S ad News Theorem t least n MRMW mull- reader/mull- writer registers are needed to solve deadlock- free mutual exclusion. Theorem (First - Threads) Theorem: Deadlock- free mutual exclusion for threads requires at least mull- reader mull- writer registers Proof: assume one register suffices and derive a contradiclon (So mullple writers don t help) rt of Multiprocessor 63 rt of Multiprocessor 64 Two Thread ExecuLon Threads run, reading and wrilng R Deadlock free so at least one gets in S write(r) R S overing State for One Register lways Exists In any protocol has to write to the register before entering S, so stop it just before write(r) over of register R: thread is about to write to R and R s value allows other threads to enter S overing state: ll shared registers are covered rt of Multiprocessor 65 rt of Multiprocessor 66

12 /8/ Proof: ssume over of Proof: ssume over of S write(r) runs, possibly writes to the register, enters S S write(r) S Runs, first obliteralng any trace of, then also enters the crilcal seclon rt of Multiprocessor 67 rt of Multiprocessor 68 Theorem Proof: ssume over of Deadlock- free mutual exclusion for 3 threads requires at least 3 mull- reader mull- writer registers has run from idle state unll about to write to R has run from idle state unll about to write to R is free to enter S Write(R ) Write(R ) We show later that such a state can be reached Only registers rt of Multiprocessor 69 7 Run Solo Obliterate Traces of Write(R ) Write(R ) Write(R ) Write(R ) S Writes to one or both registers, enters S S Other threads obliterate evidence that entered S rt of Multiprocessor 7 rt of Multiprocessor 7

13 /8/ Mutual Exclusion Fails overing State an e Reached S Write(R ) S Write(R ) S looks empty*, so another thread can get in Proved: a contradiclon starlng from a covering state for registers laim: a covering state for registers is reachable from any state where S is empty Problem: Registers may have mullple writers So threads may interfere So how can we be sure that two threads may each cover a register at the same Lme? (*) Looks empty: There is no informalon in any shared register that is not idle rt of Multiprocessor 74 overing State for Two overing State for Two Write(R ) If we run through S 3 Lmes, must return twice to cover some register, say R In each case: is outside S and has not wrien to any shared register since leaving idle state 75 Write(R ) Write(R ) Write(R ) Start with covering register R for the st Lme Run unll about to write to R This must be possible Else let write R only do that, let enter S, let write R This state looks the same to as one where did nothing 76 overing State for Two overing State for Two Write(R ) Write(R ) Write(R ) Write(R ) We were here: Start with covering register R for the st Lme Run unll about to write to uncovered R re we done? NO! could have written to R So S no longer looks empty This could prevent from entering S! rt of Multiprocessor 77 rt of Multiprocessor 78 3

14 /8/ overing State for Two InducLvely We an Show Write(R ) Write(R ) Run obliterating traces of in R Run again until it is about to write to R Remember wrote R then (maybe) R then R again fter each write enters S, exits S, enters idle state So after writing to R, leaves S so S looks empty to Now we are done Write(R ) There is a covering state Where k threads not in S cover k distinct registers Proof follows when k = n - rt of Multiprocessor Write(R ) 8 Write(R ) Summary of Lecture In the 96 s many incorrect solutions to starvation-free mutual exclusion using RW-registers were published Today we know how to solve FIFO n thread mutual exclusion using n RW-Registers This is optimal within factor ut n registers for n thread mutual exclusion is not a scalable solution Solution: Hardware supported atomic registers oming up: Which registers, what can they do? rt of Multiprocessor 8 4