Engineering Robust Server Software

Transcription

1 Engineering Robust Server Software Scalability

2 Lock Free Data Structures Atomics operations work great when they do what you need E.g., increment an int What about more complicated things? E.g., No hardware support for atomically adding to a BST Lock Free Data Structures Good when few write conflicts Generally based on atomic CAS Freeing memory makes things hard Much easier if we don't need to free things (GCed languages) 2

3 Lock Free LinkedList class LinkedList { class Node{ public: const int data; std::atomic<node *> next; Node(int d): data(d), next(nullptr) { Node(int d, Node * n): data(d), next(n) { ~Node() { ; std::atomic<node*> ; ; 3

4 Lock Free LinkedList data: 4 data: 8 data: 12 4

5 Lock Free LinkedList addfront(3) data: 4 data: 8 data: 12 data: 3 5

6 Lock Free LinkedList data: 4 data: 8 data: 12 void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 6

7 Lock Free LinkedList data: 4 data: 8 data: 12 next void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 7

8 Lock Free LinkedList data: 4 data: 8 data: 12 next temp data: 3 void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 8

9 data: 42 Lock Free LinkedList data: 4 data: 8 data: 12 next temp data: 3 Another thread just did a racing update! void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 9

10 data: 42 Lock Free LinkedList data: 4 data: 8 data: 12 next temp data: 3 void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); so CAS fails (!= temp) while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 10

11 data: 42 Lock Free LinkedList data: 4 data: 8 data: 12 next temp data: 3 void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 11

12 data: 42 Lock Free LinkedList data: 4 data: 8 data: 12 next temp data: 3 Suppose no other racing writes, so CAS succeeds void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 12

13 data: 42 Lock Free LinkedList data: 4 data: 8 data: 12 next temp data: 3 Suppose no other racing writes, so CAS succeeds void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 13

14 data: 42 Lock Free LinkedList data: 4 data: 8 data: 12 data: 3 void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 14

15 data: 42 Lock Free LinkedList data: 4 data: 8 data: 12 data: 3 void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); So what memory order do these need? 15

16 Lock Free LinkedList Thread 0 For the the CASes, what can you say about happens-before? Thread 1 temp = allocate memory store temp.data = x store temp.next = next CAS. CAS What memory ordering does that sound like? void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_???)) { temp->next.store(next, std::memory_order_???); 16

17 Acquire/Release Semantics What we want is acquire/release semantics Load: acquire Store: release store A store B store C store D (release) When load (acquire) receives value from store (release) load D (acquire) All prior stores in the releasing thread become visible-side effects In acquiring thread (only) Effectively: establishes happens-before for all these stores 17

18 Lock Free LinkedList void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_acq_rel)) temp->next.store(next, std::memory_order_???); 18

19 Lock Free LinkedList What about this store? What ordering do we need for it? void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_acq_rel)) temp->next.store(next, std::memory_order_???); 19

20 Lock Free LinkedList What about this store? What ordering do we need for it? - temp is still private to this thread - the next thing we are going to do is CAS w/ acq_rel [will ensure this update is visible to any thread reading the new value] So we can use relaxed void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_acq_rel)) temp->next.store(next, std::memory_order_relaxed); 20

21 Lock Free LinkedList What about this load? What ordering do we need for it? void addfront(int x) { Node * next =.load(std::memory_order_???); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_acq_rel)) temp->next.store(next, std::memory_order_relaxed); 21

22 Lock Free LinkedList What about this load? What ordering do we need for it? - Don't care about relationship to other values until after CAS - CAS will perform acquire [and fail if has changed] So we can use relaxed again void addfront(int x) { Node * next =.load(std::memory_order_relaxed); Node * temp = new Node(x, next); while (!.compare_exchange_weak(next, temp, std::memory_order_acq_rel)) temp->next.store(next, std::memory_order_relaxed); 22

23 Lock Free LinkedList: addsorted void addsorted(int x) { std::atomic<node *> * ptr = &; Node * newnode = new Node(x); Node * nextnode; do { while (true) { nextnode = ptr->load(std::memory_order_acquire); if (nextnode == nullptr nextnode->data > x) { break; ptr = &nextnode->next; newnode->next.store(nextnode, std::memory_order_relaxed); while (!ptr->compare_exchange_weak(nextnode, newnode, std::memory_order_acq_rel)); 23

24 Lock Free LinkedList: addsorted void addsorted(int x) { std::atomic<node *> * ptr = &; Node * newnode = new Node(x); Node * nextnode; Why acquire? do { while (true) { nextnode = ptr->load(std::memory_order_acquire); if (nextnode == nullptr nextnode->data > x) { break; ptr = &nextnode->next; newnode->next.store(nextnode, std::memory_order_relaxed); while (!ptr->compare_exchange_weak(nextnode, newnode, std::memory_order_acq_rel)); 24

25 Why Not Just SC? void addsorted(int x) { std::atomic<node *> * ptr = &; What if we make it all SC? Node * newnode = new Node(x); Node * nextnode; do { while (true) { nextnode = ptr->load(std::memory_order_seq_cst); if (nextnode == nullptr nextnode->data > x) { break; ptr = &nextnode->next; newnode->next.store(nextnode, std::memory_order_seq_cst); while (!ptr->compare_exchange_weak(nextnode, newnode, std::memory_order_seq_cst)); 25

26 Why Not Just Do SC? If we use SC, it will be right Too weak of a memory ordering -> bugs on some hardware Cost of SC? 26

27 Why Not Just Do SC? If we use SC, it will be right Too weak of a memory ordering -> bugs on some hardware Cost of SC? Slower How much slower? 27

28 Why Not Just Do SC? If we use SC, it will be right Too weak of a memory ordering -> bugs on some hardware Cost of SC? Slower How much slower? x86: not too much (already very strong memory consistency) Power8: 2x 4x (depending on data size) 28

29 Lock Free LinkedList: removefront bool removefront(int & outdata) { Node * temp =.load(std::memory_order_acquire); if (temp == nullptr) { return false; Node * next = temp->next.load(std::memory_order_relaxed); while (!.compare_exchange_weak(temp, next, std::memory_order_acq_rel)) if (temp == nullptr) { return false; next = temp->next.load(std::memory_order_relaxed); if (temp!= nullptr) { outdata = temp->data; return true; What is wrong with this code? return false; 29

30 Lock Free LinkedList: removefront bool removefront(int & outdata) { Node * temp =.load(std::memory_order_acquire); if (temp == nullptr) { return false; Node * next = temp->next.load(std::memory_order_relaxed); while (!.compare_exchange_weak(temp, next, std::memory_order_acq_rel)) if (temp == nullptr) { return false; next = temp->next.load(std::memory_order_relaxed); if (temp!= nullptr) { outdata = temp->data; return true; return false; Leak memory here (temp is last reference to node) or is it? 30

31 Difficulty with freeing memory Thread 0 temp Thread 0 does temp =.load; data: 4 data: 8 data: 12 Thread 1 31

32 Difficulty with freeing memory Thread 0 temp Thread 0 does next = temp->next.load; next data: 4 data: 8 data: 12 Thread 1 32

33 Difficulty with freeing memory Thread 0 temp Thread 1 does temp =.load next data: 4 data: 8 data: 12 Thread 1 temp 33

34 Difficulty with freeing memory Thread 0 temp Thread 0 does (successful) CAS on next data: 4 data: 8 data: 12 Thread 1 temp 34

35 Difficulty with freeing memory Thread 0 temp Note: if Thread 0 does not delete temp, Thread 1 will get next, then fail its CAS, then correctly re-get the. next data: 4 data: 8 data: 12 Thread 1 temp 35

36 Difficulty with freeing memory Thread 0 temp but if thread 0 deletes temp next data: 8 data: 12 Thread 1 temp 36

37 Difficulty with freeing memory Thread 0 temp Thread 1 has temp dangling when it does temp->next.load next data: 8 data: 12 Thread 1 temp 37

38 Difficulty with freeing memory Thread 0 temp we could move it to another list of trash [Note that thread 1's CAS will fail] next data: 4 data: 8 data: 12 trash Thread 1 temp 38

39 Difficulty Cleaning Up Memory How can we clean up our trash list? Delete "later"? How long is later? Must ensure no other thread still reference it Could we just recycle the nodes? I.e., allocate nodes out of the trash for this same list? ONLY if we can ensure no other threads still reference it.. Why? Otherwise might have old pointer to address X Reallocate node at X CAS succeeds (X=X) even though we've updated list 39

40 Freeing Data in LF DS Option 1: count threads operating in DS If count == 1, only this thread -> free nodes Difficulty: May never have only 1 thread in DS Option 2: require all threads to finish one operation after update Only can see stale data if in same operation as update Difficulty: Thread may not be doing any operations Option 3: track set of threads operating in DS Difficulty: complicated Option 4: highly scalable R/W locks R = normal operations, W = exclusive operations [freeing] Difficulty: need high read scalability 40

41 Freeing Data in Lock Free DSes GC makes it easy Stop the world Considers root sets from all threads So much simpler in Java 41

42 More Complex Deletions? What about deleting from an arbitrary position? Delete specific value, index, 42

43 More Complex Deletions? Thread 0 Thread 1 todel 4 todel 8 curr curr data: 1 data: 4 data: 8 data: 12 What about deleting from an arbitrary position? Delete specific value, index, Turns out to be more complex.. 43

44 More Complex Deletions? Thread 0 Thread 1 todel 4 todel 8 curr curr data: 1 data: 4 data: 8 data: 12 What would thread 0 do by itself? 44

45 More Complex Deletions? Thread 0 Thread 1 todel 4 todel 8 curr curr data: 1 data: 4 data: 8 data: 12 What would thread 0 do by itself? Atomic CAS is on 1's next field. 45

46 More Complex Deletions? Thread 0 Thread 1 todel 4 todel 8 curr curr data: 1 data: 4 data: 8 data: 12 What would thread 1 do by itself? 46

47 More Complex Deletions? Thread 0 Thread 1 todel 4 todel 8 curr curr data: 1 data: 4 data: 8 data: 12 What would thread 1 do by itself? Atomic CAS is on 4's next 47

48 More Complex Deletions? Thread 0 Thread 1 todel 4 todel 8 curr curr data: 1 data: 4 data: 8 data: 12 What happens if we do both at the same time? Do any CASes fail? 48

49 More Complex Deletions? Thread 0 Thread 1 todel 4 todel 8 curr curr data: 1 data: 4 data: 8 data: 12 What happens if we do both at the same time? Undid 8's delete Do any CASes fail? No: both succeed Similar problem with racing adds + deletes Note: our removefront only works if we only have addfront (what race is there if we also have addsorted?) 49

50 Lock Free BST Add

51 Lock Free BST Add 17 Option 1: CAS 24's left pointer

52 Lock Free BST Add 17 Option 1: CAS 24's left pointer But what if we want to rebalance? 4 24 How would deletes work? - Saw complex for LLs 17 52

53 Lock Free BST Add 17 Option 1: CAS 24's left pointer 24 Rebalanced. How would we do that lock free?

54 Another Option Option 2: Do functional update, and only CAS the root For any operation (add, or delete) 54

55 Add 17 Option 2: Do functional update Lock Free BST temp

56 Add 17 Option 2: Do functional update Lock Free BST temp Atomic CAS /temp

57 Add 17 Option 2: Do functional update Lock Free BST temp Atomic CAS /temp Free unused nodes

58 Lock Free BST Add 17 Option 2: 42 Do functional update Atomic CAS /temp Free unused nodes Done

59 Add 17 Option 2: Do functional update Lock Free BST temp Atomic CAS /temp What if CAS fails?

60 Add 17 Option 2: Do functional update Lock Free BST temp Atomic CAS /temp What if CAS fails? - Head has changed - Conflicting writes - Discard temp tree - Do add again to 17 60

61 Add 17 Option 2: Do functional update Lock Free BST temp Atomic CAS /temp Free unused nodes For nodes seen by other threads just as hard as in LLs.. 61

62 BST: State Re-creation What about all those extra nodes we make? Isn't that inefficient? How much state do we have to recreate for an N node tree? 62

63 BST: State Duplication What about all those extra nodes we make? Isn't that inefficient? How much state do we have to recreate for an N node tree? O(lg(N)) Note that a LinkedList would have O(N) state duplication Could use this idea there, just less efficient.. 63

64 BST: O(lg(N) State Duplication Triangles = arbitrarily large subtree (imagine thousands or millions of nodes) 64

65 BST: O(lg(N) State Duplication Only state along actual add path is duplicated Remainder is shared 12 65

66 Rebalancing? Rebalancing also only affects addition path. Can use same idea: share unaffected state v N+3 r N+2 L N+2 N L N+1 v N+1 N N+1 r N N(-1) N(-1) N N(-1) N(-1) 66

67 Lock Free Data Structures: Wrap Up Upside: scalability Downside: complexity Think carefully about races Think carefully about memory ordering Deletions are generally hardest Freeing memory is tricky 67