Distributed Systems. Day 7: Time and Snapshots [Part 1]

Size: px

Start display at page:

Download "Distributed Systems. Day 7: Time and Snapshots [Part 1]"

Dwayne Merritt
5 years ago
Views:

2 Distributed Systems Day 7: Time and Snapshots [Part 1]

3 Delete Professor Theo!!!! 10am Delete Professor Theo!!!! Server 1 I hate Professor Theo!! But Why? 4pm I hate Professor Theo!! 4:01pm But Why? I hate Professor Theo!! But Why? Server 2 Delete Professor Theo!!!!

4 10am John had an accident!!! He s well now J Server 1 John had an accident!!! Status Comment: Good News!! 4pm 5pm He s well now J Status Comment: Good News!! John had an accident!!! He s well now J Server 2 Status Comment: Good News!!

5 Server 1 Server 2 Server 3

6 Agenda Time Synchronization (NTP, PTP) Challenges and Limitations Distributed Systems Notions Total ordering, partial ordering Logical Time Logical Clocks Vector Clocks

7 How do Clocks works?

8 Clocks and Clock Skew Time is a function of a crystal s oscillations Several oscillations == one second Crystal does not always oscillate at right frequency. So time becomes faster or slower Terminology: describing differences in clocks Skew: difference between two clocks! Drift: counting at different frequency than intended 10 oscillations == 1 sec System Quartz Crystal oscillate Clock 1 Clock 2 10:00am 10:05am Skew = 10am 10:05am

9 Synchronizing Clocks

10 PTP: Precision Time Protocol What time is it? Time 10 am! One client synchronizes with an authoritative clock Through exchange of network messages Problems: Delay between messages Delay in processing Server may be wrong!!!

11 How does your phone get its Time?

12 Clock Synchronization Challenges One client synchronizes with an authoritative clock Through exchange of network messages Problems: Delay between messages Delay in processing Server may be wrong!!! Network Transfer time Network Transfer time Time??? Time: 10 am! 9:30am! 9:50am! 10:00am! 10:20am!

13 Clock Synchronization Common protocols: NTP, PTP Cheap and inaccurate across long distances More accurate protocols: GPS, atomic clocks Requires expensive hardware Signaling issues when indoors Do we really need time precision? What do we care about?

Clock Synchronization Common protocols: NTP, PTP Cheap and inaccurate across long distances More accurate protocols: GPS Expensive and hard to retrofit at global scale Signaling issues when indoors

14 Clock Synchronization Common protocols: NTP, PTP Cheap and inaccurate across long distances More accurate protocols: GPS Expensive and hard to retrofit at global scale Signaling issues when indoors Do we really need time precision? What do we care about?

15 Ordering Total ordering Each server can order events (in same order) Possible with a global entity (e.g. global clock) Partial ordering:!(total ordering) Some ordering that s not total ordered

16 Client sends to server 1 and server 2 Server 1 Server 2 Total ordered Server 1 Server 2 Total ordered Server 1 Server 2 Total ordered Server 1 Server 2 Not Total ordered Recall ``total ordering events processed in the same order in all servers.

17 Linearizability: The Golden Standard. Total order: all servers process message in same order Linearizability: special case of total order Order is based on real time Client sends to server 1 and server 2 Server 1 Server 2 Total ordered Server 1 Server 2 Total ordered

18 10am John had an accident!!! All servers process in same order Different servers Can process in different orders but events from clients in same order 4pm He s well now J Cut myself! Headed to the Dr. Status Comment: Bummer!!!! Cut myself! Headed to the Dr. Status Comment: Bummer!!!! Cut myself! Headed to the Dr. Status Comment: Bummer!!!! Cut myself! Headed to the Dr. I m fixed!!!! I m fixed!!!! I m fixed!!!! I m fixed!!!! Status Comment: Bummer!!!! Status Comment: Good News!! Status Comment: Good News!! Status Comment: Good News!! Status Comment: Good News!! 11am 5pm Status Comment: Hope He s Okay!!!! Status Comment: Good News!! Server 1 Server 2 Server 1 Server 2 Total Order: order based on global time FIFO Order: order based on client time

19 How can we implement total/fifo ordering? System Assumptions No server (node) failure. Network is reliable: messages are delivered eventually Messages are delivered in-order

20 Implementing Total Ordering Key insight: need a third party to assign IDs Step 2: ID_i++ Global Sequencer Process Pseudocode Before sending message, m, ask for an ID Include ID with messages Send to all Servers!!! Step1: GetID() Step 2: sendmsg (ID_i) Server 1 Global Sequencer Pseudocode Start ID =0 On receive ID request, Increment ID (so, ID++) Assign ID to Process clients Step3: sendmsg (m, ID_i) Server Pseudocode On receive message place in queue Order Queue by ID process Queue in order of ID clients Server 2

21 Implementing FIFO Ordering Local sequencer: Process locally maintain ID ID is monotonic Client Pseudocode Increment ID Assign ID to message Send message to all servers ID_A++; sendmsg (m, ID_A) Server 1 Server Pseudocode Maintain a queue for each client Order message by ID -- process in order This is FIFO Order ID_B++; clients clients sendmsg (m, ID_B) Client maintain different IDs: local and monotonic Server 2

22 How can we implement total/fifo ordering? Current: System Assumptions No server (node) failure. Network is reliable: Messages are delivered eventually New: System Assumptions Synchronous failures. Network is unreliable: Messages can be lost Messages can be duplicated

23 Do We Always Need Total Ordering? For some applications: yes!!! My back account For others: ``causal ordering Dependency between events are preserved A à B: on all servers A is processed before B. New Post: 1 Comment on Post Like post New Post: 2 Comment on Post Like post Make file1 Write to file1 Make file2 Delete file2 Add $100 Add $100 How do I capture and enforce causal ordering??? New Post: 3 Comment on Post Like post

24 Logical Clocks Each process maintains a ID i.e., Clock P1 A C E G Initially set to 0 Rules for updating Clock Each event increments the ID Event: Send msg, recvmsg, or run function Include ID in every message On Receiving a message: ID = max (my_id, ID-in-msg) ++ Only way to exchange information is through message exchange IDs are only exchanged through message exchange P2 B A C E G D F B D F If x->z, then clock(z) > clock(x)

25 Ordering with Logical Clocks Which ordering do you get? Total? FIFO? Causal? P1 A C m1 E G m2 Is B before or after A? Is C before or after D? Is F before or after E? If e->e, then clock(e ) > clock(e) Logical clocks do help with inferring ordering: can confirm P2 B A 1 C 2 E 3 G 5 P1 D F B 1 D 3 F 4 P2

26 Vector Clocks [0,0] Each process maintains an array of ID Array is of size N (N= # of processes) All entries initialized to 0 Array is called a Vector clock (VC) P1 [0,0] P2 A B C m1 D E F G m2 Size (VC i ) == N VC i [i] Number of events at P i VC i [h] = K Process P i is aware of the first k events at P h If VC x <= VC z then x->z (x happens before z) A [?,? ] C [?,? ] E [?,? ] G [?,? ] B [?,? ] D [?,? ] F [?,? ]

27 Vector Clocks [0,0] Each process maintains an array of ID Array is of size N (N= # of processes) All entries initialized to 0 Rules for updating Vector Clock for process Z Each event increments VC z [z] So each event increments processes clock in vector Event: Send msg, recv msg, or run function Include ENTIRE vector in every message On Receiving a message from process X This message will have VC x [] X s vector clock Z updates its clock for each entry: VC z [K] = max (VC z [K], VC x [k]) ++ Only way to exchange information is through message exchange IDs are only exchanged through message exchange P1 [0,0] P2 A B A [?,? ] C [?,? ] E [?,? ] G [?,? ] C m1 D E G m2 F B [?,? ] D [?,? ] F [?,? ] If VC x <= VC z then x->z (x happens before z)

28 Ordering with Vector Clocks Which ordering do you get? Total? FIFO? Causal? P1 A C m1 E G m2 Is B before or after A? Is C before or after D? Is F before or after E? P2 B A [ 1, 0 ] C [ 2, 0 ] E [ 3, 0 ] G [ 4, 3 ] D F B [ 0, 1 ] D [ 2, 2 ] F [ 2, 3 ] If VClock(e ) x <= z >= then VClock(e) x->z (x happens then e->e before z) Vector clocks are enough for FIFO and Causal ordering P1 P2

29 Terminology Review Total order: All servers process events in same order Linearizability: total order according to global time Requires: magical global time or a global mutex FIFO order: all servers process events in order generated by client More practical: you can trust client clock to be monotonic Logical clock: counters Vector clock: a vector of logical clocks

30 Logical Time Vector clocks provider better ordering than logical clocks HOWEVER!! Expensive scales with # of processes Network message: include vector clocks Processes: must maintain vector clocks Total ordering is a close second but too costly Achievable with global sequencer Ideal is Linearizability but impractical FIFO ordering: easy but less meaningful Causal ordering: goal for most distributed systems Achievable with vector clocks

Distributed Systems. Day 9: Replication [Part 1]

Distributed Systems. Day 9: Replication [Part 1] Distributed Systems Day 9: Replication [Part 1] Hash table k 0 v 0 k 1 v 1 k 2 v 2 k 3 v 3 ll Facebook Data Does your client know about all of F s servers? Security issues? Performance issues? How do clients