Spanner: Google s Globally- Distributed Database

Transcription

1 Spanner: Google s Globally- Distributed Database Google, Inc. OSDI 2012 Presented by: Karen Ouyang

2 Problem Statement Distributed data system with high availability Support external consistency!

3 Key Ideas Distributed data system with high availability Supports external consistency! Enabling technology: TrueTime API

4 Server Organization datacenters have one or more zones

5 Server Organization handles moving data across zones assigns data to spanservers used by clients to locate spanservers serves data to clients

6 Spanserver Stack between 100 and 1000 instances (key:string, :int64) string

7 Spanserver Stack set of replicas: Paxos group writes initiate Paxos protocol at leader; reads from any sufficiently up-to-date replica

8 Spanserver Stack supports distributed transactions contains state for two-phase locking

9 supports distributed transactions contains state for two-phase locking transactions with 1+ group: two-phase commit select coordinator leader from leaders

10 TrueTime API Exposes clock uncertainty by expressing time as an interval Uses GPS and atomic clocks Time master machines per datacenter Client polls multiple masters to compute time interval

11 TrueTime API time TT.now() earliest latest 2ϵ

12 Consistency Ensure external consistency by ensuring order All transactions are assigned Data written by T is ed with s

13 Two-phase locking: assign s at any time that locks are held Assign s to Paxos writes in increasing order across leaders A leader only assigns s within its leader lease; leader leases are disjoint

14 Transactions: two-phase commit Two transactions start commit T 1 start commit T 2 Assign commit s with s 1 < s 2 How?

15 Start: commit is after time of commit request at server or: t abs (e 2 server ) s

16 Commit wait: cannot see data committed by T until s (assigned ) has passed pick s = TT.now().latest s

17 Commit wait: cannot see data committed by T until s (assigned ) has passed pick s = TT.now().latest s wait until s < TT.now().earliest

18 Commit wait: cannot see data committed by T until s (assigned ) has passed pick s = TT.now().latest s wait until s < TT.now().earliest commit wait

19 Commit wait: cannot see data committed by T until s (assigned ) has passed pick s = TT.now().latest s wait until s < TT.now().earliest commit wait

20 s 1 < t abs (e commit 1 ) s 1 < t abs (e commit 1 ) < t abs (e start 2 ) s 1 < t abs (e commit 1 ) < t abs (e start 2 ) < t abs (e server 2 ) s 1 < t abs (e commit 1 ) < t abs (e start 2 ) < t abs (e server 2 ) s 2 s 1 < s 2

21 Two-phase commit coordinator leader

22 Two-phase commit: client begins coordinator leader

23 Two-phase commit coordinator leader

24 Two-phase commit coordinator leader choose prepare

25 Two-phase commit log prepare record in Paxos coordinator leader choose prepare

26 Two-phase commit log prepare record in Paxos coordinator leader send prepare choose prepare

27 Two-phase commit log prepare record in Paxos coordinator leader send prepare choose prepare choose commit

28 Two-phase commit log prepare record in Paxos log commit in Paxos coordinator leader send prepare choose prepare choose commit

29 Two-phase commit log prepare record in Paxos log commit in Paxos coordinator leader send prepare choose prepare choose commit commit wait done

30 Two-phase commit log prepare record in Paxos log commit in Paxos coordinator leader send prepare notify choose prepare choose commit commit wait done

31 Two-phase commit log prepare record in Paxos log commit in Paxos coordinator leader send prepare notify choose prepare choose commit commit wait done

32 Two-phase commit log prepare record in Paxos log commit in Paxos log outcome in Paxos coordinator leader send prepare notify choose prepare choose commit commit wait done

33 Two-phase commit log prepare record in Paxos log commit in Paxos log outcome in Paxos coordinator leader send prepare notify choose prepare choose commit commit wait done

34 Read-Only Transactions Serving reads at a Replica tracks safe time t safe : can read t t safe Define t safe = min(t Paxos, t TM ) Assigning s to RO transactions Simplest: assign s read = TT.now().latest May block; should assign oldest that preserves external consistency

35 Microbenchmarks Two-phase commit scalability

36 Microbenchmarks Effect of killing servers on throughput

37 Performance TrueTime F1, Google s advertising backend Automatic failover High standard deviation for latency?

38 Final Thoughts Implemented at a large scale (F1)! Commit wait is pretty clever Very dependent on clocks Security?

39 References Corbett et al. Spanner: Google s Globally-Distributed Database. Proc. Of OSDI