Flaco Lock Service. RESTful Distributed Locking Over HTTP. Jose Falcon Bryon Jacob

Transcription

1 Flaco Lock Service RESTful Distributed Locking Over HTTP Jose Falcon Bryon Jacob 1

2 About Me Jose Falcon Graduate University of Texas Expected graduahon: May 2010 Research interests: Programming languages ArHficial intelligence 2

3 Outline What is synchronizahon? Locks to the rescue! Why Flaco? Flaco Lock Server Flaco Locks Reliability With Unreliability Demo using curl Flaco Lock Client FiTng HomeAway Conclusion QuesHons 3

4 What is synchronizahon? 4

5 SynchronizaHon A problem typically found in mulh threaded (concurrent) environments EssenHally a problem of coordinahon May appear in many different forms 5

6 Too Much Milk 6

7 What s the Problem? Bought too much milk; overworking! CollecHon of workers, all capable of execuhng the same task, work without coordinahon Single task executed once by some worker 7

8 Address 123 Main St. null Aus0n, TX Telephone # Telephone # null null Address null Lost Data LisHng Address Telephone # Address 123 Main St. null Aus0n, TX Telephone # Telephone # null null (800) Telephone # null (800)

9 What s the Problem? Accessing shared object without considering changes made by other users The data is inconsistent and out of date! Prevent users from accessing stale data 9

10 Locks to the rescue! 10

11 Not Too Much Milk went for milk 11

12 Address 123 Main St. null Aus0n, TX Telephone # Telephone # null null Address 123 Main St. Aus0n, TX Persistent Data LisHng Address Telephone # Address null 123 Main St. Aus0n, TX Telephone # Telephone # null null (800) Telephone # (800) null 12

13 Locks One soluhon to the synchronizahon problem Simple and intuihve Requires: Atomic lock/unlock operahons Extreme reliability CooperaHon All users must respect the lock! 13

14 Why Flaco? 14

15 Current SoluHon At HomeAway locks are created, on demand, in a database Problems: Burden to create Requires database connechon Overhead for simple applicahons which don t rely on a DB 15

16 What We Want RESTful service for easily creahng / sharing locks across mulhple applicahons Usable in any applicahon No dependency on: Database Any specific language AddiHonally: Java client library for easily using Flaco 16

17 Flaco Lock Server 17

18 Flaco Lock Server Centralized service for manipulahng logical named locks Simplifies the creahon of locks All possible locks logically exist Stores locks in RAM, no need for a database Convenient access through REST API Any program/language supporhng HTTP requests can synchronize using Flaco 18

19 Flaco Lock Service ApplicaHons hmp request hmp request hmp request Flaco Lock Server hmp request 19

20 Asynchronous Basic operahons: inquire acquire renew release wait signal signalall Flaco Server Details 20

21 Inquire Returns informahon regarding the current state of the lock Current owner Current waiters on the ready queue Current waiters on condihon queues 21

22 Acquire Two types of acquisihon: Poll Callback Poll Simply awards the lock if it is available Callback Timed acquisihon with a callback server 22

23 Callback Acquire If the lock is available, award ownership Else queue the requester in the ready queue The server will nohfy the requester when she is eligible for acquisihon at the given callback release( foo ) ( foo, newpid) ( foo, pid) You now own the lock on foo! failed acquisihon, enqueued acquire( foo, 9000, 5) 23

24 Renew Limited Hme ownership Must renew lease prior to lease Hmeout Failure to renew results in auto release Ensures liveliness of owner 24

25 Release Removes current owner Awards ownership to next alive waiter Sends nohficahons to callback URLS 25

26 CondiHon Variables Implicit condihon All locks have an implicit condihon Explicit condihon Support arbitrary condihons on names Allow complex interachons between owners 26

27 Wait, Signal/All Wait Removes ownership of the lock Placed on condihon wait queue Signal/All Removes single/(all) waiter(s) from the condihon wait queue to the ready queue 27

28 REST API Base URL: hmp://flaco.homeaway.com/api/<lockname> GET Inquires about the state of the lock POST Mapped to various achons: acquire renew... 28

29 AcquisiHon Status Determine if a given PID is shll pending acquisihon Base URL: hmp://flaco.homeaway.com/api/status/<pid> 29

30 Flaco Locks 30

31 Flaco Lock Details Lock security Reentrancy PID may not mulh block Hierarchical locks 31

32 Lock Security Public/Private key paradigm PID/CredenHals Each enhty that acquires a lock must have a server unique PID ManipulaHons on a lock require lock credenhals 32

33 Reentrant Locks Requests for the same lock by the same PID increment a reentrancy depth Releasing a lock decrements this count Locks are released when the depth reaches zero 33

34 MulH Block Owners that are logically blocked cannot funchon as if they are not May not: Wait twice on the same lock Acquire/wait on another lock 34

35 trips Hierarchical locks trips/1234/ trips/1234/calendar trips/ trips/1235/ 1234 calendar

36 Fair! Locks are handed out according to when they were requested! Removes possibility of starvahon trips/1234/ trips/1234/calendar trips/ trips/1235/ 36

37 Reliability With Unreliability 37

38 Problem How do we maintain lock reliability with an inherently unreliable network connechon?! 38

39 Reliability Not unique to Flaco! All distributed locking systems using a network connechon will encounter this problem What if A client stops communicahng? Then she never renews her lease and the server will remove her as owner (at some point), guaranteed Ensures progress The server goes down? A backup server picks up where the other leu off 39

40 Demo using curl 40

41 Flaco Lock Client 41

42 What We Want Java library for interachng with Flaco AbstracHon of the REST API Replace local synchronizahon with distributed synchronizahon without significant code change 42

43 Lock Client Interface A lock client interacts with a lock server Single method: getlock(lockname) Returns a distributed lock object for easy manipulahon 43

44 Distributed Lock Interface Basic interface: lock() lockinterruphbly() trylock(hmeout) unlock() getnamedcondihon(condihonname) dosynchronized(callable) check() Standard Java Lock Interface 44

45 Java Client API Java implementahon handles all interachon with the Flaco Server Maintains an auto renew thread Manages internal callback server 45

46 getnamedcondihon Distributed version of Java s getnewcondihon by providing names to condihons Returns a distributed condihon object for the provided name Prevents busy waihng 46

47 dosynchronized Distributed version of synchronized keyword Executes some computahon within the scope of the lock lock.lock() lock.dosynchronized(new Callable<Void> { try { // perform some complicated achon return callable.run(); }); } finally { lock.unlock(); } 47

48 check Determines if the lock object shll holds the lock at the server 48

49 Code Snippet LockClient lc = new FlacoLockClient( flacoserver ); DistributedLock mydistrlock = lc.getlock( foobar ); mydistrlock.lock(); try { // set the phone number to (800) } finally { mydistrlock.unlock(); } Standard locking template in Java 49

50 InterrupHng Code ExecuHon Imagine a broken connechon with the server How can we interrupt the execuhng code?! Possible, though difficult Rely on throwing excephons on lock manipulahons to break execuhon All operahons on a lock or condihon variable may throw a LostLockExcep0on indicahng that the lock has been lost at the server 50

51 LockClient lc Example mydistrlock.lock(); try { distributedcount++; // do some complicated action here mydistrcond.signalall() } catch (LostLockException e) { // undo some complicated action here distributedcount ; } finally { mydistrlock.unlock(); } 51

52 FiTng HomeAway 52

53 Advantages of Flaco Extremely easy to use No database dependency No language dependency REST API Performance benefits (hypothesis) 53

54 HomeAway Core service Improve reliability of feed processing Expose synchronizahon to scripts Use in Mesa Distributed file system currently being built Outside of HomeAway Flaco will be open source 54

55 Thanks Bryon Jacob (mentor) Alex Victoria Raul Mireles The enhre services team HomeAway 55

56 Thank you!! QuesHons?? 56