CTDB Remix - Dreaming the Fantasy

Size: px

Start display at page:

Download "CTDB Remix - Dreaming the Fantasy"

Octavia Dickerson
5 years ago
Views:

1 CTDB Remix I: Dreaming the Fantasy Samba Team IBM (Australia Development Labs, Linux Technology Center) SambaXP 2017

2 CTDB Project Motivation: Support for clustered Samba Multiple nodes active simultaneously Communication between nodes (heartbeat, failover) Distributed databases between nodes Features: Volatile and Persistent databases Cluster-side messaging for Samba IP failover and load balancing Service monitoring Community: ctdb.samba.org git.samba.org/samba.git wiki.samba.org/index.php/ctdb and Clustered Samba

3 Overview Dreaming the Fantasy How did we get here? Evolving the design Laying the foundations New Architecture Designing the Reality

4 How did we get here?

5 2013 Recap SambaXP 2013

6 2013 Recap SambaXP 2013 Introduced lock helper, event helper

7 2014 Recap CTDB merges with Samba CTDB merged into Samba tree (Nov 2013) CTDB standalone waf build (Jun 2014) CTDB build integrated in toplevel build (Nov 2014)

8 2015 Recap Parallel Database Recovery Protocol marshalling New abstractions New Communication framework (tevent req based async) New Client Code Database recovery helper Re-implemented ctdb tool using new client API

9 2015 Recap Parallel Database Recovery Protocol marshalling New abstractions New Communication framework (tevent req based async) New Client Code Database recovery helper Re-implemented ctdb tool using new client API Introduced natgw helper

10 2016 Recap Introduced killtcp helper, lvs helper

11 2016 Recap Introduced killtcp helper, lvs helper Event daemon New abstractions - run proc, sock daemon Event Protocol Event client code Event handling daemon

12 Evolving the design

13 Evolving the design Identifying CTDB functions

14 Evolving the design Identifying CTDB functions Eventscripts Monitor Attach/Detach Freeze/Thaw Vacuuming Calls Messages Unix Socket Tunables Disable/Enable Healthy/Unhealthy Ban/Unban ReleaseIP TakeIP IPreallocated Locking Traverse Migration Transaction Transports ctdb daemon Controls Election Recovery Recovery Lock IP failover IP allocation Consistency recovery daemon

15 Evolving the design Grouping CTDB functions Calls Controls Messages Healthy/Unhealthy Eventscripts Monitor Start/Stop Unix Socket Transports TakeIP/ReleaseIP/IPreallocated IP assignment Disable/Enable Recovery Election Banning Consistency Attach/Detach Transactions Freeze/Thaw Migrations Locking Traverse Vacuuming Protocol ctdb daemon recovery daemon

16 Evolving the design New subsystems Service manager Protocol IP failover Cluster manager Protocol Clustered database ctdb daemon recovery daemon

17 Evolving the design Redesign of server code

18 Evolving the design Redesign of server code First approach Main concern is the transport

19 Evolving the design Redesign of server code First approach Main concern is the transport Motivation Avoid m*n*n connections

20 Evolving the design Redesign of server code First approach Main concern is the transport Develop parallel transport and proxies (scalability) Motivation Avoid m*n*n connections

21 Evolving the design Redesign of server code First approach Main concern is the transport Develop parallel transport and proxies (scalability) Convert CTDB transport to use proxy Motivation Avoid m*n*n connections

22 Evolving the design Redesign of server code First approach Main concern is the transport Develop parallel transport and proxies (scalability) Convert CTDB transport to use proxy Motivation Avoid m*n*n connections Unix datagrams (SambaXP 2015) tmsgd - fd passing (SambaXP 2016)

23 Evolving the design Redesign of server code First approach Main concern is the transport Develop parallel transport and proxies (scalability) Convert CTDB transport to use proxy Motivation Avoid m*n*n connections Unix datagrams (SambaXP 2015) tmsgd - fd passing (SambaXP 2016) Proxy design never took off

24 Evolving the design Redesign of server code

25 Evolving the design Redesign of server code Second approach Split code out and create separate daemons

26 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction

27 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction First candidate event daemon

28 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction First candidate event daemon eventd

29 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction First candidate event daemon eventd Does not need any CTDB infrastructure

30 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction First candidate event daemon eventd Does not need any CTDB infrastructure run proc abstraction

31 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction First candidate event daemon eventd Does not need any CTDB infrastructure run proc abstraction New protocol - request, reply

32 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction First candidate event daemon eventd Does not need any CTDB infrastructure run proc abstraction New protocol - request, reply Easy testing

33 Evolving the design Redesign of server code Second approach Split code out and create separate daemons Avoid boilerplate, most daemons to use unix domain sockets sock daemon abstraction First candidate event daemon eventd Does not need any CTDB infrastructure run proc abstraction New protocol - request, reply Easy testing run event abstraction

34 Laying the foundations

35 Laying the foundations CTDB State management Solved differently for different things tickles - Protocol to sync tickle lists nfs locks - Using persistent database Persistent databases are slow We are in the business of clustered databases New database model?

36 Laying the foundations CTDB State management Solved differently for different things tickles - Protocol to sync tickle lists nfs locks - Using persistent database Persistent databases are slow We are in the business of clustered databases New database model? Replicated database State information needed during lifetime of CTDB Volatile (CLEAR IF FIRST) Replicated (Re-use existing API) Uses g lock and transactions

37 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test

38 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test g lock test persistent replicated

39 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test on disk tmpfs g lock test fcntl persistent replicated

40 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test g lock test on disk tmpfs fcntl mutexes persistent replicated

41 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test g lock test on disk tmpfs fcntl mutexes persistent on disk tmpfs fcntl replicated

42 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test g lock test on disk tmpfs fcntl mutexes persistent on disk tmpfs fcntl mutexes replicated

43 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test g lock test on disk tmpfs fcntl mutexes persistent replicated on disk tmpfs fcntl mutexes on disk tmpfs fcntl

44 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test g lock test on disk tmpfs fcntl mutexes persistent replicated on disk tmpfs fcntl mutexes on disk tmpfs fcntl mutexes 80 89

45 Laying the foundations Database Performance 2 nodes (Intel Xeon E5620, RHEL6), 30 second test g lock test on disk tmpfs fcntl mutexes persistent on disk tmpfs fcntl mutexes replicated on disk tmpfs fcntl mutexes Why are transactions with mutexes so slow?

46 Laying the foundations Node-to-node communication

47 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based

48 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues

49 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message

50 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message Fire and forget

51 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message Fire and forget Tunnels

52 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message Fire and forget Tunnels New packet type CTDB REQ TUNNEL

53 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message Fire and forget Tunnels New packet type CTDB REQ TUNNEL Uses existing CTDB transport

54 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message Fire and forget Tunnels New packet type CTDB REQ TUNNEL Uses existing CTDB transport Register tunnels with tunnel id (new controls)

55 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message Fire and forget Tunnels New packet type CTDB REQ TUNNEL Uses existing CTDB transport Register tunnels with tunnel id (new controls) Client API to encapsulate packets

56 Laying the foundations Node-to-node communication CTDB REQ MESSAGE Can carry abritrary data Used by recovery daemon, samba SRVID based Several issues Multiple processes can get same message Fire and forget Tunnels New packet type CTDB REQ TUNNEL Uses existing CTDB transport Register tunnels with tunnel id (new controls) Client API to encapsulate packets New daemons can use new protocol

57 Laying the foundations Rethinking subsystem design

58 Laying the foundations Rethinking subsystem design Loose coupling to CTDB

59 Laying the foundations Rethinking subsystem design Loose coupling to CTDB Notifications (subsystem )

60 Laying the foundations Rethinking subsystem design Loose coupling to CTDB Notifications (subsystem ) Actions (subsystem )

61 Laying the foundations Rethinking subsystem design Loose coupling to CTDB Notifications (subsystem ) Actions (subsystem ) State transition graphs

62 Laying the foundations Rethinking subsystem design Loose coupling to CTDB Notifications (subsystem ) Actions (subsystem ) State transition graphs Candidates Cluster Manager Service Monitoring IP Failover

63 New Architecture

64 Cluster Manager

65 Cluster Manager Split cluster manager code

66 Cluster Manager Split cluster manager code Node 0 Node 1 Node 2 Cluster Manager Cluster Manager Cluster Manager CTDB CTDB CTDB

67 Cluster Manager Split cluster manager code Keep it loosely coupled Node 0 Node 1 Node 2 Cluster Manager Cluster Manager Cluster Manager CTDB CTDB CTDB

68 Cluster Manager Split cluster manager code Keep it loosely coupled Node 0 Node 1 Node 2 Cluster Manager Cluster Manager Cluster Manager Glue Glue Glue CTDB CTDB CTDB

69 Cluster Manager Split cluster manager code Keep it loosely coupled Support 3rd party replacements (e.g. etcd) Node 0 Node 1 Node 2 Cluster Manager Cluster Manager Cluster Manager Glue Glue Glue CTDB CTDB CTDB

70 Cluster Manager Split cluster manager code Keep it loosely coupled Support 3rd party replacements (e.g. etcd) Node 0 Node 1 Node 2 Cluster Manager Cluster Manager Cluster Manager Glue Glue Glue CTDB CTDB CTDB

71 Service Manager and IP Failover

72 Service Manager and IP Failover Separate service manager code (eventd +... )

73 Service Manager and IP Failover Separate service manager code (eventd +... ) Node 0 Node 1 Node 2 Service Manager Service Manager Service Manager CTDB CTDB CTDB

74 Service Manager and IP Failover Separate service manager code (eventd +... ) Separate IP failover code Node 0 Node 1 Node 2 Service Manager Service Manager Service Manager CTDB CTDB CTDB

75 Service Manager and IP Failover Separate service manager code (eventd +... ) Separate IP failover code Node 0 Node 1 Node 2 IP Failover IP Failover IP Failover Service Manager Service Manager Service Manager CTDB CTDB CTDB

76 Service Manager and IP Failover Separate service manager code (eventd +... ) Separate IP failover code IP failover as a service Node 0 Node 1 Node 2 IP Failover IP Failover IP Failover Service Manager Service Manager Service Manager CTDB CTDB CTDB

77 Service Manager and IP Failover Separate service manager code (eventd +... ) Separate IP failover code IP failover as a service Node 0 Node 1 Node 2 IP Failover IP Failover IP Failover Service Manager Service Manager Service Manager CTDB CTDB CTDB

78 Questions / Comments

The State of Samba (June 2011) Jeremy Allison Samba Team/Google Open Source Programs Office

The State of Samba (June 2011) Jeremy Allison Samba Team/Google Open Source Programs Office jra@samba.org jra@google.com What is Samba? Provides File/Print/Authentication services to Windows clients from