Architecting for Failure in a Containerized World. Tom Faulhaber Infolace

Transcription

1 Architecting for Failure in a Containerized World Tom Faulhaber Infolace

2

3

4

5 How can container tech help us build robust systems?

6 Key takeaway: an architectural toolkit for building robust systems with containers

7 The Rules Decomposition Orchestration and Synchronization Managing Stateful Apps

8 Simplicity

9 Simple means: Do one thing!

10 The opposite of simple is complex

11 Complexity exists within components

12 Complexity exists between components

13 Example: a counter x Counter Service Counter Service

14 Example: a counter Load Balancer Counter Service Counter Service

15 State + composition = complexity

16 Part 1: Decomposition

17 Rule: Decompose vertically

18 App Server Service #1 Service #2 Service #3

19 App Server

20 Rule: Separation of concerns

21 Example: Logging App Logging Server Core Code Config Logging Driver

22 Example: Logging App Core Code Logging Server StdOut Logger Config Logging Driver

23 Aspect-oriented programming

24 Rule: Constrain state

25 Session Store Relational DB

26 Rule: Battle-tested tools

27 Redis MySQL

28 Rule: High code churn Easy restart

29 Rule: No start-up order!

30 a b c d time

31 a b c d x time

32 a b c x x x d x time

33 a b c x x x d x time

34 a b c d time

35 a b c d time

36 a b c d time

37 Rule: Consider higher-order failure

38 The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Managing Stateful Apps

39 Part 2: Orchestration and Synchronization

40 Rule: Use Framework Restarts

41 Mesos: Marathon always restarts Kubernetes: RestartPolicy=Always Docker: Swarm always restarts

42 Rule: Create your own framework

43 Mesos Master Framework Driver Mesos Agent Framework Executor Mesos Agent Framework Executor Mesos Agent Framework Executor

44 Rule: Use Synchronized State

45 Synchronized State Tools: - zookeeper - etcd - consul Patterns: - leader election - shared counters - peer awareness - work partitioning

46 Rule: Minimize Synchronized State

47 Even battle-tested state management is a headache. (Source:

48 The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Use framework restarts Create your own framework Use synchronized state Minimize synchronized state Managing Stateful Apps

49 Part 3: Managing Stateful Apps

50 Rule (repeat!): Always use battle-tested tools! (State is the weak point)

51 Rule: Choose the DB architecture

52 Option 1: External DB Execution cluster Database cluster

53 Option 1: External DB Pros Somebody else s problem! Can use a DB designed for clustering directly Can use DB as a service Cons Not really somebody else s problem! Higher latency/no reference locality Can t leverage orchestration, etc.

54 Option 2: Run on Raw HW App App App Marathon Mesos Marathon Mesos Marathon Mesos HDFS HDFS HDFS

55 Option 2: Run on Raw HW Pros Use existing recipes Have local data Manage a single cluster Cons Orchestration doesn t help with failure Increased management complexity

56 Option 3: In-memory DB App App App MemSQL MemSQL MemSQL Marathon Mesos Marathon Mesos Marathon Mesos

57 Option 3: In-memory DB Pros No need for volume tracking Fast Have local data Cons Bets all machines won t go down Bets on orchestration framework Manage a single cluster

58 Option 4: Use Orchestration Mesos Mesos Mesos App App App Marathon Marathon Marathon Cassandra Cassandra Cassandra

59 Option 4: Use Orchestration Pros Orchestration manages volumes One model for all programs Cons Currently the least mature Not well supported by vendors Have local data Single cluster

60 Option 5: Roll Your Own Mesos Master Mesos Mesos Mesos Framework App App App Marathon Marathon Marathon ImageMgr ImageMgr ImageMgr

61 Option 5: Roll Your Own Pros Very precise control You decide whether to use containers Have local data Cons You re on your own! Wedded to a single orchestration platform Not battle tested Can be system aware

62 Rule: Have replication

63 The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Use framework restarts Create your own framework Use synchronized state Minimize synchronized state Managing Stateful Apps Battle-tested tools Choose the DB architecture Have replication

64 Fin

65 References Rich Hickey: Are We There Yet? ( There-Yet-Rich-Hickey) Simple Made Easy ( Made-Easy-QCon-London-2012) David Greenberg, Building Applications on Mesos, O Reilly, 2016 Joe Johnston, et al., Docker in Production: Lessons from the Trenches, Bleeding Edge Press, 2015

66 The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Use framework restarts Create your own framework Use synchronized state Minimize synchronized state Managing Stateful Apps Battle-tested tools Choose the DB architecture Have replication