UPGRADING A MESOS CLUSTER

Size: px

Start display at page:

Download "UPGRADING A MESOS CLUSTER"

Loreen Sutton
6 years ago
Views:

THE DREAM Web Service Mesos 0.28.1 Mesos 0.29.

9 DEPRECATION & UPGRADES From version 1.0 onward: All stable versions are loosely compatible Loose compatibility requires no dependence on deprecated functionality This means that intermediate minor releases can be skipped in a rolling upgrade as long as no deprecated features are being used! 2016 Mesosphere, Inc. All Rights Reserved. 9

12 PROTOCOL BUFFERS Messages sent between master and agent are Google protobuf messages Members of these messages may change: Can be added or removed Field type may change from optional to required or vice-versa 2016 Mesosphere, Inc. All Rights Reserved. 12

13 PROTOCOL BUFFERS Messages sent between master and agent are Google protobuf messages : message ReservationInfo { required string principal = 1; } Members of these messages may change: Can be added or removed Field type may change from optional to required or vice-versa , with checks to ensure it s set: message ReservationInfo { optional string principal = 1; } : checks removed 2016 Mesosphere, Inc. All Rights Reserved. 13

14 PROTOCOL BUFFERS : message ReservationInfo { required string principal = 1; } When an incompatible message is received by a master or agent, it can be silently dropped! , with checks to ensure it s set: message ReservationInfo { optional string principal = 1; } : checks removed 2016 Mesosphere, Inc. All Rights Reserved. 14

15 MESOS APIs Mesos defines APIs for developers to code against: Native scheduler/executor interface (libmesos) HTTP scheduler/executor API interface Authenticators Authorizers Isolators Anonymous HTTP operator API 2016 Mesosphere, Inc. All Rights Reserved. 15

16 MESOS APIs Mesos defines APIs for developers to code against: Native scheduler/executor interface (libmesos) HTTP scheduler/executor API interface Authenticators Authorizers Isolators Anonymous HTTP operator API This includes the SchedulerDriver and ExecutorDriver APIs Provides C++, Java, and Python libraries for writing frameworks/executors. Schedulers and executors should make use of the same library that the Mesos master/agent are linked against Mesosphere, Inc. All Rights Reserved. 16

17 MESOS APIs Mesos defines APIs for developers to code against: Native scheduler/executor interface (libmesos) HTTP scheduler/executor API interface Authenticators Authorizers Isolators Anonymous HTTP operator API New APIs, which will eventually replace the native interfaces Temporary scheduler and executor shims in Java will enable drop-in replacement Frameworks and executors can simply adhere to the appropriate versioned API, beginning with v1 in Mesos Mesosphere, Inc. All Rights Reserved. 17

18 MESOS APIs Mesos defines APIs for developers to code against: Native scheduler/executor interface (libmesos) HTTP scheduler/executor API interface Authenticators Authorizers Isolators Anonymous HTTP operator API Mesos supports both master and agent modules Hooks are provided: their placement within the scheduler/executor life cycle may change Should be recompiled with the appropriate libmesos 2016 Mesosphere, Inc. All Rights Reserved. 18

19 MESOS APIs Mesos defines APIs for developers to code against: Native scheduler/executor interface (libmesos) HTTP scheduler/executor API interface Authenticators Authorizers Isolators Anonymous HTTP operator API For Mesos 0.28 and earlier, the operator HTTP endpoints are not versioned. The changelog of each release covers changes. Beginning with Mesos 1.0, the operator API is versioned. The API versions correspond to major release versions Mesosphere, Inc. All Rights Reserved. 19

20 ABI COMPATIBILITY Language bindings and module libraries should all be built using the same environment and dependencies Anything depending on libmesos should use the latest, upgraded version 2016 Mesosphere, Inc. All Rights Reserved. 20

22 UPGRADE PROCEDURE Scheduler ZooKeeper Quorum 1. Recompile Mesos modules 2. Masters: rolling upgrade 3. Agents: rolling upgrade 4. Upgrade frameworks Leading Master Mesos Master Mesos Master Agent Agent Task Task 2016 Mesosphere, Inc. All Rights Reserved. 22

23 UPGRADE PROCEDURE Scheduler ZooKeeper Quorum 1. Recompile Mesos modules 2. Masters: rolling upgrade 3. Agents: rolling upgrade 4. Upgrade frameworks Leading Master Mesos Master Mesos Master Agent Agent Task Task 2016 Mesosphere, Inc. All Rights Reserved. 23

24 UPGRADE PROCEDURE 1. Recompile Mesos modules s should be built using the same versions of dependencies as the Mesos master/agent. s should use the same libmesos as the master/agent. Note: bundled dependencies in Mesos may change from one version to the next! 2016 Mesosphere, Inc. All Rights Reserved. 24

25 UPGRADE PROCEDURE Scheduler ZooKeeper Quorum 1. Recompile Mesos modules 2. Masters: rolling upgrade 3. Agents: rolling upgrade 4. Upgrade frameworks Leading Master Mesos Master Mesos Master Agent Agent Task Task 2016 Mesosphere, Inc. All Rights Reserved. 25

26 UPGRADE PROCEDURE 2. Masters: rolling upgrade 1. Stop the master process 2. Remove old master, install new 3. Start the new, upgraded master 4. Wait for the replicated log to finish recovering: Successfully recovered registrar 5. Return to step #1 on another node 2016 Mesosphere, Inc. All Rights Reserved. 26

27 UPGRADE PROCEDURE Scheduler ZooKeeper Quorum 1. Recompile Mesos modules 2. Masters: rolling upgrade 3. Agents: rolling upgrade 4. Upgrade frameworks Leading Master Mesos Master Mesos Master Agent Agent Task Task 2016 Mesosphere, Inc. All Rights Reserved. 27

28 UPGRADE PROCEDURE 3. Agents: rolling upgrade 1. Stop the agent process 2. Remove old agent, install new 3. Start the new, upgraded agent 4. Wait for the agent to re-register and complete recovery: Finished recovery 5. Return to step #1 on another node 2016 Mesosphere, Inc. All Rights Reserved. 28

29 UPGRADE PROCEDURE Scheduler ZooKeeper Quorum 1. Recompile Mesos modules 2. Masters: rolling upgrade 3. Agents: rolling upgrade 4. Upgrade schedulers Leading Master Mesos Master Mesos Master Agent Agent Task Task 2016 Mesosphere, Inc. All Rights Reserved. 29

30 UPGRADE PROCEDURE 4. Upgrade frameworks 1. Stop scheduler 2. If needed, push new executor 3. Start upgraded framework 4. Allow framework to re-register with the master; this will trigger the reregistered() callback Frameworks should be built using the same libmesos as the new master and agent. Highly available frameworks may require a rolling upgrade. Frameworks will interact with masters and agents of two different versions during upgrade! 2016 Mesosphere, Inc. All Rights Reserved. 30

32 FURTHER ADVICE Upgrade the leading master last. If the leading master is upgraded first, a leader election will occur. Scheduler ZooKeeper Quorum Leading Master Mesos Master Mesos Master Agent Agent Task Task 2016 Mesosphere, Inc. All Rights Reserved. 32

33 FURTHER ADVICE Upgrade the leading master last. If the leading master is upgraded first, a leader election will occur. Scheduler ZooKeeper Quorum This implies a minimum of two leader elections during upgrade. Mesos Master Leading Master Mesos Master Upgrading the leader last permits an upgrade with just one election Agent Task Agent Task 2016 Mesosphere, Inc. All Rights Reserved. 33

34 FURTHER ADVICE Backup ZooKeeper state ZooKeeper is used for master leader election, and some frameworks may store state here Scheduler ZooKeeper Quorum A successful upgrade will not impact the ZooKeeper ensemble, but a backup provides insurance Leading Master Mesos Master Mesos Master Agent Agent Task Task 2016 Mesosphere, Inc. All Rights Reserved. 34

37 FURTHER ADVICE Don t apply configuration and binary changes to the cluster simultaneously. 1. If a config change is needed to avoid deprecated features, do this first 2. Upgrade binaries 3. Upgrade configurations to use new features if needed 2016 Mesosphere, Inc. All Rights Reserved. 37

38 POTENTIAL PITFALLS A flapping master or agent process may occur when they re run under an init system. The program repeatedly fails, while the init system keeps restarting Mesosphere, Inc. All Rights Reserved. 38

39 POTENTIAL PITFALLS A flapping master or agent process may occur when they re run under an init system. If the master/agent configuration isn t compatible with the new binary after upgrade, it will flap. The program repeatedly fails, while the init system keeps restarting. If the agent cannot reach the master, it will flap Mesosphere, Inc. All Rights Reserved. 39

40 VISIT OUR BOOTH Located at D1, near the main entrance FirstName Office Hours Tuesday 1-3pm Wednesday 4:40-5:15pm Learn more by visiting DCOS.io and Mesosphere.com 2016 Mesosphere, Inc. All Rights Reserved. 40

SECURING A MARATHON INSTALLATION 2016

MesosCon EU 2016 - Gastón Kleiman SECURING A MARATHON INSTALLATION 2016 2016 Mesosphere, Inc. All Rights Reserved. 1 Gastón Kleiman Distributed Systems Engineer Marathon/Mesos contributor gaston@mesosphere.io