OpenStack Cluster Zero-Downtime Upgrade ft. Kolla

Transcription

1 Copyright 2017 Fujitsu Vietnam Limited OpenStack Cluster Zero-Downtime Upgrade ft. Kolla 2017 May 11 Duong Ha-Quang and Hieu LE Fujitsu Vietnam Limited

2 Who are we? 1 Copyright 2017 Fujitsu Vietnam Limited Duong Ha-Quang Software Engineer at Fujitsu Vietnam Core reviewer of Kolla duonghq@vn.fujitsu.com IRC: duonghq Hieu LE Software Engineer at Fujitsu Vietnam Official Vietnam OpenStack UG organizer hieulq@vn.fujitsu.com IRC: hieulq

3 2 Copyright 2017 Fujitsu Vietnam Limited Introduction Some reviews and thought about zero-downtime upgrade for OpenStack services. Ideas in this presentation are just concept. PoC in Kolla.

4 Agenda 3 Copyright 2017 Fujitsu Vietnam Limited 1. OpenStack Upgrading overview OpenStack upgrade assertion tag OpenStack rolling upgrade requirements 2. From minimal to zero 3. Zero downtime upgrade proposal in Kolla Kolla support for configuration management Kolla support for OSM Proposal/Demo

5 4 Copyright 2017 Fujitsu Vietnam Limited OpenStack Upgrading One of the most demand feature for every system Cold upgrade is easy Service-level agreement is more strictly nowadays, we need decreasing downtime of the system. Source:

6 5 Copyright 2017 Fujitsu Vietnam Limited Review of current upgrading model BlueGreen deployment and Canary release Rolling upgrade

7 BlueGreen deployment 6 Copyright 2017 Fujitsu Vietnam Limited Old Router User(s) New

8 Canary release 7 Copyright 2017 Fujitsu Vietnam Limited Old Router User(s) New

9 8 Copyright 2017 Fujitsu Vietnam Limited Rolling upgrade Eliminates the need to restart all services on new code simultaneously. Requires mixed-version services work together properly in mid-upgrade. May have downtime of some services at a time. X + 1 A(X) X + 1 B(X) C(X)

10 OpenStack rolling upgrade requirements 1. Online Schema Migration (OSM) 2. Maintenance Mode 3. Live Migration 4. Multi-version Interoperability 5. Graceful Shutdown 6. Upgrade Orchestration 7. Upgrade Gating 8. Project Tagging Copyright 2017 Fujitsu Vietnam Limited

11 OpenStack upgrade assertion tag 10 Copyright 2017 Fujitsu Vietnam Limited TC of OpenStack defines five upgrade-related tags: 1. assert:supports-upgrade 2. assert:supports-accessible-upgrade 3. assert:supports-rolling-upgrade 4. assert:supports-zero-downtime-upgrade 5. assert:supports-zero-impact-upgrade

12 11 Copyright 2017 Fujitsu Vietnam Limited From minimal downtime to zero-downtime upgrade Zero-downtime upgrade requires to enhance: Configuration management (CM) New config Deprecated config Removed config RPC pinning RPC Database migration (OSM) Old version New version Service Upgrading database Icons made by Freepik from is licensed by CC 3.0 BY Icons made by Google from is licensed by CC 3.0 BY Icons made by Madebyoliver from is licensed by CC 3.0 BY

13 Two main approaches for OSM Trigger-based E.g. Keystone, Glance Other: Facebook [1] Triggerless E.g. Neutron Binary log-based [2] [1] / [2] 12 Copyright 2017 Fujitsu Vietnam Limited

14 Zero downtime upgrade proposal 13 Copyright 2017 Fujitsu Vietnam Limited Database online schema migration: 2 candidate solutions 1. Buffer requests in upgrade period. 2. Utilize checkpoint/snapshot and binary log of database.

15 Zero downtime upgrade proposal (1) 14 Copyright 2017 Fujitsu Vietnam Limited Buffering HTTP and RPC requests in upgrade period. Step 0 Step 1 Step 2 HA/LB stack HA/LB stack HA/LB stack Requests buffer x Requests buffer Service (ver X) Service (X X + 1) Service (X + 1) Database (ver X) Database (X X + 1) Database (X + 1) Upgrading Requests are resent with original order

16 15 Copyright 2017 Fujitsu Vietnam Limited Zero downtime upgrade proposal (1) Buffering requests in upgrade period There are 2 request types need buffering: RESTful HTTP requests from user and inter-projects. Internal service RPC requests (through MQ) Requests must be put in buffer in received order for replay correctly (best with timestamp)

17 16 Copyright 2017 Fujitsu Vietnam Limited Zero downtime upgrade proposal (1) Buffering requests in upgrade period Pros: From user s POV: no service perceivable downtime if migration time is short enough. Cons: From user s POV: system is lag when requests are queued in buffer. If migration time is long (mainly in database migration), some requests can timeout [1]. Buffer can be very large if database migration time is long. Next idea [1]

18 17 Copyright 2017 Fujitsu Vietnam Limited Zero downtime upgrade proposal (2) Utilize checkpoint/snapshot and binary log of database. Database Create checkpoint Turn on binary log New data (Y) Before Checkpoint (X) Migrate to next release Recorded by binary log X in new schema Shutdown database-related service New data (Y) in old schema X in new schema Migrate using binary log Database in new schema Start new version and bring up system Turn off binary log downtime Can be eliminated with previous approach

19 Zero downtime upgrade proposal (2) Utilize checkpoint/snapshot and binary log of database. Pros: Internal system downtime is much smaller than previous approach (only downtime for delta change vs whole database). Cons: From user s POV: there is a short downtime. Implementation is more complicated than previous approach. 18 Copyright 2017 Fujitsu Vietnam Limited

20 Zero downtime upgrade proposal 19 Copyright 2017 Fujitsu Vietnam Limited Two candidate methods can be combined to get advantage from both methods: - Use 2 nd approach but add buffer layer and turn on when database is shutdown Zero downtime from user s POV, only a bit lag. Create checkpoint, binary log Migrate current db to new schema Turn on buffer Migrate new data to new database Turn off binary log Finish Idea 1 Idea 2 Database is turned off

21 20 Copyright 2017 Fujitsu Vietnam Limited PoC in Kolla Kolla s mission is to provide production-ready containers and deployment tools for operating OpenStack clouds. Three official deliverables: kolla(-image) -> Docker images kolla-ansible -> Deploy OpenStack using Ansible kolla-kubernetes -> Deploy OpenStack inside k8s cluster

22 Kolla support for configuration management 21 Copyright 2017 Fujitsu Vietnam Limited Kolla-Ansible has implemented mechanism for configuration management functions: Configurations overridden [1] Kolla-Kubernetes posed good potential to automated CM [1]

23 22 Copyright 2017 Fujitsu Vietnam Limited Kolla support for OSM (1) For OpenStack projects had OSM native-supported Patches for Neutron and Keystone OSM are in progress

24 Kolla support for OSM (2) 23 Copyright 2017 Fujitsu Vietnam Limited For OSM unsupported project Implement above ideas at HA/LB layer. Request buffer: Intermission/OpenResty

25 24 Copyright 2017 Fujitsu Vietnam Limited PoC scenario for 1 st approach PoC for HTTP requests buffering Intermission/OpenResty configuration proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; Intermission is bound to VIP:5000 and VIP:35357, HAProxy > 5050, > Scenario: Continuously send 200 create and delete network request to Ocata cluster Upgrade Neutron to master code-based while requests are sending.

26 Demonstration 25 Copyright 2017 Fujitsu Vietnam Limited Used scripts: Rolling upgrade with Kolla, we have downtime here PoC buffer request with Kolla

27