Rolling upgrade Performance between OpenStack Deployed in VMs and Containers

Transcription

1 OpenStack Summit 2017 Boston Thursday, May 11 th 9:50am 10:30am Rolling upgrade Performance between OpenStack Deployed in VMs and Containers Lujin Luo Hieu LE Nam Nguyen Hoai 0

2 Who are we Lujin Luo Software Engineer at Fujitsu Ltd. IRC: lujinluo Hieu LE Software Engineer at Fujitsu Vietnam Ltd. IRC: hieulq Nam Nguyen Hoai Software Engineer at Fujitsu Vietnam Ltd. IRC: namnh 1

3 What is a Rolling Upgrade V1 V2 V1 V2 controller 2

4 What is a Rolling Upgrade Load Balancer V1 V2 V1 V2 V1 V2 Minimize service downtime controller controller controller 3

5 Why do Rolling Upgrades matter Cold upgrade Large Downtime Blue/Green and Canary upgrade model High hardware cost 4

6 What Rolling Upgrades are not Zero Downtime Upgrade [1] Zero Impact Upgrade [2] [1]. [2]. 5

7 What blocks Rolling Upgrades Incompatibilities RPC communication changes Database schema changes controller 6

8 How to implement Rolling Upgrades Functionalities RPC versioning Oslo.versionedobjects Database triggers Online schema/data migration 7

9 Project statuses Nova Neutron Cinder Glance Keystone Swift Under develop ment Ceilometer Ironic Heat Barbican Designate Etc Under develop ment Under develop ment 8

10 Architecture of VM-deployed OpenStack Load Balancer Public network Controller Node #1 Controller Node #2 Controller Node #3 Management and Data network Compute Nodes Network Nodes Storage nodes 9

11 Load Balancer Architecture of container-deployed OpenStack Provisioning node (Kolla) Public network Management and Data network Docker private registry Controller nodes Compute nodes Network nodes Storage nodes 10

12 Rolling Upgrades Test Scenarios Upgrade projects and order Keystone, Glance, Nova, Neutron, Cinder [1] Upgrade release stable/newton to stable/ocata [1] 11

13 Rolling Upgrades Test Scenarios Simulation scripts Asynchronous Continuous CRUD API requests Prepare OpenStack resources before upgrading Drain upgrading node from load balancer C C C C C C C C C C Load Balancer 0.3s Controller Node #1 Controller Node #2 Controller Node #3 12

14 Rolling Upgrades Test Scenarios Downtime estimation Add up multiple slots of downtime Failed request Successful request t N : time requests was responded from server R t 1 R t 2 R t 3 R t 4 R t 5 R t 6 R t 7 R t 8 R t 9 Δt 1 = t 5 t 2 Δt 2 = t 8 t 7 Downtime: Δt = Δt 1 + Δt 2 13

15 Rolling Upgrades Test Scenarios General steps to upgrade each project Expand database schema Set up database triggers (if necessary) Rolling upgrade control plane services Contract database schema (if necessary) Rolling upgrade data plane services (if necessary) NOTE: All above steps can be automated via tools like Ansible or bash scripts 14

16 Rolling Upgrades in VMs Test environment setup Install 5 projects Newton release distro Prepare Ocata release local repo Deploy an Ocata release DevStack for database migration Upgrade steps Follow each project s upgrade procedure 15

17 Rolling Upgrades in containers Option 1: $ Kolla-ansible upgrade command Problem: The ansible engine removes all old containers with the same name and bring up new containers with the same name controller controller controller 16

18 Rolling Upgrades in containers Option 2: In-place upgrade of containers Problem: Kolla-deployed containers are restricted with root privileges New package controller controller controller 17

19 Rolling Upgrades in containers Option 3: Stop N release containers deployed by Kolla Start O release containers by operators Problem 1: Manually started containers are lack of environment settings Problem 2: Kolla changed log containers from heka to fluentd 18

20 Rolling Upgrades in containers Test environment setup Step 1: Deploy Ocata release OpenStack with Kolla Step 2: Stop and rename the 5 projects Ocata release containers Step 3: Deploy Newton release OpenStack with Kolla 19

21 Rolling Upgrades in containers General steps to upgrade each project For each service in each node, stop Newton container and start Ocata container immediately Special tricks to do database migration (deploy another devstack environment and reconfig its db to Kolla cluster) 20

22 Rolling Upgrades in containers $ docker run normal container upgrade Create union R/W view Mount volumes Set configurations Start the container $ docker start our workarounds Start the container Very fast compared with container start time Start time depends on specific service running in container 21

23 Rolling Upgrades Results Downtime Keystone Glance Cinder Nova Neutron VMs (s) Containers (s)

24 Rolling Upgrades Analysis Full shutdown and restart of Control Plane in Nova Load Balancer Running Nova instances are never affected by upgrade Rolling upgrade compute nodes with Instances live migration 23

25 Rolling Upgrades Analysis Neutron-server zero downtime upgrade Oslo.versionedobject: under development Contract scripts are banned since Ocata release 24

26 Rolling Upgrades Analysis Cinder-volume active/passive HA Configure both active and passive cinder-volume to listen to the same queue Increase timeouts between cinder-api and cinder-volume 25

27 Rolling Upgrades Analysis Downtime difference between VMs and containers VMs upgrade time: upgrade/unpack packages and restart services Containers upgrade time: restart services 26

28 Rolling Upgrades Analysis Delay/latency of request responses Keystone Glance Neutron Cinder 27

29 Conclusions Upgrade Neutron, Cinder, Glance, Keystone with no downtime Rolling upgrades in Containers have less downtime 28

30 Remaining tasks in Rolling Upgrades Implement Rolling Upgrades in remaining projects Barbican, Designate, Trove, Manila, Sahara, etc Implement Rolling Upgrades support in Kolla Support Keystone, Neutron, Glance s native rolling upgrades support Blueprint 1: Apply service upgrade procedure Blueprint 2: Keystone rolling upgrade role Blueprint 3: Zero downtime upgrade 29

31 Future work Zero Downtime Upgrade Native zero downtime feature implemented in each project Using containers to do zero downtime upgrade OpenStack cluster Zero-downtime upgrade with Kolla 11:00am-11:40am Hynes Convention Center - Level 3 - Ballroom B Zero Impact Upgrade No response delay/latency Skip release Upgrade 30

32 Appendix Upgrade scripts and testbed information Special thanks Dang Van Dai Nova, Neutron, Cinder, Keystone, Glance communities on IRC 31

33