The Road to Rolling Upgrade of Intel Private Cloud Huang, Shuquan May 2015
Agenda Intel Private Cloud Enterprise Upgrade Strategy Rehearsal Plan Roll back Plan Validation Plan Automation & local CI Detail Steps of Rolling Upgrade Result 2
OpenStack - Intel IT Convergence Platform Enterprise Hosting Lab Hosting New business Hosting OpenStack Existing Infrastructure (Proprietary Hypervisor & Storage) New Infrastructure (KVM, Open Source Storage) Open Stack provides a convergence opportunity for IT Hosting
Enterprise Upgrade Strategy Do we need to upgrade our existing Cluster? Track the related Bug & Blueprint List. How to perform a upgrade? Version by version other than skip version. Rolling upgrade. Upgrading each component of the cluster, piece by piece, eventually give us a cloud running on the new version Minimal downtime of each component What s the right timing of upgrade? Based on monitor data, find out a proper timing Long term upgrade strategy 4
Rehearsal Plan VM environment Come up with local repo and config file changes Can t use VLAN mode in VM environment Beta environment Validate the local repo and config file changes Refine the upgrade procedure in similar topology & architecture There are still some mismatch between beta and production, such as DB, Storage, etc. Production environment 5
Rollback Plan Ability to rollback to a pre-upgrade state if things fail Backup DB Config file Repo Restore 6
Validation Plan Congress?? Don t gather all tables. Keystone, for example. Data is stored in memory. Checking result is print as log which is not friendly to read. Script to collect the status of the Cluster from API and diff the result after performing a upgrade. 7
Automation & Local CI Confi file converter Input: previous version config file Output: new version config file Automation upgrade multiply nodes by leveraging local CI Local Git Repo Polling changes Jenkins Puppet Master After deployment, run Tempest to validate changes Deploy new cluster by using some version source code and cluster topology OpenStack Cluster Manual Trigger CI or Configure polling stratigies Admin 8
Detail Steps of Rolling Upgrade Preparation Backup config files, DB Build Icehouse local repo, including source code, pip packages, config file template Edit the /etc/mysql/my.cnf file, change MySQL default character set to UTF-8. Stop puppet services in Cluster Check haproxy status. Select Controller01 to upgrade and keep Controller02 running. Upgrade keystone, glance, nova in Controller01 Upgrade nova, neutron, cinder in compute/network node Stop corresponding services on Controller02, and start it on Controller01 9
keystone 1. Edit the /etc/keystone/keystone.conf file for compatibility for Icehouse: Add the [database] section. Move the connection key from the [sql] section to the [database] section. bind_host = 0.0.0.0 change to public_bind_host=0.0.0.0 & admin_bind_host=0.0.0.0 2. Stop the services: # service keystone stop 3. Uninstall the Havana package & code # pip uninstall y keystone pythonkeystoneclient 4. Install the Icehouse package 5. Upgrade the database: # keystone-manage token_flush # keystone-manage db_sync 6. Start the services. # service keystone start 10
glance 1. Before upgrading the Image Service database, you must convert the character set for each table to UTF-8. 2. Edit the /etc/glance/glance-api.conf and /etc/glance/glance-registry.conf files for compatibility with Icehouse: Add the [database] section. Rename the sql_connection key to connection and move it to the [database] section 3. Edit the /etc/glance/glance-api.conf sql_idle_timeout = 3600 -> idle_timeout = 3600 notifier_strategy = noop -> notification_driver = noop rabbit_ha_queues = true -> comment out In the /etc/glance/glance-api.conf file, add RabbitMQ message broker keys to the [DEFAULT] section. [DEFAULT]... rpc_backend = rabbit rabbit_host = controller rabbit_password = RABBIT_PASS... Replace RABBIT_PASS with the password you chose for the guest account in RabbitMQ. 4. Stop the services: # service glance-api stop # service glance-registry stop 5. Uninstall the Havana package & code # pip uninstall y glance python-glanceclient 6. Install the Icehouse package 7. Upgrade the database: # glance-manage db_sync 8. Start the services: # service glance-api start # service glance-registry start 11
nova services in controller 1. Edit the /etc/nova/nova.conf file rpc_backend = nova.rpc.impl_kombu -> rpc_backend = rabbit. libvirt_inject_password = false -> inject_password = false libvirt_inject_key=false -> inject_key -> false libvirt_inject_partition=-2 -> inject_partition = -2 [upgrade_levels] compute=icehouse-compat 2. Edit the /etc/nova/api-paste.ini file and comment out or remove any keys in the [filter:authtoken] section beneath the paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory statement. 6. Upgrade the database: # nova-manage db sync 7. Start the services: # service nova-api start # service nova-scheduler start # service nova-conductor start # service nova-cert start # service nova-consoleauth start # service nova-novncproxy start 3. Stop the services: # service nova-api stop # service nova-scheduler stop # service nova-conductor stop # service nova-cert stop # service nova-consoleauth stop # service nova-novncproxy stop 4. Uninstall the Havana package & code # pip uninstall y nova python-novaclient 5. Install the Icehouse package 12
neutron-server 1. Before upgrading the Networking database, you must convert the character set for each table to UTF-8. 2. Populate the /etc/neutron/plugins/ml2/ml2_conf.ini file with the equivalent configuration for your environment. Note: Do not edit the /etc/neutron/neutron.conf file until after the conversion steps. Warning: Because the conversion script cannot roll back, you must perform a database backup prior to executing the following commands. 3. Stop the service: # service neutron-server stop 4. Uninstall the Havana package & code # pip uninstall y neutron python-neutronclient 5. Install the Icehouse package 6. Keep the Havana configuration in the /etc/neutron. Make sure the correct setting of ml2_conf.ini already exist in the /etc/neutron/plugins/ml2. 7. Upgrade the database: # neutron-db-manage --config-file /etc/neutron/neutron.conf \ --config-file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini stamp Havana Note: make sure the file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini existing. service_plugins = neutron.services.l3_router.l3_router_plugin.l3routerplugin # neutron-db-manage --config-file /etc/neutron/neutron.conf \ --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade icehouse 8. Perform the conversion from OVS to ML2, replace NEUTRON_DBPASS with the password you chose for the database.: # python -m neutron.db.migration.migrate_to_ml2 openvswitch \ mysql://neutron:neutron_dbpass@controller/neutron 9. Edit the /etc/neutron/neutron.conf file to use the ML2 plug-in and enable network change notifications: [DEFAULT]... core_plugin = ml2 service_plugins = router... notify_nova_on_port_status_changes = True notify_nova_on_port_data_changes = True nova_url = http://controller:8774/v2 nova_admin_username = nova:q nova_admin_tenant_id = SERVICE_TENANT_ID nova_admin_password = NOVA_PASS nova_admin_auth_url = http://controller:35357/v2.0 Note: Replace SERVICE_TENANT_ID with the service tenant identifier (id) in the Identity service and NOVA_PASS with the password you chose for the nova user in the Identity service. 10. Edit the /etc/default/neutron-server to use ML2 plugin NEUTRON_PLUGIN_CONFIG="/etc/neutron/plugins/ml2/ml2_conf.ini" 11. Start Networking services: # service neutron-server start 13
neutron-*-agents 1. Restart Networking services: # service neutron-dhcp-agent stop # service neutron-l3-agent stop # service neutron-metadata-agent stop # service neutron-plugin-openvswitch-agent stop 2. Uninstall the Havana package & code # pip uninstall y neutron pythonneutronclient 3. Install the Icehouse package 4. Edit the /etc/neutron/neutron.conf file to use the ML2 plug-in: [DEFAULT] core_plugin = ml2 service_plugins = router 5. Populate the /etc/neutron/plugins/ml2/ml2_conf.ini file with the equivalent configuration for your environment. 6. Clean the active OVS configuration: Create the /etc/init/neutron-ovs-cleanup.conf # ln -sv /lib/init/upstart-job /etc/init.d/ neutron-ovscleanup # service neutron-ovs-cleanup restart 7. Edit /etc/init/neutron-plugin-openvswitchagent.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.in 8. Start Networking services: # service neutron-dhcp-agent start # service neutron-l3-agent start # service neutron-metadata-agent start # service neutron-plugin-openvswitch-agent start 14
neutron-plugin-openvswitch-agent 1. Restart Networking services: # service neutron-plugin-openvswitch-agent stop 2. Uninstall the Havana package & code # pip uninstall y neutron python-neutronclient 3. Install the Icehouse package 4. Edit the /etc/neutron/neutron.conf file to use the ML2 plug-in: [DEFAULT] core_plugin = ml2 service_plugins = router 5. Populate the /etc/neutron/plugins/ml2/ml2_conf.ini file with the equivalent configuration for your environment. 6. Clean the active OVS configuration: 7. Create the /etc/init/neutron-ovs-cleanup.conf # ln -sv /lib/init/upstart-job /etc/init.d/ neutron-ovscleanup # service neutron-ovs-cleanup restart 8. Edit /etc/init/neutron-plugin-openvswitchagent.conf 9. --configfile=/etc/neutron/plugins/ml2/ml2_conf.in 10. Restart Networking services: # service neutron-plugin-openvswitch-agent start 15
nova-compute 1. Edit the /etc/nova/nova.conf file rpc_backend = nova.rpc.impl_kombu -> rpc_backend = rabbit. libvirt_inject_password = false -> inject_password = false libvirt_inject_key=false -> inject_key -> false libvirt_inject_partition=-2 -> inject_partition = -2 2. Edit the /etc/nova/api-paste.ini file and comment out or remove any keys in the [filter:authtoken] section beneath the paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory statement. 3. Stop the services: # service nova-compute stop 4. Uninstall the Havana package & code # pip uninstall y nova python-novaclient 6. Start the services: # service nova-compute start 7. Remove the compatible mode of compute service on the controller node Edit the /etc/nova/nova.conf file [upgrade_levels] #compute=icehouse-compat 5. Install the Icehouse package 16
cinder 1. Stop the services: # service cinder-api stop # service cinder-schedule stop 2. Uninstall the Havana package & code # pip uninstall y cinder python- cinder client 3. Install the Icehouse package 4. Upgrade the DB # cinder-manage db sync 5. Start the services: # service cinder-api start # service cinder-schedule start 1. Stop the services: # service cinder-volume stop 2. Uninstall the Havana package & code # pip uninstall y cinder python- cinder client 3. Install the Icehouse package 4. Start the services: # service cinder-volume start 17
Result Issue Convert glance and neutron DB to utf8 in Mysql Galera Cluster conversion from OVS to ML2 (sequences matters) DHCP tap port is tagged to 4095 VM s tap device is lost in br-int 18
Disclaimer Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at [intel.com]. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Include as footnote in appropriate slide where performance data is shown: o Configurations: [describe config + what test used + who did testing] o For more information go to http://www.intel.com/performance. Intel, the Intel logo, {List the Intel trademarks in your document} are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. 2015 Intel Corporation. 19