OpenStack Summit Tokyo Multi location implementation of OpenStack Hyuntae Park <hyuntae-park@gmo.jp> Team Manger / OpenStack Engineer Naoto Gohko <naoto-gohko@gmo.jp> IT Architect Enginner
# Agenda History of our services using OpenStack Overview of Multi Region Original extensions to OpenStack Multi Region supported Domain
# History of our services using OpenStack 2012-2013 - 2015 - Diablo Grizzly Juno V1 Havana IPv6 / GRE on Virtual Network V2 V3 V1,V2 Block Storage Object Storage
Running Infrastructure Physical Servers 25294 137223 1508 Physical Server Running VM Created VM
# Newly Released ConoHa Multi Region SSD Only Scalability API Simple and competitive pricing
# ConoHa has data centers in 3 Locations Tokyo Singapore Sanjose
# Description of Multi Region Share single KeyStone in multiple OpenStack environment Tokyo Region Keystone Keystone Singapore Region Keystone Neutron Glance Nova Neutron Glance Nova Compute node Compute node Compute node Compute node Block Storage(Cinder) object Storage (swift) Block Storage(Cinder) object Storage (swift)
# Specs OS and OpenStack Versions CentOS 7.1 x86_64 Juno (RDO) Maria DB Multi Region Setting Connect to Tokyo KeyStone from All regions. Add each region endpoints to Tokyo KeyStone. Did not need to modify OpenStack code.
# Multi Region Design 1 Singapore Tokyo SanJose Keystone API API Management Keystone API API Management Keystone API API Management User/tenant token Nova Cinder Neutron Glance Nova-compute Nova Cinder Neutron Glance Nova-compute Nova Cinder Neutron Glance Nova-compute
# Multi Region Design 2 Singapore Tokyo Sanjose Tokyo Singapole SanJose Keystone API Management API Keystone API Management API Keystone API Management API User/tenant User/tenant User/tenant User/tenant DB Replication User/tenant DB Replication User/tenant token DB Replication token DB Replication token Token Token Token Nova Nova-API Cinder Neutron Glance Nova-compute Nova Nova-API Cinder Neutron Glance Nova-compute Nova Nova-API Cinder Neutron Glance Nova-compute
# Issues and Restrictions on Multi Region User-registration is possible in Japan only VPN performance issue Issues on replicating token table.
# User-registration is possible in Japan only Singapore Do not create/delete users Tokyo Tokyo Our Customer base User administration SanJose Do not create/delete users Singapole R/W Keystone API API Management API Management Keystone API READ Keystone API API Management READ/WRITE Keystone API API Management API Management Keystone API READ R/W User/tenant DB Replication User/tenant DB Replication User/tenant Token Token Token
# VPN Performance issue MNGLine / STGLine 10Gbps VPN 10Mbps (measurement value)
# Bloat access tokens Too many tokens will be created from each components. VM Create! VM Create! VM Create! Nova Get/token Neutron Get/token KeystoneDB API Management Keystone API OpenStack Cluster Nova Neutron Glance Get/token Cinder Get/token Cinder Glance Nova user token:001 Neutron Token:002 Glance Token:003 Cinder Token:004 Nova user token:002 Neutron Token:003 Glance Token:004 Cinder Token:005 Nova user token:006 Neutron Token:007 Glance Token:008 Cinder Token:009
# Issues on replicating token table. 100 year expires token We fixed it so that any tokens can be used for each components. Setting example.conf [keystone_authtoken] token= 100 year expires token [neutron_authtoken] token= 100 year expires token [glance_authtoken] token= 100 year expires token [cinder_authtoken] token= 100 year expires token
# Now Feature Singapore Tokyo SanJose Keystone API API Management Keystone API API Management Keystone API API Management User/tenant User/tenant User/tenant User/tenant DB Replication User/tenant DB Replication User/tenant token DB Replication token DB Replication token Token Token Token Nova Cinder Nova Cinder Nova Cinder Neutron Glance Neutron Glance Neutron Glance Nova-compute Nova-compute Nova-compute
# Replication design Singapore Region MariaDB Galera Cluster Tokyo Region MariaDB Galera Cluster SanJose Region MariaDB Galera Cluster Keystone DB(M) Keystone DB(S) Keystone DB(M) Keystone DB(S) Keystone DB(M) Keystone DB(S) MariaDB Replication MariaDB Replication
# How to add new region Singapore Tokyo Region Tokyo Region SanJose Region New Region OpenStack Component Cluster OpenStack Component Cluster OpenStack Component Keystone Cluster Keystone API Keystone Keystone API Keystone Keystone API Keystone Keystone OpenStack Component Cluster User/tenant User/tenant User/tenant User/tenant Token Token Token Token OpenStack Component VM Template Neutron Nova Nova Neutron OpenStack Component VM Template Nova Neutron Nova Neutron Glance Cinder Glance Cinder Glance Cinder Glance Cinder
# Multi Region can provide... Tokyo Region Glance /var/lib/glance/images/{tenant-id}/example.qcow2 Sanjose Region Glance /var/lib/glance/images/{tenant-id}/example.qcow2 qcow2 image qcow2 image Symbolic Link http://glance.conoha.io/{uuid}/example.qcow2 Web Share Glance image download
# Feature plans Singapore Customer Base Create/Delete user Tokyo Customer Base Create/Delete user Sanjose Customer Base Create/Delete user Tokyo API Management Keystone Sync All Keystone Tokyo API Management Keystone Sync All Keystone Tokyo API Management Keystone KeystoneDB KeystoneDB KeystoneDB API Management Keystone API API Management Keystone API API Management Keystone API OpenStack Cluster User/tenant Nova-Compute on VM Token Personal VM OpenStack Cluster User/tenant Nova-Compute on VM Token Personal VM OpenStack Cluster User/tenant Nova-Compute on VM Token Personal VM GMO PaaS GMO PaaS GMO PaaS
21
22
23
24 Glance OpenStack Diablo on CentOS 6.x Shared codes Nova Keystone OpenStack Glizzly on Ubuntu 12.04 OpenStack Havana on CentOS 6.x Shared codes Nova network Quantam ovs + gre tunnel overlay Nova Keystone Glance Keystone Glance Cinder OpenStack Juno on CentOS 7.x Keystone Ceilometer Nova Baremetal compute Ceilometer Neutron Glance Neutron Nova Designate Keystone Glance Cinder Glance Cinder LBaaS Keystone Ceilometer Baremetal compute GSLB Nova Ceilometer Neutron Nova Neutron Ironic LBaaS LBaaS Shared cluster Swift cluster Swift Swift Swift Swift Swift Upgrade Juno
25 è
26
ConoHa: based on OpenStack Juno (IaaS) Multiple region openstack cluster Tokyo / Singapore / San Jose... and so on Full SSD storage Multiple keystone service domain support ConoHa and Next service (now in development)... OEM etc. LB as a Service: LVS-DSR (original) DNS as a service : OpenStack Designate OpenStack API and additional RESTful API Multiple Languages web panel support Japanese, ConoHa, English, Korean, Mandarin Chinese
29 è
30
31 è
32
33 è
34 Domain Product Prefix name space gnc ConoHa gnc zjp JP OEM- 1 zjp zsg SG OEM- 1 zsg...... OEM- n......
Adding domain(2nd): cloud.z.com We released 2nd service on same Juno infra. (2015/10/20 ) 35
36 ConoHa Dashboard gncu- vpsadmin ConoHa web panel API gncu72074485. ConoHa public API gncu72074485. z.com public API zjpu82073435. Geographic endpoints Keystone DB openstack original Endpoint (services_auth_url) Default domain (Admin only) ConoHa domain(gnc) (admin_auth_url) (user_auth_url) Admin & Internal Endpoint gnc domain Public Endpoint gnc domain user/tenant token (admin_auth_url) Admin & Internal endpoint Z.Com domain Z.Com domain (user_auth_url) Public endpint Z.Com domain Sync (region region) Openstack Component nova cinder neutron glance glance
37
38
39
40 è
41 è Example) Domain gnc : conoha.io [n-gohko@corpscan01~]$ dig +short plastic-machine.red ns ns-a3.conoha.io. ns-a1.conoha.io. ns-a2.conoha.io. [n-gohko@corpscan01~]$ dig +short plastic-machine.red soa ns-a1.conoha.io. postmaster.example.org. 1441328665 3600 600 86400 3600
42 è
43 è
44 è Public images (a) Domain Public images (b) è
45 è è
46 è è
47
48 è SaaS service endpoint (SMTP/POP/IMAP/DB/CPanel) Mail hosting service Database hosting service Web(Cpanel) hosting service Wordpress hosting service Domain specific: none Request for KeyStone Admin endpoint(ex. GET /v2.0/token/{tokenid}) need to access the Domain side endpoint
49
50
51
52 Geographic endpoints JP (auth_url) ConoHa keystonea pi SG (auth_url) ConoHa keystone api US (auth_url) ConoHa keystone api Keystone DB sync sync user/tenant user/tenant user/tenant token token token regiontyo1 OSA1 SIN1 SJC1 ConoHa IaaS SaaS IaaS SaaS IaaS SaaS IaaS SaaS
53 Geographic endpoints JP (auth_url) ConoHa keystonea pi JP OEM-1 (auth_url) JP OEM- 1 keystonea pi SG (auth_url) ConoHa keystone api US (auth_url) ConoHa keystone api Keystone DB sync sync user/tenant user/tenant user/tenant token token token regiontyo1 OSA1 SIN1 SJC1 IaaS SaaS JP OEM- 1 IaaS ConoHa SaaS IaaS SaaS IaaS SaaS
54 Geographic endpoints JP (auth_url) ConoHa keystonea pi JP OEM-1 (auth_url) JP OEM- 1 keystonea pi SaaS only OEM-2 (auth_url) SaaS only OEM- 2 keystonea pi SG (auth_url) ConoHa keystone api SaaS only OEM-2 (auth_url) SaaS only OEM- 2 keystonea pi US (auth_url) ConoHa keystone api SaaS only OEM-2 (auth_url) SaaS only OEM- 2 keystonea pi Keystone DB sync sync user/tenant user/tenant user/tenant token token token regiontyo1 OSA1 SIN1 SJC1 IaaS SaaS JP OEM- 1 IaaS SaaS ConoHa IaaS SaaS IaaS SaaS SaaS only OEM- 2 SaaS only OEM- 2 SaaS only OEM- 2 SaaS only OEM- 2
55 Geographic endpoints (auth_url) ConoHa keystonea pi JP (auth_url) z.com keystonea pi 1)token get 2)return token & Endpint list ConoHa client Keystone DB user/tenant token regiontyo1 OSA1 nova api db scheduler novncproxy conductor vps paas vps paas 3)req: create VM 4)return: vm created &generate VM ID
56 Geographic endpoints (auth_url) ConoHa keystonea pi JP (auth_url) Z.com keystonea pi ConoHa Client 1)token GET 2)return & エンドポイントリスト Z.com Client Keystone DB user/tenant token regiontyo1 nova api db scheduler novncproxy conductor vps paas nova api db scheduler novncproxy conductor vps OSA1 paas 3)VM 作成の要求 4)VM 作成 &VM の ID 発行
57 J
Fin.
T49 GMO Internet, Inc.
60 MARKET PLACE booth: T49 GMO Internet Inc.