How is GMO Internet using OpenStack for Public Cloud. OpenStack Days Taiwan Jul 12, 2016 Hironobu Saitoh GMO Internet, Inc.

Size: px

Start display at page:

Download "How is GMO Internet using OpenStack for Public Cloud. OpenStack Days Taiwan Jul 12, 2016 Hironobu Saitoh GMO Internet, Inc."

Mervin Francis
6 years ago
Views:

1 How is GMO Internet using OpenStack for Public Cloud OpenStack Days Taiwan Jul 12, 2016 Hironobu Saitoh GMO Internet, Inc.

2 Agenda About GMO Internet Public Clouds Why OpenStack? Technical Background

Company Overview Founded 1991 Based in Tokyo, Japan 4600+ Employees in

37 Million Customers Business Segment Domain registration, Hosting,

3 Company Overview Founded 1991 Based in Tokyo, Japan Employees in over 9 countries 84 Group Companies 7.37 Million Customers Business Segment Domain registration, Hosting, Online advertising, Internet Securities Japan OpenStack UsersGroup(JOSUG) board member sine 2014

4 Public Clouds We are offering 3 public clouds built on OpenStack Since 2015 Version: Juno Since 2014 Version: Juno, Havana Since 2013 Version: Havana, Grizzly

5 Public Clouds 3 Public Clouds (built on OpenStack) 1,400+ nodes 3 availability regions (JP, US, SG) (GMO AppsCloud also has Korean region) Over 15,000+ customers Use version Diablo, Grizzly, Havana and Juno

6 Features Servers Virtual Machine Managed Mail Managed MySQL Storages Networking Auto Backup Scale up/down VM Snapshot Block Storage Private Network Object Store IPv4 ISO Image Management Communicating Different Regions Load balancer Free DNS GeoDNS(GeoRouting) A P I Global Network(IPv4/IPv6 dual stack) Region Japan(Tokyo) Singapore The United States(San Jose)

7 Components Service Usage Region Keystone Account management JP, SG, US Nova Virtual machine JP, SG, US Neutron Private networking Floating IP JP, SG, US Cinder Block storage JP, SG, US Swift Object store JP Glance VM-Image Auto Backup JP, SG, US Designate Free DNS, GeoRouting JP Ceilometer Collect user statics JP Heat Initialize VM JP, SG, US Horizon For staff use only JP

8 Providing OpenStack APIs Our customers may use OpenStack API on all regions. Public Clouds Tools and SDKs API CyberDuck gophercloud

9 Why OpenStack? Large set of services Services for building the public cloud were already implemented Modular architecture goes well with our team Develop components of public cloud in tandem Open Source Software Developed specific implementation of our service by ourselves.

10 Tools ConoHa Vagrant provider Vagrant 1.6+ plugin. Provisioning machines within ConoHa. novassh A client program for OpenStack Nova. Inspired by ec2ssh.

11 Architecture of our public clouds OpenStack Days Taiwan Jul 12, 2016 Naoto Gohko GMO Internet, Inc.

12 Public Clouds We are offering multiple public cloud services.

13 Running Infrastructure Physical (2015/10) Servers Physical Server Running VM Created VM

14 OpenStack service development team

15 Cloud service development team: Now(2016) Cloud service development team: (abount 30 people) OpenStack Neutron team: 4 people Neutron driver / modification / engineering Cloud API development team: 5 people Public API validation program OpenStack modification / scaduler programing / keystone Cloud Infra. development team: 11 people Security engineering / glance driver / cinder driver / nova additional extensions / construction of OpenStack infra. Applicatoin cloud service development team: 5 people Billing engineering / staff tools / GMO AppsCloud web GUI Additional engineering team: many people (30 ~) QA Team / Server Engineering Team / GUI development Team Network Engineering Team / SaaS development Team CRM backend and billing Team

16 Cloud service development team: Office(2016) #1 Neutron Team And Cloud API Team Cloud Infra. Team And AppsCloud Team

17 Cloud service development team: Office(2016) #2 Neutron Team And Cloud API Team Cloud Infra. Team And AppsCloud Team

18 Limied number of people. But, we have to run a lot of OpenStack service clusters.

19 Service developmemt history by OpenStack

GMO Internet, Inc.: VPS and Cloud services Onamae.com VPS (2012/03) : http://www.onamae-server.com/ Forcus: global IPs; provided by simple "nova-network" tenten

20 GMO Internet, Inc.: VPS and Cloud services Onamae.com VPS (2012/03) : Forcus: global IPs; provided by simple "nova-network" tenten VPS (2012/12) Share of OSS by Group companies in Vietnam ConoHa VPS (2013/07) : Forcus: Quantam(Neutron) overlay tenant network GMO AppsCloud (2014/04) : OpenStack Havana based 1 st region Enterprise grade IaaS with block storage, object storage, LBaaS and baremetal compute was provided Onamae.com Cloud (2014/11) Forcus: Low price VM instances, baremetal compute and object storage ConoHa Cloud (2015/05/18) OpenStack Diablo on CentOS 6.x Forcus: ML2 vxlan overlay, LBaaS, block storage, DNSaaS(Designate) and original services by keystone auth GMO AppsCloud (2015/09/27) : 2 nd region by OpenStack Juno based Shared codes Nova Keystone Glance OpenStack Glizzly on Ubuntu OpenStack Havana on CentOS 6.x Enterprise grade IaaS with High IOPS Ironic Compute and Neutron LBaaS Shared codes Nova network Quantam ovs + gre tunnel overlay Nova Keystone Glance Keystone Glance Cinder OpenStack Juno on CentOS 7.x Keystone Ceilometer Nova Baremetal compute Ceilometer Neutron Glance Neutron Nova Designate Keystone Glance Cinder Glance Cinder LBaaS Keystone Ceilometer Baremetal compute GSLB Nova Ceilometer Neutron Nova Neutron Ironic LBaaS LBaaS Shared cluster Swift cluster Swift Swift Swift Swift Swift Upgrade Juno

21 OpenStack Swift: shared cluster

Swift Hardware: Object nodes Boot: SSD x2 HDD: 4TB x12 E3-1230 v3 @ 3.

22 Swift Hardware: Object nodes Boot: SSD x2 HDD: 4TB x12 E GHz Memory 16GB 10GbE x2 (SFP+) (Intel NIC) ASUSTeK COMPUTER INC. RS300-H8-PS12

23 Hardware: LVS-DSR and reverse-proxy(layer7) nodes Boot: SSD x2 E GHz Memory 16GB 10GbE NIC x1 (Intel NIC) Supermicro microblade 8 blade nodes type

24 Hardware: swift-proxy nodes Boot: HDD x6 (1.7TB) Ceilometer Log disk (Swift all request billing data) 2.40GHz x2 CPU Memory 64GB NIC: 10GbE SFP+ x2(intel NIC) System x3550 M3 (old IBM) Hardware: account/container-server nodes Boot: HDD x2 Account/Container storage: SSD x2 2.40GHz x2 CPU Memory 64GB NIC: 10GbE SFP+ x2(intel NIC) System x3550 M3 (old IBM)

25 Swift cluster (Havana to Juno upgrade) SSD storage: container/account server at every zone

26 OpenStack Swift cluster (5 zones, 3 copy) Xeon E GHz Memory 16GB LVS-DSR LVS-DSR Xeon E GHz Memory 16GB HAProxy(SSL) HAProxy(SSL) Xeon E GHz x 2CPU Memory 64GB keystone keystone swift proxy swift proxy Xeon E GHz x 2CPU Memory 64GB, SSD x 2 swift account swift container Xeon E GHz x 2CPU Memory 64GB, SSD x 2 swift account swift container Xeon E GHz x 2CPU Memory 64GB, SSD x 2 swift account swift container Xeon E GHz x 2CPU Memory 64GB, SSD x 2 swift account swift container Xeon E GHz x 2CPU Memory 64GB, SSD x 2 swift account swift container swift objects swift objects swift objects swift objects swift objects Xeon E GHz swift objects Xeon E GHz swift objects Xeon E GHz swift objects Xeon E GHz swift objects Xeon E GHz swift objects

Swift cluster: multi-auth and multi-endpoint Juno Z.

keystone Havana AppsCloud swift proxy keystone Grizzly ConoHa

swift container swift container swift container swift container

account Juno swift objects swift objects swift objects swift

27 Swift cluster: multi-auth and multi-endpoint Juno Z.com Juno AppsCloud swift proxy keystone Juno ConoHa swift proxy keystone Havana AppsCloud swift proxy keystone Grizzly ConoHa swift proxy keystone swift proxy keystone Havana swift container swift container swift container swift container swift container To swift account swift account swift account swift account swift account Juno swift objects swift objects swift objects swift objects swift objects swift objects swift objects swift objects swift objects swift objects

28 Swift shared cluster: ex) the Internet keystone ConoHa VPS cluster (Grizzly) keystone LVS-DSR LVS-DSR reverse-proxy reverse-proxy AppsCloud cluster (Havana) keystone Swift-Proxy Swift-Proxy AppsCloud cluster (Juno) [filter:keys tone] ## res eller_prefix = [filter:keys tone] res eller_prefix = [filter:keys tone] res eller_prefix = gac_ [filter:keys tone] ## res eller_prefix = [filter:keys tone] res eller_prefix = [filter:keys tone] res eller_prefix = gac_ keystone ConoHa Cloud cluster (Juno) [filter:keys tone] res eller_prefix = nc_ [filter:ceilometer] res eller_prefix = zc_ [filter:keys tone] res eller_prefix = zc_ [filter:ceilometer] res eller_prefix = zc_ [filter:keys tone] res eller_prefix = nc_ [filter:ceilometer] res eller_prefix = zc_ [filter:keys tone] res eller_prefix = zc_ [filter:ceilometer] res eller_prefix = zc_ keystone Z.com Cloud (Juno) << account-s erver << container-s erver << object-s erver ( 2 x 5 nodes )

29 OpenStack history of computing environment

30 OpenStack service: Onamae.com VPS(Diablo) Oname.com VPS(Diablo) Service XaaS model: VPS (KVM, libvirt) Network: 1Gbps Network model: Flat-VLAN (Nova Network), without floting IP(no L3) IPv4 only Public API None (only web-panel) Glance Public image only.

overlay IPv6/IPv4 dualstack Network: 10GE wired(10gbase-t) Public API: None

31 OpenStack service: ConoHa(Grizzly, 2013/07) ConoHa(Grizzly) Service XaaS model: VPS + Private networks (KVM + ovs) Network model: Flat-VLAN + Quantam ovs-gre overlay IPv6/IPv4 dualstack Network: 10GE wired(10gbase-t) Public API: None (only web) Glance Only Public image Cinder: None ObjectStorage Swift (After Havana)

But When the scale becomes large, Localization occurs to a specific node of the

32 OpenStack service: ConoHa(Grizzly) Grizzly Quantam Network: It was using the initial version of the Open vswitch full mesh GREvlan overlay network with LinuxBridge Hybrid But When the scale becomes large, Localization occurs to a specific node of the communication of the GRE-mesh-tunnel (with under cloud network(l2) problems) (Broadcast storm?)

OpenStack service: GMO AppsCloud(Havana) Service XaaS model: KVM compute + Private VLAN networks + Cinder + Swift Network: 10Gbps wired(10gbase SFP+) Network model: IPv4 Flat-VLAN + Neutron

33 OpenStack service: GMO AppsCloud(Havana) Service XaaS model: KVM compute + Private VLAN networks + Cinder + Swift Network: 10Gbps wired(10gbase SFP+) Network model: IPv4 Flat-VLAN + Neutron LinuxBridge(not ML2) + Cisco Nexsu L2 sw/port driver Brocade ADX L4-LBaaS original driver Public API Provided the public API Ceilometer (Billing) Glance : Provided(GlusterFS) Cinder : HP 3PAR(Active-Active Multipath original) + NetApp ObjectStorage : Swift cluster Bare-Metal Compute Modifiyed cobbler bare-metal deploy driver Cisco Nexsus switch bare-metal networking driver (L2 tenant NW)

34 OpenStack service: GMO AppsCloud model Neutron LinuxBridge model(very Fast, simple is Best) this cloud is optimized services for the GAME server. public network Vlan network Vlan network NIC vlan NIC vlan NIC vlan NIC vlan bridge bridge bridge bridge tap tap vnic vm vnic compute compute

35 Cisco Nexsus L2 sw/port manage driver(self made) L2 resource is limited / SW CPU MAC ADDRESS VLAN per Network VLAN per Port Cis co Nexs us Fabric Nexus 5k s (VPC) SW Manage NW Allowed VLAN to trunked port is allowed only VLAN to be used in LinuxBridge in VM/Baremetal Compute node. Baremetal : link aggregation port Port discovery using by lldp Switch/Port API server Nexus 2k: FEX s (dual homed) Cisco Nexsus NX-OS Server: LACP : port-channel Active-Active link aggreration Compute node Baremetal Compute node Fully redundant s erver (Act-Act link aggreration) OpenStack Manage NW

36 Nova-baremetal(havana)/Ironic(juno) ansible Baremetal networking Bonding NIC + lldp discovery Taged VLAN allowd VLAN + dhcp native VLAN

37 GMO AppsCloud(Havana/Juno)

38 Public API security and load balance: LVS-DSR L7 reverse-proxy API validation wrapper

39 public API Endpoint L7:reverse proxy Swift Proxy OpenStack API OpenStack API for input validation Web panel(httpd, php) API wrapper proxy (httpd, php Framework: fuel php) Customer sys API Customer DB Ceilometer API Keystone API Nova API Neutron API Glance API Cinder API

40 public API: step 1, step 2) Public API global network s tep 1) s tep 2) the Cloud (Internet) VMx2 LVS heatbeat LVS heatbeat elvs 01 elvs 02 LVS-DSR (act-s tby) api-revers e-proxy01 HAProxy ext-api-wrapper01 php + httpd - keystone - nova - cinder - neutron - glance - account api-revers e-proxy02 HAProxy step 3) step 1) LVS-DSR (L4) is received https(tcp/443) packet, then forward api-reverse-proxy real IP s. VMx2 ext-api-wrapper02 OpenStack Management network php + httpd - keystone - nova - cinder - neutron - glance - account step 4) control-nodes 01 - keystone API - nova API - cinder API - neutron API - glance API control-nodes 02 step 2) HAProxy has valid API ACL and backend server configurations. IF HAProxy allowed POST /v2.0/tokens, then the request call to ext-api-wrapper0[12]. - keystone API - nova API - cinder API - neutron API - glance API

41 public API: step 3), step 4) Public API global network s tep 1) s tep 2) the Cloud (Internet) VMx2 LVS heatbeat LVS heatbeat elvs 01 elvs 02 LVS-DSR (act-s tby) api-revers e-proxy01 HAProxy ext-api-wrapper01 php + httpd - keystone - nova - cinder - neutron - glance - account api-revers e-proxy02 HAProxy step 3) VMx2 ext-api-wrapper02 OpenStack Management network php + httpd - keystone - nova - cinder - neutron - glance - account step 4) step 3) ext-api-wrapper0 [12], it is a php program. request URI and header, and the input value of json of the body was confirmed by php, and then call the real OpenStack API as the next processing. step 4) OpenStack API that is checked the input value will be run. control-nodes 01 - keystone API - nova API - cinder API - neutron API - glance API control-nodes 02 - keystone API - nova API - cinder API - neutron API - glance API

42 OpenStack Juno cluster: ConoHa (Juno) and Z.com cloud AppsCloud (Juno)

43 # ConoHa has data centers in 3 Locations Tokyo Singapore Sanjose

Management Keystone API READ Keystone API API Management READ/WRITE Keystone API API Management API

44 # User-registration is possible in Japan only Singapore Do not create/delete users Tokyo Tokyo Our Customer base User administration SanJose Do not create/delete users Singapole R/W Keystone API API Management API Management Keystone API READ Keystone API API Management READ/WRITE Keystone API API Management API Management Keystone API READ R/W User/tenant DB Replication User/tenant DB Replication User/tenant Token Token Token

45 What s the problems abount Multi-Cluster? The cost to operate Multi version Openstack have increased It is difficult to upgrade or add new features Managing/operating multiple sites of OpenStack is a headache.

47 OpenStack Juno: 2 service cluster, released Mikumo = 美雲 = Beautiful cloud Mikumo ConoHa New Juno region released: 10/26/2015 Mikumo Anzu

48 OpenStack Juno: 2 service cluster, released Service model: Public cloud by KVM Network: 10Gbps wired(10gbase SFP+) Network model: Flat-VLAN + Neutron ML2 ovs-vxlan overlay + ML2 LinuxBridge(SaaS only) IPv6/IPv4 dualstack LBaaS: LVS-DSR(original) Public API Provided the public API (v2 Domain) Compute node: ALL SSD for booting OS Without Cinder boot Glance: provided Cinder: SSD NexentaStore zfs (SDS) Swift (shared Juno cluster) Cobbler deply on under-cloud Ansible configuration SaaS original service with keystone auth , web, CPanel and WordPress Service model: Public cloud by KVM Network: 10Gbps wired(10gbase SFP+) Network model: L4-LB-Nat + Neutron ML2 LinuxBridge VLAN IPv4 only LBaaS: Brocade ADX L4-NAT-LB(original) Public API Provided the public API Compute node: Flash cached or SSD Glance: provided (NetApp offload) Cinder: NetApp storage Swift (shared Juno cluster) Ironic on under-cloud Compute server deploy with Ansible config Ironic baremetal compute Nexsus Cisco for Tagged VLAN module iomemory configuration

49 OpenStack Cinder Block storage: ConoHa: NexentaStor(SDS) AppsCloud: NetApp

50 NexentaStor zfs cinder: ConoHa cloud(juno) Compute

NetApp storage: GMO AppsCloud(Havana/Juno) If you are using the same Cluster ontap NetApp a Glance and Cinder storage, it is possible to offload a copy of the inter-service of OpenStack as the

51 NetApp storage: GMO AppsCloud(Havana/Juno) If you are using the same Cluster ontap NetApp a Glance and Cinder storage, it is possible to offload a copy of the inter-service of OpenStack as the processing of NetApp side. Create volume from glance image ((glance the image is converted (ex: qcow2 to raw) required that does not cause the condition) Volume QoS limit: Important function of multitenant storage Uppper IOPS-limit by volume

52 OpenStack Ironic: Only AppsCloud: Undercloud Ironic deploy Multi-tenant Ironic deploy

53 Ironic with undercloud: GMO AppsCloud(Juno) For Compute server deployment. Kilo Ironic and All-in-one Compute server: 10G boot Clout-init: network Compute setup: Ansible Under-cloud Ironic(Kilo): It will use a different network and Ironic Baremetal dhcp for Service baremetal compute Ironic(Kilo). (OOO seed server) Trunk allowed vlan, LACP

allowed VLAN Clout-init: network + lldp Network: Nexsus Cisco Allowd VLAN

54 Ironic(Kilo) baremetal: GMO AppsCloud(Juno) Boot baremetal instance baremetal server (with Fusion iomemory SanDisk) 1G x4 bonding + Tagged allowed VLAN Clout-init: network + lldp Network: Nexsus Cisco Allowd VLAN security Ironic Kilo + Juno: Fine Ironic Python driver Whole Image write Windows: OK

55 OpenStack Juno: 2 service cluster, released Service model: Public cloud by KVM Network: 10Gbps wired(10gbase SFP+) Network model: Flat-VLAN + Neutron ML2 ovs-vxlan overlay + ML2 LinuxBridge(SaaS only) IPv6/IPv4 dualstack LBaaS: LVS-DSR(original) Public API Provided the public API (v2 Domain) Compute node: ALL SSD for booting OS Without Cinder boot Glance: provided Cinder: SSD NexentaStore zfs (SDS) Swift (shared Juno cluster) Cobbler deply on under-cloud Ansible configuration SaaS original service with keystone auth , web, CPanel and WordPress Service model: Public cloud by KVM Network: 10Gbps wired(10gbase SFP+) Network model: L4-LB-Nat + Neutron ML2 LinuxBridge VLAN IPv4 only LBaaS: Brocade ADX L4-NAT-LB(original) Public API Provided the public API Compute node: Flash cached or SSD Glance: provided (NetApp offload) Cinder: NetApp storage Swift (shared Juno cluster) Ironic on under-cloud Compute server deploy with Ansible config Ironic baremetal compute Nexsus Cisco for Tagged VLAN module iomemory configuration

56 Fin.

OpenStack Summit Tokyo Multi location implementation of OpenStack

OpenStack Summit Tokyo Multi location implementation of OpenStack Hyuntae Park Team Manger / OpenStack Engineer Naoto Gohko IT Architect Enginner # Agenda History