MCP Standard Configuration. version latest

Transcription

1 MCP Standard Configuration version latest

2 Contents Copyright notice 1 Preface 2 Intended audience 2 Documentation history 2 Cloud sizes 3 OpenStack environment sizes 3 Kubernetes cluster sizes 3 StackLight LMA and environment sizes 4 Ceph cluster sizes 4 Minimum hardware requirements 6 Infrastructure nodes disk layout 9 Networking 10 Server networking 10 Access networking 10 Switching fabric capabilities 12 NFV considerations 13 StackLight LMA scaling 14 StackLight LMA server roles and services distribution 14 StackLight LMA resource requirements per cloud size 14 OpenStack environment scaling 16 OpenStack server roles 16 Services distribution across nodes 17 Operating systems and versions 18 OpenStack compact cloud architecture 19 OpenStack small cloud architecture 21 OpenStack medium cloud architecture with Neutron OVS DVR/non-DVR 23 OpenStack medium cloud architecture with OpenContrail 26 OpenStack large cloud architecture 29 Kubernetes cluster scaling 33 Small Kubernetes cluster architecture 33 Medium Kubernetes cluster architecture , Mirantis Inc. Page i

3 Large Kubernetes cluster architecture 36 Ceph hardware requirements 38 Ceph cluster considerations 38 Ceph hardware considerations , Mirantis Inc. Page ii

4

5 Copyright notice 2018 Mirantis, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. No part of this publication may be reproduced in any written, electronic, recording, or photocopying form without written permission of Mirantis, Inc. Mirantis, Inc. reserves the right to modify the content of this document at any time without prior notice. Functionality described in the document may not be available at the moment. The document contains the latest information at the time of publication. Mirantis, Inc. and the Mirantis Logo are trademarks of Mirantis, Inc. and/or its affiliates in the United States an other countries. Third party trademarks, service marks, and names mentioned in this document are the properties of their respective owners. 2018, Mirantis Inc. Page 1

6 Preface This documentation provides information on how to use Mirantis products to deploy cloud environments. The information is for reference purposes and is subject to change. Intended audience This documentation is intended for deployment engineers, system administrators, and developers; it assumes that the reader is already familiar with network and cloud concepts. Documentation history This is the latest version of the Mirantis Cloud Platform documentation. It is updated continuously to reflect the recent changes in the product. To switch to any release-tagged version of the Mirantis Cloud Platform documentation, use the Current version drop-down menu on the home page. 2018, Mirantis Inc. Page 2

7 Cloud sizes The Mirantis Cloud Platform (MCP) enables you to deploy OpenStack environments and Kubernetes clusters of different scales. This document uses the terms small, medium, and large clouds to correspond to the number of virtual machines that you can run in your OpenStack environment or the number of pods that you can run in your Kubernetes cluster. This section describes sizes of MCP clusters. OpenStack environment sizes All cloud environments require you to configure a staging environment which mimics the exact production setup and enables you to test configuration changes prior to applying them in production. However, if you have multiple clouds of the same type, one staging environment is sufficient. Environments configured with Neutron OVS as a networking solution require additional hardware nodes called network nodes or tenant network gateway nodes. The following table describes the number virtual machines for each scale: Environment size OpenStack environment sizes Number of virtual machines Number of compute nodes Compact Up to 1,000 Up to 50 3 Small 1,000-2, Number of infrastructure nodes Medium 2,500-5, (Neutron OVS or OpenContrail) Large 5,000-10, (OpenContrail only) Kubernetes cluster sizes Kubernetes and etcd are lightweight applications that typically do not require a lot of resources. However, each Kubernetes components scales differently and some of them may have limitations on scaling. Based on performance testing, an optimal density of pods is 100 pods per Kubernetes Node. Note If you run both Kubernetes clusters and OpenStack environments in one MCP installation, you do not need to configure additional infrastructure nodes. Instead, the same infrastructure nodes are used for both. The following table describes the number of Kubernetes Master nodes and Kubernetes Nodes for each scale: Small cloud 2018, Mirantis Inc. Page 3

8 Cluster size Number of pods Number of Nodes Number of Master nodes Small 2,000-5, Medium 5,000-20, Large 20,000-50, StackLight LMA and environment sizes Number of infrastructure nodes The following table contains StackLight LMA sizing recommendations for different sizes of monitored clouds. Environment size Number of monitored nodes OpenStack environment sizes Number of StackLight LMA nodes Compact Up to 50 0 (colocated with VCP) Notes and comments StackLight LMA is installed on virtual machines running on the same KVM infrastructure nodes as VCP. Small StackLight LMA is installed on dedicated infrastructure nodes running Prometheus long-term storage and/or InfluxDB, Elasticsearch, and the Docker Swarm mode cluster. Medium StackLight LMA is installed on dedicated infrastructure nodes running Prometheus long-term storage and/or InfluxDB, Elasticsearch, and the Docker Swarm mode cluster. Large StackLight LMA is installed on 3 dedicated infrastructure nodes running Prometheus long-term storage and/or InfluxDB and Elasticsearch, and on 3 bare metal servers running the Docker Swarm mode cluster. The Elasticsearch cluster can be scaled up to 5 nodes that rises the number of StackLight LMA nodes to 8. Ceph cluster sizes The following table summarizes the number of storage nodes required depending on cloud size to meet the performance characteristics described in Ceph cluster considerations. Ceph storage nodes per cloud size 2018, Mirantis Inc. Page 4

9 Cloud size Number of compute nodes Number of virtual machines Number of storage nodes Compact Up to 50 Up to 1,000 Up to 9 (20-disk chassis) Small ,000-2, (20-disk chassis) Medium ,000-4, (20-disk chassis) Large ,000-10, (36-disk chassis) Number of Ceph Monitors Seealso Ceph hardware considerations 2018, Mirantis Inc. Page 5

10 Minimum hardware requirements When calculating hardware requirements, you need to plan for the infrastructure nodes, compute nodes, storage nodes, and, if you are installing a Kubernetes cluster, Kubernetes Master nodes and Kubernetes Nodes. Note For more details about services distribution throughout the nodes, see: OpenStack environment scaling and Kubernetes cluster scaling. The following tables list minimal hardware requirements for the corresponding nodes of the cloud: MCP foundation node Parameter Value CPU RAM Disk Network 1 x 12 Core CPU Intel Xeon E5-2670v3 1 x 14 Core CPU Intel Xeon Gold 5120 (Skylake-SP) 32 GB 1 x 2 TB HDD 1 x 1/10 GB Intel X710 dual-port NICs OpenStack infrastructure node Parameter Value CPU 2 x 12 Core CPUs Intel Xeon E5-2670v3 2 x 12 Core CPUs Intel Xeon Gold 6126 (Skylake-SP) RAM Disk Network 2 x 14 Core CPUs Intel Xeon Gold 5120 (Skylake-SP) 256 GB 2 x 1.6 TB SSD Intel S3520 or similar 2 x 10 GB Intel X710 dual-port NICs OpenStack compute node Parameter Value 2018, Mirantis Inc. Page 6

11 CPU 2 x 12 Core CPUs Intel Xeon E5-2670v3 2 x 12 Core CPUs Intel Xeon Gold 6126 (Skylake-SP) RAM Disk Network 2 x 14 Core CPUs Intel Xeon Gold 5120 (Skylake-SP) 256 GB 1 x 2TB HDD for system storage 2 x 960 GB SSD Intel S3520 for VM disk storage 1 2 x 10 GB Intel X710 dual-port NICs 1 Only required if local storage for VM ephemeral disks is used. Not required if network distributed storage is set up in the cloud, such as Ceph. 2018, Mirantis Inc. Page 7

12 Ceph storage node Parameter Value CPU RAM Disk Boot drives Network 2 x 8 Core CPU Intel E5-2630v4 or 2 x 8 Core CPU Intel Xeon Silver GB 4 x 200 GB SSD Intel S3710 for journal storage 20 x 2 TB HDD for data storage 2 x 128 GB Disk on Module (DOM) or 2 x 150 GB SSD Intel S3520 or similar 1 x 10 GB Intel X710 dual-port NICs or similar Kubernetes Master node or Kubernetes Node Parameter Value CPU 2 x 12 Core CPUs Intel Xeon E5-2670v3 2 x 12 Core CPUs Intel Xeon Gold 6126 RAM Disk Network 2 x 14 Core CPUs Intel Xeon Gold GB 2 x 960 GB SSD Intel S3520 or similar for system storage 2 x 10 GB Intel X710 dual-port NICs 2018, Mirantis Inc. Page 8

13 Infrastructure nodes disk layout Infrastructure nodes are typically installed on hardware servers. These servers run all components of management and control plane for both MCP and the cloud itself. It is very important to configure hardware servers properly upfront because changing their configuration after initial deployment is costly. For instructions on how to configure the disk layout for MAAS to provision the hardware machines, see MCP Deployment Guide: Add a custom disk layout per node in the MCP model. Consider the following recommendations: Layout Mirantis recommends using the LVM layout for disks on infrastructure machines. This option allows for more operational flexibility, such as resizing the Volume Groups and Logical Volumes for scale-out. LVM Volume Groups According to Minimum hardware requirements, an infrastructure node typically has two SSD disks. These disks must be configured as LVM Physical Volumes and joined into a Volume Group. The name of the Volume Group is the same across all infrastructure nodes to ensure consistency of LCM operations. Mirantis recommends following the vg01 naming convention for the Volume Group. LVM Logical Volumes The following table summarizes the recommended Logical Volume schema for infrastructure nodes. Follow the instructions in the MCP Deployment Guide to configure this in your cluster model. Logical Volume schema for infrastructure nodes Logical Volume path Mount point Size /dev/vg01/root '/' 70 GB /dev/vg01/gluster /srv/glusterfs 200 GB /dev/vg01/images /var/lib/libvirt/images/ 700 GB 2018, Mirantis Inc. Page 9

14 Networking This section describes the key hardware recommendations on server and data networking, as well as switching fabric capabilities and NFV considerations. Server networking The minimal network configuration for hardware servers includes two dual-port 1/10 Gbit Ethernet Network Interface Cards (NICs). The first pair of 1/10 Gbit Ethernet interfaces is used for the PXE, management, and control plane traffic. These interfaces should be connected to access switch in 1 or 10 GbE mode. The following options are recommended to configure this NIC in terms of bonding and distribution of the traffic: Use one 1/10 GbE interface for PXE or management network, and the other 1/10 GbE interface for control plane traffic. This is the most simple and generic option recommended for most use cases. Create a bond interface with this pair of 1/10 GbE physical interfaces and configure it to the balance-alb mode, following the Linux Ethernet Bonding driver documentation. This configuration allows using the interfaces separately from bond for PXE purposes and does not require support from switch fabric. Create a bond interface with this pair of 1/10 GbE interfaces in 802.3ad mode and set load balancing policy to transmission hash policy based on TCP/UDP port numbers (xmit_hash_policy encap3+4). This configuration provides for better availability of the connection. It requires that switching fabric supports LACP Fallback capability. See Switching fabric capabilities for details. The second NIC with two interfaces is used for the data plane traffic and storage traffic. On the operating system level, ports on this 1/10 GbE card are joined into an LACP bond (Linux bond mode 802.3ad). Recommended LACP load balancing method for this bond interface is transmission hash policy based on TCP/UDP port numbers (xmit_hash_policy encap3+4). This NIC must be connected to an access switch in 10 GbE mode. Note The LACP configuration in 802.3ad mode on the server side must be supported by the corresponding configuration of switching fabric. See Switching fabric capabilities for details. Seealso Linux Ethernet Bonding Driver Documentation Access networking 2018, Mirantis Inc. Page 10

15 The top of the rack (ToR) switches provide connectivity to servers on physical and data-link levels. They must provide support for LACP and other technologies used on the server side, for example, 802.1q VLAN segmentation. Access layer switches are used in stacked pairs. Mirantis recommends installing the following 10 GbE switches as the top of the rack (ToR) for Public, Storage, and Tenant networks in MCP: Juniper QFX S 48x 1/10 GbE ports, 6x 40 GbE ports Quanta T3048-LY9 48x 1/10 GbE ports, 4x 40 GbE ports Arista 7050T-64 48x 1/10 GbE ports, 4x 40 GbE QSFP+ ports The recommended IPMI switch is Dell PowerConnect 6248 (no stacking or uplink cards required) or a Juniper EX4300 Series switch. The following 1/10 GbE switches are recommended to install as ToR for PXE and Management networks in MCP: Juniper EX T-BF Arista 7010T-48 QuantaMesh T1048-LB9 The following diagram shows how a server is connected to the switching fabric, and how the fabric itself is configured. For external networking with OpenContrail, the following hardware is recommended: 2018, Mirantis Inc. Page 11

16 Juniper MX104 Virtual s vmx or vsrx for lower bandwidth requirements (up to 80 Gbps) Switching fabric capabilities The following table summarizes requirements for the switching fabric capabilities: Switch fabric capabilities summary Name of requirement LACP TCP/UDP hash balance mode Multihome server connection support LAG/port-channel links Description Level 4 LACP hash balance mode is recommended to support services that employ TCP sessions. This helps to avoid fragmentation and asymmetry in traffic flows. There are two major options to support multihomed server connections: Switch stacking Stacked switches work as a single logical unit from configuration and data path standpoint. This allows you to configure IP default gateway on the logical stacked switch to support multi-rack use case. Multipath Link Aggregation Groups MLAG support is recommended to allow cross-switch bonding across stacked ToR switches. Using MLAG allows you to maintain and upgrade the switches separately without network interruption for servers. On the other hand, the Layer-3 network configuration is more complicated when using MLAG. Therefore, Mirantis recommends using MLAG with plain access network topology. The number of supported LAGs/port-channel links per switch must be twice the number of ports. Take this parameter into account so that you can create the required number of LAGs to accommodate all servers connected to the switch. Note LACP configurations on access and server levels must be compatible with each other. In general, it might require additional design and testing effort in every particular case, depending on the models of switching hardware and the requirements to networking performance. The following capabilities are required from switching fabric to support PXE and BOOTP protocols over LACP bond interface. See Server networking for details. Switch fabric capabilities summary Name of requirement Description 2018, Mirantis Inc. Page 12

17 LACP fallback mode Servers must be able to boot with bootp/pxe protocol over a network connected through LACP bond interface in the 802.3ad mode. LACP fallback capability allows you to dynamically assemble an LAG based on status of member NICs on the server side. Upon the initial boot, the interfaces are not bonded, LAG is disassembled, and the server boots normally over a single PXE NIC. This requirement applies to the multihomed Management network option only. Note that different switch vendors have different notations for the said functionality. See links to the examples below. Seealso Configuring LACP Fallback on Arista switches Forcing MC-LAG Links or Interfaces With Limited LACP Capability to Be Up Configure LACP auto ungroup on Dell switches NFV considerations Network function virtualization (NFV) is an important factor to consider while planning hardware capacity for the Mirantis Cloud Platform. Mirantis recommends separating nodes that support NFV from other nodes to reserve them as Data Plane nodes that use network virtualization functions. The following types of the Data Plane nodes use NFV: 1. Compute nodes that can run virtual machines with hardware-assisted Virtualized Networking Functions (VNF) in terms of the OPNFV architecture. 2. Networking nodes that provide gateways and routers to the OVS-based tenant networks using network virtualization functions. The following table describes compatibility of NFV features for different MCP deployments. NFV for MCP compatibility matrix Type Host OS Kernel Huge pages DPDK SR-IOV NUMA CPU pinning Multiqueue OVS Xenial 4.8 Yes No Yes Yes Yes Yes Kernel vrouter DPDK vrouter DPDK OVS Xenial 4.8 Yes No Yes Yes Yes Yes Xenial 4.8 Yes Yes No Yes Yes No (version 3.2) Xenial 4.8 Yes Yes No Yes Yes Yes 2018, Mirantis Inc. Page 13

18 StackLight LMA scaling The MCP StackLight LMA enables the user to monitor all kinds of MCP clusters, including OpenStack, Kubernetes, and Ceph environments. This section describes the distribution of StackLight LMA services and hardware requirements for different sizes of clouds. StackLight LMA server roles and services distribution The following table lists the roles of StackLight LMA nodes and their names in the Salt Reclass metadata model: Server role name StackLight LMA nodes Server role group name in Reclass model Description StackLight LMA metering node mtr Servers that run Prometheus long-term storage and/or InfluxDB. StackLight LMA log storage and visualization node log Servers that run Elasticsearch and Kibana. StackLight LMA monitoring node mon Servers that run the Prometheus, Grafana, Pushgateway, Alertmanager, and Alerta services in containers in Docker Swarm mode. StackLight LMA resource requirements per cloud size Compact cloud The following table summarizes resource requirements of all StackLight LMA node roles for compact clouds (up to 50 compute nodes). Virtual server roles Resource requirements per StackLight LMA role for compact cloud # of s CPU vcores per Memory (GB) per mon mtr log Small cloud Disk space (GB) per The following table summarizes resource requirements of all StackLight LMA node roles for small clouds ( compute nodes). 2018, Mirantis Inc. Page 14

19 Virtual server roles Resource requirements per StackLight LMA role for small cloud # of s CPU vcores per Memory (GB) per mon mtr log Medium cloud Disk space (GB) per The following table summarizes resource requirements of all StackLight LMA node roles for medium clouds ( compute nodes). Virtual server roles Resource requirements per StackLight LMA role for medium cloud # of s CPU vcores per Memory (GB) per Disk space (GB) per mon SSD mtr SSD log SSD Large cloud Disk type The following table summarizes resource requirements of all StackLight LMA node roles for large clouds ( compute nodes). Virtual server roles Resource requirements per StackLight LMA role for large cloud # of s CPU vcores per Memory (GB) per Disk space (GB) per mon SSD mtr SSD log SSD Disk type 2018, Mirantis Inc. Page 15

20 OpenStack environment scaling The Mirantis Cloud Platform (MCP) enables you to deploy OpenStack environments at different scales. This document defines the following sizes of environments: small, medium, and large. Each environment size requires a different number of infrastructure nodes. The Virtualized Control Plane (VCP) services are distributed among the physical infrastructure nodes for optimal performance. This section describes VCP services distribution, as well as hardware requirements for different sizes of clouds. OpenStack server roles Components of the Virtualized Control Plane (VCP) have roles that define their functions. Each role can be assigned to a specific set of virtual servers which allows to adjust the number of s with a particular role independently of other roles providing greater flexibility to the environment architecture. The following table lists the OpenStack roles and their names in the SaltStack formulas: Server role name OpenStack infrastructure nodes Server role group name in SaltStack formulas Description Infrastructure node kvm Infrastructure KVM hosts that run MCP component services as virtual machines. Network node gtw Nodes that provide tenant network data plane services. DriveTrain Salt Master node cfg The Salt Master node that is responsible for sending commands to Salt Minion nodes. DriveTrain / StackLight OSS node cid Nodes that run in StackLight OSS and DriveTrain services in containers in Docker Swarm mode. RabbitMQ server node msg Nodes that run the message queue service RabbitMQ. Database server node dbs Nodes that run the database cluster called Galera. OpenStack controller nodes ctl Nodes that run the Virtualized Control Plane service, including the API servers and scheduler components. OpenStack compute nodes cmp Nodes that run the hypervisor service and VM workloads. OpenStack monitoring database nodes mdb Nodes that run the Telemetry monitoring database services. 2018, Mirantis Inc. Page 16

21 Proxy node prx Nodes that run reverse proxy that exposes OpenStack API, dashboards, and other components externally. Contrail controller nodes ntw Nodes that run the OpenContrail controller services. Contrail analytics nodes nal Nodes that run the OpenContrail analytics services. StackLight LMA log nodes log Nodes that run the StackLight LMA logging and visualization services. StackLight LMA database nodes mtr Nodes that run the StackLight database services. StackLight LMA and OSS nodes mon Nodes that run the StackLight LMA monitoring services and StackLight OSS (DevOps Portal) services. Seealso StackLight LMA server roles and services distribution Services distribution across nodes The distribution of services across physical nodes depends on their resources consumption profile and the number of compute nodes they administer. Mirantis recommends the following distribution of services across nodes: Distribution of services across nodes in an OpenStack environment Service Physical server group Virtual aodh-api kvm yes mdb aodh-evaluator kvm yes mdb aodh-listener kvm yes mdb aodh-notifier kvm yes mdb VM role group ceilometer-agent-central kvm yes mdb or ctl 2 ceilometer-agent-notification kvm yes mdb or ctl 2 ceilometer-api kvm yes mdb or ctl 2 cinder-api kvm yes ctl cinder-scheduler kvm yes ctl 2018, Mirantis Inc. Page 17

22 cinder-volume kvm yes ctl DriveTrain 3 kvm yes cid glance-api kvm yes ctl glance-registry kvm yes ctl gnocchi-metricd 4 kvm yes mdb horizon/apache2 kvm yes prx keystone-all kvm yes ctl mysql-server kvm yes dbs neutron-dhcp-agent 5 gtw no N/A neutron-l2-agent 5 cmp, gtw no N/A neutron-l3-agent 5 gtw no N/A neutron-metadata-agent 5 gtw no N/A neutron-server 5 kvm yes ctl nova-api kvm yes ctl nova-compute cmp no N/A nova-conductor kvm yes ctl nova-scheduler kvm yes ctl OSS Tools 3 kvm yes cid panko-api 4 kvm yes mdb rabbitmq-server kvm yes msg Operating systems and versions The following table lists the operating systems used for different roles in the Virtualized Control Plane (VCP): Operating systems and versions Server role name Server role group name in the SaltStack model Infrastructure node kvm xenial/16.04 Network node gtw xenial/16.04 StackLight LMA monitoring node mon xenial/16.04 Stacklight LMA metering and tenant telemetry node mtr xenial/16.04 Ubuntu version 2018, Mirantis Inc. Page 18

23 StackLight LMA log storage and visualization node log xenial/16.04 DriveTrain Salt Master node cfg xenial/16.04 DriveTrain StackLight OSS node cid xenial/16.04 RabbitMQ server node msg xenial/16.04 Database server node dbs xenial/16.04 OpenStack controller node ctl xenial/16.04 OpenStack compute node cmp xenial/16.04 (depends on whether NFV features enabled or not) Proxy node prx xenial/16.04 OpenContrail controller node ntw trusty/14.04 for v3.2, xenial/16.04 for v4.x OpenContrail analytics node nal trusty/14.04, xenial/16.04 for v4.x OpenStack compact cloud architecture A compact OpenStack cloud includes up to 50 compute nodes and requires you to have at least three infrastructure nodes. A compact cloud includes all the roles described in OpenStack server roles. The following diagram describes the distribution of VCP and other services throughout the infrastructure nodes. 2(1, 2, 3) The mdb role is for Pike, the ctl role is for Ocata. 3(1, 2) DriveTrain and OSS Tools services run in the Docker Swarm Mode cluster. 4(1, 2) Gnocchi and Panko services are added starting Pike. 5(1, 2, 3, 4, 5) Services are related to Neutron OVS only. 2018, Mirantis Inc. Page 19

24 The following table describes the hardware nodes in an OpenStack environment of an MCP cluster, roles assigned to them, and resources per node: Physical server roles and hardware requirements Node type Role name Number of servers Infrastructure nodes kvm 3 OpenStack compute nodes cmp up to 50 The following table summarizes the VCP virtual machines mapped to physical servers. Virtual server roles Resource requirements per VCP and DriveTrain roles Physical servers # of s CPU vcores per Memory (GB) per ctl kvm01 kvm02 kvm msg kvm01 kvm02 kvm dbs kvm01 kvm02 kvm prx kvm02 kvm cfg kvm mon kvm01 kvm02 kvm mtr kvm01 kvm02 kvm log kvm01 kvm02 kvm Disk space (GB) per 2018, Mirantis Inc. Page 20

25 cid kvm01 kvm02 kvm gtw kvm01 kvm02 kvm TOTAL Note The gtw VM should have four separate NICs for the following interfaces: dhcp, primary, tenant, and external. It simplifies the host networking: you do not need to pass VLANs to VMs. The prx VM should have an additional NIC for the Proxy network. All other nodes should have two NICs for DHCP and Primary networks. OpenStack small cloud architecture A small OpenStack cloud includes up to 100 compute nodes and requires you to have at least 6 infrastructure nodes, including additional servers for KVM infrastructure and RabbitMQ/Galera services. The Stacklight LMA services are distributed across 3 additional infrastructure nodes. See detailed resource requirements for StackLight LMA in StackLight LMA resource requirements per cloud size. A small cloud includes all the roles described in OpenStack server roles. The number of nodes is the same for both OpenContrail and Neutron OVS-based clouds. However, the roles on network nodes are different. The following diagram describes the distribution of VCP and other services throughout the infrastructure nodes for Neutron OVS-based small clouds. 2018, Mirantis Inc. Page 21

26 The following table describes the hardware nodes in MCP OpenStack, roles assigned to them, and resources per node: Physical server roles and quantities Node type Role name Number of servers Infrastructure nodes (VCP, OpenContrail) kvm 3 Infrastructure nodes (StackLight LMA) kvm 3 OpenStack compute nodes cmp Staging infrastructure nodes kvm 6 Staging OpenStack compute nodes cmp 2-5 The following table summarizes the VCP virtual machines mapped to physical servers. Virtual server roles Resource requirements per VCP and DriveTrain roles Physical servers # of s CPU vcores per Memory (GB) per ctl kvm01 kvm02 kvm Disk space (GB) per 2018, Mirantis Inc. Page 22

27 msg kvm01 kvm02 kvm dbs kvm01 kvm02 kvm prx kvm02 kvm cfg kvm cid kvm01 kvm02 kvm gtw kvm01 kvm02 kvm TOTAL The following table summarizes the OpenContrail virtual machines mapped to physical servers (optional). This is only needed when OpenContrail is used for OpenStack Tenant Networking. Virtual server roles Resource requirements per OpenContrail roles (optional) Physical servers # of s CPU vcores per Memory (GB) per ntw kvm01 kvm02 kvm nal kvm01 kvm02 kvm TOTAL Disk space (GB) per Note A vcore is the number of available virtual cores considering hyper-threading and overcommit ratio. Assuming an overcommit ratio of 1, the number of vcores in a physical is roughly the number of physical cores multiplied by 1.3. Note The gtw VM should have four separate NICs for the following interfaces: dhcp, primary, tenant, and external. It simplifies the host networking: you do not need to pass VLANs to VMs. The prx VM should have an additional NIC for the Proxy network. All other nodes should have two NICs for DHCP and Primary networks. OpenStack medium cloud architecture with Neutron OVS DVR/non-DVR A medium OpenStack cloud includes up to 200 compute nodes and requires you to have at least 9 infrastructure nodes, including additional servers for KVM infrastructure and RabbitMQ/Galera services. For 2018, Mirantis Inc. Page 23

28 Neutron OVS as a networking solution, 3 bare-metal network nodes are installed to accommodate network traffic as opposed to virtualized controllers in the case of OpenContrail. The Stacklight LMA services are distributed across 3 additional infrastructure nodes. See detailed resource requirements for StackLight LMA in StackLight LMA resource requirements per cloud size. A medium cloud includes all roles described in OpenStack server roles. The following diagram describes the distribution of VCP and other services throughout the infrastructure nodes. The following table describes the hardware nodes in MCP OpenStack, roles assigned to them, and resources per node: 2018, Mirantis Inc. Page 24

29 Physical server roles and quantities Node type Role name Number of servers Infrastructure nodes (VCP) kvm 6 Infrastructure nodes (StackLight LMA) kvm 3 Tenant network gateway nodes gtw 3 OpenStack compute nodes cmp Staging infrastructure nodes kvm 9 Staging tenant network gateway nodes gtw 3 Staging compute nodes cmp 2-5 The following table summarizes the VCP virtual machines mapped to physical servers. Virtual server roles Physical servers Resource requirements per VCP role # of s CPU vcores per Memory (GB) per ctl kvm01 kvm02 kvm dbs kvm01 kvm02 kvm msg kvm04 kvm05 kvm prx kvm04 kvm TOTAL Disk space (GB) per Virtual server roles Resource requirements per DriveTrain role Physical servers # of s CPU vcores per Memory (GB) per cfg kvm cid kvm01 kvm02 kvm TOTAL Disk space (GB) per Note A vcore is the number of available virtual cores considering hyper-threading and overcommit ratio. Assuming an overcommit ratio of 1, the number of vcores in a physical is roughly the number of physical cores multiplied by , Mirantis Inc. Page 25

30 OpenStack medium cloud architecture with OpenContrail A medium OpenStack cloud includes up to 200 compute nodes and requires you to have at least 9 infrastructure nodes, including servers for KVM infrastructure and RabbitMQ/Galera services. For OpenContrail as a networking solution, a virtualized OpenContrail Controller is installed onto the infrastructure nodes instead of bare metal network nodes as in the case of Neutron OVS. The Stacklight LMA services are distributed across 3 additional infrastructure nodes. See detailed resource requirements for StackLight LMA in StackLight LMA resource requirements per cloud size. The following diagram describes the distribution of VCP and other services throughout the infrastructure nodes. Note The diagram displays the OpenContrail 4.0 nodes that have the control, config, database, and analytics services running as plain Docker containers managed by docker-compose. In OpenContrail 3.2, these services are not Docker-based. The following table describes the hardware nodes in MCP OpenStack, roles assigned to them, and resources per node: 2018, Mirantis Inc. Page 26

31 Physical server roles and quantities Node type Role name Number of servers Infrastructure nodes (VCP) kvm 6 Infrastructure nodes (StackLight LMA) kvm 3 Infrastructure nodes (OpenContrail) kvm 3 OpenStack compute nodes cmp Staging infrastructure nodes kvm 12 Staging OpenStack compute nodes cmp , Mirantis Inc. Page 27

32 The following table summarizes the VCP virtual machines mapped to physical servers. Virtual server roles Physical servers Resource requirements per VCP role # of s CPU vcores per Memory (GB) per ctl kvm01 kvm02 kvm dbs kvm01 kvm02 kvm msg kvm04 kvm05 kvm prx kvm04 kvm TOTAL Disk space(gb) per Virtual server roles Resource requirements per DriveTrain role Physical servers # of s CPU vcores per Memory (GB) per cfg kvm cid kvm01 kvm02 kvm TOTAL Disk space(gb) per Virtual server roles Resource requirements per OpenContrail role Physical servers # of s CPU vcores per Memory (GB) per ntw kvm07 kvm08 kvm nal kvm07 kvm08 kvm TOTAL Disk space(gb) per Note A vcore is the number of available virtual cores considering hyper-threading and overcommit ratio. Assuming an overcommit ratio of 1, the number of vcores in a physical is roughly the number of physical cores multiplied by , Mirantis Inc. Page 28

33 OpenStack large cloud architecture A large OpenStack cloud includes up to 500 compute nodes and requires you to have at least 12 infrastructure nodes, including dedicated bare-metal servers for RabbitMQ, OpenStack API services, and database servers. Use OpenContrail as a networking solution for large OpenStack clouds. Neutron OVS is not recommended. The StackLight LMA services are distributed across additional 6 infrastructure nodes, including 3 bare metal nodes for the Docker Swarm Mode cluster. See detailed resource requirements for StackLight LMA in StackLight LMA resource requirements per cloud size. A large cloud includes all roles described in OpenStack server roles. The following diagram describes the distribution of VCP and other services throughout the infrastructure nodes. 2018, Mirantis Inc. Page 29

34 Note The diagram displays the OpenContrail 4.0 nodes that have the control, config, database, and analytics services running as plain Docker containers managed by docker-compose. In OpenContrail 3.2, these services are not Docker-based. 2018, Mirantis Inc. Page 30

35 The following table describes the hardware nodes in MCP OpenStack, roles assigned to them, and resources per node: Physical server roles and quantities Node type Role name Number of servers Infrastructure nodes (VCP) kvm 9 Infrastructure nodes (OpenContrail) kvm 3 Monitoring nodes (StackLight LMA) mon 3 Infrastructure nodes (StackLight LMA) kvm 3 OpenStack compute nodes cmp Staging infrastructure nodes kvm 18 Staging OpenStack compute nodes cmp 2-5 The following table summarizes the VCP virtual machines mapped to physical servers. Virtual server roles Physical servers Resource requirements per VCP role # of s CPU vcores per Memory (GB) per ctl kvm02 kvm03 kvm04 kvm05 kvm dbs kvm04 kvm05 kvm msg kvm07 kvm08 kvm prx kvm07 kvm TOTAL Disk space (GB) per Virtual server roles Resource requirements per DriveTrain role Physical servers # of s CPU vcores per Memory (GB) per cfg kvm cid kvm01 kvm02 kvm TOTAL Disk space(gb) per Resource requirements per OpenContrail role 2018, Mirantis Inc. Page 31

36 Virtual server roles Physical servers # of s CPU vcores per Memory (GB) per ntw kvm10 kvm11 kvm nal kvm10 kvm11 kvm TOTAL Disk space (GB) per Note A vcore is the number of available virtual cores considering hyper-threading and overcommit ratio. Assuming an overcommit ratio of 1, the number of vcores in a physical is roughly the number of physical cores multiplied by , Mirantis Inc. Page 32

37 Kubernetes cluster scaling As described in Kubernetes cluster sizes, depending on the anticipated number of pods, your Kubernetes cluster may be a small, medium, or large scale deployment. A different number of Kubernetes Nodes is required for each size of cluster. This section describes the services distribution across hardware nodes for different sizes of Kubernetes clusters. Small Kubernetes cluster architecture A small Kubernetes cluster includes 2,000-5,000 pods spread across roughly physical Kubernetes Nodes and 3 Kubernetes Master nodes. Mirantis recommends separating the Kubernetes control plane that includes etcd and the Kubernetes Master node components from Kubernetes workloads. In addition, dedicate separate physical servers for shared storage services GlusterFS and Ceph. Running these components on the same physical servers as the control plane components may result in insufficient cycles for etcd cluster to stay synced and allow the kubelet agent to report their statuses reliably. 2018, Mirantis Inc. Page 33

38 The following diagram displays the layout of services per physical node for a small Kubernetes cluster. 2018, Mirantis Inc. Page 34

39 Medium Kubernetes cluster architecture A medium Kubernetes cluster includes 5,000-20,000 pods spread across roughly physical Kubernetes Nodes and 6 Kubernetes Master nodes. Mirantis recommends separating the Kubernetes control plane that includes etcd and the Kubernetes Master node components from Kubernetes workloads. While Kubernetes components can run on the same host, run etcd on dedicated servers as the etcd workload increases due to constant recording and checking pod and kubelet statuses. You can place shared storage services GlusterFS and Ceph on the same physical nodes as Kubernetes control plane components. 2018, Mirantis Inc. Page 35

40 The following diagram displays the layout of services per physical node for a medium Kubernetes cluster. Large Kubernetes cluster architecture A large Kubernetes cluster includes 20,000-50,000 pods spread across roughly physical Kubernetes Nodes and 9 Kubernetes Master nodes. Mirantis recommends to separate the Kubernetes control plane that includes etcd and Kubernetes Master node components from the Kubernetes workloads. Note While Kubernetes components can run on the same host, run etcd on dedicated servers as the etcd workload increases due to constant recording and checking pod and kubelet statuses. 2018, Mirantis Inc. Page 36

41 In addition, Mirantis recommends placing kube-scheduler separately from the rest of the Kubernetes control plane components due to the high number of Kubernetes pod turnover and rescheduling, kube-scheduler requires more resources than other Kubernetes components. You can place shared storage services GlusterFS and Ceph on the same physical nodes as Kubernetes control plane components. The following diagram displays the layout of services per physical node for a large Kubernetes cluster. 2018, Mirantis Inc. Page 37

42 Ceph hardware requirements Mirantis recommends to use the Ceph cluster as primary storage solution for all types of ephemeral and persistent storage. A Ceph cluster that is built in conjunction with MCP must be designed to accommodate: Capacity requirements Performance requirements Operational requirements This section describes differently sized clouds that use the same Ceph building blocks. Ceph cluster considerations When planning storage for your cloud, you must consider performance, capacity, and operational requirements that affect the efficiency of your MCP environment. Based on those considerations and operational experience, Mirantis recommends no less than nine-node Ceph clusters for OpenStack production environments. Recommendation for test, development, or PoC environments is a minimum of five nodes. See details in Ceph cluster sizes. Note This section provides simplified calculations for your reference. Each Ceph cluster must be evaluated by a Mirantis Solution Architect. Capacity When planning capacity for your Ceph cluster, consider the following: Total usable capacity The existing amount of data plus the expected increase of data volume over the projected life of the cluster. Data protection (replication) Typically, for persistent storage a factor of 3 is recommended, while for ephemeral storage a factor of 2 is sufficient. However, with a replication factor of 2, an object can not be recovered if one of the replicas is damaged. Cluster overhead To ensure cluster integrity, Ceph stops writing if the cluster is 90% full. Therefore, you need to plan accordingly. Administrative overhead To catch spikes in cluster usage or unexpected increases in data volume, an additional 10-15% of the raw capacity should be set aside. The following table describes an example of capacity calculation: 2018, Mirantis Inc. Page 38

43 Example calculation Parameter Value Current capacity persistent Expected growth over 3 years Required usable capacity 500 TB 300 TB 800 TB Replication factor for all pools 3 Raw capacity With 10% cluster internal reserve With operational reserve of 15% Total cluster capacity 2.4 PB 2.64 PB 3.03 PB 3 PB Overall sizing When you have both performance and capacity requirements, scale the cluster size to the higher requirement. For example, if a Ceph cluster requires 10 nodes for capacity and 20 nodes for performance to meet requirements, size the cluster to 20 nodes. Operational recommendations A minimum of 9 Ceph OSD nodes is recommended to ensure that a node failure does not impact cluster performance. Mirantis does not recommend using servers with excessive number of disks, such as more than 24 disks. All Ceph OSD nodes must have identical CPU, memory, disk and network hardware configurations. If you use multiple availability zones (AZ), the number of nodes must be evenly divisible by the number of AZ. Perfromance considerations When planning performance for your Ceph cluster, consider the following: Raw performance capability of the storage devices. For example, a SATA hard drive provides 150 IOPS for 4k blocks. Ceph read IOPS performance. Calculate it using the following formula: number of raw read IOPS per device X number of storage devices X 80% Ceph write IOPS performance. Calculate it using the following formula: number of raw write IOPS per device X number of storage devices / replication factor X 65% Ratio between reads and writes. Perform a rough calculation using the following formula: 2018, Mirantis Inc. Page 39

44 read IOPS X % reads + write IOPS X % writes Note Do not use this formula for a Ceph cluster that is based on SSDs only. Contact Mirantis for evaluation. Storage device considerations The expected number of IOPS that a storage device can carry out, as well as its throughput, depends on the type of device. For example, a hard disk may be rated for 150 IOPS and 75 MB/s. These numbers are complementary because IOPS are measured with very small files while the throughput is typically measured with big files. Read IOPS and write IOPS differ depending on the device. onsidering typical usage patterns helps determining how many read and write IOPS the cluster must provide. A ratio of 70/30 is fairly common for many types of clusters. The cluster size must also be considered, since the maximum number of write IOPS that a cluster can push is divided by the cluster size. Furthermore, Ceph can not guarantee the full IOPS numbers that a device could theoretically provide, because the numbers are typically measured under testing environments, which the Ceph cluster cannot offer and also because of the OSD and network overhead. You can calculate estimated read IOPS by multiplying the read IOPS number for the device type by the number of devices, and then multiplying by ~0.8. Write IOPS are calculated as follows: (the device IOPS * number of devices * 0.65) / cluster size If the cluster size for the pools is different, an average can be used. If the number of devices is required, the respective formulas can be solved for the device number instead. Ceph hardware considerations When sizing a Ceph cluster, you must consider the number of drives needed for capacity and the number of drives required to accommodate performance requirements. You must also consider the largest number of drives that ensure all requirements are met. The following list describes generic hardware considerations for a Ceph cluster: Create one Ceph Object Storage Device (OSD) per HDD disk in Ceph OSD nodes. Allocate 1 CPU thread per OSD. Allocate 1 GB of RAM per 1 TB of disk storage on the OSD node. Do not use RAID arrays for Ceph disks. Instead, all drives must be available to Ceph individually. Storage controllers with support for host bus adapter (HBA) or just-a-bunch-of-disks (JBOD) mode shoulud be selected for Ceph OSD nodes. Preferably, configure storage controllers on OSD nodes to work in the HBA or initiator target (IT) mode. 2018, Mirantis Inc. Page 40

45 If the RAID controller does not support HBA mode, you can configure the JBOD mode with disks exposed individually. Confirm that disks are exposed directly to the operating system by checking the device parameters. The Ceph monitors can run in virtual machines on the infrastructure nodes, as they use relatively few resources. Place Ceph write journals on write-optimized SSDs instead of OSD HDD disks. Use at least 1 journal device per 5 OSD devices. See the detailed hardware requirements to Ceph OSD storage nodes in Minimum hardware requirements. The following table provides an example of input parameters for a Ceph cluster calculation: Example of input parameters Parameter Value Virtual size 40 GB Read IOPS 14 Read to write IOPS ratio 70/30 Number of availability zones 3 For 50 compute nodes, 1,000 s Number of OSD nodes: 9, 20-disk 2U chassis This configuration provides 360 TB of raw storage and with cluster size of 3 and 60% used initially, the initial amount of data should not exceed 72 TB (out of 120 TB of replicated storage). Expected read IOPS for this cluster is approximately 20,000 and write IOPS 5,000, or 15,000 IOPS in a 70/30 pattern. Note In this case performance is the driving factor, and so the capacity is greater than required. For 300 compute nodes, 6,000 s Number of OSD nodes: 54, 36-disks chassis The cost per node is low compared to the cost of the storage devices and with a larger number of nodes failure of one node is proportionally less critical. A separate replication network is recommended. For 500 compute nodes, 10,000 s Number of OSD nodes: 60, 36-disks chassis You may consider using a larger chassis. A separate replication network is required. 2018, Mirantis Inc. Page 41

46 Note For a large cluster, three monitors is the upper recommended limit. If scale-out is planned, deploy five monitors initially. 2018, Mirantis Inc. Page 42