IBM System x Solution for IBM Cloud Manager with OpenStack Reference architecture

Transcription

1 IBM System x Solution for IBM Cloud Manager with OpenStack Kai Huang Jun Zeng Yi Xuan Huang IBM Systems and Technology Group ISV Enablement September 2014

2 Table of contents Introduction... 3 Business problem and business value... 3 Business problem... 3 Business value... 4 Requirements... 5 Functional requirements... 5 Non-functional requirements... 6 Architectural overview... 7 Component model... 8 Compute pool Storage pool Controller Support services Platform Resource Scheduler Operational model IBM System x3650 M IBM System x3550 M IBM System x3650 M4 BD IBM System Networking RackSwitch G8124E IBM System Networking RackSwitch G Deployment diagram and components Deployment considerations Controller, compute, and storage servers configuration Networking Solution sizing System availability Full rack system sample Solution scaling and expansion Authorization and automation Best practices Appendix 1. Bill of materials (BOM) Appendix 2. Resources Trademarks and special notices... 35

3 Introduction OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a data center, all managed through a dashboard that gives administrators control while empowering users to provision resources. OpenStack continues to gain significant traction because of the rapid adoption of cloud and the open source development environment. This document describes the reference architecture for OpenStack from IBM: IBM Cloud Manager with OpenStack 4.1. IBM Cloud Manager with OpenStack is an easy to deploy, simple to use, cloud management software tool that is based on OpenStack with IBM enhancements that feature a self-service portal for workload provisioning, virtual image management, and monitoring, as well as an advanced scheduling component. It is an innovative, cost-effective approach that also includes automation, metering, and security for the virtualized cloud environment. This reference architecture is primarily targeted at managed service providers (MSPs), cloud service providers (CSPs), and enterprise private clouds that require a complete solution for cloud infrastructure as a service (IaaS). This document provides planning, design considerations, and reference configurations for implementing IBM Cloud Manager with OpenStack 4.1 on IBM System x products, so as to lower the barriers of adoption, such as maintenance cost, complexity, and a lack of convergence that are traditionally associated with the OpenStack technology. The reference architecture focuses on achieving competitive performance and workload density while minimizing procurement costs by optimizing the configuration of IBM System x hardware and software, plus optimized configurations for cloud deployment. OpenStack, along with other IBM branded software, is deployed on Systems x products to instantiate compute, network, and management services. The solution empowers IT administration by enabling virtual machine migration, resource balancing, easy configuration, and resilience against certain types of failures. This configuration can be extended to multiple racks for additional workload capacity or be modified slightly to accommodate specific customer needs. The intended audience of this document is IT professionals, technical architects, sales engineers, and consultants. It will assist them in planning, designing, and implementing IBM Cloud Manager with OpenStack 4.1 on IBM System x products. The reader of this document is expected to have a basic knowledge of Red Hat Enterprise Linux and OpenStack. Business problem and business value The following sections outline the value proposition of the IBM Cloud Manager with OpenStack System x reference architecture. Business problem Virtualization and cloud have both achieved tremendous growth on x86 server hardware in recent years. Initially, virtualization provided immediate relief to server sprawl by enabling consolidation of multiple workloads onto a single server. As the hypervisor space evolved and cloud proved to be a more costeffective means for deploying IT, independent software vendors (ISVs) moved up the software stack as a means of differentiating themselves while driving customer lock-in and profit. MSPs and CSPs face intense challenges that drive them to look for economic but scalable cloud solutions that enable them to offer competitively priced IT services while easily growing capacity to meet customer Page 3 of 36

4 demand. OpenStack provides an open source alternative for cloud management that is increasingly used to build and manage cloud infrastructures to support enterprise operations. The OpenStack free license also significantly reduces the initial acquisition and expansion costs (for example, when scaling out the configuration). However, strong technical skill is required to build a robust cloud service based on OpenStack simply because it is extremely flexible and offers a wide array of deployment and configuration options. Moreover, the increasing complexity of such a cloud cluster as it scales out also introduces the risk of an unbalanced infrastructure, which might lead to performance issues, insufficient network bandwidth, and increased exposure to security vulnerabilities. Business value The IBM Cloud Manager with OpenStack reference architecture solves the previously described problems by providing a blueprint for accelerating the deployment of an enterprise level OpenStack cloud environment on IBM System x hardware. This reference architecture provides: Consolidated and fully integrated hardware resources with balanced workload for compute, network, and storage An aggregation of compute and storage hardware, delivering a single, virtualized resource pool, which can be customized to different compute/storage ratios to meet the requirements of various solutions A self-service portal for workload provisioning, allowing continuous integration, testing, and delivery Ease of scaling (both vertical and horizontal) based on business need. Storage pool can be extended at runtime. Minimized single points of failure in networking, delivering continuous access to virtual machines Redundancy and full hardware utilization Rapid OpenStack cloud deployment, including updates, patches, security, and usability enhancements with enterprise-level support from IBM Unified management and monitoring for kernel-based virtual machines (KVMs) The reference architecture reduces the complexity of OpenStack in the solution by outlining a validated configuration that scales and delivers minimum levels of redundancy across servers, storage, and networking to help enable certain levels of fault tolerance. In this reference architecture, virtual machines can be migrated among clustered host servers to support resource rebalancing and scheduled maintenance. Therefore, IT operations staff can minimize service downtime and deployment risks. Page 4 of 36

5 Requirements This section describes the functional and non-functional requirements that are addressed by the IBM System x solution for IBM Cloud Manager with OpenStack. Functional requirements Table 1 lists the functional requirements for this reference architecture. Requirement Description Supported by Mobility Resource provisioning Multitenancy Management portal Metering Table 1. Functional requirements Workload is not tied to any physical location Virtual machines, virtual storage and virtual network can be provisioned on demand Resources are segmented based on tenancy Web-based dashboard for workloads management Collect measurements of used resources to allow billing Host-assisted virtual machine (VM) migration Enabled VM is booted from volume and run on different host OpenStack compute service OpenStack block storage service OpenStack network service Built-in multitenancy in OpenStack IBM Cloud Manager with OpenStack dashboard for most of daily management operations Metering functions in IBM Cloud Manager Page 5 of 36

6 Non-functional requirements Table 2 lists the non-functional requirements for this reference architecture. Requirement Description Supported by OpenStack environment Scalability Load balancing Fault tolerance Supports the current OpenStack edition Solution components scale for growth Workload is distributed evenly across servers Single component error will not lead to whole system unavailability OpenStack Icehouse release through IBM Cloud Manager with OpenStack 4.1 Red Hat 6.5 operating system with KVM hypervisor inside Compute nodes and storage nodes can be scaled separately within a rack and across racks Network interfaces are teamed and load balanced Use of OpenStack scheduler for balancing compute and storage resources Hardware architecture ensures that computing service, storage service and network service will be automatically switched to remaining components, as long as the controller node is still functional. Compute node and storage node are redundant Data are stored on multiple drives, so no single drive failure can cause loss of data Physical footprint Compact solution IBM System x server, network, and software integrated into one rack with validated performance and reliability Provides 1U compute node option Ease of installation Ease of management/operations Reduced complexity for solution deployment Reduced complexity for solution deployment Chef server provides greater flexibility and control over how you deploy OpenStack in your cloud Separately available installation and configuration guide Optional deployment services IBM Cloud Manager with OpenStack dashboard for monitoring and GUI based operations Readily available education and user documentation Support Available vendor support Hardware warranty and software support are included with component products Page 6 of 36

7 Flexibility Security High performance Solution supports variable deployment methodologies Solution provides means to secure customer infrastructure Solution components are high-performance Hardware and software components can be modified or customized to meet a variety of unique customer requirements Provides both local and shared storage for workload Security is integrated in the IBM System x hardware, such as trusted platform module (TPM) Networks are isolated by virtual LAN (VLAN) and virtual extensible LAN (VXLAN) for virtual networks Provides 24 to 32 mixed workload (two vcpu, 6GB vram, 100 GB disk) per host Table 2. Non-functional requirements Architectural overview The IBM Cloud Manager with OpenStack 4.1 is based on the OpenStack Icehouse release. It includes all of Icehouse s features, plus a more robust management platform, heterogeneous hypervisor support, and default drivers for major IBM servers and storage devices. The following diagram depicts how IBM Cloud Manager incorporates OpenStack to provide an enterprise-level cloud platform and its related interface. Page 7 of 36

8 Figure 1. Primary components in IBM Cloud Manager with OpenStack Figure 1 shows the main components in IBM Cloud Manager with OpenStack, and how it interacts with the KVM hypervisor. IBM Cloud Manager with OpenStack uses an IaaS gateway, which is a lightweight proxy middleware container capable of providing pluggable adaptation to normalize interactions across multiple IaaS cloud provider vendors. The Cloud Manager user interface (UI) allows users to easily operate the cloud infrastructure. A Cloud Manager API is available for advanced users to maintain the cloud environment. The Cloud Manager API provides advanced functions over the default OpenStack commands to control the allocation of resources, such as provisioning servers, resizing existing servers, providing network configurations, billing, accounting and metering support, and providing request and approval workflow support. Component model Table 3 lists the core components of the OpenStack framework. Component Code name Description Compute service Nova Provision and manage large networks of virtual machines, creating a redundant and scalable cloud computing platform. It is both hardware and hypervisor-agnostic, and has a distributed and asynchronous architecture providing fault tolerance and tenantbased isolation. Block storage Cinder Provides persistent block storage for virtual machine instances. The storage on the instances is nonpersistent; hence any data generated by the instance is destroyed once the instance is terminated. Cinder uses persistent volumes that are attached to instances for data longevity, and it is possible for instances to boot from a Cinder volume rather than from a local image. Virtual network Neutron OpenStack Networking is a pluggable "networking as a service" framework for managing networks and IP addresses. It supports several flexible network models, including Dynamic Host Configuration Protocol (DHCP) and VLAN. Image management Glance Provides discovery, registration, and delivery services for virtual disk images. The images can be stored on multiple back-end storage units and are cached locally to reduce image staging time. Authentication Keystone Provides a central and unified authorization mechanism for all OpenStack users and services across all projects. It supports integration with existing authentication services such as Lightweight Directory Access Protocol (LDAP). Telemetry Ceilometer Provides efficient collection of metering data, in terms of processor and network costs, to deliver a unique point of contact for billing systems to acquire all of the measurements needed to establish customer billing across all current OpenStack core components. An administrator can configure the type of data collected to meet operating requirements. Dashboard Horizon An extensible, web-based application that runs as a Hypertext Transfer Protocol (HTTP) service that enables cloud administrators and users to control and provision compute, storage, and Page 8 of 36

9 networking resources. Table 3. Core components Except for the core components introduced in Table 3, OpenStack also defines the concepts that help the administrator further manage the tenancy or segmentation in a cloud environment, as shown in Table 4. Name Tenant Availability Zone Host Aggregate Description Table 4. OpenStack tenancy concepts The OpenStack system is designed to have multitenancy on a shared system. Tenants (or projects, as they are also called) are isolated resources that consist of separate networks, volumes, instances, images, keys, and users. Quota controls can be applied to these resources on a tenant basis. In OpenStack, an availability zone allows a user to allocate new resources with defined placement. The instance availability zone defines the placement for allocation of virtual machines, and the volume availability zone defines the placement for allocation of virtual block storage devices. A host aggregate further partitions an availability zone. It consists of key-value pairs assigned to groups of machines and can be used in the scheduler to enable advanced scheduling. The components and concepts described in Table 3 and Table 4 are organized as shown in Figure 2. Page 9 of 36

10 Figure 2. IBM Cloud Manager with OpenStack component model Compute pool The compute pool (shown at the lower-right side of Figure 2) consists of multiple compute servers that are virtualized by OpenStack Nova to provide a redundant and scalable cloud computing environment. IBM Cloud Manager with OpenStack provides Nova drivers to enable virtualization on standard x86 servers and IBM Power and System z servers, with support for multiple hypervisors. Network addresses are automatically managed and associated with instances. Compute nodes inside the compute pool can be grouped to one or more host aggregate according to the business need. Storage pool A storage pool (shown at the lower-left side of Figure 2) consists of multiple storage servers that provide persistent block storage resources from their local drives. The OpenStack Cinder virtualizes the pool of these block storage devices and provides users with a single API interface to request and consume those Page 10 of 36

11 resources without requiring any knowledge of where their storage is actually deployed and how block storage volume is allocated. Unlike the ephemeral storage of deployed instances that reside on a compute pool, the persistent block storage is attached to the instance as a virtual iscsi device and managed independently of instances. Controller The controller (shown at the upper-left side of Figure 2) is a central administrative server where users can access and provision cloud-based resources from a self-service portal. It is also a proxy and schedules the services running on the compute pool and storage pool, and it contains important information in cloud environment settings or its database. Support services Support services (shown at the upper-right side of Figure 2) are peripheral servers that are running Domain Name System (DNS), DHCP, or Network Time Protocol (NTP) services that support the core services in the OpenStack cloud environment. There is also an optional utility for deploying the initial or incremental cloud environment with the Chef framework. Platform Resource Scheduler IBM Cloud Manager with OpenStack provides default schedulers for workload scheduling. For a more advanced scheduling schema, choose IBM Platform Resource Scheduler (PRS) instead (shown at the upper-left side of Figure 2). Platform Resource Scheduler consists of several add-on components, as detailed in the following list, to OpenStack that provide advanced resource scheduling and optimization functionality. This module is licensed separately and includes: Enterprise Grid Orchestrator (EGO) provides the underlying system infrastructure to control and manage cluster resources. EGO also manages the supply of resources, enabling multiple applications to share infrastructure according to a resource distribution plan. Scheduler Plug-in a resource scheduler add-on component that functions as the bridge between EGO and OpenStack. Runtime Policy engine a library acting as a simple, generic workload manager that uses EGO for resource monitoring and scheduling decisions to provide resource optimization with customized goals, such as packing, load balancing, and energy optimization by powering hypervisor on/off. Page 11 of 36

12 Platform Resource Scheduler is used for scheduling resources for OpenStack to make decisions on where to deploy an instance with specified resources and selection criteria, such as migrating and resizing, and identifying which resource needs to be optimized among the cluster. The placement policies are applied at deployment time, runtime, or both, depending on specific policy type, as given in the following list: Packing packs workloads on the fewest physical servers to reduce fragmentation and energy usage Striping spreads workloads across as many physical servers as possible to reduce the impact of failure Load-balance allocates physical servers with the lowest load to new workloads Memory-balance places VMs on the hosts with the most available memory Affinity specifies that certain VMs be placed on the same host or few hosts Anti-affinity places workloads close to critical resources, such as storage Resource overcommit intentionally overcommits resources to maximize utilization Operational model The IBM Cloud Manager with OpenStack reference architecture solution is implemented on various system components described in this section. The solution uses the IBM System x3650 M4, IBM System x3550 M4, and IBM System x3650 M4 BD servers installed with the Red Hat Linux 6.5 operating system, and IBM Cloud Manager with OpenStack 4.1. The network configuration consists of the IBM System Networking RackSwitch G8124E and G8052. IBM System x3650 M4 The IBM System x3650 M4 server is a 2U two-socket server with outstanding reliability, availability, and serviceability (RAS) and high-efficiency design for business-critical applications and cloud deployments. It offers a flexible, scalable design and simple upgrade path to 16 hard disk drives (HDDs) or solid-state drives (SSDs), with up to six PCIe Gen 3 slots and up to 768 GB of memory. Its onboard Ethernet solution provides four standard embedded Gigabit Ethernet ports and two optional embedded 10 Gb Ethernet ports without occupying PCIe slots. Combined with the Intel Xeon processor E v2 product family, the IBM x3650 M4 server offers an outstanding level of density and performance that lowers the total cost of ownership (TCO) per virtual machine. Its flexible design and great expansion capabilities solidify dependability for any kind of virtualized workload, with minimal downtime. Figure 3. IBM x3650 M4 Page 12 of 36

13 IBM System x3550 M4 The IBM System x3550 M4 server is a cost- and density-balanced 1U, 2-socket business-critical server with up to eight HDDs or SSDs, plus an optical drive and up to 768GB of memory. The onboard Ethernet solution provides four standard integrated Gigabit Ethernet ports and two optional embedded 10 Gb Ethernet ports without occupying PCIe slots. Its outstanding RAS and high-efficiency design improves the cloud environment and saves operational costs. The System x3550 M4 server is an ideal choice for a scalable cloud environment as a compact package without compromising performance and reliability. Figure 4. IBM x3550 M4 IBM System x3650 M4 BD The IBM System x3650 M4 BD server offers a cost-effective, high-capacity storage solution with exceptional energy-smart design, leadership virtualization, and powerful systems management. It supports up to fourteen 3.5-inch hot-swap drive bays and provides maximum internal storage density of up to 56 TB in a 2U form factor. The IBM x3650 M4 BD server is designed to provide exceptional value and flexibility to meet big data and storage virtualization requirements. By consolidating storage and server into one system, it offers easy management and saves floor space and power consumption for storage-demanding applications. Figure 5. IBM x3650 M4 BD IBM System Networking RackSwitch G8124E The IBM System Networking RackSwitch G8124E delivers exceptional performance that is both lossless and low-latency. In addition, RackSwitch G8124E delivers excellent cost savings and a feature-rich design when it comes to virtualization, Converged Enhanced Ethernet (CEE)/Fibre Channel over Ethernet (FCoE), Internet Small Computer System Interface (iscsi), high availability, and enterprise-class Layer 2 and Layer 3 functionality. With support for 10 Gb, this 24-port switch is designed for clients who are leveraging 10 Gb Ethernet already or have plans to in the future. It is the first top-of-rack 10 Gb switch for IBM System x designed to support IBM Virtual Fabric, which provides the ability to dynamically allocate bandwidth per virtual network interface card (vnic) in increments of 100 MB, while being able to adjust over time without downtime. Page 13 of 36

14 Figure 6. IBM RackSwitch G8124E IBM System Networking RackSwitch G8052 The IBM System Networking RackSwitch G8052 is a 1 Gb top-of-rack data center switch that delivers linerate Layer 2/3 performance at a very attractive price. It has 48 10/100/1000BASE-T RJ45 ports and four 10 Gb Ethernet SFP+ ports, and it includes hot-swap redundant power supplies and fans standard, minimizing your configuration requirements. Unlike most rack equipment that cools from side to side, the RackSwitch G8052 has a choice of rear-to-front or front-to-rear airflow that matches server airflow. Figure 7. IBM RackSwitch G8052 Deployment diagram and components There are many models for deploying an OpenStack environment. To achieve the highest usability and flexibility, you can assign each node a specific role and place the corresponding OpenStack components on it. The controller node plays the central role in the cluster, running IBM Cloud Manager with OpenStack, and has four roles in supporting OpenStack: The controller node acts as an API layer, and it listens to all service requests (Nova, Glance, Cinder, and so on).the requests first land on the controller node. Then the requests are forwarded to a compute node or storage node, through messaging services, for underlying compute workload or storage workload. The controller node is the messaging hub, where all messages follow and route through, for cross-node communication. The controller node acts as the scheduler to determine the placement of a particular virtual machine, based on a specific scheduling driver. The controller node acts as the network node, which manipulates the virtual networks created in the cloud cluster. However, it is not mandatory that all four roles reside on the same physical server. OpenStack is by design a natural distributed framework. Such all-in-one controller placement is used for simplicity and ease of deployment. In product environments, it is possible and might be preferable to move one or more of these roles and relevant services to another node for security, performance, or manageability concerns. On the compute nodes, the installation includes the essential OpenStack computing service (openstacknova-compute), along with the neutron-openvswitch-agent which enables software-defined networking. An optional metering component (ceilometer) can be placed to collect resource usage information for billing or auditing. On a storage node, the cinder-volume service is installed. It enables the storage node to allocate a logical volume (in LVM) in a local disk array and expose it as an iscsi target, which in turn is mounted Page 14 of 36

15 onto virtual machines as requested. Refer to Figure 8 for detailed placement of components on physical servers. Figure 8. Software components placement on the nodes Page 15 of 36

16 Deployment considerations This section describes some important considerations to keep in mind when deploying IBM System x reference architecture for IBM Cloud Manager with OpenStack. Controller, compute, and storage servers configuration Within the OpenStack cloud environment, managed by IBM Cloud Manager with OpenStack, System x3650 M4 and System x3550 M4 servers are used as controller and compute nodes, and System x3650 M4 BD servers are used as storage nodes. The System x3650 M4 dual-socket 2U rack server can be expanded to handle a variety of workloads. The x3650 M4 BD server is designed specifically for Big Data. Each System x3650 M4 BD server has 56 TB storage at maximum, which is ideal to fulfill the external storage requirement. The x3550 M4 dual-socket 1U rack server, when combined with high input/output operations per second (IOPS) SSDs, provides superior high I/O performance and outstanding workload density. Table 1 lists the functional requirements for this reference architecture. In this System x3550 M4 rack server configuration, two extra IBM S3700 SSDs are used with LSI MegaRAID CacheCade Pro 2.0 for read/write caching. IBM S3700 Serial Attached Technology Attachment (SATA) 2.5-inch MLC Enterprise SSDs for IBM System x employ MLC NAND flash memory with High Endurance Technology and a 6 Gbps SATA interface to provide an efficient solution with industry-leading performance. They can be fully rewritten up to ten times per day throughout their entire five-year life expectancy. SSDs are used as a dedicated pool of Redundant Array of Independent Disks (RAID) controller cache, to improve the performance of the disk array of a server s six HDDs. Table 5, Table 6, and Table 8 outline the server configurations, all configured with network high availability. The System x3650 M4 server is configured for general workloads in this reference architecture. It is balanced on computing, memory and storage, and cost competitive at a per VM base. Memory or disks can be added to accommodate workloads that require additional memory or capacity. Component Specification Quantity Description Processor Intel E v2 2.6GHz 2 8-core high-performance processor Memory 1866MHz 16 GB RDIMM 8 128GB total Drive 600 GB 10k RPM SAS HDD 8 RAID-10 1 Gb NIC Onboard Gigabit Ethernet Controller 1 4 integrated Ethernet ports 10 Gb NIC Dual Port 10GbE SFP+ Ethernet Controller 1 Dual-port Ethernet adapter Table 5. x3650 M4 compute node/controller node Page 16 of 36

17 The System x3550 M4 server in this reference architecture is configured for I/O intensive workloads to maximize performance of storage. It offers an ideal combination of HDD s capacity and SSD s high performance. Component Specification Quantity Description Processor Intel E v2 2.6GHz 2 8-core high-performance processor Memory 1866MHz 16 GB RDIMM GB total Drive 200 GB Enterprise SSD 2 As caching devices for RAID controller Drive 900 GB 10k RPM SAS HDD 6 RAID-10 1 Gb NIC Onboard Gigabit Ethernet Controller 1 4 integrated Ethernet ports 10 Gb NIC Dual Port 10GbE SFP+ Ethernet Controller 1 Dual-port Ethernet adapter Table 6. x3550 M4 compute node The performance testing results in Table 7 demonstrate substantial improvement of I/O performance with SSD caching when utilizing SSD hardware and caching software. It makes this System x3550 M4 configuration a cost-effective option for I/O-intensive workloads, because it can sustain a high density of this kind of workloads. Disk groups Random Read IOPS (4KB) Random Write IOPS (4KB) 8 HDDs (RAID-10) HDDs (RAID-10) with 2 SSDs (RAID 1 in CacheCade) % improvement % improvement Table 7. Performance with SSD cache The System x3650 M4 BD server provides large storage capacity with up to 14 HDD drives. Storage capacity can be shared by multiple compute nodes by exposing its local disk array through OpenStack Cinder service as virtual iscsi devices. The four SSDs are also used as caching devices to achieve optimal performance. Component Specification Quantity Description Processor Intel E v2 2.1GHz 1 6-core high-performance processor Memory 1866MHz 16 GB RDIMM 4 64 GB total Drive 200 GB Enterprise SSD 4 As caching devices for RAID controller Drive 4TB 7200 RPM SATA HDD 10 RAID-10 1 Gb NIC Onboard Gigabit Ethernet Controller 1 3 integrated Ethernet ports 10 Gb NIC Dual Port 10GbE SFP+ Ethernet Controller 1 Dual-port Ethernet adapter Table 8. x3650 M4 BD, storage node When building an OpenStack cloud around these hardware options, more than one server of each type is required to meet the minimum availability requirements. Page 17 of 36

18 Networking Combinations of physical and virtual isolated networks are configured at the host, switch, and storage layers to meet isolation requirements. Different network data traffic can be physically isolated into different switches according to the physical switch ports and server ports assignment. In a single physical switch, VLANs are used to provide logical isolation between the various management, storage, and data traffic. A key element is to properly configure the switches to maximize available bandwidth and reduce congestion. However, based on individual environment preferences, there is flexibility regarding how many VLANs are created and what type of role-based traffic they handle. The Logical network configuration section presents VLANs assignment reference. There are two kinds of VLAN configuration modes switch ports that are used for the message queue and iscsi traffic, and the management network that can be configured in access mode. This limits that port to a single VLAN, and the Ethernet frame is tagged with the VLAN ID at the switch. The switch ports used for the compute node virtual switch connection must be configured in trunk mode. It allows for more than one VLAN tagging package to pass in a physical switch port. In this case, different instances can be isolated in different networks using different VLANs. Logical network configuration The VLAN settings are configured as shown in Table 9 and Table 10 for this reference architecture. Each VLAN is used to create the traffic isolation based on traffic type, and to isolate inbound and outbound management traffic from other traffic on the internal corporate network. Every network is unique. So your VLAN configuration addressing scheme must be configured to fit your specific enterprise needs. Network Name Description VLAN 10 Internal data network Used for iscsi traffic and controller message communication VLAN 11 Switch Management Network Used for switch management communication Table 9. G8124E 10 Gb Ethernet switch VLAN settings Network Name Description VLAN 11 Internal management network Used for switch and host IMM management communication VLAN Instance private network Used for isolating the tenants networks Table 10. G Gb Ethernet switch VLAN settings Solution sizing The following section outlines sizing guide for different workloads and scenarios. Page 18 of 36

19 VM workload definition This reference architecture assumes the VM workloads as shown in Table 11. The mixed workload is calculated by blending small, medium and large VM flavos. This information serves as a baseline only to measure your real workload and is not meant to represent any specific application. In any deployment scenario, workloads are unlikely to have the same characteristics, and workloads might not be balanced between hosts. You must use the appropriate scheduler for compute and storage and measure its real performance to meet the target service level agreement (SLA). vcpu vram Storage IOPS 2 6 GB 100 GB 100 Table 11. VM workloads parameters When sizing the solution, calculate the amount of resources required based on the amount of workload expected, multiplied by maximum size of that workload, plus a 10 percent buffer for management overhead. For better resource utilization, consider putting similar workloads on the same guest OS type in the same host to use memory or storage de-duplication. For example: Virtual CPU (vcpu) = Physical Cores * CPU allocation ratio 1 (4:1), 2 Virtual Memory (vram) = Physical Memory * (RAM allocation ratio ) * (100% - OS reserved) Using the formula above, you can calculate the usable virtual resources from the hardware configuration of compute nodes, as shown in Table 12. System x3650 M4 System x3550 M4 vcpu 16 Cores * 4 = 64 vcpu 16 Cores * 4 = 64 vcpu vram 128GB*150%*(100%-10%)= 172GB 128GB*150%*(100%-10%) = 172GB Storage Space Storage IOPS (r/w mixed) Table 12. Virtual resource 8*600GB(RAID-10) = 2.4 TB ~3000 ~5200 6*900GB(RAID-10)+SSD Cache = 2.7 TB 1 CPU allocation ratio indicates how many virtual cores can be assigned to a node for each physical core. 4:1 is a balanced choice for performance and cost effectiveness. 2 RAM allocation ratio is used to allocate virtual resources in excess of what is physically available on a host, through compression or de-duplication technology. The hypervisor uses it to improve infrastructure utilization the RAM allocation ratio = (virtual resource / physical resource) formula. Page 19 of 36

20 Using the mixed workload requirement generated from Table 11(2.15 vcpu, 5.9 GB vram, 100GB local disk, 100 IOPS), it is easy to calculate the expected VM density (the smallest number determines the number of VMs that can be supported per compute server, that is, 24 VMs per host), as shown in Table 13: Page 20 of 36

21 VMs / x3650 M4 VMs / x3550 M4 CPU Memory Disk Space Disk IOPS 64/2 = /6 = T / 100GB = 24 ~3000/100 = 30 64/2 = /6 = T / 100GB = 27 ~5200/100 = 52 Table 13. Resource-based VM density By applying the calculated density of a single host, this reference architecture can be tailored to fit various sizes of cloud cluster, as shown in Table 14: Entry Medium Large Controller Node N/A 3 One System x3650 M4 One System x3650 M4 Compute Node Four System x3550 M4 Four System x3650 M4 Three System x3550 M4 Storage Node One System x3650 M4 BD Two System x3650 M4 BD Eleven System x3650 M4 Four System x3550 M4 Two System x3650 M4 BD Total VMs Total IOPS Local: ~20000 Table 14. Deployment sizes Network virtualization Shared: ~7000 Local: ~28000 Shared: ~14000 Local: ~53000 Shared: ~14000 For network isolation between tenants, the VLAN can be used (up to 4096 IDs). Virtual Extensible LAN (VXLAN) can be used if the tenant number in the cloud environment is expected to exceed the maximum number of IDs. VXLAN is a network virtualization technology that addresses scalability problems associated with large cloud computing deployments by using a VLAN-like technique to encapsulate MACbased OSI layer 2 Ethernet frames within layer 3 UDP packets. Open vswitch (OVS) is a fully featured, flow-based implementation of a virtual switch and can be used as a platform in Software-Defined Networking (SDN). It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (such as NetFlow, sflow, SPAN, RSPAN, CLI, LACP, and 802.1ag). Open vswitch supports VXLAN overlay networks. It takes Layer 2 traffic and encapsulates it into Layer 3 packets to extend the 4096 tag 3 Use one of the System x3550 M4 nodes for both controller node and compute node Page 21 of 36

22 limitation in network isolation and extend the Layer 2 virtual network across physical boundaries between different data centers in a Layer 3 network. It is the default network service in this reference architecture. Page 22 of 36

23 System availability When an error occurs in the system, the source of the fault can be either hardware or software dependent. Failure of a component leads to different impacts on the system, which is generally summarized as follows: Power Distribution Unit (PDU) failure Power supply is redundant, allowing for failure of one PDU and no loss of power. Compute node failure Causes all VMs running on the node to fail. However, the failure does not affect the VM create operation. The VM can still be provisioned on the compute node that remains functioning. Controller node failure Causes all services running on the node to fail. The VMs running on the cluster are still running and accessible, but cannot be manipulated until the controller node is recovered. Storage node failure Causes VM loss connectivity to the volumes provisioned by the failed storage node and consequent errors in the guest OS. However, it does not affect the volume create operation; volumes can still be provisioned on the storage node that remains functioning. Network Interface Card (NIC) failure Causes a single interface to a compute node or storage node to fail, along with the associated paths through that network interface to the compute node or storage node, to fail. The connectivity to that node remains, but the bandwidth, either external or internal, is cut in half. However, half of the traffic to the host is rerouted via the switch aggregation cable. This also applies to a failure of the Ethernet cable connected to this interface. Switch failure Causes an associated path through that switch to the compute node or storage node to fail. The connectivity to that node remains, but the bandwidth, either external or internal, is cut in half. Switch aggregation cable failure Causes an associated path between two switches to fail. Disk failure Loses only one redundant copy of data in the RAID-10 array. In most cases, the cloud environment is still operational despite errors that persist in the system. The scheduler can discover the broken service and automatically route the new resource request to the remaining functional compute nodes or storage nodes. The workload on the failed node is disconnected. After the hardware or software component is replaced or repaired, a restart of OpenStack services and related system services might be required to start the restored component and join the system. The controller node can be a single point of failure (SPOF). However, there are solutions that can be used to make it more robust, such as regular backup configuration and database, mounting shared storage to the controller node, building a high-availability controller cluster, and so on. Such solutions are out of the scope of this document. Full rack system sample The three types of server configurations previously described can be built up to a full rack. Page 23 of 36

24 Figure 9 and Table 15 show a balanced configuration of compute, storage, networking and power. VM capacity is calculated on the number of hosts in the solution. Refer to the Solution sizing section for information on how to estimate the VM density. Because it is impractical to take into account every possible combination of these options, in a real scenario, you should modify this rack to best fit the workloads that are running on your cloud environment. Components Capacity Power 4 PDUs Networking 10 Gb VM capacity KVM and Monitor >360 (medium workload) 1 Controller node Compute node 1 15 Storage node 2 Local storage 37 TB (RAID-10) Local IOPS >100 External storage 40 TB (RAID-10) Table 15. Full rack system capacity Page 24 of 36

25 Figure 9. Full rack system configuration Page 25 of 36

26 Figure 10 shows the corresponding network topology for such a rack system. Figure 10. Network topology diagram At the physical host layer, there are five 1GbE Ethernet devices (onboard NICs with one port for the IMM, two ports for external access, and two ports reserved) and two 10GbE Ethernet devices for each cloud compute node (one Emulex 10 Gb NIC with two Ethernet ports, for internal access). At the storage node, there are four 1GbE Ethernet devices (onboard NICs with one port for the IMM and three ports reserved), and two 10GbE Ethernet devices for each cloud compute node (one Emulex 10 Gb NIC with two Ethernet ports, for internal access). At the physical switch layer, there are two redundant G port, 1GbE Ethernet switches for VM connectivity/external access and internal management, and two G8124E 24-port, 10GbE Ethernet Page 26 of 36

27 switches for controller node, compute node, and storage node connectivity. Linux NIC bonding is used to provide fault tolerance to the communication networks. Ethernet traffic is isolated by type through the use of VLANs (virtual LANs) on the switch. Solution scaling and expansion The following section outlines compute node scale up, multi rack expansion, and storage scale out. Compute node scale up To obtain higher VM density and IOPS, you can choose an upgraded System x3650 M4 model. Table 16 shows three kinds of compute node configurations based on System x3650 M4. Component Standard Configuration Advanced Configuration Premium Configuration Processor 2 x 8C 2 x 8C 2 x 10C Memory 128 GB 192 GB 256 GB Disk 8x600 GB 10k RPM 12x600 GB 10k RPM 16x600 GB 10k RPM 1 Gb NIC Gb NIC VMs IOPS >3000 >3600 >4200 Table 16. Standard and enhanced compute node configurations based System x3650 M4 Multiple rack expansion If the workload demands exceed the capacity of one rack, you can have this OpenStack cloud span multiple racks to increase its capacity by adding more racks. There are two options for deploying a controller node. You can either give each rack its own controller node, so that resources between racks are isolated, or you can allow a single controller node in the first rack to manage all nodes in multiple racks centrally, so that resource scheduling crosses the rack boundary. In the latter case, deactivate API and scheduler services on the controller on other racks and point all services to the address of the controller node in the first rack, as shown in Figure 11. Each rack has its own PDU and TOR switches and can operate separately regardless of the health status of other racks. This is set in /etc/nova/nova.conf as node_availability_zone = {zone name}. You can create other availability zones based on business need. The details for creating and managing host aggregate or availability zones and other services are not covered in this document. Page 27 of 36

28 Figure 11. Software components deployed across racks Storage scale out To increase shared storage space, simply add new storage nodes into the cluster and duplicate the configuration of the existing storage node, without any administration effort or rebooting the storage service. The increased storage capacity is automatically discovered by the controller and is added into the existing storage pool. Authorization and automation IBM Cloud Manager integrates comprehensive workflow management functions, such as when a new user needs to request an account from the system administrator to be a member of a tenant (a synonym of project in OpenStack). Operation authorization can be applied to a project so that deployment and other actions are first approved by the system administrator. There is also a dashboard that displays the requesting user, request status, action that is requested, and date of the request. For example, users can withdraw a request from the approval queue at any time. If an administrator rejects or modifies a user request, the administrator can add comments so the user can resolve any issues and resubmit the request. For recycling of resources, IBM Cloud Manager has a configurable task that allows administrators to identify expired instances and the length of time for which to keep them after they have expired. After they have expired, instances are destroyed automatically. Best practices A set of proven planning and deployment techniques contributes significantly to successful OpenStack deployment and operation. Proper planning includes sizing of needed server resources (CPU and memory), storage (space and IOPS), and core software services to support the infrastructure. This Page 28 of 36

29 information can then be implemented using following best practices to achieve optimal performance and availability for the solution. When implementing the solution, measure the actual resource usage of workload first and adjust the memory or disks accordingly to avoid imbalanced resource utilization. If the controller node is getting saturated, try to move services in the controller node to other nodes. In this case, use separate host name for the address of each service, for example, ER/nova', even though they might point to the same IP most of the time. Perform periodic backup of the database and configuration in the controller node to minimize service downtime in case of hardware failure. Set the compute and storage quota on each tenant properly, so that the resource is not exhausted due to a misconfigured or resource-starved virtual machine. To allow for planned maintenance or unplanned failures, provide enough physical server resources to handle all VMs in an N-1 configuration. Before planned maintenance, gracefully shut down the running services to avoid inconsistent states or data corruption. Because iscsi traffic and other data traffic between compute nodes and storage nodes share the same 10Gb network link, enable the Jumbo Frame feature both on the host and the switch to allow the maximum throughput. If two switches are connected using the Inter-Switch Link (ISL) protocol, the port number for the inter-switch connection must be at least two or more than the ports used in each switch to maintain network performance in case a single switch or its uplink fails. Use different VLANs for different data traffic, and to isolate traffic between tenants. If more than 4090 VLANs are required, consider a VXLAN solution instead, for tenant isolation. This reference architecture, combined with IBM s enterprise-class hardware and software, prepares IT administrators to successfully meet virtualization performance and growth objectives by deploying clouds efficiently and reliably. Page 29 of 36

30 Appendix 1. Bill of materials (BOM) This sample bill of materials lists the typical components of the IBM System x solution for IBM Cloud Manager with OpenStack. PN Description Quantity Controller or Compute x3550 M4 7914AC1 IBM System x3550m4 4 A1H3 IBM System x3550 M4 2.5" Base Without Power Supply Select storage devices - no IBM-configured RAID required 4 A347 ServeRAID M5110 SAS/SATA Controller for IBM System x 4 A1WY ServeRAID M5100 Series 1 GB Flash/RAID 5 Upgrade for IBM System x 4 A282 IBM 900 GB 10K 6Gbps SAS 2.5" SFF HS HDD 24 A4FL S GB SATA 2.5" MLC HS Enterprise SSD for IBM System x 8 A228 IBM System x Gen-III Slides Kit 4 A229 IBM System x Gen-III CMA 4 A1HG x3550 M4 4x 2.5" HDD Assembly Kit 4 A1HN x3550 M4 plus 4x 2.5" HDD Assembly Kit 4 A1H6 IBM System x 550W High Efficiency Platinum AC Power Supply 4 A1HL x3550 M4 PCIe Gen-III Riser Card 2(1 x16 FH/HL Slot) 4 A1HJ x3550 M4 PCIe Riser Card 1 (1 x16 LP Slot) 4 A4XH Emulex Dual Port 10GbE SFP+ VFA IIIr for IBM System x 4 A1HP System Documentation and Software in US English 4 A3QL 16 GB (1x16 GB, 2Rx4, 1.5V) PC CL MHz LP RDIMM 32 A2U6 IBM System x Advanced Lightpath Kit IBM Single Cable USB Conversion Option (UCO) m, 10A/ V, C13 to IEC 320-C14 Rack Power Cable 4 A3WR Intel Xeon Processor E v2 8C 2.6GHz 20 MB Cache 1866MHz 4 A3X9 Intel Xeon Processor E v2 8C 2.6GHz 20MB Cache 95W W/Fan Rack Installation of 1U Component U Bracket for Emulex 10GbE Virtual Fabric Adapter for IBM System x 4 A22C ServeRAID M5100 Series 875mm Flash Power Module Cable 4 A1HB x3550 M4 System Level Code 4 Page 30 of 36

31 PN Description Quantity A1HD x3550 M4 Agency Label GBM 4 A3XM IBM System x3550 M4 Planar (Refresh) 4 Controller or Compute x3650 M4 7915AC1 IBM System x3650 M4 12 A1JT x3650 M4 PCIe Riser Card 1 (1 x8 FH/FL + 2 x8 FH/HL Slots) 12 A4XH Emulex Dual Port 10GbE SFP+ VFA IIIr for IBM System x 12 A1H5 IBM System x 750W High Efficiency Platinum AC Power Supply m, 10A/ V, C13 to IEC 320-C14 Rack Power Cable Select Storage devices - no IBM-configured RAID required 12 A2N2 ServeRAID M5110e SAS/SATA Controller for IBM System x 12 A1WY ServeRAID M5100 Series 1 GB Flash/RAID 5 Upgrade for IBM System x 12 A2XD IBM 600 GB 10K 6Gbps SAS 2.5" SFF G2HS HDD 96 A3VC Intel Xeon Processor E v2 8C 2.6GHz 20 MB Cache 1866MHz 95W 12 A3VW Intel Xeon Processor E v2 8C 2.6GHz 20 MB Cache 95W 12 A1KF IBM System x3650 M4 2.5" Base without Power Supply 12 A1JX x3650 M4 8x 2.5" HS HDD Assembly Kit 12 A3V6 IBM System x3650 M4 Planar (IVB Refresh ) 12 A228 IBM System x Gen-III Slides Kit 12 A229 IBM System x Gen-III CMA 12 A1JZ System Documentation and Software-US English 12 A2U6 IBM System x Advanced Lightpath Kit IBM Single Cable USB Conversion Option (UCO) 12 A3QL 16 GB (1x16 GB, 2Rx4, 1.5V) PC CL MHz LP RDIMM Rack Installation >1U Component 12 A22C ServeRAID M5100 Series 875mm Flash Power Module Cable 12 A1L7 x3650 M4 Power Supply Filler 12 A1LA x3650 M4 System Level Code 12 A1LB x3650 M4 Agency Label GBM 12 A2R0 X3650 M4 Mini SAS Cable 820MM 12 Storage Node x3650 M4 BD Page 31 of 36