Building a Flexible OpenStack* Cloud from the Ground Up

Transcription

1 white paper Building a Flexible OpenStack* Cloud from the Ground Up Contributors Dr. Yih Leong Sun Senior Software Cloud Architect, Intel OTC Michael Kadera Cloud and Data Center Manager, Intel OTC John Geier Cloud & Data Center Engineer, Intel OTC Table of Contents Defining Requirements and Constraints for the Architecture...1 Planning for Growth of the Environment...3 Entry Stage...3 Intermediate Stage...3 Enterprise Stage Establishing the Rack-Level Design...4 Specifying Requirements against Available Resources...4 Per-Rack Power Usage...4 Rack Density...4 Refinements to the Rack-Level Design...4 Compute Resources...5 Management Resources...5 Networking Resources...5 Providing a Flexible Lab Topology...6 Compute Flexibility...6 Storage Flexibility...6 Network Flexibility...6 Deployment Mechanisms...6 Design Best Practices for Flexible OpenStack Clouds...6 Compute Best Practices...6 Storage Best Practices...7 Network Best Practices...7 Deployment and Management Best Practices...7 Conclusion...7 This paper describes the process of designing and building a flexible OpenStack* environment that takes advantage of Intel platform features and supports open-ended customer requirements. It outlines steps taken at the Intel Open Source Technology Center to create a lab cloud that supports OpenStack development. A well-designed OpenStack environment must not only meet the present needs of the developers and end users it supports but also offer the flexibility and scalability to satisfy future requirements as broadly as possible. It follows that, while balancing those factors with budget limitations and other practical constraints is vital, the common practice of starting the design process with a budget discussion is not ideal. Instead, it is a best practice to begin by identifying what the cloud needs to do and creating a preliminary design to meet those needs. Once that work is complete, practical limitations can be thoroughly identified and accommodated as the design is refined and finalized. This focus helps preserve meeting the present and future capabilities needed from the environment as the core priority of the process. A well-designed, flexible cloud provides long-term support for end users and their applications as their needs evolve. Just as the business it supports is dynamic and ever-shifting, an OpenStack environment must be able to adapt to the need for ongoing change. Defining Requirements and Constraints for the Architecture Identifying core design requirements begins by establishing what users will need to accomplish using the OpenStack environment being built. Some examples of the types of capabilities to be supported by the compute, storage, and network infrastructure are illustrated in Figure 1. At this stage, designers must consider beyond the known applications to be hosted on the system, including factors such as runtimes, data access, and services for identity and asset management. As usage scenarios are outlined, interlocking sets of requirements are typically revealed. For example, one aspect of service monitoring relates to charging for usage, if applicable, including chargeback to other business units within a company or billing to external users. Depending on the details of these arrangements, they may have implications with regard to Sarbanes-Oxley or other regulatory frameworks. Data access, export restrictions, and privacy regulations may also

2 Building a Flexible OpenStack* Cloud from the Ground Up 2 impact multi-tenancy strategies, resulting in the logical or physical partitioning of users and data to remain compliant. Compliance efforts, in turn, will impact mechanisms for how users, applications, and systems access specific resources and how rights that govern that access are established and maintained. Additional aspects of gathering requirements include the following: Devices and external systems to be supported, including clients such as PCs, smartphones, and endpoints on the Internet of Things, as well as cloud-based resources and enterprise back-end systems. Data and infrastructure interoperability, including functionality related to analytics and interactions between public and internal big data resources. Orchestration of resources and operational management, maintaining optimal functioning of both physical and virtual resources, including facilitating end-user support for breakfix and adds, moves, and changes. Making this process more complex, each of these factors must take into account not only current needs but also projections for the future. In the case of the lab cloud at the Intel Open Source Technology Center, the purpose was to support internal developers and architects globally as they write, test, and push OpenStack code upstream to the community at large. In this sense, Intel s production environment can be a downstream consumer of the work done in this lab and by other contributors. The corresponding goals of this work are to enhance the growth and stability of OpenStack for the enterprise, making the use of cloud resources as simple and effective as possible. The lab-cloud development effort also includes work to enable enterprise capabilities such as rolling upgrades, bare metal provisioning, and service plane high availability (HA), as well as Intel platform capabilities such as trusted compute pools, Intel AES New Instructions, Intel Advanced Vector Extensions 2, and single-root I/O virtualization. In addition, design projects such as the Intel Power Thermal Aware Solution help foster efficient management strategies for the data center based on the latest platforms. The lab environment also supports enablement for cloud-related technologies such as the Data Plane Development Kit (DPDK), Open vswitch*, and OpenDaylight*. This effort is part of Intel s ongoing commitment to expanding its participation in OpenStack and related projects. Because the user base for this lab environment is global, the ability to make changes to the environment s configuration remotely on demand is critical. To avoid the need for physically moving cables and equipment to support ongoing project-specific changes, the team established that capabilities such as KVM/PDU over IP, Intelligent Platform Management Interface (IPMI), managed switches, and software-defined infrastructure must play a significant role in the solution. Desktops Cloud Servers Smartphones Tablets Figure 1. Example requirements for the architecture. Devices Monitoring Content Finance Communications Identity Data IoT Devices Runtime Laptops Asset Management Capabilities Compute Storage Network Infrastructure

3 Building a Flexible OpenStack* Cloud from the Ground Up 3 Planning for Growth of the Environment The design of an OpenStack environment must be based on a scalable rack design that allows both the cloud s capacity and its corresponding features and capabilities to grow over time. This progression also corresponds to the reality of budget limitations, which may significantly limit what an organization is able to build in the early stages of an OpenStack deployment. Nevertheless, fiscal soundness requires that the initial design consider the long-term lifespan of the environment, which may be considered in a three-stage model, as illustrated in Figure 2. Enterprise Entry Intermediate Single Availability Zone Scale Single Availability Zone Multiple Availability Zones Provisioning Minimal Dual Availability Zones Rolling Upgrades High Availability Single Availability Zone Rolling Upgrades Enhanced Scalability Core Functionality Figure 2. Growth phases of the solution. Operational High Availability Flexible Software-Defined Netowork and Storage Entry Stage A rudimentary deployment of OpenStack at the entry stage includes initial provisioning and getting core cloud functionality in place, typically in a single rack comprising a single availability zone. One could consider this a minimum viable cloud, meaning the minimal set of capabilities required to deliver basic cloud capabilities while remaining fully extensible and scalable for long-term evolution and growth. The physical infrastructure of this relatively simple implementation serves as a building block that can be used as a template for additional racks as they are added later. To support user needs, failover testing for HA may be undertaken at this stage. Intermediate Stage An expansion of the initial OpenStack rollout uses multiple racks and establishes the scalability of services the environment offers. This may include the fundamental use of a secondary availability zone, including the intentional introduction of network latency as part of application testing. Initial testing of OpenStack rolling upgrades may take place at this stage within a single availability zone, allowing administrators to upgrade each component, piece by piece, in support of the goal of migrating to the most up-to-date version of OpenStack with minimal downtime. Testing of operational HA allows for improving the efficiency of patching and other updates to the system by minimizing the delay associated with moving virtual machines (VMs) around when taking physical nodes offline. Optimizing this functionality before the environment becomes too large can generate significant savings later. Enterprise Stage As the OpenStack environment continues to grow, it may encompass a large number of racks, each typically patterned after the design established in the entry stage. The environment may become significantly more complex at this stage, extending to multiple availability zones or regions. Availability zones can be used to divide a compute deployment into multiple logical groups that provide redundancy at the power supply or network layer; regions provide a more discrete separation across multiple sites. Refinements to support for rolling upgrades play a significant role in ensuring that the data plane and control plane remain available at all times. A significant and ongoing challenge at the enterprise stage is to scale the environment s capabilities to push the limits of OpenStack services with massive data sets and extensive groups of users. As the scope of workloads and other requirements becomes more extensive and complex, software-defined networking and storage help the environment gracefully handle the need for change while taking optimal advantage of physical data center resources.

4 Building a Flexible OpenStack* Cloud from the Ground Up 4 Establishing the Rack-Level Design The preliminary physical rack-level design for the OpenStack lab cloud was based on an ideal case and did not account for constraints such as those associated with a specific physical facility or budget. It included 30 servers per rack, with one full-rack battery backup unit (BBU) sufficient to support all of the components in two server racks. This design enabled high rack density and a small overall physical footprint. Management infrastructure included KVM and PDU capabilities that could be carried out over IP, as well as IPMI, to allow administration from anywhere in the world. Specifying Requirements against Available Resources Refinement of the preliminary design considered factors that ranged from power and cooling capacity to the weight capacity of the raised-floor room where the racks would be housed and the available budget. To facilitate these calculations and measurements, a spreadsheet was developed to document the cost, electrical power requirements, and weight of all components that could potentially be used in the solution. These included the entire bill of materials. Per-Rack Power Usage To gauge the ability of available power sources to support a given hardware configuration, it is necessary to consider all components of the solution, including supporting hardware such as local disk drives. The design must consider both power consumption while all hardware is operating at maximum capacity and usage spikes when equipment is starting up. Further, the total consumption budget must include an extra margin to allow for changes to the system configurations over time. Considering all those factors, power usage may be determined by either of two methods: Calculated, using specifications published by the equipment manufacturer. Measured, based on samples of the equipment similar to that intended to be used in the final design. Either of these approaches requires consideration of all system components, including options such as various backplanes and add-in cards, as well as the inclusion of switches and other ancillary parts of the environment. In determining the power requirements for this project, the team elected to use a combination of these methods to enable a more comprehensive approach. For example, the team was unable to test some parts of the environment such as switches and KVMs at full load without having final hardware configurations or full quantities of equipment available. To overcome that limitation, the team relied on manufacturer specifications. On the other hand, both methods were used to determine power consumption of the servers. The facility available to house the lab systems provided a power budget of 60 amps per rack, which was a key consideration in terms of how much equipment could be used per rack. To test power draw (including during usage spikes), the team used Prime95* for Linux*, a stress-testing tool designed to place load on all a system s processor cores and memory with heavy use of both integer and floatingpoint instructions. A custom script was used to read and write quasi-random data to the local drives. Rack Density In addition to power availability, the other primary factors that determine how much equipment can be placed in each rack are the cooling and weight capacity of the physical facility. Cooling requirements are largely a factor of power consumption, and sufficient cooling capacity is necessary in order to utilize the full power budget, regardless of the amperage available in the facility. Weight is especially a factor in the case of a raised-floor environment (which was used for this project). The full-rack BBUs in particular weighed approximately 2,000 pounds each, far exceeding the 1,000-pound capacity for the corresponding footprint on the raised floor. Networking equipment also contributes to resource requirements, particularly in terms of power draw and cost, which may help dictate what network capabilities are available for the solution. Some of the calculations around rack density require a relatively specialized skill set, so the team consulted with a facilities engineer. Refinements to the Rack-Level Design Based on the findings of the refinement process described above, the team found that the power, cooling, and weight constraints required the number of systems per rack to be decreased from 30 to 15. In addition, the weight of BBUs led to the limitation that only network and management resources (that is, switches and KVM devices) in the solution are supported. Cost efficiency called for the use of BBUs with capacity sufficient to handle three racks per unit. The high-level hardware configuration of racks in the solution is illustrated in Figure 3. Smart Power Distribution Units (2x per rack) 1x 48-port 10GbE switch (SFP+, managed) 2x 24-port 1GbE switches (unmanaged) 5x 1U servers (8 drive bays each) 10x 2U servers (26 drive bays each, enterprise RAID) Battery Backup Unit (1x per three racks) KVM console (1x per rack) KVM over IP (1x per rack) Figure 3. Hardware components in each rack.

5 Building a Flexible OpenStack* Cloud from the Ground Up 5 Compute Resources The initial design includes five 1U servers and ten 2U servers per rack, based primarily on cost and weight considerations. The 1U servers are used for compute and light storage, while the extra drive bays in the 2U servers provide flexibility for increased storage capacity in addition to compute, with the ability to add more drives in the future without affecting the rack design. This increase in drives was also included in the power-consumption budget. The 1U systems were calculated with 8 bays full, and the 2U systems were calculated with 26 bays full. The 2 TB SATA drive power requirements were used for all drive-expansion calculations. The smaller physical size of the 1U servers also leaves space in each rack for potentially adding equipment to the solution in the future. For example, this equipment might include adding 4U servers based on the Intel Xeon processor E7 product family or additional networking capacity. Each server is based on two Intel Xeon processors E v3, with a clock speed of 2.3 GHz, 45 MB cache, and 18 cores per socket. Other main components include 128 GB of DDR4 RAM, four 10-Gigabit Ethernet (10GbE) ports and two Gigabit Ethernet (1GbE) ports, a combination of solid-state drives (SSDs) and SATA-based hard disk drives (HDDs), a trusted platform module, and a remote management module. While only three of the 10GbE ports per node are currently in use, cost efficiencies made it a sound decision to buy two-port network adapters, providing additional future capacity. Management Resources The solution s management infrastructure includes a 16- port KVM to support the 15 servers with remote BIOS access and remote boot media. Each rack is also supplied with two network-controllable power distribution units (PDUs), which support power cycling and power metering, on a pernode, per-pdu-circuit, and total-pdu basis. IPMI capability is also included to provide advanced autonomous server management and monitoring. Networking Resources The solution s primary networking infrastructure is based on one 48-port managed 10GbE switch per rack, which provides capacity for three connections to each server. Because of the reduction from the preliminary design to the final one of 30 servers per rack to just 15, less switch capacity is needed, which allowed the team to buy a more sophisticated switch than might otherwise be possible. As a result, the solution has a highly flexible set of capabilities for VLANs, allowing one each for public (lab network), management, and storage and backup. The numbers of network ports on each server could also be increased dramatically, if desired, such as by using four-port converged network adapters. By increasing switch capacity as well, many additional use cases could be explored. For example, NIC bonding on each subnet could provide higher throughput and redundancy for mission-critical usages, while multiple 10GbE switches could provide redundancy in case one of the switches failed. In addition to the 10GbE switch, each rack is outfitted with two 1GbE switches that support management and related functionality. The first of these provides a private network for the KVM, PDUs, and BBUs. The second 1GbE switch is used for the baseboard management controller (BMC) services and out-of-band connections. 10GbE Lab Network VLAN Management VLAN Storage VLAN (Optional) Use-Case VLAN 1GbE 1GbE KVM/PDU/BBU Network BMC Network Figure 4. Networks included in the solution design.

6 Building a Flexible OpenStack* Cloud from the Ground Up 6 Providing a Flexible Lab Topology The lab design offers the flexibility for supporting multiple development configurations, which can have a compute, storage, or network focus. The ability to remotely provision and configure the environment according to the needs of a given project contributes to this flexibility. Compute Flexibility The use of standardized servers helps create a homogeneous environment where any node can stand in for any other one, allowing repeatability by groups running tests on different hardware within the lab. This consideration is behind the approach of configuring both the 1U and the 2U servers with the same processors, memory, storage, and networking hardware as well. Significant flexibility is provided by the fact that the 15 nodes in each rack can be provisioned as needed for various usage scenarios. Specifically, three nodes are typically set aside for use as controllers, and the remaining 12 servers can be set up for any combination of compute, storage, and networking resources desired and for infrastructure support such as network booting, bastion services, and provisioning. Storage Flexibility As mentioned above, significant space, power, and cooling is available on the systems and in the rack to expand local storage as emerging needs dictate, with as many as 26 drive bays included in the 2U servers, of which only six are currently in use. Adding more hot-swappable HDDs or SSDs can therefore easily and inexpensively provide additional local storage at scale, without affecting the core rack design. The storage back-end for the environment is currently based on Nova, Glance, and Cinder, with support for hardware RAID and logical volume management (LVM). Additional storage functionality is also under consideration to support various storage options that developers may want to use. The team is currently working on integrating Ceph* block storage into the environment, and future projects will address Swift object storage and Manila file-share-management. Network Flexibility The robustness of the 10GbE top-of-rack switches and Intel Ethernet Converged Network Adapters X520 allows developers to make use of advanced networking capabilities. Robust VLAN support allows a single rack or a larger environment that includes multiple racks to be segmented as needed to allow separate experiments to proceed in parallel at the same time. Bonded network interfaces provide options for increasing uplink speed (up to 40 Gb) and/or reliability, depending on the bonding architecture used. Jumbo frames increase network performance by allowing a larger data payload to be included in each network packet. Deployment Mechanisms There are multiple options available for provisioning and deployment within the environment. One approach is to conduct automated provisioning of physical hosts with Bifrost and Ironic, with automated deployment of OpenStack services using Ansible. The initial design uses Linux Bridge as the underlying networking plug-in agent for neutron, and network bonding is also configured. Using other tools such as Kolla, Fuel, or RDO Manager is also possible; the flexible network design of the environment supports testing various setup changes to improve the robustness of these approaches. Design Best Practices for Flexible OpenStack Clouds Flexibility must be a core design precept from the beginning of a successful OpenStack implementation, rather than a concept to be observed after the fact. Insight early in the process about how customers will want to use the cloud can go a long way toward defining requirements for the infrastructure. Based on that information, the design should build outward from an understanding of OpenStack architecture and dependent services. Gathering complete information about the constraints to be addressed by the design is also critical. Areas to be considered include physical space available for the construction, operation, and growth of the environment, power and cooling resources, weight capacity, budget limitations, and requirements for security and compliance. As a related matter, implementation teams must identify how they will provision, service, and maintain the physical hardware in the solution. Patching, upgrades, disaster recovery, and business continuity must all be addressed. This planning will also include determining the actual location of resources, the people who will have physical and remote access to them, and what aspects of the solution lend themselves to automation. Compute Best Practices Testing multiple virtualization technologies (for example, KVM, VMware, Microsoft HyperV*, or Xen*) may reveal advantages in terms of the capabilities supported, as well as performance and reliability in various usage scenarios. Present and future requirements should be identified and documented as part of this design decision; for example, challenges may exist with live migration in OpenStack environments using KVM. In addition, comparison of local versus shared storage for VM ephemeral disks may reveal significant operational advantages depending on specific workloads. Depending on the needs and characteristics of the workload, it may be valuable to fine-tune host over-provisioning. OpenStack s default overcommit ratios are 16:1 for CPU and 1.5:1 for RAM, meaning for example that Nova schedules 16 virtual cores for each physical core. Over-commitment of compute resources can allow a greater number of instances to coexist on a given cloud topology, although each of those instances will achieve lower performance because of the smaller allocation of resources per instance.

7 Building a Flexible OpenStack* Cloud from the Ground Up 7 Storage Best Practices Choosing server hardware with ample physical slots for HDDs or SSDs allows for flexibility in the deployment of local storage for the solution. In addition, it is worthwhile to consider the fact that Glance can support a number of back-end storage types (for example, LVM, NFS, Ceph, and Cinder), adding further to the potential flexibility of storage on OpenStack clouds. Network Best Practices Examining the options for network topology is an important aspect of OpenStack design. The standard topology includes public, management, and storage networks; including separate networks for deployment or API access may also be desirable. When confronting the complexity of networking with OpenStack, it is worth considering the use of providernetwork versus tenant-network models. Specifically, provider networks are provisioned by an OpenStack administrator (as it requires changes to physical network infrastructure) and can be shared among tenants, while tenant networks are provisioned by the tenants. Deployment and Management Best Practices Automation is a critical aspect of streamlining the day-today operation of OpenStack environments, particularly with regard to making the ongoing changes that underlie flexibility of the overall environment. It is worthwhile to invest time and effort to research and test tools for common activities. Operating System deployment, network configuration, and disk configuration can be automated using tools that include Preboot execution Environment (PXE) and IPMI. OpenStack services deployment can be automated by using tools that include Ansible and Kolla. Management of OpenStack users, projects, and subnets can be automated using approaches such as Ansible scripts. Conclusion Rather than following the common practice of starting with a project budget and building outward from that point, the approach followed here relies on first gathering user requirements and then developing a preliminary design. Once that first design is in place, it is refined as needed, according to constraints in areas such as power, cooling, weight, and budget. This prioritization helps keep capabilities, rather than constraints, at the center of design considerations. Compute, storage, and networking resources are designed at a per-rack level, providing a building block that can be repeated as the solution environment scales out. Building from a foundation based on user requirements, Intel Open Source Technology Center has designed and built an OpenStack cloud with the ability to scale to 100 or more nodes and to flexibly support emerging requirements in the years to come. While this environment is specifically geared toward the development of OpenStack features as part of Intel s commitment to the technology, it provides a methodology that can help other organizations pattern their own efforts. Learn more: intel.com/openstack

8 Building a Flexible OpenStack* Cloud from the Ground Up Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Check with your system manufacturer or retailer or learn more at intel.com. No computer system can be absolutely secure. Copyright 2016 Intel Corporation. All rights reserved. Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and other countries. *Other names may be trademarks of their respective owners. 0816/NKH/MESH/PDF US