DELL EMC VXRACK SYSTEM SDDC TM TECHNOLOGY OVERVIEW

Transcription

1 DELL EMC VXRACK SYSTEM SDDC TM TECHNOLOGY OVERVIEW ABSTRACT VxRack System SDDC powered by VMware Cloud Foundation is a turnkey, rack-scale, hyper-converged engineered system with fully integrated hardware and software provides an agile and flexible infrastructure foundation that IT organizations can leverage as part of their IT transformation into an IT-as-a-Service operating model. This is accomplished through system features such as workload domains and automated infrastructure deployments that are comparable to public cloud offerings, providing elasticity and scalability while reducing costs. This paper provides an overview of VxRack SDDC, describes its major components, and describes its value as a turnkey Software-Defined Data Center system. February 2017 Document H15859 WHITE PAPER 1

2 The information in this publication is provided as is. Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copy, and distribution of any software described in this publication requires an applicable software license. Copyright 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners. Published in the USA 2/17 VxRack SDDC Whitepaper. Dell EMC believes the information in this document is accurate as of its publication date. The information is subject to change without notice. 2

3 TABLE OF CONTENTS EXECUTIVE SUMMARY...4 INTENDED AUDIENCE...4 INTRODUCTION...4 ENGINEERED SYSTEM OVERVIEW...6 CLOUD FOUNDATION SOFTWARE STACK...7 vsphere... 8 VMware vcenter Server... 8 VMware vsphere ESXi... 8 vsphere Virtual Networking... 9 vcenter Server Features for Availability and Performance... 9 NSX NSX Components vsan vsan Datastore vsan Storage Policy-Based Management vsan Features SDDC MANAGER Simplified Resource Management with Workload Domains Lifecycle Management vrealize SUITE VMWARE S CLOUD MANAGEMENT PLATFORM (CMP) HORIZON SUITE VMWARE S VDI AND APP VIRTUALIZATION PLATFORM HARDWARE CONFIGURATION HARDWARE CONFIGURATION HARDWARE COMPONENTS Cabinets Server Nodes Server Node Storage Options Network Switches NETWORK TOPOLOGY NETWORK TOPOLOGY FOR SINGLE-RACK CONFIGURATION NETWORK TOPOLOGY FOR MULTI-RACK CONFIGURATION TRADITIONAL AND LEAF-SPINE NETWORK ARCHITECHTURE CONCLUSION

4 EXECUTIVE SUMMARY The Dell EMC VxRack System 1000 family consists of hyper-converged, rack-scale engineered systems with integrated networking to achieve the scalability and management requirements of traditional and cloud-native workloads. The VxRack 1000 family includes the Dell EMC VxRack System with FLEX and the Dell EMC VxRack System SDDC. Each system is built for fast and easy automated infrastructure deployment and/or private cloud infrastructure as a service (IaaS) architectures. VxRack 1000 systems tightly integrate hardware with the software and management layer for a turnkey, pre-integrated system. The Dell EMC VxRack System SDDC (VxRack SDDC) with VMware Cloud Foundation is a new class of hyper-converged infrastructure that provides a quick and simple path to a VMware-based software-defined data center (SDDC) environment. The hyper-converged VxRack SDDC is designed for enterprise-scale deployments of virtual infrastructure, IaaS, and virtual desktops infrastructure (VDI) in a VMware environment. Each VxRack SDDC system is based on a standardized architecture that combines Dell EMC and Cisco hardware with pre-loaded, pre-integrated software components in a complete and validated system. The VxRack SDDC is a turnkey VMware SDDC system. INTENDED AUDIENCE This document is intended for Dell EMC sales and field personnel, partners, and customers involved in designing, acquiring, and managing a VxRack SDDC solution. It may also be a useful resource for System Administrators and Dell EMC Solutions Architects. INTRODUCTION The IT infrastructure market is undergoing unprecedented transformation. The most significant transformation is reflected by two major trends: a deployment trend toward converged and hyper-converged infrastructure (HCI) and a design trend toward software-defined data centers (SDDCs). Both are responses to the IT realities of infrastructure clutter, complexity, and high cost; they represent attempts to simplify IT and reduce the overall cost of infrastructure ownership. Today s legacy infrastructure environments are typically comprised of multiple hardware and software products from multiple vendors, with each product offering a different management interface and requiring different training. Each product in this type of legacy stack is likely to be grossly overprovisioned, using its own resources (CPU, memory, and storage) to address the intermittent peak workloads of resident applications. The value of a single shared resource pool, offered by server virtualization, is still generally limited to the server layer. All other products are islands of overprovisioned resources that are not shared. Therefore, low utilization of the overall stack results in the ripple effects of high acquisition, space, and power costs. Too many resources can be wasted in traditional legacy environments. In the modern data center, converged infrastructure and SDDC are cornerstones of virtually every customer s IT strategy. Converged and hyper-converged infrastructures mean that multiple pre-engineered and pre-integrated components operate under a single controlled architecture with a single point of management and a single source for end-to-end support. HCI provides a localized single resource pool that enables a higher overall resource utilization than with a legacy island-based infrastructure. Overall acquisition cost is lower and management is simplified. In the data center, HCI typically has a smaller footprint with less cabling and can be deployed much faster and at lower total cost than traditional infrastructure. Industry infrastructure deployment is transforming as customers begin to shift from a build to a buy approach. This deployment shift is being driven by the need for IT to focus limited economic and human capital resources on driving business innovation, resulting in less resources available to focus on infrastructure. While a build-your-own deployment strategy can achieve a productive IT infrastructure, this strategy can be difficult and lengthy to implement and vulnerable to higher operating costs, and susceptible to greater risk related to component integration, configuration, qualification, compliance, and management. A buy deployment strategy for HCI provides the benefits of previously integrated, configured, qualified, and compliant components. Buying an HCI system provides a single optimized IT solution which is quick and easy to deploy. A buy deployment strategy for HCI provides a simple and effective alternative to build-your-own, and it has been widely adopted. The software-defined data center (SDDC) is a significant design trend driving an unprecedented transformation of the IT infrastructure market. SDDC is VMware s architectural vision for the modern, emerging software-centric data center. Dell EMC and VMware share this vision and together are developing new technologies to make this vision a reality for customers. While the concept is still evolving, the SDDC is a software-centric architectural design for the data center based on virtualization and automation. It logically defines all data center infrastructure services by applying the widely successful principles of server virtualization abstraction, isolation, and 4

5 pooling to the remaining network and storage infrastructure services. SDDC management is automated through policy-based software that controls resources both on premises and off premises. With SDDC, traditional enterprise applications can be supported in a more flexible and cost-effective manner. SDDC represents the epitome of the agile digital business model, where pooled resources adapt and respond to shifting application requirements. The SDDC architectural vision is being driven by VMware, a thought leader in the industry. See Figure 1 below for a typical SDDC Architecture. Figure 1: Software-Designed Data Center (SDDC) Architecture In the SDDC, virtualized servers represent a familiar software-defined IT model, in which hypervisors running on a cluster of hosts allocate hardware resources to virtual machines (VMs). In turn, VMs can function with a degree of autonomy from the underlying physical hardware. Software-defined storage (SDS) and software-defined networking (SDN) are based on a similar premise: physical resources are aggregated and dynamically allocated based on predefined policies with software abstracting control from the underlying hardware. The result is the logical pooling of compute, storage, and networking resources. Physical servers function as a pool of CPU resources hosting VMs, while network bandwidth is aggregated into logical resources, and pooled storage capacity is allocated by specified service levels for performance and durability. Once the SDDC has abstracted resources, SDDC services make the data center remarkably adaptable and responsive to business demands. In addition to virtualized infrastructure, the SDDC includes automation, policy-based management, and hybrid cloud services. The policy-based model insulates users from the underlying standardized technology, and policies balance and coordinate resource delivery. Resources are allocated where needed, absorbing utilization spikes while maintaining consistent and predictable performance. Conceptually, SDDC encompasses more than the IT infrastructure itself; it also represents an essential departure from traditional methods of delivering and consuming IT resources. The SDDC vision requires changes in the IT organization technology, people, and process which then enables a business s transformation to the cloud and toward a digital transformation as a whole. In the SDDC, infrastructure, platforms, and software become cloud computing services and are called Infrastructure-as-a-Service (IaaS), Platform-asa-Service (PaaS) and Software-as-a-Service (SaaS). SDDC is the fundamental architecture that underpins the most sophisticated cloud computing environments. Cloud services (IaaS, PaaS, SaaS) are accessed via a cloud management platform (CMP) that provides end users with a self-service catalog of IT services with financial transparency. The VMware SDDC architectural vision includes CMP layers running on top of SDDC infrastructure layers to create a private or hybrid cloud. This requires a SDDC infrastructure with an extremely high level of efficiency and serviceability, such as the VxRack SDDC system. VxRack SDDC is the simplest and most complete underlying infrastructure system needed when building a SDDC with private or hybrid cloud computing services. VMware s cloud management platform, 5

6 vrealize Suite, when combined with VxRack SDDC delivers the most complete solution for a private or hybrid cloud computing environment. VxRack SDDC, a SDDC engineered system designed as one by Dell EMC and powered by VMware Cloud Foundation, is comprised of innovative technology from three leading industry providers Dell EMC, Cisco, and VMware. VxRack SDDC provides a tightly integrated, turnkey hardware/software system with virtualized computing, networking and storage. VxRack SDDC provides standardization, modular scale, lifecycle management, and industry-best support for on-demand IT services that further accelerate a business transformation to SDDC and cloud computing. VxRack SDDC is not just evolutionary, but revolutionary. ENGINEERED SYSTEM OVERVIEW The VxRack SDDC hyper-converged infrastructure system, shown in Figure 2, is powered by VMware s Cloud Foundation, an integrated virtualization software platform that includes vsphere, NSX, vsan and SDDC Manager. The hyper-converged VxRack SDDC is a system which is sized and optimized for the customer workload, and manufactured to customer specifications. The system arrives on site ready to be configured and integrated into the data center network, and to provision virtual servers within hours. This new hyper-converged system is designed for companies that want a cloud platform with improved economics and security. VxRack SDDC has been developed for companies that intend to standardize on VMware virtualization and cloud technologies and want a premier foundation for automated data center operations. VxRack SDDC is an agile system that enables customers to build an IaaS platform based on a complete SDDC architecture. Its modular design allows datacenter scalability and flexibility for next-generation cloud workloads and enterprise applications. It s designed for deployments involving large numbers of VMs and provides the following features: Automated bring-up delivers best-in-class day 0 experience from power-on to system ready in a few hours Low provisioning complexity with simple-to-use functionality, optimized to consolidate thousands of applications with proven VMware technologies Flexible and variable configuration options to support a variety of application workloads Horizontal scaling by adding, re-assigning, and extending nodes on the fly to extend compute, storage, and networking resources exponentially Easy day-to-day management and one-click software upgrades Pre-configured, pre-loaded, pre-tested, and fully optimized hardware and software stack, delivered as a fully assembled and single vendor supported system Enhanced networking and security capabilities built-in and automatically configured A complete, turnkey system, the Dell EMC hyper-converged VxRack SDDC provides the easiest and fastest way to stand up a VMware based SDDC while at the same time introducing the performance, flexibility, and automation required to make IT departments more agile. VxRack SDDC powered by VMware Cloud Foundation delivers a rack scale hyper-converged system that simplifies the transformation to the SDDC and cloud computing. 6

7 Figure 2: VxRack SDDC CLOUD FOUNDATION SOFTWARE STACK The VxRack SDDC architecture defines data center resources in terms of software. It minimizes compute, network, and storage hardware constraints and increases infrastructure service agility. This is the evolution from simple server virtualization to complete virtualization and automation of the data center. VMware SDDC software provides virtualization and automation across fully integrated components. VMware Cloud Foundation provides the core software platform for this virtualization and automation in the VxRack SDDC. Cloud Foundation has been designed to extend and optimize the familiar VMware vsphere experience with robust software-defined solutions for storage and networking as well as simple management. It provides the opportunity to leverage a single standardized SDDC platform for both on-premises private cloud and off-premises public clouds which ensures compatibility for a true hybrid cloud architecture. It s a complete SDDC solution that provides a highly available, resilient, on-demand infrastructure. Figure 3: VMware Cloud Foundation 7

8 Cloud Foundation is a unified SDDC software platform that combines leading virtual-computing technology into a natively and fully integrated, enterprise-ready infrastructure for private and public clouds. It s made up of vsphere, vsan, and NSX, and is managed by SDDC Manager as shown in Figure 3 above. The following sections examine the Cloud Foundation software components as implemented in VxRack SDDC. vsphere VMware vsphere, the compute virtualization component of VxRack SDDC, delivers server virtualization within a highly available, resilient, and efficient on-demand infrastructure. vsphere is an established, industry-leading software platform with the following advantages: Continuous availability and fault tolerance with data protection and replication Simplified customer experience for automation and management at scale Optimal hypervisor supported by a broad ecosystem and scalable management Comprehensive built-in security for protecting data, infrastructure, and access Load-balanced workloads and prioritized access to resource for top performance and compliance Rapid provisioning and deployment of workloads and desktops VxRack SDDC leverages two key components of vsphere: ESXi and vcenter. ESXi is the hypervisor software which is installed directly onto physical VxRack SDDC nodes, the nodes are powered by the Dell EMC PowerEdge server platform. Each physical server functions as a virtual ESXi server (ESXi host) made up of partitioned logical servers referred to as virtual machines (VMs). VMs are configured on top of the ESXi server. VMware vcenter is a centralized management application that is used to manage the vsphere ESXi servers (hosts) and VMs. vcenter communicates with each ESXi host using an agent that relays tasks to perform management operations directly on the hosts. VMWARE vcenter SERVER The vcenter Server software provides the primary point of management for server virtualization and vsan, and is the enabling technology for advanced capabilities such as vmotion, DRS and HA. vcenter scales to enterprise levels where a single vcenter can support up to 1000 ESXi servers and 10,000 VMs. vcenter supports a logical hierarchy of data centers, clusters, and hosts, which allows resources to be segregated by use case or line of business and also be re-allocated dynamically as needed. vcenter is a single interface that provides a number of services, including task scheduling, statistics logging, alarm and event management, and VM provisioning and configuration. vcenter also provides distributed services such as vsphere vmotion, vsphere DRS, and vsphere HA. VxRack SDDC uses vcenter for its own system management cluster as well as for user provisioned workload cluster management. VMWARE vsphere ESXI VMware vsphere ESXi is an enterprise-class hypervisor that deploys and services VMs. Figure 4 illustrates vsphere ESXi basic architecture. ESXi partitions a physical server into multiple secure and portable VMs that can run side by side on the same physical server. Each VM represents a complete system with processors, memory, networking, storage, and BIOS so any operating system (guest OS) and applications can be installed and run on the virtual machine without any modification. The hypervisor provides physical hardware resources, dynamically to support the operation of VMs. The hypervisor enables the VMs to operate independently from the underlying physical hardware. For example, a virtual machine can be moved from one physical host to another. Also, the VM s virtual disks (VMDKs) can be moved from one type of physical storage to another without affecting the functioning of the virtual machine. ESXi also isolates VMs from one another, so when a guest operating system running in one VM fails, other VMs on the same physical host are unaffected and continue to run. VMs share access to CPUs, and the hypervisor is responsible for CPU scheduling. In addition, ESXi assigns each VM a region of usable memory and provides shared access to the physical network cards and disk controllers associated with the physical host. 8

9 Figure 4: vsphere ESXi Architecture vsphere VIRTUAL NETWORKING vsphere also provides a rich set of virtual networking capabilities, which are managed through vcenter. Virtual-switch technology allows communication among ESXi servers and among VMs in a cluster, using the same protocols that would be used over physical switches. The virtual switch supports VLANs and forwards frames at the data-link layer and virtual Ethernet adapters each have their own IP and MAC address. As a result, VMs have the same properties as physical machines from a networking perspective. The virtual adapters connect both VMs and the ESXi server console to external networks. VxRack SDDC clusters use the VMware Virtual Distributed Switch (VDS), a single switch that spans across multiple ESXi hosts in the same cluster. This switch enables VMs to maintain consistent network configurations as they migrate across multiple hosts. VDS is configured in vcenter Server at the data-center level and makes the configuration consistent across all ESXi hosts. vcenter Server stores the state of distributed ports in the vcenter Server database. Networking statistics and policies migrate with VMs when the VMs are moved from host to host. As discussed in upcoming sections, vsan relies on VDS for its storage virtualization functionality, and VDS is also the underlying technology which NSX leverages for its network virtualization functionality. vcenter SERVER FEATURES FOR AVAILABILITY AND PERFORMANCE VxRack SDDC leverages the following vsphere software suite technologies to ensure a high level of availability and load balanced performance during planned and unplanned system outages. vmotion for VM migration Distributed Resource Scheduler (DRS) for VM load balancing vsphere HA for node failover protection vmotion. VMware vmotion enables live migration of running VMs from one physical server to another with no downtime, continuous service availability, and complete transaction integrity. vmotion is a key enabling technology for creating a dynamic, automated, and self-optimizing data center. vmotion continuously and automatically allocates VMs within resource pools. It also improves availability by conducting maintenance without disrupting business operations. The advanced capability for migrating workloads without disruption is one of the features that distinguish the VxRack SDDC system from other hyper-converged systems. Distributed Resource Scheduler. The Distributed Resource Scheduler (DRS) balances computing capacity across a collection of VxRack SDDS resources that have been aggregated into logical pools. It continuously balances and optimizes compute resource 9

10 allocation among the VMs. When a VM experiences an increased workload, DRS evaluates the VM priority against user-defined resource-allocation rules and policies. If justified, DRS allocates additional resources. It can also be configured to dedicate consistent resources to the VMs of particular business-unit applications to meet SLAs and business requirements. DRS allocates resources to the VM either by migrating the VM to another server with more available resources or by making more resources for the VM on the same server by migrating other VMs off that server. In VxRack SDDC, all ESXi hosts are part of a vmotion network. The live migration of VMs to different node servers is completely transparent to end users through VMotion. DRS adds tremendous value to the VxRack SDDC clusters by automating VM placement, ensuring consistent and predictable application-workload performance. vsphere HA. vsphere HA failover technology protects VMs with rapid recovery from outages, providing cost-effective high availability for VM applications. vsphere HA uses a Fault Domain Manager (FDM) agent to proactively monitor host availability and power state. When a host fails, vsphere HA restarts the affected VMs on another host. Once vsphere HA is configured, all workloads are protected. No actions are required to protect new VMs, and applications and VMs need no special software. vsphere HA provides several points of protection for applications including: Circumventing any server failure by restarting the VMs on other hosts within the cluster. Continuous monitoring of VMs and resetting of any detected VM failures. Protecting against datastore accessibility failures and providing automated recovery for affected VMs. Affected VMs restart on other hosts that still have datastore access. Protecting VMs against network isolation by restarting them if their host becomes isolated on the management or VMware vsan network. This protection is provided even if the network has become partitioned. NSX Network design and planning for network growth when deploying a SDDC is critical to maintaining operational performance as the environment scales. VxRack SDDC integrates the latest in networking at rack scale both physical and software defined. Physical networking consists of a Cisco leaf-spine topology with top-of-rack (ToR) and spine switches. Software-defined networking (SDN) consists of VMware NSX and VMware SDDC Manager which provide network configuration, control, and management of the physical network elements. NSX network virtualization delivers the operational model of a VM to VxRack SDDC s network infrastructure. As an integral component in the VxRack SDDC architecture, NSX SDN injects improved security into the entire data center infrastructure. With NSX, network functions including switching, routing, and firewalling are embedded in the hypervisor and distributed across the environment. This effectively creates a network hypervisor that acts as a platform for virtual networks and services as shown in Figure 5. 10

11 Figure 5: NSX Software-Defined Networking NSX virtual networks leverage automated, policy-based provisioning and multi-tenant isolation to simplify network management, even for complex multi-tier network topologies and multiple workload domains. NSX reproduces the entire network model in software, enabling any network topology to be created and provisioned in seconds. Users can create multiple virtual networks with diverse requirements, leveraging a combination of the services offered via NSX, to build inherently more agile and secure environments. NSX COMPONENTS The NSX virtualized SDN is built on several key components described below. NSX Manager. NSX Manager provides a central point to deploy and configure the virtualized network components, including controller cluster systems and VMware Installation Bundles (VIB) for ESXi, VXLAN, logical switching, logical firewall, and logical routing. NSX Manager can also deploy and configure Edge gateway systems and its services. NSX Manager is deployed as a single VM and controller clusters are installed within the VxRack SDDC management domain infrastructure. Each workload domain has an NSX Manager VM per vcenter Server VM with NSX virtual switches installed in the hypervisors. SDDC Manager uses NSX APIs to configure a VXLAN with the IP parameters specified for the workload domain, and to configure the ToR ports associated with the servers. NSX Logical Distributed Router. The Logical Distributed Router (LDR) is responsible for forwarding and routing all packets through the virtualized SDN networks. The NSX LDR can provide virtual network segmentation. Creating multiple VMware NSX LDRs enables multi-tenancy or separate security zones. Each LDR can create virtual switches that function in the same context as a physical VLAN. The LDR is accessed independently on both the Control Plane and Data Plane. Each VMware host has a copy of the NSX LDR running in the hypervisor. All the gateway interfaces and IP addresses are distributed throughout the VMware cluster. This allows VMs to directly access their default gateway at the local hypervisor. NSX Controller. NSX Controller (NSX-C) provides the Control Plane functionality to distribute logical routing, so VXLAN network information can reach the underlying hypervisor. Controllers are deployed as virtual appliances and should reside in the same vcenter that connects to NSX Manager. NSX-C nodes are deployed in sets of three within a cluster and they divide the workload equally in an active-active cluster scenario. Additionally, NSX-C removes the dependency on multicast routing in the physical network and suppresses broadcast traffic in VXLAN networks. 11

12 NSX Edge Services Gateway. The NSX Edge Services Gateway (ESG) offers a rich set of services that include network address translation, routing, load balancing, firewall, L2/L3VPN and DHCP/DNS relay. Individual services can be deployed, configured, and consumed on demand. The ESG is a virtual machine, deployed via NSX Manager and accessed using the vsphere web client. Distributed Firewall. NSX provides a complete L2 L4 stateful distributed firewall that runs in the ESXi hypervisor kernel. Because the firewall is a function of the ESXi kernel, it provides massive throughput and performs at near line-rate speeds. The distributed firewall service (DFW) installs in the kernel by deploying the kernel VIB in conjunction with the VMware Inter-networking Service Insertion Platform (VSIP). VISP is responsible for monitoring and enforcing security policies on all traffic flowing through the Data Plane and provides anti-spoofing functionality and traffic redirection for third-party appliance extensions and services. DFW protects virtual-to-virtual or virtual-to-physical traffic. DFW policies can also restrict traffic between VMs and external networks. Individual VMs without a firewall-protection requirement can be added to the DFW exclusion list. NSX delivers significant advantages to the VxRack SDDC system including, improved security, on-demand service delivery, faster and more agile deployment, and operational efficiency. NSX allows IT organizations to break free of the constraints and limitations of hardware-based data center networking infrastructure. Customers can choose to leverage these features with VxRack SDDC. vsan vsan is a Software Defined Storage (SDS) technology that includes deep integration with vsphere and the VMware ecosystem to make it a compelling, effective storage solution, well suited for VxRack SDDC. vsan decouples software from the underlying hardware and it implements a notably efficient architecture, built directly into ESXi hypervisor. This distinguishes vsan from solutions that typically install a virtual storage appliance (VSA) that runs as a guest VM on each host. Embedding vsan into the ESXi-kernel layer has clear advantages in performance and memory requirements. It has very little impact on CPU utilization (less than 10 percent) and self-balances based on workload and resource availability. vsan aggregates locally attached disks of VxRack SDDC nodes to create a pool of distributed shared storage. This enables ondemand provisioning and consumption according to policy. VxRack SDDC supports multiple vsphere clusters with up to 64 nodes per cluster and up to a total of 192 nodes in a single VxRack SDDC system. Storage characteristics are configured using Storage Policy Based Management (SPBM), which allows VM object-level policies to be set and modified on the fly to control storage provisioning and day-to-day management of storage service-level agreements (SLAs). vsan DATASTORE vsan technology creates a virtual SAN from the local datastore on clustered ESXi instances. The solution provides a single vsan datastore spanning all the hosts within each vsphere cluster. In the VxRack SDDC, it leverages the local disk storage on the Dell PowerEdge servers, which function as ESXi cluster nodes. The resulting virtual SAN becomes shared storage for the ESXi hosts contributing storage. vsan is preconfigured for the management domain when VxRack SDDC is initialized and managed through vcenter. After the initialization of the management domain, the additional vsan datastore for user workloads are dynamically provisioned by SDDC manager from the selected nodes to create a vsphere cluster for that workload domain at the time of workload domain creation. The VxRack SDDC system total storage capacity depends on the storage disk configuration of the VxRack SDDC server nodes. Each node contains ten storage devices in two separate disk groups configured for either hybrid storage, with flash SSD drives for the cache tier and HDD drives for the capacity tier, or for all-flash storage, with flash SSDs for both the cache and capacity tiers. vsan STORAGE POLICY-BASED MANAGEMENT vsan is entirely policy driven and designed to simplify storage provisioning and management. It automatically and dynamically matches requirements with underlying storage resources based on VM-level storage policies. vsan policies define VM storage requirements for performance and availability. They determine how storage objects are provisioned and allocated within the datastore to guarantee the required level of service. vsan policies also define storage-specific attributes for VM objects, including availability and replication settings. 12

13 vsan FEATURES Today s data center management teams have high expectations of their storage platforms in terms of scalability, availability, performance, and efficiency. vsan delivers the same advanced technology found in the most robust storage systems. Scalability. The vsan distributed architecture suits the modular architecture of VxRack SDDC. Customers can non-disruptively scale out by adding nodes for capacity and performance. Availability. As a core VxRack technology, vsan is tightly integrated with vsphere s availability feature set, including vsphere HA, DRS, vmotion, Fault Tolerance, and snapshot technology. vsan enforces availability specifically at the storage level with efficient data-replication functionality and Fault Domain technology, which protects the environment from rack-level failures. Performance. vsan s tight kernel-level integration with vsphere gives vsan a notable advantage in performance as it reduces CPU and memory overhead. Furthermore, as part of the VMware SDDC software stack, vsan dynamically self-tunes, adjusting to ongoing changes in workload conditions to load balance storage resources, ensuring each VM adheres to its defined storage policies. In addition, vsan s flash-optimized design minimizes storage latency by accelerating read and write IO with built-in caching. The caching tier for each disk group functions as a read/write buffer for hybrid configurations and a write buffer for allflash configurations. Efficiency. Storage-capacity requirements continue to grow exponentially and vsan uses data deduplication and compression to increase capacity utilization at a lower cost for all-flash configurations. Meanwhile, vsan deduplication and compression have only a minimal impact on CPU overhead and memory. Typical virtual-server workloads with highly redundant data such as full-clone virtual desktops or homogenous-server operating systems benefit most from data deduplication and compression. vsan s kernel-level integration with vsphere not only enhances performance, but it places the storage close to the application and removes the complexity of storage management and provisioning. vsan sits directly in the IO data path, delivering the highest levels of performance without taxing CPU or memory resources. It s the optimal SDS backbone for the VxRack SDDC. SDDC MANAGER SDDC Manager is a new, innovative system management solution for VxRack SDDC designed to deliver a radically simplified user experience. SDDC Manager automates critical operations across physical and virtual infrastructure such as system initial build-up, configuration of servers and switches, auto-discovery of new physical capacity, resource provisioning, and lifecycle management of hardware and software components. SDDC Manager serves as the primary interface for an operator s day-to-day tasks and provides an integrated view of both the physical and virtual infrastructure. It complements well-known VMware management tools such as vcenter Server and vrealize Operations that continue to be available for advanced administration tasks and integration with third-party software tools. SIMPLIFIED RESOURCE MANAGEMENT WITH WORKLOAD DOMAINS The value of an SDDC platform is simple resource management. In VxRack SDDC, physical compute, storage and network infrastructure becomes part of a single shared pool of virtual resources that is managed as one system using the SDDC Manager. From this shared pool, customers can carve out separate pools of capacity called workload domains, each with its own set of specified CPU, memory and storage requirements to support various workloads types. Two types of customer workload domains are supported currently, Virtual Desktop Infrastructure (VDI) and Virtual Infrastructure (VI). As new physical capacity is added to the VxRack SDDC, added resources are automatically recognized by SDDC Manager and made available for consumption. The entire system is managed as one thereby removing any physical constraints of a single physical server or rack. SDDC Manager abstracts and aggregates physical resources into these customer defined logical workload domains which are actually physical vsphere clusters provisioned from available nodes in the VxRack SDDC system. SDDC Manager then automates the configuration of each physical vsphere cluster, creating a dedicated vcenter server, the vsan datastore, and virtual network based on the underlying configuration parameters specified by the administrator. Workload domains are a policy-driven approach for defining performance, availability, and security parameters. SDDC Manager automatically implements a deployment workflow to translate the workload domain specifications into the underlying pool of resources. Through the automation of tasks and workflows, that SDDC Manager simplifies the provisioning, monitoring, and ongoing management of both the logical and physical resources of the VxRack SDDC. 13

14 In the VxRack SDDC, system management is designed to function via a management domain. The management domain is a special purpose workload domain which runs Cloud Foundation infrastructure components including vcenter Servers, PSCs, NSX Managers, SDDC Manager, LCM, ISVM, vr Ops, vr LI. The management domain consumes the first four nodes in every rack, however, management domain nodes in racks 2-8 are for back up purposes only. See Figure 6. Figure 6: SDDC Manager abstracts and aggregates physical resources into workload domains. Each node in the management domain cluster runs a separate infrastructure services VM (ISVM) in a highly available distributed architecture. These ISVMs host internal system-level functions that work in conjunction with SDDC Manager to track inventory, coordinate activities and provide a persistent datastore. DRS anti-affinity rules are used to ensure that the ISVMs run on separate hosts. The management domain cluster includes a dedicated vcenter Server and a pair of Platform Services Controllers (PSCs) VMs. All workload domains include a dedicated vcenter Server logically connected to the management domain cluster Platform Services Controller VM. As shown in Figure 7, the four ISVMs are pinned to the four ESXi hosts (using DRS anti-affinity rules) while the other VMs will float between the four ISVM hosts in the management cluster. PSC1 and PS2 are the Platform Service Controller VMs. LCM1 and LCM 2 are Lifecycle Manager VMs. These LCM VMs are responsible for storing patches and upgrade bundles as well as applying these updates to the workload domains. As new updates become available admins are notified to schedule a time to download the updates and apply them. With Cloud Foundation individual workload domains can be updated independently. The entire patch/upgrade process is fully automated using SDDC Manager. 14

15 Figure 7: Management Domain LIFECYCLE MANAGEMENT SDDC Manager also streamlines and automates lifecycle management for VxRack SDDC components as shown below in Figure 8. Given the complexities of validating new firmware and patches in an interconnected, hyper-converged infrastructure, conventional methods of performing upgrades and patches are unfortunately prone to configuration and implementation errors. SDDC Manager automates upgrades and patch management for both the logical and physical infrastructure. Lifecycle management can be applied to the entire infrastructure or to individual workload domains. vmotion and DRS allow administrators to perform upgrades while the system and the VMs remain up and running. VxRack SDDC comes with 24/7 world-class support for the entire system, both the hardware and software, through a single Dell EMC phone number. Figure 8: SDDC Manager Lifecycle Management for VxRack SDDC 15

16 vrealize SUITE VMWARE S CLOUD MANAGEMENT PLATFORM (CMP) Most software-defined data centers will be hybrid. Workloads will be a mix of traditional and modern application architectures. They will be provisioned in an increasingly virtualized mix of physical and virtual environments managed both on-premises in private clouds and in off-premises in public clouds. The concept of a cloud management platform has evolved as a response to this complex set of management requirements. VMware s vrealize cloud management platform delivers the management capabilities to effectively manage the complete lifecycle of services delivered in a hybrid IT environment. VMware s vrealize cloud management platform includes: vrealize Operations which provides intelligent health, performance, capacity, and configuration management. vrealize Operations offers performance and health monitoring and capacity planning for the VxRack SDDC as well as custom dashboards, capacity modeling, and customized alerting. These insights help administrators maintain compliance and efficiently detect and resolve any issues that may arise. Each physical rack contains a management domain cluster with an instance of the vrealize Operations virtual appliance. vrealize Log Insight which provides real time log management and log analysis. vrealize Log Insight lets administrators monitor physical and virtual infrastructure to avoid failures and performance issues. vrealize Log Insight provides centralized log aggregation and analysis with search and filter capabilities for the entire VxRack SDDC system. This provides the ability to monitor all workload domains from a single place. Each physical rack contains a management domain cluster with an instance of the vrealize Log Insight virtual appliance. When deploying VxRack SDDC in a multi-rack setup, the boot process federates the vrealize Log Insight virtual appliances together for redundancy and scalability. vrealize Log Insight is configured to receive and process log events for every device in the rack including servers, switches, and PDUs. vrealize Automation which automates delivery of personalized infrastructure, applications and custom IT services. vrealize Business for Cloud which automates costing, usage metering, and service pricing of virtualized infrastructure. vrealize Operations and vrealize Log Insight can be purchased with the VxRack SDDC and deployed automatically via VMware SDDC Manager. vrealize Operations and vrealize Log Insight are fully integrated components of the VxRack SDDC system. This provides administrators the ability to monitor operations of both the physical and virtual components of VxRack SDDC through the single interface of SDDC Manager. vrealize Automation and vrealize Business for Cloud are not deployed and managed by SDDC Manager today, but are planned to be integrated into future releases of Cloud Foundation and therefore VxRack SDDC. As shown in Figure 9, vrealize Automation and Business for Cloud are externally integrated which means they can be deployed outside of VxRack SDDC and then consume the VxRack SDDC pooled infrastructure. External integration of vrealize Automation and vrealize Business for Cloud with VxRack SDDC provides a true private cloud environment with a full featured CMP (cloud management platform) with multi-tenancy, self-service catalog capability, charge-backs and other key features of cloud computing. 16

17 Figure 9: Private Cloud built on VxRack SDDC and vrealize Suite HORIZON SUITE VMWARE S VDI AND APP VIRTUALIZATION PLATFORM Most IT environments offer end-user computing as a service. VMware Horizon Suite, the platform for workforce mobility, connects end users to their data and applications on any device without sacrificing IT security and control. IT can transform technology silos of desktops, data and applications into centralized IT services and improve operational efficiency, security and agility through policybased management of those services. Horizon Suite is the market-leading desktop-virtualization, end user computing solution and includes Horizon View and App Volumes. VMware Horizon View (Horizon) is VMware s VDI and desktop-management environment. Horizon provisions user desktops using a flexible and secure delivery model. The desktop environments are accessed by the user from almost any device, including mobile devices, with the security and resiliency of the datacenter. Because the application software and data components reside in the datacenter, traditional security, backup, and disaster recovery approaches may be applied. If a user's device is lost or the hardware fails, the recovery is straight forward. The user simply restores the environment by logging in using another device. With no data saved on the user's device, if the device is lost or stolen, there is much less chance that critical data could be retrieved and compromised. Figure 10 below shows how Horizon View encapsulates the OS, applications, profiles, and user data into isolated layers and dynamically assembles desktops on demand to provide users with a personalized view of their individual environments. 17

18 Figure 10: Highly Available and Secure Desktops Availability and security, along with ease of management and support, are compelling reasons for moving from traditional physical desktops and laptops to VDI. VMware App Volumes provides application and user management and monitoring with enterprise-scale capabilities across virtual desktop and published/remote application environments powered by VMware Horizon. VMware App Volumes supports real-time application delivery to virtualized desktop environments. With Horizon 7 and App Volumes, IT can build a real-time application delivery system that ensures all applications are centrally managed. Applications are delivered to virtual desktops through VMDK virtual disks without modifying the VM or applications themselves and can be scaled out to virtual desktops with superior performance, at lower costs and without compromising end-user experience. Horizon Add-Ons are deployed in VDI workload domains on the VxRack SDDC. 18

19 HARDWARE CONFIGURATION VxRack SDDC implements an integrated, scalable hardware architecture (Figure 11), ready to deploy as a complete SDDC infrastructure. Minimum configurations begin with eight nodes and scale to 192 nodes in eight racks. Figure 11: VxRack SDDC Integrated and Scalable Architecture The VxRack SDDC system includes industry-leading Cisco Nexus IP network switches for ToR and Spine switches and Dell PowerEdge servers that run the VMware SDDC stack: ESXi hypervisor, NSX software-defined networking (SDN), and vsan softwaredefined storage (SDS). A VxRack SDDC configuration contains a set of compute and storage components as well as fixed network resources. The minimum single-rack configuration has eight server nodes, four of which are reserved for the management domain cluster, one Dell Networking S3048-ON switch for management and a pair of Cisco Nexus 9372PX-E switches. Storage is based solely on the associated nodes, and the management layer uses three VxRack SDDC Controllers with a native VMware vsan for high availability of the management infrastructure. Multiple-rack configurations add a pair of Nexus 9332PQ switches in the second rack as a spine layer backbone. Two spine switches maximum are needed for each system. This provides inter-rack connectivity at multiple 40GB links. The server or node connectivity to the ToR switching is 10GB via the Nexus 9372PX-E, and Nexus 9332PQ switches accommodate customer uplinks (10GB or 40GB). Customers can dynamically scale VxRack SDDC as demands increase. To simplify future expansion, best practice is to size the initial configuration with the maximum number of ports and include cross-rack networking. HARDWARE CONFIGURATION The VxRack SDDC includes the following hardware components: Panduit cabinet (or rack) with an IPI appliance and two PDUs Support for a maximum configuration of 192 Dell PowerEdge R630 servers (or 8 fully populated racks) 19

20 o o Minimum of eight server nodes per rack Maximum of 24 server nodes per rack Two Cisco 9372PX-E Top of the Rack (ToR) switches per rack Two Cisco 9332PQ spine switches included in multi-rack configurations only One Dell Networking S3048-ON switch per rack for management HARDWARE COMPONENTS VxRack SDDC takes advantage of the most reliable enterprise-quality server hardware from Dell and the industry s top network switches from Cisco. VxRack SDDC uses Panduit intelligent cabinets for rack hardware, and system components are pre-installed and configured within the rack. CABINETS In each VxRack SDDC, the compute and network-layer components are pre-installed and distributed within Panduit cabinets as shown in Figure 12. Distributing the components balances out the power draw and reduces the size of the required PDUs. This distributed design improves flexibility during upgrades or expansion. In addition, each Panduit cabinet has an Intelligent Physical Infrastructure (IPI) appliance which allows users to collect and monitor environmental data, power, and security. Figure 12: VxRack SDDC uses Panduit cabinets equipped with an IPI appliance. 20

21 SERVER NODES VxRack SDDC nodes are high-density, two-socket Dell PowerEdge R630 servers (picture in Figure 13) which utilize the latest Intel Broadwell based CPUs. These single unit (1RU), rack-mount servers come in six configurations with either hybrid or all-flash storage. Currently, hybrid and all-flash storage cannot be mixed in the same VxRack SDDC. Figure 13: Dell PowerEdge 630 VxRack SDDC Node SERVER NODE STORAGE OPTIONS Local disks within the Dell PowerEdge R630 Server provide storage in the VxRack SDDC. Each server has direct attached storage (DAS) that becomes aggregated by vsan to create a single datastore shared among all the VMs running on the VxRack SDDC Workload Domain clusters. Each node contains ten disks in two separate vsan disk groups configured either as a hybrid storage with two Solid State Drives (SSDs) for the cache tier and eight Hard Disk Drives (HDD) for capacity tier or as all-flash storage with two SSDs for the cache tier and eight SSDs for the capacity tier. Figure 14 lists current VxRack SDDC node configurations with Dell PowerEdge R630 as of the writing of this paper. As Dell EMC continues to qualify and build new Dell EMC PowerEdge systems these choices will grow. Check with your Dell EMC account team for any new VxRack SDDC node configurations. 21

22 Figure 14: VxRack SDDC Node Configurations NETWORK SWITCHES VxRack SDDC comes with Cisco Nexus 9372PX-E switches installed in pairs at the top of the rack. Expansion racks require a pair of spine switches in the second rack only to allow connectivity between racks and inter-rack east-west traffic. Both the Nexus 9372PX-E switches and the Nexus 9332PQ spine switches have two links to the Dell S3048-ON management switch to communicate with out-ofband components. The table in Figure 15 lists each VxRack SDDC network switch and their functions. 22

23 Switch Switch Type Function Ports Cisco Nexus 9372PX-E Leaf Switch/ToR Switch Deployed in pairs in each rack Northbound traffic to customer network is through these ToR switches in the 1 st rack only Customer data uplink 48x 10GbE + 6x 40GbE40GbE One 40Gbps QSFP per 9372PX-E switch to connect to each of the two Spine switches in multi-rack configuration Four 10GbE links between each pair of 9372PX-E 2x 10GbE links between server and each 9372PX-E For 9372PX-E in the 1 st rack only - up to four 10GbE or 40GbE links to customer data center switches Cisco Nexus 9332PQ Spine Switch (multi-rack configurations) Inter-rack communications Deployed in a pair in 2 nd rack only Only 2 needed per VxRack SDDC system 3240GbE 1x 40Gbps QSFP to connect to each of the 9372PX-E ToR switches 2x 40Gbps QSFP to connect to the other 9332PQ Dell Switch S3048-ON Management Switch Single switch deployed on each rack Runs the SDDC Manager Hardware Management Service (HMS) which allows for physical switch configuration using the native CLI/API 48x 1GbE + 4x 10GbE 1x 1GbE link to each server nodes, 9372PX- E and 9332PQ (if exists) in the same rack 1x 10GbE link to each 9372PX-E in the same rack Figure 15: VxRack SDDC Switches The ToR switches in the first rack connect to the data center aggregation switch using up to 4 10 or 40Gbps links per 9372PX-E. They also run a Layer 3 protocol like OSPF or BGP to connect to the data center network, allowing access between the VxRack SDDC rack and the customer network. If one ToR switch fails, a second ToR on the first rack can take over. In addition, features like ECMP are used for spraying traffic across links for path redundancy and load balancing. 23

24 NETWORK TOPOLOGY Each rack configuration uses a pair of Cisco Nexus 9372PX-E ToR switches and a single Dell Switch 3048-ON management switch. VxRack SDDC nodes connect to the ToR switches through 10G SFP+ (one per switch) to communicate with the in-band components of the infrastructure. The 9372PX-E switches have two links to the management switch to communicate with the out-of-band components. Expansion rack(s) requires a single pair of Cisco Nexus 9332PQ switches for the spine network. These always reside in rack #2 and allow connectivity between all racks. Each expansion rack also has two Cisco Nexus 9372PX-E ToR switches that connect to the Nexus 9332PQ switches in rack #2. The server nodes connect to the Ethernet component of the network layer. The southbound interconnects link to the Cisco Nexus switches in the Ethernet network through 10GbE port channels. All racks are part of the same VxRack SDDC management domain, up to the supported maximum scale-out configuration of 192 nodes, or eight racks. NETWORK TOPOLOGY FOR SINGLE-RACK CONFIGURATION The diagram in Figure 16 illustrates the single-rack configuration network design. Figure 16: Single Rack Network Design 24

25 NETWORK TOPOLOGY FOR MULTI-RACK CONFIGURATION The diagram in Figure 17 illustrates the multi-rack configuration network design. Figure 17: Multi-Rack Network Design TRADITIONAL AND LEAF-SPINE NETWORK ARCHITECHTURE In a conventional data center network hierarchy, hosts at the access layer connect to network switches which, in turn, rely on routing services (data center services as well services outside the data center, including the Internet) from a third tier. This traditional coreaggregate-access (three-tier) network model is efficient for traffic that travels north-south, which is traffic that travels in and out of the data center. North-south traffic typically has a lot of remote client/server communication. This traditional network architecture is usually built for redundancy and resiliency and is still very widely used for service-oriented types of traffic that travel north-south. However, the trends in traffic patterns are changing as the workloads in today s data centers are changing. Traffic in the new SDDC data center typically is east-west traffic, or server-to-server traffic. This traditional network model becomes ineffective and prone to bottlenecks for highly scalable SDDC infrastructures. Figure 18 shows the traditional network data center topology with a three-layer architecture: the access layer, where users connect to the network; the aggregation layer, where access switches intersect; and the core, where aggregation switches interconnect to each other and to networks outside of the data center. In this traditional network architecture, if a server connected to the left-most access switch needs to communicates with a server connected to the right-most access switch, this east-west communication must travel all the way to the core switch and back down again. Clearly this is not the most efficient path and will cause more latency while consuming more bandwidth. If a cluster of servers (which can number in the hundreds, or even thousands) is performing a resource-intensive calculation in parallel, this traditional network architecture may introduce unpredictable latency and/or a lack of bandwidth. Extremely powerful servers may be performing calculations, but if the servers can t talk to each other efficiently because of a bottleneck in the network architecture, the network architecture must evolve. 25

26 Figure 18: Traditional Network Architecture In modern SDDC data centers, compute and storage infrastructure alterations change the predominant network traffic patterns from north-south to east-west. To address network bottleneck because of the shift from north-south to east-west traffic, one solution is to create a Spine and Leaf network architecture, also known as a Distributed Core. This architecture has two main components: Spine switches and Leaf switches as shown in Figure 19. Figure 19: Spine and Leaf Network Architecture Spine switches are like the core, but instead of being a large, chassis-based switching platform, the spine is composed of many highthroughput Layer 3 switches with high port density. Leaf switches are like the access layer; they provide network connection points for 26

27 servers, as well as uplink to the spine switches. The most important part of this architecture is that every leaf switch connects to every spine switch in the fabric. This point is important because no matter which leaf switch a server is connected to, it always has to cross the same amount of devices to get to another server (unless the other server is located on the same leaf). This keeps the latency down to a predictable level and provides much greater overall bandwidth. VxRack SDDC uses a unique version of a spine and leaf network design. In Cloud Foundation all the physical racks are on the same L2 network. The ToR switches are dual connected to two spine switches through redundant 40Gbps links configured in a Multi-Chassis Link Aggregation Group (MC-LAG). Spanning Tree Protocol is not used because looping is avoided using MC-LAG. Layer 3 (northbound traffic) is only supported with the 9372PX-E switches in the first rack. SDDC Manager configures a Switched VLAN Interface (SVI) for each requested VLAN and configures a static route between the two 9372PX-E ToR switches in the first rack and the upstream router. Full redundancy is accomplished by setting up ibgp between ToR switches and an ebgp between each of the two 9372PX-E ToR switches in the first rack and the upstream router. VxRack SDDC powered by VMware Cloud Foundation integrates with an existing data center network infrastructure and provides full uplink compatibility to existing switches such as Cisco, Juniper, and Brocade. L3 or L2 uplink connectivity is supported, however, L3 is preferred. The VMware SDDC Manager provides support for configuring, controlling, and managing the physical network elements, and software defined networking is delivered through VMware NSX. In Figure 20 VxRack SDDC powered by VMware Cloud Foundation physical switches features are shown. Each rack contains two ToR switches and all hosts (Server 1 24) in the physical rack are dual connected to these two ToR switches with 10Gb links. Each NIC on all hosts is connected to one of the two ToR switches in a Multi-Link Aggregation (MLAG) bond configuration. MLAG is a type of link aggregation group (LAG) with constituent ports that terminate on separate chassis, primarily for the purpose of providing redundancy in the event one of the chassis fails. They operate as if they are connected to a single, logical switch. The BMC port on each host is connected to the management switch over a 1G connection used for out-of-band (OOB) management. The ToR switches are connected to each other over 2 links for control traffic and redundancy. The ToR switches are further connected to spine switches in a dual-mlag (Multi-Link Aggregation ) configuration. The data path between the hosts across multiple racks can tolerate a failure of one link or one ToR switch or one spine switch using link aggregation between hosts and ToR switches and also between ToR switches and spine switches. For uplink connectivity, ToR switches are connected to the existing LAN environment using 40 or 10 GbE with MLAG/Channel over 160 GbE. 27

28 Figure 20: VxRack SDDC powered by Cloud Foundation Physical Networking 28