Dell EMC VxRack System FLEX Architecture Overview Document revision 1.6 October 2017
Revision history Date Document revision Description of changes October 2017 1.6 Editorial updates Updated Cisco Nexus 9332PQ network topology content Added Cisco Nexus 93180YC-EX Switch June 2017 1.5 Inserted additional illustrations in the Network topology section Updated physical network illustration General content updates throughout publication March 2017 1.4 Updated elevations and physical and logical network diagrams December 2016 1.3 Updated compute components Updated sample configuration elevation diagrams October 2016 1.2 Updated VxRack System product name Updated to support ScaleIO 2.0 Updated management software information Removed System infrastructure topic July 2016 1.1 Updates to overview graphics in the System architecture and components section Revised Virtual networks section Revised Base configurations and scaling section Revised elevation drawings in the Sample configurations section January 2016 1.0 Initial release Revision history 2
Contents Introduction...4 System architecture and components...5 Compute...9 Network... 9 Network topology... 9 Storage...16 Virtualization...17 VMware vsphere ESXi... 17 VMware vcenter Server...18 Management... 19 Base configurations and scaling...20 System features... 21 VxRack system management software...22 Sample configuration... 23 VxRack System FLEX with eight VxRack 1U High-Density Compute enclosures... 23 Additional references... 24 Network components...24 Storage components... 24 Virtualization components... 24 3 Contents
Introduction This document describes the high-level design of the VxRack System FLEX. This document also describes the hardware and software components that Dell EMC includes in each VxRack FLEX. The target audience for this document includes sales engineers, field consultants, advanced services specialists, and customers who want to deploy a virtualized infrastructure using VxRack FLEX. Refer to the Glossary for a description of terms specific to Converged Systems. Introduction 4
System architecture and components VxRack System FLEX contains compute, network, storage, virtualization, and management resources. The following table shows the components of VxRack System FLEX: Resource Components Compute VxRack 2U Capacity Compute enclosure VxRack 2U Capacity Storage only enclosure VxRack 1U High-Density Compute enclosure VxRack 1U High-Density Storage only enclosure Network Cisco Nexus 3172TQ Management Switch Cisco Nexus 3132Q-X or Cisco Nexus 3164Q or Cisco Nexus 93180YC- EX Switch ToR Access Switches Cisco Nexus 9332PQ Aggregation Switch Storage ScaleIO software-defined storage Virtualization VMware vsphere ESXi VMware vcenter Server VMware vsphere Web Client VMware Single Sign-On (SSO) Service Management Vision software 5 System architecture and components
The following illustration provides a high-level overview of the physical connectivity of VxRack FLEX components with Cisco Nexus 93180YC-EX Switches: System architecture and components 6
The following illustration provides a high-level overview of the physical connectivity of VxRack FLEX components with Cisco Nexus 31xxQ Switches: 7 System architecture and components
The following illustration provides an overview of the logical network components in the VxRack FLEX architecture: This illustration shows a single rack only. Systems with multiple racks contain multiple aggregation and access switches. System architecture and components 8
Compute VxRack System FLEX enclosures are two socket, 1 RU or 2 RU rack mount enclosures built for production-level applications. VxRack FLEX can be deployed in numerous system sizes. The minimum can be as small as four enclosures, while six is recommended to optimize ScaleIO performance. Add VxRack FLEX enclosures in any increment of four enclosures, as required, up to hundreds of nodes in a single ScaleIO cluster. Network The Cisco Nexus switches provide 10 or 25 GbE or 40 GbE IP connectivity between the VxRack System FLEX and the external network. The Cisco Nexus 31xx Series Switches provides 4 x 10 GbE connectivity using breakout cables to connect to the VxRack FLEX enclosures and VxRack Controller nodes. The Cisco Nexus 93180YC-EX Switch provides 4 x 10 GbE connectivity to the VxRack FLEX enclosures and VxRack Controller nodes using a 1:1 connection between the NIC and switch. VxRack FLEX includes Cisco Nexus 3132Q-X, Cisco Nexus 3164Q, or Cisco Nexus 93180YC-EX top of rack (ToR) switches, as well as the optional Cisco Nexus 9332PQ aggregation switches. VxRack FLEX includes a Cisco 3172TQ switch used for out-of-band management. Network topology The default non-aci network design contains a pair of connected Cisco Nexus 3132Q-X Switches, Cisco Nexus 3164Q Switches, or Cisco Nexus 93180YC-EX Switches for the ToR access layer. The Cisco Nexus 3132Q-X Switch supports a maximum of 44 nodes. When density exceeds 44 nodes, up to five switch pairs can be bundled and attached to a pair of Cisco Nexus 9332PQ aggregation switches, supporting up to 220 nodes (total node count includes VxRack Controllers). 9 System architecture and components
The following illustration shows a single rack access network design of the Cisco Nexus 3132Q-X Series Switches: The following table provides details about the Cisco Nexus 3132Q-X single rack access network design: Cisco Nexus 3132Q-X access network design Access switch pair 32 x 40GE ports: 4 x 40GbE ports UL to customer 2 x 40GbE ports vpc peer 2 x 40GbE ports for management cluster 2 x 40GbE ports for future expansion 22 x 40GbE ports DL to nodes (88 10GE ports) 44 supported servers (including VxRack Controller servers, if applicable) System architecture and components 10
Cisco Nexus 3132Q-X access network design 5.5:1 oversubscription The Cisco Nexus 3164Q Switch supports a maximum of 100 nodes. When density exceeds 100 nodes, up to two switch pairs can be bundled and attached to a pair of Cisco Nexus 9332PQ aggregation switches, supporting up to 200 nodes (total node count includes VxRack Controllers). The following illustration shows a single rack access network design of the Cisco Nexus 3164 Series Switch: The following table provides details about the Cisco Nexus 3164Q single rack access network design: Cisco Nexus 3164Q access network design 64 x 40GE ports: 8 x 40GbE ports UL to customer 2 x 40GbE ports vpc peer 11 System architecture and components
Cisco Nexus 3164Q access network design 2 x 40GbE ports for management cluster 2 x 40GbE ports for future expansion 50 x 40GbE ports DL to nodes (200 10GE ports) 100 supported servers (including VxRack Controller servers, if applicable) 6.25:1 oversubscription The number of nodes supported by the Cisco Nexus 93180YC-EX Switch depends on how it is configured. It has the following variations: Connecting to Cisco Nexus 9332PQ aggregation Switch with single uplink Connecting to Cisco Nexus 9332PQ aggregation Switch with dual uplinks System architecture and components 12
The following illustration and table shows network design of the Cisco Nexus 93180YC-EX Switch: The number of supported nodes depends on the configuration of the Cisco Nexus 93180YC-EX Switch. There are four variations: Cisco Nexus 93180YC-EX Access Switch connecting to 9332PQ Aggregation Switch with single uplink Cisco Nexus 93180YC-EX Access Switch connecting to 9332PQ Aggregation Switch with dual uplinks Cisco Nexus 93180YC-EX Access Switch not connecting to Cisco Nexus 9332PQ Aggregation Switch 6-40/100GbE ports 2 x 40GbE ports UL to customer 2 x 40GbE ports vpc peer 48-10/25GbE ports 6 x 10GbE port VxRack Controller node 13 System architecture and components
Cisco Nexus 93180YC-EX Access Switch not connecting to Cisco Nexus 9332PQ Aggregation Switch 42 x 10GbE ports DL to server 21 FLEX Node servers 6:1 oversubscription Cisco Nexus 93180YC-EX access Switch connecting to Cisco Nexus 9332PQ Aggregation Switch with Single Uplink 6 x40/100ge ports 2 x 40GbE ports UL to aggregation switch pair 2 x 40GbE ports vpc peer 48 x10/25ge ports 6 x 10GbE ports for VxRack Controller 42 x 10GbE ports to DL to nodes 21 VxRack FLEX node servers 6:1 oversubscription Maximum of 14 pairs of Cisco Nexus 93180YC-EX Switches Cisco Nexus 93180YC-EX Access Switch connecting to Cisco Nexus 9332PQ Aggregation Switch with Dual Uplinks 6 x40/100ge ports 4 x 40GbE ports UL to customer 2 x 40GbE ports vpc peer 48 x10/25ge ports 6 x 10GbE ports for VxRack Controller 42 x 10GbE ports to DL to nodes 21 VxRack FLEX node servers 6:1 oversubscription Maximum of 7 pairs of Cisco Nexus 93180YC-EX Switches System architecture and components 14
The following illustration shows an aggregation/access network design of the Cisco Nexus 9332PQ and Cisco Nexus 3132Q-X switches that are ready to scale to multiple racks: The following table provides details about the Cisco Nexus 9332PQ single rack aggregation network design: Cisco Nexus 9332PQ aggregation network capacity Aggregation layer Aggregation switch 15 System architecture and components
Cisco Nexus 9332PQ aggregation network capacity 32 x 40GE ports: 4 x 40GbE ports UL to customer 2 x 40GbE ports vpc peer connectivity 2 x 40GbE ports (customer UL, data protection, etc.) 4-8 x 40GbE ports for future expansion 20 x 40GbE DL ports (Cisco Nexus 3132Q-X) 16 x 40GbE DL ports (Cisco Nexus 3164Q) 5:1 oversubscription (Cisco Nexus 3132Q-X) 4:1 oversubscription (Cisco Nexus 3164Q) The compute layer connects to the Ethernet component of the network layer. The southbound interconnects connect to the Cisco Nexus switches in the Ethernet network through 10 GbE port channels using breakout cables. The storage layer consists of ScaleIO software-defined storage using the direct attached storage (DAS) of each VxRack System enclosure. VxRack Controller servers connect to the ToR switches using 10G SFP+ (two to each switch), using breakout cables, to communicate with the in-band components of the infrastructure. They have one link to the Cisco Nexus 3172TQ management switch to communicate with the out-of-band components. Cisco Nexus 9332PQ leaf/aggregation network pair capacity connected by Cisco Nexus 93180YC-EX Switch (Single link) 32 x 40GE ports: 2 x 40GbE ports vpc peer 2 x 40GbE ports uplink to customer 28 x 40GbE ports to VxRack Controller 14 cabinets of 24 servers - 3 Controllers = 333 SIO Nodes 14:1 oversubscription Cisco Nexus 9332PQ leaf/aggregation network pair capacity connected by Cisco Nexus 93180YC-EX Switch (Dual link) 32 x 40GE ports: 2 x 40GbE ports vpc peer 2 x 40GbE ports uplink to customer 28 x 40GbE ports to VxRack Controller 7 cabinets of 24 servers - 3 Controllers = 165 SIO Nodes 7:1 oversubscription Storage VxRack System FLEX uses the software-defined storage technology. ScaleIO is a software-only solution that uses the local disks and LAN of existing VxRack System nodes to create a virtual SAN. It is designed to scale to thousands of nodes. System architecture and components 16
VxRack FLEX supports the following ScaleIO 2.0 features: Extended 5-node MDM cluster LDAP user authentication In-flight checksum Public and private key pair authentication Instant maintenance mode ESRS 3.0 integration For more information about ScaleIO, refer to Dell EMC Support. Virtualization VMware vsphere is the virtualization platform that provides the foundation for the private cloud. The core VMware vsphere components are the VMware vsphere ESXi and VMware vcenter Server for management. VMware vsphere 6.0 includes a pair of Platform Service Controller Linux appliances to provide the Single Sign-On (SSO) service. The hypervisors are deployed in a cluster configuration. The cluster allows dynamic allocation of resources, such as CPU, memory, and storage. The cluster also provides workload mobility and flexibility with the use of VMware vmotion and Storage vmotion technology. VMware vsphere ESXi VMware vsphere ESXi is a lightweight hypervisor that runs on the VxRack Controller and on the VxRack System FLEX compute nodes. It requires very little space to run (less than 6 GB of storage required to install) and has minimal management overhead. VMware vsphere ESXi does not contain a console operating system. It boots from the SATADOM, a small SATA3 flash memory module which is inserted into the server board SATA connector to provide solid state storage capacity that simulates a hard disk drive (HDD). For the enclosures, VMware vsphere ESXi boots from the SATADOM on the local node. VxRack Controllers boot from an SD card. Cluster configuration VMware vsphere ESXi hosts and their resources are pooled together into clusters. These clusters contain the CPU, memory, network, and storage resources available for allocation to VMs. vsphere clusters can scale up to a maximum of 64 hosts. Up to 1000 hosts can be supported in one VMware vcenter instance. Some advanced CPU functionality might be unavailable if more than one enclosure model is running in a given cluster. Data stores VxRack FLEX supports a single data store type: block level storage using VMFS. 17 System architecture and components
The maximum size per VMFS5 volume is 64 TB (50 TB VMFS3 @ 1 MB). The maximum VMDK file size is 62 TB. Each host/cluster can support a maximum of 255 volumes. Virtual networks Virtual networking in VxRack FLEX is managed by a VMware vsphere Distributed Switch (VDS). VMware vcenter Server VMware vcenter Server is a central management point for hypervisors and VMs. VMware vcenter Server Virtual Appliance (vcsa) is leveraged to manage the virtual infrastructure. VMware Update Manager is installed on a 64-bit Windows Server and runs as a service to assist with host patch management. There are two vcsas in VxRack System FLEX: one for the VxRack Controller subsystem and one for customer use, which contains all of the ScaleIO nodes. vcsa provides functionality which includes: Cloning of VMs Template creation VMware vmotion and VMware Storage vmotion Initial configuration of VMware Distributed Resource Scheduler (DRS) and VMware vsphere high-availability clusters vcsa also provides monitoring and alerting capabilities for hosts and VMs. System administrators can create and apply alarms to all managed objects in VMware vcenter Server, including: Data center, cluster, and host health, inventory, and performance Data store health and capacity VM usage, performance, and health Virtual network usage and health Databases The back-end database that supports VUM is a virtual machine running a Microsoft SQL database. The back-end database that supports VMware vcenter Server Appliance on VMware vsphere ESXi 6.0 is an embedded postgres database. Authentication VMware Single Sign-On (SSO) Service integrates multiple identity sources including AD, Open LDAP, and local accounts for authentication. System architecture and components 18
Supported features Dell EMC supports the following VMware vcenter Server features: VMware SSO Service VMware vsphere Web Client (used with Vision software) VMware vsphere High Availability VMware DRS VMware vsphere Distributed Switch (VDS) VMware Fault Tolerance VMware vmotion VMware Storage vmotion VMware Enhanced vmotion Compatibility (EVC) Resource Pools Storage DRS (capacity only) Storage-driven profiles (user-defined only) Distributed power management (up to 50 percent of VMware vsphere ESXi hosts) VMware Syslog Service VMware Core Dump Collector VMware vcenter Web Services Management The VxRack FLEX management stack consists of the following software components: Vision software ScaleIO GUI VMware vsphere Web Client ESRS vsphere Update Manager (VUM) Microsoft SQL Server 19 System architecture and components
Base configurations and scaling VxRack System FLEX has a base configuration that is a minimum set of hyperconverged and storage components, as well as fixed network resources. These components are integrated within one or more 28-inch 42 RU cabinets. Within the base configuration, the following hardware aspects can be customized: Hardware Compute Network Storage Management Minimum set Four nodes minimum. Dell EMC recommends at least six nodes to realize performance and resiliency features of ScaleIO SDS. One Cisco Nexus 3172TQ and one pair of Cisco Nexus 3132Q-X, Cisco Nexus 3164Q, or Cisco Nexus 93180YC-EX Switches. No minimum. Storage is software-defined SAN storage that leverages local disks to build a virtual SAN storage pool. Three VxRack Controller servers with HA support and VSAN for software scale-out. VxRack FLEX is a scale-out solution that allows you to add VxRack System enclosures with various CPU, memory, and drive options. ScaleIO software-defined SAN storage allows scaling from a base configuration to a large-scale deployment. Together, the components offer balanced CPU, I/O bandwidth, and storage capacity relative to the compute and storage arrays in the VxRack System. All components have N+N or N+1 redundancy. These resources can be scaled up as necessary to meet increasingly stringent application workload requirements. To scale up storage and compute resources, add nodes to the environment. Optionally, add expansion cabinets with additional resources. System architecture and components 20
System features VxRack System FLEX is a modular hyper-converged platform that allows extreme scalability and flexibility for next generation cloud applications and mixed workloads. VxRack FLEX is designed for deployments involving large numbers of VMs and users. It provides the following features: Low provisioning complexity with simple-to-use functionality Flexible and variable configuration options Horizontal scaling by adding, moving and removing nodes on-the-fly to extend direct attached storage (DAS) capacity exponentially Step-sized building blocks for future data center environments Pre-configured, pre-loaded, pre-tested, and fully optimized IT stack, delivered as a fully assembled and supported system 21 System features
VxRack System management software Dell EMC recommends using Vision software as your primary management interface for VxRack Systems. Vision software includes the Vision dashboard, which provides system inventory, health monitoring, and RCM compliance management; and the Vision shell, which provides network configuration and other capabilities necessary to manage a VxRack System. For more information about Vision software, refer to cpsdocs.dellemc.com. VxRack System management software 22
Sample configuration VxRack System FLEX with eight VxRack 1U High-Density Compute enclosures Cabinet elevations vary based on the specific configuration requirements. These are provided for sample purposes only. For specifications for a specific VxRack FLEX design, consult your varchitect. 23 Sample configuration
Additional references References to related documentation for virtualization, compute, network and storage components are provided. Network components Network component information and links to documentation are provided. Product Cisco Nexus 3132Q-X Cisco Nexus 3164Q Cisco Nexus 3172T-Q Cisco Nexus 9332PQ Cisco Nexus 93180YC-EX Link to documentation http://www.cisco.com/c/en/us/products/collateral/switches/ nexus-3132q-switch/data_sheet_c78-729482.html http://www.cisco.com/c/en/us/products/collateral/switches/ nexus-3000-series-switches/datasheet-c78-731524.html http://www.cisco.com/c/en/us/products/collateral/switches/ nexus-3000-series-switches/data_sheet_c78-729483.html http://www.cisco.com/c/en/us/support/switches/nexus-9332pqswitch/model.html http://www.cisco.com/c/en/us/products/switches/nexus-93180yc-exswitch/index.html Storage components Storage component information and links to documentation are provided. Product Description Link to documentation ScaleIO Converges storage and compute resources into a single-layer architecture, aggregating capacity and performance, simplifying management, and scaling to thousands of nodes. http://www.emc.com/storage/scaleio/ index.htm Virtualization components Virtualization component information and links to documentation are provided. Product Description Link to documentation VMware vcenter Server VMware vsphere ESXi VMware Single Sign- On (SSO) Service Provides a scalable and extensible platform that forms the foundation for virtualization management. Virtualized infrastructure for Vblock Systems. Virtualizes all application servers and provides VMware High Availability (HA) and Dynamic Resource Scheduling (DRS). Provides VMware-specific authentication services. http://www.vmware.com/products/ vcenter-server/ www.vmware.com/products/vsphere/ blogs.vmware.com/kb/2012/10/ vsphere-sso-resources.html Additional references 24
The information in this publication is provided "as is." Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software described in this publication requires an applicable software license. Copyright 2016, 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners. Published in the USA in October 2017. Dell EMC believes the information in this document is accurate as of its publication date. The information is subject to change without notice. 25 Copyright