VMware vsan Network Design-OLD November 03, 2017

VMware vsan Network Design-OLD November 03, 2017 1

Table of Contents 1. Introduction 1.1.Overview 2. Network 2.1.vSAN Network 3. Physical Network Infrastructure 3.1.Data Center Network 3.2.Oversubscription Considerations 3.3.Host Network Adapter 3.4.Virtual Network Infrastructure (VMkernel) 3.5.Virtual Switch 3.6.NIC Teaming 3.7.Multicast 3.8.Network I/O Control 3.9.Jumbo Frames 4. Switch 4.1.Switch Discovery Protocol 5. Network 5.1.Network Availability 6. iscsi Target 6.1.iSCSI Networking Design Considerations 6.2.Quality of Service (QoS) 6.3.VMkernel Port Guidance 7. Conclusion 7.1.Conclusion 8. About the Author 8.1.About the Author 9. Vendor Specific Guidance 9.1.Cisco ACI 2

1. Introduction This document is targeted toward virtualization, network, and storage architects interested in deploying VMware vsan solutions. 3

1.1 Overview vsan is a hypervisor-converged, software-defined storage solution for the softwaredefined data center. It is the first policy-driven storage product designed for VMware vsphere environments that simplifies and streamlines storage provisioning and management. vsan is a distributed, shared storage solution that enables the rapid provisioning of storage within VMware vcenter Server as part of virtual machine creation and deployment operations. vsan uses the concept of disk groups to pool together locally attached flash devices and magnetic disks as management constructs. Disk groups are composed of a cache device and several magnetic or flash capacity devices. In hybrid architectures, flash devices are used as read cache and write buffer in front of the magnetic disks to optimize virtual machine and application performance. In all flash, the cache device endurance is leveraged to allow lower cost capacity devices. The vsan datastore aggregates the disk groups across all hosts in the vsan cluster to form a single shared datastore for all hosts in the cluster. vsan requires a correctly configured network for virtual machine I/O as well as communication among cluster nodes. Since the majority of virtual machine I/O travels the network due to the distributed storage architecture, highly performing and available network configuration is critical to a successful vsan deployment. This paper gives a technology overview of vsan network requirements and provides vsan network design and configuration best practices for deploying a highly available and scalable vsan solution. 4

2. Network vsan is an integral part of an overall VMware vsphere network configuration and therefore cannot work in isolation from other vsphere network services. 5

2.1 vsan Network The hosts in a vsan cluster must be part of a vsan network and must be on the same subnet regardless whether the hosts contribute storage or not. vsan requires a dedicated VMkernel port type and uses a proprietary transport protocol for vsan traffic between the hosts. The vsan network is an integral part of an overall vsphere network configuration and therefore cannot work in isolation from other vsphere network services. vsan utilizes either VMware vsphere Standard Switch (vss) or VMware vsphere Distributed Switch (vds) to construct a dedicated storage network. However, vsan and other vsphere workloads commonly share the underlying virtual and physical network infrastructure. Therefore, the vsan network must be carefully designed following general vsphere networking best practices in addition to its own. The following sections review general guidelines that should be followed when designing a vsan network. These recommendations do not conflict with vsphere Networking Best Practices. 6

3. Physical Network Infrastructure This section discusses the physical network infrastructure recommendation for a successful vsan deployment. 7

3.1 Data Center Network The traditional access-aggregation-core, three-tier network topology was built to serve north-south traffic in and out of a data center. Convergence and virtualization has changed datacenter traffic patterns to include an east-west flow. While the three-tier network offers great redundancy and resiliency, it limits overall bandwidth by as much as 50% due to critical network links being oversubscribed. The Spanning Tree Protocol (STP) is implemented to prevent network looping. However, as virtualization and cloud computing evolves, more data centers have adopted the leaf-spine topology for data center fabric simplicity, scalability, bandwidth, fault tolerance, and quality of service (QoS). vsan is compatible with both topologies regardless how the core switch layer is constructed. 3.2 Oversubscription Considerations East West and Throughput Concerns VMware vsan requires low latency and ample throughput between the hosts, as reads may come from any host in the cluster, and writes must be acknowledged by two hosts. For simple configurations utilizing modern, wire speed, top of rack switches, this is a relatively simple consideration as all ports can speak wire speed to all ports. As clusters are stretched across datacenters (perhaps using the vsan fault domains feature), the potential for over-subscription become a concern. Typically, the largest demand for throughput is during a host rebuild or host evacuation as potentially all hosts may be requesting to send and receive traffic at wire speed to reduce the time of the action. The larger the capacity consumed on each host, the more important the over subscription ratio becomes. A host with only 1Gbps and 12TB of capacity would take over 24 hours to refill with data. Leaf-Spine In traditional leaf-spine architecture, due to the full mesh topology and port density constraints, leaf switches are normally oversubscribed for bandwidth. For example, a fully utilized 10GbE uplink utilized by the vsan network in reality may only achieve 2.5Gbps throughput on each node when the leaf switches are oversubscribed at a 4:1 ratio and vsan traffic needs to go across the spine, as illustrated 8

in Figure 1. The impact of network topology on available bandwidth should be considered when designing your vsan cluster. The leaf switches are fully meshed to the spine switches with links that could either be switched or routed, these are referred to as Layer 2 and Layer 3 leaf-spine architectures respectively. vsan over layer 3 networks is currently supported. VMware Recommends: Consider using layer 2 multicast for simplicity of configuration and operations Here is an example of how over commitment can impact rebuild times. Let us assume the the above design is used with 3 fault domains, and data is being mirrored between cabinets. In this example each host has 10TB of raw capacity, with 6TB of it being used for virtual machines protected by FTT=1. In this case we will also assume 3/4ths (or 30Gbps) of the available bandwidth is available for rebuild. Assuming no disk contention bottlenecks it would take approximately 26 minutes to rebuild over the over subscribed link. If the capacity needing to rebuild was increased to 12TB of data, and the bandwidth was reduced to only 10Gbps, then the rebuild would take at a minimum 156 minutes. Any time capacity increases, or bandwidth between hosts is decreased the time for rebuilds becomes longer. VMware Recommends: Minimize oversubscription to reduce opportunities for congestion during host rebuilds or high throughput operations. Equal-Cost-Multi-Path (ECMP) A number of vendors have implemented Ethernet fabrics that eliminate the need for spanning tree to prevent loops, and employ layer 2 routing mechanisms to best use the shortest paths as well as supplemental paths for added throughput. SPB (Shortest Path Bridging) or TRILL ("Transparent Interconnection of Lots of Links") are commonly used, but often with proprietary extensions. vsan is 9

compatible with these topologies, but be sure to design adequate east west traffic within each vsan cluster. Cisco FEX/Nexus 2000 It should be noted that fabric extending devices such as the Cisco Nexus 2000 product line have unique considerations. These devices lack the ability for port to port direct traffic on the same switch, and all traffic must travel through the uplink to the Nexus 5000 or 7000 series device and back down. While this will increase port to port latency, the larger concern is large throughput operations (such as a host rebuild) will potentially put pressure on the over subscribed uplinks back to the switch. Non-Stacked top of rack switches and Cisco Fabric Interconnects. VMware Recommends: Deploy all hosts within a fault domain to a low latency wire speed switch or switch stack. When multiple switches are used, pay attention to throughput of the links between switches. Deployments with limited or heavily oversubscribed throughput should be carefully considered. Flow Control Pause Frames are related to Ethernet flow control and are used to manage the pacing of data transmission on a network segment. Sometimes, a sending node (ESXi/ESX host, switch, etc.) may transmit data faster than another node can accept it. In this case, the overwhelmed network node can send pause frames back to the sender, pausing the transmission of traffic for a brief period of time. vsan manages congestion by introducing artificial latency to prevent cache/buffer exhaustion. Since vsan has built-in congestion management, disabling flow control on VMkernel interfaces tagged for vsan traffic is recommended. Note Flow Control is enabled by default on all physical uplinks. For further information on Flow Control see KB 1013413. VMware Recommends: Disable flow control for vsan traffic. Security Considerations VMware vsan like other IP storage traffic, is not encrypted and should be deployed to isolated networks. VLAN s can be leveraged to securely separate vsan traffic from virtual machine and other networks. Security can also be added at a higher layer by encrypting data in guest in order to meet security and compliance requirements. 3.3 Host Network Adapter VMware Recommends: Each vsan cluster node should follow these practices: At least one physical NIC must be used for vsan network. One or more additional physical NICs are recommended to provide failover capability. The physical NIC(s) can be shared amongst other vsphere networks such as virtual machine network and vmotion network. Logical Layer 2 separation of vsan VMkernel traffic (VLANs) is recommended when physical NIC(s) share traffic types. QoS can be provided for traffic types via Network IO Control (NIOC). 10GbE NIC or larger is strongly recommended for vsan, and a requirement for all flash vsan. If 1GbE NIC is used for hybrid configurations, VMware recommends it to be dedicated for vsan. 10

Larger than 10Gbps such as 25/40/100Gbps is supported as long as your edition of vsphere supports it. 3.4 Virtual Network Infrastructure (VMkernel) To enable the exchange of data in the vsan cluster, there must be a VMkernel network adapter for vsan traffic on each ESXi host. This is true even for hosts that do not contribute storage to vsan. For each vsan cluster, a VMkernel port group for vsan should be created in the vss or vds, and the same port group network label should be used to ensure labels are consistent across all hosts. Unlike multiple-nic vmotion, vsan does not support multiple VMkernel adapters. 3.5 Virtual Switch VMware vsan supports both vss and vds virtual switches. It should be noted that vds licensing is included with vsan and licensing should not be a consideration when choosing a virtual switch type. vds is required for dynamic LACP (Link Aggregation Control Protocol), LBT (Load Based Teaming), LLDP (Link Layer Discovery Protocol), bidirectional CDP (Cisco Discovery Protocol), and Network IO Control (NIOC). vds is preferred for superior performance operational visibility, and management capabilities. VMware recommends: Deploy vds for use with VMware vsan. vcenter and vds Considerations VMware fully supports deploying a vcenter that manages a cluster on top of the storage cluster. Starting with vsphere 5.x static port groups became the default port group type for vds, and will persist assignment to a virtual machine through a reboot. In the event vcenter is unable to be bind to the vds a pre-created ephemeral port group, or a vss can be leveraged to restore access to the vcenter Server. 3.6 NIC Teaming The vsan network can use teaming and failover policy to determine how traffic is distributed between physical adapters and how to reroute traffic in the event of adapter failure. NIC teaming is used mainly for high availability, but not load balancing when the team is dedicated for vsan. However, additional vsphere traffic types sharing the same team could still leverage the aggregated bandwidth by distributing different types of traffic to different adapters within the team. vsan supports all vss and vds supported NIC teaming options. Load Based Teaming Route based on physical NIC load, also known as Load Based Teaming (LBT), allows vsphere to balance the load on multiple NIC s without a custom switch configuration. It begins balancing similar to Virtual Port ID, but will dynamically reassess physical to virtual NIC bindings every 30 seconds based on congestion thresholds. To prevent impact on port change settings such as Cisco s portfast or HP admin-edge-port on ESXi host facing physical switch ports should be configured. With this setting, network convergence on these switch ports will happen fast after the failure because the port 11

will enter the Spanning tree forwarding state immediately, bypassing the listening and learning states. Additional information can be found on different teaming policies in the vsphere Networking Documentation. IP Hash/LACP An additional failover path option is the IP hash based policy. Under this policy, vsan, either alone or together with other vsphere workloads, is capable of balancing load between adapters within a team, although there is no guarantee of performance improvement for all configurations. While vsan does initiate multiple connections, there is no deterministic balancing of traffic. This policy requires the physical switch ports to be configured for a port link aggregation technology or port-channel architecture such as Link Aggregation Control Protocol (LACP) or EtherChannel. Only static mode EtherChannel is supported with the vsphere Standard Switch. Dynamic LACP is additionally supported with vsphere Distributed Switch, as well as additional hashes. LAGs using the vds and dynamic LACP enable advanced hashes that allow using things such as source and destination port. These hashes will allow for potentially balancing of traffic that is split across multiple connection sessions between the same two hosts. Note, you will need to see what your switch supports and even within the same vendor or product family different ASICs may only support specific options. VMware Recommends: Use Load Based Teaming or LACP for load balancing. 3.7 Multicast IP multicast sends source packets to multiple receivers as a group transmission. Packets are replicated in the network only at the points of path divergence, normally switches or routers, resulting in the most efficient delivery of data to a number of destinations with minimum network bandwidth consumption. For specifics on multicast please see VMware vsan Layer 2/Layer 3 Network Topologies. vsan uses multicast to deliver metadata traffic among cluster nodes for efficiency and bandwidth conservation. Multicast is required for VMkernel ports utilized by vsan. While Layer 3 is supported, Layer 2 is recommended to reduce complexity. All VMkernel ports on the vsan network subscribe to a multicast group using Internet Group Management Protocol (IGMP). IGMP snooping configured with an IGMP snooping querier can be used to limit the physical switch ports participating in the multicast group to only vsan VMkernel port uplinks. The need to configure an IGMP snooping querier to support IGMP snooping varies by switch vendor. Consult your specific switch vendor/model best practices for IGMP snooping configuration. If deploying a vsan cluster across multiple subnets, be sure to review best practices and limitations in scaling Protocol Independent Multicast (PIM) dense or sparse node. A default multicast address is assigned to each vsan cluster at time of creation. When multiple vsan clusters reside on the same layer 2 network, the default multicast address should be changed within the additional vsan clusters to prevent multiple clusters from receiving all multicast streams. Similarly, multicast address ranges must 12

be carefully planned in environments where other network services such as VXLAN also utilize multicast. VMware KB 2075451 can be consulted for the detailed procedure of changing the default vsan multicast address. Isolating each clusters traffic to its own VLAN will remove possibility for conflict. VMware Recommends: Isolate each vsan cluster's traffic to its own VLAN when using multiple clusters. 3.8 Network I/O Control vsphere Network I/O Control (NIOC) can be used to set quality of service (QoS) for vsan traffic over the same NIC uplink in a vds shared by other vsphere traffic types including iscsi traffic, vmotion traffic, management traffic, vsphere Replication (VR) traffic, NFS traffic, Fault Tolerance (FT) traffic, and virtual machine traffic. General NIOC best practices apply with vsan traffic in the mix: For bandwidth allocation, use shares instead of limits as the former has greater flexibility for unused capacity redistribution. Always assign a reasonably high relative share for the Fault Tolerance resource pool because FT is a very latency-sensitive traffic type. Use NIOC together with NIC teaming to maximize network capacity utilization. Leverage the vds Port Group and Traffic Shaping Policy features for additional bandwidth control on different resource pools. VMware Recommends: Follow these recommendation for vsan: Do not set a limit on the vsan traffic; by default, it is unlimited. Set a relative share for the vsan resource pool based on application performance requirements on storage, also holistically taking into account other workloads such as bursty vmotion traffic that is required for business mobility and availability. Avoid reservations as they will share unused traffic only with other management types (vmotion, Storage, etc.) but not with Virtual Machine networking needs. 3.9 Jumbo Frames vsan supports jumbo frames, but does not require them. VMware testing finds that using jumbo frames can reduce CPU utilization and improve throughput, however, with both gains at minimum level because vsphere already uses TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) to deliver similar benefits. In data centers where jumbo frames are already enabled in the network infrastructure, jumbo frames are recommended for vsan deployment. If jumbo frames are not currently in use, vsan alone should not be the justification for deploying Jumbo Frames. VMware Recommends: Use the existing MTU/Frame size you would otherwise be using in your environment. 13

4. Switch Switch discovery protocols allow vsphere administrators to determine which switch port is connected to a given vss or vds. 14

4.1 Switch Discovery Protocol Switch discovery protocols allow vsphere administrators to determine which switch port is connected to a given vss or vds. vsphere supports Cisco Discovery Protocol (CDP) and Link Layer Discovery Protocol (LLDP). CDP is available for vsphere Standard Switches and vsphere Distributed Switches connected to Cisco physical switches. When CDP or LLDP is enabled for a particular vsphere Distributed Switch or vsphere Standard Switch, you can view properties of the peer physical switch such as device ID, software version, and timeout from the vsphere Client. VMware Recommends: Enable LLDP or CDP in both send and receive mode. 15

5. Network The vsan network should have redundancy in both physical and virtual network paths and components to avoid single points of failure. 16

5.1 Network Availability For high availability, a vsan network should have redundancy in both physical and virtual network paths and components to avoid single points of failure. The architecture should configure all port groups or distributed virtual port groups with at least two uplink paths using different NICs that are configured with NIC teaming, set a failover policy specifying the appropriate active-active or active-standby mode, and connect each NIC to a different physical switch for an additional level of redundancy. VMware Recommends: Redundant uplinks for vsan and all other traffic. 17

6. iscsi Target iscsi networking design with vsan 6.5 18

6.1 iscsi Networking Design Considerations iscsi best practices generally mirror vsan and existing Best Practices for Running VMware vsphere on iscsi. Specific vsan guidance is available iscsi Target Usage Guide. Configuration considerations Like vsan, iscsi does not require Jumbo Frames, but will support them if used. IPv6 as well as IPv4 is supported. When configuring a target you will select a VMkernel port, and the same numbered port will be used on all hosts. For this reason the cluster should have a uniform configuration with all VMkernel ports configured with the same MTU. Unlike iscsi initiator ports, you do not need to tag a port as iscsi, as the target service will manage adding the service to all ports in the cluster. VMware Recommends: Use isolated unique VLANs and VMkernel ports for performance as well as security. Security CHAP with bidirectional handshakes is fully supported by the iscsi target service. Performance and Availability Performance in general will be improved by using more targets, as it will increase access to VMkernel ports, as well as target queue depth. It should be noted that all connections for a target will connect to the same target VMkernel port on a single host. Initial connections can be made to any host, and an iscsi redirect will be used to send the connection to the host owning IO access for a given target. As 19

additional targets are created they will be balanced across the cluster. Any host can take over access in the event of failure of a given target. 6.2 Quality of Service (QoS) Class of service (CoS) and DSCP tags can be used to prioritize iscsi traffic. Consult with your switching vendor for best practices for configuring and tagging VLANs. 20

6.3 VMkernel Port Guidance The VMware iscsi Service, is designed with several assumptions and best practices of design. All hosts will contribute a VMkernel port for the target All VMkernel ports for a target will use the same number (IE a target will use vmk2 on all hosts). Different VMkernel ports can be used for different targets or they can be shared All imitators can see all VMkernel ports used in the cluster A/B network separation (with different non-routed subnets for different VMkernel ports for the same target) are not supported. The VMkernel ports used for iscsi will ideally be used only for iscsi traffic. 21

7. Conclusion vsan Network design should be approached in a holistic fashion, taking into account other traffic types utilized in the vsphere cluster in addition to the vsan network. 22

7.1 Conclusion vsan Network design should be approached in a holistic fashion, taking into account other traffic types utilized in the vsphere cluster in addition to the vsan network. Other factors to consider should be the physical network topology, and the overprovisioning posture of your physical switch infrastructure. vsan requires a 1GbE network at the minimum for hybrid clusters and 10Gbps for all flash clusters. As a best practice, VMware strongly recommends 10GbE network for vsan to avoid the possibility of the network congestion leading to degraded performance. A 1GbE network can easily be saturated by vsan traffic and teaming of multiple NICs can only provide availability benefits in limited cases. If 1GbE network is used, VMware recommends it be used for smaller clusters, and be to be dedicated to vsan traffic. To implement a highly available network infrastructure for vsan, redundant hardware components and network paths are recommended. Switches can be configured either in uplink or stack mode, depending on switch capability and your physical switch configuration. vsan supports both vsphere Standard Switches and vsphere Distributed Switches. However, VMware recommends the use of vsphere Distributed Switches in order to realize network QoS benefits offered by vsphere NIOC. When various vsphere network traffic types must share the same NICs as vsan, separate them onto different VLANs and use shares as a quality of service mechanism to guarantee the level of performance expected for vsan in possible contention scenarios. 23

8. About the Author John Nicholson is a Senior Technical Marketing Manager in the Storage and Availability Business Unit. 24

8.1 About the Author John Nicholson is a Senior Technical Marketing Manager in the Storage and Availability Business Unit. He focuses on delivering technical guidance around VMware vsan solutions. John previously worked in architecting and implementing enterprise storage and VMware solutions. Follow John on Twitter: @Lost_Signal Appendix Multicast Configuration Examples Multicast configuration examples should be used only as a reference. Consult with your switch vendor as configuration commands may change between platforms and versions. Cisco IOS (Default is IGMP snooping on) switch# configure terminal switch(config)# vlan 500 switch(config vlan)# no ip igmp snooping switch(config vlan)# do write memory Brocade ICX (Default is IGMP snooping off) Switch#configure Switch(config)# VLAN 500 Switch(config vlan 500)# multicast disable igmp snoop Switch(config vlan 500)# do write memory Brocade VDX Guide (See guide for vsan VDX configuration) HP ProCurve (Default is IGMP snooping on) switch# configure terminal switch(config)# VLAN 500 ip IGMP 25

switch(config)# no VLAN 500 ip IGMP querier switch(config)# write memory References 1. vsan Product Page http://www.vmware.com/products/virtual-san/ 2. VMware vsan Hardware Guidance http://www.vmware.com/files/pdf/products/vsan/vmware-tmd-virtual-san- Hardware-Guidance.pdf 3. VMware NSX Network Virtualization Design Guide http://www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualizationdesign-guide.pdf 4. Understanding IP Hash Load Balancing, VMware KB 2006129 5. Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/HP switches, VMware KB 1004048 6. Changing the multicast address used for a VMware vsan Cluster, VMware KB 2075451 7. Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environment, VMware KB 2055140 8. IP Multicast Technology Overview http://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/ip_multicast/ White_papers/mcst_ovr.pdf 9. Essential vsan: Administrator s Guide to VMware vsan by Cormac Hogan, Duncan Epping 10. VMware Network I/O Control: Architecture, Performance and Best Practices http://www.vmware.com/files/pdf/techpaper/vmw_netioc_bestpractices.pdf 26

9. Vendor Specific Guidance This section includes links and resources to specific networking vendors. 27

9.1 Cisco ACI Cisco and Cisco Application Centric Infrastructure (ACI) fabric Guidance General Cisco Guidance VMware vsan with Cisco UCS Architecture documents can be found here. Cisco ACI Version Guidance 2.1(1h) is the Cisco ACI release that introduced IGMP snoop static group support and IGMP snoop access group support. VMware has identified this as a minimum release version for a successful vsan experience. You should always consult Cisco's documentation for Minimum and Recommended Cisco ACI and APIC Releases. By default, IGMPsnooping is enabled on the bridge domain. Note Layer 3 IPv6 multicast routing is not supported by APIC IGMP Snooping. In layer 2, IPv6 multicast will be flooded to the entire bridge domain. For these reasons we do not recommend IPv6 with CIsco ACI. For these reasons is is currently not recommended to use IPv6 and Cisco ACI for vsan multicast traffic. For more information on Multicast with Cisco ACI see Cisco APIC and IGMP Snoop Layer 2 Multicast Configuration Below are sample screenshots taken from a Cisco ACI fabric that was configured for VMware vsan. Sample EPG for VMware vsan Sample IGMP snooping configuration Sample IGMP Querier Sample verification of igmp snooping configuration. 28