Reference Architecture DataStream Architecting DataStream Network Document # 317-0026NA Version 1.03, January 2016 www.cohodata.com
Abstract This document provides an overview of data center networking concepts in order to assist in making storage network design decisions. Audience Intended for individuals responsible for implementing DataStream storage. Prerequisite Knowledge Basic knowledge of data center networking concepts. Scope This document addresses the following topics: Isolating NFS traffic Load balancing MTU Spanning tree Physical and logical network topologies Additional Help For additional information about the information contained in this reference guide, we recommend that you first consult your third party switch manufacturer documentation. You may also contact Coho Data technical support; see the Coho Data support services page for more information. 317-0026NA Architecting DataStream Network 02.6.1.03 1
Isolating NFS traffic It is good practise to separate different traffic types using 802.1Q VLAN tagging. Segregation can prevents unauthorized access, broadcast storms, and provides flexibility to implement policies based on VLAN ID. Management Traffic The management network is designed to provide access to vsphere administration services. Management traffic should be on a dedicated management VMkernel port and VLAN. Traffic should be routed only to networks that need to configure and manage vsphere. Virtual Machine Traffic Virtual machine (VM) traffic should be isolated on its own network, separate from the vsphere Management Network. vmotion Traffic It is recommended to have vmotion traffic on a different VMkernel port(s) as your storage and management traffic. vmotion traffic should be isolated from the rest of the network to prevent man-in-the-middle (MitM) attacks as vmotion traffic is sent in plain text. Fault Tolerance Logging Traffic Fault Tolerance (FT) logging traffic is unencrypted and contains VM guest network and storage I/O data. It is recommended to use an isolated VM port group to avoid MitM attacks. IP Storage Traffic It is recommended to isolate VM Storage traffic (i.e. NFS & iscsi) from the rest of the production VM, Management, vmotion, and FT Logging traffic to avoid any network congestion/performance issues. Storage traffic should not be routable to other networks via a layer 3 router/switch. Storage traffic should be using a VMkernel port(s) and an isolated VLAN ID. See the Resources page for additional reading on network isolation practices and the VMkernel networking layer. 317-0026NA Architecting DataStream Network 02.6.1.03 2
VLAN tagging A cost effective and simple way to isolate NFS traffic is to use VLAN tags. The 802.1Q open standard (VLAN tagging) was developed to allow physical switches to be divided into virtual LANs. Only ports which the same VLAN membership will forward traffic to each other. When 802.1Q tagging is implemented, a 4 byte tag is added to the ethernet frame. Within those 4 bytes, 12 bits are reserved specifically for a VLAN ID. Ethernet Frame with 802.1Q Tag. Frame53(70bytesonwire, 70bytescaptured) EthernetII, Src: 00:40:05:40:ef:24, Dst: 00:60:08:9f:b1:f3 802.1qVirtualLAN 000...=Priority:0...0...=CFI:0... 000000100000=ID: 100 Type: IP(0x0800) InternetProtocol, SrcAddr: 131.151.32.129(131.151.32.129), DstAddr: 131.151.32.21 (131.151.32.21) TransmissionControlProtocol, SrcPort: 1173(1173), DstPort: 6000(6000), Seq: 0, Ack: 128,Len:0 Example PCAP of Ethernet Frame with VLAN ID 100. 317-0026NA Architecting DataStream Network 02.6.1.03 3
vswitches and VLAN tags It s highly recommended that NFS traffic is isolated from other network traffic. NFS traffic should be isolated on vswitches by using either isolated vswitches or dedicated vswitch port groups, with a configured VLAN ID. How vswitches handle tagged and untagged traffic not only depends on the vswitch configuration but also on how the upstream physical switch is configured. External Switch Tagging In this configuration, the connected physical switch ports are configured in access mode. This means that the physical switch removes the VLAN tags from the frame before forwarding the traffic to the virtual switch. For frames originating from the VMkernel, the vswitch does not add a VLAN tag to the frame, rather, it forwards the traffic as untagged to the connected physical switch. In turn, the physical switch adds a tag to the frame with a configured VLAN ID of the connected access port. In this case, the vswitch port groups cannot use VLAN IDs. This scenario is only appropriate if a dedicated vswitch is used for NFS VMkernel traffic. Virtual Guest Tagging In this configuration, the physical switch ports are configured in trunk mode and pass traffic to the vswitch, with the VLAN tags intact. The virtual switch also leaves the VLAN tags in place and passes the frame to the VM s NIC or a VMkernel port. In this case, the VM or VMkernel port must be configured specifically to handle tagged VLAN traffic; if not, the traffic is dropped. Traffic originating from the VMkernel or from a VM is sent with a VLAN tag already in place. This configuration is typically used for traffic sniffing, rather than for NFS VMkernel deployments. Virtual Switch Tagging In this configuration, the VMkernel port group has a configured VLAN ID. The physical switch s ports are configured in trunk mode and pass the traffic to the vswitch, with the VLAN tags intact. The vswitch inspects the tag s VLAN ID and forwards the frame to the port group with the same VLAN ID. Before forwarding the frame, the vswitch strips the 802.1Q tag from the frame. For frames originating from the VMkernel, the vswitch adds the VLAN ID, of originating port group, to the frame before forwarding it to the upstream physical switch. Virtual switch tagging is generally the recommended method of isolating NFS traffic. 317-0026NA Architecting DataStream Network 02.6.1.03 4
Port Types The port type configuration of a switch port or port channel dictates how that devices treats VLAN tags. There are two basic port types: 1. Access 2. Trunk Access Port A port which allows only frames for a single VLAN is called an access port. Access ports can either allow tagged traffic or untagged traffic but not both. If the access port is assigned a VLAN and an untagged frame enters an access port, an ingress action, the frame will have the VLAN tag added by the switch. The VLAN ID that is added to the frame is the configured VLAN of that port. Typically client endpoints will send an untagged frame to the switch, which in turn, adds a tag and forwards the Frame. For example, if an untagged frame enters port 21 and port 21 is a member of VLAN 100, port 21 adds a 802.1Q tag with VLAN ID 100 to the frame. Access Port Adding a VLAN Tag. If a tagged frame is forwarded to an access port, the switch will inspect the tag and if the VLAN ID matches the VLAN membership of the access port, the tag is removed and the frame is forwarded. Access Port Removing VLAN Tag. If you don t explicitly assign an access port to a VLAN, by default, the port will be a member of the default VLAN ; typically VLAN 1. VLAN 1 is reserved for internal use, all traffic on VLAN 1 will be sent and received without a 802.1Q VLAN tag. Some switches have a feature called switchport host, this feature is not compatible with LAGs (aka port channels) and typically should not be for DataStream connected ports. 317-0026NA Architecting DataStream Network 02.6.1.03 5
Trunk Port A trunk port can carry untagged packets simultaneously, with the 802.1Q tagged packets, from multiple VLANs. Trunk ports handle tagged and untagged traffic in a different manner than access ports. When a client endpoint or upstream switch forwards a frame to a trunk port, the trunk port will inspect the 802.1Q tag s VLAN ID and if the port is a member of the tagged vlan, the frame is forwarded with the tag in intact. Trunk Native A trunk port can also send and receive untagged packets. The untagged VLAN is called the NATIVE VLAN. A native VLAN ID must be explicitly specified on the trunk. For example, if the trunk port s native VLAN is VLAN 5. Traffic from VLAN 5 will be handled in the same manner as if it were handled by an access port. If trunk port is also a member of VLAN 100, 200, and 300, traffic from these VLANs would have to arrive with a VLAN tag in place and will remain tagged when forwarded by the trunk port. 317-0026NA Architecting DataStream Network 02.6.1.03 6
DataStream VLAN config DataStream appliances can be either configured with or without a VLAN ID VLAN tagging for the NFS network can be configured by Clicking Change Storage Network Settings, from the DataStream UI, under Settings > Networking. Without a specified VLAN ID, the DataStream switches will forward untagged frames. Connected switch ports should be configured in mode access with a specified VLAN ID. Adding a VLAN ID is the equivalent of configuring the the DataStream client ports in mode trunk. The DataStream switches will receive and forward NFS traffic with a 802.1Q tag in place. Connected switch ports should also be configured as trunks. DataStream OS 2.6.x - Settings > Networking. See the DataStream user guide for more information on how to configure DataStream network settings. 317-0026NA Architecting DataStream Network 02.6.1.03 7
Link Aggregation Group (LAG) Also known as port channels, LAGs bundle physical interfaces into a single aggregated logical interface. LAGs are implemented to add redundancy eliminating single points of failure to increase availability. A basic LAG bundles two or more interfaces on a switch into a single logical link. If one interface goes down, the other will continue sending traffic. The diagram below shows two switches (switch 01 and switch 02) each has a local LAG that merges 2 ports into a single logical port. Single chassis LAG. A multi-chassis LAG (vpc, VSS, MLAG) merges ports on two or more switch chassis into a single logical link eliminating single points of failure at the switch chassis level. The diagram below shows two switches (switch 01 and switch 02) each has a local LAG that merges 2 ports into a single logical port. These two local logical ports are then merges into single local port that spans both switch chassis. Switches with Multi-Chassis LAG. 317-0026NA Architecting DataStream Network 02.6.1.03 8
Link Aggregation Control Protocol Active LAGs use a control protocol to manage peering with an neighboring LAG. DataStream appliances support active LAGs using Link Aggregation Support Protocol (LACP) as well as LAGs without a control protocol, Static LAGs (mode on). Active LACP Ports in a LAG configured to use active LACP will transmit LACPDUs. LACPDUs will transmitted even if the port s counterpart is not configured to transmit LACPDUs, and even if the port s counterpart is not even configured to use LACP. Passive LACP Ports in a LAG configured to use passive LACP will only transmit LACPDUs. LACPDUs are a response to a LACPDU request. Static LAG Static LAGs do not use LACPDUs and will not respond to or transmit LACAPDU packets. Load-Balancing Traffic is load balanced across interfaces in a LAG in a variety of ways. Typically load-balancing policies are based on layer two or layer three fields suchs as source or destination MAC, source or destination IP, or the IP header s protocol field. The type of load-balancing configured on a intermediary switch or a vswitch will dictate the type of topology that should be used. 317-0026NA Architecting DataStream Network 02.6.1.03 9
VMkernel NIC Teaming VMware vsphere vswitches and vswitch port groups support the configuration of NIC teams. NIC teams provide a hot failover in the event of a hardware failure. Depending on the NIC teaming load balancing policy selected, the upstream switch may or may not need to have LAGs configured. The advantages and disadvantages of using vswitch IP hash load balancing policy are outlined in VMware KB 2006129. Deployments with a NIC teaming load balancing policy configured to Route based on IP hash require link aggregation (LAG) to be configured on any ESXi connected switches. Deployments where the VMkernel NIC teaming load balancing policy is configured to Route based on source MAC hash or Route based on the originating virtual port ID, the connected do not require LAG to be configured on the intermediary switch. For DataStream deployments (direct connect or with intermediary switch) route based on source MAC hash is the recommended VMkernel NIC team load balancing policy. 317-0026NA Architecting DataStream Network 02.6.1.03 10
IP Hash Intermediary Switch When using IP hash load balancing, the DataStream Arista switch ports must be LAGed. Ports can be merged into LAGs using the DataStream UI. When selecting a port on the primary switch, the corresponding port on the secondary switch is automatically selected. The cabling must be connected in a way that is conducive to this workflow. DataStream switches support LACP active, LACP passive, and static LAGs. DataStream OS 2.6.x - Hardware > Switches View. Route Based on IP Hash. 317-0026NA Architecting DataStream Network 02.6.1.03 11
It is not necessary to match the load-balancing policy between the intermediary switch or the ESXi vswitch. Regardless of what is configured on the intermediary switch it is recommended that you use routed based on source MAC hash or route on based on virtual port ID for your vswitch NIC teaming load balancing. MAC Hash On Intermediary Switch When using MAC hash load-balancing on the intermediary switch, the DataStream Arista switch ports do not need to be LAGed. MAC Hash on Intermediary Switch. MAC hash is the preferred load-balancing policy for most DataStream deployments. 317-0026NA Architecting DataStream Network 02.6.1.03 12
Spanning-Tree A huge benefit MLAG (LAG across multiple chassis) is the ability to turn a redundant active/standby (with two switches) path to an endpoint into a reducanent active/active path. This change in capabilities has to do with how spanning tree views LAGs. In this topology spanning tree will block one of the active paths to the DataStream switches. The end-end path is redundant, but in active/standby. LAGs are Local and Independent on each Intermediary Switch. Active path Disabled by STP 317-0026NA Architecting DataStream Network 02.6.1.03 13
In this topology the LAG appears to be a single switch to spanning tree (STP). Because the multi-chassis LAG domain appears to STP as a single switch, there are no blocked ports. The redundant path is active/active. LAG is Used to Create a Single Logical Intermediary Switch. Active path Disabled by STP Applies to Arista MLAG, Cisco vpc, Cisco VSS, cisco StackWise, Brocade VLAG, HP IRF, and many more vendor specific implementations. 317-0026NA Architecting DataStream Network 02.6.1.03 14
STP Port Types Because the DataStream switches act as endpoints their client facing ports are configured as STP port-type edge. Edge ports do not send BPDU packets. It is recommended that the intermediary switch ports connected to the DataStream switches are configured as STP port-type edge. This configuration prevents the DataStream switches from becoming part of the existing networks STP topology. Cisco IOS switches may prompt with the following error: {%PM-4-ERR_DISABLE: channel-misconfig(stp) errordetectedongi0/35, putting Gi0/35inerr-disablestate} To fix: switch(config-if)#spanning-treeportfastdisabled Jumbo Frames Jumbo frames are ethernet frames with an MTU larger than 1500 bytes. To enable jumbo frames the MTU must be changes from the default 1500 bytes to a larger size. Typically 9000 (vsphere) or 9216 (Cisco). Arista switches allow jumbo frames by default. The DataStream UI provides an avenue for configuring a global MTU value for your entire DataStream cluster. 317-0026NA Architecting DataStream Network 02.6.1.03 15
Coho Data Support Services Every effort was made to ensure a trouble-free experience. Should you experience any 1 difficulty with your Coho Data products, please contact Coho Data Technical Support. The Coho Data support center can be reached in the following ways: North America 855-786-2646 Email support@cohodata.com Web www.cohodata.com/support Document Feedback We appreciate any feedback about this document. Whenever possible or applicable, please include the document number (on front page), the title, version number, and the specific chapters or paragraphs. Send your comments to: documentation@cohodata.com. Note: All comments become the property of Coho Data. 1 Please consult your technical support agreement for more information about your support tier. 317-0026NA Architecting DataStream Network 02.6.1.03 16
Resources Coho Data DataStream User Guide (see the Coho Data Customer Support Portal ) Cisco Nexus vpc Reference Architecture DataStream Arista MLAG Reference Architecture UCS Direct Connect Reference VMware Understanding IP Hash Load Balancing (KB 2006129) Adopting Sound Network Isolation Practices Setting Up VMkernel Networking - VMkernel Networking Layer 317-0026NA Architecting DataStream Network 02.6.1.03 17
About the Author Urs is a Solutions Architect at Coho Data. Urs specializes in automation, virtualization, and networking design. Urs has several years experience supporting enterprise hardware and software solutions across a variety of distinguished companies. Prior to working as an Solutions Architect, he as an Escalation Engineer at Coho Data, he also spent several years as a Systems Administrator before joining Coho Data. Urs is a VMware VCP-DCV, VCP-DT, and holds several more distinguished industry certifications. 2016 Coho Data. All rights reserved. The Coho Data logo, DataStream, and MicroArray are trademarks of Coho Data in Canada, United States and other jurisdictions. All other trademarks, service marks, and trade names referenced in this document are those of their respective owners. No part of this document or other Coho Data document assets may be reproduced without the express written consent of Coho Data. Every effort has been made to ensure that the information in this document is accurate; however, errors and/or omissions in content are possible. In no event shall Coho Data be liable for incidental or consequential damages arising from use of this document or the software and hardware described in this document. Content is subject to change without prior notice. 317-0026NA Architecting DataStream Network 02.6.1.03 18