Best Practices for Deploying VMw are and Dell EqualLogic

Transcription

1 Best Practices for Deploying VMw are and Dell EqualLogic Andy Banta iscsi Development VMware William Urban, VCP Product Solutions Engineer Dell Dell Storage Forum: EqualLogic User Conference San Diego, October 4-8, 2010

2 Notices & Disclaimers Copyright 2010 by Dell, Inc. No part of this document may be reproduced or transmitted in any form without the written permission from Dell, Inc. This document could include technical inaccuracies or typographical errors. Dell may make improvements or changes in the product(s) or program(s) described herein at any time without notice. Any statements regarding Dell s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. References in this document to Dell products, programs, or services does not imply that Dell intends to make such products, programs or services available in all countries in which Dell operates or does business. Any reference to an Dell Program Product in this document is not intended to state or imply that only that program product may be used. Any functionality equivalent program, that does not infringe Dell s intellectual property rights, may be used. The information provided in this document is distributed AS IS without any warranty, either expressed or implied. Dell EXPRESSLY DISCLAIMS any warranties of merchantability, fitness for a particular purpose OR INFRINGEMENT. Dell shall have no responsibility to update this information. The provision of the information contained herein is not intended to, and does not, grant any right or license under any Dell patents or copyrights. Dell, Inc. 300 Innovative Way Nashua, NH USA 2

3 Network, Network, Netw ork! 3

4 As with all things, it starts with the Infrastructure Switching Infrastructure forms the backbone of all VMware and EqualLogic installations Flow Control Jumbo Frames (Not required but supported) Enterprise Class switching infrastructure Sufficient buffers for port groups Sufficient bandwidth between switches for intergroup communication 4

5 As with all things, it starts with the Infrastructure End Node Ports should use Port Fast/Spanning Tree/FST to allow for control module failover to happen in correct time Array doesn t support VLAN Tagging but will work in 10G environment that has it and will unpack the packet as long as Array Ports are untagged Multitude of User Conference sessions dedicated to Networking Topics 5

6 St o rage Sizing and Performance 6

7 RAID Type Considerations RAID 5 Slowest Raid type and Rebuild Type Useful when space is of the essence RAID 50 Most commonly used, best space/ performance ratio Moderate Rebuild Impact RAID 10 Fastest Read/Write at the cost of space Minimal Rebuild Impact RAID 6 Excellent reads and availability 7

8 Volume Size and Limitations While VAAI improves LUN locking there are still only so many operations per VMFS volume Configure volume types to match IO types Is there R10 SAS/ SSD datastores? Slower IO on Raid 50 SATA? 2TB is still the size limit for VMFS datastores 8

9 Virtualization changes how connections to the array work Because each ESX host needs to talk to the volume, connection counts can become increasingly high if not adequately configured Formula: Number of ESX Servers * Number of Sessions (2,4,6 etc) * Number of Volumes + overhead for iscsi in the guest and snapshots increased connection count to 1024 per pool for 4096 per group 9

10 Data Drives 10

11 Performance Tuning for Data Drives Three ways to connect Data Drives VMDK on VMFS iscsi in the Guest (Sometimes called Direct Connect) Raw Device Mapping (RDM) 11

12 VMDK on VMFS Advantages Easy to configure Viewable in Virtual Center (clones, templates, storage vmotion, etc) Doesn t require Storage Team interaction Allows for tiering of data drives to different VMFS volumes/pools based on workload Uses vsphere MPIO (New in 4.x) for high bandwidth Apps Disadvantages No host integration with ASM/ME If separated from Datastore that OS resides on no ASM/VE integration No real isolation of Data for Data Protection/ DR strategies 12

13 VMDK on VMFS Best Practices Configure virtual disk with new SCSI adapter, change adapter type to Paravirtual SCSI adapter (2008/2003/RHEL5) Align data partition in OSes that do not automatically offset 64k (W2k3 for example) 13

14 iscsi in the Guest VM Advantages Overcome 2TB Limits of VMFS/ RDM Utilize host integration software such as ASM/ME and off host backup Isolates the data drives for Data Protection/DR strategies Can be mounted on physical or other virtual machines Uses same best practices from physical environments Allows for tiering of data drives to different volumes/pools based on workload Disadvantages Isn t visible to Virtual Center Requires Storage Team intervention More VM CPU overhead as the volume is only seen in the VM Needs to be considered for DR Plans separate from VMs 14

15 iscsi in the Guest VM Best Practices For VM MPIO Configure Active/Standby VM Nics since ESX doesn t guarantee splitting bandwidth Use DSM in Windows 15

16 Raw Device Maps Advantages Easy to configure One volume = One RDM Viewable in Virtual Center Storage View Allows for tiering of data drives to different volumes/pools based on workload Uses vsphere MPIO (New in 4.x) for high bandwidth Apps Isolates data from OS and a lot of 3 rd party apps work with RDMs Disadvantages No host integration with ASM/ME or ASM/VE Each ESX server needs connectivity so connection count concerns 16

17 RDM Best Practices Configure virtual disk with new SCSI adapter, change adapter type to Paravirtual SCSI adapter (2008/2003/RHEL5) Align data partition in OSes that do not automatically offset 64k (W2k3 for example) 17

18 So which one do I use? All of them offer advantages whether based on administrative overhead, simplicity, or host integration No one solution is right, use them all to meet the business goals you have 18

19 iscsi in VMw are 19

20 MPIO Configurations with different versions of vsphere vsphere 4.0 Patch 5 or Update 2 Required (iscsi connection drops on same subnet) Patch ESX SG KB# :1 vmkernel to physical nic binding (New update to best practices forthcoming) vsphere 4.1 Enterprise or Enterprise Plus license Dell EqualLogic MEM 1:1 vmkernel to physical nic binding (setup.pl configures this for you) vsphere 4.1 without E/ EP license 1:1 vmkernel to physical nic binding (can still use setup.pl) 20

21 iscsi Improvements and Features

22 Multiport Performance Improvements New vswitch Port Use Model for Performance 1- to- 1 Relationship vmknic to PNIC All ESX iscsi NICs Share iscsi Name Each vmknic Has Different IP Info Allows Multiple Sessions with Same Target Therefore Multiple Paths to Same LUN Storage- side PSA Takes Care of Path Policies

23 Different HBA Models Independent iscsi HBAs The Current Qlogic HBA Model HBA Configuration Does Not Use ESX Configuration HBA has all IP and iscsi Discovery Information Therefore, No ESX Networking Configuration is needed Management Handled by the HBA This requires a plugin for the HBA

24 Different HBA Models Dependent iscsi HBAs Configuration Stored with ESX SW Initiator HBA borrows IP Identity of a VMKNIC IP Address, MAC, VLAN ESX Does Discovery, Authentication iscsi Traffic not seen on the VMKNIC Management Handled by ESX No plugin is required for the HBA

25 ESX iscsi Implementation

26 ESX iscsi Management Implementation

27 iscsi Configuration Exam ples

28 Configuring iscsi Multipathing Optional Configuration Not Enabled by Default Configurations from prior releases still work Referred To In Many Places as Port Binding Available for the ESX SW Initiator and Broadcom HBAs Supported on Other Dependent Adapters As They re Available Uses the vsphere Client and CLI for configuration Can be done entirely on the command line Limitations No Ability to Use iscsi SAN Routing

29 iscsi With NIC Teaming Conventional NIC teaming in ESX

30 Configuring ESX iscsi Overview Each iscsi NIC on the Host is a Storage Path Endpoint Different than Pre- vsphere 4 Where Host was One Endpoint ESX Initiator Still a Single vmhba Each Uplink has an Associated VMKNIC Works with One or Mutliple vswitches Legacy or vds Supported NIC Teaming Policy of One Active Uplink and No Standby Links

31 ESX iscsi Multipath Configuration

32 Multipath Configuration Storage Systems Presenting Multiple Portals End Up with as Many Paths as the Number of ESX Ports Times the Number of Storage Ports.

33 Configuring ESX iscsi Overview Each VMKNIC Has Different IP Configuration VMKNICs Associated With the iscsi HBA iscsi Uses Each VMKNIC as a Storage Port Storage Multipathing Sees Separate Paths to Storage Works with NMP, EqaulLogic MEM and Other Storage Vendors

34 Multipath Configuration Two paths through the two ports

35 Broadcom iscsi HBA Configuration New Support in ESX 4.1 Supported with Licensed Broadcom Adapters based for 1Gbit based for 10Gbit Licensing Provided by OEM Vendors Configuration Similar to ESX iscsi Mulipathing Configuration Each Broadcom Port is a Separate vmhba Limited to 1 VMKNIC for iscsi Use Per NIC

36 Broadcom iscsi HBA Configuration Limitations No Jumbo Frames 64 Session per HBA max iscsi Traffic NOT Recorded or Figured Into NIC Traffic Stats

37 Multipath Configuration

38 Set Active Uplink for VMKNIC Just Like ESX iscsi Configuration, Move All But One Adapter from Active to Unused

39 Broadcom iscsi Configuration Identify Adapters and Uplinks

40 Broadcom iscsi Configuration Configure VMKNICs on vswitch

41 Broadcom iscsi Configuration Associate VMKNICs w ith Broadcom HBAs

42 Broadcom iscsi Configuration HBAs Configured and Ready to Use

43 vds iscsi Configuration Similar to Legacy vswitch Require vds iscsi Implementation to Work VMware and Cisco Nexus Include Implementation Useful On Systems With Limited Ethernet Ports For Example: Blade Systems with 2 10G Ports, Only Limit of One Uplink per VMKNIC Portgroup Can t Move between Uplinks VMKNICs Associated with Unpopulated Uplinks Are Down

44 vds iscsi Configuration Cisco Nexus 1000V Configuration Make Modifications to Port Profile Map Portgroup to VMware Portgroup Name n1000v(config- port- prof)# vmware port- group VMK- portgroup- name Add iscsi- multipath Capability to the Port Profile n1000v(config- port- prof)# capability iscsi- multipath See Cisco Nexus 1000v Documentation

45 vds Configuration Example Start at the vcenter Home page and select networking

46 vds Configuration Example Edit the portgroup for the iscsi vmknic

47 vds Configuration Example Move all but one uplink to unused adapters

48 vds Configuration Example Create vds Port Group by selecting New dvswitch and creating a new port group

49 vds Configuration Example Update vds PG Policy for Teaming and Failover, Ensure only one Active dvuplink is associated with this port group

50 ibft Configuration Information ibft NIC Boot Capability Initial Support for Broadcom NIC Certification Plan Available Current Certification for 1G, 10G Broadcom and 10G Intel adapters ESXi No ibft Support for ESX Classic Limitation of Dependent HBA using ESX Configuration ESXi Network Configuration Automatically Uses ibft Config

51 iscsi Configuration Best Practices

52 Jumbo Frames Must Be Configured End- to- End, Three Places in ESX Configuration

53 Jumbo Frames vmknics Must Be Removed and Re- added Use vmkping to Test MTU Size vmkping s 9000 I vmk# <IP address>

54 Ethernet Congestion iscsi Storage Is Intended to Be Shared Links Are Shared Also TCP Guarantees Reliable Delivery Done by Retransmitting Packets That Are Corrupt or Lost Even with Flow Control, Ethernet Switches Drop Packets Most Switches Show Statistics ESX Retransmission Statistics Are Available VLANs Are Virtual Separation Physical Oversubscription Still Causes Losses

55 Ethernet Congestion Oversubscribed Links Cause Dropped Packets and Retransmissions

56 Ethernet Congestion Use Adequate Links for the Load

57 Delayed ACK

58 Delayed ACK Delayed ACK Not Supported by Some Storage Systems Should Always Be Enabled When It Can Be False Performance Reporting File Copy in the COS or Guest OS is Deceptive Queues Aren t Used Fully Latency, Rather Than Throughput Measured Delayed ACK Latency Worse When TCP Session Isn t Heavily Used IOmeter or Blaster Are Better Performance Testers Console O/S Never Good For Measuring Performance

59 NMP Round Robin Tuning Default Round Robin Policy Sends 1000 I/Os to Each Path Many Storage Systems Don t Queue That Much Use CLI to Change Policy

60 NMP Round Robin Tuning

61 NMP Round Robin Tuning Disadvantages ESX Multipath Switching Uses Some Cycles Can Severely Degrade Storage System Caching Some Storage Vendors Use Cache on Each Path

62 Tuning iscsi Login Parameters Parameters Are Very Storage Dependent Pretty Well Tuned, Don t Expect Miracles Parameters InitialR2T Ack Before Sending First Data InitialBurst Size of First Transfer MaxR2T Number of Outstanding Transfers LoginTimeout, Retrys DataDigest, HeaderDigest Trade Performance for Integrity LogoutTimeout NoOpTimeout Changes Failover Behavior

63 iscsi Configuration Pitfalls

64 Broadcom iscsi Configuration Pitfalls NIC Not Licensed for iscsi Use No vmhba Associated with bnx2i Driver in esxcfg- scsidevs a Messages inv vmkernel log bnx2i: LOM is not enabled to offload iscsi connections, dev=vmnic# bnx2i: iscsi not supported, dev=vmnic# OEMs License NICs So ftw are- enabled by some No ESX utilities for lic ensing Hardware- enabled by some

65 Broadcom iscsi Configuration Pitfalls Very Bad Read Performance Out of Order Frames on the Wire Reassembled on Broadcom Chip Enable Flow Control to Clear This Up

66 iscsi Command Line Tools

67 iscsi Troubleshooting Tools Command Line tools for tracking down problems esxcli esxtop vmkiscsi- tool /proc nodes Switch and router utilities

68 iscsi Troubleshooting Tools

69 Ethernet Congestion Use Switch Statistics Or esxtop Network Options

70 EqualLogic Session Information

71 Qlogic HBA /proc Node

72 ESX iscsi Adapter /proc Node

73 Broadcom HBA /proc Node

74 Want to learn more? More resources can be found online at - > Sign into the EqualLogic Customer Support - > Resources Drop Down Tech Reports, On Demand Webcasts and more! 74

75 Questions? Come see the SRM Session Wednesday 2:10-3:10 Come see the EqualLogic MPIO Tools and learn how to configure and install Microsoft DSM and VMware MEM Tuesday 4:30-5:30 Come see the Data Protection and Disaster Recovery with Auto Snapshot Manager/ VMware Edition Tuesday 9:45-10:45 or Thursday 10:15-11:15 Stop by the expo for hands on training/demo/discussion Find us anytime to chat! 75