Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 STO2115BU vsphere Storage Best Practices Thorbjoern Donbaek Rohan Pasalkar #VMWorld #STO2115BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. 2

3 Agenda ESXi Storage Architecture Overview vsphere Storage Configuration and Best Practices PSA Best Practices iscsi Configuration Best Practices VMFS Best Practices NFS Best Practices Questions Please hang onto your questions till end 3

4 ESXi Storage Architecture Overview VMworld 2017 Content: Not for publication

5 ESXi Storage Architecture Overview The flow of an IO VM initiates IO through VMM layer FSS layer can fork into VMFS, NFS etc. RDM goes directly to PSA Device layer VMFS pushes command to PSA Queuing and IO scheduling happens in PSA Device Layer NMP and third party multi-pathing plugins in PSA layer Sub-plugins of NMP are SATP and PSP vscsi FSS + FDS VMFS, PSA Device layer PSA Plugins Application Guest OS vmm Path + Adapter vnvme SATP RDM NMP PSP What we will focus on here Talk focus on VMFS, NFS, PSA, iscsi Native Driver 5

6 vsphere Storage Configuration and Best Practices

7 PSA Best Practices

8 Planned LUN/Disk Removal Best Practice APD is the loss of all paths to a target Best practice: avoid single points of failure in topology to avoid APD PDL is the removal of a LUN at a target In vsphere 5.5 and onwards, a device in PDL will be automatically removed when not in use, but please use planned removal In vsphere 5.5 and onwards, you can terminate a VM on PDL Best practice: Whether device removal reports APD or PDL the planned removal process should be used and is the same First unmount all VMFS volumes on the LUN(s), then detach LUN(s) before finally removing the LUN(s)/target see KB for the exact process vsphere Datastore PDL APD vsphere Datastore 8

9 SCSI Scheduler (pre vsphere 6.0) SCSI Scheduler (post vsphere 6.0) Single queue per <VM, SCSI-LUN> Shares, IOPS limit, reservation settings are cumulatively set for the VM Fairness across VMs but not across VMDKs Virtual disks vsphere VMworld 2017 LUN Virtual disks A queue for every VMDK Shares, IOPS limit, reservation are restricted per-vmdk Fairness across VMs and VMDKs Settings must be done per Virtual disk/vmdk. This mechanism is enabled by default, but can be disabled using the following documentation. Virtual disks vsphere/6.0/com.vmware.vsphere.storage.doc/guid-99c31be3- D39D-4DA1-89A3-EC9DEA5CC24E.html vsphere LUN Virtual disks Content: Not for publication 9

10 SATP and PSP Best Practices SATP should only be changed per storage vendors recommendation/guides Reason is that this is what has been qualified by the storage vendor Always use a PSP that has been qualified by the storage vendor If PSP_RR is one of the qualified options, use PSP_RR Rotates through all active paths Provides best performance with good load spread Now also works with WSFC Best PSP_RR practice Only change settings like iops, bytes, useano per array vendor or VMware recommendations vsphere Datastore 10

11 LUN Count Best Practice Spread your load across many LUNs Helps spread load across paths too Use at least 2x-3x the number of paths Using more LUNs will increase the number of commands outstanding at the array (up to adapter limit) This will increase throughput as long as the array can benefit from more parallelism and you are not bottlenecked on other things (path/adapter bandwidth, CPU ) For heavy IO VMs consider having only one or few VMs per LUN Also consider more LUNs per VM and in extremes more virtual HBAs per VM Active LUNs vsphere No active LUNs Active LUNs vsphere 11

12 Adaptive Queuing by Throttling LUN Queue Depth The VMkernel can throttle the LUN Queue Depth (QD) when congestion is detected Enabled by QFullSampleSize and QFullThreshold values QD reduced when seeing BUSY or QUEUE FULL status for more than QFullThreshold out of QFullSampleSize IOs VMkernel will gradually restore the QD as congestion subsides In vsphere 5.1 patch 1 and onwards, QFullSampleSize and QFullThreshold can be set both globally and per device Per device setting takes precedence if both are set Better as different arrays can benefit from different thresholds vsphere IOs failing with BUSY or QUEUE FULL vsphere Further details are documented in KB Shared LUNs 12

13 WSFC (Clustering) Best Practices WSFC best practices are generally documented in the vsphere guide: Setup for Failover Clustering and Microsoft Cluster Service vsphere/6.5/com.vmware.vsphere.mscs.doc/guid-1a2476c0-ca66-4b80-b6f9-8421b html New: Ensure that the LUN number seen by both hosts is the same. If the LUN number is the same then after adding the RDM disk on one host, the user can add the same RDM disk by selecting "Existing Hard Disk. If the LUN numbers are different then the option Existing Hard Disk can t be used, instead the option "Add RDM disk needs to be used. This is being documented in a KB article. VMware L10 vsphere Add as RDM Shared SCSI Disk L11 VMware Add as RDM L10 vsphere 13

14 iscsi Configuration Best Practices or distribution

15 iscsi Configuration Best Practices ESX support 3 types of iscsi adapters: Software, dependent and independent ESX allows simultaneous use of all 3 types of adapters Mix of SW and HW adapters to access same target not allowed IPv4 & IPv6 Supported on all 3 adapters Mix of IPv4 & IPv6 not recommended Software iscsi provides near line Rate Throughput (10 Gb) HW iscsi Provides Low CPU ESX Utilization 1Gb NIC minimum, prefer 10Gb NIC for multiple iscsi LUNS for better throughput Recommended to use dedicated LAN for iscsi traffic for better security and performance Recommend Jumbo Frames (MTU 9000) enabled E2E for better throughput. Enable digest as a robust mechanism to detect error & IPSec/IPv6/CHAP for enhanced security ibft iscsi boot is supported for both IPv4 and IPv6 under BIOS/UEFI boot mode 15

16 NIC Teaming vs Port Binding Avoid NIC teaming for iscsi To utilize NIC teaming, two or more network adapters must be uplinked to a virtual switch. The main advantages of NIC teaming are increased network capacity and passive failover in the event one of the adapters in the team goes down Best practice for iscsi is to avoid NIC teaming for iscsi traffic and instead use port binding. Port binding fails over I/O to alternate paths based on SCSI sense codes and not just network failures. Also, port bindings gives administrators the opportunity to load-balance I/O over multiple paths to the storage device NIC teaming provides fault tolerance only at NIC/port level. VMworld 2017 Please note that port binding is not supported with NIC teaming Content: Not for publication 16

17 iscsi Port Binding Port binding is used in iscsi when multiple VMkernel ports for iscsi reside in the same broadcast domain and IP subnet to allow multiple paths to be created with target Routing is also supported with port binding from vsphere 6.5 onwards With port binding we try to connect to each target portal from all bound ports, so make sure that all portals are reachable from all bound ports else you may experience discovery/login/rescan delays 17

18 When Not to Use Port Binding Unless you use routing, port binding should not be used when Array Target iscsi ports are in a different broadcast domain and IP subnet VMkernel ports used for iscsi connectivity exist in a different broadcast domain, IP subnet and/or vswitch Per vmk gateway (routing) is required for this to work (available in vsphere 6.5) see next slide VMworld 2017 Content: Not for publication 18

19 iscsi Routing Using Separate Gateway per vmkernel Port and Static Routes From vsphere 6.5 onwards, separate gateway per vmkernel port can be configured To see gateway information per vmkernel port # esxcli network ip interface ipv4 address list Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS vmk DHCP true vmk DHCP true vmk DHCP true With the above, you can use port binding to reach targets in different subnets. You can also configure static routes when initiators and targets are in different subnets 19

20 VMFS Best Practices

21 VMFS Best Practices and Considerations VMFS Datastore Management Storage Space Reclamation (UNMAP) in VMFS-6 Thin vs Thick Storage Provisioning Storage Hardware Acceleration VMFS Snapshot Best Practices and Considerations VMFS Heartbeat and ATS VMFS Troubleshooting 21

22 VMFS Datastore Management (VMFS5 vs VMFS6) Features and Functionality VMFS3/5 VMFS6 4KB Sector readiness No Yes Automatic Space Reclamation No Yes Space Reclamation from guest OS Manual Space Reclamation through esxcli Limited 512e Storage Devices Yes. Not supported on Local 512e Devices. Yes Default Snapshots Access for ESXi host version 6.0 and earlier Yes VMFSSparse for virtual disks < 2TB. SEsparse for virtual disks >= 2TB. Yes Yes Yes SEsparse Note: VMFS3 is deprecated starting ESXi 6.0. No new VMFS3 datastore creation is allowed. We strongly recommend to upgrade your datastore to VMFS5/6. No 22

23 VMFS Datastore Management (Considerations) Migration to VMFS6 version No in-place upgrade from VMFS3 / VMFS5 to VMFS6. Please use storage vmotion to migrate VMs from VMFS3 / VMFS5 datastore to VMFS6 datastore. Following PowerCLI Update-VmfsDatastore makes it easier for setups with large VM to upgrade to VMFS6. Recommendation: Use VMFS6 if compatibility with previous vsphere version is not a requirement. Datastore Extents Best practice: Use homogeneous storage devices (512n or 512e) as extents for spanned VMFS datastore. Storage DRS VMworld 2017 VMFS5 and VMFS6 can coexist in the same datastore cluster. Content: Not for publication Best practice: Use homogeneous storage devices (512n or 512e) for all datastores in the cluster. 23

24 Automatic Space Reclamation for VMFS6 Automatic Unmap Features in VMFS6 Support added for SCSI version 6 and SPC-4 extending UNMAP support to Linux GOS as well. Space reclamation priority is default set to Low (25MB per sec). Space reclamation is automated. Prerequisites for Linux/Windows GOS unmap support The virtual disk must be thin-provisioned. Virtual machine hardware must be of version 11 ( version 13 for Linux GOS) or later. Recommendation Automatic unmap is not supported on storage arrays with UNMAP granularity greater than 1MB. Please use CLI for these storage arrays to reclaim the free space. All GOS generated unmaps may not translate to exact space reclamation ( e.g. non-aligned unmaps). We recommend to use optimize drive or fstrim ( for Linux). 24

25 UNMAP & Thin and Thick Provisioning (VMFS6) LUN VMDK UNMAP GOS UNMAP VMFS Performance Impact Thin Thin UNMAP Supported Automatic UNMAP Asynchronous UNMAP processing with minimal performance impact. Thin Thick No UNMAP Automatic UNMAP Asynchronous UNMAP processing with minimal performance impact. Thick Thin UNMAP Supported No UNMAP Just meta data updates. No performance impact. Thick Thick No UNMAP No UNMAP No Impact VMworld 2017 Content: Not for publication 25

26 Thin vs Thick Storage Provisioning Virtual Disk Type Storage Space Allocation In Guest UNMAP Support I/O Performance* (*First Write) Virtual Disk Creation Time Eager Zeroed Thick Virtual Disk Upfront Lazy Zeroed Thick Virtual Disk Upfront Thin Virtual Disk On demand as virtual disk is written No No Yes Highest Less than EZT Less than LZT Higher Medium Instant 26

27 Storage Hardware Acceleration Best Practice: Use hardware acceleration supports if supported by backend Storage Device. For XCOPY, default transfer size is 4MB. Support for different sizes for different vendors. Recommendation: Please consult your storage vendor for recommended setting for your usage. In a storage DR configuration, please ensure the remote Storage Array also has ATS enabled. Reference: Frequently Asked Questions for vstorage APIs for Array Integration Refer KB VMworld 2017 Content: Not for publication 27

28 VMFS Snapshot Best Practices and Considerations Best Practices for using Snapshot VMs with snapshot has performance penalty compared with VMs without snapshot. Reduce the amount of time you run on snapshot. Limit the number of snapshot levels active at a given point in time for optimal performance. To help reduce consolidation time run VMs for lesser duration on snapshot. When consolidation fails, some of the delta disk are left behind. Periodically go and remove them. Considerations: SESparse is default snapshot type for VMFS6. VMworld 2017 Content: Not for publication If you migrate a VM with the VMFSsparse snapshot to VMFS6, the snapshot format changes to SEsparse. 28

29 VMFS Heartbeat and ATS VMFS Heart Beat VMFS relies on on-disk locks to arbitrate access to shared storage. It uses on-disk heartbeat to indicate liveness of hosts using the filesystem Starting ESX 5.5x VMFS HB started using ATS. Not using ATS exposes the issue of Delayed write ESX 6.5 (and ESX 6.0 U3) introduces hardening of ATS HB algorithm to handle erroneous miscompares from underlying storage. Best Practice: Use ATS for Heart Beat. 29

30 VMFS Troubleshooting VOMA Using vsphere On-Disk Metadata Analyzer Check vmfs, lvm metadata errors voma m vmfs -f check -d /vmfs/devices/disks/<device> Fix vmfs, lvm metadata errors voma m vmfs -f fix -d /vmfs/devices/disks/<device> Collect metadata dump voma -f dump -d /vmfs/devices/disks/<device> -D <path for dump file> Recommendation: Power off VM or migrate them to different datastore before using VOMA. Please refer KB for more information. 30

31 NFS Best Practices

32 NFS Configuration Best Practices (NFSv3 & NFSv4.1) Network considerations: Minimum NIC speed 1Gb (recommended speed 10Gb) Use separate vmkernel port group for NFS traffic Avoid multiple hops between ESXi host and NFS server Configure same MTU for all the devices in the path (recommended MTU 9000) Recommendation: Configure with no Single Point NFS Storage Of Failure. VMware EXi Switch 32

33 NFS Multi-pathing Best Practices (NFSv4.1 Only) Configure shares for Multi-pathing (Session Trunking) VMware Exi vmk x Interface in Same Subnet y z NFS41 Storage vswitch1 vmk 1 VMware Exi vmk 2 vswitch x x y y Interface in Different Subnet NFS Storage 33

34 NFS Security Best Practices (NFSv3 & NFSv4.1) NFSv3: ESXi server mount the NFS shares with root access To address this concern, either use dedicated LAN or VLAN NFSv4.1: NFSv41 supports kerberos, for User authentication (krb5) Integrity of RPC headers (krb5i) 34

35 Questions?

36

37

38 ESXi Storage Architecture Overview CONFIDENTIAL 38