Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Size: px

Start display at page:

Download "Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme"

Lucas Daniel
6 years ago
Views:

1 SER1143BU A Deep Dive into vsphere 6.5 Core Storage Features and Functionality Cormac Hogan Cody Hosterman VMworld 2017 Content: Not for publication #VMworld #SER1143BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. 2

3 Introduction

4 Welcome from Cormac and Cody Cormac Director and Chief Technologist Cody Technical Director for VMware Solutions Pure 4

5 Agenda Slide 1 Limits 2 VMFS-6 3 VAAI (ATS Miscompare and UNMAP) 4 SPBM (SIOCv2 and vsphere VM Encryption) 5 NFS v4.1 6 iscsi 7 NVMe 5

6 vsphere 6.5 Storage Limits

7 vsphere 6.5 Scaling and Limits Paths ESXi hosts now support up to 2000 paths Increase from the 1024 per host paths supported previously Devices ESXi hosts now support up to 512 devices Increase from the 256 devices supported per host previously Multiple targets are required to address more than 256 devices This does not impact Virtual Volumes (aka VVols), which can address 16,383 VVol per PE 7

8 vsphere 6.5 Scaling and Limits 512e Advanced Format Device Support Capacity limits are now an issue with 512n (native) sector size used currently in disk drives New Advanced Format (AF) drives use a 4K native sector size for higher capacity These 4Kn devices are not yet supported on vsphere For legacy applications and operating systems that cannot support 4KN drives, new 4K sector size drives that run in 512 emulation (512e) mode are now available These drives will have a physical sector size of 4K but the logical sector size of 512 bytes These drives are now supported on vsphere 6.5 for VMFS and RDM (Raw Device Mappings) 8

9 512n/512e # esxcli storage core device capacity list [root@esxi-dell-e:~] esxcli storage core device capacity list Device Physical Logical Logical Size Format Type Blocksize Blocksize Block Count naa.624a9370d4d78052ea564a7e MiB 512n naa.624a9370d4d78052ea564a7e MiB 512n naa.624a9370d4d78052ea564a7e MiB 512n naa.624a9370d4d78052ea564a7e MiB 512n naa.55cd2e404c31fa MiB 512e naa.500a07510f86d6bb MiB 512e naa.500a07510f86d MiB 512e naa.5001e f MiB 512n 512e (emulated) 512n (native) 9

DSNRO The setting Disk.SchedNumReqOutstanding aka No of outstanding IOs with competing worlds has changed in behavior DSRNO can be set to a maximum of: 6.

10 DSNRO The setting Disk.SchedNumReqOutstanding aka No of outstanding IOs with competing worlds has changed in behavior DSRNO can be set to a maximum of: 6.0 and earlier: and on: Whatever the HBA Device Queue Depth Limit is Allows for extreme levels of performance VMworld 2017 Content: Not for publication 10

11 VMFS-6

12 VMFS-6: On-disk Format Changes File System Resource Management - File Block Format VMFS-6 has two new internal block sizes, small file block (SFB) and large file block (LFB) The SFB size is set to 1MB; the LFB size is set to 512MB These are internal concepts for files only; the VMFS block size is still 1MB Thin disks created on VMFS-6 are initially backed with SFBs Thick disks created on VMFS-6 are allocated LFBs as much as possible For the portion of the thick disk which does not fit into an LFB, SFBs are allocated These enhancements should result in much faster file creation times Especially true with swap file creation so long as the swap file can be created with all LFBs Swap files are always thickly provisioned VMFS-6 12

13 VMFS-6: On-disk Format Changes Dynamic System Resource Files System resource files (.fdc.sf,.pbc.sf,.sbc.sf,.jbc.sf) are now extended dynamically for VMFS-6 Previously these were static in size These may show a much smaller size initially, when compared to previous versions of VMFS, but they will grow over time If the filesystem exhausts any resources, the respective system resource file is extended to create additional resources VMworld 2017 VMFS-6 can now support millions of files / pointer blocks / sub blocks (as long as volume has free space) VMFS-6 Content: Not for publication 13

14 vmkfstools 500GB VMFS-5 Volume # vmkfstools P v10 /vmfs/devices/<device id> 14

15 vmkfstools 500GB VMFS-6 volume In VMFS-6, Sub Blocks for are used for Pointer Blocks. That's why Ptr Blocks max is shown as 0 here Large file blocks 15

16 VMFS-6: On-disk Format Changes File System Resource Management - Journaling VMFS is a distributed journaling filesystem Journals are used on VMFS when performing metadata updates on the filesystem Previous versions of VMFS used regular file blocks as journal resource blocks In VMFS-6, journal blocks tracked in a separate system resource file called.jbc.sf. Introduced to address VMFS journal related issues on previous versions of VMFS, due to the use of regular files blocks as journal blocks and vice-versa E.g. full file system, see VMware KB article VMFS-6 16

17 New Journal System File Resource VMFS-5 VMFS-6 17

18 VMFS-6: VM-based Block Allocation Affinity Resources for VMs (blocks, file descriptors, etc.) on earlier VMFS versions were allocated on a per host basis (host-based block allocation affinity) Host contention issues arose when a VM/VMDK was created on one host, and then vmotion was used to migrate the VM to another host If additional blocks were allocated to the VM/VMDK by the new host at the same time as the original host tried to allocate blocks for a different VM in the same resource group, the different hosts could contend for resource locks on the same resource This change introduces VM-based block allocation affinity, which will decrease resource lock contention VMFS-6 18

19 VMFS-6: Parallelism/Concurrency Improvements Some of the biggest delays on VMFS were in device scanning and filesystem probing vsphere 6.5 has new, highly parallel, device discovery and filesystem probing mechanisms Previous versions of VMFS only allowed one transaction at a time per host on a given filesystem; VMFS-6 supports multiple, concurrent transactions at a time per host These improvements are significant for fail-over event, and Site Recover Manager (SRM) should especially benefit Requirement to support higher limits on number of devices and paths in vsphere 6.5 VMFS-6 19

20 Hot Extend Support Prior to ESXi 6.5, VMDKs on a powered on VM could only be grown if size was less than 2TB If the size of a VMDK was 2TB or larger, or the expand operation caused it to exceed 2TB, the hot extend operation would fail This required administrators to typically shut down the virtual machine to expand it beyond 2TB The behavior has been changed in vsphere 6.5 and hot extend no longer has this limitation VMworld 2017 Content: Not for publication This is a vsphere 6.5 improvement, not specific to VMFS-6. This will also work on VMFS-5 volumes. 20

21 Upgrading to VMFS-6 No direct in-place upgrade of filesystem to VMFS-6 available. New datastores only. Customers upgrading to vsphere 6.5 release should continue to use VMFS-5 datastores (or older) until they can create new VMFS-6 datastores Use migration techniques such as Storage vmotion to move VMs from the old datastore to the new VMFS-6 datastore VMworld 2017 Content: Not for publication 21

22 VMFS-6 Performance Improvements with resignature (discovery and filesystem probing) 22

23 23

24 VAAI vsphere APIs for Array Integration VMworld 2017 Content: Not for publication

region that is updated is the time stamp, which tells others that this host is alive When the host is down,

25 ATS Miscompare Handling (1 of 3) The heartbeat region of VMFS is used for on-disk locking Every host that uses the VMFS volume has its own heartbeat region This region is updated by the host on every heartbeat The region that is updated is the time stamp, which tells others that this host is alive When the host is down, this region is used to communicate lock state to other hosts VMworld 2017 ATS Content: Not for publication 25

26 ATS Miscompare Handling (2 of 3) In vsphere 5.5 U2, we started using ATS for maintaining the heartbeat ATS is the Atomic Test and Set primitive which is one of the VAAI primitives Prior to this release, we only used ATS when the heartbeat state changed For example, we would use ATS in the following cases: Acquire a heartbeat Clear a heartbeat Replay a heartbeat Reclaim a heartbeat We did not use ATS for maintaining the liveness of a heartbeat VMworld 2017 This change for using ATS to maintain liveness of a heartbeat appears to have led to issues for certain storage arrays ATS Content: Not for publication 26

27 ATS Miscompare Handling (3 of 3) When an ATS Miscompare is received, all outstanding IO is aborted This led to additional stress and load being placed on the storage arrays In some cases, this led to the controllers crashing on the array In vsphere 6.5, there are new heuristics added so that when we get a miscompare event, we retry the read and verify that there is a miscompare If the miscompare is real, then we do the same as before, i.e. abort outstanding I/O If the on-disk HB data has not changed, then this is a false miscompare In the event of a false miscompare: VMFS will not immediately abort IOs VMFS will re-attempt ATS HB after a short interval (usually less than 100ms) ATS 27

28 An Introduction to UNMAP UNMAP via datastore VAAI UNMAP was introduced in vsphere 5.0 Enables ESXi host to inform the backing storage that files or VMs had be moved or deleted from a Thin Provisioned VMFS datastore Allows the backing storage to reclaim the freed blocks No way of doing this previously, resulting in stranded space on Thin Provisioned VMFS datastores UNMAP 28

29 Automated UNMAP in vsphere 6.5 Introducing Automated UNMAP Space Reclamation In vsphere 6.5, there is now an automated UNMAP crawler mechanism for reclaiming dead or stranded space on VMFS datastores Now UNMAP will run continuously in the background UNMAP granularity on the storage array The granularity of the reclaim is set to 1MB chunk Automatic UNMAP is not supported on arrays with UNMAP granularity greater than 1MB Auto UNMAP feature support is footnoted in the VMware Hardware Compatibility Guide (HCL) UNMAP 29

Some Considerations with Automated UNMAP Only is issued to VMFS datastores that are VMFS-6 and have powered-on VMs Can take 12-24 hours to fully reclaim Default behavior is

30 Some Considerations with Automated UNMAP Only is issued to VMFS datastores that are VMFS-6 and have powered-on VMs Can take hours to fully reclaim Default behavior is turned on, but can be turned off on the host (host won t participate) EnableVMFS6Unmap or on the datastore (no hosts will reclaim it) VMworld 2017 Content: Not for UNMAP publication 30

31 An Introduction to Guest OS UNMAP UNMAP via Guest OS In vsphere 6.0, additional improvements to UNMAP facilitate the reclaiming of stranded space from within a Guest OS Effectively, ability for a Guest OS in a thinly provisioned VM to tell the backing storage that blocks Backing storage to reclaim this capacity, and shrink the size of the VMDK VMworld 2017 UNMAP Content: Not for publication 31

32 Some Considerations with Automated UNMAP TRIM Handling UNMAP work at certain block boundaries on VMFS, whereas TRIM does not have such restrictions While this should be fine on VMFS-6, which is now 4K aligned, certain TRIMs converted into UNMAPs may fail due to block alignment issues on previous versions of VMFS Linux Guest OS SPC-4 support Initially in-guest UNMAP support to reclaim in-guest dead space natively was limited to Windows 2012 R2 Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs With SPC-4 support introduced in vsphere 6.5, Linux Guest OS es will now also be able to issue UNMAPs UNMAP 32

33 Automated UNMAP Limits and Considerations Guest OS filesystem alignment VMDK alignment is aligned on 1 MB block boundaries However un-alignment may still occur within the guest OS filesystem This may also prevent UNMAP from working correctly A best practice is to align guest OS partitions to the 1MB granularity boundary UNMAP 33

34 Known Automated UNMAP Issues vsphere 6.5 Tools in guest operating system might send unmap requests that are not aligned to the VMFS unmap granularity. Such requests are not passed to the storage array for space reclamation. Further info in KB article This issue is addressed in vsphere 6.5 P01 vsphere 6.5 P01 Certain versions of Windows Guest OS running in a VM may appear unresponsive if UNMAP is used. Further info in KB article This issue is addressed in vsphere 6.5 U1. UNMAP 34

35 UNMAP in action 35

36 36

37 SPBM Storage Policy Based Management VMworld 2017 Content: Not for publication

The Storage Policy Based Management (SPBM) Paradigm VMworld 2017 Content: Not for SPBM is the foundation of VMware's Software Defined Storage vision Common framework to allow

38 The Storage Policy Based Management (SPBM) Paradigm VMworld 2017 Content: Not for SPBM is the foundation of VMware's Software Defined Storage vision Common framework to allow storage and host related capabilities to be consumed via policies. publication Applies data services (e.g. protection, encryption, performance) on a per VM, or even per VMDK level 38

Creating Policies via Rules and Rule Sets Rule A Rule references a combination of a metadata tag and a related value, indicating the quality or quantity of the capability that is desired These two

39 Creating Policies via Rules and Rule Sets Rule A Rule references a combination of a metadata tag and a related value, indicating the quality or quantity of the capability that is desired These two items act as a key and a value that, when referenced together through a Rule, become a condition that must be met for compliance Rule Sets A Rule Set is comprised of one or more Rules A storage policy includes one or more Rule Sets that describe requirements for virtual machine storage resources Multiple Rule Sets can be leveraged to allow a single storage policy to define alternative selection parameters, even from several storage providers 39

40 VAIO vsan, VVOLs, VMFS 40

41 SPBM and Common Rules for Data Services provided by hosts - VM Encryption - Storage I/O Control v2 41

42 VMworld new features introduced with vsphere Encryption - Storage I/O Control v2 Content: Not for publication Implementation is done via I/O Filters 42

43 Storage I/O Control v2 VM Storage Policies in vsphere 6.5 has a new option called Common Rules These are used for configuring data services provided by hosts, such as Storage I/O Control and Encryption. It is the same mechanism used for VAIO/IO Filters or distribution 43

44 vsphere VM Encryption vsphere 6.5 introduces a new VM encryption mechanism It requires an external Key Management Server (KMS). Check the HCL for supported vendors This encryption mechanism is implemented in the hypervisor, making vsphere VM encryption agnostic to the Guest OS This not only encrypts the VMDK, but it also encrypts some of the VM Home directory contents, e.g. VMX file, metadata files, etc. Like SIOCv2, vsphere VM Encryption in vsphere 6.5 is policy driven 44

45 vsphere VM Encryption I/O Filter Common rules must be enabled add vsphere VM Encryption to a policy. Only setting in the custom encryption policy is to allow I/O filters before encryption. 45

46 VM Encryption and SIOC policy 46

47 47

48 NFS v4.1 Improvements

49 NFS v4.1 Improvements Hardware Acceleration/VAAI-NAS Improvements NFS 4.1 client in vsphere 6.5 supports hardware acceleration by offloading certain operations to the storage array. This comes in the form of a plugin to the ESXi host that is developed/provided by the storage array partner. Refer to your NAS storage array vendor for further information. Kerberos IPv6 Support NFS v4.1 Kerberos adds IPV6 support in vsphere 6.5. VMworld 2017 Content: Not for Kerberos AES Encryption Support NFS v4.1 Kerberos adds Advanced Encryption Standards (AES) encryption support in vsphere 6.5 NFSv4.1 publication 49

50 iscsi Improvements

51 iscsi Enhancements ISCSI Routing and Port Binding ESXi 6.5 now supports having the iscsi initiator and the iscsi target residing in different network subnets with port binding UEFI iscsi Boot VMware now supports UEFI (Unified Extensible Firmware Interface) iscsi Boot on Dell 13th generation servers with Intel x540 dual port Network Interface Card (NIC). VMworld 2017 iscsi Content: Not for publication 51

52 NVMe Support

NVMe (1 of 2) Virtual NVMe Device New virtual storage HBA for all flash SAN/vSAN storages New Operating Systems now leverage multiple queues with NVMe devices Virtual NVMe device allows VMs to take

53 NVMe (1 of 2) Virtual NVMe Device New virtual storage HBA for all flash SAN/vSAN storages New Operating Systems now leverage multiple queues with NVMe devices Virtual NVMe device allows VMs to take advantage of such in-guest IO stack improvements Improved performance compared to Virtual SATA device on local PCIe SSD devices Virtual NVMe device provides 30-50% lower CPU cost per I/O Virtual NVMe device achieve 30-80% higher IOPS 53

54 NVMe (2 of 2) Supported configuration information of virtual NVMe device. Number of Controllers per VM 4 Enumerated as nvme0,, nvme3. Number of namespaces per controller 15 Each namespace is mapped to a virtual disk. Enumerated as nvme0:0,, nvme0:15 Maximum queues and interrupts 16 1 admin + 15 I/O queues Maximum queue depth 256 4K in-flight commands per controller Supports NVMe Specification v1.0e mandatory admin and I/O commands Interoperability with all existing vsphere features, except SMP-FT 54

56 Cormac Hogan

vsphere 6.5 Storage December 15, 2017

vsphere 6.5 Storage December 15, 2017 December 15, 2017 1 Table of Contents 1. Core Storage Whitepaper 1.1.Storage Limit Improvements 1.2.Pluggable Storage Architecture (PSA) Improvements 1.3.VMFS-6 1.4.UNMAP 1.5.Storage I/O Control v2 1.6.vSphere