Hitachi Data Systems Hitachi NAS NFS with VMware vsphere Best Practices Guide

Transcription

1 Hitachi Data Systems MK-92HNAS028-03

2 Hitachi, Ltd. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or stored in a database or retrieval system for any purpose without the express written permission of Hitachi, Ltd. Hitachi, Ltd., reserves the right to make changes to this document at any time without notice and assumes no responsibility for its use. This document contains the most current information available at the time of publication. When new or revised information becomes available, this entire document will be updated and distributed to all registered users. Some of the features described in this document might not be currently available. Refer to the most recent product announcement for information about feature and product availability, or contact Hitachi Data Systems Corporation at Notice: Hitachi, Ltd., products and services can be ordered only under the terms and conditions of the applicable Hitachi Data Systems Corporation agreements. The use of Hitachi, Ltd., products is governed by the terms of your agreements with Hitachi Data Systems Corporation. March 2017 Hitachi Data Systems Page ii

3 Hitachi Data Systems products and services can be ordered only under the terms and conditions of Hitachi Data Systems applicable agreements. The use of Hitachi Data Systems products is governed by the terms of your agreements with Hitachi Data Systems. Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries. Hitachi Data Systems is a registered trademark and service mark of Hitachi, Ltd., in the United States and other countries. Archivas, Dynamic Provisioning, Essential NAS Platform, HiCommand, Hi-Track, ShadowImage, Tagmaserve, Tagmasoft, Tagmasolve, Tagmastore, TrueCopy, Universal Star Network, and Universal Storage Platform are registered trademarks of Hitachi Data Systems Corporation. AIX, AS/400, DB2, Domino, DS8000, Enterprise Storage Server, ESCON, FICON, FlashCopy, IBM, Lotus, OS/390, RS6000, S/390, System z9, System z10, Tivoli, VM/ESA, z/os, z9, zseries, z/vm, z/vse are registered trademarks and DS6000, MVS, and z10 are trademarks of International Business Machines Corporation. All other trademarks, service marks, and company names in this document or website are properties of their respective owners. Microsoft product screen shots are reprinted with permission from Microsoft Corporation. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit ( Some parts of ADC use open source code from Network Appliance, Inc. and Traakan, Inc. Part of the software embedded in this product is gsoap software. Portions created by gsoap are copyright Robert A. Van Engelen, Genivia Inc. All rights reserved. The software in this product was in part provided by Genivia Inc. and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the author be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this software, even if advised of the possibility of such damage. The product described in this guide may be protected by one or more U.S. patents, foreign patents, or pending applications. Notice of Export Controls Export of technical data contained in this document may require an export license from the United States government and/or the government of Japan. Contact the Hitachi Data Systems Legal Department for any export compliance questions. March 2017 Hitachi Data Systems Page iii

4 Document revision level Revision Date Description MK-92HNAS March 2013 First Publication MK-92HNAS December 2013 Revision 1, replaces and supersedes MK-92HNAS MK-92HNAS August 2015 Revision 2, replaces and supersedes MK-92HNAS028-01, previously titled as Hitachi NAS Platform Best Practices Guide for NFS with VMware vsphere. MK-92HNAS March 2017 Revision 3, replaces and supersedes MK-92HNAS Contact Hitachi Data Systems 2845 Lafayette Street Santa Clara, California North America: References Hitachi NAS Platform File Services System Administration Guide (available from the NAS Manager GUI help) Best Practices for Running VMware vsphere on Network-Attached Storage (NAS) Best Practices for Running VMware vsphere on Network Attached Storage by VMware VMware Knowledge Base Article , Configuring Flow Control on ESX and ESXi VMware Knowledge Base Article , Definition of the advanced NFS options March 2017 Hitachi Data Systems Page iv

5 Table of Contents Introduction... 1 Overview... 1 VMware vsphere on NFS... 1 NFS and virtual machines introduction... 2 HDS storage system terminology... 2 System Drives... 2 Storage pools... 2 File systems... 2 NAS Virtual Volumes... 2 Enterprise Virtual Server... 2 Storage configuration from raid group to file system... 3 Storage configuration recommendations... 4 Tiered file system configuration example for NAS and VMware with HDT/HRT Flash/SAS pool... 5 Tiered file system configuration example for NAS and VMware with HDP pool... 6 General NAS and VMware recommendations... 7 NAS release... 7 NAS file system configuration recommendations... 7 NFS exports... 8 File system size... 8 File system utilization... 8 NAS thin provisioning... 8 Storage pools... 8 NAS Virtual Volumes (ViVols)... 9 OS alignment for boot and data disks... 9 NAS Tiered File System... 9 NAS deduplication ESXi v5.x recommended settings Networking recommendations Jumbo frames Minimizing latency Link aggregation Miscellaneous network features Storage provisioning basics within VMware NFS volume provisioning VMware ESXi NFS configuration Creating a NAS datastore NAS NFS with VMware vsphere Best Practices Guide March 2017 Hitachi Data Systems

6 Dynamically growing NFS file systems Network connectivity basics in VMware Enabling network connectivity and configuring the VMkernel Hitachi Virtual Infrastructure Integrator for VM backup and recovery High availability and replication VMware vsphere Metro Storage Cluster VMware Site Recovery Manager (SRM) and NAS VMware API for Array Integration for NAS support Full File Clone Fast File Clone Reserve Space Extended Stats Migrating Virtual Machines to NAS datastores using Storage vmotion Configuring NAS autoexpansion with vcenter Creating the basics Enabling file system thin provisioning Creating a VM thin provision Setting the file system quota mode Creating quotas Soft limit quotas Hard limit quotas Setting the file system quota mode Additional NAS and VMware best practices considerations NAS file system Allocation on Demand versus Allocate Now Allocation on Demand Allocate Now Allocation type considerations EVS failover timeout on guest operating systems EVS failover timeout for Linux EVS failover timeout for Windows VMware VMDK thin provisioning Summary NAS NFS with VMware vsphere Best Practices Guide March 2017 Hitachi Data Systems

7 Introduction This document covers VMware best practices specific to Hitachi NAS storage and is supplemental to the VMware NFS Best Practices document published by VMware, in collaboration with Hitachi Data Systems. See the HDS Support or HDS Community website for the latest version of this best practices guide. Overview HDS has taken an evolutionary step in network storage by architecting a hardware based file system and operating system into a hardware-based appliance that is capable of serving data at high throughput and IOPS rates. This approach, combined with an advanced feature set and storage virtualization, makes Hitachi Network-Attached Storage file controllers an ideal storage solution for VMware environments, serving vsphere and vcloud workloads. Although the hardware architecture is unique, the NAS 4000 series and VSP (F)G series platforms provide standards-based SMB/CIFS, NFS, iscsi, FTP, and NDMP protocols for file access and data services. The platforms advanced feature set includes file clone and file system snapshots, replication, enterprise virtual servers and file systems, thin provisioning, dynamic volume expansion, transparent data migration, and a global namespace. These features are enabled on a robust Hitachi SAN platform which provides the back end storage plus additional rich data services such as mirroring, flash acceleration and global active volumes between datacenters. In addition, rich management software such as Hitachi Virtual Infrastructure Integrator and plugins/connectors such as Hitachi Storage Connecter for vrealize Orchestrator provide the necessary native integration for VMware admins. This range of capabilities within the platform solution provides the feature flexibility to meet the various requirements of VMware based environments. VMware vsphere on NFS Using VMware vsphere with the NFS protocol is a fully supported well adopted storage option. You can use all the vsphere-supported features and related VMware products, including: vmotion Storage vmotion VMware Consolidated Backup (VCB) Storage I/O Control Support VAAI for NAS Support VVOL / VASA Support Site Recovery Manager (SRM) VMware Distributed Resource Scheduling (DRS) Fault tolerance Clustered datastores High Availability March 2017 Hitachi Data Systems Page 1

8 The NFS protocol is a mature, stable protocol, and the NAS storage platform provides excellent NFS performance through a unique, hardware-accelerated platform. NFS and virtual machines introduction Virtual machines (VMs) are stored on NFS datastores and their disk drives exist as files that are referred to as virtual machine disks (VMDKs). Each VM becomes a folder within a datastore. The NFS datastore is mapped to an NFS export/folder and NFS datastores appear in the vsphere client similar to the way other VMFS datastores. Guest Operating Systems of all types (Windows, Linux etc ) can be hosted on NFS datastores. Guest Operating Systems make no distinction whether they are running on a VMFS or NFS datastore. HDS storage system terminology Some specific terms related to the HDS storage solution are briefly defined here. If you are already familiar with the configuration of HDS systems, you can skip this section. System Drives The NAS system uses RAID technology as a foundation for data protection. The backend storage subsystem is configured using an appropriate data protection scheme, and the resulting volumes are presented as logical units for the NAS system to use. These logical units are referred to as System Drives (SDs) and are combined to create storage pools. Storage pools The NAS system combines SDs to create a virtualized pool of storage (which is also known as a span in the system CLI). Storage pools contain one or more NAS file systems that you can share through the NFS protocol. You can dynamically extend the storage pool by adding additional SDs at any time. File systems File systems are provisioned within storage pools, and they can grow independently of one another, according to guidelines set by system administrators. File systems contain files, directories, and Virtual Volumes. NAS Virtual Volumes NAS Virtual Volumes (ViVols) are directories in a file system. You can apply a quota to each Virtual Volume. NAS Virtual Volumes may also be used as self-contained units for file replication or internal data migration. NAS Virtual Volumes simplify tracking of space used by a directory hierarchy. When quotas are set, they can be used to display only the amount of space you want the vsphere software to recognize. NAS Virtual Volumes and quotas provide tremendous provisioning flexibility, as well as simple, zero-impact space accounting. Enterprise Virtual Server An Enterprise Virtual Server (EVS) encapsulates one or more IP addresses, one or more file systems, and one or more shares. Currently, each NAS system or cluster can present up to 64 virtual servers. Virtual servers can migrate between physical cluster nodes, similar to how VMware VMs migrate between ESXi hosts. An EVS appears on the network as a standard NFS file server. March 2017 Hitachi Data Systems Page 2

9 Storage configuration from raid group to file system The following illustration demonstrates the storage configuration for the Hitachi NAS platforms based on the recommended Tiered (Flash) File system (TFS) configuration. This next illustration shows non-tfs configuration and the relationship between storage components. March 2017 Hitachi Data Systems Page 3

10 Storage configuration recommendations There are several different options for storage configuration for NAS/VMware, including HDP, HDT/HRT and TFS based implementations and combinations of these technologies. In some cases, a site has different file system and backend pool configurations in order to match SLAs of groups of VMs to the appropriate file system and associated NFS datastores. This section contains some general recommendations and also two recipe configurations as follows: Tiered file system configuration example for NAS and VMware with HDT/HRT Flash/SAS pool Tiered file system configuration example for NAS and VMware with HDP pool The following general storage configuration is recommended for VMware vsphere environments. For optimized storage configuration used extensively as part of large scale testing initiatives, use the TFS-based configuration shown here. For recommended RAID levels: HDP Pools Workload\Storage Array VSP Gx00 and Fx00 Random with heavy Read RAID6 6+2 Sequential with heavy Read RAID6 6+2 Heavy Write RAID6 6+2 or RAID10 2+2, 4+4 Thin provisioning for HDP Pools was introduced in NAS release Since this release, thin provisioned DP-VOLs are recommended for the following reasons: It is faster and easier to expand NAS file systems and pools. Data rebalancing is automatically performed by the dynamic provisioning pool more efficiently. Prior to the support of thin provisioned SDs, new SDs had to be presented by the NAS platform and the storage pool had to be expanded by a stripeset. The file system then had to be rebalanced. When using HDP or HDT/HRT, follow these guidelines: Dedicate an HDP or HDT/HRT pool to each NAS storage pool. Use Hitachi Dynamic Provisioning (HDP) except when using HDT/HRT. March 2017 Hitachi Data Systems Page 4

11 Tiered file system configuration example for NAS and VMware with HDT/HRT Flash/SAS pool This example demonstrates a general server workload for a Tiered File System (TFS) configuration for NAS and VMware with HDT/HRT FMD Flash and SAS pool. In the example, Hitachi Storage was configured with FMD flash drives and 10K SAS drives grouped and presented as follows: 26 RAID-6 (6D+2P) 10K parity groups (PGs). 1 RAID-5 (7D+1P) FMD Parity Group (recommended configuration is RAID-5 for FMDs). 1 HDT/HRT pool was created from the 26 RAID-6 (6D+2P) 10K PGs and 1 RAID-6 (7D+1P) FMD PGs TB thin provisioned DP-Vols were created from the single HDT/HRT Pool. The Tiering Policy was set to Level 1, so blocks written to these DP-Vols were always written to Tier 1, which is the FMDs TB thin provisioned DP-Vols were created from the single HDT/HRT Pool. The Tiering Policy was set to All, so blocks written to these DP-Vols were dynamically placed. March 2017 Hitachi Data Systems Page 5

12 The following configuration was used for NAS: 32 system drives (SDs) were created from the 32 8 TB DP-Vols that were provisioned from the 10K SAS drives. 16 SDs were created from the 16 HDP Vols from the FMDs. One Storage Pool was created from the 48 system drives. Two file systems with tiered file system configuration (TFS) were created from the storage pool. 2 NFS exports were created from each file system as mount points for the ESXi datastore With this configuration, both tiered file system (TFS) tier 0 metadata as well as VMware VMDK data are being served from flash drives. Tiered file system configuration example for NAS and VMware with HDP pool This example demonstrates a general server workload for a Tiered File System (TFS) configuration for NAS and VMware with HDP pool configuration. In the example, Hitachi Storage was configured with 10K SAS drives grouped and presented as: Thirteen RAID-6 (6D+2P) parity groups (PG) One dynamic provisioning pool was created from the thirteen parity groups Sixteen 8TB thin provisioned DP-VOLs were created from the single dynamic provisioning pool Hitachi Storage was also configured with FMDs grouped and presented as: One RAID-5 (3D+1P) parity group (PG) One dynamic provisioning pool was created from the single parity group Eight 4TB thin provisioned DP-VOLs were created from the single dynamic provisioning pool March 2017 Hitachi Data Systems Page 6

13 The following configuration was used for Hitachi NAS Platform: Sixteen system drives (SD) were created from the sixteen DP VOLs that were provisioned from the 10K SAS drives Eight system drives (SD) were created from the eight DP VOLs from the FMDs One storage pool for a Tiered File System (TFS) configuration was created from the 24 system drives as listed here. This storage pool creation pattern takes future file system expansion into consideration. The file system should be expanded with eight SDs at a time. Stripeset 0 (Tier 0 - metadata) 8 FMD 4TB SDs Stripeset 1 (Tier 1) 8 10K SAS 8TB SDs Stripeset 2 (Tier 1) 8 10K SAS 8TB SDs Note: The use of two stripesets can easily be consolidated into one stripeset of 16 SD's. Current guidelines suggest this is a more optimal solution. One tiered file system was created from the storage pool. Four NFS Exports were created as mount points for the ESXi datastore. General NAS and VMware recommendations NAS release NAS 12.7 or higher is recommended for VMware environments. Features such as bitmap readahead, sparse metadata and checkpoint block freeing help to enhance VMware performance. NAS file system configuration recommendations In general, a 4KB file system block size is recommended and considered the default. Disable the quick start option for NAS read ahead when VM I/O profiles are primarily random on a NAS file system. If the I/O profile is sequential, leave the default read options set to enabled. To do this, where <file_system> is the name of the file system: o o Random: read-ahead file-system <file_system> --quick-start disable Sequential: read-ahead file-system <file_system> --default Turn off access time on each file system (and new file systems) if used exclusively to host NFS datastores to improve I/O performance. To turn off this feature, use the command: fs-accessed-time --file-system <file_system> off Note: Use the Virtual Infrastructure Integrator s Hitachi NAS NFS Best Practice Compliance Dashboard to check all settings for VMware environments, and to make any recommended configuration changes. March 2017 Hitachi Data Systems Page 7

14 NFS exports Do not export the root of file system. Instead, create a directory to be exported. Use Access Configuration set as follows: rw,norootsquash for all the vsphere servers that need to access them (VMKernel IP addresses) * (noaccess) for everything else File system size File system size affects backup, restore and file system checking. Larger file systems typically take longer to backup and recover. The NAS file system employs journaling, maintains redundant copies of key structures and can generally recover and bring itself to a consistent state. In rare situations, like any other file system, it may need to be checked. File system utilization Maintain at least 15% free space in each file system used by ESXi hosts. It is recommended to set file system usage warning at 80%. NAS thin provisioning Recommended provisioning is to use file-system-thin if a file system is set to auto-expand. There are situations, especially with small chunk sizes, where a VM cannot be created or expanded even though there is space available. In this case, the VMware server is not aware of this and will report an error. Currently, this is a CLI-only option: filesystem-thin <filesystem-instance-name> [off on] Storage pools Do not mix disk types in the same storage pool. Limit ownership of all file systems that are created on a storage pool to one EVS. Configure a minimum of four System Drives (SD) in a storage pool. Configure one LU\LDEV for each RAID group, consuming all space (if possible). March 2017 Hitachi Data Systems Page 8

15 NAS Virtual Volumes (ViVols) Note: NAS Virtual Volumes are different from VMware VVols. Using NAS Virtual Volumes (ViVols) is optional NAS ViVols provide directory level quotas that allow for increasing and decreasing the visible capacity of data store concurrently without having to make changes to the underlying file system. The same considerations for number of file systems apply to number of Virtual Volumes. NAS snapshots apply to the whole file system, not individual Virtual Volumes. NAS object replication operates at the file system level. Take this into consideration when using the Hitachi Virtual Infrastructure Integrator vaulting feature. Using Virtual Volumes does not change other best practices such as file system utilization. OS alignment for boot and data disks Virtual Infrastructure Integrator s Compliance check verifies that all NAS hosted Virtual Disks are properly aligned. If not, Virtual Infrastructure Integrator identifies the specific Virtual disk that is not properly aligned. For this feature to work correctly, use NAS OS v12.4 or higher with Virtual Infrastructure Integrator v3.1 or higher. Windows partitions on 2008 or newer should align automatically. Older versions will require realignment. Partitions on older Linux versions and other operating systems may require alignment. Alignment should be in multiples of the NAS file system block size (4KB) and should be done prior to OS installation for boot disks. When migrating existing VM s onto NAS datastores, the alignment may be off. Realigning VMs is possible. There are third-party tools available to assist such as UberAlign. Note: HDS does not warrant, endorse or recommend the use of UberAlign. This utility is mentioned only to illustrate an example. NAS Tiered File System The NAS Tiered File System (TFS) feature enhances extensive metadata access especially in large VM environments. FMDs or Solid-State Drives (SSDs) are strongly recommended for Tier 0. Tier 0 can also use SAS drives. However, when using SAS, RAID10 is required. On average, Tier 0 should represent minimum 5% of the capacity of the file system. March 2017 Hitachi Data Systems Page 9

16 When considering the number of file systems to create: In general, less is more. However, this must be balanced with SLAs, risk, flexibility and other business requirements. Consider your risk profile. For example, a single very large file system with all of your capacity puts more data at risk and can lengthen recovery operations. Conversely, too many small file systems can strand capacity and increase management effort. For instance, a system with 2PB of capacity may use sixteen 128TB file systems. The file systems are large but the amount of data at risk when compared to the overall capacity is relatively small. A system with 100TB capacity generally should not configure a single file system of the same size. Replication requirements: A file system may host various applications with different replication RPO/RTO requirements. It may be better to have separate file systems in these situations. Snapshot requirements: Similar to replication requirements, different applications may have different needs that are better served with separate file systems. For example, one application may require long retention (which will use more file system space) while others may not. So it would make sense to separate these if capacity utilization is a concern. Deduplication occurs within a file system. To maximize dedupe results, it is best to have data expected to be similar hosted on the same file system. NAS deduplication The NAS system supports primary deduplication of data on the NAS file systems. VM type files, such as VMware VMDK files, are ideal for deduplication. As discussed previously, the NAS file system supports VMDK thin provisioning. NAS does not deduplicate the sparse portion of the VMDK because it is not using any space. For example: Assume a file system containing 100 thin VMDKs of 10GB, each deployed from the same template. o o o Total space utilization reported by the NAS system (by default): 1000GB Each VMDK contains 1GB of actual data, while the rest is sparse Each VMDK contains identical data After deduplication, the NAS system would report the following: o o Deduplicated: 99GB Total utilization: 901GB With sparse file support enabled, the NAS system would report the following total space utilization: o o Before deduplication: 10GB After deduplication: 1GB March 2017 Hitachi Data Systems Page 10

17 ESXi v5.x recommended settings Verify and if necessary change the NFS MaxVolumes and TcpipHeapsize ESXi host settings according to the VMware Knowledgebase article available here. For VMware ESXi 5.x, verify and if necessary change the following ESXi host settings as follows: NFS.HeartbeatFrequency = 12 NFS.DiskFileLockUpdateFreq = 10 NFS.LockUpdateTimeout = 5 Note: For ESXi v6.0 and later, the default settings are acceptable and no changes are required. Networking recommendations The following network configurations are recommended for VMware vsphere environments: 10Gb Ethernet (10Gbe) 10Gbe dedicated connections for the Ethernet storage network between NAS and hypervisor and Layer 2 switches Session Sharing / Multipathing with IP Addresses and NAS Enterprise Virtual Server (EVS) Different network for management and data To achieve an equivalent to multipathing in VMware/ NFS environments (that is, effective load sharing and multiple sessions across multiple available physical connections/adapters): When NAS is deployed in a clustered scenario, dedicated EVSs should be deployed on each node, and multiple IP addresses should be assigned to each EVS. NAS NFS datastores should be provisioned in vcenter in a round-robin fashion against the EVSs multiple IP addresses to ensure parallelism on the NAS and increased IP connectivity from the hypervisor. This results in higher overall throughput on the storage network. IP hashing configured on the ESXi NFS network adapters to spread the IP load across the multiple 10Gbe links (IP hashing is used within the VMware vswitch for active/active interface utilization on the hypervisor). This requires link aggregation or port-channeling (static) to be configured at the network switch. In addition to IP hashing, a separate option is Load-Based Teaming (LBT). See the VMware NFS paper in the References section for more details. The LBT option can be deployed only by using distributed switches in VMware. March 2017 Hitachi Data Systems Page 11

18 Jumbo frames Jumbo frames are now considered optional. If enabled, ensure each infrastructure device throughout the storage network (end-to-end) is configured to pass 9000 MTU jumbo frames. On the NAS server, log in to the console and perform the following actions: To update the configuration to set the MTU for TCP and non-tcp packets to 9000, use the command ipadv -m n Use the ifconfig command to verify the setting has been applied: NAS:$ ifconfig ag1 Link encap:1 HWaddr f8-a8 Link:UP Admin:UP MTU:9000 Metric:1 txqueuelen:64 To check that jumbo frames are being sent and received, and to see the traffic flow in the CLI, use the command: nim nim-mib grep JUMBO March 2017 Hitachi Data Systems Page 12

19 Minimizing latency For vsphere 5.0, all Ethernet storage network components should reside on the same subnet in HDS NAS configurations (hypervisor VMkernels, NAS EVS etc ). Link aggregation NAS interfaces are configured as vlags to the upstream switches (for example, Brocade VDX or Cisco Nexus) to ensure connection load-balancing as well as availability. Do not use the NAS round robin port level load-balancing. Instead, use the normal default setting. Miscellaneous network features Flow control is recommended when vsphere servers are using 1Gbe to connect to NAS 10Gbe datastores. Note: Flow control needs to be enabled end-to-end. Spanning tree is unnecessary when utilizing VCS fabric with Brocade VDX switches. Storage provisioning basics within VMware VMware vsphere supports NFS, FC and iscsi protocol based datastores. These storage options give you flexibility in setting up the disk requirements based on your business needs, cost, performance etc NFS volume provisioning There are various different methods of creating NFS exports on NAS servers: using the NAS Manager, the NAS CLI, Hitachi Command Suite or Hitachi Storage Advisor. Note: This document does not cover file system creation. See the Hitachi NAS Platform File Services System Administration Guide for information about file system and export creation. March 2017 Hitachi Data Systems Page 13

20 VMware ESXi NFS configuration ESXi host supports the NFSv3 protocol in order to enable communication between an NFS client and NFS server. The client issues requests for information from the server and the server replies with the result. For NAS, NFSv3 is required. The NFS client that is built into the ESXi host enables you to access the NFS server and use NFS volumes to store virtual machine disks (VMDKs). The following illustration shows vsphere configured with NFS datastores. Creating a NAS datastore To create NFS storage: 1. In the VMware Virtual Infrastructure client, choose ESXi. 2. Select Configuration > Storage (SCSI, SAN, and NFS) > Add Storage. 3. In the Storage Type window, select Network File System, and then click Next. March 2017 Hitachi Data Systems Page 14

21 4. In the Locate Network File System window, enter the NAS server name, folder, and datastore name, and then click Next. 5. The following illustrations show the NFS volume setup. March 2017 Hitachi Data Systems Page 15

22 Important: Ensure that you mount datastores with the same volume label on all vsphere ESXi hosts within VMware high availability (HA) environments. Dynamically growing NFS file systems To increase the storage on the NAS server in a VMware vsphere environment, you can expand the NFS file system on the NAS system. To expand an NFS file system: 1. In the NAS Manager, navigate to Home > Storage Management > File System. 2. Select a file system and click Details to display the File System Details window. 3. Click Expand to display the Expand File System window. 4. In the Expand File System window, enter the new size of file system, and then click OK. 5. Refresh the storage on each ESXi host. 6. From the ESXi host, navigate to Hardware > Configuration. 7. Select Storage and then click Refresh to display the new size of NFS file. The expanded storage is ready for use. March 2017 Hitachi Data Systems Page 16

23 Network connectivity basics in VMware Enabling network connectivity and configuring the VMkernel The IP protocol storage uses the TCP/IP stack as its foundation for communication. The stack includes iscsi and NFS (NAS) for ESXi hosts. A VMkernel uses the TCP/IP protocol stack to handle the transport of data. To create a VMkernel: 1. In the VMware Virtual Infrastructure client, select an ESXi host. 2. Select the Configuration tab, and then click the Networking command link to add networking and click Next to continue. 3. To choose a VMkernel, select one of the physical network cards and click Next. 4. Enter VMkernel in the Network Label text box and click Next. 5. Enter the IP address and the subnet mask. 6. To provide the VMkernel default gateway, click Edit, and enter the gateway address. 7. Click OK to continue to the confirmation display. Click Next and then Finish. Note: The VMkernel network should be set up in a private network or with a unique VLAN ID that provides network isolation. March 2017 Hitachi Data Systems Page 17

24 Hitachi Virtual Infrastructure Integrator for VM backup and recovery Hitachi Virtual Infrastructure Integrator provides a management console plug-in and associated software for VMware vcenter, accessed through the vsphere Web client. With the Virtual Infrastructure Integrator, the management of virtual machine (VM) backup, restore, cloning for NFS and VMFS datastores is greatly simplified. The Virtual Infrastructure Integrator provides the following key features: Efficient, scalable, consistent, and managed VM backup on NFS and VMFS datastores Fast, space efficient VM clones in 95% less time and space Visibility to Hitachi Data Systems NFS/VMFS storage services serving VMware for effective management Logically and reliably protects hundreds of VMs Assured implementation enablement that compares the existing Hitachi Data Systems- VMware configuration against best practices Specific to backup capabilities, the Virtual Infrastructure Integrator: Uses Hitachi Data Systems storage based file clone technology and HDS HTI technology to deliver VM level snapshots Applies schedule and retention policies Application-consistent backups for assured recovery Provides direct access to scalable VM backup and recovery process to VM administration Augments traditional backup and provides better VM RTO/RPO. Provides enhanced data protection through vaulting March 2017 Hitachi Data Systems Page 18

25 For more information, see the Hitachi Virtual Infrastructure Integrator software datasheet. High availability and replication High availability solutions and replication technology are important to every organization. NAS offers both configurations within the datacenter, campus, metro and geographic datacenters. It provides replication solutions to support business continuity, data protection, and disaster recovery. VMware vsphere Metro Storage Cluster NAS platforms are certified to support VMware vsphere Metro Storage Cluster configurations for high availability active-active datacenters. The technology is enabled with either Hitachi Global Active Device configuration or Synchronous Disaster Recovery (SyncDR) Cluster solution depending on release level. March 2017 Hitachi Data Systems Page 19

26 VMware Site Recovery Manager (SRM) and NAS NAS offers replication at both the file and object level to address data protection and business continuity. The Virtual Infrastructure Integrator offers vaulting using NAS Object Replication. NAS integrates with VMware vcenter Site Recovery Manager (SRM) by providing a Storage Replication Adapter (SRA). The NAS SRA for VMware Site Recovery Manager is available for vsphere 5.x and 6.x environments. For more information, contact Hitachi Data Systems or VMware support as appropriate. VMware API for Array Integration for NAS support VMware API for Array Integration (VAAI) for NAS is an adapter plugin that storage vendors create to provide storage offload services for VMware vsphere/vcenter environments using NFS datastores. This is similar to VAAI for FC block storage. However, VAAI-NAS supports additional primitives. Hitachi Data System s VAAI for NAS adapter takes advantage of capabilities inherent in the HDS Enterprise NFS/NAS architecture such as hardware file system, VM level hardware snapshot, and clone operations with File Clone technology. The VAAI-NAS adapter is used by VMware vcenter for provisioning and operational tasks such as Clone VM and Deploy VM from Template operations to dramatically speed up those important operations by offloading to the NAS platform. VAAI-NAS is also used by VMware Horizon View to offload and speed up desktop provisioning and recompose operations. Finally, vcloud Director takes advantage of VAAI-NAS, similar to vcenter, to also dramatically speed up provisioning and VM deployments such as deploying VM/vApp instances from templates. Operations mentioned in this section now take seconds, compared to minutes without VAAI-NAS. The NAS supports calls from a VAAI for NAS adapter as follows: Full File Clone Fast File Clone Reserve Space Extended Stats The VAAI plug-in is available for download from either VMware or HDS Support. The VAAI for NAS plug-in must be installed on each ESXi host that you require to use the VAAI NAS primitives. You can use VMware vsphere update manager to update multiple hosts automatically. Note: Thin Provisioned VMDK creation is recommended when using VAAI for NAS for maximum performance and space efficiency benefits. March 2017 Hitachi Data Systems Page 20

27 Full File Clone The Full File Clone API enables the ESXi host to offload a cold clone operation or template deployments to the storage array. Full File Clone is used when the source and target are on different datastores. One of the important benefits of using Full File Clone on NAS is that NAS will recognize sparseness (blocks allocated but not used) in VMDKs, which reduces the time to create Full File Clones. An important point to note is that this primitive does not support Storage vmotion. This is true for any NAS provider s VAAI for NAS based on vsphere 5.1. Storage vmotion on NFS datastores continue to use the VMkernel software data mover as this is quite efficient. The primitive can be used only when the virtual machine is powered off. Fast File Clone With the Fast File Clone API, it is possible to offload provisioning and cloning of VMs using linked clone technology within the NAS platform. That is, when Fast File Clone API is called by VMware vcenter or Horizon View or vcloud Director, this feature takes advantage of NAS File Clone to provide fast, space-efficient clones of a source VM or source template. Again, operations now take seconds compared to minutes without VAAI-NAS. When a VMware admin does provisioning tasks within VMware vcenter such as Clone VM or Deploy VM from Template operations, it will use VAAI-HNAS to dramatically speed up those important operations by offloading them to the NAS. March 2017 Hitachi Data Systems Page 21

28 Specific to VMware Horizon View 5.2 and unique HDS support for VCAI (View Composer Array Integration) primitive support, desktop provisioning also calls VAAI-HNAS to offload operation to NAS. Finally, with the release of vsphere 5.1 and with VMware vcloud Director 5.1, this primitive is fully supported for VMware vcloud vapps when VAAI-NAS is enabled on the datastore and Fast Provisioning Using Linked Clones is selected to speed up provisioning operations. Reserve Space Reserve Space is another VAAI for NAS primitive. Without VAAI for NAS, you cannot pre-allocate or zero out space for Virtual Machine Disk formats (VMDKs) on NFS without using the CLI. Historically, the only option available was to build thin VMDKs on NFS or manually create zeroed VMDKs using the CLI. With the introduction of Reserve Space, you can create thick VMDKs on NFS datastores. However, VAAI for NAS Reserve Space does not behave like VAAI for block. It does not direct the array to do the zeroing on its behalf. When creating a VMDK on a VAAI for NAS array, selecting Flat sends a Space Reserve NAS VAAI command to the array guaranteeing that the space will be available. This is equivalent to VMware vsphere VMFS (Virtual Machine File System) lazyzeroedthick, and the blocks are zeroed on first write. However, selecting Flat pre-initialized also sends a Space Reserve NAS VAAI command, but it does ESXi-based zero writing to the VMDK. This is equivalent to a VMFS Thick Provision Eager Zeroed (eagerzeroedthick). Note: VAAI NAS Reserve Space enables you to create virtual disks in lazyzeroedthick or eagerzeroedthick formats on NFS datastores on arrays that support Reserve Space. However, when you check the disk type on the Virtual Machine Properties dialog box, the Disk Provisioning section always shows eagerzeroedthick as the disk format no matter which format you selected during the disk creation. ESXi does not distinguish between lazy zeroed and eager zeroed virtual disks on NFS datastores. March 2017 Hitachi Data Systems Page 22

29 For NAS, Thin Provisioned VMDK or ThickLazy VMDK creation is recommended. With NAS hardware-accelerated file system and primary deduplication and the HDS allocation mechanism, these provide the optimum provisioning performance, security, and space efficiency and mitigate the need to use ThickEager VMDKs where previously required. ThickEager VMDKs can continue to be used if necessary. ThickLazy VMDKs will report as ThinProvisioned VMDKs in vcenter since due to the space efficiency features in the HDS VAAI implementation. (Competitors might report ThickEager VMDKs for ThickLazy VMDKs.) Extended Stats Extended Stats enables you to query how much space a VMDK actually consumed on an NFS datastore. vmkfstools extendedstat <vmdk file> For example, you might create a 100GB thin VMDK, but actually consume only 25GB of space on the array. This was an issue vsphere previously did not address. Moreover, this was not a necessary feature for VMFS because vsphere understands VMFS, but was needed for NFS. Thin provisioning primitives were introduced with vsphere 5.0, including features such as raising an alarm when a thin provisioning volume reached 75% of capacity at the back end, thin provisioningstun and, the UNMAP primitive. However these thin provisioning primitives are for SCSI only. The VAAI space-threshold alarm is supported only on SCSI datastores. Similarly, the VAAI thin provisioning-stun was introduced to detect out-of-space conditions on SCSI LUNs. However, for NAS datastores, NFS servers can already return an out-of-space error that should be propagated up the stack. This should induce a virtual machine stun similar to VAAI thin provisioning. This operation does not need the VAAI NAS plug-in, and should work on all NFS datastores, whether or not the hosts have VAAI enabled. The vsphere Storage DRS also uses this event. After the alarm is triggered, the vsphere Storage DRS no longer considers those datastores as destinations for initial placement or ongoing loadbalancing of virtual machines. Finally, the UNMAP primitive is also for SCSI. The reclaiming of dead space is not an issue on NAS arrays. March 2017 Hitachi Data Systems Page 23

30 The next illustration shows how to verify that an NAS NFS datastore is configured for hardware acceleration. The hardware acceleration is provided by the VAAI adapter for NAS. It also shows the storage capabilities that are recognized for that particular datastore. Migrating Virtual Machines to NAS datastores using Storage vmotion When using NAS NFSv3 storage as VMware datastores, there are several options to select from for virtual disk format. These options are only available when the NAS VAAI plugin is installed on the ESXi Server. The best practice for NAS is to use Thin Provisioned VMDKs instead of Thick Eager VMDKs. By using this option, the capacity utilization after migration to NAS is reduced. During the Storage vmotion workflow, ensure Thin Provision is selected as the virtual disk format. To confirm that the VM was migrated using Thin Provisioned format, use the stat command and/or vmkfstools from an ESXi host for the VMDK(s). From the vsphere CLI, navigate to the /vmfs/volume/ directory and run either command. March 2017 Hitachi Data Systems Page 24

31 Examples: VMDK after Storage vmotion with Thin Provisioning: /vmfs/volumes/46a2299b-2f7b7351/vm-eagerzeroblock # stat vm-eagerzeroblockflat.vmdk File: vm-eagerzeroblock-flat.vmdk Size: Blocks: IO Block: regular file /vmfs/volumes/46a2299b-2f7b7351/vm-eagerzeroblock # vmkfstools --extendedstat vm-eagerzeroblock.vmdk Capacity bytes: Used bytes: , Unshared bytes: VMDK after Storage vmotion with Thick Eager Zero: vmfs/volumes/d da139ba1/vm-eagerzerothick2 # stat vm-eagerzerothick2- flat.vmdk File: vm-eagerzerothick2-flat.vmdk Size: Blocks: IO Block: regular file /vmfs/volumes/d da139ba1/vm-eagerzerothick2 # vmkfstools --extendedstat vm-eagerzerothick2.vmdk Capacity bytes: Used bytes: , Unshared bytes: With Thin Provisioning, in the example above, the system consumes blocks. Displaying a used space of 21GB compared to the original 34GB. With Thick Eager, blocks will be consumed and the system will report about 34GB used space. March 2017 Hitachi Data Systems Page 25

32 Configuring NAS autoexpansion with vcenter Configuring autoexpansion on the NAS server increases the space capacity in VMware for file systems. There are different options for managing capacity, including setting limits on Virtual Volume quotas and setting the file system quota mode. Creating the basics First, create a filesystem, Virtual Volume, NFS export and Datastore: 1. Create the file system in NAS Manager as shown in the example below: March 2017 Hitachi Data Systems Page 26

33 2. Optionally, create a Virtual Volume as shown in the example below: If you do not create a Virtual Volume, you need to create an NFS Export as shown in the example below: March 2017 Hitachi Data Systems Page 27

34 3. In vcenter, navigate to Home -> Storage and add the Datastore as shown in the example below: Looking at the Datastore properties (below), the capacity is the currently allocated space, not the expansion limit. This presents challenges when creating larger VMs because VMware is not aware of the NAS server s autoexpansion feature and that there is additional capacity available. March 2017 Hitachi Data Systems Page 28

35 4. Run the disk file usage command on the NAS server to display the current capacity: NAS server[evs1]:$ df -f fs-vm1 ID Label Size Used Snapshots Deduped Avail Thin FS Type fs-vm GB GB (13%) NA 0 B (0%) GB (87%) No 4 KB,WFS-2,128 DSBs,dedupe enabled 5. Check the status of thin provisioning on the NAS server: NAS server [EVS1]:$ filesystem-thin fs-vm1 filesystem-thin: executing on cluster node 1, though the EVS in context (1) is currently on cluster node 2 Thin provisioning is disabled Enabling file system thin provisioning Enabling thin provisioning on a file system enables vsphere to see the capacity available up to the NAS autoexpansion limit. Use the filesystem-thin command as shown in the example below: NAS server [EVS1]:$ filesystem-thin fs-vm1 on filesystem-thin: executing on cluster node 1, though the EVS in context (1) is currently on cluster node 2 Success Now, in vcenter, you can see the additional capacity: The df command also shows the additional capacity. March 2017 Hitachi Data Systems Page 29

36 Creating a VM thin provision Once the file system has been thin provisioned, you can create a VM with a vmdk disk thin provision to see the impact on the Datastore usage: Setting the file system quota mode You can set the amount of quota space that is charged by a file using the fs-quota-mode command. There are two options: File based - this mode charges the file size of a file to quotas. For example, if a file has a length of 1GiB, then 1GiB of usage is charged to the appropriate quotas. G [EVS1]:$ fs-quota-mode -f fs-vm1 File system fs-vm1 is using file size based quotas Usage based - this mode charges the on disk usage of a file to quotas. For example, if a sparse file has a length of 1GiB, but has only written to 100KiB of that file, then 100KiB is charged to the quota. Only user data blocks are charged to the quota - metadata associated with the file is not charged. G [EVS1]:$ unmount fs-vm1 File system successfully unmounted. G [EVS1]:$ fs-quota-mode -f fs-vm1 --usage-based File system fs-vm1 is using usage based quotas G [EVS1]:$ mount fs-vm1 Request to mount file system "fs-vm1" submitted successfully. File system "fs-vm1" successfully mounted. March 2017 Hitachi Data Systems Page 30

37 In vcenter, the 10TiB hard limit is still in place but the usage is much smaller (60KB instead of 100GB). Creating quotas If you created a Virtual Volume, Virtual Volume quotas can be created with a soft limit or a hard limit. Soft limit quotas In the example below, a soft limit of 10TB is created for the quota. March 2017 Hitachi Data Systems Page 31

38 The nfs-export command displays the 20TB capacity: NAS server[evs1]:$ nfs-export list vm1 Export name: /vm1 Export path: /vm1 File system label: fs-vm1 File system size: GB File system free space: GB File system state: formatted = Yes mounted = Yes failed = No thin provisioned = Yes File system thin size: TB File system thin free space: TB vcenter also displays the 20TB capacity: However, the quota command displays the 10TB soft limit: NAS server[evs1]:$ quota list fs-vm1 Type : Explicit Target : ViVol: vm1 Usage : 100 GB Limit : 10 TB (Soft) March 2017 Hitachi Data Systems Page 32

39 Hard limit quotas In the example below, a hard limit of 10TB is created for the quota: The nfs-export command displays the full 20TB capacity: NAS server[evs1]:$ nfs-export list vm1 Export name: /vm1 Export path: /vm1 File system label: fs-vm1 File system size: GB File system free space: GB File system state: formatted = Yes mounted = Yes failed = No thin provisioned = Yes File system thin size: TB File system thin free space: TB March 2017 Hitachi Data Systems Page 33

40 However, vcenter displays the 10TB capacity: The quota command also displays the 10TB hard limit: G [EVS1]:$ quota list fs-vm1 Type : Explicit Target : ViVol: vm1 Usage : 100 GB Limit : 10 TB (Hard) Setting the file system quota mode You can set the amount of quota space that is charged by a file using the fs-quota-mode command. There are two options: File based - this mode charges the file size of a file to quotas. For example, if a file has a length of 1GiB, then 1GiB of usage is charged to the appropriate quotas. G [EVS1]:$ fs-quota-mode -f fs-vm1 File system fs-vm1 is using file size based quotas Usage based - this mode charges the on disk usage of a file to quotas. For example, if a sparse file has a length of 1GiB, but has only written to 100KiB of that file, then 100KiB is charged to the quota. Only user data blocks are charged to the quota - metadata associated with the file is not charged. G [EVS1]:$ unmount fs-vm1 File system successfully unmounted. G [EVS1]:$ fs-quota-mode -f fs-vm1 --usage-based File system fs-vm1 is using usage based quotas G [EVS1]:$ mount fs-vm1 Request to mount file system "fs-vm1" submitted successfully. File system "fs-vm1" successfully mounted. March 2017 Hitachi Data Systems Page 34

41 In vcenter, the 10TiB hard limit is still in place but the usage is much smaller (60KB instead of 100GB). Additional NAS and VMware best practices considerations NAS file system Allocation on Demand versus Allocate Now When creating NAS file systems, there are two options: Allocation on Demand and Allocate Now. Either option can be used. This section discusses the pros and cons for each option. Allocation on Demand With Allocation on Demand, whole chunks of capacity are allocated from a storage pool to a file system when initially created. The initial allocation, and automatic growth limit is predetermined. The growth limit can be modified at any time. Pros: Allows for more flexibility if file system capacity requirements are not well understood at creation or if capacity requirements change. Chunks are only used as required. The storage pool has available space to create future file systems. NAS file system thin provisioning presents the growth limit (capacity) to the clients, not the actual size. March 2017 Hitachi Data Systems Page 35

42 Cons: Actual file system utilization can be higher (because less capacity is allocated). For example, if an initial file system size is 150GB with a growth limit of 500GB, and there is 100GB of data, in this model the utilization rate would be ~66% vs 20%. Allocate Now With Allocate Now, all chunks from the storage pool are allocated on file system creation. Pros: All capacity is allocated up front. The file system can still be expanded (provided there are unused chunks in the storage pool). Cons: Once chunks are allocated they cannot be reclaimed. Allocation type considerations In general, when using Allocate on Demand, ensure the initial allocation is sufficient to span all stripesets in the storage pool (if multiple are present) and that it leaves sufficient free space after any VM s are moved onto the file system. When using Allocate Now, ensure the allocated capacity is sufficiently large enough to span all stripesets in the storage pool (if multiple are present). The Allocate now option may be preferred over the Allocate on Demand option in order to prevent a possible decrease in performance. March 2017 Hitachi Data Systems Page 36