Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd

Transcription

1 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd How Dell EMC provides the storage density and compute power to maximize the benefits of S2D and the advance features sets in Windows Server ****This paper is now deprecated**** To view the new deployment guide, see To view more information about Storage Spaces direct, see: Dell Engineering November 2016 A Dell EMC Reference Architecture

2 Revisions Date November 2016 Description Initial release THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Copyright 2016 Dell Inc. All rights reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. 2 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

3 Table of contents Revisions Introduction Audience Assumptions Hardware overview Server overview Switch overview Storage Spaces Direct overview Deployment scenarios Cluster Shared Volumes File system Virtual disk Storage pool Software Storage Bus Storage Bus Cache Configuring your cluster Installing the operating system and hardware update Joining the node to an Active Directory Domain Installing Windows Server roles and features (for hyper-converged only) Validating the cluster Creating a failover cluster Setting file share witness Enabling Storage Spaces Direct Creating a Cluster Shared Volume Optimizing the storage pool Conclusion A BIOS configuration A.1 Memory Operating Mode A.2 System Profile A.3 UEFI Boot Mode A.4 Trusted Platform Module (TPM) A.5 Secure Boot Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

4 B Configuration details C Terminology Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

5 1 Introduction This guide provides an overview for deploying Storage Spaces Direct (S2D), a new feature introduced in Microsoft Windows Server 2016, using the latest Dell EMC PowerEdge R730xd servers and Dell EMC Networking S4048 switch. The Dell EMC R730xd provides customers the storage density and compute power to maximize the benefits of S2D and the advance features sets in Windows Server Storage Spaces Direct (S2D) Dell EMC PowerEdge R730xd server Dell EMC Networking S4080 switch 5 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

6 2 Audience Relative to delivering the hyper-converged environment, the objectives of this document are to define: Technical design for the solution Hardware used to support the design Constraints, which are relevant to the design Relevant risks, issues, assumptions and concessions referencing existing ones where possible. Breakdown of the design into key elements such that the reader receives an incremental or modular explanation of the design 6 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

7 3 Assumptions The following are assumptions regarding the state of the Dell EMC PowerEdge R730xd: The reader has working knowledge to deploy, manage and update the Dell EMC PowerEdge R730xd. For additional documentation, see the Dell EMC PowerEdge R730xd Manuals page. The R730xd has been updated to the latest firmware and drivers. For the latest drivers and firmware visit the PowerEdge R730xd product support page. The configuration outlined in Appendix B is being used to enable Storage Spaces Direct. The BIOS settings outlined in Appendix A have been implemented in your solution. The following are assumptions about the state of Microsoft Windows Server 2016 deployment: The reader has working knowledge to deploy, manage, and update the operating system. Microsoft Windows Server 2016 Datacenter Edition with the Desktop Experience (Full GUI) has already been installed and is ready for configuration. The latest fixes and patches have already been applied to the operating system using Windows Updates. The following assumptions have been made about the network state: The servers have access to a domain controller/dns/dhcp server that has already been configured. A Hyper-V switch with Switch Embedded Teaming (Set) for the RDMA adapters is available. Hyper-V virtual NIC for RDMA adapters has been created and RDMA has been enabled and configured to route across subnets. Data Center Bridging (DCB) has been configured on the nodes and on the network switch. Jumbo frames with the 9014 MTU frame size has been configured across the network. Priority Flow Control (PFC) has been configured on all nodes and the network switch. RDMA virtual NICs have been configured with static IP addresses, subnet masks, and default gateways. All management traffic is converging over the RDMA network adaptors and a separate virtual NIC has been configured to handle the management traffic 7 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

8 4 Hardware overview 4.1 Server overview The goal is to maximize server-based storage and performance using the Dell EMC PowerEdge R730xd, part of the PowerEdge 13 th generation servers. The 2U rack mounted R730xd provides a high storage density solution that allows for (front of server), (internal) and (back of server) drive slots to help you achieve your storage needs. With 2 CPU sockets available and a wide range of CPU options the R730xd also provides the capabilities to match your computational needs. A four node cluster was used to take advantage of the failover capabilities in S2D. A two-tiered configuration was used, which consists of the Storage Bus Cache (SBC). The SBC is used as cache and a hot data tier. Virtual disks are then created from capacity devices and used as the cold data tier. In this implementation, we used 800 GB SATA SSDs for caching and 4 TB SATA HDDs for the capacity tier. The Mellanox ConnectX-3 Pro was used to take advantage of Remote Direct Memory Access (RDMA). RDMA allows for significantly increased throughput and lower latency by performing direct memory transfers between servers. Storage Spaces Direct uses SMB for all intra-node communication and uses RDMA with SMB Direct to provide tremendous performance benefits. For server component details, see Appendix B. 4.2 Switch overview The Dell EMC Network S port provides a 10 GbE-capable network switch for storage, cluster, and client traffic. Redundant switches are not required, but are recommended for high-availability (HA). If connecting to two separate network switches and implementing switch embedded teaming (SET), both switches require access to all subnets so that failover can occur. You must configure two identical switches and provide redundant paths for all communication. DCB must be configured as it and provides enhancements to the Ethernet protocol, which improves functionality for data center networks. To take advantage of the Mellanox RDMA over Converged Ethernet (ROCE) network adapters, Priority Flow Control (PFC) and Enhanced Transmission Selection (ETS) are required. PFC and ETS were configured on all nodes and all network switches interconnecting the nodes. For additional information enabling DCB is available at: Data Center Bridging (DCB) Overview. 8 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

9 5 Storage Spaces Direct overview Storage Spaces Direct (S2D) is a new software defined storage feature of Microsoft Windows Server 2016 and is included as part of the Datacenter Edition of the operating system at no additional cost. Storage Spaces Direct (S2D) uses a share-nothing storage architecture where multiple servers with attached drive devices are combined in a cluster to create a highly available and scalable storage subsystem. Overview of Storage Spaces Direct 5.1 Deployment scenarios There are two deployment scenarios for Storage Spaces Direct. The disaggregate and hyper-converged deployments both focus on providing storage for Hyper-V IaaS (infrastructure as a service) for service providers and enterprises. In this guide, we focus on hyper-converged configurations; however, we have included notes for disaggregate solutions as well. In a hyper-converged infrastructure (HCI) the compute and storage are resourced from the same cluster of servers to provide a highly-virtualized solution. This deployment option simplifies the manageability and scalability since additional nodes will increase both storage and compute resources. Virtual machines are stored on local CSVs and therefore do not require a scale-out file server. HCI deployments are best suited for small to mid-size environments. In an S2D disaggregate solution, the Hyper-V servers (compute components) are separate from the Scale- Out File Server (storage component). The VMs use a Scale-Out File Server to store their files through the network using the SMB3 protocol. This deployment allows for independent scaling of the compute and storage components and is a good fit for any workload that is compatible with SMB storage. Hyper-V running directly on the nodes is not supported on disaggregated deployments. 9 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

10 5.2 Cluster Shared Volumes Cluster Shared Volume File System (CSVFS) sit above ReFS to bring all the mounted volumes into a single namespace accessible through any node. You must have one cluster-shared volume (CSV) for every VD. You should add CSVs in multiples of the cluster nodes to enable the CSVs to load balance across the cluster. For example, if you have a 2 node configuration, you should have a minimum of 2 CSVs. In this paper, we discuss a 4-node cluster and uses 4 CSVs. 5.3 File system ReFS provides the file system in which the Hyper-V VM files are stored and is the default file system recommended when using Storage Spaces Direct. ReFS provides several advantages over NTFS such as automatic error detection and correction, and many optimizations for Hyper-V virtual hard drive operations. In addition, ReFS is required for using multi-resilient virtual disks. Note: ReFS does not support deduplication. NTFS is still supported with Storage Spaces Direct, but you only use it if required by a specific workload or if deduplication is required. NTFS does not support Multi-Resilient virtual disks. Throughout this paper we will focus on ReFS. 5.4 Virtual disk Virtual disks are created from free capacity in a storage pool and you can configure virtual disks with attributes such as resiliency and storage tiers. You can configure virtual disk with various resiliency types mirror, parity or multi-resilient. Virtual disk resiliency types Resiliency Type Capacity Efficency File System Numbers of Nodes Required Mirror 33% NTFS and ReFS 3+ Parity Varies (50%+) NTFS and ReFS 4+ Multi-resilient Varies (50%+) ReFS only 4+ Multi-resilient virtual disks are a new feature with Storage Spaces Direct and combines the performance improvements of mirroring with the capacity efficiency of parity. A virtual disk formatted with Resilient File System (ReFS) and at least four nodes is required to use multi-resilient virtual disks. When a multi-resilient virtual disk is created, two tiers are designated, one as a mirror tier and the other as a parity tier. When data is de-staged from the SBC, it is written to the mirror tier since it is best at handling random IO. Periodically, 10 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

11 ReFS automatically rotates data from the mirror tier to the parity tier in large sequential chunks, which overwrites entire parity stripes eliminating the Read-Modify-Write (RMW) cycle and reduces the impact of parity calculations. Sizing guidance for the mirror tier and parity tier of a multi-resilient virtual disk can vary depending on the workload characteristics, but the mirror tier must be sized to accommodate the active working set. Note: If you do not know the size of the active working set, a good general rule of thumb is to size the mirror tier as 20 percent of the total volume size. Note: Virtual disks do not support thin provisioning. 5.5 Storage pool A storage pool is a logical grouping of storage devices that enables the aggregations of storage capacity. Optimally, all storage devices should be in a single pool, but multiple storage pools can be used if necessary. Using the automatic enable-clusters2d command automatically places all available storage into one pool. 5.6 Software Storage Bus The Software Storage Bus (SSB) is automatically configured when Storage Spaces Direct is enabled. The SSB is a virtual storage bus that spans all nodes that make up the Storage Spaces Direct cluster. The SSB allows each node of the cluster to see the disks across all other nodes in the cluster. The SSB uses SMB and SMB Direct for communication between nodes. 5.7 Storage Bus Cache The Storage Bus Cache (SBC) is scoped to each node and is a per node cache. Storage devices are categorized as caching devices or capacity devices. Caching devices can consist of SSD or NVMe storage devices, while capacity devices are typically SSD or HDD storage devices. Capacity devices are bound to caching devices in a round robin manner and will be reassigned automatically if there is a caching device failure or topology change. When caching devices are configured SBC automatically consumes all capacity on the caching devices except 32GB, which is used for storage pool and virtual disk metadata. Any storage devices designated as SBC will be obfuscated by Storage Spaces Direct and will not show up as usable capacity. In Server Manager, they may be displayed as 32 GB capacity devices, however the remaining capacity is being actively used. The SBC is divided into a read and write cache depending on the storage device configuration. The SBC is not part of a virtual disk and data resiliency is achieved by writing multiple data copies to the SBC in different nodes of the cluster. 11 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

12 When Storage Spaces Direct is enabled by default, it automatically selects the highest performance devices available and assign them as caching devices. However, if Storage Spaces Direct was enabled and automatic configuration was disabled, caching devices can be manually assigned using this PowerShell command: Get-StoragePool S2D Get-PhysicalDisk? MediaType -eq SSD Set-PhysicalDisk - Usage Journal For more information, see Storage Spaces Direct in Windows Server Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

13 6 Configuring your cluster 6.1 Installing the operating system and hardware update The first step is to ensure that Windows Server 2016 Datacenter is installed and activated on all nodes. Make sure to install all Microsoft updates and patches that are available. As a common best practice ensure your server firmware and drivers are up to date. 6.2 Joining the node to an Active Directory Domain All cluster nodes must be joined to the same domain. This step requires domain credentials and requires a reboot. Continue to log in to an account with domain administration credentials for the remaining steps. To join a node to an Active Directory domain: 1. Open Server Manager. 2. On the left, select Local Server. 3. Click the domain (default WORKGROUP). 13 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

14 4. On the System Properties dialog box, click Change. 5. On the Computer Name/Domain Changes dialog box, click Domain. 14 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

15 6. Type your domain credentials. You will be asked for authentication. 7. This is also the time to rename the node if desired (recommended). 8. A reboot is required. 6.3 Installing Windows Server roles and features (for hyper-converged only) Install the Failover-Clustering feature and the Hyper-V roles including the management tools specified for deploying the nodes in a hyper-converged architecture. Do not perform this step if deploying as a disaggregated architecture. This step requires a reboot. Use PowerShell to install the features and roles you need. Install-WindowsFeature -Name Hyper-V -IncludeManagementTools Install-WindowsFeature -Name File-Services, Failover-Clustering - IncludeManagementTools Note: You can configure a SOFS from the Failover Cluster Manager by adding a role to the cluster. 15 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

16 6.4 Validating the cluster Before you begin to create the Failover Cluster, use the following PowerShell command to validate the cluster and to determine whether this configuration of servers is set up correctly to support failover. Test-Cluster -Node $node1,$node2,$node3,$node4 -Include "Storage Spaces Direct",Inventory,Network,"System Configuration" 6.5 Creating a failover cluster Use the following PowerShell command to create a failover cluster: New-Cluster $ClusterName -Node $node1,$node2,$node3,$node4 -NoStorage 6.6 Setting file share witness You can configure a cluster file share witness to maintain a cluster or storage quorum when there is a node or network communication failure where nodes continue to operate but can no longer communicate between one another. More information is found at Understanding Quorum Configurations in a Failover Cluster. Use this link to learn more on how to Configure a File Share Witness on a Scale-Out File Server Use this link to learn more on configuring Cloud Witness, a new type of Failover Cluster quorum witness being introduced in Windows Server Enabling Storage Spaces Direct Storage Spaces Direct is not enabled by default. You must manually enabled it using PowerShell. Note: Use PowerShell when configuring Storage Spaces Direct as not all options and capabilities are available when using the GUI interface in Failover Cluster Manager or Server Manager. Dell EMC recommends running a script to clean and online all disks. An official Microsoft PowerShell script is available here: By default, when enabling Storage Spaces Direct, it automatically performs some initial configuration. The automatic configuration performs the following steps: Creates a single storage pool named S2D on <ClusterName> using all eligible storage devices Configures the storage bus cache using the most performance storage devices (SSD) Creates two default tiers, one called Capacity and the other called Performance Automatically analyzes all available storage devices and configures them for each tier 16 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

17 Creates a single storage pool Configures the storage bus cache Creates two default tiers Automatically analyzes and configures all available storage devices Initial configuration steps. To enable Storage Spaces Direct with automatic configuration run the PowerShell Command: Enable-ClusterS2D To verify if Storage Spaces Direct is enable on the node run the PowerShell Command: Get-ClusterS2D Note: Dell EMC recommends that you use the verbose flag to watch the progress in case of failure. For the latest information to enable S2D or manually configure S2D, refer to the Microsoft Tech Center. Enable-clusters2d -Autoconfig:0 -SkipEligibilityChecks $pd = Get-PhysicalDisk? CanPool -eq $true New-storagepool -StorageSubSystemFriendlyName clus* -friendlyname S2D - ProvisioningTypeDefault Fixed -PhysicalDisks $pd Get-StoragePool S2D Get-PhysicalDisk? MediaType -eq SSD Set-PhysicalDisk - Usage Journal $MT = New-StorageTier StoragePoolFriendlyName S2D -FriendlyName MT -MediaType HDD -ResiliencySettingName Mirror -PhysicalDiskRedundancy 2 $PT = New-StorageTier StoragePoolFriendlyName S2D -FriendlyName PT -MediaType HDD -ResiliencySettingName Parity -PhysicalDiskRedundancy Creating a Cluster Shared Volume At this point, you should have a fully functional cluster, storage pool, and S2D should be running correctly. It is now time to configure the storage. The following PowerShell is an example configuration for a 4-node system. You can modify the code for other applications. 17 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

18 We need to store the information concerning the storage pool and the storage tiers. $poolname = Get-StoragePool S2D* $perf = Get-storagetier MT $capa = Get-storagetier PT To set up the volumes, change the values for the FriendlyName (name of your CSV). In addition, you must change the size of the tiers to match with the storage capacity of your system. New-Volume -StoragePool $poolname -FriendlyName FriendlyName -FileSystem CSVFS_REFS -StorageTiers $perf, $capa -StorageTierSizes XGB, XGB -Verbose Next, run the command once for each CSV. In this case, we ran the command four times incrementing FriendlyName each time. 6.9 Optimizing the storage pool Over time, as data is written and deleted, the distribution of data within the storage pool may become unbalanced. Running an optimize operation rebalances the data across the Storage Spaces Direct cluster. Additionally, as storage devices are added or removed from the pool, optimizing the existing data to use the new disks results in better storage efficiency and performance across the pool. Optimization requires resources to run, so while the heat map is updated daily, running extra optimizations may put strain on the system. You can run an optimize operation on demand using the following PowerShell command: Optimize-StoragePool <PoolName> You can see the status of an on-going optimize job with the following PowerShell command: Get-StorageJob? Name eq Optimize 18 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

19 7 Conclusion Storage Spaces Direct on Windows Server 2016 introduces many improvements that enables your ability to create highly available, highly scalable software-defined storage while simplifying the deployment and management of SDS systems. This evolution in Microsoft SDS offerings unlocks the ability to use directly attached local storage and the ability to use SATA and NVME devices. Throughout this document we focused on showcasing the ability of the Dell EMC PowerEdge R730xd server to provide the flexibility, compute, and storage requirements to fully take advantage of the new SDS offering on Windows Server Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

20 A BIOS configuration Ensure the following BIOS configurations are set in the server BIOS before proceeding with any additional steps. The following BIOS configuration were set in settings in the server BIOS A.1 Memory Operating Mode 1. To confirm this setting, when you turn on the server power, press F2 to enter System Setup. 2. Click System BIOS. 3. Click Memory Settings. 4. In the Memory Operating Mode drop-down list, ensure Optimizer Mode is selected. A.2 System Profile 1. To confirm this setting, when you turn on the server power, press F2 to enter System Setup. 2. Click System BIOS. 3. Click System Profile Settings. 4. In the System Profile drop-down list, ensure Performance is selected. A.3 UEFI Boot Mode 1. To confirm this setting, when you turn on the server power, press F2 to enter System Setup. 2. Click System BIOS. 3. Click Boot Settings 4. Ensure UEFI is selected for the Boot Mode. A.4 Trusted Platform Module (TPM) 1. If ordered with your server (optional), ensure TPM is enabled. 2. To confirm this setting, when you turn on the server power, press F2 to enter System Setup. 3. Click System BIOS. 4. Click System Security. 5. Ensure On is selected for the TPM Security. A.5 Secure Boot 1. Secure boot is not required for DHMS, but if it is implemented, you must enabled it in the BIOS. Secure boot configuration is outside the scope of this document. 2. To confirm this setting, when you turn on the server power, press F2 to enter System Setup. 3. Click System BIOS. 4. Click System Security. 5. Ensure Enabled is selected for the Secure Boot. 20 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

21 B Configuration details The component list shown in Table 2 is a minimum requirement. Minimum requirement list Component Detail Platform Dell EMC PowerEdge R730xd (2U, 16 x 3.5 and 2 x 2.5 rear drives) TPM TPM 2.0 CPU Memory 2 x Intel Xeon CPU E GB RDIMM, 2400MT/s, Dual rank, x8 Data Width OS drive 480GB SATA RI SSD (Intel S3510) NDC NIC System management Power supply Storage adapter Intel 1350 QP 1 Gb Mellanox ConnectX-3 Pro 10 Gb SFP+ idrac8 Enterprise Redundant 750 W HBA330 SSD (caching) 4 x 2.5 SATAL 800 GB (Intel S3710) HDD (capacity) OS Networking switch 12 x 3.5 SATA: 4 TB (Seagate Makara BP, Toshiba Tomcat R, HGST Aries K Plus) Windows Server 2016 Datacenter Edition 2 x S4048 (48 port 10 GbE SFP+) 21 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

22 C Terminology Resiliency Refers to the method by which a virtual disk (VD) is protected against physical disk failures. There are four types of resiliency options for Microsoft Storage Spaces Direct Simple, Mirrored, Parity, and Multi-Resilient. Simple VD o Data is striped across the physical disks. o No resiliency provided. o No protection against physical disk failures. Mirrored VD o Data is striped across physical disks o An extra one or two copies of the data is also written to a second set of physical disk to protect against physical disk failures. o A two-way mirror can tolerate one physical disk failure o A three-way mirror can tolerate two physical disk failures Parity VD o A three-way mirror can tolerate two physical disk failures o There are two types of parity options single parity and dual parity. Single parity writes one copy of parity information protecting against one hard drive failure. Dual parity writes two copies and protects against two hard disk drive (HDD) failures. Multi-Resilient VD o Contains two tiers. The first tier is a mirrored tier and the second tier is a parity tier A mirrored tier is optimized for performance Parity tier is optimized for capacity. o Data is automatically tiered between the two tiers using ReFS. Scale-Out File Server (SOFS) A role that enables a cluster to act as an active-active file server, which increases performance and provides highly available SMB file shares. Hyper-V A role that enables a server to host multiple guest virtual machines (VMs) that share the underlying hardware resources by virtualizing the processors, memory, and storage. Storage node A physical server, which is a member of a SOFS failover cluster. Compute node A physical server, which is a member of a failover cluster running a compute workload (for example, Hyper-V and SQL) Storage pools A group of physical disks, which allows you to manage and use the combined disk space of all the physical disks in the storage pool. Caching devices A storage device designated to be used for the Storage Bus Cache. These devices offer the most performance and are either SSDs or NVMe. 22 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00

23 Capacity devices - A storage device designated to be used for data storage and will be the devices used to create virtual disks. These devices are either SSDs or HDDs. Software storage bus (SSB) Created when Storage Spaces Direct is enabled. A virtual storage bus spanning all the servers that make up the cluster. This feature is used so that each server can see and use all available storage drives internal to each server. Storage Bus Cache (SBC) Created when Storage Spaces Direct is enabled. This is a per server cache which is used to accelerate reads and writes for the capacity devices in the server. 23 Microsoft Storage Spaces Deployment on Dell EMC PowerEdge R730xd A00