This guide provides configuration settings and considerations for Hosts running VMware ESXi.

Transcription

1 The Host Server VMware ESXi Configuration Guide May 2018 This guide provides configuration settings and considerations for Hosts running VMware ESXi. Basic VMware storage administration skills are assumed including how to connect to iscsi and Fibre Channel target ports and the discovering, mounting and formatting of disk devices. Also see: This is the official statement from DataCore about differences between the information in this document and VMware's own Hardware Compatibility List The Data Infrastructure Software Company

2 Table of contents Changes made to this document 3 VMware ESXi compatibility lists 4 ESXi Operating system versions 4 VMware ESXi Path Selection Policies (PSP) 6 VStorage API for Array Integration (VAAI) support 7 VMware VVOL VASA API VSphere Metro Storage Clusters (VMSC) 8 The DataCore Server's settings 9 Operating system type 9 Port roles 9 Multipathing support ALUA support 9 10 Serving a Virtual Disk for the first time Serving a Virtual Disk to more than one Host port DataCore Servers in an ESX Virtual Machine 11 The VMware ESXi Host's settings 12 ISCSI connections 13 TCP Ports 13 Advanced settings 13 DiskMaxIOSize 13 Enable VM Component Protection (VMCP) 13 Adjust the Round Robin IOPS limit 13 VMware Path Selection Policies 14 Configuring the Round Robin Path Selection Policy 16 Configuring the Fixed Path Selection Policy 18 Configuring the Most Recently Used Path Selection Policy 20 Known issues 22 ESXi 6.x (includes 6.0.x and 6.5.x) 23 ESXi 5.x (includes 5.0.x, 5.1.x and 5.5.x) 28 Appendix A 32 Preferred Server & Preferred Path settings 32 Appendix B 34 Reclaiming storage from Disk Pools 34 Appendix C 38 Moving from Most Recently Used to either Round Robin or Fixed Path Selection Policy 38 Previous Changes 39 Page 2

3 Changes made to this document The most recent version of this document is available from here: All changes since February 2018 The VMware ESXi Host's settings Advanced Settings Enable VM Component Protection (VMCP) VMware ESXi 6.x only. Enable VM Component Protection (VMCP) on your HA cluster to allow the cluster to react to all paths down and permanent device loss conditions by restarting the VMs. Known Issue affects all versions of ESXi ESXi hosts need to perform a rescan whenever Virtual Disks are unserved See and Without a rescan on the Host, ESXi will continue to send SCSI commands to DataCore Server Frontend Ports for LUNs that are no longer served. This causes the DataCore Server to have to send back an appropriate ILLEGAL_REQUEST SCSI response each time the missing LUN is probed for by the Host. In extreme cases, when large numbers of Virtual Disks are unserved, the amount of SCSI commands generated by this send-and-response will significantly affect performance any Host that is using the Front End Port(s) for existing Virtual Disks VMware Path Selection Policies - Configuring the Round Robin Path Selection Policy While it is still possible to use VMware ESXi s built-in, generic 'VMW_SATP_ALUA' rule, e.g: VMW_SATP_ALUA system tpgs_on Any array with ALUA support DataCore are now recommending that this custom SATP rule be used instead: esxcli storage nmp satp rule add -V DataCore -M "Virtual Disk" -s VMW_SATP_ALUA -c tpgs_on -P VMW_PSP_RR -O iops=10 Important notes for users that were already using the previously documented custom SATP rule: The difference between the old and new custom rule is the addition of the -O iops=10 Remove the existing custom rule before trying to add the new one e.g: esxcli storage nmp satp rule remove -V DataCore -M "Virtual Disk" -s VMW_SATP_ALUA -c tpgs_on -P VMW_PSP_RR While it is possible to adjust the existing rule using the command line - see - this is not persistent over a reboot, therefore DataCore do not recommend following the command in the VMware KB article. All previous changes Please see page 39 Page 3

4 VMware ESXi compatibility lists ESXi Operating system versions SANsymphony 9.0 PSP 4 Update 4 (1) 10.0 (all versions) ESXi Version With ALUA Without ALUA With ALUA Without ALUA 5.x Qualified Not Qualified Qualified Not Qualified 6.x Qualified Not Qualified Qualified Not Qualified Notes: Qualified vs. Not Qualified vs. Not Supported See the next page for definitions. DataCore Server Front-End Port connections Fibre Channel and iscsi are supported. VMware VAAI (VStorage API for Array Integration) compatibility See the table on page 6 for more ESXi version-specific information. Reclaiming storage from DataCore Disk Pools See Appendix B: 'Reclaiming Storage from Disk Pools' on page 34 for version-specific instructions. 1 SANsymphony-V 8.x and all versions of SANsymphony 9.x before PSP4 Update 4 are now End of Life. Please see: End of Life Notifications Page 4

5 Qualified vs. Not Qualified vs. Not Supported VMware ESXi compatibility lists Qualified This combination has been tested by DataCore and all the host-specific settings listed in this document applied using non-mirrored, mirrored and Dual Virtual Disks. Not Qualified This combination has not yet been tested by DataCore using Mirrored or Dual Virtual Disks types. DataCore cannot guarantee 'high availability' (failover/failback, continued access etc.) even if the host-specific settings listed in this document are applied. Self-qualification may be possible please see Technical Support FAQ #1506 Mirrored or Dual Virtual Disks types are configured at the users own risk; however, any problems that are encountered while using VMware versions that are 'Not Qualified' will still get root-cause analysis. Non-mirrored Virtual Disks are always considered 'Qualified' - even for 'Not Qualified' combinations of VMware/SANsymphony. Not Supported This combination has either failed 'high availability' testing by DataCore using Mirrored or Dual Virtual Disks types; or the operating System's own requirements/limitations (e.g. age, specific hardware requirements) make it impractical to test. DataCore will not guarantee 'high availability' (failover/failback, continued access etc.) if the host-specific settings listed in this document are applied. Mirrored or Dual Virtual Disks types are configured at the users own risk. Self-qualification is not possible. Mirrored or Dual Virtual Disks types are configured at the users own risk; however, any problems that are encountered while using VMware versions that are 'Not Supported' will get best-effort Technical Support (e.g. to get access to Virtual Disks) but no root-cause analysis will be done. Non-mirrored Virtual Disks are always considered 'Qualified' even for 'Not Supported' combinations of VMware/SANsymphony. VMware versions that are End of Support Life (EOSL) / Availability (EOA) / Distribution (EOD) For versions that are listed as 'Not Supported', self-qualification is not possible. For versions that are listed as 'Not Qualified', self-qualification may be possible if there is an agreed support contract with VMware as well. Please contact DataCore Technical Support before attempting any self-qualification. For any problems that are encountered while using VMware versions that are EOSL, EOA or EOD with DataCore Software, only best-effort Technical Support will be performed (e.g. to get access to Virtual Disks). Root-cause analysis will not be done. Non-mirrored Virtual Disks are always considered 'Qualified'. Page 5

6 VMware ESXi Path Selection Policies (PSP) VMware ESXi compatibility lists VMware Path Selection Policy ESXi Version Most Recently Used (MRU) (use without ALUA enabled) Fixed (use with ALUA enabled) Round Robin (RR) (use with ALUA enabled) 5.x Not Tested Tested/Works Tested/Works 6.x Not tested Tested/Works Tested/Works Notes: Tested/Works vs Not Tested This table only applies to combinations that are listed as 'Qualified' on page 4. ESXi version 6.x Fixed and RR PSPs are both listed on VMware's Hardware Compatibility List. MRU is not. ESXi version 5.x Only RR PSP is listed on VMware's Hardware Compatibility List. Both Fixed and MRU are not. Page 6

7 VMware ESXi compatibility lists VStorage API for Array Integration (VAAI) support SANsymphony 9.0 PSP 4 Update 4 (1) 10.0 (all versions) ESXi Version VAAI VAAI 5.x Tested/Works Tested/Works 6.x Tested/Works Tested/Works Notes: VAAI-specific commands that are supported by the DataCore Server: Atomic Test & Set (ATS) Clone Blocks/Full Copy/XCOPY Zero Blocks/Write Same Block Delete/SCSI UNMAP Tested/Works vs. Does not Work This table only applies to combinations that are listed as 'Qualified' on page 4. Reclaiming storage from DataCore Disk Pools See Appendix B: 'Reclaiming Storage from Disk Pools' on page 34 for version-specific instructions. 1 SANsymphony-V 8.x and all versions of SANsymphony 9.x before PSP4 Update 4 are now End of Life. Please see: End of Life Notifications Page 7

8 VMware ESXi compatibility lists VMware VVOL VASA API 2.0 SANsymphony 9.0 PSP 4 Update PSP3 and earlier 10.0 PSP 4 and greater ESXi Version VASA support for VVOL VASA support for VVOL VASA support for VVOL 5.x VVOL Support Not Available VVOL Support Not Available Not Supported 6.x VVOL Support Not Available VVOL Support Not Available Tested/Works Notes: Tested/Works vs. Not Supported VMSC support only applies to combinations that are listed as 'Qualified' on page 4. Configuration specific notes Please refer to 'Getting Started with the DataCore VASA Provider' section from the Online Help: VSphere Metro Storage Clusters (VMSC) Notes: Qualified vs. Not Supported VMSC support only applies to combinations that are listed as 'Qualified' on page 4. Selfqualification may be possible for combinations that are listed as 'Not Qualified' - please contact DataCore Technical Support. Virtual Disks that were formatted using VMFS5 are supported. Virtual Disks formatted with earlier versions of VMFS and have then been upgraded to VMFS5 are not supported. Page 8

9 The DataCore Server's settings Also see: Video: Configuring ESX Hosts in the DataCore Management Console Operating system type When registering the Host choose the 'VMware ESXi' menu option. See the Registering Hosts section from the SANsymphony Help: Port roles Ports used for serving Virtual Disks to Hosts should only have the Front End (FE) role enabled. Mixing other Port Role types may cause unexpected results as Ports that only have the FE role enabled will be turned off when the DataCore Server software is stopped (even if the physical server remains running). This helps to guarantee that any Hosts do not still try to access FE Ports, for any reason, once the DataCore Software is stopped but where the DataCore Server remains running. Any Port with the Mirror and/or Back End role enabled do not shut off when the DataCore Server software is stopped but still remain active. Multipathing support The Multipathing Support option should be enabled so that Mirrored Virtual Disks or Dual Virtual Disks can be served to Hosts from all available DataCore FE ports. Also see the Multipathing Support section from the SANsymphony Help: WebHelp/Hosts.htm Non-mirrored Virtual Disks and Multipathing Non-mirrored Virtual Disks can still be served to multiple Hosts and/or multiple Host Ports from one or more DataCore Server FE Ports if required; in this case the Host can use its own multipathing software to manage the multiple Host paths to the Single Virtual Disk as if it was a Mirrored or Dual Virtual Disk. Note: Hosts that have non-mirrored Virtual Disks served to them do not need Multipathing Support enabled unless they have other Mirrored or Dual Virtual Disks served as well. Page 9

10 The DataCore Server's settings ALUA support The ALUA support option (Asymmetrical Logical Unit Access) should be enabled if required and if Multipathing Support has been also been enabled (see above). Please refer to the Operating system compatibility table on page 4 to see which combinations of VMware ESXi and SANsymphony support ALUA. More information on Preferred Servers and Preferred Paths used by the ALUA function can be found on in Appendix A on page 32. Serving a Virtual Disk Serving a Virtual Disk for the first time DataCore recommends that before serving Virtual Disks for the first time to a Host, that all DataCore Front-End ports on all DataCore Servers are correctly discovered by the Host first. Then, from within the SANsymphony Console, the Virtual Disk is marked Online, up to date and that the storage sources have a host access status of Read/Write. Serving a Virtual Disk to more than one Host port DataCore Virtual Disks always have their own unique Network Address Authority (NAA) identifier that a Host can use to manage the same Virtual Disk being served to multiple Ports on the same Host Server and the same Virtual Disk being served to multiple Hosts. See the SCSI Standard Inquiry Data section from the online Help for more information on this: While DataCore cannot guarantee that a disk device's NAA is used by a Host's operating system to identify a disk device served to it over different paths generally we have found that it is. And while there is sometimes a convention that all paths by the same disk device should always using the same LUN 'number' to guarantees consistency for device identification, this may not be technically true. Always refer to the Host Operating System vendor s own documentation for advice on this. DataCore's Software does, however always try to create mappings between the Host's ports and the DataCore Server's Front-end (FE) ports for a Virtual Disk using the same LUN number (3) where it can. The software will first find the next available (lowest) LUN 'number' for the Host- DataCore FE mapping combination being applied and will then try to apply that same LUN number for all other mappings that are being attempted when the Virtual Disk is being served. If any Host-DataCore FE port combination being requested at that moment is already using that same LUN number (e.g. if a Host has other Virtual Disks served to it from previous) then the software will find the next available LUN number and apply that to those specific Host- DataCore FE mappings only. 3 The software will also try to match a LUN 'number' for all DataCore Server Mirror Port mappings of a Virtual Disk too, although the Host does not 'see' these mirror mappings and so this does not technically need to be the same as the Front End port mappings (or indeed as other Mirror Path mappings for the same Virtual Disk). Having Mirror mappings using different LUNs has no functional impact on the Host or DataCore Server at all. Page 10

11 The DataCore Server's settings DataCore Servers in an ESX Virtual Machine See the article Hyperconverged and Virtual SAN Best Practices guide: Page 11

12 The VMware ESXi Host's settings The following are the Host-specific settings that need to be configured directly on the Host Server. Note: Older versions (of VMware ESXi) may require different Host settings when compared to newer versions. When a setting or configuration change is listed for one version but not another then is only required for that specific version of VMware ESXi. If you have upgraded from an older version and a specific setting is no longer documented for your newer version then assume that no further changes are needed for those settings that are no longer listed in your newer version and they should be left as they were. Page 12

13 The VMware ESXi Host's settings ISCSI connections TCP Ports Open TCP Port 3260 for iscsi connections to a DataCore Server. Advanced settings DiskMaxIOSize From the vsphere Web Client, select the ESX Host and choose the Configure Tab. Then under the System options choose Advanced System Settings and verify that the following setting is set: Disk.DiskMaxIOSize = 512 Enable VM Component Protection (VMCP) VMware ESXi 6.x only. Enable VM Component Protection (VMCP) on your HA cluster to allow the cluster to react to all paths down and permanent device loss conditions by restarting the VMs. This will help to speed up the recovery process with automation should you encounter another APD/PDL storage event. For guidelines on setting up VM component protection see F01F7EB8-FF9D-45E2-A093-5F56A788D027.html Adjust the Round Robin IOPS limit Adjusting the limit down from the default setting can provide a positive impact to performance when using the Round Robin Path Selection Policy. DataCore recommends changing the IOPS down to a value of 10 from the default value of See Configuring the Round Robin Path Selection Policy on page 16 for more details. Page 13

14 VMware Path Selection Policies Which PSPs are qualified? Please refer to VMware ESXi Path Selection Policies (PSP) compatibility list on page 6. Which PSP does DataCore Software recommend? DataCore does not recommend one particular policy over another; one user s installation and configuration of SANsymphony will be different to another's. Note: Some PSPs are not supported by VMware themselves for certain types of Virtual Machine Operating Systems. DataCore cannot take responsibility for these VMware-unsupported Virtual Machine/PSP combinations should any issues occur. Also see Changing the PSP type on an already-served Virtual Disk As long as the same Storage Array Type Plug-in (SATP) being used on the Host is the same for the new PSP there is nothing that needs to be done on the DataCore Server. If the current SATP is different to what the new PSP requires (for example, moving from 'Most Recently Used' to 'Round Robin'), then DataCore recommend that you unserve the Virtual Disks first, delete the old SATP value adding the new SATP and serve them back again. Note: Changing the SATP type may also require that the ALUA option also be changed as well on the Host, within the SANsymphony Console, from its current setting. In which case see the After changing the settings section of changing multipath or ALUA support settings for hosts from the SANsymphony Help: Using different PSPs for the same Virtual Disk on multiple Hosts While this is technically possible, it is not supported and DataCore cannot guarantee the behavior of the VMware ESXi Hosts in this case. Always use the same PSP for the same Virtual Disk on all VMware ESXi Hosts that it is served to. Page 14

15 VMware Path Selection Policies Which Storage Array Type Plug-in (SATP) should I use? Please refer to the following pages to determine which SATP and how to configure it for the particular PSP you wish to use. Note: Auto-detection by VMware ESXi to choose the correct Path Selection Policy and/or Storage Array Plugin Type to use for a given Virtual Disk can be inconsistent in older versions and VMware ESXi may, for example, default to Most Recently Used by default for any Virtual Disk mapped to the Host regardless if the ALUA option had been enabled or not. This type of mistake will cause unexpected results during any failover event. It is, therefore, important to always verify manually, that both the correct PSP and SATP have been selected. This can be done directly on the VMware vsphere client GUI or by running the following command at the VMware ESXi console: esxcli storage nmp device list grep -A 7 ^naa\.60030d9 This looks for all devices that contain DataCore s unique NAA identifier - part of any SANsymphony Virtual Disk s SCSI Standard Inquiry Data. See the SCSI Standard Inquiry Data section from the SANsymphony Help: The additional -C switch for the grep command will display an additional 3 lines of output above and below the line that the string searched for appears on which should then include the Path Selection Policy and the Storage Array Type Plugin. Increase this number to display more of the Virtual Disk s properties as required. If using SANsymphony s own VMware vcenter Integration, searching by the NAA identifier is the only way to list the Virtual Disks on the command line. Page 15

16 VMware Path Selection Policies Configuring the Round Robin Path Selection Policy Use the SATP 'VMW_SATP_ALUA' with the claim option 'tpgs_on'. While it is possible to use VMware ESXi s built-in VMW_SATP_ALUA' rule, DataCore recommend that you create a custom VMW_SATP_ALUA SATP rule instead VMware ESXi s built-in VMW_SATP_ALUA' rule VMW_SATP_ALUA system tpgs_on Any array with ALUA support Creating a custom VMW_SATP_ALUA SATP rule To create a custom rule run the following command on the ESXi Host s console esxcli storage nmp satp rule add -V DataCore -M "Virtual Disk" -s VMW_SATP_ALUA -c tpgs_on -P VMW_PSP_RR -O iops=10 Verify the custom rule has been set correctly esxcli storage nmp satp rule list -s VMW_SATP_ALUA grep DataCore The response should look like something this (1) VMW_SATP_ALUA DataCore Virtual Disk user tpgs_on VMW_PSP_RR iops=10 This custom SATP rule can be used for all Virtual Disks from any DataCore Servers when using the Round Robin PSP. Important Notes: Round Robin is only supported with the ALUA option enabled on the Host from within the SANsymphony Management Console. Also see Changing multipath or ALUA support settings for hosts from the SANsymphony Help: 1 This example is taken from VMware ESXi version 6.5 Page 16

17 VMware Path Selection Policies The DataCore Server Preferred Server when using Round Robin Also see the Preferred Servers section from the SANsymphony Help: DataCore recommends configuring Hosts with an explicit DataCore Preferred Server or leave the 'Auto select' setting configured and not to use the All preferred server setting at least to start with. When the Host's Preferred Server setting is either 'Auto Select', or is using an explicitly named DataCore Server, only the Host paths that are connected to either the first DataCore Server listed in the Virtual Disk's properties (for 'Auto Select') or the named DataCore Server respectively are set as Active Optimized. The Host's paths connected to the other DataCore Server are set as 'Active Non-optimized'. VMware Hosts will only send IO to 'Active Optimized' paths when there is a choice between that or 'Active Non-optimized'. Caution is advised when using the All setting as this allows the VMware Host to send I/O to all paths on all DataCore Servers for any served Virtual Disk. While this may seem preferable, in configurations where there are significant path distances between servers (e.g. across remote sites) or where the speed of links between remote servers is significantly slower than links between local servers on the same site, it is possible that longer I/O wait times are encountered between paths to remote servers which will then cause additional delays for I/O within the same request but sent via paths to local servers resulting in overall significant I/O latency. Testing is strongly advised whenever the All setting is required. Please see Appendix A Notes on Preferred Server and Preferred Path settings on page 32 for a more details explanation when using the All setting. Page 17

18 Configuring the Fixed Path Selection Policy Use the SATP 'VMW_SATP_ALUA' with the claim option 'tpgs_on'. VMware Path Selection Policies The Fixed PSP can be configured using either the 'default' SATP type or by configuring a custom SATP rule. Using the default SATP type It is possible to use VMware ESXi s built-in, generic 'VMW_SATP_ALUA' rule VMW_SATP_ALUA system tpgs_on Any array with ALUA support Using a custom SATP rule To create a custom rule run the following command on the ESXi Host s console esxcli storage nmp satp rule add -V DataCore -M "Virtual Disk" -s VMW_SATP_ALUA -c tpgs_on -P VMW_PSP_FIXED Verify the custom rule has been set correctly esxcli storage nmp satp rule list -s VMW_SATP_ALUA grep DataCore The response should look like something this (1) VMW_SATP_ALUA DataCore Virtual Disk user tpgs_on VMW_PSP_FIXED This custom SATP rule can be used for all Virtual Disks from any DataCore Servers when using the Fixed PSP. Note: Fixed PSP is only qualified with the ALUA option enabled on the VMware Host from within the DataCore Server's Console. See Changing multipath or ALUA support settings for hosts from the SANsymphony Help: 1 This example is taken from VMware ESXi version 5.5 Page 18

19 VMware Path Selection Policies Which Preferred Server setting on the DataCore Server should I use with Fixed PSP? Unlike Round Robin, where DataCore recommend (initially) to not use the 'All' Preferred Server setting, when using the Fixed PSP the 'All' setting is mandatory and no other Preferred Server setting is qualified. This is because the Fixed PSP always required an 'Active Optimized' path to failover or failback to for it to work as expected. Note: The Fixed PSP will not send IO to all 'Active Optimized' paths like the Round Robin PSP does. The actual 'active' path used by the VMware Host that is using the Fixed PSP is configured on the ESX Host directly and is not controlled by the DataCore Server. Please refer to VMware's own documentation on how to configure the 'active' path when using the Fixed PSP. Please see Appendix A Notes on Preferred Server and Preferred Path settings on page 32 for a more details explanation when using the All setting. Also see the Preferred Servers section from the SANsymphony Help: Page 19

20 VMware Path Selection Policies Configuring the Most Recently Used Path Selection Policy Use the SATP 'VMW_SATP_DEFAULT_AA' with no claim option set. The Fixed PSP can be configured using either the 'default' SATP type or by configuring a custom SATP rule. Using the default SATP type It is possible to use VMware ESXi s built-in, generic 'VMW_SATP_DEFAULT_AA' rule VMW_SATP_DEFAULT_AA fc system Fibre Channel Devices Using a custom SATP rule To create a custom rule run the following command on the ESXi Host s console esxcli storage nmp satp rule add -V DataCore -M "Virtual Disk" -s VMW_SATP_DEFAULT_AA -P VMW_PSP_MRU Verify the custom rule has been set correctly esxcli storage nmp satp rule list -s VMW_SATP_DEFAULT_AA grep DataCore The response should look like something this (6) VMW_SATP_DEFAULT_AA DataCore Virtual Disk user VMW_PSP_MRU This custom SATP rule can be used for all Virtual Disks from any DataCore Servers when using the Fixed PSP. Note: Most Recently Used is only qualified without the ALUA option enabled on the VMware Host from within the DataCore Server's Console. Se: Changing multipath or ALUA support settings for hosts from the SANsymphony Help: 6 This example is taken from VMware ESXi version 5.5 Page 20

21 VMware Path Selection Policies Which Preferred Server setting on the DataCore Server should I use with Most Recently Used? Because the ALUA option is not supported when using the Most Recently Used PSP it must never be enabled on the Host. The Preferred Server setting, which controls the ALUA state of a given path to a Host from a DataCore Server, will therefore be ignored by the Host. Note: The actual 'active' path used by the VMware Host that is using the Most Recently Used PSP is configured on the ESX Host directly and is not controlled by the DataCore Server. Please refer to VMware's own documentation on how to configure the 'active' path when using the Most Recently Used PSP. Also see the Preferred Servers section from the SANsymphony Help: Page 21

22 Known issues The following is intended to make DataCore Software customers aware of any issues that may affect performance, access or generally give unexpected results under certain conditions when VMware ESXi is used with SANsymphony. Some of the issues here have been found during DataCore s own testing but many others are issues reported by DataCore Software customers, where a specific problem had been identified and then subsequently resolved. DataCore cannot be held responsible for incorrect information regarding VMware products. No assumption should be made that DataCore has direct communication with VMware regarding the issues listed here and we always recommend that users contact the VMware directly to see if there are any updates or fixes since they were reported to us. For Known issues for DataCore s own Software products, please refer to the relevant DataCore Software Component s release notes. Page 22

23 ESXi 6.x (includes 6.0.x and 6.5.x) When using Converged Network Adaptors When using QLogic's Dual-Port, 10Gbps Ethernet-to-PCIe Converged Network Adaptor Disable both the adaptor's BIOS and the 'Select a LUN to Boot from' option. When connecting ESXi Hosts to DataCore Servers After upgrading to VMware ESXi 6.0 Update 3 ESX paths will only report as 'Active'. No paths will report as 'Active (I/O)' - regardless of the Path Selection Policy. VMware have verified this as a cosmetic bug in ESXi that does not affect IO of either the ESX Hosts or VMs and that their engineering team is currently working on a solution. Also see: Example before ESXi 6.0 Update 3: Example after ESXi 6.0 Update 3: Note: Use the ESXi 'esxtop' command (e.g. using the 'd' or 'u' switches) to show actual activity on the expected paths and/or devices. Page 23

24 Known issues ESX 6.x Sharing the same physical connection for Host Front end and DataCore Mirror Ports may result in unexpected behavior when a failure occurs on that physical connection. When Virtual Disks are served to two or more ESX Hosts; make sure that all Host to DataCore Server connections i.e. Front End Ports - do not also share the same 'physical' connection as any DataCore Server to DataCore Server connections i.e. Mirror ports. For example using a single physical 'Inter Switch Link' between the DataCore Servers across two site locations where a Virtual Disk is also served to an ESX Host over the same Inter Switch Link. Should a failure occur on that single 'physical' connection, this will cause both the Mirror I/O (between the DataCore Servers) and the Host I/O (between the Host and the DataCore Server) to fail at the same time. Even though the DataCore Server sends a 'correct' SCSI notification to the ESX Hosts - LUN_NOT_AVAILABLE - ESX will continue to try to access all Virtual Disks even though DataCore would normally expected the ESX Host to report either a 'Permanent Device Loss' (PDL) or an 'All-Paths-Down' (APD) event. No attempt to 'failover' (ESX HA) or move the VM (ESX Fault Tolerant) will be attempted and the ESX Host will lose access to the Virtual Disk. DataCore cannot support a configuration where ESX Hosts are served Virtual Disks over the same Physical Link as the DataCore Servers send their Mirror I/O. ESXi hosts need to perform a rescan whenever Virtual Disks are unserved See and Without a rescan on the Host, ESXi will continue to send SCSI commands to DataCore Server Frontend Ports for LUNs that are no longer served. This causes the DataCore Server to have to send back an appropriate ILLEGAL_REQUEST SCSI response each time the missing LUN is probed for by the Host. In extreme cases, when large numbers of Virtual Disks are unserved, the amount of SCSI commands generated by this send-and-response will significantly affect performance any Host that is using the Front End Port(s) for existing Virtual Disks ESXi hosts experience degraded IO performance on iscsi network when Delayed ACK is 'enabled' on ESXi its software iscsi initiator. See for more specific information and how to disable the 'Delayed ACK' feature on ESXi Hosts. A reboot of the ESXi Host will be required. Storage PDL responses may not trigger path failover in vsphere and 6.0 Update 1. This has now been fixed by VMware. See VHBAs and other PCI devices may stop responding when using Interrupt Remapping. See Under heavy load the VMFS heartbeat may fail with 'false' ATS miscompare message. The ESXi VMFS 'heartbeat' used to use normal 'SCSI reads and writes' to perform its function. A change in the heartbeat method - released in ESXi 5.5 Update 2 and ESXi 6.0 uses ESXi's VAAI ATS commands instead directly to the storage array (i.e. the DataCore Server). DataCore Servers do not require (and so do not support) these ATS commands. DataCore therefore recommend disabling the VAAI ATS heartbeat setting see If your ESXi Hosts are connected to other storage arrays contact VMware to see if it is safe to disable this setting for these arrays. Page 24

25 Known issues ESX 6.x ESX Hosts when not using VAAI's Atomic Test and Set / Compare and Write Significant numbers of VMs running on the same Virtual Disk may result in excessive SCSI reservation requests leading to reservation conflicts between Hosts sharing the Virtual Disk which may lead to increased I/O latency. Reduce the number of running Virtual Machines on a single Virtual Disk and that ESX Hosts with the closest IO path to the DataCore Server all access the same, shared Virtual Disk as this will also help to reduce the potential for excessive SCSI Reservation conflicts. Also see: Applies only to SANsymphony versions 10.0 PSP 6 Update 5 and earlier ESX Hosts with IP addresses that share the same IQN connecting to the same DataCore Server Front-end port is not supported (this also includes ESXi 'Port Binding'). The Front End port will only accept the first connection from a given IQN that attempts to login to it where a unique ISCSI Session ID (ISID) is created for that connection. All subsequent connections that then come from a different NIC that happens to share the same IQN as the first login, will causes a ISID conflict and will be rejected by the DataCore Server. After that, no further iscsi logins will be possible for this IQN. This may cause unexpected disconnects between the Host and the DataCore Server for those connections. It is important to note that if the first successful connection gets disconnected for any reason (e.g. by a SCSI reset ), then one of the other NICs - sharing the same IQN - may re-attempt a login and, if successful, will take the session for itself. This will now block the previously-connected NIC from being able to re-connect and it will now remain disconnected. See the following pages for example of qualified and not-supported configurations: Example 1 A qualified configuration An ESX Host (ESX1) has four different Network Interfaces; each with its own IP address but all with the same IQN: (iqn.esx1) (iqn.esx1) (iqn.esx1) (iqn.esx1) There are, in this example, two DataCore Servers each with two Front-end ports with their own corresponding IP adresses and IQNs: (iqn.dcs1-1) (iqn.dcs1-2) (iqn.dcs2-1) (iqn.dcs2-2) Each Network Interface of the ESX Host should connect to a separate Front-end Port on both DataCore Servers; (iqn.esx1) ISCSI Fabric (iqn.dcs1-1) (iqn.esx1) ISCSI Fabric (iqn.dcs1-2) (iqn.esx1) ISCSI Fabric (iqn.dcs2-1) Page 25

26 Known issues ESX 6.x (iqn.esx1) ISCSI Fabric (iqn.dcs2-2) Also note that this kind of set up will make things simpler to manage and troubleshoot if connection problems occur in the future. There is no case in the above example where the other Network Interface on ESX1 is also trying to connect to the other Front-end port on the same DataCore Server (i.e. there are no multiple ISCSI session connections). Example 2 An non-supported configuration Using the same values as the qualified example above; (iqn.esx1) ISCSI Fabric (iqn.dcs1-1) (iqn.esx1) ISCSI Fabric (iqn.dcs1-2) (iqn.esx1) ISCSI Fabric (iqn.dcs1-1) (iqn.esx1) ISCSI Fabric (iqn.dcs2-2) In this case, both of the Network Interfaces from ESX1 have been configured to connect to the same Network Interface on DataCore Server 1 in this case iqn.dcs1-1 (highlighted in red) which will not work as expected. DataCore Server1 will accept only one of the connections and the other will be rejected; any subsequent interruption over that iscsi connection may then result in either of the two ESX Network Interfaces being able to (re)connect to iqn.dcs1-1, forcing the other ESX connection to be rejected. In other words, there is no guarantee that the ESX1 connection that was previously logged into iqn.dcs1-1 will be able to reconnect if disconnected for any reason and if the other Network Interface logs in before it. A solution in this case may be Teaming the NICs together as the teamed connections will then only have a single IP address and be recognized by the DataCore Server as a single NIC. Page 26

27 Known issues ESX 6.x When running a Microsoft Windows Cluster in an ESX Virtual Machine Running Microsoft Cluster Services in a Virtual Machine on Virtual Disks that have more than one Front End mapping to each DataCore Server may cause unexpected loss of access. A fix is available from VMware. See for more information. Unable to access filesystem for MSCS cluster nodes after vmotion. This has now been fixed by VMware. See The SCSI-3 Persistent Reserve tests fail for Windows 2012 Microsoft Clusters running in VMware ESXi Virtual Machines. This is expected. See specifically read the 'additional notes' (under the section 'VMware vsphere support for running Microsoft clustered configurations'). ESXi/ESX hosts with visibility to RDM LUNs being used by MSCS nodes with RDMs may take a long time to start or during LUN rescan. See Page 27

28 ESXi 5.x (includes 5.0.x, 5.1.x and 5.5.x) When using Converged Network Adaptors When using QLogic's Dual-Port, 10Gbps Ethernet-to-PCIe Converged Network Adaptor Disable both the adaptor's BIOS and the 'Select a LUN to Boot from' option. When connecting ESXi Hosts to DataCore Servers Sharing the same physical connection for Host Front end and DataCore Mirror Ports may result in unexpected behavior when a failure occurs on that physical connection. When Virtual Disks are served to two or more ESX Hosts; make sure that all Host to DataCore Server connections i.e. Front End Ports - do not also share the same 'physical' connection as any DataCore Server to DataCore Server connections i.e. Mirror ports. For example using a single physical 'Inter Switch Link' between the DataCore Servers across two site locations where a Virtual Disk is also served to an ESX Host over the same Inter Switch Link. Should a failure occur on that single 'physical' connection, this will cause both the Mirror I/O (between the DataCore Servers) and the Host I/O (between the Host and the DataCore Server) to fail at the same time. Even though the DataCore Server sends a 'correct' SCSI notification to the ESX Hosts - LUN_NOT_AVAILABLE - ESX will continue to try to access all Virtual Disks even though DataCore would normally expected the ESX Host to report either a 'Permanent Device Loss' (PDL) or an 'All-Paths-Down' (APD) event. No attempt to 'failover' (ESX HA) or move the VM (ESX Fault Tolerant) will be attempted and the ESX Host will lose access to the Virtual Disk. DataCore cannot support a configuration where ESX Hosts are served Virtual Disks over the same Physical Link as the DataCore Servers send their Mirror I/O. ESXi hosts experience degraded IO performance on iscsi network when Delayed ACK is 'enabled' on ESXi its software iscsi initiator. See for more specific information and how to disable the 'Delayed ACK' feature on ESXi Hosts. A reboot of the ESXi Host will be required. ESXi hosts need to perform a rescan whenever Virtual Disks are unserved See and Without a rescan on the Host, ESXi will continue to send SCSI commands to DataCore Server Frontend Ports for LUNs that are no longer served. This causes the DataCore Server to have to send back an appropriate ILLEGAL_REQUEST SCSI response each time the missing LUN is probed for by the Host. In extreme cases, when large numbers of Virtual Disks are unserved, the amount of SCSI commands generated by this send-and-response will significantly affect performance any Host that is using the Front End Port(s) for existing Virtual Disks VHBAs and other PCI devices may stop responding when using Interrupt Remapping. See Page 28

29 Known issues ESX 5.x Under heavy load the VMFS heartbeat may fail with 'false' ATS miscompare message. The ESXi VMFS 'heartbeat' used to use normal 'SCSI reads and writes' to perform its function. A change in the heartbeat method - released in ESXi 5.5 Update 2 and ESXi 6.0 uses ESXi's VAAI ATS commands instead directly to the storage array (i.e. the DataCore Server). DataCore Servers do not require (and so do not support) these ATS commands.datacore therefore recommend disabling the VAAI ATS heartbeat setting see If your Hosts are connected to other storage arrays contact VMware to see if it is safe to disable this setting for these arrays. ESX Hosts when not using VAAI's Atomic Test and Set / Compare and Write Significant numbers of VMs running on the same Virtual Disk may result in excessive SCSI reservation requests leading to reservation conflicts between Hosts sharing the Virtual Disk which may lead to increased I/O latency. Reduce the number of running Virtual Machines on a single Virtual Disk and that ESX Hosts with the closest IO path to the DataCore Server all access the same, shared Virtual Disk as this will also help to reduce the potential for excessive SCSI Reservation conflicts. Also see: Applies only to SANsymphony versions 10.0 PSP 6 Update 5 and earlier ESX Hosts with IP addresses that share the same IQN connecting to the same DataCore Server Front-end port is not supported (this also includes ESXi 'Port Binding'). The Front End port will only accept the first connection from a given IQN that attempts to login to it where a unique ISCSI Session ID (ISID) is created for that connection. All subsequent connections that then come from a different NIC that happens to share the same IQN as the first login, will causes a ISID conflict and will be rejected by the DataCore Server. After that, no further iscsi logins will be possible for this IQN. This may cause unexpected disconnects between the Host and the DataCore Server for those connections. It is important to note that if the first successful connection gets disconnected for any reason (e.g. by a SCSI reset ), then one of the other NICs - sharing the same IQN - may re-attempt a login and, if successful, will take the session for itself. This will now block the previously-connected NIC from being able to re-connect and it will now remain disconnected. See the following pages for example of qualified and not-supported configurations: Example 1 A qualified configuration An ESX Host (ESX1) has four different Network Interfaces; each with its own IP address but all with the same IQN: (iqn.esx1) (iqn.esx1) (iqn.esx1) (iqn.esx1) There are, in this example, two DataCore Servers each with two Front-end ports with their own corresponding IP adresses and IQNs: (iqn.dcs1-1) (iqn.dcs1-2) Page 29

30 Known issues ESX 5.x (iqn.dcs2-1) (iqn.dcs2-2) Each Network Interface of the ESX Host should connect to a separate Front-end Port on both DataCore Servers; (iqn.esx1) ISCSI Fabric (iqn.dcs1-1) (iqn.esx1) ISCSI Fabric (iqn.dcs1-2) (iqn.esx1) ISCSI Fabric (iqn.dcs2-1) (iqn.esx1) ISCSI Fabric (iqn.dcs2-2) Also note that this kind of set up will make things simpler to manage and troubleshoot if connection problems occur in the future. There is no case in the above example where the other Network Interface on ESX1 is also trying to connect to the other Front-end port on the same DataCore Server (i.e. there are no multiple ISCSI session connections). Example 2 An non-supported configuration Using the same values as the qualified example above; (iqn.esx1) ISCSI Fabric (iqn.dcs1-1) (iqn.esx1) ISCSI Fabric (iqn.dcs1-2) (iqn.esx1) ISCSI Fabric (iqn.dcs1-1) (iqn.esx1) ISCSI Fabric (iqn.dcs2-2) In this case, both of the Network Interfaces from ESX1 have been configured to connect to the same Network Interface on DataCore Server 1 in this case iqn.dcs1-1 (highlighted in red) which will not work as expected. DataCore Server1 will accept only one of the connections and the other will be rejected; any subsequent interruption over that iscsi connection may then result in either of the two ESX Network Interfaces being able to (re)connect to iqn.dcs1-1, forcing the other ESX connection to be rejected. In other words, there is no guarantee that the ESX1 connection that was previously logged into iqn.dcs1-1 will be able to reconnect if disconnected for any reason and if the other Network Interface logs in before it. A solution in this case may be Teaming the NICs together as the teamed connections will then only have a single IP address and be recognized by the DataCore Server as a single NIC. Page 30

31 Known issues ESX 5.x When running a Microsoft Windows Cluster in an ESX Virtual Machine Running Microsoft Cluster Services in a Virtual Machine on Virtual Disks that have more than one Front End mapping to each DataCore Server may cause unexpected loss of access. A fix is available from VMware. See for more information. The SCSI-3 Persistent Reserve tests fail for Windows 2012 Microsoft Clusters running in VMware ESXi Virtual Machines. This is expected. See specifically read the 'additional notes' (under the section 'VMware vsphere support for running Microsoft clustered configurations'). ESXi/ESX hosts with visibility to RDM LUNs being used by MSCS nodes with RDMs may take a long time to start or during LUN rescan. See Page 31

32 Appendix A Preferred Server & Preferred Path settings See the Preferred Servers and Preferred Paths sections from the SANsymphony Help: Without ALUA enabled If Hosts are registered without ALUA support, the Preferred Server and Preferred Path settings will serve no function. All DataCore Servers and their respective Front End (FE) paths are considered equal. It is up to the Host s own Operating System or Failover Software to determine which DataCore Server is its preferred server. With ALUA enabled Setting the Preferred Server to Auto (or an explicit DataCore Server), determines the DataCore Server that is designated Active Optimized for Host IO. The other DataCore Server is designated Active Non-Optimized. If for any reason the Storage Source on the preferred DataCore Server becomes unavailable, and the Host Access for the Virtual Disk is set to Offline or Disabled, then the other DataCore Server will be designated the Active Optimized side. The Host will be notified by both DataCore Servers that there has been an ALUA state change, forcing the Host to re-check the ALUA state of both DataCore Servers and act accordingly. If the Storage Source on the preferred DataCore Server becomes unavailable but the Host Access for the Virtual Disk remains Read/Write, for example if only the Storage behind the DataCore Server is unavailable but the FE and MR paths are all connected or if the Host physically becomes disconnected from the preferred DataCore Server (e.g. Fibre Channel or iscsi Cable failure) then the ALUA state will not change for the remaining, Active Nonoptimized side. However, in this case, the DataCore Server will not prevent access to the Host nor will it change the way READ or WRITE IO is handled compared to the Active Optimized side, but the Host will still register this DataCore Server s Paths as Active Non-Optimized which may (or may not) affect how the Host behaves generally. Page 32

33 Appendix A - Preferred Server & Preferred Path settings In the case where the Preferred Server is set to All, then both DataCore Servers are designated Active Optimized for Host IO. All IO requests from a Host will use all Paths to all DataCore Servers equally, regardless of the distance that the IO has to travel to the DataCore Server. For this reason, the All setting is not normally recommended. If a Host has to send a WRITE IO to a remote DataCore Server (where the IO Path is significantly distant compared to the other local DataCore Server), then the WAIT times accrued by having to send the IO not only across the SAN to the remote DataCore Server, but for the remote DataCore Server to mirror back to the local DataCore Server and then for the mirror write to be acknowledged from the local DataCore Server to the remote DataCore Server and finally for the acknowledgement to be sent to the Host back across the SAN, can be significant. The benefits of being able to use all Paths to all DataCore Servers for all Virtual Disks are not always clear cut. Testing is advised. For Preferred Path settings it is stated in the SANsymphony Help: A preferred front-end path setting can also be set manually for a particular virtual disk. In this case, the manual setting for a virtual disk overrides the preferred path created by the preferred server setting for the host. So for example, if the Preferred Server is designated as DataCore Server A and the Preferred Paths are designated as DataCore Server B, then DataCore Server B will be the Active Optimized Side not DataCore Server A. In a two-node Server group there is usually nothing to be gained by making the Preferred Path setting different to the Preferred Server setting and it may also cause confusion when trying to diagnose path problems, or when redesigning your DataCore SAN with regard to Host IO Paths. For Server Groups that have three or more DataCore Servers, and where one (or more) of these DataCore Servers shares Mirror Paths between other DataCore Servers setting the Preferred Path makes more sense. So for example, DataCore Server A has two mirrored Virtual Disks, one with DataCore Server B, and one with DataCore Server C and DataCore Server B also has a mirrored Virtual Disk with DataCore Server C then using just the Preferred Server setting to designate the Active Optimized side for the Host s Virtual Disks becomes more complicated. In this case the Preferred Path setting can be used to override the Preferred Server setting for a much more granular level of control. Page 33