The DataCore Server. Best Practice Guidelines. August The Data Infrastructure Software Company

Transcription

1 The DataCore Server August 2017 The Data Infrastructure Software Company

2 Table of contents Changes made to this document 4 Overview 5 Which versions of SANsymphony does this apply to? 6 High level design objectives 6 The DataCore Server 8 BIOS 8 CPUs 9 Power 11 System Memory 11 A summary of the recommendations for this section 13 Fibre Channel connections 14 Multi-ported vs Single-ported adaptors 14 ISCSI connections 15 Multi-ported vs Single-ported adaptors 15 Link aggregation and SCSI Load Balancing 15 NIC teaming 15 Spanning Tree Protocols (STP/RSTP/MSTP) 15 A summary of the recommendations for this section 17 RAID controllers and Storage Arrays 18 A summary of the recommendations for this section 19 The DataCore Server's operating system 20 Windows system settings 20 Windows service packs, updates, security and hot fixes 22 Installing third-party software on the DataCore Server 23 Upgrading (or reinstalling) the Windows OS 24 Summary of software configuration recommendations 25 Upgrading SANsymphony 26 TCP/IP Networking 27 The SANsymphony Console 27 The SANsymphony Server Group 28 Other IP-based protocols 29 NIC teaming 29 Windows Hosts file / DNS settings 30 Firewalls and network security 30 A summary of the recommendations for this section 31 SANsymphony features 32 Disk Pools 32 Disk Pool settings 32 The number of physical disks in a Disk Pool 34 Configuring RAID sets for use in a Disk Pool 35 Auto-Tiering considerations 35 A summary of recommendations for Disk Pools 36 Replication 37 Page 2

3 Replication settings 37 Data Compression 37 Transfer Priorities 37 The Replication buffer 38 Replication connections 39 Other Replication considerations 41 A summary of recommendations for Replication 42 Snapshot 43 Snapshot settings 43 Considerations for Disk Pools 43 Considerations for 'Preferred' DataCore Servers 43 Number of Snapshots per Virtual Disk 44 Previous Changes 45 Page 3

4 Changes made to this document The most recent version of this document is available from here: All changes since September 2015 Added New sections The DataCore Server - BIOS A new section has been added - this information was previously documented in DataCore Support's FAQ 1467 Disk Pools A new section has been added detailing best practices when configuring Disk Pools. Snapshot A new section has been added detailing best practices when configuring Snapshot. Updated The DataCore Server - CPU Many parts of this section have been updated and expanded including the specifics regarding recommended numbers of CPUs in both Physical and Virtual DataCore Servers. The information in this document has been re-organized (since the last edition) into more defined 'hardware' and 'software' sections to make finding specific information simpler and some of the sections have been re-titled. This document has been reviewed for SANsymphony 10.0 PSP 6. All previous changes Please see page 42. Page 4

5 Overview Each DataCore implementation will always be unique and giving advice that applies to every installation is therefore difficult. The information in this document should be considered as a set of general guidelines and not necessarily strict rules. DataCore cannot guarantee that following these best practices will result in a perfect solution there are simply too many separate (and disparate) components that make up a complete SANsymphony installation, most of which are out of DataCore Software s control but these guidelines will significantly increase the likelihood of a more secure, stable and high-performing SANsymphony installation. This guide assumes that DataCore-specific terms - e.g. Virtual Disk, Disk Pool, CDP and Replication etc. including their respective functions will be understood. It does not replace DataCore technical training nor does it remove any need to use a DataCore Authorized Training Partner and assumes fundamental knowledge of Microsoft Windows and the SANsymphony suite of products. Page 5

6 Which versions of SANsymphony does this apply to? This document applies to SANsymphony-V 9.0 PSP4 Update 4 [1] and all versions of SANsymphony 10.x. Any recommendations that are specific to a particular version of SANsymphony will be clearly indicated within the text. High level design objectives The design objective of any SANsymphony configuration is always an end-to-end process from the user to the data and in terms of availability, redundancy and performance these cannot only be considered individually as are not isolated functions but integrated parts of the entire infrastructure. The following information provides some high level recommendations which are often overlooked or forgotten about. Best Practices are not Pre-requisites Some of the Best Practices listed here may not be applicable to your installation or cannot be applied because of the physical limitations of your installation for example, it may not be possible to install any more CPUs or perhaps add more PCIe Adaptors or maybe your network infrastructure limits the number of connections possible between DataCore Servers and so on. Therefore, each set of recommendations is accompanied by a more detailed explanation so that when there is a situation where a best practice cannot be followed, there is an understanding of what and how that may limit your SANsymphony installation. Also see DataCore Training How to become a DataCore Certified Implementation Engineer Avoid complexity Overly complicated environments may appear to cover all possible situations, but the more complex the design, the more likely unforeseen errors will occur and can make the system difficult to maintain and support especially when adding to it in future. A simple approach is recommended whenever possible. 1 SANsymphony-V 8.x and all versions of SANsymphony-V 9.x before PSP 4 Update 4 are now end of life. Page 6

7 High level design objectives Avoid single points of failure Dependencies between major components can impact a whole environment such as switches, fabrics, storage, etc. any of which can become a single point of failure. Distribute components across separate racks, in separate rooms, buildings and even across sites. Keep storage components away from the public network. Avoid connecting redundant devices to the same power circuit. Use redundant UPS protected power sources and connect every device to it. A UPS back-up does not help much if it fails to notify a Host to shut down because that Host s management LAN switch itself was not connected to the UPS backed power circuit. Use redundant network infrastructures and protocols where the failure of one component does not make access unavailable to others. Do not forget environmental components. A failed air conditioner may collapse all redundant systems located in the same datacenter. Rooms on the same floor may be affected by a single burst water pipe (even though they are technically separated from each other). Monitoring and event notification Highly available systems keep services alive even if half of the environment has failed but these situations must always be recognized and responded to in a timely manner so that they can be fixed as soon as possible by the personnel responsible to avoid further problems. Documentation Document the environment properly; keeping it up-to-date and accessible. Establish 'shared knowledge' between at least two people who have been trained and are familiar with all areas of the infrastructure. User access Make sure that the difference between a 'normal' server and a DataCore Server is understood. A DataCore Server should only be operated by a trained technician. Page 7

8 The DataCore Server BIOS Many server manufacturers often have their own 'Low Latency Tuning' recommendations. DataCore always recommend follow those recommendations settings unless they contradict the following DataCore-recommended BIOS settings below. Note that server manufacturers may use different names or terminology for many of the settings listed below. Always consult the vendor's own server documentation if you are unsure. When running SANsymphony in a Virtual Machine Set the all hypervisors server BIOS settings 'as if' it were a 'physical' DataCore Server according to the following recommendations. Collaborative power control This should always be disabled where possible as it can interfere or override many of the BIOS settings listed in this section. This includes disabling features such as 'Power Saving' (or 'Low Power mode') and 'Overclocking' (sometimes known as 'Speedstepping'). Command per Clock (CPC Override/Mask) This should be enabled as it can increase the overall speed of memory transfer. Hemisphere Mode Hemisphere mode can improve memory cache access both to and from the memory controllers but the server hardware usually has to have memory modules of the same specification to take advantage of this setting. This should be set this to Automatic. Intel CPU specific options Turbo Boost Intel Turbo Boost can, in extreme cases, over-clock a CPU leading to erratic behavior that will impact the SANsymphony's I/O Scheduler. Turbo Boots should always be disabled. Hyper-Threading For Intel CPUs manufactured after 2014, Hyper-Threading should be enabled. This will help increase the potential number of threads that can be created for SANsymphony's I/O Scheduler. For Intel CPUs manufactured before 2014 Hyper-Threading should be disabled. In testing DataCore discovered that for older CPUs Hyper-Threading could cause the SANsymphony I/O Scheduler to behave erratically Page 8

9 Hardware The DataCore Server Non-Uniform Memory Access (NUMA) Snoop options Enable the Early Snoop option (if available). Cluster options Disable all NUMA Cluster option settings. Power saving settings (C-States) General This will put a CPU into a low power state when idle. SANsymphony relies on all CPUs being available all of the time to be able to continuously poll for any Target IO received by the DataCore Server and not have to wait for an interrupt command (IRQ) to take the CPU out of the idle state, which will cause an unnecessary delay. Leaving C-states enabled may also increase memory latency. Disable all C-states (this is sometimes known as C0 State). Static High Set the power management settings to static high to disable any additional CPU power saving. CPUs Note: unless specified, a CPU in this section can refer to a single physical core, a single hyperthreaded core or a single virtual CPU (e.g. when running a DataCore Server inside a Virtual Machine). Type and Speed Any x64 processor (except for Intel s Itanium family) is supported for use in a DataCore Server but use server-class processors rather than those intended for workstations. DataCore Software has not found any significant performance differences between the CPU manufacturers for processors with similar architectures and frequencies. Even so, faster (higher frequency) CPUs are always preferred over slower one, as they can process more instructions per second. We also recommend using faster cores over more (slower) cores. Additional CPU Sockets may also be necessary to use all of the available PCIe/Memory-Sockets on the server s motherboard. Please consult your server vendor. What is the recommended number of CPUs in a DataCore Server? The DataCore Poller, DataCore Server Ports and Logical Processors The 'DataCore Poller' will attend to the need of all Server Ports in a DataCore Server that use a DataCore Driver (including Fibre Channel, iscsi and Loopback Ports regardless of their Port Role). Page 9

10 Hardware The DataCore Server While 'most' workloads will only ever require a few Pollers, SANsymphony will create additional Pollers, if necessary (and automatically) up to a maximum of one Poller per DataCore Port; but will always limit the number to an amount less than the number of Logical Processors listed in Windows' Task Manager. Requirements 2 Physical (or 4 Hyper-Threaded) CPUs as a prerequisite for the base SANsymphony Software 1 additional CPU for each fibre channel port or 3 for each pair of iscsi ports [1] that use the DataCore driver (regardless of the port s role). 1 additional CPU for both the Live and Performance Recording features. 2 additional CPUs for the Replication feature Example: For a DataCore Server with 4 Fibre Channel Ports (2 Mirror and 2 Backend Ports) and 2 iscsi connections (2 Front-end Ports) we would recommend 9 CPUs in total 9 physical CPUs or 5 with Hyper-Threading. I.e. 2 CPUs for the minimum requirement of SANsymphony; 1 CPU for each of the Fibre Channel Ports (for a total of 4) and 3 CPUs for the 2 iscsi Ports. If either of SANsymphony s Live or Performance Recording functionality was required then an additional CPU would be recommended. Hyper-Threaded CPUs Even if the DataCore Server is using Hyper-Threading, for performance considerations, the minimum requirement of a DataCore Server to have at least 2 CPUs should always mean 2 physical CPUs in the server and not 2 Hyper-threaded CPUs (i.e. just 1 physical CPU with the Hyper-threaded function enabled). Running Hyper-V alongside SANsymphony Also known as a DataCore s Virtual SAN solution - where only the Hosts will be running inside Virtual Machines - additional Physical CPUs may be needed by Hyper-V to be able to create enough virtual processors used by those Host Virtual Machines. Please consult Microsoft s Hyper-V documentation to see how many additional cores will be required. Running SANsymphony in a virtual machine The minimum number of [Virtual] CPUs for running a DataCore Server inside a Virtual Machine is 2 the same as running in a physical server - but the additional CPU calculations used for 1 ISCSI connections generally have a much larger overhead, when compared to fibre channel ports, due to the extra work required encapsulating and de-encapsulating of SCSI data to and from their IP packets. Page 10

11 Hardware The DataCore Server physical DataCore Servers (see the previous section above), cannot be applied to virtual DataCore Servers. There will always be contention from other vcpus on other Virtual Machines (i.e. that are being used for Hosts) and even if the Hypervisor could dedicate physical CPUs just for the use of the DataCore Server s virtual machine it would almost certainly come at a cost for those same Host virtual machines, which would be restricted to the remaining CPUs on the Hypervisor for all their workloads. Even if the same number of vcpus were to be created to match those of an equivalent physical DataCore Server, there is still no guarantee that all of these vcpus would be used at the same rate and throughput as a physical DataCore Server would with physical CPUs. Also see Installing a DataCore Server within a Virtual Machine Power Use redundant and uninterruptable power supplies (UPS). Also see UPS Support System Memory Amount The minimum amount of memory for a DataCore Server is 8GB. The recommended amount of memory for a DataCore Server depends entirely on the size and complexity of the SANsymphony configuration. Use the DataCore Server Memory Considerations document available from the Support Website to calculate the memory requirement for the type, size and complexity of the SANsymphony configuration, always allow for plans of future growth. Also see DataCore Server Memory Considerations Type To avoid any data integrity issues while I/O is being handled by the DataCore Server s own Cache, ECC Memory Modules should be used. Page 11

12 Hardware The DataCore Server If a Server CPU's uses NUMA architecture then all the physical memory modules should have the same specification. See the NUMA Group Size Optimization and Hemisphere Mode entries from the BIOS section on page 9 for more information. Page 12

13 Hardware The DataCore Server A summary of the recommendations for this section BIOS CPU Collaborative power control should be disabled. CPC Override/Mask should be disabled. Hemisphere Mode should be set to Automatic. Intel Turbo Boost should be disabled. Intel Hyper-Threading should be enabled. Power saving (C-states) should all be disabled. Static High should be enabled. NUMA Group Size Optimization/Node Interleaving should be enabled and set to Flat (if the option is available). Use server class processors. Always enable Hyper-Threading (if available). The recommended number of CPUs for a DataCore Server 2 Physical (or 4 Hyper-Threaded) CPUs as a prerequisite for the base SANsymphony Software 1 additional CPU for each fibre channel port or 3 for each pair of iscsi ports [ ] that use the DataCore driver (regardless of the port s role). 1 additional CPU for both the Live and Performance Recording features. 2 additional CPUs for the Replication feature When Hyper-V is implemented on the DataCore Server additional physical CPUs will be needed for the Hyper-V Virtual Machines configured as Hosts. Power Use redundant power supplies. Use an uninterruptable power supply (UPS). System Memory See DataCore Server Memory Considerations from the Support Website: Use ECC Memory. Enable CPC Settings. Page 13

14 Hardware Fibre Channel Connections Fibre Channel connections Multi-ported vs Single-ported adaptors Adapters used for Fibre Channel connections are usually available in either single, dual or quadport configurations and often there is little difference in performance capabilities when, for example, comparing two single-ported adapters with one dual-port adaptor or two dual-port adaptors with a one quad-port adapter [1]. There is however a significant redundancy implication when using single adapters with multiple ports on them as most hardware failures usually end up affecting all ports rather than just one. Using adapters that have a smaller number of ports on them minimizes the risk of multiple failures happening at the same time. Use independent switches for redundant fabrics; this also prevents network-loops making spanning tree protocols obsolete and simplifies the overall iscsi implementation. Also see Qualified Hardware Components Known Issues - Third Party Hardware and Software 1 This assumes that there are always an adequate number of PCIe Lanes available in the PCI Slot being used for the adapter. Please refer to your server hardware vendor s own documentation for this. Page 14

15 Hardware ISCSI Connections ISCSI connections Multi-ported vs Single-ported adaptors Adapters used for iscsi connections are usually available in either single, dual or quad-port configurations and often there is little difference in performance capabilities when, for example, comparing two single-ported adapters with one dual-port adaptor or two dual-port adaptors with a one quad-port adapter [1]. There is however a significant redundancy implication when using single adapters with multiple ports on them as most hardware failures usually end up affecting all ports rather than just one. Using adapters that have a smaller number of ports on them minimizes the risk of multiple failures happening at the same time. Link aggregation and SCSI Load Balancing For better performance use faster, separate network adaptors and links instead of teaming multiple, slower adaptors. For high availability use more, individual network connections and multipath I/O software than rely on either teaming or spanning tree protocols to manage redundancy. Just like Fibre Channel environments, that use independent switches for redundant fabrics, this also prevents network-loops making spanning tree protocols obsolete and simplifies the overall iscsi implementation. Fundamentally, SCSI load-balancing and failover functions are managed by Multipath I/O protocols [1] ; TCP/IP uses a completely different set of protocols for its own load-balancing and failover functions. When SCSI commands, managed by Multipath I/O protocols but carried by TCP/IP protocols are combined (i.e. iscsi), then interaction between the two protocols for the same function can lead to unexpected disconnections or even complete connection loss. NIC teaming NIC teaming is not recommended for iscsi connections as it adds more complexity (without any real gain in performance); and although teaming iscsi Targets - i.e. Front-end or Mirror ports - would increase the available bandwidth to that target, it still only allows a single target I/O queue rather than, for example, two, separate NICs which would allow two, independent target queues with the same overall bandwidth. Spanning Tree Protocols (STP/RSTP/MSTP) None of the Spanning Tree Protocols - STP, RSTP or MSTP are recommended on networks used for iscsi as they may cause unnecessary interruptions to I/O; for example, other, 1 Mirrored Virtual Disks that are configured to use multiple iscsi Mirror paths on the DataCore Server are, by default, autoconfigured to be managed by Microsoft s MPIO using the Round Robin with Subset Policy. Page 15

16 Hardware ISCSI Connections unrelated devices generating unexpected network-topology causing STP to re-route iscsi commands inappropriately or even blocking them completely from their intended target. Also see Qualified Hardware Components Known Issues - Third Party Hardware and Software Page 16

17 Hardware ISCSI Connections A summary of the recommendations for this section ISCSI connections Use many adapters with a smaller numbers of ports on them as opposed to fewer adapters with larger numbers of ports. Use faster, separate network adaptors instead of NIC teaming. Do not use NIC teaming or STP protocols with iscsi connections. Use more, individual network connections (with Multipath I/O software) to manage redundancy. Use independent network switches for redundant iscsi networks. Page 17

18 Hardware RAID controllers and Storage Arrays Configure the DataCore Server s own Boot Disk for redundancy; use RAID 1 (usually simpler, less complex to configure and has less overhead than RAID 5). RAID and Storage Array controllers used to manage physical disks in Disk Pools need to be able to handle I/O from multiple Hosts connected to the DataCore Server; so for high performance/low latency hardware, use independent buses or backplanes designed for heavy workloads ( workstation hardware is not usually designed for such workloads). A low-end RAID controller will deliver low-end performance. An integrated (onboard) RAID controller that is often supplied with the DataCore Server may only be sufficient to handle just the I/O expected for the boot drive. Controllers that have their own dedicated CPU and cache are capable of managing much higher I/O workloads and many more physical disks. Consult with your storage vendor about the appropriate controller to meet your expected demands. If both 'fast' and 'slow' disk types share the same disk controller in the storage array (e.g. an SSD sharing the same disk controller as a SAS RAID5 set), then the slower disks on that controller can hold up I/O to the faster disks. DataCore recommend is to have a separate disk controller for each differential of disk speed type. If there is no choice but to mix different disk speed types on the same disk controller - for example mixing SSD with SAS - then in this example make sure the SAS disk have 'no RAID' (or RAID0) configured and use SANsymphony's Disk Pool mirroring feature as this should be faster than hardware RAID mirroring. Also see Storage Array Guidelines for DataCore Servers Mirroring Disk Pools Qualified Hardware Components Known Issues - Third Party Hardware and Software Page 18

19 Hardware RAID controllers and Storage Arrays A summary of the recommendations for this section RAID controllers and Storage Arrays Use high performance/low latency hardware with independent buses or backplanes designed for heavy workloads ( workstation hardware is not usually designed for such workloads). Avoid using onboard disk controllers for anything other than the DataCore Servers own Boot Disks. Configure the DataCore Server s Boot Disk for RAID1. Use a separate controller for physical disks used in either Disk Pools or for the Replication buffer. If mixing fast and slow disks on the same controller then use Disk Pool mirroring instead of hardware RAID on the slower disks. Also see Storage Array Guidelines for DataCore Servers from the Support Website: Page 19

20 The DataCore Server's operating system Use non-oem versions of Microsoft Windows to avoid the installation of unnecessary, thirdparty services that will require extra system resources and so potentially interfere with SANsymphony. Also see Software Requirements Windows system settings Network Time Protocol (NTP) Synchronize the system clocks for all DataCore Servers in the same Server and Replication Groups. Although the DataCore Server s system clock has no effect on any Host, Mirror or Storage I/O operations, synchronizing them with each other within a Server Group will help avoid unexpected behavior for operations within SANsymphony that is, potentially, timesensitive. Examples include: SANsymphony Console Tasks that use a Scheduled Time trigger. Continuous Data Protection s retention time settings. Time-sensitive licenses (e.g. that contain fixed, expiration dates either for evaluation or migration purposes). It will also help stop misleading Configuration Conflict warnings between DataCore Servers in the same Server Group (e.g. after a planned DataCore Server reboot). It is also recommended to synchronize all host s system clocks as well as any SAN or Network switch hardware clocks (if applicable) with the DataCore Servers. This can be especially helpful when using DataCore s VSS on a host (where backup and restore operations may take place); but generally to help with any troubleshooting where a host s own system logs need to be checked against those of a DataCore Server. Many SAN events often occur over very short periods of time (e.g. Fibre Channel or ISCSI disconnect and reconnection issues between Hosts and DataCore Servers). Page 20

21 Software The DataCore Server's operating system Power Options Select the High Performance power plan under Control Panel\Hardware\Power Options. Startup and Recovery/System Failure The SANsymphony installer will by default enable the recommended setting automatically Kernel Memory Dump. No additional settings are required. See the section Virtual Memory/Page File on the following page for more information. Windows Error Reporting (WER) for user-mode dumps Enable WER and configure it for Full dumps on the DataCore Server this is not configured by default. User-mode dumps are especially useful to help analyze problems that occur for any SANsymphony Management or Windows Console issues (i.e. non-critical, Graphical User Interfaces) if they occur. Also see Collecting User-Mode Dumps Virtual Memory/Page File SANsymphony does not use the page file for any of its critical operations. No additional settings are required and the default page file settings should be used. The default size of the page file created by Windows is determined by the amount of Physical Memory installed in the DataCore Server along with the type of memory dump that is configured. The SANsymphony installer will change the memory dump type to Kernel Memory Dump ensure that if any crash analysis is required from the DataCore Server that the correct type of dump file is generated. A DataCore Server that has a small boot disk and large amounts of physical memory may end up with a Page File that fills the boot disk after the installation. In this case, it is still recommended to keep the Kernel Memory Dump setting but manually enter a custom value for the page file size as large as is practically possible (for your boot disk) by unchecking the Automatically manage paging file size for all drives option. Page 21

22 Software The DataCore Server's operating system Windows service packs, updates, security and hot fixes Never configure the DataCore Server Windows Update settings to Install updates automatically if applicable to your network use the Check for updates but let me choose when to install them option. Also see the section Windows service packs, updates, security and hot fixes below. Service packs that have passed qualification for use with SANsymphony can be found in the SANsymphony minimum requirements page. Never install a Windows service pack that has not been qualified on a DataCore Server. Because of the number of these that are released between full service packs, individual Windows updates and security fixes (available via the regular Windows Update Service) they are never qualified separately; however, DataCore Software always apply the latest updates or security fixes to all their development servers as soon as they become available. This means that unless explicitly listed in the Configuration Notes section of the SANsymphony Software release notes, we recommend that all Windows updates and security fixes that are currently available from Microsoft should be applied whenever possible. Hotfixes that are not distributed via WUS Occasionally, Microsoft will make available hotfixes that are not distributed via the normal Windows Update Service, and sometimes these are applicable to the DataCore Server and should be applied. A full list of all hotfixes that must be applied to a DataCore Server can be found on the SANsymphony minimum requirements web page. Hotfixes that are not listed on the SANsymphony minimum requirements web page should never be applied to the DataCore Server but if there is a specific requirement to install an unlisted hotfix (e.g. by Microsoft s own technical support or hardware vendor) please contact DataCore Customer Support for further advice. Also see SANsymphony s minimum requirements Software Downloads and Documentation - SANsymphony release notes Known Issues - Third-party Hardware and Software Page 22

23 Software The DataCore Server's operating system Software updates via Windows Update Service (WUS) for third-party drivers Sometimes, third-party vendor drivers (e.g. Fibre Channel drivers that are being used for Backend connections or NIC drivers used for iscsi connections) may be distributed via the Windows Update Service. Any that is known to cause problems when using SANsymphony will be listed in the Known Issues - Third-party Hardware and Software document. Installing third-party software on the DataCore Server It is recommended not to install third-party software on a DataCore Server. SANsymphony requires significant amounts of system memory as well as CPU processing; it will also prevent certain system devices (e.g. Disk devices) from being accessed by other software components that may be installed on the DataCore Server which may lead to unexpected errors from those other software components. The purpose of the DataCore Server should not be forgotten and trying to run the DataCore Server as a Domain Controller or as a Mail Server/Relay for example, as well as SANsymphony, must not be done as this will affect the overall performance and stability of the DataCore Server. DataCore recognize that certain types of third-party software are required to be able to integrate the DataCore Server onto the user s network. These include: Virus scanning applications UPS software agents The server vendor s own preferred hardware and software monitoring agents In these few cases, and as long as these applications or agents do not need exclusive access to components that SANsymphony needs to function correctly (i.e. Disk, Fibre Channel or iscsi devices), then it is possible to run these alongside SANsymphony. Always consult the third-party software vendor for any additional memory requirements their products may require and refer to the Known Issues - Third-party Hardware and Software document for any potential problems with certain types of third-party software that have already been found to cause issues or need additional configuration. DataCore Support may ask for third-party products to be removed in order to assist with Troubleshooting. Also see Changing Cache Size DataCore Server Memory Considerations Known Issues - Third-party Hardware and Software Page 23

24 Software The DataCore Server's operating system Qualified Software Components Upgrading (or reinstalling) the Windows OS Versions of Windows that have passed qualification for a specific version of SANsymphony will be listed in both the SANsymphony Software release notes and the SANsymphony minimum requirements page. Never upgrade in-place to a newer version of Windows operating system, for example upgrading from Windows 2008 to Windows 2012 or upgrading from Windows 2012 to Windows 2012 R2; even if the newer version is considered qualified by DataCore the upgrade will stop the existing SANsymphony installation from running. Instead of an in-place upgrade the DataCore Server s operating system must be installed as new. R2 versions of a particular Windows Operating System also need to be qualified for use on a DataCore Server. Any R2 versions of Windows that have passed qualification for a specific version of SANsymphony will be listed in both the SANsymphony Software release notes and the SANsymphony minimum requirements page. Also see How to reinstall or upgrade the DataCore Server's Windows Operating System SANsymphony s minimum requirements Software Downloads and Documentation - SANsymphony release notes Page 24

25 Software The DataCore Server's operating system Summary of software configuration recommendations The DataCore Server's operating system Use non-oem versions of Microsoft Windows. Windows system settings Synchronize the system clocks for all DataCore Servers in the same Server Group, any associated Replication Groups and Hosts. Use the High Performance power plan. Use Kernel Memory Dumps for System Failure settings later versions of SANsymphony set this automatically during installation. Configure Windows Error Reporting (WER) for Full user-mode dumps Use the default Page File settings. Windows service packs, updates, security and hot fixes Never configure Windows Update settings to Install updates automatically Only use service packs that have been qualified. Apply all updates and security fixes available from Microsoft including Cumulative Updates and Update Rollups. Always apply the Windows hot fixes listed in the SANsymphony minimum requirements web page. Do not apply hot fixes that are not included as part of a normal Windows Update Service and are not listed in the SANsymphony minimum requirements web page. Do not apply any Windows updates for (third-party) drivers to Fibre Channel HBAs using the DataCore driver. Installing third-party software on the DataCore Server It is recommended not to install third-party software on a DataCore Server. Upgrading (or reinstalling) the Windows Operating System Only use versions of the Windows operating system that have been qualified. Never upgrade in place newer versions of Windows operating system. Only use R2 versions of Windows that have been qualified. Never upgrade in place versions of Windows to its R2 equivalent. Page 25

26 Software Upgrading SANsymphony All the instructions and considerations for updating existing versions of SANsymphony or when upgrading to a newer, major version of SANsymphony are documented in the SANsymphony Software release notes. The SANsymphony Software release notes are available either as a separate download or come bundled with the SANsymphony software. Also see Software Downloads and Documentation - SANsymphony release notes Page 26

27 Software TCP/IP Networking SANsymphony s Console, the VMware vcenter Integration component Replication and Performance Recording function (when using a remote SQL Server) all use their own separate TCP/IP session. To avoid unnecessary network congestion and delay as well as losing more than one of these functions at once should any problems occur with one or more network interfaces, we recommend using a separate network connection for each function. The SANsymphony Console On the DataCore Server Even if the SANsymphony Console is used to log in to one of the other DataCore Servers in the group (i.e. an unelected node ) that other server will still connect directly to the controller node to make configuration changes or to display the information in its own SANsymphony Console. This means that all DataCore Servers in the same Server Group must have a routable TCP/IP connection to each other so that if the controller node moves to a different server, then the new controller node must also be able connect to all of the remaining DataCore Servers in the group [1]. On a Workstation Workstations which only have the SANsymphony Console component installed cannot become controller nodes and never directly send or receive configuration information for any Server Group they connect to. Just like an unelected node the workstation will connect to the controller node to make configuration changes or to display the information in its own SANsymphony Console (See Understanding the Controller Node concept on the previous page). This means that even if the workstation is on a separate network segment from the DataCore Servers (e.g. in a different vlan) it must still be able to send and receive TCP/IP traffic to and from all the DataCore Servers in that vlan. 1 Re-election of the controller node takes place if the node is shutdown or if it becomes unavailable on the network to the rest of the Server Group for any reason. Page 27

28 Software TCP/IP Networking The SANsymphony Server Group The 'Controller Node' concept Where a Server Group has two or more DataCore Servers in it, one of them will be designated as the controller node for the whole group. The controller node is responsible for managing what is displayed in the SANsymphony Console for all DataCore Servers in the Server Group for example; receiving status updates for the different objects in the configuration for those other DataCore Servers (e.g. Disk Pools, Virtual Disks and Ports etc.), including the posting of any Event messages for those same objects within the SANsymphony console. The controller node is also responsible for the management and propagation of any configuration changes made in the SANsymphony Console regardless of which DataCore Server s configuration is being modified, and makes sure that all other DataCore Servers in the Server Group always have the most recent and up-to-date changes. The election of which DataCore Server is to become the controller node is decided by the SANsymphony software automatically and whenever; A DataCore Server is removed from or added to the existing Server Group The existing controller node is shutdown The existing controller node becomes unreachable via the TCP/IP network to the rest of the Server Group (e.g. an IP Network outage). The decision on which DataCore Server becomes the controller node is decided automatically between all the Servers in the Group and cannot be manually configured. It is also important to understand that the controller node does not manage any Host, Mirror or Back-end I/O (i.e. in-band connections) for other DataCore Servers in the Server Group. In-band I/O is handled by each DataCore Server independently of the other Servers in the Server Group, regardless if it is the elected controller or not. Nor does it send or receive Replication data configured for another DataCore Server in the same Server Group, although it will manage all Replication configuration changes and Replication status updates regardless if it is the Source Replication Server or not. Page 28

29 Software TCP/IP Networking The Connection Interface setting Except for iscsi I/O, all other TCP/IP traffic sent and received by a DataCore Server is managed by the SANsymphony Connection Interface setting. This includes: When applying SANsymphony configuration updates to all servers in the same Server Group. Any UI updates while viewing the SANsymphony Console, including state changes and updates for all the different objects within the configuration (e.g. Disk Pools, Virtual Disks, Snapshots and Ports etc.). Configuration updates and state information to and from remote Replication Groups Configuration updates when using SANsymphony s VMware vcenter Integration component. SQL updates when using a remote SQL server for Performance Recording The Connection Interface s default setting ( All ) means that SANsymphony will use any available network interface on the DataCore Server for its host name resolution, this is determined by the Windows operating system and how it has been configured and connected to the existing network. It is possible to change this setting, and choose an explicit network interface (i.e. IP Address) to use for host name resolution instead, but this requires that the appropriate network connections and routing tables have been set up correctly and are in place. SANsymphony will not automatically retry other network connections if it cannot resolve to a hostname using an explicit interface. We recommend leaving the setting to All and use the appropriate Hosts file or DNS settings to control host name resolution. Other IP-based protocols While it is technically possible to share ISCSI I/O, Replication data and the SANsymphony Console s own inter-node traffic over the same TCP/IP connection, for performance as well as losing more than one of these functions at once, we recommend using dedicated and separate network interfaces for each iscsi port. NIC teaming Inter-node communication For all TCP/IP traffic where Multipath I/O protocols are not being used (i.e. non-iscsi traffic), we recommend to use NIC teaming to provide redundant network paths to other DataCore Servers. Page 29

30 Software TCP/IP Networking We also recommend that each NIC that is teamed is in its own separate network and that failover mode is used rather than load balancing as there is no specific performance requirement for Inter-node communication as the TCP/IP and using fail over mode means that configuring and managing the network connections and switches is simpler. It also makes troubleshooting any future connection problems easier as well. ISCSI connections See ISCSI Hardware on page on page 15 for more information about NIC teaming with iscsi. Spanning Tree Protocols (STP/RSTP/MSTP) See ISCSI Hardware on page on page 15 for more information about STP with iscsi. ISCSI connections See ISCSI Hardware on page on page 15 for more information iscsi. Windows Hosts file / DNS settings There is no preference for managing DataCore Server host name resolution between using the local Hosts file or DNS. Either method can be used. DataCore do recommend however using Host Name resolution over just using IP addresses as it is easier to manage any IP address changes that might occur, planned or unexpected, by being able to simply update any Hosts file or DNS entries instead of reconfiguring a Replication group or remote SQL server connection for Performance Recording (i.e. manually disconnecting and reconnecting), which is disruptive. When using a Hosts file, do not add any entries for the local DataCore Server but only for the remote DataCore Servers and do not add multiple, different entries for the same server (e.g. each entry has a different IP address and/or server name for the same server) as this will cause problems when trying to (re)establish network connections. Firewalls and network security The SANsymphony software installer will automatically reconfigure the DataCore Server s Windows firewall settings to allow it to be able to communicate with other DataCore Servers in the same Server or Replication groups. No additional action should be required. If using an external firewall solution or another method to secure the IP networks between servers then refer to the Windows Security Settings Disclosure for the full list of TCP Ports required by the DataCore Server and ensure that connections are allowed through. Also see Windows Security Settings Disclosure Page 30

31 Software TCP/IP Networking A summary of the recommendations for this section ISCSI connections See ISCSI Hardware on page on page 15 for more information iscsi. Other required TCP/IP connections Leave the DataCore Server s Connection Interface setting as default ( ALL ). Use either Hosts or DNS settings to control all host name resolution for the DataCore Server. Use a managed Hosts file (or DNS) instead of just using IP addresses. Never install a Windows service pack that has not been qualified. Any Windows updates and security fixes that are currently available from Microsoft s Windows Update Service should be applied whenever possible. For Firewalls and other network security requirements please refer to Windows Security Settings Disclosure via the online help: Webhelp/windows_security_settings_disclosure.htm TCP/IP Network Topology Ensure DataCore Servers in the same Server Group all have routable TCP/IP connections to each other at all times. Ensure Workstations that are only using the SANsymphony Console has a routable TCP/IP connection to all DataCore Servers in the Server Group at all times. If using SANsymphony s VMware vcenter Integration component, ensure the server running the vcenter has a routable TCP/IP connection to all DataCore Servers in the Server Group at all times. Do not route non-iscsi TCP/IP traffic over iscsi connections. Use dedicated and separate TCP/IP connections for each of the following: The SANsymphony Console Replication transfer Performance Recording when connecting to a remote SQL server. Use NIC teaming, in failover mode, for inter-node connection redundancy for non- ISCSI TCP/IP traffic with separate networks for each NIC. Use independent network switches for redundant TCP/IP networks used for inter-node communication between DataCore Servers. Page 31

32 Software SANsymphony features Disk Pools Also see Disk Pool settings The following section applies to all types of Disk Pools, including normal, shared, SMPA and Bulk Pools. The Disk Pool Catalog The Disk Pool Catalog is an index that is used to manage information about each Storage Allocation Unit's location, its allocation state and its relationship to the storage source it is allocated for within the Disk Pool. The Catalog is stored on one or more of the physical disks in the Disk Pool - also used for Storage Allocation Unit (SAU) data - but in a location that only the Disk Pool driver can access. Each DataCore Server's Disk Pool has its own, independent Catalog regardless if the Disk Pool is shared or not. Information held in the Catalog includes: Whether an SAU is allocated to a Storage Source or not The Physical Disk within the Disk Pool that an allocated SAU can be located The Storage Source an allocated SAU 'belongs' to The Virtual Disk's Logical Block Address (LBA) that the allocated SAU represents when accessed by a Host. Whenever an SAU is allocated, reclaimed or moved to a new physical disk within the same Disk Pool, the Catalog is updated. It is important that Catalog updates happen as fast as possible so as to not to interfere with other I/O within the Disk Pool; for example, if the Catalog is being updated for one SAU allocation and another Catalog update for a different SAU is required, then this other Catalog update will have to wait for a short time before its own index can be updated. This can be noticeable when a lot of SAUs need to be allocated all within a very short time; and while the Disk Pool will try to be as efficient as possible when handling multiple updates for multiple SAU, there is an additional overhead while the Catalog is updated for each new allocation before the I/O written to the SAU is considered complete. This can, in extreme cases, result in unexpected I/O latency during periods of significant SAU allocations. Page 32

33 Software Disk Pool recommendations Therefore we recommend that the Catalog be located on the fastest disk possible within the Disk Pool. Where is the Catalog located? The Catalog is always stored within 1GB of the start of a physical disk's LBA space and always on the first physical disk added to the Disk Pool. If a second physical disk is added to the Disk Pool, then a backup copy of the Catalog is stored on that second physical disk the Catalog on the first physical disk is then considered as the primary copy. Tier assignment of the physical disk has no influence on where the Catalog will be stored. Any further physical disks added to the Disk Pool will not be used to store further copies. If the physical disk that holds the primary Catalog is mirrored within the Disk Pool, then the physical disk used as the mirror will now hold the backup Catalog. If the backup Catalog was previously on another physical disk in the Disk Pool before the mirroring took place, then then this other (non-mirrored) physical disk will no longer hold the backup Catalog. There is only ever a maximum of two copies of the Catalog in a Disk Pool at any time. Also see: It is important that any Catalog update can happen as fast as possible to not interfere with other I/O within same the Disk Pool, as in extreme cases where a lot of sudden, intensive I/O from a Host requires a lot of Catalog updates, can end up causing increased I/O wait times for other SAU read and write requests for all Hosts that use the same Disk Pool. DataCore recommend therefore that all Disk Pools have 'dedicated' physical disks used just for storing the primary and secondary Disk Pool Catalogs and that these physical disks are as fast as possible. As the Catalog is located within the first 1GB of the physical disk used to store it and as there is a minimum Disk Pool requirement of any physical disk to have enough free space to allocate at least one SAU, that this 'dedicated' physical disk be 2GB in size; 1GB for the Catalog itself and 1GB for the largest SAU possible within the disk pool (see previous section on Storage Allocation Unit size in this chapter). How the Catalog location is managed during Disk Decommissioning If the physical disk that holds the primary copy of the Catalog is removed then the backup copy on the remaining second physical disk in the Disk Pool will be automatically 'promoted' to be the primary Catalog and, if available, a backup copy will be written to the 'third' physical disk added to the Disk Pool. If the physical disk that holds the backup copy of the Disk Pool is removed then a new backup copy of the Catalog will be written to the 'next available' physical disk in the Disk Pool. The location of the primary copy remains unchanged. Page 33