A TPC Standby Solution User's Guide

Transcription

1 A TPC Standby Solution User's Guide Achim Christ IBM Systems Lab Services & Training European Storage Competence Center Version February 2013 Copyright IBM Corporation, 2013

2 Revision History Version Author Date Comment 1.0 Achim Christ Initial Version Authors / Reviewers Name Organization Address Achim Christ IBM Systems Lab Services & Training achim.christ@de.ibm.com Walter Morton IBM Global Technology Services walter_morton@at.ibm.com Christian Bolik IBM TPC Development BOLIK@de.ibm.com Copyright IBM Corporation, 2013

3 Table of Contents Revision History... 2 Authors / Reviewers Disaster Recovery Considerations Stand-By Concept for TPC Server Requirements for TPC Server Stand-By Concept for Brocade CIM Agent Requirements for Brocade CIM Agent Step-By-Step Instructions Partition / LUN Overview Setup Procedure for TPC Server Failover / Failback Procedure for TPC Server Recovery Procedures for Disk Failures on TPC Server Setup Procedure for Brocade CIM Agent Failover / Failback Procedure for Brocade CIM Agent Test Documentation Used Hardware and Software Setup Procedure Test Documentation Failover / Failback Procedure Test Documentation Appendix Copyright IBM Corporation, 2013

4 1 Disaster Recovery Considerations Availability needs of more and more datacenter administrators require their TPC environment to be set up in a redundant way. TPC Standard Edition (SE) may or may not be used as a monitoring tool, only it may not have direct impact on disaster recovery capabilities of the managed devices (storage subsystems, switches and monitored servers). Nevertheless, since e.g. the collected performance data is considered business critical, it is necessary to assure continuous data collection from all monitored devices. In case of a component failure, failure of the supporting infrastructure (power drop, etc.) or a site disaster, data collection needs to continue as quickly as possible using stand-by components in a backup datacenter. To achieve this goal, a SAN-boot based redundancy concept was developed to allow fast switching of server hardware. The central TPC server (including database, etc.) is equipped with a Fiber Channel HBA to allow mirroring internal hard disks into the SAN infrastructure. Using traditional disaster recovery techniques such as subsystem mirroring (e.g. using IBM Metro Mirror / EMC SRDF replication), availability of the data can then be assured across sites. In case the productive TPC environment fails, a prepared stand-by environment can quickly resume productive operations without loss of (monitoring) data. In order to activate the stand-by environment, manual actions are necessary (reboot of stand-by server) a fully automated failover can not be achieved with this technique. This paper describes an exemplary realization of this concept, as implemented for multiple European clients. It serves the purpose of acting as a blueprint for developing tailored operational documents and guidelines. Please note that implementation of such a redundancy concept does not lift the requirement to implement a backup strategy, leveraging traditional techniques such as backup / archive to longterm (tape-) storage. Implementation of such concepts is not covered by the paper at hand please consult the TPC Advanced Topics Redbook (SG ) for an implementation example using TSM. Beside the TPC server itself, CIM agents are required to allow data collection from monitored devices and are thus considered part of the TPC solution. For DS8000 storage subsystems and Cisco fabric devices, CIM agents are embedded in the hardware no additional disaster recovery considerations need to be taken. For Brocade fabric devices, an external CIM agent needs to be implemented (a.k.a. Proxy model). Disaster recovery considerations of the Brocade CIM agent will be characterized separately. The following sections describe the redundancy concepts in detail and give step-by-step instructions for setup, failover- and failback procedures. Page 4 of 30 Copyright IBM Corporation, 2013

5 1.1 Stand-By Concept for TPC Server Both, the primary (active) and secondary (stand-by) TPC server are equipped with Dual-Port Fiber Channel HBAs. Both servers are dual-fabric attached and have SAN storage configured. The assigned SAN storage (LUN count and size) corresponds the layout of the local harddisks / partitions. This configuration allows mirroring (RAID 1) of local harddisks and SAN LUNs using regular operating system functions, in this case Windows Volume Manager. While the system partition(s) are mirrored between local harddisks and SAN LUNs, partitions used for the database repository are placed on SAN LUNs, only without an additional mirror on local harddisks. This allows for improved performance of the database (main performance influencer for overall TPC solution), as well as additional flexibility for future expansion of the required partitions. During normal operation, the secondary TPC server boots off a so-called 'Maintenance Partition'. The maintenance partition does not contain any productive (TPC-) software components but only serves the purpose of monitoring the stand-by hardware (e.g. using Tivoli / IBM Director). Static IP addresses are required for both, the primary and secondary TPC server. Those IP addresses need to be valid in both datacenters: in case of a failover, the secondary TPC server will boot off the primary server's partition (mirrored SAN disk), including all operating system (network-) settings and software stack. It will be configured to use the static IP address of the primary server this needs to be available (properly routed, etc.) in the backup datacenter as well... With this configuration, failures on local harddisks and failures of the SAN infrastructure are completely hidden assuring continuous data collection without manual intervention. Only after Page 5 of 30 Copyright IBM Corporation, 2013

6 failure of the complete server or in case of a site disaster, manual actions are necessary to reestablish data collection. In case of a failure of the primary TPC server or the primary datacenter, the following actions need to be taken: 1. Shutdown of the primary and secondary TPC server (primary only if possible). 2. Reconfiguration (site-switch) of subsystem mirroring target LUNs of the production mirror in the secondary datacenter need to become accessible and reverse mirroring back to the primary datacenter needs to be started (if possible). 3. Reconfiguration of secondary server HBA BIOS to now boot off production LUNs instead of maintenance LUNs. 4. Boot secondary server and (re-)establish mirror to internal disks. 5. Check that TPC data collection is being restarted... failover is complete. The procedure will be described in greater detail in the following chapters Requirements for TPC Server Stand-by server hardware (identical with production HW) 2 x Static IP addresses valid in both sites 2 x Single-Port Fiber Channel HBAs SAN storage corresponding TPC server disk layout (LUN count and size) 1.2 Stand-By Concept for Brocade CIM Agent No SAN-boot based redundancy concept is implemented for the Brocade CIM agents. In case of a component failure, manual reconfiguration of the remaining CIM agents needs to be performed. If a CIM agent server fails, and it can not be repaired / replaced in time, the remaining CIM agent needs to be reconfigured to now also monitor the devices that were previously monitored by the failed agent. Alternatively, if stand-by server hardware is available, a stand-by CIM agent can be prepared to be started in case of a failure on the primary CIM agent. This makes reconfiguration of the surviving CIM agent needless, making the failover procedure much quicker and less error-prone. Page 6 of 30 Copyright IBM Corporation, 2013

7 After starting the stand-by CIM agent or reconfiguring the surviving CIM agent, TPC configuration needs to be adapted as well. The devices managed by the failing CIM agent need to be rediscovered using the remaining CIM agent(s) (which is/are now also managing the 'missing' devices). After the discovery ran successfully, performance monitors that failed when the CIM agent became unreachable need to be restarted. Detailed instructions can be found in 'TPC User's Guide' and 'TPC Installation and Configuration Guide'. Note: TPC will not automatically switch data collection from one CIM agent to another. Instead, manual intervention is required in all cases: performance monitoring jobs for the affected devices need to be stopped and restarted after successfully rediscovering the devices through the remaining CIM agent. In case of a failure of one of the Brocade CIM agents, the following actions need to be taken: 1. Reconfiguration of the surviving CIM agent configuration files and scripts have been prepared for that purpose (involves restarting the agent). Alternatively: 1. Start stand-by CIM agent with prepared configuration (identical to failed agent configuration). 2. Reconfiguration of TPC rediscover missing devices through surviving CIM agent and restart performance data collection. The procedure will be described in greater detail in the following chapters Requirements for Brocade CIM Agent Preparation of failover scripts on both CIM agent servers Optional: Server hardware for stand-by CIM agent installation Page 7 of 30 Copyright IBM Corporation, 2013

8 2 Step-By-Step Instructions After outlining the stand-by concept on a high level in the previous chapter, this chapter describes the necessary procedures for setup, failover and failback in greater detail, providing concrete step-by-step instructions for all necessary actions. Procedures for the TPC server are being described separately from procedures for the Brocade CIM Agent, due to the different concepts implemented. 2.1 Partition / LUN Overview The following partitions are available on the TPC servers: Primary TPC Server Partitions Partition Description Path Mirrored System Contains operating system and all productive applications: C:\ Yes (local and SAN) Windows 2003 IBM TPC IBM DB2 QLogic SANsurfer Database Contains default database instance (DB2) C:\DB2 No (SAN only) Secondary TPC Server Partitions Partition Description Path Mirrored Maintenance Contains operating system and applications for hardware monitoring: C:\ No (local only) Windows 2003 QLogic SANsurfer Page 8 of 30 Copyright IBM Corporation, 2013

9 2.2 Setup Procedure for TPC Server Following is the step-by-step procedure that needs to be applied in order to prepare a TPC server for disaster recovery. On a high level, the following steps need to be performed for both servers (active and stand-by): Fiber-attach the server and assign SAN storage Create a mirror between local harddisk(s) and SAN disk(s) Configure server to boot off SAN disk by default Following is the detailed step-by-step procedure to apply to both TPC servers (active and standby) in order to prepare for disaster recovery: 1. Physical installation of Fiber Channel HBAs and fabric attachment. 2. Software installation of Fiber Channel HBAs according to storage support matrix. (install driver, eventually update BIOS / firmware, etc.) 3. Installation of multipathing software (e.g. SDD-DSM) according to storage support matrix. 4. Logical configuration of storage (LUN-assignment / mapping and subsystem mirroring) and SAN fabric (zoning). 5. Setup of Windows Volume Manager mirror: a. Initialization of SAN disks as 'Dynamic Disks'. b. Conversion of local disks to 'Dynamic Disks'. Note: Conversion of system partition (C:) requires server reboot c. Creation of a mirror for each partition on local disks on the corresponding SAN disk. Wait until all mirrors are established and synchronized (this can take up to some hours). 6. Server reboot access of Fiber Channel HBA BIOS by pressing <Ctrl+Q> during boot process. Following settings to be changed: a. Enable HBA BIOS b. Enable 'Selectable Boot' c. Choose LUN with system partition (C:) as 'Selectable Boot Device' 7. Another server reboot access of server's BIOS by pressing <F1> during boot process. Following settings to be changed: a. Primary boot device: HDD0 when HBA BIOS is enabled, this corresponds the SAN disk b. Secondary boot device: HDD1 - when HBA BIOS is enabled, this corresponds the local disk Page 9 of 30 Copyright IBM Corporation, 2013

10 8. After a final server reboot, the server attempts to access and boot off the SAN disks. Only if that is not possible (SAN disk not accessible) the local disks are used instead... In addition to the above procedure, it is now possible to add an additional partition to hold the database repository. According to the stand-by concept, this partition does not need to be mirrored to local harddisks to allow for greater performance and flexibility. To add an additional partition for the database repository, follow this procedure: 1. Attach an additional SAN LUN to the TPC server and be sure it becomes visible. Initialize and format the disk assign it a temporary drive letter such as X: 2. Stop DB2 and all associated services. 3. Copy the contents of C:\DB2 to the newly created drive, e.g. X:\. 4. Unassign the temporary drive letter of the new SAN LUN and mount it to the NTFS folder C:\DB2. The folder needs to be empty, so delete it's contents prior to mounting the SAN LUN. 5. Restart DB2 or reboot the server. When it becomes necessary to expand the partition available to the database repository in the future, simply attach a new LUN with the desired size and repeat the above procedure to migrate the database repository to the new LUN. With this procedure, the primary server is now fully set up and configured. The same procedure needs to be applied to the secondary server as well. The stand-by server does not hold any TPC software components but only hardware monitoring tools other than that, the procedure to be applied is exactly the same as outlined above. Note: Make sure the primary and secondary server each have their own SAN disks attached during installation and setup, and have no access to the other server's LUNs. Making LUNs visible to both servers at the same time can lead to serious data corruption which will require reinstallation. In order to prepare for complete site disasters, subsystem-based mirroring functions (Enterprise Copy Services such as IBM Metro Mirror / EMC SRDF replication) need to be set up in order to replicate production data to the backup datacenter. Configuration, ongoing monitoring and reconfiguration in case of a failover is not covered by the document at hand. Nevertheless, the according procedures play a vital role in the TPC Stand-By Concept... clear documentation and proper testing should be considered mandatory. Page 10 of 30 Copyright IBM Corporation, 2013

11 2.3 Failover / Failback Procedure for TPC Server In case the primary TPC server fails, the secondary server can be reconfigured to take over responsibilities of the failed server. That way, TPC can continue e.g. performance data collection very quickly. Note: The secondary TPC server will boot the exact same operating system image that was previously used by the primary server. It will come up with same Hostname, Network Configuration, IP address, etc. - the surrounding infrastructure needs to support this configuration with e.g. proper network routing, name resolution, etc. In order to perform the so-called 'Failover' to the stand-by hardware, SAN storage needs to be reconfigured (flip subsystem mirror direction) and the stand-by server needs to be rebooted: During normal operations, the stand-by server is booted off the maintenance partition, which is being mirrored between local harddisk and SAN LUN. If, instead, the productive TPC partition is to be booted, settings in Fiber Channel HBA BIOS need to be changed. In the 'Selectable Boot' settings section the productive LUN needs to be chosen instead of the maintenance LUN. Reference list of boot LUNs (configure in HBA BIOS <Ctrl+Q> Selectable Boot Devices): Server / Site Productive LUN Maintenance LUN Primary <FILL_IN_LUN_ID> <FILL_IN_LUN_ID> Secondary <FILL_IN_LUN_ID> <FILL_IN_LUN_ID> After restarting the Windows system, TPC will automatically continue data collection. After verification that data collection was successfully re-established, Windows Volume Manager needs to be reconfigured: After a failover, Windows Volume Manager will be in state 'Failed Redundancy'. The internal harddisk of the primary server (reported in status 'Missing') needs to be deleted from the configuration. Instead, a new mirror on the internal harddisk of the secondary TPC server needs to be established. This will require re-synchronization of the mirror, which can take up to some hours... Following is the detailed step-by-step procedure to follow in order to activate the stand-by hardware and perform a failover: 1. Shutdown of the primary TPC server if possible. 2. Shutdown of the secondary TPC server (running the maintenance partition). 3. Reconfiguration (site-switch) of subsystem mirroring to make the production LUNs in the backup datacenter accessible. 4. Boot secondary server access of Fiber Channel HBA BIOS by pressing <Ctrl+Q> during boot process. Following settings to be changed: Page 11 of 30 Copyright IBM Corporation, 2013

12 a. Choose LUN with productive system partition (C:) as 'Selectable Boot Device'. Consult the 'Reference list of boot LUNs' above for information which LUN to choose. 5. Reboot secondary server it will come up with the productive TPC environment. 6. Check that TPC data collection is restarting all performance monitors should be in a 'Running' state (blue status icon). 7. In Windows Volume Manager, re-establish mirror of SAN LUNs to internal harddisks: a. Local disks of the primary server, previously used as mirror for production partition, will appear with status 'Missing' on the secondary server. Remove those disks. b. Local disks of the secondary server will appear with status 'Foreign'. Import those disks. c. Creation of a mirror for each partition on SAN disks on the corresponding local disk. 8. Re-establish reverse subsystem mirroring back to the primary datacenter if possible. With this procedure, the secondary server now took over all responsibilities of the (failed) primary server data collection will continue as normal. Once the primary server has been repaired / replaced, a failback to the primary datacenter can be performed using the same procedure as described above. Note: Make sure the primary and secondary server each have their own SAN disks (either productive or maintenance) attached at any given time, and have no access to the other server's LUNs. Making LUNs visible to both servers at the same time can lead to serious data corruption which will require re-installation. Page 12 of 30 Copyright IBM Corporation, 2013

13 2.4 Recovery Procedures for Disk Failures on TPC Server In case of a failure of the TPC server's local harddisk, the following procedure needs to be applied in order to recover the machine and re-establish mirroring. Note that in this case it is not necessary to perform a full site failover. After a failure of one of the mirrored drives (either local disk or SAN LUN), the device will appear in state 'Failed Redundancy' as shown below: The failed disk will appear in state 'Missing', and the mirror will appear in state 'Failed Redundancy'. These state changes will also be reflected in Windows Event Log in addition, administrative users connecting to the console session of the TPC server will receive a Login Notification. To recover from this situation, the following procedure needs to be applied: 1. If the local harddisk failed, shutdown the TPC server and physically replace the failed disk drive. Restart the server prior to proceeding. In case of a failure of the SAN LUN, fix the problem. 2. In Windows Volume Manager, re-establish mirroring: a. The failed disk will appear in state 'Missing'. Remove the mirror on this disk, then remove the missing disk. b. Initialize the new disk drive and choose to convert to 'Dynamic Disk'. c. (Re-)Create a mirror for each partition on the new drive that was just initialized. Wait until the mirror is fully synchronized. Page 13 of 30 Copyright IBM Corporation, 2013

14 2.5 Setup Procedure for Brocade CIM Agent Following is the step-by-step procedure that needs to be applied in order to prepare the Brocade CIM agents for disaster recovery. The following steps need to be performed: 1. If stand-by server hardware is available: Set up stand-by CIM agents for both productive CIM agents. Clone configuration (provider.xml), but disable services. Alternatively: 1. If no stand-by server hardware is available: Prepare alternative configuration (provider.xml), including all entries from both existing configuration files. Distribute this alternate configuration file to both CIM Agent servers 2. In both cases: Prepare failover- and failback-scripts to either start the stand-by CIM agent services, or replace configuration files with the alternate version prior to restarting the surviving CIM agent. 2.6 Failover / Failback Procedure for Brocade CIM Agent Following is the step-by-step procedure to follow in order to perform a failover of a Brocade CIM agent: 1. If stand-by server hardware is available: Start stand-by CIM agent with prepared configuration (identical to failed agent configuration). Alternatively: 1. If no stand-by server hardware is available: Reconfiguration of the surviving CIM agent configuration files and scripts have been prepared for that purpose (involves restarting the agent). 2. In both cases: Reconfiguration of TPC: a. Stop all failing Performance Monitors (if possible) b. Run Discovery Job c. Check Managed Devices of remaining CIMOM to also include the devices previously managed by failed CIMOM d. Run Probe Job for devices previously managed by failed CIMOM e. Restart all failed Performance Monitors Page 14 of 30 Copyright IBM Corporation, 2013

15 3 Test Documentation After outlining the stand-by concept in detail in the previous chapter, this chapter documents tests of the concept that have been performed in the IBM Systems Lab in Mainz, Germany. The stepby-step procedures will be documented in detail, including used hardware and software components. 3.1 Used Hardware and Software Following is the description of Hardware and Software used during the tests in the IBM Systems Lab in Mainz: Server: IBM SSPC (2805MC2) Operating System: Microsoft Windows Server 2003 R2 Enterprise Edition Service Pack 2 HBA: 2 x QLogic QLE2460 4Gb Fiber Channel HBA (IBM P/N 39R6525) Management Software: QLogic SANsurfer Management Application for Windows b31 BIOS and Firmware: Multi-boot image for 4Gb HBAs 1.69 Driver: STOR Miniport 32-bit Driver (x86) 8Gb/4Gb HBAs Multipathing Software: SDD-DSM See the appendix for detailed description (including output of msinfo and DSA) of the server used during the tests. 3.2 Setup Procedure Test Documentation Following is the detailed description of the setup procedure followed during the tests in the IBM Systems Lab in Mainz: Prior to the actual SAN-boot tests, latest IBM System X hardware drivers (including System Flash BIOS Update, etc.) and Windows operating system patches were installed. Information about IBM hardware drivers for System X can be obtained from the following URL: ibm.com/systems/support/supportsite.wss/selectproduct? taskind=2&brandind= &familyind= &typeind= &m odelind=0&osind= &psid=bm&continue.x=1&matrix=y It is strongly recommended to update the system with the latest version of the drivers available from the above URL. Page 15 of 30 Copyright IBM Corporation, 2013

16 Note that you will not be able to use the UpdateXpress System Pack from the above URL since it will not recognize server model See the appendix for detailed description (including output of msinfo and DSA) of the server used during the tests. 1. Physical installation of Fiber Channel HBAs and fabric attachment. After the server is equipped with both single-port HBA cards, it is Fiber connected to the FC switch. The storage subsystem (IBM DS8000 was used during the tests) also needs to be Fiber connected to the same switch or fabric. Note that in a production environment it is strongly recommended to establish two fully redundant paths both HBA cards should be Fiber connected to separate switches in separate physical fabrics. 2. Software installation of Fiber Channel HBAs according to storage support matrix. (install driver, eventually update BIOS / firmware, etc.) The tests in the IBM Systems Lab in Mainz were performed using the following QLogic software components. Note that all required QLogic software can be obtained from the following URL: duct_detail.aspx?oemid=376 STOR Miniport 32-bit Driver (x86) 8Gb/4Gb HBAs QLogic SANsurfer Management Application for Windows b31 Multi-boot image for 4Gb HBAs 1.69 First, the Windows HBA driver needs to be installed. Download and extract the STOR Miniport Driver from the above URL and refer to the readme.txt file Chapter 4.3 for installation instructions. You will need to access Windows Device Manager by clicking Start Settings Control Panel System, selecting the Hardware tab and clicking the Device Manager button. Expand the SCSI and RAID controllers branch to locate the QLogic Fiber Channel Adapter. Right-click one of the HBA entries and select Properties from the context menu. Next, select the Driver tab and click Update Driver to start the Upgrade Device Driver Wizard. Choose to Install from a list or specific location (Advanced) and point to the location where you extracted the STOR Miniport Driver. Repeat this procedure for both QLogic Fiber Channel Adapters. After installation of the Windows driver for both HBAs, the SANsurfer management application needs to be installed. Using SANsurfer, Firmware and BIOS of the installed HBAs can be updated with the downloaded Multi-boot image. To do so, from within SANsurfer select an HBA port, then select the Utilities tab and click on Update Entire Image. Point SANsurfer to the file Q24AF169.BIN you extracted from the Multi-boot image to start the Firmware / BIOS update. Page 16 of 30 Copyright IBM Corporation, 2013

17 Here is the HBA information as reported by SANsurfer after the update (see appendix for further details): Page 17 of 30 Copyright IBM Corporation, 2013

18 6. Installation of multipathing software (e.g. SDD-DSM) according to storage support matrix. Installation of multipathing software needs to accord with the used storage subsystem. Install the multipathing software (such as SDD-DSM or Powerpath) that comes with the storage subsystem of choice. Refer to the vendor documentation for additional details. During the tests in the IBM System Lab in Mainz, DS8000 storage subsystems were used. Thus, the SDD-DSM multipathing driver was installed. 7. Logical configuration of storage (LUN-assignment / mapping and subsystem mirroring) and SAN fabric (zoning). The tests in the IBM Systems Lab in Mainz were performed using an IBM DS8000 storage subsystem. The subsystem was configured as outlined below: Name ID Model WWNN ============================================= DS8k-SLE07 IBM KA362 9B FFFBA0 Name desc ID Model WWNN DS8k-SLE07_2 8 x 300GB/15k IBM KA360 9B FFFBA0 Page 18 of 30 Copyright IBM Corporation, 2013

19 Memory MTS 64.0 GB IBM KA362 Storage: Extpools: p0,p1 Volgrp: it_austria/07/06/09 V9 SCSI Map 256 Flash_iTAustria/07/06/09 V11 SCSI Map 256 Lss: 30:31 VolID: 3000,3100 Name ID datatype extpool cap vgpr ============================================================== SSPC_Boot_ FB 512 P v9 Hostconnect: Name ID WWPN HostTy volgrpid =============================================================== SSPC_Boot_p1/07/06/ B320896C2 Win2003 V9 SSPC_Boot_p2/07/06/ E08B948A91 Win2003 V9 Refer to the vendor documentation for configuration details and instructions for the subsystem of choice. Note that configuration of subsystem replication mechanisms is beyond the scope of this document. The Brocade Switch was configured (zoned) to establish 2 logical paths from the TPC server to the DS8000 subsystem, using 2 DS8000 ports in different adapter cards and enclosures. Here is the logical switch configuration: Effective configuration: cfg: itaustria zone: SSPC_1_Boot_DS8k_SLE07_2 50:05:07:63:08:14:4b:a0 21:00:00:1b:32:08:96:c2 zone: SSPC_2_Boot_DS8k_SLE07_2 50:05:07:63:08:19:4b:a0 21:00:00:e0:8b:94:8a:91 Refer to the vendor documentation for configuration details and instructions for the switch of choice. 8. Setup of Windows Volume Manager mirror: The SAN LUN(s) configured on the storage subsystem should now be visible at the server. In order to verify this, access Windows Device Manager by clicking Start Settings Control Panel System, selecting the Hardware tab and clicking the Device Manager button. Expand the Disk drives branch and verify the following entries: Page 19 of 30 Copyright IBM Corporation, 2013

20 Adaptec Array SCSI Disk Device The internal RAID controller IBM Multi-Path Disk Device The SAN LUN(s) as presented by the multipathing driver IBM SCSI Disk Device IBM SCSI Disk Device The individual paths to the SAN LUN Be sure that Windows Device Manager properly displays all configured LUNs, as illustrated below: a. Initialization of SAN disks as 'Dynamic Disks'. Next, open Windows Computer Management panel. To do so, click Start Programs Administrative Tools Computer Management. Select Storage Disk Management from the left-hand navigation tree. A wizard will come up, as shown below: Page 20 of 30 Copyright IBM Corporation, 2013

21 If the wizard does not automatically come up, right-click the disk that corresponds the SAN LUN (e.g. Disk 1) and select Initialize. Complete the wizard, choosing to initialize and convert the SAN LUN (e.g. Disk 1). The disk needs to be converted to dynamic you can either choose to convert it during initialization, or you can manually convert the disk afterwards by right-clicking it and selecting Convert to Dynamic Disk... b. Conversion of local disks to 'Dynamic Disks'. Note: Conversion of system partition (C:) requires server reboot Right-click the disk that corresponds the local RAID controller (e.g. Disk 0) and select Convert to Dynamic Disk... Complete the wizard, choosing to convert the local disk (the disk that corresponds the SAN LUN should have been converted already). After initialization of the SAN LUN and conversion of both disks to dynamic, the Disk Management panel should look similar to the window shown below: Page 21 of 30 Copyright IBM Corporation, 2013

22 c. Creation of a mirror for each partition on local disks on the corresponding SAN disk. Wait until all mirrors are established and synchronized (this can take up to some hours). Right-click a partition on the local disk (e.g. Disk 0 partitions are marked yellow) and select Add Mirror... from the context menu. Complete the Add Mirror wizard and choose to create a mirror on the disk that corresponds the SAN LUN (e.g. Disk 1). Repeat this procedure for all partitions on the server's local disks. Computer Management panel should now display two mirrors for the chosen partition (e.g. C:), which are now marked red: Page 22 of 30 Copyright IBM Corporation, 2013

23 Note that the SAN LUN (e.g. Disk 1) will remain in state 'Warning' until synchronization of the mirror is complete. This may take up to some hours, depending on workload and disk-speed. 9. Server reboot access of Fiber Channel HBA BIOS by pressing <Ctrl+Q> during boot process. Following settings to be changed: a. Enable HBA BIOS b. Enable 'Selectable Boot' c. Choose LUN with system partition (C:) as 'Selectable Boot Device' The HBA BIOS can be enabled either by pressing <Ctrl+Q> during the boot process of the server, or using SANsurfer. When using SANsurfer, select an HBA port and click on the Parameters tab to bring up the HBA parameters panel. From the HBA Parameters settings section choose to Enable HBA Port BIOS as shown below: Page 23 of 30 Copyright IBM Corporation, 2013

24 In addition to enabling HBA Port BIOS, switch to the Boot Device Selection settings section and enable Selectable Boot. Choose the SAN LUN that corresponds the system partition (C:) as (Primary) Boot Port Name, as shown below: Page 24 of 30 Copyright IBM Corporation, 2013

25 Repeat this procedure for both HBA Ports of the server. Be sure to save the changes by clicking the Save button. 10. Another server reboot access of server's BIOS by pressing <F1> during boot process. Following settings to be changed: a. Primary boot device: HDD0 when HBA BIOS is enabled, this corresponds the SAN disk b. Secondary boot device: HDD1 - when HBA BIOS is enabled, this corresponds the local disk When the HBA Port BIOS is enabled, it will present the SAN LUN to the server's BIOS as HDD0. All existing HDD devices (local disks / RAID devices) will be renamed the HDD device number will be increased by one. With HBA Port BIOS disabled, the local harddisk is considered device HDD0. When enabling HBA Port BIOS, the local harddisk is renamed to HDD1. The HBA will present the SAN LUN as HDD0. In order to boot from SAN per default, the primary boot device needs to be set to HDD0. In order to boot from local harddisk in case the SAN LUN is not available, secondary boot device needs to be set to HDD1. Page 25 of 30 Copyright IBM Corporation, 2013

26 11. After a final server reboot, the server attempts to access and boot off the SAN disks. Only if that is not possible (SAN disk not accessible) the local disks are used instead... The same procedure needs to be applied to the standby TPC server as well. Note that configuration of subsystem replication mechanisms is beyond the scope of this document. After both TPC servers are configured to boot from SAN LUNs, the used storage subsystems need to be configured to replicate a copy of the TPC SAN LUNs to the opposing data center. Refer to the vendor documentation for configuration details and instructions for the subsystem of choice. Page 26 of 30 Copyright IBM Corporation, 2013

27 3.3 Failover / Failback Procedure Test Documentation Following is the detailed description of the failover procedure followed during the tests in the IBM Systems Lab in Mainz: 1. Shutdown of the primary TPC server if possible. During a planned failover, the primary TPC server should be shut down cleanly. This is, of course, not possible in case of an unplanned failover or a disaster. Windows crash recovery mechanisms will allow to recover from such a situation, but startup will take longer then after a clean shutdown Shutdown of the secondary TPC server (running the maintenance partition). 3. Reconfiguration (site-switch) of subsystem mirroring to make the production LUNs in the backup datacenter accessible. Configuration of subsystem replication mechanisms is beyond the scope of this document. Prior to being able to boot from the secondary copy of the TPC SAN LUN in the backup site, replication to those LUNs needs to be stopped in order to make the LUNs accessible. Refer to the vendor documentation for configuration details and instructions for the subsystem of choice. 4. Boot secondary server access of Fiber Channel HBA BIOS by pressing <Ctrl+Q> during boot process. Following settings to be changed: a. Choose LUN with productive system partition (C:) as 'Selectable Boot Device'. Consult the 'Reference list of boot LUNs' above for information which LUN to choose. After switching the replication direction of the TPC SAN LUNs in the previous step, the maintenance LUNs in the backup site will not be accessible any more. The standby server will not be able to boot from those LUNs. Instead, HBA Port BIOS settings need to be changed so that the standby server boots from the production LUNs that are now accessible in the backup site. To do so, access the QLogic HBA BIOS by pressing <Ctrl+Q> during the boot process, select one of the two adapters and open the 'Configuration Settings' 'Selectable Boot Settings' section. Choose the production LUN as the primary boot device refer to the Reference list of boot LUNs' above for information which LUN to choose. Save the settings and repeat the same procedure for the second HBA. Exit HBA BIOS and choose to reboot the server. 5. Reboot secondary server it will come up with the productive TPC environment. 6. Check that TPC data collection is restarting all performance monitors should be in a 'Running' state (blue status icon). Page 27 of 30 Copyright IBM Corporation, 2013

28 7. In Windows Volume Manager, re-establish mirror of SAN LUNs to internal harddisks: a. Local disks of the primary server, previously used as mirror for production partition, will appear with status 'Missing' on the secondary server. Remove those disks. After booting the standby server from the production SAN LUN, Windows Volume Manager will report the local disk of the primary server as 'Missing' in the standby server, and the production partition in state 'Failed Redundancy' as shown below: To remove the missing disk, right-click the production partition and select Remove Mirror. Complete the wizard and be sure to remove the mirror on the missing disk. b. Local disks of the secondary server will appear with status 'Foreign'. Import those disks. After booting the standby server from the production SAN LUN, Windows Volume Manager mirror needs to be reconfigured to discard the local copy of the maintenance partition and create a new copy of the production partition instead. After booting from the production LUN, the local disk of the standby server will be marked 'Foreign', as shown below: Page 28 of 30 Copyright IBM Corporation, 2013

29 Right-click the local disk that is marked 'Foreign' and choose to Import Foreign Disks... Complete the wizard. After importing the foreign disk, Windows Volume Manager will display two identical partitions, as shown below: Page 29 of 30 Copyright IBM Corporation, 2013

30 Note that only one of the two partitions is assigned a drive letter (C:) - in this case the partition on Disk 1. Delete the other partition (the one not assigned a drive letter, in this case on Disk 0) by right-clicking it and selecting Delete Volume... c. Creation of a mirror for each partition on SAN disks on the corresponding local disk. Now that the mirror on the missing disk was removed, the foreign disk was imported and the obsolete maintenance partition was removed, a new mirror for the production partition can be created on the local disk. To do so, right-click the production partition and choose Add Mirror. Complete the wizard and choose to create a mirror on the disk that corresponds the local harddisk (e.g. Disk 0). Note that the local disk (e.g. Disk 0) will remain in state 'Warning' until synchronization of the mirror is complete. This may take up to some hours, depending on workload and disk-speed. 8. Re-establish reverse subsystem mirroring back to the primary datacenter if possible. Once production was re-established in the backup site, it is necessary to establish reverse subsystem mirroring back to the primary datacenter. Configuration of subsystem replication, however, is beyond the scope of this document. Refer to the vendor documentation for configuration details and instructions for the subsystem of choice. Page 30 of 30 Copyright IBM Corporation, 2013