EMC BUSINESS CONTINUITY SOLUTION FOR GE HEALTHCARE CENTRICITY PACS-IW ENABLED BY EMC MIRRORVIEW/CE

Transcription

1 White Paper EMC BUSINESS CONTINUITY SOLUTION FOR GE HEALTHCARE CENTRICITY PACS-IW ENABLED BY EMC MIRRORVIEW/CE Applied Technology EMC GLOBAL SOLUTIONS Abstract This white paper provides an overview of a GE Healthcare Centricity PACS-IW environment built on servers with EMC CLARiiON storage. It describes the tested configuration environment and identifies the key results of testing. This white paper covers EMC MirrorView, EMC MirrorView/Cluster Enabler, and CLARiiON storage. March 2011

2 Copyright 2011 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Part Number H8186 2

3 Table of Contents Executive summary... 5 Key benefits... 5 Business case... 6 Solution overview... 6 Key results/recommendations... 7 Introduction... 8 Overview... 8 Purpose... 8 Audience... 8 Terminology... 8 Technology overview... 9 Environment components GE Healthcare Centricity PACS-IW Physical environment Hardware components Software components Storage EMC MirrorView Microsoft clustering EMC MirrorView/Cluster Enabler Configuration Setting up Microsoft Cluster to use MNS with FSW Main database cluster configured for MirrorView with Cluster Enabler Main controller cluster configured for MirrorView with Cluster Enabler Integrad Control Panel CLARiiON MirrorView configuration Centricity PACS-IW Viewer Interface login Patient study list Patient study retrieval Progression bar explanation Purple bar Light green bar Dark green bar Sample case study Test and validation Primary cluster node failure

4 Observations Main controller node failure Main database node failure Primary site storage failure Main controller and main controller node failure Conclusion Summary Findings Next steps References Product documentation

5 Executive summary Healthcare providers are continuing to invest in radiology and cardiology Picture Archiving and Communication System (PACS) to predict, diagnose, treat, and monitor disease. With each new technology advancement for managing modalities, healthcare provider IT organizations are challenged to meet increasing demands for high availability and scalability of systems and networks. Increased security and automated tools to simplify the complexity of managing this expanding imaging environment are also needed. This has resulted in a higher demand on the IT infrastructure to provide high availability, business continuity, security, and tools to support these clinical applications. EMC offers several key infrastructure components to economically deliver highly available applications to healthcare IT consumers. These components, when combined with clinical and business application software from EMC partners, deliver continuous information availability and protection to enable uninterrupted hospital operations. The EMC Business Continuity Solution for GE Healthcare Centricity PACS-IW enabled by EMC is an example of this sort of collaboration between EMC and GE Healthcare. Centricity PACS-IW is a web-based solution for all modalities and healthcare environments that offers a wide range of functions at high speed, based on the latest international standards for small and medium-size hospitals and imaging centers. For the first time, GE PACS-IW users can deploy a solution that is tested and integrated with their application with automated failover and recovery processes that can deliver repeatable recovery point objective (RPO) and recovery time objective (RTO). Key benefits This solution provides important benefits, including: Built on EMC tiered-networked storage platforms that are highly scalable and are able to support the needs of smaller and medium-sized hospitals up to the largest academic hospitals and Integrated Delivery Networks (IIDNs). Delivers five 9s of availability (99.999% uptime), plus the reliability, scalability, and performance needed by the healthcare enterprise. Provides a standard and highly automated failover and recovery solution for all GE PACS-IW users. Quick failover following an unplanned outage tested at less than 3 minutes in physical environment. Dynamically maps computing resources to the healthcare enterprise. This results in a lowering of IT costs due to the ability to treat the data center as a single pool of processing. Meets RPOs and RTOs for business continuity and disaster recovery plans with information management and protection software. Maintains the privacy and security of patient records, confidential s, and other communications. Provides secure and rapid recall of historical images and meets regulatory requirements for records retention. 5

6 This information infrastructure, built for reliability, supports both short-term access and longterm archiving and can be integrated with cardiology, radiology, and other clinical and business systems to automate workflow and streamline operations. Business case This white paper provides a technical architecture for current and prospective customers of GE Healthcare Centricity PACS-IW who are responsible for providing an IT infrastructure for clinical applications, and have a technical understanding of the GE PACS-IW application and IT impact on business continuity. EMC enterprise business continuity solutions ensure that your applications and data are available during planned and unplanned outages. EMC enterprise business continuity solutions provide tiered storage platforms, software, and services that ensure high availability and robust data protection and are designed to fit budgetary and technology constraints. All work together to help you survive an unplanned outage and continue your business operations. Solution overview The PACS-IW solution leverages Microsoft Cluster Services with EMC Cluster Enabler to maintain business continuity for the application and EMC MirrorView replication technology for the storage array. Cluster Enabler (CE) for Microsoft Failover Clusters is a software extension of failover cluster functionality. Cluster Enabler allows Windows Server 2003 and 2008 (including R2) Enterprise and Datacenter editions running Microsoft Failover Clusters to operate across multiple connected storage arrays in geographically distributed clusters. In Windows Server 2003 the failover clusters are called server clusters and use Microsoft Cluster Server (MSCS). Each cluster node is connected through a storage network to the supported storage arrays. Cluster Enabler expands the range of cluster storage and management capabilities while ensuring full business continuance protection. A Fibre Channel connection from each cluster node is made to its own storage array. Two connected storage arrays provide automatic failover of mirrored volumes during a Microsoft failover cluster node failover. Cluster Enabler protects data from storage, system, and site failures, 24 hour a day, 7 days a week, and 365 days per year. While Cluster Enabler protects against node failures at either primary or secondary sites, it also allows for scheduled maintenance by allowing cluster resources to fail over and fail back between sites. MirrorView/S is a synchronous product that mirrors data in real time between local and remote storage systems. MirrorView/A is an asynchronous product that offers extendeddistance replication based on a periodic incremental-update model. It periodically updates the remote copy of the data with all the changes that occurred on the primary since the last update. MirrorView also offers consistency groups, a unique consistency technology for the midrange market that replicates write-order dependent volumes. Using this technology, MirrorView maintains write ordering across secondary volumes in the event of an interruption of service to one, some, or all of the write-order dependent volumes. 6

7 Key results/recommendations By deploying the EMC Business Continuity Solution for GE Healthcare Centricity PACS-IW, a hospital can consistently meet RPOs and RTOs for business continuity and disaster recovery plans while maintaining the privacy and security of patient records. This is achieved by creating highly automated processes for failover and for restarting applications and the storage platforms supporting those applications. This goal of a rapid restart was achieved consistently in less than 3 minutes in the test environment created, avoiding extremely costly outages that can take hours, or even days, from which to recover. 7

8 Introduction Overview This white paper provides an overview of a GE Healthcare Centricity PACS-IW environment built on CLARiiON storage. It demonstrates cluster node and storage business continuity scenarios using with Microsoft Clustering Services. The goal of this white paper is to assist GE Healthcare and EMC customers in understanding the capabilities of this solution. As their PACS-IW environment grows they can make the necessary changes to continue to support their business continuity and disaster recovery strategy and requirements. Purpose This white paper describes and documents how to implement a business continuity and disaster recovery solution in support of PACS-IW. It will help GE and EMC engineers and customers understand the best practices for deploying PACS-IW on an EMC CLARiiON array in a business continuity and disaster recovery configuration. Audience This document is intended for the EMC Global Practice, Product Engineering, Field Council, Global Solutions, Marketing, and Demo teams, EMC Training and Services organizations, as well as current and prospective GE Centricity PACS-IW customers who are responsible for providing an IT infrastructure for clinical applications. Terminology Table 1 defines terms used in this document. Table 1 Terminology Term MNS/FSW MSCS MV/CE MV/S PACS RPO RTO SAN Definition Majority Node Set/FileShare Witness Microsoft Cluster Services MirrorView Cluster Enabler MirrorView/Synchronous or MirrorView/S Picture Archiving and Communication System Recovery point objective. RPO is the point in time (prior to an outage) that systems and data must be restored to. Recovery time objective. RTO is the period of time after an outage in which the systems and data must be restored to the predetermined (RPO). Storage area network. A high-speed special-purpose network connecting various sorts of storage devices with servers, typically to support a larger network of users. 8

9 Technology overview The following illustration shows the architecture of the solution. Figure 1 Architecture diagram Figure 1 shows a typical GE PACS-IW installation in a physical environment for a disaster recovery scenario. It consists of two geographically dispersed clusters for both main controller and main database nodes on CLARiiON storage. Replication between sites is implemented using MirrorView/S functionality. The main controller provides the simple webbased user interface and the main database holds all of the patient data. 9

10 Environment components This section identifies and briefly describes the technology and components used in the infrastructure environment. GE Healthcare Centricity PACS-IW GE Healthcare Centricity PACS-IW is a standards-based, single user interface, web-based PACS. It handles Computed Tomography (CT), Magnetic Resonance (MR), ultrasound (US), nuclear medicine (NM), computerized radiography (XA), PET/CT scan (PT), and many other kinds of exams. Centricity PACS-IW uses native tools for paperless operations (document scanning), print pages (key images), industry-standard IHE Portable Data Imaging (PDI) CD burning, and referring physician access for imaging results. The web-based PACS-IW can be configured in different ways to fulfill customer s requirements regarding data safety and system uptime. It can also be adopted to fit into the current IT environment managing multi-site installations, bandwidth limitations, and security requirements: Multi-site powered, for scalable growth Maximum speed with cross-site streaming of images Scalable for expanding businesses Protect patient data transfer using encrypted protocols Table 2 lists the details of the PACS-IW clusters. Table 2 PACS-IW cluster details Term Main database cluster Main controller cluster Cluster Name GE-MVDB-CLU MVCE-CLU Member Nodes GE-MV-DB1, GE-MV-DRDB1 GE-MV-MC1, GE-MV-DR-MC1 Majority Node Set GE-MVCE-DB GE-MVCE-MC 10

11 Figure 2 Main controller and main database clusters 11

12 Physical environment The environment consisted of two Microsoft clusters spread across two sites. The hardware and software components are listed below. Hardware components 4 servers with quad-core Intel Xeon x5570 processors, 2.93 Ghz CPU, 144 MB memory CLARiiON CX4-120 x 2 Network switch x 2 FC switch x 4 GE PACS-IW licensing USB dongle x 2 Software components Unisphere on CLARiiON FLARE 30 MirrorView/S software enabler Cluster Enabler Base Cluster Enabler Plugin Solutions Enabler Solutions Enabler Base License Solutions Enabler License for Cluster Enabler with CLARiiON PowerPath 5.3 Microsoft SQL Server 2005 SP2 GE Centricity PACS-IW Storage The storage configuration consisted of two EMC CX4-120 arrays using Unisphere (FLARE 30), one array per site. The primary and secondary sites consisted of dual fabrics offering redundancy across switches. Both sites had IP network switches. EMC MirrorView EMC MirrorView /S enabler was installed on each of the CLARiiON CX4-120 arrays. Primary and secondary mirror pairs were created and data was synchronized between arrays. Microsoft clustering The main database cluster consisted of one node per site, each physical node using Windows 2003 SP2 x64 and SQL 2005 SP2 installed. The main controller cluster consisted of one node per site, each physical node using Windows 2003 SP2 x86 and each node with GE Centricity PACS-IW software installed. The PACS-IW licensing requirements are satisfied by using a USB dongle connected to each of the main controller nodes. Access to the PACS-IW Viewer was achieved via the main controller cluster virtual IP address. 12

13 EMC MirrorView/Cluster Enabler supports the quorum model types in Table 3. Table 3 Supported model types Microsoft Windows Server Model type 2003 Majority Node Set (MNS) MNS with File Share Witness (FSW) 2008 Node Majority Node and File Share Majority For the purpose of the GE PACS-IW use case, the MNS with FSW quorum model was selected. An MNS quorum cluster can only run when the majority of the cluster nodes are available; a two-node MNS quorum cluster is unable to sustain the failure of any cluster node. This is because the majority of a two-node cluster is two. To sustain the failure of any one node in an MNS quorum cluster, you must have at least three devices that can be considered as available. The MNS with FSW cluster model is recommended for clusters with special configurations. It works in a similar way to Node and Disk Majority, but instead of a witness disk, this cluster uses a witness file share. If you use Node and File Share Majority, at least one of the available cluster nodes must contain a current copy of the cluster configuration before you can start the cluster. Otherwise, you must force the starting of the cluster through a particular node. Note Windows Server 2003 only supports two-node clusters with this quorum model. 13

14 Configuration Setting up Microsoft Cluster to use MNS with FSW Typically many Windows 2003 cluster configurations will be set up with Shared Quorum disk. To satisfy the Cluster Enabler requirements, the Microsoft cluster was reconfigured to use MNS with FSW. This entails adding another node into the Microsoft cluster to act as the witness for the cluster and it is recommended that this node be located at a tertiary site. Note 1. Add a new resource called MajorityNodeSet to the cluster group for each cluster, as in the following Microsoft article: 2. On the MNS node, create a new folder and update the FileShare permissions to add the Cluster Service account and allow change" and "read" permissions for the file share. Microsoft Windows documentation provides instructions on changing permissions for FileShare. 3. Update the cluster so it can reference the newly created file share witness. The following Microsoft article references how to define the MNS with FSW: 4. Do not add the MNS node into the cluster until Cluster Enabler has been configured, as the MSN does not need Cluster Enabler installed. If the node is added to the cluster at this point the MV/CE configuration will fail. It is highly recommended that the MNS nodes reside at a tertiary site, as per Microsoft recommendations. Main database cluster configured for MirrorView with Cluster Enabler Figure 3 shows the status as viewed via Cluster Enabler after installation is complete, note that the MNS node is excluded from control via CE. Figure 3 Main database cluster status 14

15 Figure 4 shows the Microsoft cluster view after MNS has been added to the cluster. The resources can only fail over and fail back between the production nodes (GE-MV-DB1 and GE- MV-DRDB1). Figure 4 Main database cluster status with MNS Main controller cluster configured for MirrorView with Cluster Enabler Figure 5 shows the status via Cluster Enabler after installation is complete; note that the MNS node is excluded from control via CE. Figure 5 Main controller cluster status 15

16 Figure 6 shows the Microsoft cluster view after MNS has been added to the cluster. The resources can only fail over and fail back between the production nodes (GE-MV-MC1 and GE- MV-DR-MC1). Figure 6 Main controller status MNS Integrad Control Panel The main controller cluster nodes also show the status of the PACS-IW resources on the Integrad Control Panel. The green status shows that the resources are online and operational from an application point of view. Figure 7 Integrad Control Panel 16

17 CLARiiON MirrorView configuration MirrorView Cluster Enabler groups the mirror pairs into consistency groups automatically during the installation and names them according to the cluster groups defined in the Microsoft cluster, that is, Integrad and SQLServer. Figure 8 Mirrors and Consistency Groups Centricity PACS-IW Viewer The GE Centricity PACS-IW Viewer provides access to patient images and reports. Interface login To verify the functionality of the PACS-IW viewer, browse to the virtual IP for the main controller cluster: and login with a user account. Figure 9 Centricity PACS-IW login screen 17

18 Patient study list Figure 10 shows the list of patient studies associated with the specific logged-in user. Figure 10 User-specific patient study list Patient study retrieval When the case study is being retrieved a monitor progress bar may be visible on the viewer screen. Progression bar explanation Purple bar The technical influencer is primarily the database server. A slowly growing purple bar indicates issues with the database query for study descriptors. Possible reasons for this progress bar are: Non-performing hard disks hosting database files, such as SATA and NAS Network latency greater than 10 ms The database has not been re-indexed for a long time 18

19 Light green bar The technical influencer is mainly the primary archive and network conditions. A slowly growing light green bar indicates issues with the image retrieval from the primary archive. Possible reasons for this progress bar are: Non-performing primary archive; required throughput 200 MB/s; (NAS should not be used as the primary archive, the preferred primary archive is direct-attached storage (DAS) or SAN; SAS or FC disks) A network connection between the node and controller of less than 1 Gb/s WAN connection between the client and node with low bandwidth (less than 4Mb/s) Dark green bar The technical influencer is primarily the client. A slowly growing dark green bar indicates issues with the performance of the client workstation. De-compression is too slow. Possible reasons for this progress bar are: CPU is at the limit of its performance. Radiologists should have workstations with 2x quad-core CPU and 4 GB memory Too many applications are running at the same time, consuming memory resources; the workstation starts outsourcing to the page file on hard disk In Windows XP, the 4 GB memory address range needs to be switched on in the boot.ini file The client hard disk is too slow; required throughput for workstations with high workload - hard disk is 120 MB/s 19

20 Sample case study Figure 11 shows typical images viewed by healthcare professionals in the PACS-IW viewer. Figure 11 Patient images viewed though Centricity PACS-IW Viewer 20

21 Test and validation Primary cluster node failure In the event of a server failure for any node owning the cluster resources at the time, EMC Cluster Enabler allows all disk-based resources to automatically fail over between sites using EMC MirrorView replication technology. The images below illustrate failover in the event of a node failure on the main controller cluster. Note that the resources are owned by the primary node (GE-MV-MC1) prior to simulated node failure. A node failure can be represented by stopping the Cluster Service, shutting down the operating system, or removing power from the node. Figure 12 Cluster status before failover Once Microsoft clustering detects that the active node has gone offline or its resources are failing, it automatically moves the resources to the secondary node available in the cluster. Figure 13 shows the node GE-MV-DR-MC1 taking ownership of the cluster resources on failure of node GE-MV-MC1. Figure 13 Failure of the GE-MV-MC1 node 21

22 EMC Cluster Enabler interacts with MirrorView and promotes the mirrored LUNs on the secondary CLARiiON array and its role is now defined as primary. Figure 14 Mirrored LUNs promoted Each of the physical disk resources has a dependency on the EMC Cluster Enabler resource. Figure 15 Physical disk properties When the EMC Cluster Enabler resource type (EMC_Integrad) comes online, all physical disks that are dependent on this resource are brought online on the secondary node. 22

23 Figure 16 Physical disks online The steps above are for node failure only, and show failover of the main controller node, but this automated process also applies to the main database cluster in the event of any node failure. Observations Main controller node failure When a main controller node fails and all resources move to the secondary node in the cluster, during this time the cluster resources need to temporarily go offline while being moved from node to node. The following observations were made: Centricity PACS-IW Viewer users are automatically logged out of the application. Note This is because the Tomcat cluster service needs to be restarted on the secondary node. PACS-IW users can re-login once the Tomcat service comes online on the secondary node and can access the patient studies list. Typically the Tomcat resource restarts within a few seconds. PACS-IW users are not able to open patient studies until all cluster resources have moved to the secondary node. Figure 17 opens if patient study list access is attempted by users while the failover is in progress. Figure 17 Study loading problem main controller Once the resources are fully online on the secondary node all access to the patient studies is now available. 23

24 In the GE PACS-IW use case environment the failover was observed to take approximately 3 minutes. Main database node failure When a main database node fails and all resources move to the secondary node in the cluster, the cluster resources need to temporarily go offline while being moved from node to node. The following observations were made: Centricity PACS-IW Viewer users are not logged out of the application. PACS-IW users are not able to open patient studies until all cluster resources have moved to the secondary node. The Study loading problem message box is displayed if patient study access is attempted by users while the failover is in progress. Once the resources are fully online on the secondary node all access to the patient studies is now available. In the GE PACS-IW use case environment the failover was observed to take approximately 3 minutes. 24

25 Primary site storage failure The images show failover in the event of a storage failure. Note that the resources are owned by the primary nodes (GE-MV-MC1 and GE-MV-DB1) prior to simulated failure. A storage failure can be represented by power outage, total fabric failure, and storage array failure. In order to simulate total storage failure, ports were blocked on both switches in the fabric, removing disk access to the storage array. Figure 18 Storage failure simulation 25

26 Once Microsoft Clustering detects that the active node has gone offline or its resources are failing it automatically moves the resources to the next available node in the cluster. In Figure 19 the nodes GE-MV-DRDB1 and GE-MV-DR-MC1 are shown taking ownership of the cluster resources on failure of the primary storage array. Figure 19 Resources move nodes EMC Cluster Enabler interacts with MirrorView and promotes the mirrored LUNs on the CLARiiON array to the secondary array. Using EMC Unisphere, the MirrorView Consistency Groups now show their role to be Primary on the secondary array. Figure 20 Consistency groups in the Primary role 26

27 Main controller and main controller node failure Storage failure at a site will result in the main controller and main database nodes failing. All resources move to the alternate nodes in the cluster at the secondary site, and during this time the cluster resources need to temporarily go offline while being moved from node to node. Centricity PACS-IW Viewer users are automatically logged out of the application Note This is because the Tomcat cluster service needs to be restarted on the secondary node. PACS-IW users can re-login once the Tomcat service comes online on the secondary node and regain access to the patient studies list. Typically the Tomcat resource restarts within a few seconds. PACS-IW users are not able to open patient studies until all cluster resources have moved to the secondary node. The Study loading problem message box is displayed if patient study access is attempted by users while the failover is in progress. Once the resources for both clusters are fully online on the secondary site all access to the patient studies is now available. In the GE PACS-IW use case environment the failover for both clusters was observed to take approximately 3 minutes. 27

28 Conclusion Summary For the first time, GE Centricity PACS-IW users are enabled to deploy an integrated solution leveraging EMC capabilities for automated failover and recovery processes that can deliver repeatable RPO and RTO. Findings Testing of the EMC Business Continuity Solution for GE Healthcare Centricity PACS-IW demonstrates that a standard, repeatable, and highly automated failover and recovery solution can be created for all GE PACS-IW users. In the test conducted in EMC s Industry Solutions Engineering laboratory, RTOs of less than 3 minutes in the physical environment were consistently achieved for the workloads tested. Cluster Enabler protects against node failures at either primary or secondary sites and allows for scheduled maintenance by allowing cluster resources, failover and failback between sites. Using MirrorView, which offers a unique consistency technology that replicates write-order dependent volumes, you can maintain write ordering across secondary volumes in the event of an interruption of service to one, some, or all of the write-order dependent volumes, speeding the recovery process in the event of an outage or disaster. Next steps To learn more about this and other solutions, contact an EMC representative or visit 28

29 References Product documentation For additional information on the products discussed in this white paper, see the following: EMC Cluster Enabler Base Component Version 4.1 Release Notes EMC MirrorView/Cluster Enabler Version 3.1 Product Guide EMC MirrorView/Cluster Enabler Plug-in Version 4.1 Release Notes EMC MirrorView Knowledgebook: FLARE 30 A Detailed Review GE Healthcare Centricity PACS-IW Complete Installation Guide GE Healthcare Centricity PACS-IW User s Manual 29