Executive Summary: Health Care Information Technology creates every day challenges for CIOs and data center managers with issues ranging from regulatory/government requirements that include Health Insurance Portability and Accountability Act (HIPAA) and the Deficit Reduction Act (DRA); to workflow and market pressure realities, like larger imaging data sets in Radiology and Cardiology and increased cost pressures. These data management requirements force the need to establish a disaster recovery strategy for any institution. Moreover, demands of Health Care IT to provide automation across clinical applications and departments increases the requirements for accessibility and uptime. Lastly, obsolescence of software and hardware with emerging technological advances and system upgrades can disrupt access to data and critical systems. Each of these factors should be considered when determining how data is managed for the enterprise and how the disaster recovery plan becomes a component of that plan. In the end, a strategy has to focus not just on the simple hardware and software associated with a make or buy decision. These simple costs represent only the upfront and often initial capital expenditure, but there are additional real costs of this infrastructure that come from the management, maintenance and upkeep expenses now demanded by the realities of the digital clinical environment. Today, a great illustration of this is in the technology of Picture Archive and Communication Systems (PACS). Recent presentations have shown that 30% of PACS sales are replacement systems due to the end of life of system components. These replacements systems require additional capital funds to migrate the legacy data to the new system in addition to the outright cost of the new solution. Selecting the right disaster recovery solution can remove this cost issue while maintaining the level of care expected by health care providers and consumers. A Storage Service Provider (SSP) can meet the market and regulatory demands placed on an organization by providing disaster recovery solutions that will remove some of the operational hurdles that accompany IT in the health care market. published July, 2008 Page 1 of 7 www.insiteone.com InSite One, Inc. 800-441-0091
Background When most people think of disaster recovery (DR) natural events like hurricanes, tornadoes, or floods come to mind. Yet almost three out of four events requiring recovery from disasters for information systems comes from the most devastating form of disaster of all - human error. In fact, natural disasters are actually much easier to deal with than those made by machine or human error because they are in front of you and can t be avoided or hidden. That said, all forms of disasters can have a catastrophic impact on a facility s ability to provide services to the community. If a facility has a properly designed DR solution, most, if not all, can be avoided as well at all cost. It is not only the operational and clinical necessity for providing care but regulatory requirements that drive the need for DR planning in healthcare. Many healthcare facilities that think they have a true DR solution in place really don t. Before you can plan for DR, it is also important to understand what DR encompasses, as the terms redundancy, disaster recovery and business continuity have all been used to describe DR solutions. Healthcare IT is fraught with buzz words and case studies that describe multiple approaches to data protection and disaster recovery. Redundancy is the most basic form of disaster recovery and uses multiple interchangeable components to perform a single function in order to cope with failures and errors. These are typically hardware-related and replicates what you have in place on-site, with no other backups other than that which is hardware-based. It requires high connectivity and multiplies the cost challenges, not just doubling the replacement costs when the existing technology reaches its life expectancy, but those costs associated with migrating the data as well. Disaster Recovery (DR) are defined as the processes, policies and procedures of restoring operations critical to the resumption of business, including regaining access to data (records, hardware, software, etc.), communications (incoming, outgoing, toll-free, fax, etc.), workspace, and other business processes after a natural or human induced disaster. It is often associated with data stored remotely from the primary use location and is defined by Return to Operations (RTO) objectives, i.e. how fast can you get the data back to its initial state. Depending on the design, the cost for a DR solution can either be incremental or one that is exponential. These range from fairly simple implementations to those that embrace high end network connectivity requirements and multiple storage solutions. Page 2 of 7
Business Continuity (BC) take DR to the next level, encompassing processes and procedures to ensure that essential functions can continue during and after a disaster and seeks to prevent interruption of mission-critical services, and to re-establish full functioning as swiftly and smoothly as possible. This begins with a contingency plan as well as incorporating a disaster recovery plan with a business resumption plan. As the most comprehensive level of DR, it has the highest value to an organization by ensuring continuity of function which in most healthcare situations is a requirement. Challenges About half of all PACS sites have true DR solutions in place. One of the most challenging areas for DR involves Michael Cannavo, Principal Picture Archive and Communication Systems (PACS). Image Management Consulting Most people associate PACS with radiology, but cardiology PACS provide comparable volumes of data equal to or sometimes even greater than their radiology counterparts. The large volumes of data create tremendous hurdles for long-term data management as well as DR. About half of all PACS sites have true DR solutions in place. By definition, a true DR solution includes electronic backups that are also sent off-site at regular intervals and usually involve using a Storage Service Provider (SSP). By using an SSP, a facility no longer needs to worry about either storing the data safely or restoring it after a declared event, keeping both the data and the system replicated off-site, and enabling continuous access to systems and data. Most DR solutions are also governed by the Health Insurance Portability and Accountability Act, otherwise known as HIPAA. As of April, 2005 HIPAA requires healthcare facilities to address three DR areas found within the Section 164.306 of its security rule, with a fourth area strongly suggested. These areas include a data backup plan, a disaster recovery plan (which is part of both DR and BC), and an emergency operations mode plan. A test and revision procedures plan is also strongly recommended but not required at this time. HIPAA requires that exact copies of electronic protected health information (PHI) be provided and these requirements are put in place in an attempt to ensure this happens. Unfortunately while many places address one or more requirements, very few address all areas. Page 3 of 7
The data backup plan establishes and implements procedures to create and maintain retrievable exact copies of electronic protected health information (PHI). The disaster recovery plan establishes (and implements as needed) procedures to restore any loss of data. The emergency mode test operations plan establish (and implements as needed) procedures to enable continuation of critical business processes for protection of the security of PHI while operating in an emergency mode. The testing and revision procedures plan implements procedures for periodic testing and revision of contingency plans. Considerations As complex as a DR or a BC plan is and with HIPAA requiring certain items to be compliant, this bodes the obvious question-should you implement your own DR or BC plan or use outside resources like an SSP? Each has its pros and cons. Keep in mind that only 25% of the costs associated with implementing a DR solution relate to the actual storage and hardware costs. While the initial and ongoing costs are related to software, service fees and FTE (full time equivalent - i.e. employee) utilization and other areas like data migration (DM) also need to be factored in. The cost of data migration alone sometimes exceeds any cost savings associated by not using an SSP, with studies being migrated from one vendor s system to another costing from $0.25-.50 each, depending on the volume being migrated, whether database reconciliation, and a host of other variables are being done or not. Add to the cost of new archival hardware, the exceptionally quick turnaround offered by most SSP s (typically 1 TB of data within 24 hours) and other benefits, the use of an off-site SSP seems at the very least a logical consideration. Prevention is another element of cost with DR that is usually not addressed, or at least not to the level it should be. Local mirroring of systems and/or data and use of disk protection technology such as RAID is the norm for healthcare IT solution providers today and is what constitutes the most basic level of DR, redundancy. DR strategies need to go beyond just the basics, including the use of surge protectors to minimize the effect of power surges on delicate electronic equipment. Page 4 of 7
Uninterruptible Power Supplies (UPS) and/or a backup generator to keep systems going in the event of a power failure are also crucial to a well-designed system, as are fire prevention measures like alarms, accessible fire extinguishers, and inert gas agents that do not harm system hardware. Anti-virus software and other security measures should also be a part of any DR solution implemented. About 30% of PACS sales today are replacement systems, in large part because the existing components have reached their end-of-life (EOL) Michael Cannavo, Principal Image Management Consulting Disasters involve the loss of access to critical clinical data regardless of the cause. One natural cause of this type of disaster comes with data migration due to technology obsolescence or replacement. About 30% of PACS sales today are replacement systems, in large part because the existing components, including both the deep and DR archive, have reached their end-oflife (EOL) and are no longer supported by the vendor or hardware provider. This change in platform creates an opportunity for data to be lost or, minimally, creating a loss of access to the data during the migration period which can be weeks or even months long, having a negative impact on delivering patient care. Technology obsolescence is the bane of most clinical systems and on-site archives used with PACS are no different. While archive and DR solutions might use the same hardware (AIT tape or DVD s, for example) differences in storage capacity related to technology upgrades create obsolescence on a regular basis. DVD technology, for example, has shown several incremental increases in capacity over the past five years (from 1.4 GB to 9.4 GB). Blu-Ray discs can store 50GB per disk, five times that of the current DVD s. If a site upgraded the DVD writers every time storage capacity increased they would be doing this at least twice annually and have an incredibly wide variety of disparate media to deal with as well, even though most disk readers are backwards compatible. Using an SSP eliminates the need to keep up with technological advancements and the associated support therein. The ongoing investments in hardware, service and support, data migration costs, FTE costs, etc. and the possible need to upgrade computer rooms as well to support a large scale on-site archive all need to be considered relative to the cost of using a third party storage service provider (SSP). Page 5 of 7
Finding available capital dollars to purchase the required hardware is also difficult in these trying times. The Deficit Reduction Act (DRA), Medicare payment reductions, and others changes in reimbursement policies have tended to impact the amount of available capital that can be spent for DR solutions. A multi-site DR solution can easily cost $75-150K if using a SAN (Storage Area Network) as a DR solution, plus ongoing support costs. Since most of the cost relating to using an SSP come out of the operating vs. capital budget, this helps with facilities that have limited capital dollars. This also frees up capital investments for other needs such as clinical applications software or digital modalities. This is also important for facilities that may acquire additional sites and need to ramp up quickly to accommodate the additional capacity requirements from a DR solutions standpoint. It is also crucial that before a new PACS system is properly implemented, the existing archived data needs to be completely migrated over to the new system. Or at least the prior two years of studies needs to be migrated. Doing this using internal resources can be draining from both a financial and a resource perspective, not to mention taking an inordinate amount of time. This also requires cooperation from the prior vendor (who may or may not be totally cooperative if their system is being replaced by a competing vendor) the replacement vendor, and, of course, the facility s IT staff as well as the PACS systems administrator (PSA). The time required to migrate data is also fairly high, and runs between 30-90 days for the average site. This adds to the costs and potentially delaying implementation. Most home grown DR solutions merely address replicating the DICOM data and not the transactional data, including the database, configuration files and other related items. While that certainly is a start, it is far from optimal with respect to the Return to Operations (RTO). Working with your applications provider and IT department up front to set expectations on RTO of the application or minimally access to clinical data is part of the business continuity plan and design. In the design of DR planning, the safety of the backup data is as important as the primary archive. Thus you need to address the physical proximity issue- making sure the backup copies of data are as far enough away that a natural or physical disaster will not also impact access to or of your DR data. Many sites today have solutions that have archived data in the same rack or side-by-side with the primary copy; this is risky at best. Page 6 of 7
Informatics experts who define DR solutions suggest that the optimal DR data backup be located at least 50 miles or more from the primary site. Having a geographically diverse DR solution is also key, as all too many DR solutions in place today involve backups that remain on-site or within the existing campus. Ideally you also want to have a DR backup that doesn t employ a medium where data can be lost through drop outs or be damaged (i.e. tape) and has at least a thirty year shelf life, especially where pediatric, mammographic, occupational medicine, and other studies will be archived for extended periods of time. Backing up data to an off-site storage site is optimal especially if that data is near-line accessible. Commonly SSP s complimenttheir archives with secondary copies on a removable media that is stored off site as well. The challenge is maintaining the integrity of that media. DR solutions should incorporate some level of insuring the data integrity so that there is confidence that the data will be accessible when it is most neededfor recovery from a disaster. So how often do you back up off-site? Daily at a minimum, hourly preferred, real time optimally. Continuous backups ensure that data restoration involves minimal data re-entry or minimal reconstruction time from log files, incremental tapes, etc. Interestingly enough restoring data quickly in times of a natural disaster isn t nearly as crucial as those found with human error where the real disaster demands immediate action. In the case of active off-site storage for DR (where the data can be obtained in real time providing the highest level of performance). Most storage service providers can restore a terabyte of date in under 48 hours- the better services can provide immediate access in under one hour. Conclusion SSP s have their place in this equation, and investigated for the value they add in an environment where resources- technical, financial, personnel- are limited and storage and backup requirements are increasing at a steady pace. Regulations and clinical necessity demand that DR plans and data protection be an integral part of all clinical systems design. It is even more integral with PACS where the volume of data keeps increasing exponentially as modalities such as CT Scans get faster and volumetric rendering requires larger and larger data sets to be stored. As the need to store images for extended periods of time also increases, the need for solid DR solutions and strategies increases as well. SSP s have their place in this equation, and investigated for the value they add in an environment where resources- technical, financial, personnel- are limited and storage and backup requirements are increasing at a steady pace. Page 7 of 7