Hot Topics in IT Disaster Recovery

Transcription

1 Risk Masters International LLC Hot Topics in IT Disaster Recovery Steven J. Ross Executive Principal A Presentation for the Middle Tennessee Chapter of ISACA

2 The Popular View of IT Disaster Recovery Today The general view Back up the data Run an annual test Recover when and if necessary Actually, there s a lot going on Acquisitions and divestitures Assessing data center risk Developing Disaster Recovery architecture Recovering from cyber attacks DRaaS 2

3 Acquisitions and Divestitures The risks in integrating or separating business continuity management capabilities Differences Risks DR programs at different levels of maturity and capability Different management expectations for recoverability, availability and resilience Different interdependencies among and between business functions and technologies Different budgets for tools, redundant equipment, alternate sites, etc. Different levels of acceptance of DR in the culture Critical business functions, applications and technologies overlooked in the plans Failure to take advantage of best practices Lowered morale of DR professionals Inadequate preparation for events of the transition, especially for IT, and unexpected events in the future Inability to build upon the two DR functions for continuous improvement in the future 3

4 Acquisitions and Divestitures, continued Top 10 transition initiatives 1. Develop an effective governance structure 2. Develop continuous improvement processes 3. Consolidate IT infrastructure recovery 4. Consolidate application recovery 5. Establish a long-term BCM vision/strategy 6. Retain best practices 7. Test new recovery procedures 8. Train personnel in new recovery procedures 9. Enhance BCM staff capabilities 10. Adopt leading-edge BCM tools 4

5 Acquisitions and Divestitures, continued 1. Develop an effective governance structure 2. Develop continuous improvement processes 3. Consolidate IT infrastructure recovery 4. Consolidate application recovery 5. Establish a long-term DR vision/strategy 6. Retain best practices 7. Test new recovery procedures 8. Train personnel in new recovery procedures 9. Enhance DR staff capabilities 10. Adopt leading-edge DR tools Business Continuity Capability Critical/Resilient Non-critical/Timeconsuming Slow/Difficult Fast/Easy Speed of Accomplishment 5

6 Assessing Data Center Risk Do data centers matter anymore? Isn t everything being done on mobile devices and standalone servers? No The real work of companies and government agencies is occurring in rooms filled with machines Centralized, organization-owned data centers Centralized colocation data centers Distributed data centers Data centers in the cloud The best form of IT Disaster Recovery is not to have a disaster at all 6

7 Assessing Data Center Risk, continued The physical layer matters, and so do the machines in it 7

8 Assessing Data Center Risk, continued Uptime Institute tier ratings Active capacity components to support the IT load Distribution paths Concurrently maintainable Tier I Tier II Tier III Tier IV N N+1 N active and 1 alternate N after any failure (2N) 2 simultaneously active No No Yes Yes Fault tolerance No No No Yes Compartmentalization Continuous cooling No No No Yes Load density dependent Load density dependent Load density dependent Class A Source: Uptime Institute Data Center Site Infrastructure Tier Standard: Topology 8

9 Assessing Data Center Risk, continued How to Assess Data Center Risk, Resilience and Sustainability Study as built blueprints Walk through the data center for a high level perspective Interview Data center manager/operators Facilities manager Security manager/guard staff Inspect Underfloor (if there is one) Above ceiling (look for brown spots) MEP (confirm ratings) Telecommunications and power lines (walk the lines) Around the outside of the building and in the vicinity (walk and drive) 9

10 Assessing Data Center Risk, continued How to Assess Data Center Risk, Resilience and Sustainability Review Maintenance logs Access logs Incident analyses Purchase orders Map what you see to standards and tier ratings Verify management s expectations for continuous service Is 5 nines enough? Unplanned Downtime per Year Seconds in a year = 31,536,000 Uptime % Seconds 99% 315, Days 2 nines % 103, Days Tier % 79, Hours Tier % 31, Hours 3 nines 99.99% 3, Hours 4 nines % Minutes 5 nines % Minutes Tier % Minutes Tier % 32 32Seconds 6 nines % 3.2 3Seconds 7 nines Source for Tier Ratings: Tier Ratings Define Site Infrastructure Performance, W. P. turner et al. 10

11 Developing Disaster Recovery Architecture What is a Disaster Recovery architecture? Architecture The character or style of building; a fundamental underlying design of computer hardware, software, or both Disaster Recovery Architecture The selection of system components and their configuration, designed to provide recoverability for some or all components used for productive purposes The analysis that produces such an architecture Rationale for choosing certain features Rationale for not choosing other alternatives Cost/benefit analysis Cost/risk analysis Feasibility and reliability 11

12 Developing Disaster Recovery Architecture, continued What is a Disaster Recovery architecture? The architecture needs to resolve all levels of the technology stack Physical location(s) Facilities Computing platforms Servers Mobile devices Storage Telecommunications Local Wide area Staffing Applications 12

13 Developing Disaster Recovery Architecture, continued What is a Disaster Recovery architecture? Implementation modes Internal Colocation Outsourced Hot site Cloud Hybrid Recovery timeframes Aligned with the BIA results Governance Future growth and design 13

14 Developing Disaster Recovery Architecture, continued Components of a Disaster Recovery architecture Implementation cost and timeframe Recovery tiers and sequence Capacity Current Planned Performance Business as usual Disruption Location Talent pool Standard of living Infrastructure Hazards Butte Odessa 14

15 Recovering from a Cyber Attack Recovery from What? Disclosure of information Plug the leak Ensure that there are no others Prevent recurrence If personal information is involved, notify government agencies Most normal operations can continue while the recovery proceeds Web site vandalism Close the site Fix the problem Return to normal operations The may be the most common occurrence, since web sites are the most exposed to anonymous users 15

16 Recovering from a Cyber Attack Recovery from What? Manipulation of data Stop all applications whose data is known to be affected Treat all other data with caution Identify and repair the cause of the problem Using DBA tools and user-based controls, identify and repair erroneous data and pointers These tools are in use for problems other than cyberattacks Determine whether some data has been deleted and if possible replace it Requires the use of backups, perhaps over many generations Restart applications, with caution In some cases, operations might continue, if new activity can be isolated from faulty existing data 16

17 Recovering from a Cyber Attack Recovery from What? Manipulation or destruction of software Identify infected software Compare current software with validated copies Hashing would be the most applicable technique In some cases, software vendors may be able to publish the signature of a known attack Recovery requires trusted images and a clean environment Logs and journals also required to roll forward Replace infected or destroyed software This may require going back as far as the last vendor release Subsequent local updates may need to be rolled forward Operations cannot continue in the affected environment until the software is replaced and possibly updated 17

18 Recovering from a Cyber Attack Recovery from What? Manipulation or destruction of data Identify affected databases Wherever software is attacked, it must be assumed that the related data was affected also However, there may be interactions among databases not reflected in software Recover manipulated or destroyed data This will require going back as far as the known date of an attack Subsequent local updates may need to be rolled forward Logs and journals are also required to roll forward Replicated data may be infected; hence the need for backups 18

19 Disaster Recovery as a Service (DRaaS) The Cloud and Disaster Recovery Service Use Eliminating Long Term Storage Archiving Rare restoration Tape libraries Backup and Recovery Point in time image Repairs Magnetic tape Disaster Recovery as a Service (DRaaS) Disaster recovery Recovery sites & staff 19

20 Disaster Recovery as a Service (DRaaS), continued Advantages of DRaaS Much faster recovery for the most critical applications, depending on SLAs selected (and paid for) As little as an hour available Greatly reduced constraint of having enough technically skilled personnel to perform a recovery Testing more frequently, with less cost and more convenience for personnel Elimination or at least reduction of the use of tape Data deduplication and compression might reduce the bandwidth for communication with the vendor No need to travel for testing or an actual recovery 20

21 Disaster Recovery as a Service (DRaaS), continued Factors to Consider Cloud vendor with a DRaaS offering or a Disaster Recovery vendor with a cloud? A cloud or a backup site? Internal replication and redundancy are critical attributes A cloud service or outsourcing? The quality of vendor personnel The quality and quantity of the vendor s data centers Locations Infrastructure Vendor-owned or colocation facility Insourcing How easily can you get your applications and data back 21

22 Disaster Recovery as a Service (DRaaS), continued Factors to Consider SLAs Can they be met Testing All at the same time? Vendor reliability (see Insourcing) Platform compatibility Storage sharing Data security/encryption In transit At rest Key management Access control 22

23 Disaster Recovery as a Service (DRaaS), continued Leading vendors it depends who you ask Source: Gartner, 2016 Source: Forrester Research, Inc

24 Feedback? 24