Agenda. Definitions. Components. Things to Consider. Exercise Approaches. Audit Approaches. Disaster Recovery / Business Continuity 1

Transcription

1 Disaster Recovery & Business Continuity 1 Agenda Definitions Components Things to Consider Exercise Approaches Audit Approaches Disaster Recovery & Business Continuity 2 Disaster Recovery / Business Continuity 1

2 Disaster Recovery "Drive thy business or it will drive thee." Benjamin Franklin ( ), American entrepreneur, statesman, scientist and philosopher "It is your business when the wall next door catches fire." Horatius (65-8 BC), Roman poet Disaster Recovery & Business Continuity 3 Definition from CobiT Disaster Recovery Planning (DRP), a key component of Business Continuity Planning (BCP), refers to the technological aspect of BCP the advance planning and preparations necessary to minimize the loss and ensure continuity of business functions in the event of a disaster. DRP comprises consistent actions to be undertaken prior to, during and subsequent to a disaster. It is built from a comprehensive planning process, involving all of the enterprise business processes. Strategies include alternate site, redundant data centers, reciprocal agreements, telecommunication links, disaster insurance, BIA and legal liabilities. Disaster Recovery & Business Continuity 4 Disaster Recovery / Business Continuity 2

3 Definition from Wikipedia Business continuity encompasses a loosely defined set of planning, preparatory and related activities which are intended to ensure that an organization's critical business functions will either continue to operate despite serious incidents or disasters that might otherwise have interrupted them, or will be recovered to an operational state within a reasonably short period. Business continuity includes three key elements: Resilience: critical business functions and the supporting infrastructure are designed and engineered in such a way that they are materially unaffected by most disruptions, for example through the use of redundancy and spare capacity Recovery: arrangements are made to recover or restore critical and less critical business functions that fail for some reason. Contingency: the organization establishes a generalized capability and readiness to cope effectively with whatever major incidents and disasters occur, including those that were not, and perhaps could not have been, foreseen. Contingency preparations constitute a lastresort response if resilience and recovery arrangements should prove inadequate in practice. Disaster Recovery & Business Continuity 5 It Happens NYC Earthquake in Nepal It is difficult to know where to begin with the historic flooding in Louisiana during the past week. There is the sheer volume of water itself based on rainfall accumulations, an estimated 4 trillion gallons of rain fell across southern Louisiana from the middle of Thursday through Saturday morning. That is roughly the same amount of water discharged by the Mississippi River into the Gulf of Mexico over the course of 80 days. Many assets were lost. Disaster Recovery & Business Continuity 6 Disaster Recovery / Business Continuity 3

4 Critical Assets Systems People Equipment Information Critical Assets Functions In a disaster, you could lose some or all of these Disaster Recovery & Business Continuity 7 Protect Your Assets BCP can be a long term competitive advantage BCP connects to the objectives of your organization What are the business plans for growth, restructuring, short/long term strategies? BCP should have the fullest possible understanding of the important processes of the business and customers and suppliers Companies with a BCP recover faster Up to 40% never recover from a disaster Disaster Recovery & Business Continuity 8 Disaster Recovery / Business Continuity 4

5 DR/BC Drivers More data than ever before, big data Connectedness, customer/user disruption and consequence More techsupported workflows Corporate reputation and image Strict data/availability requirements, regulatory issues Increased reliance on availability Disaster Recovery & Business Continuity 9 Executive Motivation Right thing to do Regulatory expectation Increased customer confidence Fear Financial Loss Market share loss Reputational impairment Disaster Recovery & Business Continuity 10 Disaster Recovery / Business Continuity 5

6 Types of Disaster Natural Earthquake Flood Hurricane Drought Twister Tsunami Cold/Heat wave Thunderstorm Mudslide Man-Made Riots War Terrorism Power outages Sprinkler system bursts Equipment sabotage Arson Epidemic Pollution Transportation accident Food poisoning Technological Database corruption Hacking Viruses Internet worms Disaster Recovery & Business Continuity 11 Business Continuity Management A series of management process and integrated plans that maintain the critical processes of an organization, should a disruption take place which impacts the ability to continue to provide key services. Business Continuity is defined as the capability of the organization to continue delivery of products or services at acceptable predefined levels following a disruptive incident. Disaster Recovery & Business Continuity 12 Disaster Recovery / Business Continuity 6

7 Business Continuity Management CM: Strategies and actions designed to protect people, property and business functions while preparing for recovery of critical processes Crisis Management IT Disaster Recovery Business Recovery IT: Resuming critical technologies (applications, communications, etc.) based on business requirements BR: Resuming critical business processes in a reasonable timeframe to a minimum acceptable level using alternatives facilities, resources, technologies, etc. Disaster Recovery & Business Continuity 13 Agenda Definitions Components Things to Consider Exercise Approaches Audit Approaches Disaster Recovery & Business Continuity 14 Disaster Recovery / Business Continuity 7

8 Components of DR / BCP Business Locations Information Technology Locations Disaster Recovery & Business Continuity 15 Questions to Ask What do you do if a business location is inoperable? What do you do if an IT location is inoperable? Disaster Recovery & Business Continuity 16 Disaster Recovery / Business Continuity 8

9 Enterprise View Enterprise Risk Management Business Continuity Management Disaster Recovery Disaster Recovery & Business Continuity 17 The Goal of Disaster Recovery To protect the entity in the case of any emergency or situation. Event Event Occurs Productivity Productivity Loss Productivity Loss Productivity Loss Time Time Disaster Recovery & Business Continuity 18 Disaster Recovery / Business Continuity 9

10 Detection Recovery DR Plan Goals Prevention Protect the organizations assets and manage risk Response Policies, procedures and actions to be followed in the event of an emergency Resumption Resumption of only the most time-sensitive business operations immediately following a disaster Recovery Implementing expanded operations to address less time-sensitive business operations immediately following an interruption or disaster Restoration The repair or relocation of the primary site and its contents and for the restoration of normal business operations at the primary site Disaster Recovery & Business Continuity 19 BCP Important Components Emergency Response Crisis Management Minutes Hours Weeks Business Continuity/IT Plans Disaster Recovery & Business Continuity 20 Disaster Recovery / Business Continuity 10

11 Best Approach All Hazards Loss of technology Loss of people DISASTER Loss of facility Loss of vendor Disaster Recovery & Business Continuity 21 Agenda Definitions Components Things to Consider Exercise Approaches Audit Approaches Disaster Recovery & Business Continuity 22 Disaster Recovery / Business Continuity 11

12 General Considerations Key staff or vendors not available during recovery Primary, secondary Ensure adequate decision making and spending authority in advance Regional communications / infrastructure may not function Recovery procedures detailed enough so alternate resources can follow if needed Recover all versus subset of required systems / functions to meet critical business processes Escalation plan and related timelines Disaster Recovery & Business Continuity 23 Things to Consider The Disaster Recovery environment must remain isolated from production No overlap of systems No overlap of technology The criterion for inclusion in the disaster recovery solution is: Does the application under consideration sustain the existing customer base given a disaster? Consider If one in 500 data centers have a major failure per year 70% of the businesses associated with these data center failures close within two years of the event The risk to a company is failure to prepare for a disaster and not the likelihood of a data center event Disaster Recovery & Business Continuity 24 Disaster Recovery / Business Continuity 12

13 Things to Consider The DR solution must be based on business driven requirements The DR solution is only as good as the network connections to it The DR solution need not mirror the production solution DR plans should be detailed enough so someone unfamiliar with the environment can read them at time of use and recover the environment Only materials present at a disaster recovery exercise can be used to recover the environment as additional materials may not be available during a true disaster The recovery of the environment requires distributed systems have their data synchronously recovered with their mainframe counterparts as applications span platforms Corporate Contingency owns the process of recovering business units to alternate locations Disaster Recovery & Business Continuity 25 Helpful Hints - DR Include critical activities that satisfy customers expectations and support overall business operations Identify the critical business information needed for these activities to succeed Review statistics on the frequency, impact and causes of downtime Identify and rank your most vulnerable business activities Adequately protect legacy systems against hacker intrusion and viruses Maintain a functional area checklist to continue business effectively in the case of a disruption or emergency Ensure change control keeps your continuity plan current with process and technology changes Place business continuity and disaster recovery on the board agenda Disaster Recovery & Business Continuity 26 Disaster Recovery / Business Continuity 13

14 Planning Strategy Continuity Life Cycle Executive Management Support & Sponsorship Compliance Monitoring & Auditing Risk Assessment & Business Impact Analysis Testing & Maintenance Continuity Life Cycle Business Continuity Strategy Design Training & Awareness Business Alignment Plan Development & Strategy Implementation Disaster Recovery & Business Continuity 27 Contingency Planning Strategy Emergency Mode Operations Communications and Logistics Tactical Response Emergency Operations Center EOC Divisional Command Centers Business Continuity Plans Disaster Recovery Plans Regular Business Operations Executive monitoring of events E S C A L A T I N G S E V E R I T Y Disaster Recovery & Business Continuity 28 Disaster Recovery / Business Continuity 14

15 DR / BCP Planning FALSE One time event Executed in a vacuum Only focused on IT systems Absolute assurance Focused on large events TRUE On going process Company culture Reasonable assurance Mitigate risks that would prevent recovery Covers all critical company processes Disaster Recovery & Business Continuity 29 Business Recovery Impact Analysis What is time sensitive? What do we have to do? Recovery Plans Alternate work solutions Alternate locations Alternate hours 3 rd party providers of services Recovery procedures What is the cost of not doing it? Resources needed with time frames Disaster Recovery & Business Continuity 30 Disaster Recovery / Business Continuity 15

16 Technical Recovery Activities and programs designed to return the technical components of an organization to an acceptable condition. Technical components are those associated with data processing and telecommunications. Disaster Recovery & Business Continuity 31 Areas of Concern Supply Chain focus (less manufacturers and suppliers) Technology virtualization & cloud (public and private) services Outsourcing of functions IT, HR, Data Centers Not employees contract specifies actions and responses Not be their only client, nor their highest priority Broader communications Doing more with less Crisis Management issues More single points of failure Disaster Recovery & Business Continuity 32 Disaster Recovery / Business Continuity 16

17 Single Points of Failure Loss of personnel and shrinking headcount More gaps from senior to junior personnel Less staff = less cross-training Retirement disaster larger than ever Less spend on technology and redundant systems Disaster Recovery & Business Continuity 33 Risk Evaluation & Control Threat Vulnerabilities Controls Chemical Spill High None Power Outage Moderate UPS/ Emergency Generator Hurricane Zero in Denver n/a Disaster Recovery & Business Continuity 34 Disaster Recovery / Business Continuity 17

18 Threats 61% of companies surveyed (Baker Tilly / Oshkosh) had to invoke their BCP. 43% had to invoke it more than once. Key Causes: Fire Natural Disaster Telecom Failure Power Outage Utility Outage IT Failure Pandemic Flood Disaster Recovery & Business Continuity 35 Top Lessons Learned 48%: Not enough training and awareness efforts across the company 37%: Plans didn t adequately address emergency communications 25%: Key staff had not been included in testing; as a result they did not know their roles and responsibilities in the plans??% Cyber attacks Disaster Recovery & Business Continuity 36 Disaster Recovery / Business Continuity 18

19 Evaluate Options Internal solutions Third party solutions Option Cost Time Internal Recovery High Fast Commercial Hot Site Medium Medium Cold site with drop shipment Medium/low Slow Cold site with vendor arrangements Low Long No recovery strategy Zero Disaster Recovery & Business Continuity 37 Disaster Recovery Plans Are there short-term backup copies of data to enable recovery from a processing failure? Are there regular backups at scheduled intervals? Are there multiple generations of backups? Are there full system backups for the operating system and application systems? Are the backups stored properly off-site and in vaults Disaster Recovery & Business Continuity 38 Disaster Recovery / Business Continuity 19

20 Emergency Response Triggers Disaster Recovery & Business Continuity 39 Successful BCP Requires clear and consistent communication Encompasses the entire Organization Informs employees communicate, where they will go and how to do their jobs Prepares the organization for disruptive events Disaster Recovery & Business Continuity 40 Disaster Recovery / Business Continuity 20

21 Data is Critical Back up relevant data on a regular basis: daily. Backup to media (tape or external hard drive), or to a remote location via the cloud. The backup process must be reliable. Management / audit should test the data to ensure that the process is actually recording all of the data onto the target backup device and where the backup is stored. There should be a test for restoring the backup at least once a year. That test should be documented, even if it is just a screenshot showing the data restored. Disaster Recovery & Business Continuity 41 Business Continuity Plans Have critical business functions been identified? Have alternate worksites been identified? Are all procedures documented, reviewed and tested? Have call-out lists been created and updated? Have technical components been identified PCs, phones, supplies, etc.? Has a dry run been executed recently? Disaster Recovery & Business Continuity 42 Disaster Recovery / Business Continuity 21

22 Business Continuity Model Awareness and Program Management Business Impact Analysis Business Recovery Planning Develop Maintain Test Demonstrated Recovery Elements of a Business Continuity Plan Risk Assessment Incident Management and Response Training Disaster Recovery & Business Continuity 43 Awareness BIA Planning Recovery Risk Incident Response Train Senior management awareness, responsibilities, commitment Existence of formal BCP program Policies, standards, roles and responsibilities Awareness and training programs Monitoring of business disruptions Third party BCP requirements Business interruption insurance coverage Disaster Recovery & Business Continuity 44 Disaster Recovery / Business Continuity 22

23 Awareness BIA Planning Recovery Risk Incident Response Train Risk assessments conducted Risk assessment methodology policies and standards Roles and responsibilities Current risk mitigation strategies and plans Risk mitigation monitoring process Senior management reviews Link to BCP and DRP Disaster Recovery & Business Continuity 45 Awareness BIA Planning Recovery Risk Incident Response Train Goal: define objectives for recovery of host computing systems that support the critical business processes Determines critical business processes and resources Establishes foundation for well reasoned and prioritized responses Focuses on reestablishing the most critical business processes in most cost effective manner Disaster Recovery & Business Continuity 46 Disaster Recovery / Business Continuity 23

24 Awareness BIA Planning Recovery Risk Incident Response Train BIA process, frequency and date of last BIA Quantitative and qualitative impacts Critical business processes and technologies Recovery time objectives and resource requirements Internal and external dependencies Result validation process Result communication and review by management Disaster Recovery & Business Continuity 47 Awareness BIA Planning Recovery Risk Incident Response Train Incident and Emergency Response documentation Disaster notification, escalation and declaration Response team notification and mobilization Response team roles, responsibilities & procedures Emergency communication capabilities Emergency operations center Communication plans (internal and external) Disaster Recovery & Business Continuity 48 Disaster Recovery / Business Continuity 24

25 Awareness BIA Planning Recovery Risk Incident Response Train Business recovery selection and implementation Current business recovery requirements Existing recovery strategies Integration of business and technology strategies On-site and off-site back up and storage strategies Contracts for relocation and alternate work space Manual procedures and data capture and recovery Disaster Recovery & Business Continuity 49 Awareness BIA Planning Recovery Risk Incident Response Train Plan documentation and sign-off process Plan components and layout (see audit checklist) Plan communication to Recovery teams, employees and public if appropriate Plan storage: storage media and use of tools Disaster Recovery & Business Continuity 50 Disaster Recovery / Business Continuity 25

26 Awareness BIA Planning Recovery Risk Incident Response Train Formal maintenance procedures and guidelines Roles and responsibilities Frequency of updates Triggers for plan updates: org, personnel changes, process and technology changes Plan quality assurance and auditing Plan update monitoring Disaster Recovery & Business Continuity 51 Awareness BIA Planning Recovery Risk Incident Response Train Formal testing procedures and guidelines Roles and responsibilities Frequency of testing and types of tests Test plan & criteria and test result documentation Post-mortem: communication of results Post-mortem: improvements based on results Monitoring of test schedules and follow up Disaster Recovery & Business Continuity 52 Disaster Recovery / Business Continuity 26

27 Awareness BIA Planning Recovery Risk Incident Response Train Formal training procedures and guidelines Roles and responsibilities BCP training program components: awareness and recovery team training Training quality assurance and auditing Training compliance tracking Disaster Recovery & Business Continuity 53 Awareness BIA Planning Recovery Risk Incident Response Train History of incidents BCP Activations (with and without DRP activation) Assessment of incurred losses and mitigations Documentation of business recovery: what, how, successes & failures, lessons Learned Communications Post-recovery improvement implantation Disaster Recovery & Business Continuity 54 Disaster Recovery / Business Continuity 27

28 Resumption at Primary Site Primary site has been declared safe by Fire Department, inspectors, other officials Test functionality with business process owners and get satisfactory response Connections to Internet and WAN have been reestablished Re-establish connections or DNS pointers to primary site Replicate data back or move the recovery system for use as the primary system Disaster Recovery & Business Continuity 55 Agenda Definitions Components Things to Consider Exercise Approaches Audit Approaches Disaster Recovery & Business Continuity 56 Disaster Recovery / Business Continuity 28

29 Exercise Approaches Disaster Recovery & Business Continuity 57 Table Top Process Walk-Through / Desk check Procedures verification Tabletop simulations Benefits Ensures documentation is current Anyone with technical expertise can execute Ensures Plan exist Drawbacks No assurance the plan works Not executing in live environment Disaster Recovery & Business Continuity 58 Disaster Recovery / Business Continuity 29

30 Simulation Process Benefits Drawbacks Simulate a disaster Unannounced scenario Interdependency review End to end walk thru Tests the communication factor call lists No assurance the plan works Not executing in live environment Disaster Recovery & Business Continuity 59 Off-Site Exercise Process Inhabit the remote location(s) Bring up infrastructure Bring up applications Process transactions WFH Benefits Ensures plans work Ensures sites are ready Provides coverage for all critical applications WFH is working Drawbacks Extensive planning Time consuming and expensive Not the full production environment Not all employees can participate Disaster Recovery & Business Continuity 60 Disaster Recovery / Business Continuity 30

31 What to Test Loss of assumed services Loss of network connectivity: Use manual processes Use alternate technology Use an outsourcing vendor Loss of Technology Loss of Vendor Coordinate test with the vendor Use alternate vendors Exercise reciprocal agreement Document results to the business plan Alternate web addresses Alternate phone numbers Disaster escalation paths Relocate to alternate site(s) Reroute phones and faxes Test ability to access the site Test alternate technology Test ability to work remote / from home via VPN while testing bandwidth Loss of Facility Disaster Recovery & Business Continuity 61 Testing Gaps Vendor personnel or backup recovery personnel cannot restore the system Port mapping / system documentation not complete / up to date Insufficient remote software / hardware support level Vendor hardware is insufficient Insufficient procedures / lack of clean updated scripts Poorly trained recovery personnel Backup not really effectiveverify successful recovery of each platform using a checklist and document verification method Application recovery not verified during the 24 hour test / inaccurate RTO Inaccurate system documentation leads to failure to meet RTO Port mapping is inaccurate / not maintained properly by hardware support Disaster Recovery & Business Continuity 62 Disaster Recovery / Business Continuity 31

32 DR Plan Key Categories Organization and assignments Critical application selection Hardware and software Facilities: Warm site / hot site; Off-site storage Awareness / training Testing Main site resumption Disaster Recovery & Business Continuity 63 Organization and Assignments Plan completeness / availability Plan updates Roles and responsibilities of DR team Disaster considerations Command center protocols Declaration criteria Timelines for each plan to execute Success factors Disaster Recovery & Business Continuity 64 Disaster Recovery / Business Continuity 32

33 Questions to Ask Where are the BCP and DRP stored and how often are they reviewed and updated? How is the shutdown process organized to minimize data loss during failover procedures? Part of the detailed DR scripts? Are the names and telephone numbers of critical vendors and contacts and required lead times for services documented. Who performs the risk assessment of potential disasters? How often? Is assessment done by appropriate individuals that understand not only the business requirements but also the potential disaster risks that exist? How are contingency plans documented based on risk assessment data that is collected? Where is evidence of this assessment stored? Disaster Recovery & Business Continuity 65 Critical Application Selection Business Impact Assessment (BIA) Technology inventory Application inventory Maximum acceptable outage Maximum acceptable downtime Disaster Recovery & Business Continuity 66 Disaster Recovery / Business Continuity 33

34 Business Impact Analysis (BIA) Establishes the value of each organizational unit or resource as they relate to the function of the total organization Provides the basis for identifying the critical resources required to develop recovery strategies Provides the order or priority for recovery Disaster Recovery & Business Continuity 67 Business Impact Analysis (BIA) BIA Produces Objective Prioritization Based on corporate risk factors Applications Reputational impact Operational impact Determining the threshold for risk Processes Regulatory / legal impact Credit / liquidity impact Cost of not doing anything Length of time until restoration Disaster Recovery & Business Continuity 68 Disaster Recovery / Business Continuity 34

35 Definitions RPO = Recovery Point Objective RTO = Recovery Time Objective Point of Last Backup EVENT Recovery Time RECOVERY POINT OBJECTIVE (RPO): The point in time to which systems and data must be recovered after an outage. (e.g. end of previous day's processing). Used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems or functions have been recovered. RECOVERY TIME OBJECTIVE (RTO): The period of time within which systems, applications, or functions must be recovered after an outage (e.g. one business day). Used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. SIMILAR TERMS: Maximum Allowable Downtime. Disaster Recovery & Business Continuity 69 BIA Data Collection Steps Calculate Application RTO List Application Dependencies Technology Required RTO List Manual Procedures Assign Process RTO Disaster Recovery & Business Continuity 70 5 Disaster Recovery / Business Continuity 35

36 BIA Information Needed Division: Function/Function Description, Business Basics Process/Process Description, Process Location, Process RTO & Rationale Process Supporting Applications & RTO Process to Application Relationships Process Supporting Applications & RPO Impacts Process to Application Linkage Process to Application Linkage Notes Intangible Impacts Financial Impacts Impact Rationale Input / Output Dependencies Internal/External Dependencies Exchanges (Input/Output) Impacts Functions Disaster Recovery & Business Continuity 71 BIA Information Needed Member Health / Safety Employee Relations Image (Protect the Brand) Member Service / Satisfaction Provider Service / Satisfaction Contractual Obligation (SLAs) Legal Liabilities HIPAA Violations (PHI) Other Regulatory Violations (SPI) Vendors / Partners Relations Disaster Recovery & Business Continuity 72 Disaster Recovery / Business Continuity 36

37 BIA Terminology Time sensitive / critical business functions and processes Business operations required for normal daily delivery of goods and services Key, essential, or critical processes No connotation of important vs. nonimportant Analysis Separation of the whole into the parts of which it is composed Examination to determine which parts are effected in a situation Disaster Recovery & Business Continuity 73 BIA Output Sequence # AppId AppName Target RTO ITG Business Application Application Type Recovery Predecessor Recovery Succesor RSA Archer Tier 1 ITG-IS Application AnyConnect Tier 1 ITG-I&R Tool AnyPlace VPN Tier 1 ITG-I&R Tool Internet Tier 1 ITG-I&R Tool Safenet Keysecure Tier 1 ITG-IS Application LDAP Security Key Lifecycle ManaTier 1 ITG-IS Application LDAP AD / Domain Controller ServTier 1 ITG-I&R Tool Directory Service BMC Atrium Discovery and DTier 1 ITG-I&R Application LDAP BMC Atrium CMDB Cisco Unified CommunicatioTier 1 ITG-I&R Application Cisco Unified CommunicatioTier 1 ITG-I&R Application Iron Port Tier 1 ITG-BPS Application LDAP SMTP (Simple Mail Transfer Protocol) LDAP Infoblox;SMTP (Simple Mail Transfer Protocol) BMC Atrium CMDB;CA SiteMinder;Certificate Services;RRD interface (RR Donnelly - Fulfillment);RSA Medgine (ASP);Vantage Membership System (U65);Virtual Desktop Infrastructure (VDI);Virtua - CVP (Cisco Voice Portal);NICE - Disaster Recovery & Business Continuity 74 Disaster Recovery / Business Continuity 37

38 Application Prioritization Critical: 0-3 days, processes defined by company Necessary: 4 10 days, money making processes Discretionary: 11+ days, non regulatory maintenance Disaster Recovery & Business Continuity 75 Questions to Ask Is prioritization of applications qualified and/or quantified in a manner approved by senior management Are hardware requirements linked to the business approved business impact assessments? Are infrastructure requirements linked to the business approved business impact assessments? How will firewall, VPN, internet service, etc. recovery be prioritized? Are non IT supported end user applications identified by the business to support recovery efforts? How will these applications be recovered - is vendor media and corresponding data files available during the recovery process? How are non application transaction processing requirements prioritized (i.e. phone system (internal / cellular), building security, etc. Disaster Recovery & Business Continuity 76 Disaster Recovery / Business Continuity 38

39 Hardware and Software Mainframe configuration Distributed configuration Backup media File back-up schedule Telecommunications schema Interdependencies Vendor lead times Disaster Recovery & Business Continuity 77 Questions to Ask Are the servers that will be used capable of full capacity processing after failover? Are operating systems currently at the proper version and are they properly configured? Is the infrastructure at the failover site capable of full capacity processing after failover? Are routers, switches and firewalls properly configured and updated to match the standard production processing environment? How will remote locations effectively connect to the failover site? Does the site have adequate bandwidth to process data for all remote sites? Disaster Recovery & Business Continuity 78 Disaster Recovery / Business Continuity 39

40 Testing Predefined execution plan Location of recovery teams Tracking of testing Issue management Post exercise updates of plans Shift turnover protocols Disaster Recovery & Business Continuity 79 Questions to Ask What is the turnaround requirement to update scripts based on knowledge gained after a DR test exercise? When testing efforts are not complete due to time constraints, how is this accounted for How often is it required to have attempted a recovery on all levels of applications? Are application owner users involved in the testing? Are all operating platforms and operating systems included? Is the inter-dependency of applications considered? Is a log maintained of all issues identified and the resolution? Disaster Recovery & Business Continuity 80 Disaster Recovery / Business Continuity 40

41 Facilities Availability Shared facilities Equipment requirements Data backups / availability Documentation: Certain manuals will be needed including user and technical manuals. Members of the recovery team may not normally do some of the business processes. Disaster Recovery & Business Continuity 81 Questions to Ask How many employees are supported by our off site facilities? Is a guarantee of availability in the contract provisions and does the site have equipment unique to the company? If the disaster is regional, how is disaster declaration handled If the site is unavailable, where will end users process transactions? Are applications installed on PC's at the recovery site or will applications need to be loaded to support end user transaction processing? Are applications that are not IT supported going to be installed by the end user or by IT? Who maintains the inventory? How will critical supplies, business documents, communications etc. be sent to the recovery site? Disaster Recovery & Business Continuity 82 Disaster Recovery / Business Continuity 41

42 Main Site Resumption Availability Cutover procedures Length of time at the site covered by contract Testing Disaster Recovery & Business Continuity 83 Questions to Ask If the hardware used in the recovery site does not support full capacity processing how long can the alternate site be used before a total loss will occur? Does the contract specify pricing for various lengths of stay at the site? Do we follow the same procedures when executing the failback process? (For IT and business transaction processing?) Is the failback process tested as part of the regular exercise? Is the failback plan table-top tested at least annually? Disaster Recovery & Business Continuity 84 Disaster Recovery / Business Continuity 42

43 Training Disaster Recovery & Business Continuity 85 Questions to Ask How often is the DR Plan reviewed with Senior Leadership Is the status of DR communicated to the Audit Committee? Are all team members required to acknowledge that they received adequate training? How often and after what events is training conducted for end users and/or team members? Are all end users required to acknowledge that they received adequate training? Are communications with external parties approved and updated? Who is responsible for damage control from a PR perspective? Disaster Recovery & Business Continuity 86 Disaster Recovery / Business Continuity 43

44 Agenda Definitions Components Things to Consider Exercise Approaches Audit Approaches Disaster Recovery & Business Continuity 87 CEB (Gartner) 2016 Hot Spots 1. Interdependence of Today s Companies: In striving to meet aggressive growth and cost targets, companies have turned to solutions that exacerbate business continuity risk. The decadeslong trend of just-in-time production has resulted in a move toward lean supply chains and single sourcing, placing greater importance on individual suppliers. Forty-six percent of respondents in a recent survey judged this as a top-three risk. This is further exacerbated by operations in emerging markets, which offer lower production costs but are often less politically stable and at higher risk for natural disasters. Disaster Recovery & Business Continuity 88 Disaster Recovery / Business Continuity 44

45 CEB (Gartner) 2016 Hot Spots 2. Inability to Adequately Insure for Adverse Events: Insurance has traditionally been one of the pillars of a robust disaster response. However, getting the right insurance coverage is becoming more difficult. As company operations are increasingly located in more insecure locations, they become exposed to what is referred to as uninsurable risk, such as political risk or pandemics. Companies must decide whether the level of uninsurable risk is justified by business needs. Because of the rise of digitalization and cloud computing, cyber insurance policies have seen a boom. Some companies are taking a cautious approach, as it remains to be seen whether cyber insurance will really cover the full extent of damages, given its currently limited capacity. Disaster Recovery & Business Continuity 89 CEB (Gartner) 2016 Hot Spots Role of Internal Audit Audit should assess the crisis management plans of the company as a whole, and in particular of key functions such as Procurement, Logistics and IT Operations. Audit should ensure the plans are coordinated across the organization, and all involve the communications team. Finally, Audit should ensure risk management is integrated into supply chain and third party relationship management. Disaster Recovery & Business Continuity 90 Disaster Recovery / Business Continuity 45

46 CEB (Gartner) 2016 Hot Spots 2016 Audit Plan Additions Incident Response Plan: Ensure BCPs have been pressure tested for different events and specific roles and responsibilities have been assigned to all stakeholders. Verify a plan is in place for internal and external communications and that it has been coordinated with all stakeholders. Critical Operational Components Audit: Conduct a risk assessment of all critical components of the supply chain and operations infrastructure to determine which components have network effects where the impact from damage could spread. Criteria could include geographic location, availability of alternate suppliers, and existence of backup systems. Insurance Coverage Review: Evaluate insurance coverage strategies to ensure all critical business operations and potential risk scenarios have been properly considered and the appropriate insurance policies have been put in place. Disaster Recovery & Business Continuity 91 CEB (Gartner) 2016 Hot Spots Key Risk Indicators Number of critical nodes in the supply chain deemed to be at risk of contagion Frequency of updates to Business Continuity and Disaster Recovery (BCDR) plans in key functions Number of tests carried out on IT infrastructure deemed business critical Levels of political and social stability in countries of operation Frequency and efficiency of staff training on disaster response Existence of alert systems and early warning systems for business operations in disaster-prone areas Disaster Recovery & Business Continuity 92 Disaster Recovery / Business Continuity 46

47 The Role of Internal Audit 1. Business Case Communication 2. Catalyst for Change 3. Sharing Risk Metrics 4. Measure Performance 5. Translate: help overcome resistance Disaster Recovery & Business Continuity 93 Audit Benefits Independent Evaluation Identify strengths and weaknesses Discover risks and mitigation strategies Recommend improvements Identify Best Practices Disaster Recovery & Business Continuity 94 Disaster Recovery / Business Continuity 47

48 Audit Considerations DR/ BCP Team Risk Assessment BIA Secondary, Tertiary Frequency Frequency Identified and empowered Quality review Measures: RTO, RPO Disaster Recovery & Business Continuity 95 Audit Considerations Cloud Vendors Annual Maintenance Documentation Testing Disaster clauses Ownership Up-to-date Frequency SOC Report IT General Controls Review and approval Not stored in one place Includes all phases Type Results Disaster Recovery & Business Continuity 96 Disaster Recovery / Business Continuity 48

49 Audit Considerations Business Continuity Policy Assess entity coverage Management acceptance and approval Periodic review and updates Monitoring controls Names BC Committee and states responsibility Business Impact Assessment/Risk Assessment Existence of BIA Criticality of processes and systems determined Periodic updates of the BIA /RA Management approval Business Process / Business Unit Recovery Plan Recovery Plans exist and are formatted to follow the BIA Recovery strategies document all required supports Employee / customer / vendor contact information is available and current High likelihood events are identified and guidance for these events exist Plans for salvage, cleanup and rebuilding are addressed Alternate operational arrangements are prearranged Disaster Recovery & Business Continuity 97 Audit Considerations IT Systems Disaster Recovery Plan IT DR Plan is aligned with the overall plan to support business operations Critical systems are identified and prioritized Offsite storage and recovery facilities are determined and are periodically assessed for viability Hardware/software inventories exist Standard and Emergency contracts exist for key vendors Alternate processing arrangements exist Recovery Testing / Staff Training Procedures for testing and training exist Testing of all functional areas has been performed Backups are performed and are periodically tested Organization members are trained regarding incident response and disaster responsibilities Disaster Recovery & Business Continuity 98 Disaster Recovery / Business Continuity 49

50 Auditing The Plan Written policies and procedures Documented business continuity and crisis management plans Documented test plans and results of testing Awareness and training materials Documented communication (e.g., letters, s) Documented meeting minutes (e.g., steering committee, crisis management team) Project plans and/or schedules for key business continuity and crisis management planning activities Results of testing activities Disaster Recovery & Business Continuity 99 Audit Focus Policies & Procedures Exercises Exercise Documentation Updated Frequency Goals & Objectives Approaches Complete Scope Assumptions Approved Reporting Participants Evaluation Disaster Recovery & Business Continuity 100 Disaster Recovery / Business Continuity 50

51 Audit Focus Points Goals and objectives met What to do differently next time Issue Logs reviewed and answered Action items identified Evaluation Disaster Recovery & Business Continuity 101 Possible Gaps Call tree notification system dysfunctional / not at vendor, call trees incomplete or not defined Persons who can declare not defined or poorly separated (or the wrong people) vendor cannot take action under contractual terms Support teams not defined / backups for key members Approval process for changes to DR Documents DR documents not current and at vendor/on secure website Vendor in same geographic area Disaster Recovery & Business Continuity 102 Disaster Recovery / Business Continuity 51

52 Possible Gaps Step by step instructions for platform owner / vendor operators are not crystal clear No clear assignment of responsibilities or documented procedures for key platform owners No clear assignment of responsibility for vendor personnel or appropriate training on platforms Backups for key personnel not defined Business impact analysis and risk assessment not current/tier of recovery is insufficient- Example: Distributor switch from call center to web application/proprietary remote order entry system Disaster Recovery & Business Continuity 103 Questions for the Auditor What is the scope of the DR Audit? What is the timing of the DR Audit? How much time is required to conduct the DR audit? How much interaction/staff time will be required? What data gathering will be required? What audit tools will be used? How can the organization prepare for the DR audit? What Standard will be used for the DR audit? Will this be an audit or an advisory engagement? Disaster Recovery & Business Continuity 104 Disaster Recovery / Business Continuity 52

53 Audit Focus Points Risk Assessment Threats and Vulnerabilities Physical & Environmental Information Systems Existing Business Continuity Capabilities How does it fit into ERM Mitigation Strategies BIA Information RTO RPO Disaster Recovery & Business Continuity 105 Audit Focus Points Plan structure and documentation Format Standard Format Understandable Easy to Read Logical Sequence Allows reader to understand process Provides insight into timing Standard Elements Strategies Processes Teams Tasks Disaster Recovery & Business Continuity 106 Disaster Recovery / Business Continuity 53

54 Audit Focus Points Plan maintenance Documentation Logs Audit Trails Signatures Standards Frequency Response to Change Distribution Processes Responsibility Plan Owners Approvals Writers Disaster Recovery & Business Continuity 107 Audit Focus Points Training Plan owner and writer training Business continuity Documentation requirements Awareness Frequency and content Breadth of program Disaster Recovery & Business Continuity 108 Disaster Recovery / Business Continuity 54

55 Audit Focus Points Exercises Exercises Documentation Reporting Types Frequency Attendance Support Documentation requirements Goals and objectives Approach Tabletop or other mechanism Announced or surprise Assumptions Scope and participants Draft Distribution Follow-up Disaster Recovery & Business Continuity 109 Audit Focus Points Brainstorm Documentation Scope Afterwards What worked What didn t Were Goals and objectives met? What to so differently next time Issue Logs reviewed and answered Action Items Identified Mitigation plans Realistic What was not done Tabletop or other mechanism Announced or surprise Assumptions Scope and participants Disaster Recovery & Business Continuity 110 Disaster Recovery / Business Continuity 55

56 Phase Phase 1 Planning and Training 1.1 Obtain DR training materials (Guides, Classes, etc.) 1.2 Obtain logistic materials (Time, Participants, Facilities) Obtain DR Exercise planning documents (Exercise Definition, Plan, Scope) Meet with DR Exercise planners to gain a better understanding of the DR exercise planning process 1.5 Inquiry about subsidiaries capabilities of recovering in the event of a disaster. Testing Procedures Evaluate that pre-exercise planning was adequate Through analysis of the training materials determine the following: Was any formal training prepared prior to the DR exercise? Were any training aids prepared prior to the DR exercise? If so, were the materials available and their existence communicated? Through analysis of the logistic materials determine the following: Were exercise participants made aware of the exercise logistics? (Including, but not limited to: Travel, Facilities, Workspace, Meals 1.3 Through analysis of the DR Exercise Planning documents determine the following: Is DR exercise scope defined? Are DR exercise objectives defined Is recovery timing/sequencing defined? Is exercise success defined? 1.4 During the meeting determine the following: Were DR Stakeholders engaged throughout the planning process? Record and document notes. Have issues identified in previous exercises been logged and addressed? Record and document notes. Clarify any issues noted. Record and document notes Through inquiry determine the following: Obtain DR history for all X Company subsidiaries. Has the subsidiary participated in a DR exercise? When was the last DR exercise? What was the outcome of the exercise? Obtain a list of all subsidiaries participating in the 2012 DR exercise? What major issues have come up so far? Is everything progressing as expected? Record and document notes. Disaster Recovery & Business Continuity 111 Phase Phase 2 Exercise Execution Observe and document general exercise activities. 2.2 Observe and document business process portion of the exercise. 2.3 Obtain the exercise population used for business processing. Testing Procedures Evaluate that the exercise follows planned procedures. 2.1 Through observation determine the following: How were plans distributed to the DR Team? Are DR Teams using their plans during recovery procedures? Record and document notes. Is the DR Command Center (Exercise Leadership) tracking issues and resolutions? Document evidence if tracked in a central database. Is the DR Command Center (Exercise Leadership) recording when activities begin and end? IS the DR Command Center (Exercise Leadership) documenting and communicating issues, progress, and issue resolution to exercise participants? Document evidence. What major issues have come up so far? Is everything progressing as expected? Record and document notes. How are DR Teams completing application checkout? 2.2 Through observation determine the following: What was the data replication strategy used (e.g. Virtual tape)? What was the synchronization date used? How are teams testing and verifying the business processing portion of the exercise? What major issues have come up so far? Is everything progressing as expected? 2.3 Through data analytics determine the following: Create a statistical report that includes, but is not limited to the following: % that completed to EOB % that DID NOT complete to EOB # of unique members # of unique groups # of unique diagnosis codes # of unique procedure codes Do the results of the statistical report analyzing the DR exercise population seem reasonable or adequate to the DR planners? Do the DR planners have any concerns or issues on the results? Disaster Recovery & Business Continuity 112 Disaster Recovery / Business Continuity 56

57 Phase Phase 3 Post Exercise Activities 3.1 Obtain a copy of the Draft Exercise Report before distribution and evaluate the accuracy of the report based on Phases 1 and Meet with ISDR Leadership and discuss the draft report and communicate potential improvement opportunities 3.3 Obtain evidence that the Exercise Report is communicated to management. Testing Procedures Evaluate that DR Exercise Results are accurately and appropriately documented and communicated to management stakeholders. Communicate AAS findings. 3.1 Through analysis of the Draft Report determine the following: Are the exercise results accurately reported? Are results traceable to DR exercise objectives? 3.2 Meet with DR leadership and clarify any questions with the Draft Report and communicate improvement opportunities. Communicate Improvement Opportunities and discuss with ISDR management. 3.3 Through analysis of the communication determine the following: Was the report communicated to the appropriate management? Was the report timely? Disaster Recovery & Business Continuity 113 John A. Gatto johnagatto@comcast.net Disaster Recovery & Business Continuity 114 Disaster Recovery / Business Continuity 57