Technical Recovery Planning

Transcription

1 Technical Recovery Planning

2 Statistics NEDRIX 2006 Annual Conference Business Continuity 52 offices in 26 countries 258 Business Continuity Plans 260 Business Continuity Manager (BCM) / Alt BCM / Team Leaders 3,000+ business alternate site exercises annually 40+ continuity presentations to external audiences annually Technology Recovery 31% increase in business applications since business applications 371 priority priority priority 3 5 large scale Data Centers 150+ technology recovery exercises annually

3 Corporate Continuity & Client Services Continuity & Client Relationship Management North America & Asia Pacific Relationship Management between Business and IT Infrastructure / Continuity Problem Resolution / Escalation Business Project Capacity Tracking Compliance Education and Monitoring Recovery Exercise Support Continuity Due Diligence Client Presentations & Requests for Proposal BCM Quarterly Meetings Continuity Information Services Continuity Software Tools development and support Compliance Database Application Continuity Database Business Continuity Planning Database (LDRPS) Continuity Reporting Program Structure Off-site Storage Program Continuity & Client Relationship Management Europe, South Africa, India Relationship Management between Business and IT Infrastructure / Continuity Problem Resolution / Escalation Business Project Capacity Tracking Compliance Education and Monitoring Recovery Exercise Support Incident Management / Client Presentations Continuity Due Diligence Technical Continuity Services Continuity Support Services Enterprise Technology Recovery Solution Development and Implementation Continuous Availability Implementations Technology Recovery Exercise Planning & Execution Evaluation of System Recoverability Incident Management/Notification/Escalation/Debrief Maintain/Manage Emergency Operations Centers Process Continuity/Dependency Analysis Communications / Internal & External Training, Awareness & Simulations / Executives & BCMs CC&CS Website Corporate Continuity Conference LDRPS Training State Street Notify

4 n1 NEDRIX 2006 Annual Conference Technical Continuity Planning Ensures complete technical continuity solutions built from implementation, to testing, through problem resolution Technical Recovery comprised of 5 components: 1. Application Information 2. Completeness of Recovery Program 3. Sufficiency of the Solution (RTA vs. RTO) 4. Recovery Exercise Status 5. Recovery Plan Status

5 Slide 4 n1 Assumptios nevilkb, 9/26/2006

6 n2 NEDRIX 2006 Annual Conference Types of Disaster Recovery (DR) Plans Datacenter Main Frame Midrange Open Systems Desktop Clients/Vendors

7 Slide 5 n2 for the purpose of the presentation, we will concentrate on Open Systems A typical STateStreet Application Number of users Number of Clients ## of transactions $$ in daily transactions nevilkb, 10/3/2006

8 n3 NEDRIX 2006 Annual Conference Types of Technical Recovery Plans Actual Recovery Plan Plan that is executed during a real incident Recovery Exercise Plan Plan that is used during a test. Comprised of actual recovery plan plus steps used to protect the production environment while testing

9 Slide 6 n3 Purpose of writing a plan: Document, real event nevilkb, 9/26/2006

10 Technical Recovery Plan Process 1. Form a project team comprised of application and technology infrastructure support (server, storage, messaging, database, network) 2. Develop an Application Topology Diagram 3. Create the Recovery Exercise Plan 4. Create the Actual Recovery Plan 5. Conduct the Exercise

11 Topology Diagram Application Components Interdependencies (clients, users, other applications, vendors, etc.) Data Feeds Timing of feeds, number of feeds, etc. Data Transport method Data Replication Requirements based on objectives User Access Recovery gaps that need to be resolved

12 Topology Diagram Sample 1

13 Topology Diagram Sample 2

14 Actual Recovery Plan Steps to failover to Disaster Recovery and failback including: redirecting network to recovered facility start the operating system bring up the database activate messaging contact technical support (internal/external) to redirect electronic feeds) start job processing etc. Supporting Standard Operating Procedures (SOPs) for steps in plan (generic SOPs) Post Plans and SOPs on Common Repository. Site redundancy across two data centers Contact information is from Data Center Operations on call list

15 Recovery Exercise Plan Establish test date and time Business Unit defines test objectives Steps to failover to Disaster Recovery and failback (as discussed on prior page) Create steps to bring down and protect production while testing including: bring down the production application perform database backup prior to exercise hold job processing and feeds shutdown network connectivity as appropriate Identify contact information for the test

16 Sample of Technical DR Plan Steps Task Actual Actions Insert sample recovery Time START OF EVENT plan steps Duration Person/ Group Comments 1 An event has occurred in a SSC production facility CRM, BCM, IT groups contacted 2 EMG assess event based on feedback from the IT and BCM community. EMG 3 EMG elevates event to Red Alert Disaster declared. EMG 4 CCCS activates the Corporate Command Center. IT and BCM conference lines activated. 5 IT Management would notify IT support groups to initiate application recoveries IT 6 CSC notifies the appropriate support groups of the Recovery Event and that the Fund Transfer application requires recovery. CSC See contact list at the end of this plan. 7 Ensure that the following interdependent applications are being recovered: Open Systems1 - XXX Open Systems2 AAA Parallel Crystal Server Interdependent applications are involved in the same loss of site 8 App Support verifies with business unit that failover to recovery can begin. Notify CSC & CCCS that Application recovery can begin. Notify CSC upon completion Application Sup por t

17 Task Actual Time NEDRIX 2006 Annual Conference Sample of Technical DR Plan Steps Actions Duration Person / Group Comments C: S: F: Helpdesk to alter DNS to swap the ip addresses associated with the production and DR hostnames. dabc03 = XXX.XXX dabc03dr = XX.XX dabc04 = XXX.XXX dabc04dr = XX.XX CSC/ HelpDesk DNS will resolve the name for dabc03, axyz01 to the ip addresses of the DR (or production) servers SOP Open Systems Support to mount the file system on DR (xxxx2412) for the following nodes: dabc03 (xxxx2412) dabc04 (xxxx2412) CSC/ Open Systems Support DR server is prepared for application and DB recovery (files mounted, vvr link updated etc) SOP DR dabc03 SOP DR dabc04 CSC contacts WHS to start the Webserver (IHS) on xxxx2412: log on to xxxx2412 cd into /usr/local/web/httpd/sites/abc type sudo./apachectl start Notify CSC upon completion CSC/ WHS SOP Intranet user access via abc.statestr.com will be enabled C: S: F: CSC notifies DBA to do the following on the DR server (svda0083): 1.Bring up Oracle DB (O03ABC3) on dabc03/ xxxx Bring up Oracle DB (O04ABC3) on dabc03/ xxxx CSC/ DBA SOP DR O03ABC3 SOP DR O04ABC3 C: S: F: CSC contacts MQ Support to do Start MQ Series for dabc03 on svda0083 and for dabc04 on xxxx2412 CSC/ MQ Messaging SOP C: S: F: CSC requests ABC Support to bring up ABC applications on the DR server xxxx2412 for dabc03 and dabc04. CSC/ ABC Support SOP

18 Conduct the Recovery Exercise Execute Recovery Exercise Plan Establish Actual Recovery Time (RTA) Obtain Business Unit feedback on success of exercise objectives Document, classify, and report issues Develop solutions to resolve issues (exercise and actual) Report Compliance (5 components)

19 Questions?