For ACP-South Texas chapter program meeting in October 2012 only. Do not cite, copy or distribute without the author's consent. 1

Transcription

1 Agenda Company and Program Overview The 4Ps of DR Planning People Property Process Performance Application Lifecycle Management Angela Mestre, MPH, MBA, CBCP, CISA, PMP Education Director, ACP - South Texas Company Profile Peace time War time Challenges, Lessons learned and Conclusion IT Continuity and Disaster y Program Milestones Established in 968, privately held Enterprise Products Company and its affiliates own significant equity interests in three publicly traded partnerships with a combined enterprise value of approximately $34 B.

2 Program Overview DR Phase I Goals ( ) IT Continuity and Disaster y Program Project in establishing the DR Program Goals & Objectives Strategy Plans & Procedures Vital Records Define strategy, policies and procedures Centralized DR Documentation Infrastructure and Data Establish infrastructure resiliency and DR capacity Establish data and server backup practices People Property Performance Process DR Planning Teams IT Essential Employees Area Business Owners Data Centers IT Assets End User Equipments Application RTO/RPO/MTD Infrastructure Resilience Lifecycle Mgmt y processes Support & Logistics Prepare Test Maintain Centralized DR Documentation Library IT Continuity and Disaster y Program Centralized SharePoint site and (for DR) Announcements Project status Critical Event Calendar Emergency Operation Sub-site Incident log & lessons learned Checklist and cheat sheets Essential employee list First responder team roster Phonebook & call tree Establish and maintain DR documentation Establish and conduct functional DR tests Business Mitigate Program Management Application IT DR Standard, Plans & SOPs DR Strategies Disaster declaration process IT Essential Employees Application system recovery plans Vital Records Library Infrastructure profile Critical application sytems and data profile DR testing documents DR failover documents Execute ongoing exercising and training Business engagement in continuity planning Program Planning & Disaster y Teams DR Oversight Committee DR Leads and Liaisons DR Testing Teams IT DR Liaisons and Business Owners Infrastructure project manager as PoC IT and Business testers IT Emergency Response Teams (ITERT) Coordinate emergency preparation and response activities Primary consist of IT Managers in each division IT Essential Employees (ITEE) Defined roles and responsibilities Primary consists of primes and backups in supporting critical systems People 2

3 People Contingency Planning DR Data Center & IT Assets Planning for Absence of IS Employees Data Center Profile Environment Site Protection Access control Identify core people and core skills Documentation & knowledge management Preparing for Workplace Closing Logistics for telecommuting Infrastructure External connectivity Resilliency (No SPoF) Logistics for office closure Logistics for emergency communications Planning for Emergency Response Operations DR Application Systems Pre-loaded Tier Apps Data backup for Tier 2 Apps All employees Corporate EOC External: vendor, customer, media, investors Telecommunications Planning for Failover Operations IT Essential Employees Property First ers 2 DR Definition, Scope Limitation, & Prioritization Requirement Identification & Implementation DR is defined as a service disruption resulting from the total loss of the Houston Maximum Tolerable Downtime (MTD) and Application RPO & RTO Data Center that last for longer than 5-7 days Infrastructure Resiliency DR Data Center was designed and built to meet a utilization level of 0-5% of normal production system capacity If failover is initiated, IT should plan to operate from DR Data Center for a minimum of 3 weeks before failback procedures are initiated TYPE RTO < 24 Hours (Mission Critical) Tier Application Systems 8-48 Hours Apps Implemented (Mission Critical) with High Availability -3 Days (Vital) Performance Automated Network Infrastructure 3-7 Days (Essential) RECOVERY STRATEGY Manual Support Center, NOC, Citrix, Internet and Intranet, and critical infrastructure application systems. DNS Alias update. Warm server, application log updates, SAN replication. Warm server, application log updates, SAN replication. from backup tapes or VTL. Warm server, application loaded, recovery from backup tapes or VTL. 3

4 Process Standardization & Documentations Application Inter-dependencies and Sequencing DR Asset Lifecycle Management DR Test Planning, Support & Logistics Incident Response Support & Logistics Ongoing Process Improvement and Governance Activities Steering committee review DR Application Lifecycle Management Formalize planning activities Corrective actions Opportunities for improvement beyond DR Prepare Mitigate Process DR Asset Approval and On-boarding Test Maintain DR Asset Approval and On-boarding DR Assessment DR Strategy Determine the business areas served and group of clients List components critical to support minimum levels of service Prioritize essential business processes and determine attributes for IT Management review and decide RTO/RPO and approve as new DR associated IT processes Business process and its criticality in operation value chain List of service dependencies and interfaces Commitments to other organizations (SLAs) Cost of failure over time Business agreement and executive approval on MTD and DR solutions Execute DR Solutions DR system setup and validation Identify the essential IT processes and functions associated with each business processes General description of process, frequency and manpower IT facilities and non-it facilities Vital/critical records Other systems, departments' dependencies application DR Options and cost estimate (cost/benefit analysis) Database & system replication Security & change management Documentation, Testing & Maintenance First year limited application level testing Second year roll into Annual DR Testing Prepare Mitigate 4

5 Energy Industry Value Chain E&P Crude Oil Production Midstream Commercial and Financial Applications (Pipeline, Barges, Trucks) Nature Gas / NGL Production (Natural Gas Pipelines) Crude app (Pipeline, Barges, Trucks) (Natural Gas Pipelines) Crude Oil Storage Natural Gas Processing Plants (NGL Pipelines) Crude Oil Refining Natural Gas Storage NGL Storage Refined Product Storage Natural Gas (pipelines) NGL Fractionation Midstream Crude app Crude Oil Storage Crude app Natural Gas Processing Plants (NGL Pipelines) Gas App Crude Oil Refining Midstream RP (Barges, RP pipelines) Refined Product Industries Gasoline addictives Power generation fuel EndConsumer Motor gasoline Petrochemicals Gas App RP (Barges, RP pipelines) 2 Days 3 Days 4 Days Gas App Natural Gas (pipelines) NGL App NGL Fractionation NGL App Application RTOs Based on Inter-dependencies Day NGL App RP App End-use Consumers 2 Hrs NGL Storage Refined Product Storage Industrial / Residential fuel Downstream Financial App Gas App Natural Gas Storage 5 Days 6 Days 7 Days Application Profile 7 Days - 2 Days Infrastructure Support Gas Apps Crude Apps Refined Apps NGL Apps 5

6 Functional Testing vs. Failover & Failback Functional DR Testing DR Test Testing objectives consist of: Network Validate DR applications and business processes Verify the adequacy of team collaboration procedures Server Validate that application documentation is current Document deficiencies and lessons learned Rehearse for continuous operations Timeline Test Prep 6/27 DR environment refreshed 6/28 Take Oracle DB Snapshots 6/29 7/5 Stop SQL DB and other backup jobs 7/ 7/6 Complete other infrastructure Preparation Testing 7/ 7/5 IT Application Testing at home 7/8 7/20 Commercial User Testing at home 7/20 7/22 Financial User Testing at home DR Resumption 7/26 7/28 Return to DR-Ready Mode Comm Data Apps Failover Active Directory (AD) Point to DR Internal DNS Records Point to DR Citrix Use DR Citrix URL Use Production Citrix URL Citrix Icon - File Manager Modify pointer Citrix Icon Outlook Do not use File Servers Modify pointer & shortcuts Restore - Outlook Do not use - SMTP Modify Failover RightFax Oracle DB Modify Snapshots Restore Restore SQL DB Aged database Restore Sequencing Required Backup Preparation Onsite backup only Modify frequency Application Startup Modify DNS aliases, SMTP, Fax Server, etc Script or manual process Test Lessons Learned From Past DR Tests IT Support Feedback Availability / Capacity / Demand Management Change Management Information Security Management DR Test Format Test on site vs. from home vs. outside of Enterprise network Need to improve process to track progress or validate test Documentation, Planning, and Communications Need to separate DR test and DR failover procedure documents Need to include test setup steps in the IT Test Plans Provide detailed support procedures for business testers Have face-to-face meetings with testers prior to test Improve 6

7 Process Improvement Initiatives Annual Documentation Refresh Checklist DR Documentation Ownership Record management Review and Signoff DR Plan and Policy Update Application Documents Server setup and startup procedures Production and DR server Mapping Failover checklist Vital records Review and Validate Infrastructure Support Documents DR Citrix farm Backup and replication User security management Update Testing Documents Test scope and procedure documents Functional test - Infrastructure preparation checklist; IT support and user test plan Failover test - Component failover test plan Process Improvement Projects (DR & Beyond) User access management Mobile telecommunication recovery planning DNS Alias Standardization DR Subnet Standardization & Isolation Change Management in DR Automation and Scripts Backup High Availability testing 28 Hurricane Planning and Response Pre-staged (Pre-Storm) Est. # in SA (Drive-ups) NOC 3 Est. # in Houston (On-sites) 3 Desktop 2 Network Server 2 2 DBA - ORACLE CIO s Announcements To all IS To area business owners Reminder to IT Essential Employees Update hotline and contacts Deploy pre-staged DR teams DBA - SQL PM Hurricanee Response Early dismissal Accommodations Communication tools Equipment and supplies Plans & procedures Plan for second waves No proactive system failover for weather events Disaster declaration is a business decision DR is defined as a service disruption resulting from the total loss of the Houston Data Center for longer than 5-7 days If failover is initiated, IT should plan to operate from DR Data Center for a minimum of 3 weeks before failback procedures are initiated Pre-Storm (72-96 Hours ) Preparation Alert Level I Equipment Supplies Setup Pre-Storm (36 Hours) Preparation Alert Level II Evacuation On-site Drive-up Pre-Storm (24 Hours) Preparation Alert Level III Emergency shutdown Transfer of control During-Storm Emergency Response Alert Level IV Stay safe Emergency shutdown Post-Storm Disaster y Damage Assessment Restoration Failover Mitigate 7

8 Timeline DR Phase I Failover Scenario 24 Hours Prior Day Day 2 Day 3 Day 4 Day 5 Day 6 Timeline DR Phase I Failback Scenario Day 7 Day Day Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Event Normal Ops Response and y Work-Around WorkAround Support Fail-back Infrastructure Applications Data/System Data Data/System Data Failover! Applications Infrastructure Support RPO RTO WRT RPO MTD RTO Normal Operations MTD DR Phase II Goals (200 -?) Phase II Activities New Objectives Strategic Review Capacity increase computing capacity at DR Data Center DR Data Center Upgrade Speed failover and failback quickly Application Failover Testing Isolation failover on individual app/server basis Automation automate manual tasks/processes Proof that we can failover and failback DR Scope Change Determine time taken for each application components Infrastructure and Data Data center partially down Increase to 25-50% of normal production system capacity Proof that reverse synchronization process is working properly Proof that regular procedures are working properly in test mode Application Systems failover with interface and component testing High-availability plan for each component for each app Proof that Users can use the system to conduct work at acceptable performance level Program Stay away from high traffic time to minimize user impact 8

9 Conclusions Strength Weakness. DR strategy and plans are based upon sound assessments 2. Integrated with corporate Safety and Emergency Operations. 3. Exercised on a regular basis 4. Continuous process improvement goes beyond DR 5. DR Phase II initiatives. Outdated BIAs 2. Lack of Business Continuity Oversight 3. Limited Executive Support in BCP 4. Not integrated with corporate Risk Management group 5. Lack of mass communication 6. DR Testing Environment is not isolated Opportunities Threats. Reorganization to incorporate Business. Lack of collaboration with supply chain Continuity vendors 2. MIR3 implementation 2. Lack of Service Level Agreements 3. Vendor support and consultation during Production Data Center build-out 9