Business Continuity Disaster Recovery & Emergency Response

Transcription

1 Business Continuity Disaster Recovery & Emergency Response Gina Gavito, Director of Emergency Response and Business Continuity Roxray Adonis, CBCP, Director of IT Business Continuity, Disaster Recovery & CERT June 2013 Our Mission To create a culture of preparedness for all types of emergencies aircraft, natural disasters and man made events. To respond effectively to the needs of customers, co workers and enterprise during emergencies. 2 1

2 The World s Leading Airline Best new aircraft order book of U.S. network carriers The most modern and fuel efficient fleet Industry leading loyalty program Voted best program for nine consecutive years FORTUNE rated UA World s Most Admired Airline Best Airline for North American Travel for 2012 by Business Traveler Magazine 3 Unmatched Route Structure World s Most Comprehensive Network International gateways to Asia and Australia, Europe, Latin America, Africa and the Middle East Optimal hub locations in 10 cities Hubs in the four largest U.S. cities Founding member of Star Alliance Provides service to 194 countries via 27 member airlines 4 2

3 United Airlines by the Numbers Flights Per Day (UA & UAX) 5,472 Airports We Operate In 381 Continents We Operate In 6 Customers Transported Per day 200,000 Customers Flown In ,000,000 Aircraft Mainline 699 Aircraft Express 554 Total number of co workers 87,867 Fleet modernization: New Boeing aircraft in Annual Goal Setting Monthly Meetings Team Collaboration and Alignment Resource Training / Sharing Education Business Impact Analysis Emergency Response Plan Development Testing Response Recovery Leverage Skills Sets Business Continuity Disaster Recovery 6 3

4 Strong Stakeholder Support Across The Organization Planning Operations Technology Safety Executive Leadership Network Operations Airport Operations Enterprise Risk Management Corporate Safety Flight Safety Security Information Technology 7 Recent Business Disruption Events System Outage 11/06/ /17/2011 Computer Outages H1N Dengue Fever 2011 H1N Pandemics Hurricanes Mexico, 2011 Irene, 2011 Isaac, 2011 Sandy, 2012 Sequester, 2013 Government Funding Natural and Man made Emergencies Tornados DFW, 2012 STL, 2011 LOS, 2011 BOM, 2011 BKK, 2008 Civil Unrest HDQ - NOC, late 90 s ABQ, 2011 Fires Earthquake NRT, SFO, 1989 Flood BKK, 2011 OMA, 2011 MSY,

5 Emergency Response Plan Corporate Emergency Operations Center (EOC) Station Emergency Response Plan Command & Control for Emergencies Coordinates all actions involved in the response Dispatches Go Team Initial Responders Provides Immediate assistance to survivors, friends and relatives Telephone Enquiry Centre (TEC) Go Team Manage Calls from Friends & Relatives Manage incoming and outgoing calls to Friends and Relatives Process the Passenger manifest verification Humanitarian Support: Staffs Family Assistance Center 100+ volunteers (scaled), Departs within 3 hours of the incident Technical Support: A/C Recovery, Scene and Investigation 9 Key Events 2011 Missouri River Flood 2012 Hurricane Sandy Planning Response Recovery Lessons Learned 10 5

6 Key Events Missouri River Flood Hurricane Sandy June September 2011 Flooding OMA airport with eight feet of water if levee breached October 2012 Largest Atlantic Hurricane on record High winds spanning more than 1,100 miles 40 airports impacted, including two hubs over nine days Nearly 6,000 flights cancelled 11 Concerns of Missouri Floods and Hurricane Sandy Omaha Airport Co worker safety Ground equipment damage Total loss of airport if flooded Sandy Co worker safety About 25,000 co workers affected by storm About 14,000 in NY/NJ Area Damage to physical assets Aircraft, ground equipment, computer rooms 12 6

7 Planning Response Recovery Same Process for Different Events ER/BC Activate EOC Protect aircraft and ground equipment Transportation & hotel logistics Flight diversion contingency plans Communication plan Report status to business leaders 24/7 airline operations monitoring Activate volunteers Manage EOC Charter flights, employee outreach Employee status line and Intranet updates Status reports Deploy volunteers File insurance claims Provide employee relief services Employee status line Status reports Airport damage assessment Reposition aircraft, ground equipment DR Activate IT Response Team Protect IT equipment Pre stage IT Go Kit equipment Review recovery script Setup IT Go Team logistics Report status to EOC and IT Leaders 24/7 IT systems monitoring Activate IT Go Team Represent IT in EOC Provide IT escalation IT issue resolution Activate backup work area Status reports to IT leaders Deploy IT Go Team to impacted sites Perform IT damage assessment Provide expenses for insurance claims Replace/repair damaged IT equipment Reposition IT equipment 13 Preparation Omaha Airport Day 1: June 14 Request from station manager for IT Action Plan IT Airport Incident Response Team formed Protecting equipment, LAN rooms Staging IT recovery equipment Extra ground equipment moved to alternate airports Vital records relocated to concourse level Day 3: June 16 Action Plans for Recovery A. Time to power down all equipment B. Time to only remove critical IT hardware C. Total Loss Shipping of additional equipment to LNK (Servers, Kiosks) Day 5 and Beyond Continued monitoring of water levels OMA Website NOAA Army Corp of Engineers Daily Operations Calls All clear given in September Day 2: June 15 Creation of Emergency Notification List Creation of recovery script Sent recovery equipment to LNK airport, about 60 miles SW Flight diversion planning Day 4: June 17 Finalized logistics for equipment delivery from LNK to OMA for recovery activities IT resource transportation logistics finalized 14 7

8 Evolution of the IT Response Process Process Improvement OMA Flood 2011 Sandy 2012 Reactive Ad hoc process Process ownership absent Undefined roles and responsibilities Unsure which teams to contact Unknown equipment availability ER/BC coordination limited Proactive Established IT Response Team Established process Clear process ownership Defined roles and responsibilities Pre staged equipment Automated emergency notification list Strong ER/BC relationship Strong IT Leadership support Formalized post incident reporting IT Go Team volunteer list 15 Preparation and Response Hurricane Sandy Friday Oct 26 Activated IT Airport Incident Response Team Reviewed recovery equipment inventory list Sandbagging of flood prone areas in EWR and IAD Sent recovery equipment to EWR and IAD in a secure storage location Sunday Oct 28 Continued monitoring of storm EOC Activated Continuation of flight cancellations Shutdown of operations at select stations High Call Center Volume Tuesday Oct 30 Continued support in EOC Select airports shut down Initial damage assessment Monitoring of IT systems Gradual restoration of operations at line stations Normal Ops Saturday Oct 27 Network Operations Center IROPs Start of flight cancellations Aircraft removal to alternate airports Airline volunteer coordination Generators staged at major hubs Monday Oct 29 Storm Landfall Shutdown of operations at select stations IT Equipment delivered to IAD, EWR delivery unsuccessful due to road closures Wednesday Oct 31 Ferry flights to reposition aircraft and deployment of airport volunteers Deployment of IT Go Team to NYC metro area Damage assessment activities 16 8

9 Customer/Co worker Communication Plan for Events Toll free number established for providing updates to co workers Daily ops calls with Network Operations Weather, operational and airport situation reports sent three times each day Daily calls with IT response teams Corporate Communications Media support IT representation on daily operations calls Business operations representation on daily IT status calls IT Leadership (CIO, VPs, MDs) briefed of situation during the morning IT status calls ER/BC Lessons Learned Early activation of EOC Inclusion of all Headquarters EOC resources for large scale events Role Clarification / expectations in EOC BC events versus AC accident events Be clear with the field that EOC is there to support Provide direct communication lines to functional areas in the EOC Best conference call frequency/structure (NOC report out; station dialogue) Anticipate Issues Coordinate solutions carefully through the station / region to avoid overlap Shift Schedules for EOC to minimize fatigue 9

10 IT Lessons Learned Be proactive Remain flexible. Not all support activities will be the same, but follow the process Expect ad hoc requests from the business on securing additional IT equipment Creation of a recovery script helped to identify additional logistical steps that otherwise might have been overlooked Websites and social media such as NOAA.gov and the Army Corp of Engineers twitter feed are valuable information sources Define the single point of escalation for DR issues Coordination with local agencies is essential Questions Thank You! 20 10