Ever Vigilant. The Pursuit of Enterprise Resiliency

Transcription

1 Ever Vigilant The Pursuit of Enterprise Resiliency Justin Williams Network Disaster Recovery - International AT&T Network Emergency Management, Preparedness & Response

2 AT&T Network Operations Mission Ensure that the AT&T Network connects people with their world everywhere they live and work and does it better than anyone else Provide Comprehensive and Seamless Customer Application Service Quality, by preventing, detecting and correcting failures in the network before they impact the service experience. 2

3 Network Scale & Scope $98 billion AT&T capital investment in the US in the last five years 4,000+ MPLS nodes serving 187 countries 955,886 Miles of network fiber 107 million Wireless Customers 402,000 Wi-Fi hotspots accessible worldwide 757 billion Text messages handled over the past year Industry Leadership AT&T Rated a Leader, the Highest Ranking in the 2013 Current Analysis Global Managed Mobility Services Report (January 2013) Frost and Sullivan 2012 North American Business Choice Award for Overall Best Mobile Workforce Management Software (September 2012) AT&T Recognized As the Top Provider of Managed IP Telephony Services in the 2012 Nemertes PilotHouse Awards (August 2012) AT&T Recognized As the Top Provider of Internet Connectivity Services in the 2012 Nemertes PilotHouse Awards (August 2012) AT&T is the Frost and Sullivan Global Leader in Providing M2M Connectivity (August 2012) AT&T recipient of the Frost & Sullivan U.S. Dedicated Internet Access Market Share Leadership Award (August 2012) AT&T is a Leader in Gartner s Magic Quadrant for Global Network Service Providers for the Eleventh Consecutive Year Setting the industry standard for network reliability 100% Fortune 1000 companies are AT&T customers over 50% of the FTSE companies, over 85% of the CAC 40 companies, over 80% of the DAX companies & over 60% of the Nikkei 225 companies. 3 million Small businesses served by AT&T 43+ petabytes of data transmitted by AT&T on an average business day 38 Internet Data Centers on four continents 3

4 External Pressures Process, Tooling, and People guard against these threats; only a handful make it through to impact the network. Software Bugs Network Changes Planned Maintenance Events Hacker Attack Indicators Cable Cuts Hurricanes/Typhoons, Natural Disasters Political / Social Impacts & Events 4 Hardware Failures

5 Securing Our Network NETWORK RISKS Am I Ready? Are You Ready? Are We Ready? PHYSICAL LOGICAL ECONOMIC POLITICAL SOCIAL 5 Weather photos courtesy of NOAA

6 AT&T Emergency Management Due to the global reach of our network, AT&T has significant experience in dealing with a number of different disaster types; preparation is the key. Phase 1 Phase 2 Phase 3 Planning and Preparation Monitoring during Event Damage Assessment and Restoration A key focus of our ongoing process is to always identify key lessons learned after each event to continue strengthening our readiness and response 6

7 AT&T Network Continuity Best Practices Assets Personnel Process Practice Practice Practice Practice Learn Monitor & Manage Events Respond, Resume, Recover, Restore 7 RCA/Process Improvement

8 AT&T Network Disaster Recovery Specially Designed Recovery Equipment More than a $600 million investment in over 320 trailers and support vehicles containing network technology, infrastructure, and support elements. 8

9 AT&T Network Disaster Recovery 9 Theodore Vail Dedicated Teamwork Well-practiced and tested team of telecom disaster recovery professionals supplemented by a large pool of volunteers who train together during quarterly field exercises.

10 AT&T Network Disaster Recovery Recovery Technology Trailers 10

11 AT&T Network Disaster Recovery Power & Infrastructure 11

12 AT&T Network Disaster Recovery Command Centers & Emergency Communications 12

13 AT&T Network Disaster Recovery Support Infrastructure 13

14 AT&T NDR Program Structure Recovery Operations Wireless Operations Technology/ Disaster Planning & Implementation Deployment N etwork D R isaster ecovery Emergency Communications Special Operations 14 Support Functions Warehousing & Maintenance Incident Management Customer/Agency Outreach

15 What s going on outside in the car park? Full simulation exercise with heavy focus on training Using four technology trailers; all technologies are being configured to match a UK site Ten Frame and ATM switches linked through high speed fiber optic links to additional trailers. Four high speed routers fully meshed with fiber optic links. Two CRS1 core routers linked to trailer 118 Fujitsu FW7500 DWDM solution layer 1. Back up generator power for all trailers. Satellite-based communications for command and control functions (voice and data). Training of new volunteers and testing of process and procedures

16 Logistical Challenges Import Export Duty / Taxes Movement of controlled equipment: overcome by export licensing Moving 30+ tons of equipment around the world Rapid deployment Global power differences Dynamic nature of events 16 16

17 AT&T s PS-Prep DHS Certification The PS-Prep certification from the U.S. Department of Homeland Security (DHS) validates that AT&T is able to maintain or recover its own business operations in the face of an emergency or disaster, whether natural, manmade, or cyber in nature. AT&Ts preparedness measures include: Documenting and practicing strategies and procedures Creating and practicing a comprehensive response, recovery, and restoration process Creation and maintenance of an incident command structure Private organizations across the country play a vital role in bolstering our disaster preparedness and response capabilities, said Secretary Napolitano. I commend AT&T for achieving PS-Prep certification, and encourage other private sector partners to work with the PS-Prep program to further enhance the readiness and resiliency of our Nation. 17

18 Recovery in Operation Case Studies New York World Trade Center Hurricane Katrina Rome Deployment Earthquake in Chile AT&T AGN Node Recovery Santiago, Chile 18

19 September 11, 2001 New York City Network Call Attempts reached a high of 431M, 30% higher than the previous of 330M. AT&T transport node in the 6 th sub-basement of the World Trade Center s South Tower was totally destroyed (equivalent of 2,000 T3s of transport capacity). Three AT&T local switches in nearby buildings damaged. 19

20 September 11, 2001 WTC Response Simultaneous Recovery/Restoration of the Destroyed Office & Support of Authorities Emergency Communications Vehicle 1 Police Plaza, New York City NDR Recovery Site for NYC/WTC Local Services Jersey City, NJ 20

21 Hurricane Katrina s Local/Wireless Impact 2005 Damaged Infrastructure 21

22 Rome Deployment:

23 March/April 2010 Chilean Earthquake / AGN Node 23

24 AT&T Network Emergency Management In Summary Emergency response situations come in many forms and create very different challenges We have a great deal of experience in dealing with a broad range of emergency situations Advance planning and preparation are essential to our ability to respond effectively AT&T has the best disaster response resources and capabilities in the industry We invest in those resources in order to maintain a high level of readiness 24 24

25 Am I Ready?

26 Are You Ready? 26

27 Are We Ready?

28 28 ENJOY THE EXERCISE TOUR & COME BACK FOR THE OPEN FORUM!

29 29