North American Portability Management, LLC LNPA Transition Contingency Rollback Industry Discussion July 12 th, 2017
Agenda Introduction Transition overview, definition of rollback, and background Rollback Governance Who is involved, how decisions will be made and communicated Rollback Approach Overview of the plan, support provided, roles and responsibilities Conclusion and Next Steps Discussion 2
About this meeting Welcome to today s presentation! Objectives Facilitators Our goal today is to introduce the contingency rollback approach and seek feedback from NPAC Users This will be the first of several meetings to prepare for potential rollback following transition Greg Chiasson Principal Chicago, IL William Reilly Director Florham Park, NJ 3
Subject Matter Experts A number of subject matter experts are present today and will support the discussion Suzanne Addington NAPM Transition Team Sprint Inc. Teresa Patton NAPM Co-Chair & Transition Team AT&T Inc. Deborah Tucker NAPM Transition Team Verizon Inc. Dave Gorton LNPA Transition Team iconectiv LLC John Malyar LNPA Transition Team iconectiv LLC Paul Mazouat LNPA Transition Team iconectiv LLC 4
Introduction 5
Transition overview The NPAC transition will take place in three groups using a Prepare, Transition, Soak, Accept process Transition groups are defined to maximize testing and minimize NPAC user impact Transition in stages, in concert with the testing in stages, is designed to minimize overall transition risk 6
What is a rollback? Rollback means to reverse a system upgrade or transition so that the original system is put back in place This is a common mechanism that provides a safety in case there are unexpected problems with a new system Transition Transition Rollback Rollback Fix In Place For small or simple systems, rollback is straightforward For complex and interconnected systems fix-in-place is frequently a better course. However, a rollback plan is necessary as the option of last resort 7
What might trigger a rollback? Rollback would only be triggered by a catastrophic service outage which could not be fixed in a reasonable period Software failure Hardware or Connectivity failure Security-based failure Prevention Third-party security audit 24x7 security monitoring and threat response Prevention: Redundant data centers with failover (geographically remote) High availability architecture Practice data transfers Prevention High availability software architecture Exhaustive pre-transition testing Individual validation with each vendor / mechanized system Development team on standby The scale and complexity of NPAC services makes any type of rollback difficult preventing and mitigating catastrophic failure risk is core to the transition program design 8
How likely is a rollback? The new NPAC uses a modern, high availability design High-availability clusters Active-active geographic redundancy One-region-percluster isolation N+ redundancy within each cluster The NPAC components and connected local systems are being validated through multiple layers of testing Development QA IQA and ITE Vendor Cert Industry Testing Group and Round Robin Testing Performance Testing Security Testing The chance that a rollback will be needed is expected to be extremely low given the NPAC architecture, extensive testing, and fix-in-place capabilities 9
Rollback Governance 10
Who is involved in making a rollback decision? Industry participants All participants in the regional transition will be informed, via regular reports on status, and should monitor their own systems for incidents and performance. Impacted participants will be consulted if an incident is reported Transition Governance Team (TGT) A Transition Governance Team will monitor the transition. This group will be responsible for making a determination that a failure is catastrophic and for the decision to rollback TGT composition NAPM LLC Project Executives NAPM LLC Transition Team Transition Oversight Manager iconectiv NPAC Executive (accountable SVP) 11
How would the decision be made? Assess iconectiv will analyze any failure and provide an estimated timeframe for restoring service The TOM will provide a readout of additional industry feedback collected from participants The TGT will evaluate the impact of continuing operations, the resolution timeframe, and the risks of conducting a rollback versus fixing the system in place Decide If a decision is reached that a failure is catastrophic and that it is in the industry s best interest to roll back, then all participants and relevant stakeholders will be notified In the event of significant issues, the TGT will also decide if future Go-Live events will go ahead as planned or be delayed A failure is deemed catastrophic when it causes a loss of key service availability for all users in one or more regions and is not repairable in a reasonable period 12
How will it work? During each regional transition, performance of the new NPAC will be monitored. If a critical failure occurs, iconectiv will begin troubleshooting and the industry will be notified As the problem scope and resolution options are developed, the TGT will assess whether the failure is catastrophic and whether to rollback Rollback will be evaluated for each region separately 13
Rollback Approach 14
How was the rollback plan selected? The Transition Oversight Manager, NAPM Transition Team, iconectiv, and Neustar conducted an extensive review of rollback options over the past 18 months Develop Options A baseline approach was developed, with a variety of automated, semiautomated, and manual alternatives Analyze Feasibility Feasibility, including impact to users, schedule, cost, and contractual components was analyzed Review Alternatives Modifications and alternatives were considered to address challenges surfaced in analysis After thorough consideration, the NAPM selected an Industry-Led rollback approach as the best option considering technical, resource, schedule, and contractual constraints 15
What is Industry-Led rollback? In the event of rollback, NPAC users will submit porting and other requests in accordance with their own processes and priorities, i.e., the industry leads the effort In industry-led rollback, The Neustar NPAC will restart service Service providers will re-connect with the Neustar NPAC and then resubmit requests that were sent after the regional cutover. Requests should be submitted in accordance with the service providers own processes. The TOM and NAPM will provide governance and facilitation iconectiv will support service providers with individualized transaction reports and an associated user guide as a supplement to the service providers own records to assist resubmission work 16
How will industry-led rollback work? Performance will be monitored and all NPAC requests logged Monitor Assess Decide Act Transition Pre-Rollback Point of Failure Rollback NAPM TOM maintains open conference bridges and Webex for communications Transition completed Industry NAPM iconectiv Neustar Maintain Neustar NPAC system in quiescent, state Begin logging NPAC requests Begin monitoring performance Critical failure reported to TGT with description of the problem and impact, to the extent known at that time iconectiv engineering works to resolve Assess failure status Transition Governance Team Announce rollback on the open conference bridge Begin allowing connections to the NPAC, with data as-of the close of transition Transaction Report provided to each participant Re-associate with Neustar NPAC Begin normal porting. Evaluate the Transaction Report and begin resubmitting transactions to re-sync through the usual interfaces (SOA, LTI, etc.) If a rollback is declared, participants will coordinate and resubmit activity according to their individual needs and priorities 17
How are transactions resubmitted? Source Data Participants should maintain a list of requests in case of a rollback iconectiv will provide participants with Transaction Reports to assist in resubmission planning Transaction Reports for participants will be posted to the iconectiv SFTP site Options for resubmitting NPAC requests SOA (using operational teams to control request creation and flow of new work as appropriate). LTI GUI Other existing processes Resubmission will be at the discretion of each participant. Participants will cooperate with each other as per usual and submit requests in the method(s) most preferred by their operational (including customer care) organizations 18
Industry roles & expectations General The Neustar NPAC will resume normal porting operations after rollback. This may create an impact for some customers and generate care trouble tickets. Industry participants are expected to coordinate, resubmit, and resolve issues cooperatively Mechanized Users Neustar network configurations must be preserved to allow quick reconnection. LSMS and SOA systems will need to be re-associated LTI Users LTI users will need to revert to their Neustar LTI credentials. Since no local system is involved, a resync is not needed but LTI users should be prepared for some trouble calls and possibly slow/delayed porting from larger, more impacted, operators Ancillary Service Users No user action is required and little impact is expected 19
What support is available in the event of rollback? Both current industry processes and supplemental support mechanisms will be provided Current service provider processes for handling errors and resubmissions iconectiv-generated transaction report iconectiv transaction report users guide NAPM / TOM coordination bridge iconectiv help desk (transaction report support) Neustar help desk (business as usual issues) In the coming months, subsequent sessions with the industry will detail and explain support mechanisms 20
What are expected preparation timelines? After the approach is finalized, iconectiv will develop and test the supporting material and processes 2016 2017 2018 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Contingency Rollback Prep Option Evaluation Process Design Transaction Report and Guide Industry Operational Process Report Spec Sample Reports Develop Test Note: Dates are subject to change Today Ready mid-march 2018 Industry participants will use the transaction report specification and sample reports to prepare their operational teams 21
What challenges are expected? A rollback event will require extra effort from the industry Resubmission of transactions Resynchronization of systems Reconciliation of issues The greatest impact will fall upon high-volume, mechanized users Largest number of resubmissions SOA/LSMS resynchronization These challenges are recognized and have been evaluated by the NAPM, which has many members that would be among the most impacted by a rollback Industry-led rollback is the selected contingency rollback approach 22
Conclusions and next steps Our session today is an introduction to contingency rollback. Your feedback and input is requested Today, during the discussion session Within the next two weeks (by July 28) through the Comments feature on the LNPA Transition tab of the napmllc.org website The TOM will host an interactive session following the August 16 TOEP webcast to discuss provided inputs Additional contingency rollback working sessions are expected to be conducted at major milestones, e.g., availability of transaction log specification, rollback users guide, transaction reports, etc. Contingency rollback updates will also be provided through future TOEP webcasts 23
Discussion 24
Thank you for your participation