High Level UKL Data Migration Approach Bulk Load, IDRs and Cutover Activity. April 2016

Transcription

1 High Level UKL Data Migration Approach Bulk Load, IDRs and Cutover Activity April 2016

2 Data Migration Approach Objective The objective of the slides is to detail the data migration approach for Bulk Load, Implementation Dress Rehearsals (IDR) and final cutover and describe the high level activities that Data Migration will undertake along with Transition team to ensure Data has been successfully migrated. In order to manage the transition during this period, a separate technical and business transition approach has been defined, which identifies what actions are being taken before, during and after the cutover period. The objective of Data transition is to migrate data that falls under Bulk data and cutover (as part of IDRs ) into SAP ISU production environment in an effective and timely manner with the highest possible quality using tested and proven approaches while taking into consideration the downstream impact on BW. The objective of Data cutover is to migrate data that falls under cutover from all relevant Legacy systems following start of Non Effective Day (NED) period into SAP ISU production environment in an effective and timely manner with the highest possible quality using the previously tested and proven approaches.

3 1. Background Overall Plan The UKL Data Migration has followed a Bulk and Delta Approach practiced through various Data Migration Test Cycles (DMTCs) which were designed to Migrate legacy Data through to a series of SAP Data Objects which serve as Target, To refine the process and timings of migration itself, To detect any defects that may be present and Include all sources of Legacy Data. (Refer to next slides) Data also has been provisioned to various downstream users such as UAT. Performance Testing and Market Trials for their testing. A number of Defects have been resolved both Data Migration and Source Quality through Technical fixes, Data Cleansing or via an Industry agreed Transformation Rule. These rules have now been incorporated into the Data Migration routines to form part of Bulk and Delta data uploads into Target environment.

4 Data Depth Sources of Data Data Map for Migration Standard Objects Data Breadth Custom Objects LPG 2k US 113 sites DM 1340 sites Other Sources of data being integrated into the SAP solution LSP 340k SSP 21.6m GT Data Daily/Non Daily Metered Sites IGT 1.96m igt Data is sourced via the IGT Repository and migrated into SAP Various SAP Object Modules (inc. Custom Build) to hold of Data Items from all Sources have been built iteratively as part of Data Migration

5 Data Cycles and Goals Stage Goal Downstream Learnings CT1 CT2 UAT DMTC1 DMTC 2.0 A DMTC 2.0 B DMTC 2.1 MT Component Test to understand relationships between each object SIT 1 Component Test to perform the first Integrated Data Object Testing SIT 2 Provide a dedicated load for use in UAT for all LDZs/Suppliers/IGTs 1 st Full Bulk (GT, IGT) Load to test performance Smaller load for finding defects ahead of UAT top-up UAT 1 N/A N/A Smaller load (220k) to find and fix defects proactively UAT 2 2 nd Full Load - Practised Load timings, Defects from UAT load are fixed and MT preparation Provide a dedicated load for MT, Ensure known defects are fixed and fallout is minimised ahead of MT. Performance Test Market Testing Object sequencing Custom Objects were needed to hold Xoserve specialist Data (e.g. Weather data) Sample testing provided functional insight into data Bulk load timings revised from 3 weeks to 6 weeks Multiple Defects uncovered through func. testing Business Testing introduced High number of defects found and rectified Performance tuning and load times achieved. Scripted Validation introduced Thorough profiling of Data and Semi automated validation was run on 6m data DMTC3 3 rd Full Load for full data breadth and Depth. Delta and NED practice None Ongoing.

6 Data Key Milestone PoaP April 2016 Go Live Feb 16 Mar 16 Apr 16 May 16 Jun 16 Jul 16 Aug 16 Sep 16 Oct 16 Data Workstream DMTC3 Defect Fixes Data Workstream has introduced additional Test Phases to perform Mini Data cycles for target areas and defects Mini Test Cycles Build & Unit test Complete Bulk Load IDR Prep C/O Data Loads Delta Load Application DMTC 3.0 Complete Bulk Data Load Start Data Validation Bulk Data Load Complete Delta Data Loads Complete Transition & Cutover Period Cutover Transition Workstream Low Level Transition Clarifications Completed (TPG) All Transition MODs Approved 19/09 Go / No Go Decision (PNSG) NEDs Non Effective Days 24/09 30/09 Complete On Track At Risk Will Miss Under Review

7 2. Data Migration Key Principles Some Key Principles to be adhered to in Bulk Load, IDR and Cutover Cycles Legacy Data Data Migration is to ensure all requisite Legacy Data is taken across from Legacy source to SAP Target system ready for Go-Live. All Data necessary for operating new world process (in line with Source to Target Mapping) is aimed to be taken across during Bulk and Cutover cycles. This includes Data that may have been highlighted for a quality issue. Data Cleanse and Transformation Rules Data Migration will ensure that any cleansed Data being sent across by market participants to Legacy systems in line with the current Data Migration plan for Bulk and IDRs is migrated into SAP solution. All agreed Transformation Rules are to be applied to the participant Data at Bulk/IDR stages. Pre Migration All known Data Defects are acted upon, fixed where necessary and proven before applying to the To Be Target System. Data Migration Code is frozen to deploy Bulk Data Load and subsequently IDR loads and strict Change Management/Governance is followed for all Data Exceptions beyond this point Bulk Migration All data migration activities that result in the transfer of data from legacy systems to Target system are reconciled to demonstrate the completeness of data through the end-to-end migration process Migrated Data is sufficiently validated to ensure it meets quality criteria and can be functionally used for processes. Details about the rejected records and corresponding error descriptions will be captured during each stage of Data migration. The process to manage such rejections & subsequently correction of these rejections is termed as Fallout Management Defect Management All fallout of Data items is managed via the Defect Management Process Defect Prioritisation is followed to ensure key Defect categories from above are prioritised and fixed in bulk load Any MPRNs that fail to make it into Target System are expected to be go through Data Migration Re-runs to ensure successful migration of all legacy Data Individual Portfolios that cannot migrate beyond this point will be published to participants for resolution as necessary Xoserve will work with market participants as necessary to find a resolution for any outstanding data migration failure Cutover In Flight Transaction Migration is expected to take place alongside transition team to migrate all in flight process data where it is data related Post Go Live, previous month's historic Invoices are expected to follow a separate Migration run into Target System

8 Data Migration Plan 1. Plan elements describing Bulk, IDR, Cutover 2. Sequential Model through Bulk Data Load, IDRs and Cutover 3. Data Controls and Exception Reporting 4. Production Controls and Release Management

9 Data Migration Plan Data migration will happen in the following 3 phases until go live and covers all data elements. Bulk Load Implementation Dress Rehearsal (IDR) Cutover (Non Effective Day window) Bulk data migration will involve the extract, transformation and load of the full scale Legacy UK Link data into SAP - ISU. This will be initiated from 16 rd May 2016 as per current plan. The high level approach to be followed is outlined below: Bulk Management - Technical Overview Defect Management

10 Data Migration Sequential Model for Bulk, IDRs and Cutover Legacy Source for Bulk & IDRs Data required by SAP resides in a number legacy sources. In some cases source data is further sub-divided into logical groupings. This is done to make the Extract, transform and Load (ETL) process more efficient and aid sequencing. The following list shows the sources (and logical sub groupings): UK Link [NDM sites] UK Link [DM sites] igt Migration Database LPG Offline data DM CSEPs Unique Sites Invoice Data Data Load Sequencing Due to the extremely large amount of data being migrated from the UK Link system it is not possible to perform a full migration during the agreed Non Effective Day (NED) window. In fact, even when fully optimised, it is expected that the end-to-end process from data cut at source to final data loaded to SAP takes approximately 6 weeks running on a 24/7 basis. As such the approach being taken is to perform a Bulk Load with a data cut Mid May. Following the data-cut, the UK Link database is automatically monitored for changes, and those changes are captured in files to be used in the migration. Other sources are smaller in nature or more difficult to automatically monitor and capture change. These sources will be captured and loaded during the NED window

11 Data Migration Sequential Model for Bulk, IDRs and Cutover The following diagram depicts the schematic sequence of load planned into the remaining cycles, by source. Bulk Load Phase IDR Phase Non-Effective Days Phase Post NED Phase UK Link [DM Sites] UK Link Delta (DM and NDM) UK Link Delta (DM and NDM) Historic Invoice Documents UK Link [NDM Sites] All other Sources - practice LPG Data Unique Sites Offline Data igt Data (Transactional) Logical Model of Data Migration through Bulk, IDRs and Cutover

12 Data Controls Data Controls are in place for Data Transition Management via Data Governance forums (Internal and External) Overall Transition Plan and controls Fallout Management Defect Management Lifecycle Data Control Approach - Checkpoints for Data are in place designed around Bulk load, IDR and Cutover Entry and Exit criteria Go/ No Go decisions are aligned to Transition plan dates Regular Progress Reporting to Industry is to be published via this group and any new governance structure Throughout a particular migration phase, there are standard and robust error handling processes to cater for errors at any stage of the migration process, (e.g. file failure due to technical issues encountered during Extraction). Any Reload and Rerun of Data is managed as part of Workstream and Programme wide Release Management controls Fallout Management will be triggered post Bulk and each IDR to manage all exceptions Fallout reporting will be undertaken by Xoserve where a Data Migration solution is not found for a particular set of MPRNs

13 Defect Management Defect Management The process of Defect Management has been practiced through various data load cycles which were designed to identify all fallout through the migration itself, to detect any defects that may be present post migration and test for known defect fixes deployed. Defect Management covers these defect types - 1. Data Migration - The failure of the Extract, Transform or Load process for technical reasons; 2. Quality where the source data is not available or is in format which does not allow it to be migrated 3. SAP functional fit - In some cases the target system itself requires data in a certain format or some mandatory data which may not fully exist in source Defect Management Lifecycle will follow previous best practice of managing Defects and will follow an Agile fix Trajectory to resolve Bulk Defects via ETL/SAP BAPI tools. Full Portfolio Reconciliation is to take place as part of standard practice and Bulk Load will follow the previous best practice of Defect analysis and communication as per MT load. Impact Assessment of known defects will be undertaken to understand scale and impact and the existing/new Industry Engagement forums will used (e.g. Data Management Group) to publish and seek resolution where necessary of any late issues encountered in transition.

14 CCC & GCC Go Live Implementation Window New UK Link, SAP BW & DES Go Live 25-Apr May May May May May Jun Jun Jun Jun Jul Jul Jul Jul Aug Aug Aug Aug Aug Sep Sep Sep Sep Oct Oct Oct Oct Oct Nov-16 ###### ###### ###### D - 18 D - 17 D - 16 D - 15 D - 14 U K L I N K P R O G R A M M E - R E V I S E D H I G H L E V E L C U T O V E R P L A N D - 13 D - 12 D - 11 D - 10 D - 9 D - 8 D - 7 D - 6 D - 5 D - 4 D - 3 D - 2 D - 1 Week Cutover Week (D) D + 1 Week D + 2 D + 3 D + 4 D + 5 D + 6 IDR 1 IDR 2 IDR 3 Old to New New File Formats DMTC 3.0 Prod Env available Bulk Data Migration (SAP ISU) SAP BW Data Load 7 Noneffective days IDR Preparation NED & Cutover Simulation Plan Updates Early Productionisation of Service Management & Infrastructure components NED & Cutover Simulation Plan Updates NED & Cutover Simulation Plan Updates Post Implementation Support Delta Data Migration for Sept 2016 Invoicing processes in Legacy UK Link Key Assumptions: SAP ISU & SAP BW Production Environments are available by 15 th May 2016 Unique Sites UAT is complete by 27 th May Non effective days MOD is approved Final ISU Bulk Data Migration can be initiated on 23 rd May 2016 ISU Bulk Data Migration can be completed in 4 weeks SAP BW Bulk load can be completed in 5 weeks Delta Data Migration can be completed in 1 week Key Principles: Minimum of 2 consecutive successful idrs 1 st of Month Go-Live Minimum of 1 idr post Market Trials of Unique Sites 4 th Contingency idr not planned. If idr2 or idr3 failed, implementation would move to a Contingency date, i.e. the 1 st of the following calendar month. Cutover Preparation Cutover Window - DDM - Config. - AQ Business QA - Data Check - Go / No Go - Go Live / CCC Impl Post Implementation Support Legend: CMS - Contact Management Service CCC - Contact Management Service Consequential Change Workstream Config - Configuration Ctgy - Contingency DDM - Delta Data Migration GCC - Gemini Consequential Change Workstream idr - Implementation Dress Rehearsal Impl - Implementation UNC MOD - Uniform Network Code Modification Proposal PT - Performance Testing QA - Quality Assurance SAP BW - System, Applications and Products Business Warehouse SAP ISU - System, Applications and Products Industry Specific Solution for Utilities GCC Code Drop Final GCC Config Management of In-flight D + 5 Closeout Post Implementation Support - Database Backup - Database Refresh - Delta Data Migration

15 Data Migration Communications Progress Against Plan Updates Regular Progress Reporting to Industry is to be published via this group, UKLC, PNSG and UKLIEF New governance structure and subsequent channels may need to considered in the coming months Key Contacts Emma Lyndon Development Manager Padmini Duvvuri Senior Project Manager