Data Migration Plan Updated (57) Fingrid Datahub Oy

Transcription

1 1 (57) Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 FI Helsinki FI Helsinki

2 2 (57) Sisällysluettelo Definitions Summary Introduction Datahub Purpose of the data migration plan The Datahub project's data migration project and its goals Logical structure of the data migration system Preliminary information survey of market parties Phases of data migration work Procurement and planning phase of data migration work Pilot phase for data migration work Data migration work in the Datahub implementation phase Datahub system commissioning phase Tasks and responsibilities Data migration process General description of the process Data being transferred Migration file data content Migration file format Data delivery responsibility and rules concerning data selection Data import, check and correction proposals Importing data to the data migration system Data to be checked Error types Quality requirements for data during the pilot and data migration phase File check Integrity check Consistency check Data export to Datahub Monitoring and monitoring tools Import and check monitoring Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

3 3 (57) End user reports Summary report for the check Error report for a check Datahub data migration reports Main user's reports Delivery statistics for migration files Party-specific quality statistics Source material quality report at the Datahub level Final data migration report Data migration deviation report Data migration process performance Change management Data migration system architecture Architecture requirements Functionality Reliability Usability Efficiency Maintainability Portability Data security Technical configuration of the data migration system Requirements for storage capacity Data protection Observations about data quality made during the specification phase Accounting point data Customer information Contract information General observations Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

4 4 (57) File history Revision Made by Date Description 0.9 Data migration working group 08/06/2016 Draft for industry comments 1.0 Data migration working group 30/09/2016 First official Finnish version /12/2016 First official English version Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 FI Helsinki FI Helsinki

5 5 (57) Definitions Terms and abbreviations Attribute Business process Consistency check Data entity Data migration iteration Data migration system Data migration working group Data model Data standard Datahub Entity File check Fingrid Datahub Oy Integrity check Iteration Migration file Pilot iteration Description One part of data included in the data entity, for example, a customer ID for customer information. A group of tasks linked to each other, which are done in order to achieve a set target, for example, the switching of a customer's supplier. A check performed by the data migration system on the entire source material. The system checks that the source material delivered by different market parties is unambiguous and non-contradictory. For example, the check requires that accounting points may only have one valid sales agreement. The consistency check is a different concept than the "integrity check" that is performed on the source material delivered by a single market party. Data in the information systems of the market parties, for example, associated with customers, accounting points and contracts. Iterations completed with all market parties that are performed after the pilot phase. A system in which the data coming from the source systems of market parties is checked and any deviations are reported back to the market parties. The data migration system provides summary information to Fingrid on progress of the migration process and which stage each market party is in. A group of 11 pilot companies (pilot group) selected by Finnish Energy that, together with Fingrid and the data migration plan partner, participates in specification of the documentation associated with data migration and in the data migration process. The data model is an abstract model that organises data attributes and specifies how they are related to each other. A specification that describes the entities and the attributes of the entities of the Datahub data model. A centralised information exchange solution for the Finnish electricity retail market. A data set comprising several attributes. A check performed by the data migration system on a single file. This includes a syntax check, logical check of data content and duplicate tests. A subsidiary established for the operational activities of Datahub and wholly owned by Fingrid. Hereinafter the shorter "Fingrid" version will be used in the specification. A check performed by the data migration system on the source material of one market party. The system ensures that the source material supplied by the market party is complete with regard to the data model in the data standard. For example, the check requires that an accounting point ID in the contract information is also found in the accounting point data (referential integrity check). The integrity check is a different concept than the consistency check, which examines the consistency of the source material at the market level. A repetition of a correction/delivery/upload cycle performed during implementation of the data migration system. A file used in data migration that is used to export data from the source system to Datahub in the specified format. An iteration performed with a pilot group, which develops the data migration system and measures its readiness. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

6 6 (57) 1 Summary In the Datahub, data migration refers to uploading the basic business process data and metering data from the source systems to the new Datahub system prior to its commissioning. The source systems are business applications of the electricity grip operators and electricity suppliers that maintain customer, accounting point, contract and metering data and information. The data migration work has two main goals: Harmonisation of the data delivered from the source systems Implementing the initial upload for the Datahub system Harmonisation of the data coming from the source systems is essential in order for the systems to be able to perform the business processes in the Datahub system after its commissioning. The source data for the data migration comes from very different business applications and some 200 companies operating in the electricity market, all of which have different practices for maintaining basic business process data. In order to harmonise the source data and ensure quality, a separate "data migration system" will be procured and serve as a pre-processing system for the data uploaded to the actual Datahub system. The market parties will deliver the source data to the data migration system as migration files in the format specified in the data standard. Once in the data migration system, the following check procedures will be performed on the source material: File check: The migration file delivered by the market party is in the format required by the data standard Integrity check: The source material delivered by the market party is internally complete from the perspective of the Datahub business processes Consistency check: The source data delivered by the market party is consistent in relation to the material delivered by the other parties. The data migration system will be built so that the market parties can perform the migration file-specific and party-specific check procedures independently regardless of deliveries by other parties. The data migration system will produce check reports that the market parties use as the basis for performing the necessary corrective measures in the source systems. The data migration system does not revise or enrich source material data, so all corrective measures are performed in the source systems. The data migration system meets the same data protection and data security requirements as the actual Datahub system. Fingrid is responsible for performing market-level tests. Based on its results, a decision is made on uploading the source material to the actual Datahub system. Datahub's business processes require that all market parties deliver complete data during the data migration. The supplier of the Datahub system is responsible for uploading the transferred data to the Datahub system. After uploading, the Datahub system will provide a data migration report that is used to ensure that the data added to Datahub corresponds to the data in the source systems. In addition, it will ensure that the uploaded data can be processed correctly in the business processes. Data migration work will begin with a pilot phase to ensure the functioning of the data migration system tools and data migration process. Data migration work during the Datahub implementation phase will focus on the market parties' readiness in relation to data quality and ensure the market parties' ability to deliver the source data within the specified time window. The preliminary schedule for data migration work has been specified as follows: 12/ /2017 Pilot phase. 10/ /2019 Implementation phase. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

7 7 (57) The market parties must take the following measures on the basis of the plan and the data standard: 1. Compiling a Datahub data migration plan. The plan should cover at least the following areas: Data migration source systems: o Customer information systems o Metering data systems o Other possible systems Source system change needs and an action plan Needs related to cleaning and enriching source material and an action plan Specification of the tools needed to create migration files and implementation plan o Specifications for data field equivalence o Specifications for value equivalence o Methods used to derive the data needed by Datahub from existing data (if the data is not directly available) A source material delivery plan that is compiled for different phases in accordance with the set targets. System environment plan (test systems, uploading systems, dependencies on other systems, etc.). Resource plan (who handles picking and uploading of the right data for Datahub). 2. Data cleaning and enrichment. Data cleaning and enrichment should be started as soon as possible, especially for the following data: Consumer customers' personal identity codes Business IDs and association IDs Address information. 2 Introduction 2.1 Datahub Electricity retail market information exchange is needed when handling various electricity market business processes. Business processes include imbalance settlement, end user move from one address to another and supplier change. These processes and the information exchange used in them must function seamlessly and efficiently from the viewpoint of different parties. In order to handle the information exchange and business processes for the above-mentioned matters, the Ministry of Employment and the Economy asked Fingrid Oyj to implement a solution to centralise all information exchange related to the electricity market in a single service. The work began on 15 April The solution is a so-called Datahub. Datahub is a database for metering data and basic information used by market parties to submit and retrieve information needed for their market processes. Datahub is a centralised information exchange solution for the retail market where information about Finland's 3.5 million electricity accounting points is stored. In the future, Datahub's information will be used by around 100 electricity retailers and over 80 distribution network companies serving electricity consumers. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

8 8 (57) Datahub facilitates metering data processing and makes the customer's contract events simpler and faster while reducing service errors. A standardised interface to electricity consumption data promotes full utilisation of smart grids and meters and provides new business opportunities. Datahub can also process and refine information saved in it. The wide use of smart, remotely readable electricity meters in Finland ensures that a lot of data is collected about each accounting point on a daily basis. This data and possible mobile applications in the future can provide electricity consumers with completely new services. Examples of these could be an application that allows the consumer to monitor the electricity use data for a city home and a summer cottage at the same time even when they are located in different parts of Finland. Datahub and smart systems also allow electricity users to participate in demand response. Demand response means balancing electricity production and consumption so that electricity use is automatically adjusted according to the load on the electricity grid. Electricity devices can be disconnected during peak consumption situations and overproduction can be discharged to devices at large properties., a subsidiary wholly owned by Fingrid, was established on 16 February 2016 to handle Datahub's operative activities and its task is to complete the electricity market's centralised information exchange solution (Datahub). 2.2 Purpose of the data migration plan This plan in intended for electricity retail market parties and their information system providers in order to support possible system changes and quality improvement required by data entities. The market parties can use this plan as the basis for compiling their own Datahub data migration plans, which assess change needs in their own business applications and plan the implementation and testing of change needs. The data migration plan describes how the required data is imported from the market party systems to the Datahub system. A more detailed specification concerning for how long a period of time and according to which terms data must be provided is available in section 5.2. The technical descriptions and instructions that were developed when compiling the data migration plan will be used in the invitation to tender for the data migration system and as technical guidelines for the market parties regarding how data should be delivered. The data migration working group has commented on the data migration plan prior to its release to all retail market parties and their IT providers for commenting. The data migration plan includes the following areas: A description of the methods and tools used to import the data entities to Fingrid's data migration system and a technical description of the information exchange format used by the Datahub supplier to enter the data in the actual system A description of the migration files that market parties use to deliver the data being transferred to Datahub Instructions on the content and delivery of migration files A description of the tool's additional functionalities used to manage and process the data (metering and monitoring, uploads/errors) A description of the repeatability of uploads and management of different delivery batches (several parties and possibly several uploads from them) Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

9 9 (57) A description of the system requirements and platform that the data migration environment will be built on. 2.3 The Datahub project's data migration project and its goals Data migration is a sub-project of the Datahub project. Data migration will be implemented in several correction/delivery/upload repetitions, which will be referred to as "iteration" later in this document. The data migration project has two main goals: Harmonisation of the data delivered by the market parties Initial Datahub system upload. The data migration project will be divided into three phases: The goal of the planning and procurement phase is to procure and set up the data migration system. The primary choice for the data migration system is a commercially available tool suitable for the purpose. If a tool suitable for the purpose cannot be found, a separate system will be created for data migration. The planning phase will involve compiling examples of data migration migration files and instructions on how the market parties must prepare themselves for the Datahub data migration. Quality criteria for the source material and quality measurement methods will also be specified. The goal of the pilot phase is to ensure the functionality of the data migration process and the performance and reliability of the data migration system. A further goal is to improve guidance related to data migration on the basis of collected experiences. The pilot phase will be completed with selected pilot companies in order to test the processes with real source data. The data migration system must be procured and available prior to initiation of the pilot phase. The goal of the Datahub implementation phase is to harmonise the data delivered by the source systems, ensure data quality, coordinate delivery of the source material, and initial Datahub system upload. The entire industry will participate in this phase. Quality and delivery reliability criteria in the Datahub implementation phase will be stricter in each consecutive iteration phase in order to achieve the final goal. In order to improve data quality, the market party data will be compared to existing registers maintained by third parties, such as the postal address information system, register of associations and trade register. In order to ensure efficient and reliable Datahub operation, the customer's personal identity code will be needed for accurate identification of private customers. This information is not available in the market party systems for all customers. A permit to supplement personal information with information from the Population Register does not exist, and this will require a change in legislation. 2.4 Logical structure of the data migration system The figure below presents the logical structure of the data migration system and its connections to other systems. Table 1 describes the different components of the system. Note that Figure 1 is not Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

10 10 (57) a process diagram. For example, the arrow from the "Check" component to the "Integration" component does not mean that integration will be performed after the check. Instead, the check component uses the integration component. FIGURE 1 LOGICAL STRUCTURE OF THE DATA MIGRATION SYSTEM In this context, market party and Datahub are abstracts of the systems that participate in the data migration process with the data migration system. TABLE 1 DATA MIGRATION SYSTEM COMPONENTS Component Data upload Data check Reporting Publishing area Description A part of the system that is responsible for uploading the migration files in xlsx format to the Data migration system and unpacking them to the Data check component. The migration files are saved in the File archive. A part of the system that is responsible for checking an individual source material entity in relation to specified rules and creating data for check reports. A part of the system that is responsible for creating a report on the check results. The report files are saved in the File archive. A data space in the data migration system to which the ready source material to be uploaded to Datahub is transferred. The data is forwarded to the Datahub system in file format. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

11 11 (57) File archive Integration A data storage where all migration files and check reports uploaded to the system are saved in a unique manner. A part of the system that is responsible for connections to external data storages that can be used in conjunction with check procedures. Rather than being stored in the data migration system, data is deleted when it is no longer needed for checking purposes. The system is completely emptied at least after every iteration phase. 2.5 Preliminary information survey of market parties A questionnaire was sent to industry parties to survey what information systems the parties currently use and the readiness of these systems in terms of Datahub system commissioning. The surveyed information systems provide source material for the initial Datahub upload and the same systems will be integrated with Datahub during its commissioning, in other words, the systems must be able to communicate with Datahub after its commissioning. The survey was conducted to ensure an efficient start to Fingrid's data migration work and to ensure that the data migration plan supports the industry's preparations for data migration and commissioning of the future Datahub system in the best possible manner. During the planning phase for the data migration work, a preliminary survey was implemented for market parties in the data migration work group. Based on the preliminary survey, the following observations were made: Not all of the attributes associated with the data described in the Datahub project's data standard are available in the market party information systems Data quality does not meet Datahub project requirements At this time, the market parties do not have the personnel resources required by the project The market party information systems require updating, version changes or replacing entire systems. 3 Phases of data migration work Data migration work is divided into the following phases: Procurement and planning phase Pilot phase Datahub implementation phase The procurement and planning phase involves setting up the data migration system where the source material checks will be performed and which will serve as the pre-processing system for Datahub system data. The pilot phase and Datahub implementation phase are divided into iteration phases. A checkpoint will be set for each iteration phase with requirements that must be met before it is possible to move Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

12 12 (57) on to the next iteration phase. The preliminary schedule for the pilot phase and the Datahub implementation phase is presented in Table 2. Months are reported with a consecutive number because it is not practical to provide an accurate schedule estimate at this stage of the project. The schedule provides some idea of the duration of the project and iteration phases. Three iteration phases are planned for the pilot phase and five iteration phases for the Datahub implementation phase. Deliveries will begin with customer information, because this area is expected to present the greatest challenges. The order of data upload is described in more detail in Figure 8 and the iteration phases and checkpoints are described in more detail in sections 3.2 and 3.3. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

13 TABLE 2 PRELIMINARY SCHEDULE FOR THE PILOT PHASE AND DATAHUB IMPLEMENTATION PHASE Datahub data migration work 12/ /2019 Data types Customer Pilot-iterations 1 3, Data migration iterations 1 5 Accounting point Pilot-iterations 1 3, Data migration iterations 1 5 Contract Pilot-iterations 1 3, Data migration iterations 1 5 Metering Pilot-iterations 1 3, Data migration iterations 1 5 Ohter Pilot-iterations 1 3, Data migration iterations 1 5 P1 P3, M0 M5 Check points month Datahub - Data migration system implementation Pilot group Pilot group Pilot group Pilot group Pilot group Analysing results Conclusions of the results of the pilot group tests Contract with the Datahub supplier Data migrations, planning Datahub system Market parties Market parties Market parties Market parties Market parties Analysing results Conclusions of the results of the market parties' tests Start of production use Production use support -> Market parties' data and system updates P0 P1 P2 P3 M0 M1 M2 M3 M4 M5 Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

14 14 (57) Deviations observed in the data migration system will be managed using the classification described in Table 3. Deviations observed in the source data will be managed according to the quality requirement levels described in Table 3. TABLE 3 CLASSIFICATION OF DEVIATIONS Class Deviation description Description 1 Prevents use 2 Seriously hampers use 3 4 Hampers use but the problem can be worked around Cosmetic, or does not hamper use The deviation completely prevents a critical function, not necessarily use of the entire system. The deviation significantly hampers system use, may prevent non-critical functions but does not completely prevent critical functions. The deviation may be serious, but a temporary solution exists that ensures that use is not significantly affected. The source data is divided into three quality requirement levels in accordance with Table 4. TABLE 4 QUALITY REQUIREMENT LEVELS FOR SOURCE DATA Level Description Fields in the data standard 1 Essential for Datahub 2 3 Essential for electricity market operations Useful data, but not essential for market and Datahub operations First-level rules for checking are specified for fields in which the value in the priority field is essential. Second-level rules for checking are specified for fields in which the value in the priority field is useful and the value in the cardinality field is 1. Third-level rules for checking are specified for fields in which the value in the priority field is useful and for which a value has been entered in the field. 3.1 Procurement and planning phase of data migration work The procurement and planning phase involves specification of detailed requirements for the data migration system and arrangement of competitive bidding for its delivery. Fingrid will set up and test the system in cooperation with the system supplier. In addition to system procurement, the data migration procurement and planning phase includes compiling data migration guidelines for the market parties. Data migration guidelines cover the following areas: 1. Market party data migration plan instructions Brief instructions for the market parties' own data migration plan. 2. Example files for migration files Example files for each migration file. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

15 15 (57) 3. Quality requirements for data Information about data migration quality categories, requirements and quality measurement methods. 4. Data migration system instructions Market party instructions, including descriptions of the check reports. 5. Data cleaning Necessary guidelines and tips by data entity. The procurement and planning phase checkpoint measures the criteria needed to move on to the data migration pilot phase. TABLE 5 PROCUREMENT AND PLANNING PHASE CHECKPOINT Checkpoint Criteria for approval P0 The data migration system is available. The functionalities needed to initiate pilot iteration phase 1 have been implemented and tested. The system can receive customer information, perform a file check and create a check report The agreements concerning processing of material subject to data protection legislation have been signed with the pilot group parties The data migration system has sufficient data security and protection and user rights to allow the market parties to transfer their data to the system Example files for data migration have been released to the industry The date when the source material will be copied from the production environment to testing has been agreed. 3.2 Pilot phase for data migration work The pilot phase involves building the data migration system and specification of the related rules for checking, data upload tools and reporting mechanisms. This phase measures the readiness of the data migration system. Market parties participating in the pilot must be able to pick data from their own systems in the format specified by the data standard. In addition, a process will be developed with the pilot group to ensure that data is picked and delivered in a coordinated manner by all pilot group members. An analysis of test results will be performed after each pilot iteration phase. The results will be reported to the market parties in the pilot group and to Fingrid. TABLE 6 GOALS OF THE PILOT ITERATION PHASES Iteration phase (checkpoint) Pilot iteration phase 1 (P1) Goals The pilot group can deliver quality level 1 (Table 4) data that complies with the data standard and migration file specification in terms of format and content. The functioning of the data migration process has been tested in practice Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

16 16 (57) Iteration phase (checkpoint) Goals Deviations of category 1 and 2 have been corrected in the data migration system The rules for check have been corrected and new rules have been added as deficiencies are identified. Deviations of category 1 and 2 in the rules for check have been corrected The functionalities needed to initiate iteration phase 2 have been implemented in the data migration system Process lead time and efficiency have been tested in relation to the criteria set in section 5.6 for functions 1, 2 and 3 (Table 18). Pilot iteration phase 2 (P2) Pilot iteration phase 3 (P3) The pilot group can deliver quality level 2 (Table 4) data that complies with the data standard and migration file specification in terms of format and content Referential integrity check between different entities and rules for check attributes that cover different entities have been tested Deviations of severity category 3 have been corrected in the data migration system The rules for check have been corrected and new rules have been added as deficiencies are identified. Deviations of severity category 1 and 2 in the rules for check have been corrected Data can be validated against external registers and errors found on the basis of this process Problems identified when using external registers have been corrected. The publishing area for the data migration system has been implemented as planned The functionalities needed to initiate iteration phase 3 have been implemented in the data migration system Process lead time and efficiency have been tested in relation to the criteria set in section 5.6 for functions 1, 2 and (Table 4). The pilot group can deliver quality level 3 (Table 4) data that complies with the data standard and migration file specification in terms of format and content The rules for check for a check performed against other parties' data have been created Methods to ensure that data is collected in a coordinated manner from each pilot group company at the same time have been created and tested Simultaneous uploading of data by several parallel users has been tested The rules for check have been corrected and new rules have been added as deficiencies are identified. Deviations of severity category 1 and 2 in the rules for check have been corrected The process lead time and efficiency have been tested and they must meet the criteria set in section 5.6 for functions 4 and 5 (Table 18) Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

17 17 (57) Iteration phase (checkpoint) Goals The project has reliable information about the expected performance of metering data picking. The following actions are performed in each phase: 1. The pilot group updates the test environment data with production data at the agreed time. 2. The pilot group uses the extraction tool supplied by the data system supplier or its own tool to pull the data out of the source system and produce a file that complies with the specified format and content. 3. The file is read into the data migration system. 4. If necessary, the source system extraction tool is corrected if errors occur during loading. 5. Check is performed on the data (does not apply to metering data, which is not checked in the data migration system but is forwarded directly to the Datahub system). 6. The party uploading the data receives a check report, which lists any deviations. 7. If necessary, new rules for check are added and existing rules corrected. 8. Problems and ideas related to use of the data migration system and data migration process are recorded. 9. The supplier of the data migration system implements changes in the data migration system that have been agreed upon with Fingrid and the data migration system supplier. 10. The data migration system is prepared for the next iteration phase. 11. The guidelines and specifications are improved and updated if necessary. The table below describes the approval criteria for each pilot phase checkpoint (Table 2), which must be met before the project can move on to the next iteration phase. TABLE 7 CHECKPOINT APPROVAL CRITERIA Checkpoint P1 P2 Criteria for approval The plan and communications mechanisms have been specified for collecting data from all market parties at the same time The fields uploaded by the pilot group to the data migration system comply with the data standard and migration file specification in terms of format and their content complies with the data standard Data prioritised as essential must be included in the migration files Rules for performing a file check can be created in the data migration system Data migration system check reports can be created Data migration system user management is functional. Fields that are marked as mandatory and which have been prioritised as essential must be included in the migration file It must be possible to create rules for integrity checks Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

18 18 (57) P3 Users can upload files to the data migration system by themselves Attributes for which the value is dependent on the value in another data entity field must be in order. It must be possible to create rules for check between parties in the data migration system The data migration process has sufficient performance capacity The data migration system has sufficient performance capacity Availability of the data migration system is sufficient. 3.3 Data migration work in the Datahub implementation phase All market parties will be included in the actual data migration phase, and at this time the market parties must be able to pick data from their own systems and the data must also meet the quality targets specified for each migration process checkpoint. This phase measures the readiness of the entire market. This data migration phase involves transferring data from the data migration system to Datahub for process testing. The schedule for Datahub implementation and the actual data migrations will be clarified during spring The following is a rough list of the central goals for each iteration phase and the actions required for them. TABLE 8 GOALS OF THE DATA MIGRATION ITERATION PHASES Iteration phase (checkpoint) Data migration iteration phase 1 (M1) Goals All of the market parties must be able to deliver quality level 1 (Table 4) data that complies with the data standard and migration file specification in terms of format and content The functionality of rules for check the data migration system has been verified when all of the market parties upload their data to the data migration system Deviations of category 1 and 2 have been corrected in the data migration system The rules for check have been corrected and new rules have been added as deficiencies are identified. Deviations of severity category 1 and 2 in the rules for check have been corrected Level 1 quality requirement errors in the transferred files have been corrected Process lead time and efficiency have been tested in relation to the criteria set in section 5.6 for functions 1, 2 and 3 (Table 18). Data migration iteration phase 2 (M2) All of the market parties must be able to deliver quality level (Table 4) data that complies with the data standard and migration file specification in terms of format and content Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

19 19 (57) Iteration phase (checkpoint) Data migration iteration phase 3 (M3) Goals Referential integrity checks between different entities and rules for check attributes that cover different entities have been performed. The check is performed between data uploaded by a market party Deviations of severity category 3 have been corrected in the data migration system The rules for check have been corrected and new rules have been added as deficiencies are identified. Deviations of severity category 1, 2 and 3 in the rules for check have been corrected Level 1 and 2 quality requirement errors in the transferred files have been corrected Process lead time and efficiency have been tested in relation to the criteria set in section 5.6 for function 4 (Table 18) The Datahub system supplier has implemented tools and mechanisms to read data from the publishing area to Datahub The Datahub system supplier has implemented the functionalities required for data migration reporting The Datahub system supplier has implemented tools and mechanisms to read metering data to Datahub The functionalities to import a Datahub data migration reports have been implemented in the data migration system A data migration final report has been implemented in the data migration system. All of the market parties must be able to deliver quality level 3 (Table 4) data that complies with the data standard and migration file specification in terms of format and content Data transfer from the data migration system to the publishing area and then to Datahub has been tested Level 1 and 2 quality requirement errors in the transferred files have been corrected Uploading of metering data to Datahub has been tested. The efficiency and lead time for the transfer process has been tested for metering data and basic data. The process must meet the criteria set in section 5.6 for functions 6, 7 and 8 (Table 18) A check run can be performed on the data from all market parties. Datahub data migration reports can be released to the market parties The data migration system is able to create a summary report for data migration. Data migration iteration phase 4 (M4) All market parties must be able to deliver the source material within the time window provided Commissioning simulation has been performed Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

20 20 (57) Iteration phase (checkpoint) Data migration iteration phase 5 (M5) Goals Testing and fluency assessment of the entire process has been performed Performance has been proven to meet the goals set for all functions in section 5.6. All market parties can deliver data that meets the quality criteria for Datahub commissioning (Table 16) The mutual reference integrities for data are in order Errors of severity category 1, 2 and 3 in the transferred files have been corrected. The following actions are performed in the data migration phase: 1. The market parties copy the source material from the production environment to the test environment (done in phases 1 3) 2. The market party uses the extraction tool supplied by the system supplier or its own tool to pull the data out of the source system and produce a file that complies with the specified format and content 3. The guidelines and specifications are improved and updated if necessary 4. The market parties upload the data to the data migration system 5. Fingrid performs a check run on all the material 6. The rules for check are changed or new ones created if necessary 7. The approved data is uploaded to the publishing area 8. The data migration system supplier makes the necessary corrections to the transfer mechanisms and tools that transfer the data to the publishing area 9. The Datahub system supplier reads the data from the publishing area to Datahub (from iteration phase 3 onwards) 10. The market parties perform spot tests on random records to ensure the data quality 11. The Datahub system supplier corrects errors in the migration tool 12. The Datahub system supplier reads the metering data from the migration files to Datahub 13. The Datahub system supplier produces a data migration report (from iteration phase 3 onwards) 14. The data migration system creates a summary report for data migration (from iteration phase 3 onwards) 15. The data in the Datahub data migration report is compared to the source material. The data is reported and possible deviations highlighted (from iteration phase 3 onwards) 16. The Datahub system supplier corrects any problems that arise during the upload. Actions for data migration iteration phases 4 and 5 are specified in the commissioning plan. The table below describes the approval criteria for each checkpoint, which must be met before the project can move on to the next iteration phase. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

21 21 (57) TABLE 9 CHECKPOINT APPROVAL CRITERIA Checkpoint Criteria for approval M0 M1 M2 M3 M4 M5 Performance-related requirements are met for functions 2 5 (Table 18) The rules for a file check exist and have been verified The rules for an integrity check exist and have been verified The rules for a consistency check exist and have been verified Guidelines related to data migration are completed and up to date. The data migration system functions acceptably. Deviations of category 3 (Table 3) have been corrected in the data migration system P1 P3 have been performed in an acceptable manner The date (or time window) when the source material will be copied from the production environment to testing has been agreed. The fields uploaded by all market parties to the data migration system comply with the data standard and migration file specification in terms of format and their content complies with the data standard Data prioritised as essential must be included in the migration files The source material corresponds to the criteria set for the iteration phase (Table 16). Fields that are marked as mandatory and which have been prioritised as essential must be included in the migration file The source material corresponds to the criteria set for the iteration phase (Table 16). Data migration process performance must meet the criteria specified in section 5.6 The source material corresponds to the criteria set for the iteration phase (Table 16) Datahub has upload tools that allow the data to be read from the publishing area to Datahub A data migration report is created in Datahub and provides the basis for comparing data saved in Datahub with data in the source systems. It has been ensured that the entire data migration can be completed within the time window provided in conjunction with commissioning. The source material can be collected from the market parties at the specified time. The source material corresponds to the criteria set for the iteration phase (Table 16). The data has been transferred to Datahub in the commissioning phase. It has been ensured that the data transferred from the publishing area corresponds to the source material quantitatively and qualitatively. The source material corresponds to the criteria set for the iteration phase (Table 16). Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

22 22 (57) 3.4 Datahub system commissioning phase A separate detailed plan for the commissioning phase is made with the industry as a part of the Datahub implementation plan. This document outlines commissioning only to the level that performance requirements for the data migration process can be assessed. Data migration is a critical part of the Datahub commissioning process. The Datahub commissioning phase begins when the initiation of new retail market processes ends and continues until data traffic with Datahub is opened. Unfinished processes will be completed or at least to the point that the PRODAT messages associated with all processes have been sent and acknowledged. The goal is to reach a situation where the basic data for market parties are harmonised. When that point has been reached, messaging will be stopped completely and the data migration process will be initiated in accordance with Figure 2. During the critical phase of commissioning, the market parties will not be able to initiate new business processes, and events will be placed in a queue to wait for data traffic to be opened in Datahub. Data traffic to Datahub will be opened when the data migration has been successfully completed and the Datahub business processes and data connections have been tested. For this reason, it is advisable to minimise the duration of data migration. FIGURE 2 TARGET TIMES FOR THE INTERRUPTION OF RETAIL MARKET MESSAGING AND FOR THE DATA MIGRATION PROCESS. 4 Tasks and responsibilities The tasks and responsibilities of the data migration system project are described in the RACI matrix (Responsibility assignment matrix). The roles are divided as follows in the matrix: R = responsible A = accountable C = consulted I = informed Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

23 23 (57) TABLE 10 DATA MIGRATION TASKS AND RESPONSIBILITIES Task Communications Communications and contact directed at data migration project market parties Communications directed at data migration project system suppliers Communications and reporting directed at the data migration project pilot group Communications for other external stakeholders in the data migration project Market party Data migration partner System supplier Datahub supplier Fingrid C/I R I C/I A/R R C/I A/R C/I R I C/I A/R R A/R Training and technical support Instructions for using the data migration system C/I A/R C/I Training for the pilot group C/I A/R C/I Training for market parties and system suppliers Training for the Fingrid project group Data migration project technical support for market parties Data migration project technical support for system suppliers Technical support for the Datahub supplier C/I A/R C/I A/R I A/R R A/R I I A/R C I I C/I Data migration system Implementation of object- and attribute-specific rules for check based on the data standard C/I A/R C/I I Integration with the Trade register C/I A/R C/I I Integration with the Population register Integration with the Postal address register Procurement of the data migration system Installation of the data migration system Data migration system maintenance and management C/I A/R C/I C/I C/I A/R C/I I Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki A/R R A/R I A/R

24 24 (57) Task Establishment and maintenance of data migration system basic data (for example, user IDs). Market party Data migration partner System supplier Datahub supplier C/I A/R R Fingrid Data check and correction Compiling migration file examples C I A/R Extraction tools to produce migration data (other than metering data) Extraction tools to produce migration data (metering data) Production of migration files in accordance with specifications Uploading migration files to the data migration system Checking uploaded data (other than metering data) Checking uploaded data (metering data) Production of a company-specific check and correction report Overall data migration project reporting Data correction and enrichment in the market parties' own systems Datahub system upload tool Data transfer from the data migration system to Datahub Required changes to the market parties' own information systems A/R C R A/R C R A/R A/R A/R A/R I A/R I A/R A/R C R A/R R R R A/R A/R I Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

25 25 (57) 5 Data migration process 5.1 General description of the process Figure 3 presents the data migration process phases. The data being transferred to Datahub is in the current systems of various market parties. FIGURE 3 DATA MIGRATION PROCESS In phase 1, each market party produces the fields specified in the data standard for the specified migration files from its own information system. The delivery and data content of migration files are described in more detail in section 5.2 Data uploading to the data migration system is addressed in section In phase 2, the basic data for market parties is uploaded to the data migration system, where rules in accordance with the Datahub data model have been specified in advance. Metering data is forwarded directly to the Datahub system. For metering data, the data migration system only serves as a file forwarding platform. In phase 3, the data uploaded by the market party is checked in relation to the data model on the basis of the rules. The data is deleted from the data migration system database when it is no longer needed for check. The database is completely emptied at least after every iteration phase. In phase 4, the data migration system produces party-specific check reports by source material. The check reports are in xlsx format and they are saved in the file archive. The parties can retrieve reports from their own user interfaces. The reports contain correction proposals that provide the basis for improving source system data. The parties can compare the reports with the data content of the source systems and verify the consistency of the basic data in the data migration system and the source systems. The preliminary data content of the check reports is described in section Virhe. Viitteen lähdettä ei löytynyt.. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

26 26 (57) In phase 5, the data is corrected or enriched in the source systems on the basis of the check reports. The original data, error type and possible correction proposal is recorded in the report. The data migration system can only provide correction proposals in certain situations. Fingrid does not require market parties to perform corrections mechanically. The iterations in phases 1 5 are performed as many times as needed until the data has successfully passed according to the rules for check. Data uploading, check and reporting is described in more detail in section Fingrid's role is to monitor how the process progresses and provide support to different market parties when necessary. The data migration system must provide Fingrid with situation reports concerning how each market party has progressed with regard to its data migration. When a market party has produced transfer data in accordance with the data model selection rules, Fingrid confirms and approves the party data for transfer to Datahub. Data migration process monitoring and the monitoring tools are described in section 5.4 In phase 6, the data that has passed the check is transferred to the data migration system publishing area. This phase requires that the technical specification for data transfer to Datahub has been completed. The specification work is done together with the Datahub system supplier. In phase 7, data transfer from the data migration system publishing area to Datahub is initiated by the Datahub main user. Data uploading to Datahub is described in the section General description of the process. In phase 8, Datahub creates migration file- and party-specific data migration reports that contain the data uploaded to Datahub. In phase 9, the data migration reports produced by Datahub are imported to the data migration system. In phase 10, the market parties compare the data exported to Datahub with data in the source system using the data migration reports obtained from Datahub. This is not an essential phase in terms of data migration, and its purpose is to verify the consistency of the source data and Datahub data. In phases 11 12, the data in the Datahub data migration report is compared with the data published in Datahub. Based on the results of the comparison, a data migration final report is created, including a summary of the transferred data and a list of deviations. 5.2 Data being transferred Figure 6 specifies the migration files that each market party produces from its own systems for transfer to the data migration system in order to be checked. The figure specifies an element like that in Figure 4 for each migration file. The data presented on a green background is critical in terms of Datahub operations. The data on a blue background is classified as 2nd priority data, which is useful for the market but not critical. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

27 27 (57) An example file is made for each migration file, with columns specified in accordance with the data standard and a sufficient number of example rows where different values are used to illustrate different cases. The migration file element is divided into three parts: 1. The attributes that uniquely identify the row and the attributes that combine the data in different migration files with each other, for example, Customer ID, Customer type and Party ID. The uniquely identifying attributes are marked in red. 2. Attributes that are purely associated with data migration, for example, migration file date and fields that improve data readability and understandability. 3. Entities specified in the data standard and their attributes are combined in the same migration file. Entity refers to the Entity column in the data standard and the value in it. FIGURE 4. THE MIGRATION FILE SPECIFIES THE UNIQUELY IDENTIFYING FIELDS, DATA MIGRATION- RELATED ATTRIBUTES AND THE ENTITIES COMBINED IN THE MIGRATION FILE. The relation between the migration files is expressed with a cardinality that specifies how many objects occur between them. The example in the figure below can be read as follows: a customer can have several or no authorisations and one authorisation can be linked to only customer entity at a time. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 forename.surname@fingrid.fi FI Helsinki FI Helsinki

28 28 (57) FIGURE 5. THE CARDINALITY EXPRESSES HOW MANY OBJECTS OCCUR IN THE RELATION BETWEEN MIGRATION FILES. Street address Postal address Phone Fax Business Identity Code FI , VAT reg. Läkkisepäntie 21 P.O.Box 530 FI Helsinki FI Helsinki

29 FIGURE 6. MIGRATION FILES PRODUCED BY THE MARKET PARTY INFORMATION SYSTEMS

30 Public [Topic] (57) Migration file data content The attributes of a single migration file and the rules related to attribute quality are described in the data standard. The heading of the migration file is formed on the basis of the data standard attributes and the entity data fields related to the migration file in question are selected for it (see Figure 6). The order of the columns in each migration file has no significance. Example files have been made for each migration file, and they contain a specified heading and example rows. The example rows clarify the problem situations related to picking, such as presenting address information in different cases. TableTable 11 lists the data standard fields that are related to data migration. The table and Figure 6 make it possible to pick the attributes for each migration file from the data standard. TABLE 11. DATA MIGRATION-RELATED FIELDS IN THE DATA STANDARD Field Attribute Migration file The field belongs to the migration file Belongs to a unique key Only information related to data migration Distribution system operator attribute. Description Attribute name used as the name of the migration file column heading C=Customer information A=Accounting point information Aa= Additional accounting point addresses M=Metering data Ma=Metering grid area information P=Party information E=Exchang point information Cn=Contract information Pu=Production unit information Pr=Product information Au=Authorisation information An attribute that is only in internal Datahub use and which does not have to be transferred; in data migration is left outside the migration file. Y=The data content of the field is included in the migration file N=The field is outside data migration The field uniquely identifies the row in the migration file and the rows in different migration files can be combined with each other on the basis of the identifying fields. The attribute is a field only needed by data migration. For example, the migration file date links the uploaded data and the migration file from which the data has been uploaded to the data migration system. The party ID is also included in every migration file so that the origin of the data can be traced. This attribute applies to the distribution system operator. The field may not necessarily be mandatory. If the field only applies to the distribution system operator, other parties can leave this field empty when delivering data. Y=Applies to the distribution system operator. N=Does not apply to the distribution system operator. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

31 Public [Topic] (57) Field Supplier attribute Third party attribute Format Technical example Field length Permitted values / code list / code Terms and dependencies Default value Priority Cardinality Description This attribute applies to the supplier. The field may not necessarily be mandatory. If the field only applies to the supplier, other parties can leave this field empty when delivering data. Y=Applies to the supplier N=Does not apply to the supplier This attribute applies to a third party. The field may not necessarily be mandatory. If the field only applies to the third party, other parties can leave this field empty when delivering data. Y=Applies to a third party N=Does not apply to a third party The format in which the data must be entered in the field. When checking data, the check procedure determines whether the value provided is in the right format. The formats used are listed below. Text = Can contain letters, numbers and special characters Number = Decimal or whole number. A comma (,) is used to separate decimals. Day = The date in the YYYY-MM-DD format. Time stamp = The moment in time in the format YYYY-MM-DDTHH:MI:SS+00:00, for example, T14:05:25. Time is expressed in UTC time. Boolean = 0 (false) / 1 (true) Code list = A list of values specified in advance, in which the permitted values are specified as a code-description pair in the data standard field. Permitted values / code list / code. The migration file always has a code. Relation = The attribute refers to another entity. Relation-type fields are not included in the migration file. An example of the provided data that observes the format and rules specified for the field. The longest permitted field length. If this is a decimal, the decimal numbers are reported after a comma, for example, 9,3 = 9 whole numbers and 3 decimals. A list of values from which one must be selected for the field if the data is mandatory. The list of values is specified as code-description pair. A code is always entered in the field. The value in the field or its mandatory nature can depend on the value of some other attribute or the dependency can extend to an attribute of a completely different migration file. If the value is unknown, the value specified in the Default value column is entered in the field. The Default value field specifies the default value code and its description in the format <code> = <description>. Classifies the attributes according to whether they are essential to Datahub operations or only useful data for the market. Essential = This attribute is mandatory and it must be given in the migration file. Useful = If this data is found in the market party's source system it can be added, but this data is not critical for Datahub. If the field contains the value 1, the field is mandatory. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

32 Public [Topic] (57) Migration file format With regard to basic data, the migration files are specified in xlsx format and they use the UTF-8 character set. The files are uploaded to the data migration system by each market party via a browserbased upload user interface. The data must be specified in a single worksheet. The rules for naming the migration files are specified in the table below. TABLE 12. RULES FOR NAMING A MIGRATION FILE IN XSLX FORMAT File name part Party ID Migration file type Date Revision Ending Description and specification of the file name part A unique party ID that is specified in the Fingrid document osapuolet.pdf Customer information Accounting point information Additional accounting point addresses Metering grid area information Party information Connection point information Contract information Production unit information Product information Authorisation information Date when the data was uploaded from the source system The date format is YYYY-MM-DD. A consecutive revision number starting with 001. Each migration file is always given a new revision number when it is taken out of the system. The type of the migration file ending, which is xlsx. In their entirety, the basic data migration file names are in the format: <Party ID>-<Migration file type>-<date>- <revision>-<ending>. Customer information ("Asiakastiedot" in Finnish) for example: MSOY-Asiakastiedot xlsx If the data in the migration file has to be divided into several files, each migration file is numbered consecutively. The migration file revision number remains the same. For example, MSOY-Asiakastiedot xlsx, MSOY-Asiakastiedot xlsx, MSOY-Asiakastiedot xlsx etc. The file format for transferring metering data is SAF, which is a file format commonly used for transferring metering data. SAF is a text-based file format in which the data is separated by a semicolon. A more detailed description of the file format is available in the migration file guidelines and a separate technical description (Enoro SAF Technical Description pdf). The files are uploaded to the data migration system by each market party. Metering data is not imported to the data migration system database, instead it is forwarded directly to the Datahub system. Due to the large amount of data, the files must be delivered in compressed format. The table below specified the rules for naming metering data migration files. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

33 Public [Topic] (57) TABLE 13. RULES FOR NAMING AN SAF FORMAT MIGRATION FILE File name part Party ID Migration file type Time stamp Consecutive number Ending Description and specification of the file name part A unique party ID that is specified in the Ediel portal Metering data The time stamp when the data was uploaded from the source system The time stamp format is YYYYMMDDHHMISS. A consecutive revision number starting with 1. The consecutive number prevents the creation of files with the same name. The number of characters in the consecutive number is not limited. The type of the migration file ending, which is saf. Thus the format for migration files containing metering data is: <Party ID>-<Migration file type>-<time stamp>- <Consecutive number>.<ending>. For example ("Metering data" in Finnish is "Mittaustiedot"). FSJ000- Mittaustiedot saf. Thus, the file is for the time: at Data delivery responsibility and rules concerning data selection Data for which another market party has maintenance responsibility can be stored in a market party's information system, for example, with regard to accounting point data, each distribution system operator is responsible for maintaining the data for accounting points in its own area. Accounting point information can be stored in electricity sales company systems, but during data migration, data is only delivered to Datahub by distribution system operators. A storage period can also be specified for the data so that, for example, contract information can be found in the systems of several different market parties. The table below presents a view on which market party delivers data in migration files in the data migration phase and on the length of time for which history data is delivered in data migration. TABLE 14. SPECIFICATION OF DATA DELIVERY RESPONSIBILITY FOR MIGRATION FILES, RULES FOR DATA SELECTION, AND HISTORY DATA TRANSFER NEED Migration file Data owner that delivers data to Datahub Rules Data for time interval Customer information Distribution system operator (grid agreement) Supplier (sales agreement) If a specific customer's information is contained in the systems of several distribution system operators, the customer information of the distribution system operator that has the most recent grid agreement with the customer is used. Customers with valid contracts. Customers with contracts that have ended within the past six weeks. Customers with future contracts (see Contract). Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

34 Public [Topic] (57) Migration file Data owner that delivers data to Datahub Rules Data for time interval If several market parties have different information for the same customer, the market parties must clarify the right information among themselves so that the distribution system operators determine it among themselves and the supplier companies clarify it with the distribution system operators. Accounting point information Distribution system operator If a distribution system operator's system has accounting points outside the distribution grid, the distribution system operator only delivers accounting points in its own area. Only main metering sites are transferred. Accounting points in use are imported. (Under construction, connected or disconnected.) Accounting points removed from use are imported if a contract specified for transfer is associated with them (see Contract information). Contract information Distribution system operator (grid agreement) Supplier (sales agreement) - Valid contracts and contracts that have ended within the past six weeks. Contracts beginning in the future. Party information All The parties only deliver their own data. Only valid data. Product information Distribution system operator (grid product) Supplier (sales product) - Valid products associated with contracts being transferred. Metering data is imported to Datahub for the validity period of contracts that are currently valid or have ended within the past six weeks. Metering data Distribution system operator Metering data is picked from the system from which metering data is reported to the suppliers. The metering data is retrieved for the validity period of the contract, but for a maximum of six years into the past. Only hourly metered metering data is collected from the metering data history. The transfer of history data with regard to metering data is described in more detail in Figure 7. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

35 Public [Topic] (57) Migration file Authorisatio n information Metering grid area information Connection point information Production unit information Contact information Data owner that delivers data to Datahub Third party Distribution system operator Distribution system operator Distribution system operator Distribution system operator (grid agreement) Supplier (sales agreement) All (Party information) Rules Authorisations are based on a third party's power of attorney to process data for the specified accounting point. The information systems do not contain powers of attorney, so the check must be handled outside the information systems. Fingrid is responsible for the procedure. Data for time interval The third party specifies the valid authorisation information needed for the migration file. - Only valid data. - Only valid data. - Only valid data. - Valid contact information associated with contracts being transferred. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

36 Public [Topic] (57) FIGURE 7. RULES RELATED TO THE TRANSFER OF METERING DATA HISTORY DATA The figure above presents the rules related to the transfer of metering history data. The red vertical line illustrates the time prior to which data is transferred to Datahub. During the data migration project, data is uploaded in several iterations, so the red vertical line can also describe the time when the data is taken out of the system for data check. 1. The accounting point contract has been valid for less than six years, so metering data is imported for the entire contract period. 2. The accounting point contract has been valid for the entire six year period. Metering data is transferred for the entire period. Metering data for a longer period is not imported to Datahub. 3. If the accounting point contract has ended less than six weeks before the data is taken out of the source system, the metering data for these accounting points is also included. Metering data is collected for the validity period of the contract after it ends, but only for a maximum of six years from the time of picking. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

37 Public [Topic] (57) 4. Only metering data within the scope of hourly time series metering is taken into consideration in metering data. If the metering method has changed during the contract validity period, only the hourly time series metered data is included in the metering data. 5. If accounting point consumption is metered in some other way than hourly time series, use of these accounting points must be balanced via balancing calculation and then switched to use hourly time series or hourly time series modelling. This must take place in conjunction with Datahub commissioning at the latest. The majority of metering data will be uploaded to Datahub well before commissioning, during the socalled "main upload of metering data". After the main uploading, only new and changed values will be uploaded in conjunction with commissioning or, alternatively, new and changed metering data can be imported to Datahub continuously after the main uploading. The exact procedure will be specified during the planning of commissioning. The mutual dependencies of the data being transferred are presented in the figure below. The migration files are uploaded to the data migration system in the order presented below so that level 1 files are uploaded first and followed by level 2 files. This makes it possible to perform referential integrity check between the data in the data migration system. Thus the lower levels are dependent on the upper level data being uploaded first. FIGURE 8. EXECUTION SEQUENCE FOR MIGRATION FILES In the pilot phase, each party in the group must have a test system in use, with data from the party's own source system production environment corresponding to an agreed time period. In the pilot phase, the data is taken from the test environment using the system provider's extraction tool in order to Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

38 Public [Topic] (57) ensure that mutual referential integrity and logic check can be performed on the market party's own data and on all market party data in the data migration system. However, individual newer migration files can be added to the data migration system, for example, directly from the production environment if there is a desire to also start data improvement in the source system in a parallel manner. In data migration during the Datahub implementation phase, the data will be taken from the production environment by each market party, starting at the agreed time and continuing back in time. 5.3 Data import, check and correction proposals Figure 9 presents data import to the data migration system by a market party. The data migration system checks the data entered by the user and provides correction proposals if necessary. A detailed description of the rules for check will be drawn up during the pilot phase and released to the industry. Based on the import, the data migration system collects a summary of the imports made by the market party in question and the data quality. Based on the summary, Fingrid will make a decision whether the market party's data meets the Datahub requirements. FIGURE 9 MIGRATION FILE IMPORT AND CHECK Importing data to the data migration system The provider of the market party information system creates a tool that will be used to produce migration files from the source system in the format specified in section 5.2. The market parties will be provided with a browser-based uploading tool for which each market party will receive its own ID. The IDs will be provided to designated persons. The migration file must include the columns specified Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

39 Public [Topic] (57) in the data standard and the heading row fields must use the attribute names specified in the standard. Traffic between the data migration system and the user (client) must be encrypted. The user can monitor the import and check phases via the browser-based user interface. (Figure 10, in Finnish) In conjunction with file import, the user interface must provide at least the following information about the import process: Migration file name and size Time stamp when import began Progress of import process (%) Time stamp when import was completed Import duration Number of imported rows. FIGURE 10 MIGRATION FILE IMPORT TO THE DATA MIGRATION SYSTEM (ILLUSTRATION) After import, the files are automatically transferred to the check phase, where the system performs a file check and a party-specific integrity check. Fingrid initiates a Datahub-level consistency check based on the party's request for approval. The user interface must provide at least the following information about the check process: File import time Progress of file check (%) Time stamp when the file check was completed File check duration Progress of integrity check (%) Time stamp when the integrity check was completed Integrity check duration Migration file status information. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

40 Public [Topic] (57) FIGURE 11 MONITORING OF THE CHECK PROCESS A consistency check is not performed on a party-specific basis, so progress of that process is not visible in the party's user interface Data to be checked Each field is checked to ensure that the data type, format and content of the field comply with the rules specified in the data standard. All of the errors found in a row are marked on a field-specific basis for each data row. The entities to be checked are specified in their entirety in the data standard. The estimated number of rows in the data to be checked is: Accounting point information: approximately 3.5 million rows. Customer information: approximately 4 5 million rows. Contract information: approximately 7.5 million rows. Other basic data: some thousands of rows. Metering data is not checked in the data migration system and is forwarded directly to the Datahub system where it is linked to the metering points. The Datahub system checks that metering data has been delivered for all active metering points and that all metering data can be linked to a metering point Error types The following is a list of the error types for which rules for check are created in the data migration system. The data migration plan does not specify the actual rules for check for different level check and project phases. A separate document is specified for the rules for check, and it lists all of the rules for check. Rules are added during the pilot phase in particular, but they can also be added during the actual data migration phase. The specified rules provide the basis for implementing rules for check in the data migration system. TABLE 15. LIST OF ERROR TYPES Error Missing data Incorrect data type Incorrect format Description The field is specified as mandatory, but no value has been entered. Note: Some of the fields are conditionally mandatory, for example, "Mandatory if the status is something other than 'under construction' and the fuse size has not been reported" The data content in the field does not correspond to the specified data type. For example, a numeric field contains letters. The data content in the field is not in the required format. For example, "<number of cables>x<number of phases>x<ampere>. Number of cables is left out if only one cable is in use." Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

41 Public [Topic] (57) Error Incorrect value Duplicate data Referential integrity error Register error Register difference Overlapping validities Logical error Description The value in the field does not correspond to the specified range of values. Data specified as unique occurs in the data more than once, which means it is not unique. The field is an ID/code field for which a corresponding value cannot be found in the object in question. For example, a contract refers to an accounting point that cannot be found. A field/data is checked against an external register, such as the Finnish Patent and Registration Office's Trade Register, but the value in question, such as a business ID, cannot be found. A field/data is checked against an external register, such as the Finnish Patent and Registration Office's Trade Register, and the key value in question is found but there is a difference in the related data. For example, the company name does not match. The validity of a data row overlaps with another data row and is thus conflicting. For example, an accounting point has several simultaneous, valid sales agreements. For example, an accounting point does not have a grid agreement for the entire validity of the sales agreement Quality requirements for data during the pilot and data migration phase Data delivered by different market parties is checked in the data migration system against the specified rules. Separate data quality goals have been set for each project checkpoint. The quality goals are different for the actual pilot phase and the data migration phase. The pilot phase focuses on creating rules for check and their functionality in the data migration process is examined. Errors in the market party's data are permitted in the pilot phase, but this does not prevent a move to the next iteration phase. The performance of the actual data migration process is also verified during the pilot phase. In the actual data migration phase, the market parties must ensure that the data has been corrected in the source system and that the data passes the check successfully. The percentage of incorrect records permitted at each checkpoint is specified on the basis of experience gained in the pilot phase. The rules for check can also be dependent on different phases of the data migration project. The check performed in different phases can be divided into three categories: 1. File check, a migration file-specific independent check 2. Integrity check, a check between migration files delivered by a market party 3. Consistency check, a check of all market party data performed for the transfer to Datahub. Preliminary quality criteria for the checkpoints of each implementation phase in Datahub are presented in Table 16. The checkpoints observe the iteration phases, which means that the next iteration phase only begins when the entire industry has achieved the quality criteria. TABLE 16 PRELIMINARY SOURCE MATERIAL QUALITY CRITERIA BY CHECKPOINT Level Description T1 T2 T3 T4 T5 Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

42 Public [Topic] (57) 1 Essential for Datahub 95.00% 98.00% 99.00% 99.90% 99.98% No Essential for electricity 2 requiremen 95.00% 98.00% 99.00% 99.50% market operations t 3 Useful data, but not essential for market and Datahub operations No requiremen t No requirem ent No requirement No requirement No requirement File check Rules for check are specified for each migration file in the data migration system, which can be specified on a levelspecific basis depending on the criticality of the field in terms of Datahub or the electricity market. The rules for check are specified, implemented and tested in the pilot phase. The mutual reference integrities in the migration files and dependencies between fields among different migration files are not taken into account in this phase. At least the following check types are performed in a file check: A value has been entered in the field (on level 1 and 2). The value complies with the list of values if a list of values has been specified. The value length is shorter or the same as the longest permitted field length if this is specified. The field value complies with the data type. The field's dependency on values in other fields if this is specified (mandatory if...). A unique ID (the same value does not occur twice). Logical time interval check (for example, the end time for validity is larger than the start time). Register comparison (Posti's address register, trade register, etc.). Unambiguous validities (no overlapping validities). Some of the checks may already be performed in conjunction with file import. In particular, there are challenges associated with checking accounting point address information. There are many accounting points without an address or for which an address can't be found in the postal address register because no mail is delivered to the address. A method to identify accounting points for which the address cannot be checked must be created in the data migration system so that the system does not create a large number of unnecessary errors. Another challenging area involves checking customer information. Many market parties can have the same customer's information in different forms. The instructions for cleaning data must specify detailed processes for determining the right customer information. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

43 Public [Topic] (57) Integrity check When a market party has delivered migration files containing dependencies (see Figure 8), referential integrity check can be performed between the migration files. For example, it must be possible to find the customer specified in a contract in the customer information imported to the data migration system with the ID provided. The comparison only targets data uploaded by the market party itself. For example, the supplier may not necessarily have access to the accounting point data referred to in the contract. In addition, rules for check are created for fields where the value depends on a value in a field in some other migration file. The rules for check data between migration files will be specified, implemented and tested during the pilot phase. Errors are permitted in the market party material, but the quality must be on a level sufficient to ensure that check process tests provide reliable results. Checking the dependencies between migration files can significantly affect the lead time for the entire process, so the pilot phase also involves determining which rules can be left out in the actual data migration phase. Errors in referential integrity and field dependency must be checked and corrected in the source system. The rules for check do not check data that is maintained in another market party's system Consistency check Once all the market parties have uploaded their material to the data migration system, the following check can be performed on the material: Referential integrity check (the accounting point ID in the supplier's contract can be found in the distribution system operator's material). Unambiguous validities (the same accounting point does not have a valid contract between two different suppliers at the same time). Functional integrity (the accounting point has a sales and grid agreement, metering time series valid for the duration of the agreements). Unique IDs (the same customer ID and customer ID type does not have two records). Rules for check are already created and tested during the pilot phase of the data migration project, but the rules for check can only be reliably tested during the data migration phase. Only approved material is published in Datahub. As a general rule, incorrect rows containing data errors in individual fields or referential integrity errors are not transferred to Datahub. Final approval of the material is done in this phase. 5.4 Data export to Datahub Once a market party's data has passed the check and it meets the quality requirements specified for data approval (see Table 5), the data is transferred from the data migration system to a separate publishing area. The Datahub system supplier loads the data from the publishing area to the fields and tables in accordance with the data model for the Datahub database. Technical specification for data transfer between the publishing area and the data migration system and Datahub will be done in cooperation with the Datahub system supplier. After the Datahub data upload, the Datahub system Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

44 Public [Topic] (57) creates so-called data migration reports, which are returned to the data migration system. The data migration reports contain all data uploaded to Datahub on a market party-specific and migration filespecific basis. The data migration system releases the data migration reports to the market parties and creates a final data migration report. The market parties can compare the data migration report data with the data in the source systems and thus verify that the data in Datahub and the source systems are consistent. In addition, the main users for the source systems make spot checks on the data in Datahub to ensure that the data has gone to the right fields in Datahub. Data check is done via a browser-based user interface for which user-specific IDs can be opened for each market party. Spot checks will be specified in more detail in the data migration phase once the Datahub system is available. The results of the spot checks are reported to Fingrid. The metering point (accounting points, production units and exchange points) metering data is forwarded directly to the Datahub system, and the data migration system only serves as a relay platform in this case. The Datahub system retrieves the metering data from the data migration system's publishing area. A metering data check requires that the basic data has been imported to the Datahub system. Datahub performs the following check on metering data: Metering data is available for the entire validity of the grid agreement for the accounting point if the metering method is hourly metering Metering data is available for the entire validity of the production unit Metering data is available for the entire validity of the connection point All metering data is linked to a basic data structure (no "orphan" time series). Datahub performs hourly imbalance settlement calculations and the following calculation results are used in metering data check: Metering grid area-specific losses Supplier-specific total series by metering grid area. 5.5 Monitoring and monitoring tools During the import and checking of migration files, the end user can monitor progress of the process from the user interface. The System also publishes various reports for its users via the user interface in different phases of data migration. The System publishes its own reports for the end users and the main user. More detailed descriptions of report content and requirements are available in Datahub Data migration Data migration service's functional requirements, section 9. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

45 Public [Topic] (57) Import and check monitoring The end user can monitor the progress of migration file import and checking via the progress bars in the user interface and detailed data. Figure 12 presents an illustration of the end user's monitoring display. The following examples of the reports are in Finnish, since the actual data migration system will also be only in Finnsih. FIGURE 12 MIGRATION FILE IMPORT AND CHECK. USER INTERFACE ILLUSTRATION End user reports Summary report for the check The browser-based user interface allows the market parties to check the situation concerning delivery and checking of the migration files. The summary report contains number and quality information about the most recently imported migration files. Error reports can be downloaded from the user interface, and they contain detailed information about the errors and their number in xslx format for selected migration files. The figure below shows an example of a market party summary report. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

46 Public [Topic] (57) FIGURE 13 MARKET PARTY SUMMARY REPORT Error report for a check The end user can examine errors that occur in data uploading in a detailed manner via the error reports produced by the System. The reports are in xlsx format and they are available from the end user's user interface. There are three types of error reports: File check error report Integrity check error report Consistency check error report. The error reports help the parties perform corrections in the source systems. The reports contain the following preliminary columns: TABLE 17 ERROR REPORT COLUMNS. No. Column Description Standard columns 1 Migration file Migration file name 2 Row number Migration file row where the error was detected 3 Quality requirement level 4 Error ID Classified on levels 1 3. The quality requirement level tells which quality requirement level the error is assigned to (Table 4). The error ID tells the error type. The error types are listed in the rules for check document published during the pilot phase. 5 Field The field where the incorrect data was found. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

47 Public [Topic] (57) n Correction proposal Migration file-specific columns <Migration file fields> The System's proposal for correction. Only written if the right value in the field can be deduced, otherwise it is left empty. The fields that are dependent on the migration file type. The values in an incorrect row are recorded as such in the error report for the check area. Rows that are rejected in a syntax check are not found in the check area, so they are not recorded in the error report. Note that the same migration file row appears several times in the error report if several errors are found in that row Datahub data migration reports Data migration reports are reports produced by the Datahub system, which help the market parties compare data content transferred to Datahub with the data content in the source system. Datahub creates reports by party and by source material in xlsx format, and they are available from the end user's user interface. Data migration reports are also used to create the final data migration report and the related deviation report that belong to the main user's tools. Statistics are created on the basis of the data in the check area. For metering data, Datahub delivers the following reports to the data migration system by distribution system operator: 1. A list of the accounting points, production units and connection points that are missing all or some metering data for the validity of the grid agreement. 2. Metering time series that Datahub was not able to link to a basic data structure. 3. Metering grid area-specific loss series. 4. Supplier-specific total series by metering grid area. The grid owner compiles reports from the data migration system's user interface and compares the results of imbalance settlement with the results calculated by its own system and clarifies the causes of the errors. The preliminary report format is xlsx, but the report format and content will be specified in more detail in cooperation with the Datahub supplier Main user's reports Delivery statistics for migration files The delivery statistics report for migration files is a main user tool for monitoring migration file deliveries by the parties and it allows the main user to monitor deliveries by market parties according to the party and source material type. The report format is html or pdf and it is only visible to the main user. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

48 Public [Topic] (57) FIGURE 14 MONITORING DELIVERY OF MIGRATION FILES Party-specific quality statistics The party-specific source material quality report allows the Datahub main user to check the quality of data delivered by the market parties according to the quality requirement level. The report format is html or pdf and 100% means error-free quality. The report content is illustrated in Figure 15. FIGURE 15 SOURCE MATERIAL QUALITY BY PARTY Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL Helsinki Helsinki

49 Public [Topic] (57) Source material quality report at the Datahub level The source material quality report at the Datahub level is a main user tool for monitoring the overall quality of the source material. The report presents the quality of source material according to source material type and quality requirement. The quality is reported as a percentage, where 100% means error-free data. FIGURE 16 SOURCE MATERIAL QUALITY REPORT AT THE DATAHUB LEVEL Final data migration report The final data migration report is a summary of the data migration results in html or pdf format. The final report shows the number of rows delivered according to source material and the number of missing or deficient rows. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL 530 etunimi.sukunimi@fingrid.fi Helsinki Helsinki

50 Public [Topic] (57) FIGURE 17 FINAL DATA MIGRATION REPORT Data migration deviation report The data migration deviation report contains detailed information about deviations observed when comparing the data published in Datahub and the Datahub data migration reports. The report format is xlsx and it is only visible to the main user. 5.6 Data migration process performance Data migration system performance and the lead time for the entire data migration process is critical, particularly in the Datahub commissioning phase. The System must process large data masses and there may be 300 simultaneous users operating in parallel. Table 18 lists each data migration process phase, the computational requirements set for performance, and the number of simultaneous sessions. Note that this plan does not set performance requirements in relation to the amounts of data. In practice, this means that the extraction tools for large systems must be more effective than the extraction tools in small systems. Katuosoite Postiosoite Puhelin Faksi Y-tunnus , ALV rek. Läkkisepäntie 21 PL Helsinki Helsinki