Data Validation in the ESS Context

Size: px

Start display at page:

Download "Data Validation in the ESS Context"

Marianna Webb
5 years ago
Views:

1 Data Validation in the ESS Context ESTP course Item 02 Luxembourg, Nov 2017 Eurostat, Unit B1

2 Presentation Outline ESS Vision 2020 Quality and compliance GSBMP Data validation Life cycle Main problems in current situation and medium term goals Validation levels 20 main types of validation rules in the ESS 2

3 ESS Validation Youtube video at 3

4 The path to ESS Vision 2020 Reg. 223/2009 ESS COM (2009)404 Vision COM (2009)433 GDP and beyond COM (2011)211 Robust quality ESSC Nov.2012 ESS.VIP Programme Reg. 99/2013 ESP "The ESS provides the EU, the world and the public with independent high quality information on the economy and society on European, national and regional levels and make the information available to everyone for decision-making purposes, research and debate." 4

of work: 1. Focus on users 2. Strive for quality 3. Harness new data sources 4.

5 In May 2014, the ESSC came to an agreement on the ESS Vision 2020 as the guiding frame for the ESS development during the years up to key areas of work: 1. Focus on users 2. Strive for quality 3. Harness new data sources 4. Promote efficiency in production processes 5. Improve dissemination and communication 5

6 ESS Vision 2020 portfolio 6

7 ESS Vision 2020 Building Blocks Communication Governance Administrative data Users Big Data Validation Data warehouses Methods Services Quality Standards Network Enterprise Architecture 7

8 Quality and compliance Countries have to provide statistics to Eurostat on time (punctuality) and of acceptable quality. Link with quality principles of the European statistics code of practice Principles 11 to 15 related to statistical output: 11: Relevance 12: Accuracy and reliability 13: Timeliness and punctuality 14: Coherence and comparability 15: Accessibility and clarity 8

9 GSBPM Global Statistical Process Model Global Statistical Business Process Model 9

10 Data validation life cycle 10

11 Main problems in current situation No clear picture of who does what Time-consuming "validation ping-pong" Possibility of validation gaps Possible subjective assessment of data quality Lack of standards and of common architecture Risk of non-consistent assessment over time, between countries and across statistical domains Possible misunderstanding on what is acceptable or not Duplication of IT development costs within the ESS Manual work due to low integration between the different tools 11

12 Medium-term goals - ESS VIP validation Medium-term goals Goal 1: Ensure the transparency of the validation procedures applied to the data sent to Eurostat by the ESS Member States. Business Outcomes Increase in the quality and credibility of European statistics Reduction of costs related to the timeconsuming validation cycle in the ESS ("validation Ping-Pong") Goal 2: Enable sharing and re-use of validation services across the ESS on a voluntary basis. Reduction of costs related to IT development and maintenance 12

Data Within an organisation Within a domain From the same source Same dataset Validation complexity Validation levels 0 to 5 (Graph B) Same file Level 0: Format & file structure Level 1: Cells,

13 Data Within an organisation Within a domain From the same source Same dataset Validation complexity Validation levels 0 to 5 (Graph B) Same file Level 0: Format & file structure Level 1: Cells, records, file Between datasets Between files Level 2: Revisions and Time series Level 2: Between correlated datasets From different sources Between domains Between different organisations Level 3: Mirror checks Level 4: Consistency checks Level 5: Consistency checks 13

14 Validation levels 0 to 5 (Graph A) Data Within a statistical authority Between statistical authorities Within a domain Between domains Level 5 Consistency checks From the same source From different sources Level 4 Consistency Checks Same dataset Between datasets Level 3 Mirror checks Same file Between files Level 2 Checks between correlated datasets Level 0 Format and File structure checks Level 1 Cells, Records, File checks Level 2 Revisions and Time series 14

15 Priority to Validation at file level - Why? Easiest to define, check, implement, configure and share in a validation service (for "pre-validation") Fastest to check, Most numerous validation rules, Most numerous to lead to severity level error (with rejection of data), lead, when not checked to tedious and burdensome manual work and "ping-pong" At least these rules should be checked prior to transmission to Eurostat 15

No worry: you'll remain anonymous Time for a Quiz Validation rules Guess the level(s) Validation rule/level 0 1 2 3 4 5 Valid code in field "aircraft type"

for country X The revisions of data between 2 versions are limited Outliers detection Export figures reported by country A towards country B correspond to

16 No worry: you'll remain anonymous Time for a Quiz Validation rules Guess the level(s) Validation rule/level Valid code in field "aircraft type" Annual data consistent with quarterly data Country code in first field consistent with data sender Eurostat data for country X is consistent with OECD data for country X The revisions of data between 2 versions are limited Outliers detection Export figures reported by country A towards country B correspond to Import figures reported by country B from country A Changes in agricultural production of olives for country "X" is consistent with neighbouring countries 16

17 20 Main types of validation rules Cover at least 80-90% of the validation rules used for ESS data Used: To structure the standard documentation that describes validation rules in domains (and as a check-list) To identify the main VTL operators needed for Validation of ESS data In Validation Rule Manager to allow an easy definition of rules based on a simple set of parameters per rules and an easy and automatic generation of VTL 17

18 Classified in 4 groups 1. Basic file structure check (level 0): => Preliminary checks needed => SDMX full coverage (structural Validation) => 4 types of rules that lead to errors in case of failure 2. Basic intra-file checks (Levels zero and 1): => SDMX large coverage => 5 types of rules that lead in most cases to errors in case of failure 3. Checks intra or inter files (levels 1 to 5): => SDMX very partial coverage => 8 types of rules that usually lead to errors in case of failure 4. Checks inter-files in same statistical domain (Level 2 and 3): => No SDMX coverage => 3 types of rules that usually lead to warnings in case of failure 18

19 Main types of validation rules - Overview Rule type Mandato ry Default Validation level Micro Severity level SDMX data E W I 19 Comments (EVA) Envelope is Acceptable X X X X X (FLF) File Format X X X X X (FDD) Fields Delimiter (X) ; X X X X Mandatory for CSV files (DES) Decimals Separator (X). X X X X Mandatory for CSV files (always. For SDMX-ML) (FDT) Field Type X X (X) (X) X X (FDL) Field Length X X X X X (FDM) Field is Mandatory or empty X (X) (X) X X (X) (COV) Codes are Valid (X) X (X) X X (X) Mandatory for key fields (RWD) Records are Without Duplicates (X) Key X X (X) X Mandatory for aggregated (tabular) data Default: No duplicate key (REP) Records Expected are Provided X X X X (X) (RNR) Records Number is in a Range X >=1 X (X) X X (X) (X) Default: at least one record (file not empty) (COC) Codes are Consistent X X (X) X X (X) (VIR) Values are in Range >=0 X X (X) X X (X) (X) Default: values are zero or positive (Min=zero) (VCO) Values are Consistent X X X X X X X (X) (X) (VAD) Values for Aggregates are consistent with Details (X) = X X X (X) (X) Mandatory if aggregates and details are provided Default: aggregates = sum of details (VNO) Values are Not Outliers X X (X) X (X) (VSA) Values for Seasonally Adjusted data are plausible X X X (X) (X) (RRL) Records Revised are Limited X (X) (X) X (X) (VRT) Values are Revised within a Tolerance level X (X) (X) X (X) (VMP) Values for Mirror data are Plausible X (X) X (X)

20 Main types of validation rules Focus on 8 rules from Group 3 Rule type Mand atory Default Validation level SDMX Micro Severity level data E W I Comments (REP) Records Expected are X X X X (X) Provided (RNR) Records Number is in a Range X >=1 X (X) X X (X) (X) Default: at least one record (file not empty) (COC) Codes are Consistent X X (X) X X (X) (VIR) Values are in Range >=0 X X (X) X X (X) (X) Default: values are zero or positive (Min=zero) (VCO) Values are Consistent X X X X X X X (X) (X) (VAD) Values for Aggregates are consistent with Details (X) = X X X (X) (X) Mandatory if aggregates and details are provided Default: aggregates = sum of details (VNO) Values are Not Outliers X X (X) X (X) (VSA) Values for Seasonally X X X (X) (X) Adjusted data are plausible 20

21 QUESTIONS? 21

22 Time for a Coffee break 22

DIRECTORS OF METHODOLOGY/IT DIRECTORS JOINT STEERING GROUP 18 NOVEMBER 2015

DIME/ITDG SG November 2015 DIRECTORS OF METHODOLOGY/IT DIRECTORS JOINT STEERING GROUP 18 NOVEMBER 2015 Item 03 of the agenda ESS Vision 2020: ESS.VIP VALIDATION 1. Purpose of the document ESS Vision 2020: