Data Quality Framework

Transcription

1 #THETA2017 Data Quality Framework Mozhgan Memari, Bruce Cassidy The University of Auckland This work is licensed under a Creative Commons Attribution 4.0 International License

2 Two Figures from 2016 The cost of poor quality data, in the US alone, in 2016 $136 billion per year $3.1 trillion per year The size of the big data market, worldwide, in 2016 Bad Data Costs the U.S. $3 Trillion Per Year, Thomas C. Redman, SEPTEMBER 22,

3 Data, Information, Value Organizations collect more data these days Data-driven decision making to gain competitive advantage Delivery of crucial information at right time Transforming data into value 3

4 Data Quality Are data trustworthy? Are data fit for purpose? Data quality is an assessment of information's fitness to serve its purpose in a given context. 4

5 Agenda Data quality Framework: definition and objectives Methodology Quality Dimensions and Measures Technical design and Implementation in the data warehouse Wrap up 5

6 The Data Quality Framework An assessment and measurement tool, integrated into organisational process, providing a benchmark for the effectiveness of any future data quality improvement initiatives and a standardized template for information on data quality both for internal and external users. The development of a data quality framework and strategy for the New Zealand Ministry of Health, Dr. Karolyn Kerr 6

7 The Data Quality Framework Objectives: Specific guides on systems and integration efforts to supply Useful and Trustworthy data Measuring the level of confidence in data continuously Continuous data quality issue diagnosis Data quality governance A strategy for defining quality objectives measuring the quality enacting improvements 7

8 The Data Quality Framework Conceptual model Governance of the data quality in the organization Roles and responsibilities Service delivery Technical model Defines different metrics to calculate Implementation 8

9 Methodology Existing Methodologies 1 Total Data Quality Management (TDQM) 2 Total Information Quality Management (TIQM) 3 Data Warehouse Quality (DWQ) 4 Data Quality Assessment (DQA) 5 The University of Auckland Data Quality Framework Methodology: combines DQA & DWQ methodologies focuses on the process of assessment, maintenance and improvement of the quality of the data stored in data warehouse 1 Batini et al Wang et al English Jeusfeld et al Pipino et al

10 Methodology 10

11 Methodology The University of Auckland Data Quality Framework Methodology: combines DQA & DWQ methodologies focuses on the process of assessment, maintenance and improvement of the quality of the data stored in data warehouse Objective assessment: Different techniques are applied to define the data quality values and levels of each critical object and then the results are compared with the acceptance criteria and goals Subjective assessment: The perceptions, information needs and data quality requirements of the stakeholders and business users are defined and measured by subjective metrics 11

12 Quality Dimensions and Measures Data quality dimensions are defined to interpret data quality into practical and measurable concepts The University of Auckland Data Governance Policy proposes Completeness Validity Consistency Accuracy And. Interpretability Accessibility Timeliness Reasonableness 12

13 Quality Dimensions and Measures Completeness dimension: Data completeness indicates if all the data necessary to meet the current and future business information demand are available in the data resource. Validity dimension: Validity is to ensure correctness and reasonableness of data. Consistency dimension: The consistency refers to the validation of semantic rules defined over a set of data items. Accuracy dimension: Data accuracy defines if the data values stored in an object are the correct values. 13

14 Quality Dimensions and Measures Accessibility dimension: Accessibility refers to the physical conditions in which users can obtain and access data when required. Interpretability dimension: Interpretability is the extent to which the data are semantically meaningful for the specific analytical purposes. Reasonableness dimension: Reasonableness refers to some agreed value to the data items that help to the understanding of the aspect of the business activity being reported. Timeliness dimension: Timeliness of information reflects the length of time between the availability of data and the event or phenomenon they describe. 14

15 Quality Dimensions and Measures 15

16 Completeness Completeness is defined as "the degree to which data values are present in the attributes that require them." Completeness Definition The proportion of stored data against the potential of "100% complete" Reference Domain integrity and Business rules which define what "100% complete" represents. Measure 7 metrics Completeness Coherence completeness 1 Coherence completeness 2 Scope Unit of Measure Purpose of Measure Type of Measure Related data quality dimension Optionality 0-100% of critical data to be measured in any data item, record, data set or database in the UoA data warehouse Percentage, Ratio Assessment, repair Assessment, Continuous and Discrete Validity and Accuracy If a data item is mandatory, 100% completeness will be achieved, however validity and accuracy checks would need to be performed to determine if the data item has been completed correctly Tuple-based completeness (compares source to target) Column-Null-based completeness Tuple-Null-based completeness (each row at entity) Metadata level Null-based completeness Schema-based completeness 16

17 Quality Dimensions and Measures 1.Discuss the business problem 2.Define the goals, priorities and the acceptance limits Goal Accepted limit Priorities 3. Metrics for the measurements Metric What to test Output (Table level, column level, ) 17

18 Quality Dimensions and Measures Completeness Pilot study: 1. The business problem: DQ for Dim Employee 2.Define the goals, priorities and the acceptance limits Goal Accepted limit Priorities Total completeness 100% 95% completeness Employee info 3. Metrics for the measurement Metric What to test Output (table or column level) Coherence completeness 1 Coherence completeness 2 Tuple-based completeness (compares source to target) Column-Null-based completeness Tuple-Null-based completeness (each row at entity) Metadata level Null-based completeness Schema-based completeness 18

19 Objective Assessment Completeness In ETL process, or specific timeline based on application Metric What to test Output Coherence completeness 1 Coherence completeness 2 Tuple-based completeness Column-Null-based completeness Count and compare the number of unique records at source & target Unique values from PS_names to DIM_EMPLOYEE Compare the not-null values populated from source fields into the target table over the related fields For x-1 to x-5 in DIM_EMPLOYEE Count the required tuples from PS_names to DIM_EMPLOYEE Calculate the completeness over each field then over the entity Table level Column level Table level Table level Column level One value for each target table One value for each field in the target table Average of the fields completeness (CC3_Employee) One value for each field in the target table Completeness % over the fields (x_i) in target table DIM_EMPLOYEE Table level Average of the fields completeness 19

20 Completeness Pilot Study Metric What to test Output Coherence completeness 1 PS_names to DIM_EMPLOYEE dim_employee Results: 100% Completeness Coherence completeness 1 Coherence completeness 2 Actual Key PS_names DIM_EMPLOYEE Business key Tuple-based completeness (compares source to target) Column-Null-based completeness Tuple-Null-based completeness (each row at entity) Compares unique values over the unique identifiers (considering the business rules) ETL Process Coherence completeness 1 Metadata level Null-based completeness Schema-based completeness 20

21 CC2_employee_x_1 CC2_employee_x_2 CC2_employee_x_3 Completeness Pilot Study Metric What to test Output Coherence completeness 1 PS_names to DIM_EMPLOYEE dim_employee Results: 100% Coherence completeness 2 For x-1 to x-5 in DIM_EMPLOYEE Column level: Ave(CC2_employee_x_i) Results: 100% Completeness Coherence completeness 1 Coherence completeness 2 PS_names y-1 y-2 y-3 y-4 y-5 DIM_EMPLOYEE x-1 x-2 x-3 x-4 x-5 Tuple-based completeness (compares source to target) Column-Null-based completeness Tuple-Null-based completeness (each row at entity) Ave(CC2_employee_x_i) Metadata level Null-based completeness Schema-based completeness Compares the not-null values populated from source fields into the target table over the related fields ETL Process Coherence completeness 2 21

22 Completeness Pilot Study Metric What to test Output Coherence completeness 1 PS_names to DIM_EMPLOYEE dim_employee Results: 100% Coherence completeness 2 For x-1 to x-5 in DIM_EMPLOYEE Column level: Ave(CC2_employee_x_i) Results: 100% Tuple-based completeness PS_names Required tuples from PS_names to DIM_EMPLOYEE DIM_EMPLOYEE dim_employee Results: 100% Completeness Coherence completeness 1 Coherence completeness 2 Tuple-based completeness (compares source to target) Column-Null-based completeness Tuple-Null-based completeness (each row at entity) Metadata level Null-based completeness Compares the required tuples at source with the tuples that have been populated ETL Process Tuple-based completeness Schema-based completeness 22

23 Consistency Pilot Study Uniqueness checking for primary keys in source tables For STAGE_HR9.PS_PERSONAL_DATA Unqi_AC = 100 % Uniqueness checking For business keys in target tables For DSS.Dim_person Unqi_BR = 99.93% Consistency Meta data Duplication Uniqueness checking STAGE_HR9.PS_PERSONAL_DATA emplid Person_key DSS.Dim_person emplid Data profiling Dependency checking Referential integrity checking test For each entity and for each set of columns X that should be unique as per the business requirements ETL Process For each entity and for each set of columns X as the business key ETL Process Domain integrity Business rules 23

24 Consistency Pilot Study Consistency Dependency checking For each defined dependency X >Y in the target entity position_nbr ----->position_descr in DSS.Dim_person integrity_1=100% Meta data Duplication Uniqueness checking Data profiling DSS.DIM_POSITION position_descr position_nbr Dependency checking Referential integrity checking test For each defined dependency X ----->Y in entity r ETL Process Domain integrity Business rules 24

25 Repository and Reports Ranges Color code 25

26 Repository and Reports 26

27 Summary Fitness and Trustworthiness of data has become very important Data quality framework defines a model of the organization data environment identifies relevant data quality dimensions and measures provides a guidance for data quality improvement Objective and Subjective assessment of data quality 27

28 Discussion