DATA QUALITY STRATEGY. Martin Rennhackkamp

Transcription

1 DATA QUALITY STRATEGY Martin Rennhackkamp

2 AGENDA Data quality Data profiling Data cleansing Measuring data quality Data quality strategy Why data quality strategy? Implementing the strategy

3 DATA QUALITY OBJECTIVES How can we achieve our business objectives? With appropriate, complete and accurate data Reduce costs in the right Increase context, profits format at the correct time and place for the right purpose. Improving Data warehouse and Business Information Quality Larry English, 1999

4 DATA QUALITY APPROACH Know Your Data (Data Profiling) Fix Your Data (Data Cleansing) Business Application Projects System Replacements Data Warehouse / ETL Data Integration Enhance Your Data (CRM, Householding) Data Quality Initiatives Root Cause Analysis & System / Process Remediation Metadata Management Data Stewardship & Governance Monitor Your Data (Data Profiling)

5 Data quality AGENDA Data profiling Data cleansing Measuring data quality Data quality strategy Why data quality strategy? Implementing the strategy

6 DATA PROFILING Software and analytical techniques thorough knowledge Data: content, structure, correctness and quality Diagnostic step in data quality management

7 ANALYSIS THRASHING Multiple meanings Cleanliness Multiple sources USERS Accuracy Completeness Timeliness History complications Audit and validation

8 DATA PROFILING TECHNIQUES Domain analysis Pattern analysis Statistical analysis Range or threshold checking Cross-table identification Business rule analysis Customer profiling

9 DOMAIN ANALYSIS Data type adherence: numeric and dates Domains: dates, numbers, alphanumeric Nulls and spaces Frequency and % of distinct values Primary and alternate key uniqueness

10 PATTERN ANALYSIS Specific data type formats: ID number, Tax number Postal code Telephone and fax numbers Dates formats Specific data values: N/A unknown

11 STATISTICAL ANALYSIS Record count Sum Mode value occurring most often Minimum / Maximum Percentiles Mean Standard deviation Hash total / control total CRC

12 RANGE ANALYSIS Ranges of valid values zero or negative numbers allowed past or future dates minimum and maximum values Thresholds salary over given value? interest rate <= 0 negative amount fields Sets of valid values discrete set of permitted values e.g. flags (Y/N), types, codes

13 CROSS-TABLE ANALYSIS Referential integrity between two tables B.ForeignKey = A.PrimaryKey Look-up values Possible PK-FK identification if not defined Primary keys Possible primary and alternate key identification Referential integrity in one table Hierarchical references

14 CROSS-TABLE ANALYSIS

15 BUSINESS RULE ANALYSIS Consistency checks of relationships: Every product in the order table in the product table? No autopsy result if date of death is blank. Accident claim reason only if insured item is a vehicle. Can only have pregnancy test if gender = female. Cannot have 0 monthly interest on fixed deposit saving account. Cannot have first payment date before activation date. Richness of profiling product s business rule detection and reporting

16 CUSTOMER PROFILING TOOLS Focus on names and descriptive data Recognize duplicates and variations Standardize international names, phone numbers and addresses USA zip code vs UK postal code USA states vs UK counties vs cantons vs provinces Parse names and addresses Propagate addresses into households

17 ON-GOING PROCESS Periodical data profiling Analysis on source systems Analysis on data warehouse Re-extraction, analysis, cleansing and re-insertion into data warehouse First step in repeating ETL processes Regular data quality benchmarks (CIO dashboard)

18 BI METHODOLOGY Technical Architecture Product Selection & Installation Project Planning Business Requirements Data Profiling Data Warehouse Design Data Warehouse Implementation ETL Specification Information Delivery Specification Data Warehouse Maintenance ETL Implementation Information Delivery Implementation Testing, Deployment & Roll-out Maintenance Project management Information management Metadata management

19 WHY DATA PROFILING? Understand the data Understand requirements Accurate database design Risk mitigation: ETL is the most under-estimated, under-budgeted part of most BI projects ETL code-load-explode Initiate spin-off data quality projects

20 Data quality Data profiling AGENDA Data cleansing Measuring data quality Data quality strategy Why data quality strategy? Implementing the strategy

21 DATA CLEANSING PROJECTS Data entry At data capture time in source system One record at a time interactive mode In contact with user or client Require changes to source system Require changes to capturers KPIs Data extraction At data transfer time from the source system One system at a time Many records at a time batch mode Data integration At data transfer time into data warehouse More than one system at a time different dynamics Many records at a time batch mode Can select best records for consolidation and merging

22 DATA CLEANSING PROJECTS Reporting / down-stream extraction When extracting from data warehouse for data marts, reporting, analytics Many records at a time batch mode Possibly far removed from source system identification Formatting, extraction and transformation rules Could be start of feedback cycle to source system Maintenance On-going across spectrum of systems On-going improvement and raising data quality standard In-place on source or data warehouse system Many records at a time batch mode May introduce reporting inconsistencies over time

23 DATA CLEANSING In transit: During operational system extract* During data warehouse input* During data warehouse / data mart reporting** In place: In operational system In data warehouse / data mart* Techniques: Reject, ignore error, correct, look-up, default value

24 FEEDBACK CYCLE Operational systems Financial Marketing Quality feedback cycle Decision makers On-line Web ERP HR Extract, transform & load Query and analysis tools Operations Data warehouse

25 AGENDA Data quality Data profiling Data cleansing Measuring data quality Data quality strategy Why data quality strategy? Implementing the strategy

26 INHERENT QUALITY MEASURES Data correctness and accuracy Definition conformance Consistency of the meaning and definition of data Completeness (coverage) All required values for all required fields Validity Data values conform to domain and business rules Accuracy to source Data agrees with data values in original source Accuracy to reality Accurate reflection of real-world object or event Precision Values to the right degree of granularity Non-duplication One-to-one correspondence between records and real-world objects or events

27 RAGMATIC QUALITY MEASURES Usefulness of data Accessibility Able to access the data on demand Timeliness Data available to support process within required time table Contextual clarity Data presentation enables understanding of its meaning Avoid misinterpretation Derivation integrity Correctness with which two/more data elements are combined Usability Ease of use of the form of data presentation Rightness Right kind of data with right degree of quality to support a given process

28 MEASURING DATA QUALITY Aspect Measure Completeness Percentage of data available Accuracy Timeliness Delivery Usability Percentage of records with errors Percentage of time data available when needed Percentage of users satisfied with delivery method Subjective: degree to which presentation is purpose-efficient

29 MEASURING DATA QUALITY Completeness Cost Accuracy Delivery Timeliness

30 CIO DASHBOARD

31 APPROACH Cultural issue Business buy-in Responsibilities Methodology, processes and procedures Quality feedback cycle Governance and controls => Data Quality Strategy

32 AGENDA Data quality Data profiling Data cleansing Measuring data quality Data quality strategy Why data quality strategy? Implementing the strategy

33 DATA QUALITY STRATEGY Get stakeholder involvement, budget and go-ahead Hand over to programme management for implementation

34 DATA QUALITY STRATEGY Context Storage Data flows Stewardship Workflow Monitoring Appendices: - Profiling: technique vs storage - Cleansing: Data detail vs storage - Ownership/Stewardship: Data entities vs storage

35 DATA QUALITY STRATEGY Context: The types and usage of data => type of cleansing required specialized data requires specialized cleansing e.g. titles, dates, numbers, addresses

36 DATA QUALITY STRATEGY Context: The types and usage of data => type of cleansing required Storage: System architecture and platforms => where & how cleansing done technology: mainframe / UNIX locality: locally, regionally, internationally formats: tables: files, spreadsheets applications: open (RDBMS), proprietary connectivity: ODBC, proprietary

37 DATA QUALITY STRATEGY Context: The types and usage of data => type of cleansing required Storage: System architecture and platforms => where & how cleansing done Data flows: Capture, movement and migration of data => cleansing opportunity Transactional updates Operational data feeds Purchased data Legacy migration Regular maintenance (batch) Special extracts

38 DATA QUALITY STRATEGY Context: The types and usage of data => type of cleansing required Storage: System architecture and platforms => where & how cleansing done Data flows: Capture, movement and migration of data => cleansing opportunity Stewardship: People and organisation => who is involved Stakeholders: buy-in, mandate, budget Data stewards: roles, responsibilities Organisational structure: impact Education: all stakeholders

39 DATA QUALITY STRATEGY Context: The types and usage of data => type of cleansing required Storage: System architecture and platforms => where & how cleansing done Data flows: Capture, movement and migration of data => cleansing opportunity Stewardship: People and organisation => who is involved Workflow: Tasks and processes involved => process improvement opportunity Eliminate non-value-add tasks Add data improvement tasks

40 DATA QUALITY STRATEGY Context: The types and usage of data => type of cleansing required Storage: System architecture and platforms => where & how cleansing done Data flows: Capture, movement and migration of data => cleansing opportunity Stewardship: People and organisation => who is involved Workflow: Tasks and processes involved => process improvement opportunity Monitoring: Measuring quality over time => Measure against baseline Controls to improve

41 PROFILING STRATEGY Storage Profiling Source system Staging area Data warehouse Data mart Report / analytics Domain analysis D:1, T:1, P:m D:1, T:y, P:m D:1, T:m, P:m D:1, T:m, P:m D:1, T:m, P:m Pattern analysis D:1, T:1, P:m D:1, T:y, P:m D:1, T:m, P:w D:1, T:m, P:w D:1, T:m, P:w Statistical analysis D:1, T:1, P:m D:1, T:y, P:m D:1, T:m, P:d D:1, T:m, P:d D:1, T:m, P:w Range analysis D:1, T:1, P:m D:1, T:y, P:m D:1, T:m, P:d D:1, T:m, P:d D:1, T:m, P:w Relationship analysis D:1, T:1, P:m D:1, T:y, P:m D:1, T:m, P:w D:1, T:m, P:w D:1, T:m, P:w Business rule analysis D:1, T:1, P:m D:1, T:y, P:m D:1, T:m, P:d D:1, T:m, P:d D:1, T:m, P:w D:development T:test P:production 1:once-off d:daily w:weekly m:monthly y:yearly

42 CLEANSING STRATEGY Storage Cleansing Source system Staging area Data warehouse Data mart Report / analytics ACCOUNT Account no Account name Ignore Ignore Reject Ignore Ignore Ignore Ignore Ignore Ignore Ignore Household Ignore Household Lookup Lookup Ignore Contact Ignore Ignore Parse Ignore Ignore Language Ignore Default Lookup Lookup Ignore Address Ignore Ignore Parse Ignore Ignore Postal code Lookup & replace Ignore Lookup Lookup Ignore Balance Ignore Ignore Ignore Ignore Ignore

43 OWNERSHIP / STEWARDSHIP Storage Entity Source system Staging area Data warehouse Data mart Report / analytics ACCOUNT CUSTOMER PRODUCT ORDER DELIVERY HR FINANCE Own: CR Cust: CR Own: CR Cust: CR Own: Prod Cust: Prod Own: Prod Cust: Prod Own: Prod Cust: Prod Own: HR Cust: CTO Own: CFO Cust: FinIT Own: CR Cust: EDW Own: CR Cust: EDW Own: Prod Cust: EDW Own: Prod Cust: EDW Own: Prod Cust: EDW Own: HR Cust: EDW Own: CFO Cust: EDW Own: CR Cust: EDW Own: CR Cust: EDW Own: Prod Cust: EDW Own: Prod Cust: EDW Own: Prod Cust: EDW Own: HR Cust: EDW Own: CFO Cust: EDW Own: CR Cust: CR Own: CR Cust: CR Own: Prod Cust: EDW Own: Prod Cust: EDW Own: Prod Cust: EDW N/A Own: CFO Cust: FinIT Own: CR Cust: CR Own: CR Cust: CR Own: Prod Cust: BI Own: Prod Cust: BI Own: Prod Cust: BI Own: HR Cust: BI Own: CFO Cust: FinIT

44 AGENDA Data quality Data profiling Data cleansing Measuring data quality Data quality strategy Why data quality strategy? Implementing the strategy

45 WHY DETAIL STRATEGY? Control redundant data Get knowledge of data available Avoid inconsistent overlapping data sets Avoid inconsistent reports Avoid late data availability Formally assign data responsibility Set and adhere to standards Get management to understand importance of data quality

46 GAIN CONTROL Set consistent and unassailable data quality controls Define data ownership and stewardship Increase trust in data Improve delivery and responsiveness Consistent use of terminology Technical and business Align with corporate strategy Get buy-in and support from group management level Group-wide vendor license power

47 Data quality Data profiling Data cleansing AGENDA Measuring data quality Data quality strategy Why data quality strategy? Implementing the strategy

48 GOVERNANCES Source systems: - BI readiness / enablement - Levels of data quality as KPIs - Data capturing quality KPIs Data warehouse: - Levels of data quality as KPIs Information usage: - Levels of data quality as KPIs - User training and certification License to drive Metadata compliance: - Source systems - Data warehousing and related processes - Information exploitation Quality control: - End-to-end data quality program - Periodic data quality measurements and feedback - Data quality measures on CIO dashboard - Data quality in all personnel s KPIs

49 RESPONSIBILITIES Business ownership of data: - Data treated as an asset - Aligned with business strategy - Thorough end-to-end process Source data alignment: - Source of data creation and maintenance - Interface requirements and service levels Analysis and certification: - Data quality and acceptance metrics - Acceptance by business-representative data stewards - Agreed-on cleansing transformations Monitoring: - On-going frequent audits - CIO dashboard Change control: - All source system changes that may affect data quality - All data warehouse changes that may affect data quality - Metadata management

50 COMPLICATIONS Existing perceptions Legacy data Informal data (Excel, Access, data on users PCs, etc) Existing tools and systems Accidental data warehouse Yesterday s requirements Existing power-play, politics, agendas and domains of control Uncooperative role-players

51 BEST PRACTICE Don t get into too much detail too soon Don t lead with long-term deliverables (Get short-term on the table and accepted, get the project started) Don t be a theorist Use practical actionable steps Don t commit more than you can deliver Avoid unproved technology Align with corporate strategy (Or fight never-ending unpopular battle) Play the P s correctly Politics and Partnership Sell strategy and approach in organisation Initially hard for buy-in and go-ahead On-going subtly to justify existence and promote workings

52 COMMUNICATION Publicize objectives: What data was corrected Future planes Report progress Users know to what extent to use which data Re-affirm process to users: Commitment to the process Build up confidence in the system Changing Perceptions

53 SUMMARY Data quality management approach Data as corporate resource Effects the bottom line End-to-end data quality process Monitor throughout Fix as close to source as possible Data profiling Key step in diagnosing and monitoring data quality Feed CIO dashboard Data quality strategy Defined contents reviewed implementation Required to establish corporate data quality management program Teamwork, partnership and politics Communication