Data Warehouse Testing Best practices to improve and sustain Data Quality Getting ready for Serious DevOps

Transcription

1 Data Warehouse Testing Best practices to improve and sustain Data Quality Getting ready for Serious DevOps Ajay Nalabhatla, QA Lead Srihari Gopisetty, Technology Manager Wells Fargo India Solutions 1

2 Abstract In the age of Digital disruption, every organization wants to transform its technology arm in to advance practices - DevOps, to and fulfill the continuous demand from Business. However, all organizations are data driven and they need to realize that the success does not rely only on faster throughput and speed but also on the ability to access the diverse, large volume of complex data at real time to make strategic decisions. Important question is Does the organization have Quality Data? On average, U.S. organizations believe 32% of their data is inaccurate -Gartner Average organization loses $8.2 million annually due to poor Data Quality. -Experian Less than 0.5% of all data is every analyzed -Forrester Even as many organizations are establishing the Data Warehouse Testing as specialized service, recent surveys indicate that much more improvements needs to be done. It is a call to-action for organizations to address Data Quality gaps 2

3 Setting the context Data are of high quality "if they are fit for their intended uses in operations, decision making and planning. J.M.Juran (Source dqglossary.com) Data Quality Issue Drivers QA Key Causes Unavailability of Complete Data No Exhaustive Validation Consistency Accuracy ETL Transformation Missing Defined Test Strategies Validity Dimension Integrity Delayed Batch SLA Lack of Tools / Accelerators Incomplete DB Objects Validation Timeliness Completeness Batch Performance Missing End -to- End QA Framework Obsolete Jobs & Records Missing Standard Process REMEMBER : Poor Data Quality = Use of Less Information for Decision Making 3

4 Where, What, Why Heterogeneous Sources Staging/ODS Data Warehouse Applications BI DB OLTP OLAP Internal Sources Ascii Xml ETL Tables Other DB Objects Views ETL Views DB Objects Reports Ebcdic External Sources Extracts Downstream Apps Extracts Downstream Apps High Level Tests Static Testing ETL Transformatio n Testing Staging /ODS Validation Data Warehouse Validation Data Quality & Objects Validation Batch Performance BI Testing / Extracts High Medium Low 4

5 QA Framework For High Quality Exhaustive validation at every intermediate check point Runbook and scheduler/dependence validations Data Integrity Validation RI checks etc. Database Objects Validation Partitions, Synonyms, Flashback etc. Heterogeneous sources validation Xml/Ascii /Ebcdic Batch Performance Execution Database Privileges validation at Table/View/Report Level BI reports UI,Data & Performance validation 5

6 Specialized Testing Database Object Validation - Partition Users View DBA s View Test Strategy Database Table Regression Day Wise Partition Interval & Merge Purge >13M Test Flow Day2 Feb 17 Feb 17 Day3 Day 31 Jan 17 Feb 18 Dec 17 Feb 18 Source Inside DB Table 1 st time Load Daily Loads Monthly Purge >13 Months 6

7 Automation Possibility = Automation possibility Extracts Market Tools Parameter Home Grown Tools External Tools Static Testing Limited Limited Source File Metadata / Layout / Fields Order Yes Macro / UNIX Shell Yes Exhaustive Validation Diff Server DB s Yes Macro / UNIX Shell Yes for Both Batch Job execution in sequence Yes - UNIX No Heterogeneous File Load & Comparison (ASCII / XML / EBCEDIC) Yes ASCII / XML only Yes Regression Testing Tables /Extracts Yes Macro / UNIX Yes Data Quality checks Yes Macro / UNIX Yes Table Metadata validation Yes Macro / UNIX Yes BI Reports Validation (Data /Graphs) Yes Only Data Yes Batch Performance Testing Yes -UNIX Yes Yes Macro using ODBC / UNIX Yes Views Validation Partition /Index validation No No Test Cases Batch Execution yes Unix / Excel Yes Automate Test Execution Scheduler yes Unix Yes 7

8 Integrate the ETL testing In DevOps Usual Architecture for ETL Static Testing ETL/DW Testing Batch Performance Testing 8

9 Demo Overview CD CT CI Static Testing ETL/DW Testing Batch Performance Testing 9

10 Sources & ETL Jobs ETL Jobs SQL - Test cases Shell Script 10

11 Data Validation Test case.sql file.sql in Script MySQL Test 11

12 Jenkins Dashboard

13 Jenkins First job Creation

14 Adding Subsequent job(s)

15 ETL Test case Execution Job

16 Dependency Scheduling for Pipeline

17 GitHub Add WebHook with Jenkins

18 Jenkins GitHub for Automatic Triggering Jenkins Scheduling using Poll Feature

19 GitHub, Add files to GitHub

20 Pipeline For Sequence Execution Batch Execution Start Green =Success Batch Execution Finish

21 Console o/p for jobs

22 Final Pipeline O/p Test Results document on Jenkins Test Results on Linux

23 Build Execution History - Useful for Batch performance testing 23

24 DevOps Readiness Require thorough knowledge on Line of Business Data flow Runbook availability following predecessors & successors Identification of suitable Test approach based on project types Pre-Requisites Availability of Test data that represent all needs Alternative analysis to avoid table unusable state issues Ensure table Referential Integrity is addressed Availability of TDM team for table refresh to previous state in case of failures Batch performance SLA prediction Focus about - Cultural, Process and Tools 24

25 Benefits Improved Business confidence on Quality Blueprint That helps companies to gear up Possible Faster Iteration, Quick Feedback, Great Collaboration Low-priced Automation possibilities Insights on various Database Object Validations Early Defect Detection 25

26 26

27 Author Biography Ajay Nalabhatla Ajay Nalabhatla works as a Data Management QA Lead at WellsFargo for a Line of Business. He has around 11+ years of experience in Quality Assurance for ETL, DWT & BI Testing. Over a decade, he has involved in various DWT projects for different Banking & Securities Clients and delivered them successfully. He was also involved in conducting the Due Diligence for various DWT clients and also suggested them many improvements in both process and Technical competencies. Ajay s holds Bachelor of Engineering in Electronics & communication from Anna University Srihari Gopisetty Srihari Gopisetty is managing the Data Management and Digital Advisory Teams for a Line of Business at Wells Fargo India Solutions. He has more than 17 years of rich experience in leading teams for BFSI Domain and Microsoft Products. Prior to Wells Fargo, he has worked with Microsoft, First Advantage Srihari s educational background includes Bachelor of Engineering in Mechanical from Gulbarga University 27

28 References & Appendix

29 APPENDIX 29

30 Test Approach - Data Migration Pre - Migration Post -Migration Steady State Analyze Db and identify the objects Tables/Indexes/views Compare all DB objects Between Legacy & New Steady state Validations for Monthly loads Segregate wave wise plan - Forklift,Consolidation, static, Dynamic tables /views Validate data for all tables b/w legacy & new identified for migration Downstream App s support Validate New tables transformation rules are as per specs Take the snapshot for Post comparison Parallel load comparison for tables between legacy & new Validate Purging process is as expected for New system Prioritize / Create Batch & collect pre-run stats Execute Batch performance testing & compare stats with Legacy Performance Monitoring for Data Load 30

31 Infrastructure Upgrade Test Strategy - ETL tool /Scheduler tool upgrade Job Types Direct SQL processing Pre Migration Test Strategy Identify the various types of jobs Identify the Priority Jobs for phase wise execution Build run book for the phases with all dependencies Collect the batch statistics Take the snapshots of relevant table data ETL Jobs File watcher / Legacy jobs using c/c++ Store Procedure jobs Post Migration Steady State Execute the identified jobs on upgrade system Validate the batch / job performance with pre-stats Regression Testing for tables Parallel Load processing & data validation Validate Job dependencies and predecessors Batch performance monitoring. Analyse the failures to understand if they are upgrade related Steady state support for downstream applications 31

32 Thank You!!! 32