Abstract. Duplicate record checks

Transcription

1

2 Profile Koushik Kadimcherla is a Test Analyst with Infosys Limited. He has 4.4 years of experience in the IT industry. Koushik has been working on Data warehouse testing projects from the past 4 years. His technical skills include C, Java, Basics of.net programming, Ab initio, UNIX Shell Scripting, Oracle 9i and Teradata Sashidhar Pamidimukkala is a Test Analyst with Infosys Ltd and has 4.4 years of experience in the IT industry. He is involved in S/W testing projects for the banking clients and has experience in testing Data warehouse applications and BI reports. Sashidhar has worked on ETL and reporting tools namely Ab initio and Business Objects. Currently Sashidhar works on test planning and test execution on Data warehouse applications.

3 Abstract Extract Transform and Load (ETL) testing involves huge volume of data to be tested across a Data warehouse. It involves various data validations across multiple heterogeneous systems. Significant amount of manual effort needs to be utilized for performing many repetitive steps as stated below: Comparing record counts of source and target Comparing hash sums of source and target Performing exhaustive data validations between source and target Duplicate record checks All the above activities consume a lot of effort in a traditional approach especially when there are billions of records for comparison thereby making the test execution a hectic and challenging job to meet the timelines with full test coverage This paper highlights the strategies which can be an efficient and cost effective solution that have been implemented based on the project experiences.

4 ETL Test Scenarios The below test scenarios come into picture while testing the above two ETL flows: Basic validations Note the record count in source & target Verify if above record counts are matching or not Note the hash sum of key fields in source & target Verify if above hash sums are matching or not Check for duplicate records SQL queries will be used on database tables and UNIX commands will be used for UNIX flat files to perform the above activities Data validations In the 1 st fig, as ODS and DW are different environments, data should be populated into excel sheets and compared using macros/excel compare formula. Sampling technique should be used if data in both the tables is very huge as excel cannot accommodate after certain limit. All the above tasks are handled manually and consume significant effort to validate the same on multiple sets of tables.

5 Challenges in traditional ETL test execution Lack of Full test coverage: Since the data comparison is done manually on a spreadsheet, only a limited sample of records will be covered leading to a possibility of missing the defects over the remaining records Complex Business Logic simulations: Handling of business logic like lookup.need to use combination of scripting,sqls and Manual Verifications Ex: EmpID field in Employee table (ODS) has to look up Employee Details table(dw) for Employee Full name. Execution time: Execution time for manual approach is always time taking and needs significant effort during cross comparisons (File and table comparisons) Meeting the deadlines: Huge volume of data to be tested across Extract-Transform-Load (ETL) objects like tables. Multiple resources should be involved to meet the timelines. Every resource should be technically competitive Ex: For ETL scenarios in previous slide, every tester should have good knowledge on SQL queries and UNIX commands

6 Proposed Solution In most of the ETL applications, UNIX servers are used as staging medium. Using shell scripting automation script could be developed for better test coverage. Automation mechanism addresses the test execution challenges with this architecture

7 Proposed Solution (Contd.) The Automation script should have below automatic capabilities to save manual effort: Read data from source object (file or table) Read data from target object (file or table) Create a clone of ODS table or UNIX file in the Data warehouse in tabular format Perform all the Extract Transform and Load test validations (record/hash counts, duplicate record checks and data validations) Generate a log file to track the test results to store the test results for record counts/hash sum and other data validation results with the help of automation mechanism for making the test results reporting process very easy and readable Syntax to run the automation script: Automation script should be designed to read 4 parameters at run time as shown below: Source object Target object Cloned Temp table layout (ODS or UNIX file clone on DW) SQL validation file (Validation SQL queries) <Automation script> <Source object> < Target object> <Temp table layout file> <SQL validation file>

8 Key Benefits Automation mechanism for ETL validations could significantly cut down the test execution effort by 80% compared to manual and repetitive execution This mechanism could be reused across several ETL testing projects with minimal customization in the automation script Creating simple log files to store the test results for record counts/hash sum and other data validation results with the help of automation mechanism would make the test reporting process very easy Helps to show good productivity with few resources Resources need not be technically competitive on SQL and UNIX

9 Results based on an ETL project for Financial Advisors Client Value Levers Business Process Lever Engineering Impact Engineering Metrics Change Enablers Reduction in cost(90 KUSD) Efficient Business reporting & reduce time to service customers by Financial Advisors (40%) Savings in Testing Effort (80%) Productivity improvement (14%) Optimized business process Automation of manual processes Automation-% of test case automated (60%)

10 Any Questions

11 Thank You!