ETL Testing Concepts:

Transcription

1 Here are top 4 ETL Testing Tools: Most of the software companies today depend on data flow such as large amount of information made available for access and one can get everything which is needed. This is where the concept of ETL and ETL Testing comes into the picture. Basically ETL is abbreviation used for Extraction, Transformation and Loading. Presently ETL Testing is performed using SQL scripting or using spreadsheets which may be time-consuming and error-prone approach. In this article, we will have detailed discussions for several concepts viz. ETL, ETL Process, ETL testing and different approaches used for it along with ETL testing tools. ETL Testing Concepts: #1) As mentioned previously ETL stands for Extraction, Transformation and Loading which are three database functions where; Extraction: Reading data from database Transformation: Converting extracted data in the required form to store into another database Loading: Writing the data into target database #2) ETL is used to transfer or migrate the data from one database to another, to prepare data marts or data warehouses Following diagram elaborates the ETL Process in precise way

2 ETL Testing Process: ETL Testing Process is similar to other testing processes that includes following stages; Identifying business requirements Test Planning Designing test cases and test data Test execution and bug reporting Summarizing reports Test closure Types of ETL Testing ETL Testing can be categorized into following categories according testing process been followed; 1) Production Validation Testing: It is also called Table balancing or product reconciliation. It is performed on data before or as it is being moved into production system in correct order 2) Source To Target Testing: This type of ETL Testing is performed to validate data values after data transformation 3) Application Upgrade: It is used to check whether the data is extracted from older application or new application or repository 4) Data Transformation Testing: Multiple SQL queries are required to be run for each and every row to verify data transformation standards 5) Data Completeness Testing: Performed to verify that the expected data should be loaded at destination as per predefined standards I would also like to compare ETL Testing with Database Testing but before that let us have a look towards types of ETL Testing with respect to database testing; 1) Constraint Testing: Testers should test whether data is mapped accurately from source to destination, while checking for it testers need to focus on key checks (constraints) such as; NOT NULL UNIQUE Primary Key Foreign Key Check NULL Default

3 2) Duplicate Check Testing: Source and target tables contains huge amount of data with frequently repeated values, in such case testers follow some database queries to find such duplication. 3) Navigation Testing: Navigation concerns with GUI of the application. User finds application friendly when he gets easy and relevant navigation throughout entire system. Tester must focus on avoiding irrelevant navigation through user point of view. 4) Initialization Testing: Initialization Testing is performed to check combination of hardware and software requirements along with platform it is installed 5) Attribute Check Testing: This testing is used to perform for verifying all attributes of source and target system that should be same From above listing one may consider that ETL Testing is quite similar to Database Testing but the fact is ETL Testing is concerned with Data Warehouse Testing and not Database Testing. There are several other facts due to which ETL Testing differs from Database Testing, let s have quick look towards it one by one. 1) The primary goal of Database Testing is to check if the data follows the rules and standards of data model where on the other hand ETL Testing checks if data is moved or mapped as expected 2) Database Testing focuses on maintaining primary key-foreign key relationship while ETL Testing verifies for data transformationas per requirement or expectation and same at source and target 3) Database Testing recognizes missing data where as ETL Testing determines duplicate data 4) Database Testing is used for data integration and ETL Testing forenterprise business intelligence reporting These are some major differences which makes ETL Testing different from Database Testing. ETL bugs are also of several types such as; Type of bug Calculation Bugs Input/output Bugs H/W bugs Description Final output wrong due to mathematical error Accepts invalid values and rejects valid values Device is not responding due to hardware issues User Interface Related to GUI of an application

4 Type of bug Description bugs Load condition bugs Denies multiple users How to create test cases in ETL Testing: The primary goal of ETL testing is to assure whether the extracted and transformed data is loaded accurately from source to the destination. ETL testing includes two documents; #1) ETL Mapping Sheets: This document contains information of source and destination tables and their references. Mapping sheet provides help to create big SQL queries while performing ETL Testing. #2) Database schema for Source and Destination table: It should be kept updated in mapping sheet with database schema to perform data validation. Best ETL Testing Tools List: Like automation testing ETL Testing can be also automated. Automated ETL Testing reduces time consumption during the testing process and helps to maintain accuracy. Given below are some ETL Testing Automation Tools that are used to perform ETL Testing more effectively and rapidly. #1) Informatica Data Validation Informatica Data Validation provides complete solution for data validation along with data integrity Reduces programming efforts and business risks due to intuitive user interface and built-in operators Identifies and prevents data quality issues and provides greater business productivity Allows 64% free trial and 36% paid service that reduces time and cost required for data validation Official Link: Informatica Data Validation #2) QuerySurge from RTTS

5 QuerySurge is an automated testing tool specifically used for data warehouse testing Verifies, converts and upgrades data through the ETL process Reduces testing time and schedules tests for specific time Builds test scenario and test suits along with configurable reports Commercial tool connects source and target data and also supports real time progress of test scenarios Official Link: QuerySurge from RTTS #3) icedq icedq is designed for ETL Testing, Data Migration Testing and Data Quality Verification Identifies data integration errors without any custom code Supports rule engine for ETL process, collaborative efforts and organized QA process Commercial tool with 30 days trial provides custom reports with alerts and notifications Official Link: icedq #4) Datagaps ETL Validator ETL Validator is data testing tool specifically for automated data warehouse testing ETL Validator is used to check Data Validity, Data Accuracy and also used to perform Metadata Testing Checks Referential Integrity, Data Integrity, Data Completeness and Data Transformation Commercial tool with 30 days requires zero custom programming and improves business productivity Official Link: Datagaps ETL Validator While performing ETL testing several things to be kept in mind by testers, some of them are listed below; Apply suitable business transformation logic Execute backend data-driven tests

6 Create and execute absolute test cases, test plans and test harness Assure accuracy of data transformation, scalability and performance Make sure ETL application reports invalid values Unit tests should be created as targeted standards Conclusion ETL Testing is not only tester s duty but it also involves developers, business analyst, database administrators (DBA) and even users. ETL Testing process became vital because it is required to make strategic decisions at regular time intervals. ETL Testing is being considered as Enterprise Testing though it requires good knowledge of SDLC, SQL queries, ETL procedures etc.