Test Automation for data teams with Tosca BI

Size: px

Start display at page:

Download "Test Automation for data teams with Tosca BI"

Grant Patterson
5 years ago
Views:

1 Data migration / DWH / BI testing Test Automation for data teams with Tosca BI By Daina Dirmaitė I Nov 13, 2018

2 Data Testing Challenges 1. Data models and data mapping documents in many ways represent project requirements and, as such, are unique to data testing. 2. The environments to be tested are complex and heterogeneous. Multiple programming languages, databases, data sources, data targets, and reporting environments are often all integral parts ofthe solution. 3. A good understanding of SQL queries, data profiling methods, Excel, and DB editors is essential. 4. Typically, data integrity issues go undetected unless domain experts discover a discrepancy in a report. At this late stage, it s difficult and time-consuming to unravel and remediate the problem. TEST TYPES: Constraint testing; Source-to-target count comparisons; Source-to-target data validation; Transformations of source data and application of business rules; Duplicate testing; Performance; and so on BENEFITS OF AUTOMATED DWH / BI TESTING: Identify data acquisition (ETL) errors; Reduce test creation and maintenance time; Expose defects earlier when they re faster and easier to resolve; Improve data quality; Eliminate delays from manual testing.

Tricentis Tosca Test Automation @ DevOps Speed Tosca is a Continuous Testing platform that accelerates testing to keep pace with Agile and DevOps.

3 Tricentis Tosca Test DevOps Speed Tosca is a Continuous Testing platform that accelerates testing to keep pace with Agile and DevOps. With the industry s most innovative functional testing technologies, Tricentis Tosca breaks through the barriers experienced with conventional software testing tools. Using Tricentis Tosca, leading companies such as HBO, Toyota, Allianz, BMW, Starbucks, Deutsche Bank, Lexmark, Orange and UBS achieve 90%+ test automation rates.

4 Testing Challenges Effective Test Planning

5 Automated E2E Testing with Tosca BI

6 Pre-screening Testing Test type Has No Empty Values Field Type Min Value Max Value Description Verifies that the column has no empty values. Checks if a field has Numeric values, depending on your selection from the drop-down menu. Returns the smallest value of the selected column. Returns the largest value of the selected column. Sum Checks if the sum of this field matches the specified value. You can use the relational operators equals =, smaller than < or greater than > to compare the current sum with the specified value. Value Range Min Length Max Length Exact Length Is Unique Row Count Checks if the values in this field match one of the specified values. Separate several values with a comma. Checks if the amount of characters in this field is greater than the specified value. Checks if the amount of characters in this field is smaller than the specified value. Checks if the amount of characters in this field is equal to the specified value. Checks if values in this field are unique. Checks if the current row count is equal to the specified value.

7 Vital Checks 1. Completeness tests: Enable count comparisons between source and target 2. Uniqueness tests: Check for the uniqueness constraint defined in the database 3. Referential integrity tests: Check that complete records have been copied and that technical and logical integrity is maintained If the database changes (e.g., a table is removed), Tricentis Tosca identifies the impacted test cases for the review.

8 Vital Checks Table test Test type Metadata Completeness Uniqueness Referential Integrity Description Compares predefined table definitions against the current table definition. Tests row counts on file or table level. Tests if there are primary key violations in target databases. Use this test if the target data source does not enforce constraints. Tests the primary and foreign key relationship in target databases.

9 Vital Checks File test Test type Has No Empty Values Field Type Min Value Max Value Description Verifies that the column has no empty values. Checks if a field has Numeric values, depending on your selection from the drop-down menu. Returns the smallest value of the selected column. Returns the largest value of the selected column. Sum Checks if the sum of this field matches the specified value. You can use the relational operators equals =, smaller than < or greater than > to compare the current sum with the specified value. Max Length Exact Length Is Unique Row Count Value Range Min Length Checks if the amount of characters in this field is smaller than the specified value. Checks if the amount of characters in this field is equal to the specified value. Checks if values in this field are unique. Checks if the current row count is equal to the specified value. Checks if the values in this field match one of the specified values. Separate several values with a comma. Checks if the amount of characters in this field is greater than the specified value.

10 Reconciliation Testing Reconciliation tests include a complete row by row comparison of two datasets in two separate systems. These datasets can be one of the following types: 1. database tables; 2. files, including files on a Hadoop system or on a Linux/Unix environment connected via SSH; 3. other sources - such as MS-Excel files - if you have installed an appropriate ODBC driver. The main goal of reconciliation testing is to confirm that the source data matches the target data. Tosca BI then helps to find mismatches where they are not expected. Possible mismatches are: 1. a row in the source dataset is not present in the target dataset; 2. a row in the target dataset is not present in the source dataset; 3. a source row matches a target row by RowKey, but not on all other columns.

11 Reconciliation Testing You want to match source table Left and target table Right: The row by row comparison delivers the following result:

12 Data Profiling BI Report Testing 1. Profiling tests validates data for logical consistency and correctness from a business perspective. 2. For example, one could automatically check that insurance contracts can only be cancelled if all outstanding invoices have been paid. Or, you could validate whether a certain business process completes within a specified period of time. 3. The profiling functionality can also be used to monitor how many data values of a certain type exist at any given point; it can alert you to out of range values as well as use results to create a trend profile over time. 1. Report testing verifies report creation and content from the end-user perspective. 2. Tests can also check access restrictions and report generation performance. 3. Report tests might involve a combination of UI and API tests, depending on how the reports are accessed. 4. For example, a test might open a Cognos report in a web browser, retrieve a value from a table in the report, and then compare it with a result retrieved from a database query.

13 Data Profiling Column Profiling Key and Join Testing Dependency Testing After basic checks like comparison of elementary, table related key performance indicators (KPIs such as number of records, sums/ means/ standard deviation of numeric fields) against expected values, column profiling provides the first cut on understanding the data in the data warehouse: Meaningful extract of the database tables attributes (columns) are examined Basic features to be checked are percentage population, uniqueness (distinction), value ranges and field lengths Key and join testing is a vital step in any DWH/BI testing project. Whilst the uniqueness of keys has been already covered in column profiling, joins between primary and foreign keys now need to be checked and invalid references need to be detected. Aligned with the arguments above, these checks can also be conducted via plain SQL statements. Tosca s predefined framework contains also a tool-set for join testing. Dependency testing focusses on the relationships of data, both within tables but also and even more important across wider ranges of the database by joining tables. For instance, if we believe that a combination of two field values [x, y] should always specifically match values in three other fields values [a, b, c], we are above all interested in any exceptions. Dependency testing can be used as a powerful early alert system for both dataquality and data processing issues.

14 Supported Technologies Supported Databases 1. All ODBC databases, e.g. Oracle, DB2, Teradata, MS SQL, Hive, Hbase; 2. Hadoop through WebHDFS. File Support 1. XML, JSON; 2. Excel; 3. Fixed & Comma Separated; Test Examples 1. Straight data move; 2. Report validation; 3. Data transformations. Sample Operations 1. File to File / DB; 2. DB to File / DB; 3. WebHDFS to DB/File; 4. DB/File to WebHDFS.

15 Key Takeaways 1. Data in its final state is the driving force behind organizational decision making; 2. Raw data is often changed and processed to reach a usable format for BI reports; Data integrity practices ensure that this DWH/BI information is attributable and accurate; 3. Data can easily become compromised if proper measures are not taken toverify it as it moves from each environment to become available to DWH/BI projects; 4. Errors with data integrity commonly arise through human errors, noncompliant operating procedures, data transfers, software defects, and compromised hardware. What expect from Tosca BI? 1. Automated data quality testing; 2. Automated testing of the entire DWH/BI processing; 3. Performance boost intest execution; 4. Business based test case definition with no hyper complex SQL; 5. Reduced test case maintenance effort; 6. Accelerated path (3 6 months) to achieve test sets with high coverage of business risk (> 90%). DATA WAREHOUSING PROJECTS CAN FAIL FOR MANY REASONS: 1. poor data architecture; 2. inconsistently defined data; 3. inability to relate data from different data sources; 4. missing and inaccurate data values; 5. inconsistent use ofdata fields; 6. unacceptable query performance and so forth. OTHER DWH / BI TESTING TOOLS: 1. DBFit: Open source database testing tool 2. icedq, QuerySurge, Zuzena: Test automation tools designed specifically for Data Warehousing and related projects; 3. Informatica Data Validation: Accelerate and automate Informatica ETL testing in both production environments and dev/ test; 4. Analytix Data Services, WhereScape, TimeXtender: DW automation tools that include test automation capabilities.

Test Automation: Agile Enablement for Business Intelligence Teams

Test Automation: Agile Enablement for Business Intelligence Teams Lynn Winterboer Agile Analytics Educator & Coach @AgileLynn www.winterboeragileanalytics.com Lynn Winterboer Colorado Native Guest Ranch