BI/DWH Test specifics

Transcription

1 BI/DWH Test specifics 26/05/2016 Page

2 me => TestMoto: inadequate test scope definition? no problem problem cold be only bad test strategy more than 16 years in IT more than 10 years experience in testing area in various roles: Test Manager, Test Coordinator, Test Analyst, Quality assurance engineer ISTQB certifications: foundation+agile+advanced TM experienced in test and analysis trainings, user consultancy, application support, quality assurance, process improvement, analysis, business analysis and test analysis co-founder of ProTest public test community (e.g. 26/05/2016

3 SELECT TestSpecifics FROM dwh_test_approach What is DWH? - data, layers, transformation steps - huge volume, long processing - DWH is not private world Test approach - data is highest priority - no data, but data error is our fast food - SQL like team communication language - development patterns - strange, but not only data are tested DWH test specifics - local point of view - group point of view - how to test business use cases Test Environment and Test Data Management - not so easy - not so clear - but it s working if you know how to do it

4 Quote1 The price of light is less than the cost of darkness. ~ Arthur C. Nielsen ~

5 What is DWH? Data warehouse (DWH): system used for reporting and data analysis, and is considered as a core component of Business Intelligence central repositories of integrated data from one or more disparate sources store current and historical data used for creating analytical reports for knowledge workers data uploaded from the operational systems data may pass through an operational data store for additional operations before it is used in the DWH source: Wikipedia

6 DWH - data, layers, transformation steps T1 (Source Landing) data transmission T2 (Landing BASE): end of day processing full snapshot in Base cross System unification T3 (Base Core): transformation with Business Logic verification that all sources has to be loaded surrogation (primary+foreing key) historization

7 DWH data layers/zones Zone Description Supported Processes Update Frequency History Management 1) Landing Source delivery layer Data delivery Daily 1 Day Base Source data vault Data historization, preparation, Data unification Daily Meta Data Driven Core Single version of truth for analytical and reporting purposes Data processing stage for all types of data and information transformations based on all types of mappings Data harmonization Reporting and analytics Daily Meta Data Driven Minimum 7 Years

8 DWH - huge volume, long processing 1. DWH contains a huge amount of data: - various business entities/areas (deal customer account provision collateral) - complex data relations (bridge tables) - saved for long time (many years) with data historization (valid from-to, time-stamps) 2. data processing ensure: - data transformation from sources to targets usable for next business analysis needs - transformation is going from one layer/zone to another - transformation is specified by business and technical mapping

9 DWH is not private world

10 Quote Test approach If you can t explain it simply, you don t understand it well enough. ~ Albert Einstein ~

11 (Tyres) Test approach Test Schedule Data availability Data sources Release scope Test Types Data processing Data changes Data features goal: improved data quality and availability Data flows

12 Data = highest priority 1. all expected data have to loaded 2. no duplicities are loaded 3. data are aggregated appropriately 4. results are based on business rules 5. data transformed within data flows: - accurately - completely - correctly - on time - integrated

13 SQL like team communication language SELECT COUNT(*) as RECORD_CNT_BASE_CUST FROM L_BASE.LC_CUST WHERE s_num <> 0 AND cob_dt = TO_DATE(' ', 'DD.MM.YYYY') Note: business testers/users in specifics roles are using SQL too (data stewards, data specialists, data analysts, )

14 Test object + mapping specification + test code Test Object: Target Table: RT_ACC Target Attribute: ACC123 Code: Mapping specification: for all (take IFRS) Everything is used upon the Group chart of accounts (DOM_GCOA) 1. The digit 1 must be 1 (1 = only assets) 2. Check position 5 + 6: a) 31, 32 or 33 is HFT (Held for Trading) Fair value b) 41, 71, 72, 73 is FV (Fair value) c) 51 is AFS (Available for Sale) Fair value d) 52 is AFS at-cost CASE WHEN substr(dacifrs.dom_gcoa,1,1) = '1' THEN e) 11 is LAR (Loans and receivables) at cost f) 61 is HTM (Held to Maturity) at cost CASE WHEN substr(dacifrs.dom_gcoa,5,2) in ('31', '32', '33', '41', '71', '72', '73','51') THEN 0 WHEN substr(dacifrs.dom_gcoa,5,2) in ('52', '11', '61') THEN 1 ELSE 2 END WHEN anc.acc000 = 11 THEN 1 ELSE 0 END ACC123

15 Test Steps + Test Results Test Steps: Static Testing: Code-Review of implementation (PL / SQL package) Plausibility Checks on resulting records after data processing SELECT ACC123, COUNT(*) AS NUM_RECORDS, CASE WHEN ACC123 IN (0, 1, 2) THEN 'OK' ELSE 'NOK' END AS BASIC_MAP_CHECK_RESULT FROM LDDADMIN.AUR_ACC_AC GROUP BY ACC123 ORDER BY ACC123 Results in the database: 31 records with ACC123 IS NULL have been subject of further investigations and discussions

16 Test Steps + Test Results

17 development test development analysis Question: What is goal of mapping test? Requirement specification mapping: TargetField = SourceField + 1 Mapping specification Implementation Implementation SELECT SELECT SQL script SQL script SQL result compare SQL result SQL result What is the goal of such test? : 1. transformed data (SQL result) 2. mapping specification 3. mapping implementation

18 Test development patterns (reusable SQL patterns) - repeated checks/sql statements allow to define SQL patterns - SQL patterns = test development unification - could be used as basis for test automation

19 Strange, but not only data are tested - primary focus => test data - BI/DWH applications have GUI too => so we need standard tests (functionality, usability, performance,...) - DWH could be surrounded by other applications => data corrections, manual data uploads, specific calculations and recalculations

20 Quote BI/DWH test specifics Not everything that can be counted counts, and not everything that counts can be counted. ~ Albert Einstein ~

21 BI/DWH Test approach 1. standard test activities have to be done: - test planning - test scope definition - traceability setup - roles & responsibilities - test scheduling but BI/DWH test specifics has to be considered

22 BI/DWH test specifics Planning & Organization: - many teams, complex environment - dependency on data processing planning Test environments, test data: - many data sources and localities - long data processing - data features & specific test types Activities: - essential technical (SQL) data check - business data confirmation during and after data processing - data quality as the ongoing activity

23 Test plan live structure example

24 BI/DWH Test approach Test Schedule Data availability Data sources Release scope Test Types Data processing Data changes Data features goal: improved data quality and availability Data flows

25 Data features Data Quality Logging and Audit Load Type components should check data quality issues components have to provide log and audit trail info components should be able to handle full loads and/or delta Loading Dependency loading dependency between source deliveries e.g. corrections cannot be loaded if original source file is missing Transformations Generation Metadata Restartability components with business transformation (framework) should be provided components should be based on a common metadata set all components access and utilize the same common metadata set components are fully restartable

26 Test types I. nonbi world: - functionality - usability - reliability - performance - supportability BI world: - Accuracy - Timeliness - Consistency - Integrity - Conformity - Completeness - Correctness

27 Test types II. Source data: 1.c. Consistency relations: 1.d. Integrity data:1.a. Accuracy timing: 1.b. Timeliness format:1.e. Conformity data:1.f. Completness data 1.g. Correctness other/derived technical test types: Plausibility Test Interface Operability Test Interface Code Review Checks on Landing-Zone Interface Regression Tests for Fundamental Source Changes Data Type Changes Data Quality Log Checks...

28 DWH test specifics - local vs group point of view Local specifics: - local data sources only Group specifics: - more data sources (multi-entity) - interactivity between local teams is more flexible - interactivity between local teams is less flexible - data processing is shorter - data processing is longer (from a few days to weeks) - data reprocessing is usually possible without limitations - data reprocessing is usually possible with limitations (e.g. different interface versions, )

29 How to test business use cases Use case testing could be based on: 1. final outputs (e.g reports, DM smart cubes): advantage: full test of business requirements disadvantage: testable only in the end of data processing (short time for test execution, data reprocessing if huge time-consumer, ) 2. ongoing outputs from data processing (e.g. core layer tables): advantage: running test before finishing of data processing, defects are better analyseable on data level disadvantages: business testers have to have technical knowledge (SQL, DB structure, ), tested outputs are sometimes not complete and not presented at the same structure/form like final outputs

30 QuoteTestEnvironment If you can t describe what you are doing as a process, you don t know what you are doing. ~ Dr. W. Edwards Deming ~

31 Test Data Management 1. based on: - data request specification (more projects) 2. not all data are processable (volume, time dependency,...) 3. test data types: - new version - data structure or content is changed - prod_copy if fresh data are needed 4. data testability depends on data processing status

32 Test Environment Management 1. based on: - data request specification - list of jobs 2. data processing is part of test environment preparation 3. test environment is shared between more projects 4. test manager have to monitor current testability

33 Data processing general example Data processing steps source_1 source_2 source_3 source_4 source_5 source_6 source_x Step1 source_3: 1. day Step2 Step3 Step4 Step5 source_3: 2 day Step6 Step7 Step8 Step9 Step10 source_3: 3 day Step11 Step12 Data source_3 testability: 1. day - testable for step1 and step2 2. day - testable to step 6 3. day - testable to step 11

34 Data processing real anonymized example

35 pictures

36 General Recap DWH: data, layers, transformation steps, huge volume, long processing SQL is essential for technical checks Business testers use SQL too Test approach = data vs data processing vs data features + test types BI/DWH test specifics Planning & Organization: - many teams, complex environment - dependency on data processing planning Test environments, test data: - many data sources and localities - long data processing - data features & specific test types Activities: - essential technical (SQL) data check - business data confirmation during and after data processing - data quality as the ongoing activity

37 Q & A

38 Many thanks for your attention. 26/05/2016

39 ? car test or wall test?

40 real gap :o)

41 nice spring leafs or bug?

42 Vocabulary Data warehouse (DWH): system used for reporting and data analysis, and is considered as a core component of Business Intelligence central repositories of integrated data from one or more disparate sources store current and historical data used for creating analytical reports for knowledge workers data uploaded from the operational systems data may pass through an operational data store for additional operations before it is used in the DWH Business intelligence (BI): "a set of techniques and tools for the acquisition and transformation of raw data into meaningful and useful information for business analysis purposes BI technologies are capable of handling large amounts of unstructured data to help identify, develop and otherwise create new strategic business opportunities allow easy interpretation of these large volumes of data provide historical, current and predictive views of business operations functions of BI technologies are reporting, online analytical processing, analytics, data mining, process mining, complex event processing, business performance management, benchmarking, text mining, predictive analytics and prescriptive analytics. source: Wikipedia

43 Vocabulary Extract, Transform and Load (ETL): process in database usage and especially in data warehousing that: Extracts data from homogeneous or heterogeneous data sources Transforms the data for storing it in the proper format or structure for the purposes of querying and analysis Loads it into the final target (database, more specifically, operational data store, data mart, or data warehouse) Data mart: the access layer of the data warehouse... is a subset of the data warehouse that is usually oriented to a specific business line or team where conformed dimensions are used The reasons why because the information in the database is not organized Also, complicated queries While transactional databases are designed to be updated, data warehouses or marts are read only. source: Wikipedia