TESTONA. Next Level Classification Tree Method and Combinatorial Test Design PK

Transcription

1 TESTONA Next Level Classification Tree Method and Combinatorial Test Design PK

2 Overview Berner & Mattner A u t o m o t i v e Tr a n s p o r t a t i o n S y s t e m s I n d u s t r i a l E m b e d d e d S y s t e m s Signalling Vehicle Control Vehicle System Machinery Engines & Energy Space & Defence 2 2

3 Berner & Mattner Overview Founded: 1979 Employees: 430 Turnover 2012: 41,7 Mio. Sites: Offerings: Sectors: TOP clients: Munich, Stuttgart, Ingolstadt, Wolfsburg, Berlin, Cologne, Vienna Products and Engineering Automotive, Transportation, Industrial Embedded Systems DAIMLER, BMW, AUDI, Siemens, Ford, VW, Bombardier, EADS, Deutsche Bahn, MAN 3

4 Assystem Group Overview Founded: 1966 Employees: Turnover 2012: Sites: Offerings: Sectors: TOP clients: 855,6 Mio Australia, Belgium, Germany, France, Great Britain, India, Yemen, Canada, Qatar, Morocco, Nigeria, Austria, Portugal, Romania, Russia, Switzerland, Spain, USA, United Arab Emirates Engineering services and consulting Energy, Automotive, Rail Transportation, Aerospace & Defence, Technology & Product Engineering Areva, Alstom, EADS, EDF, General Electric, Peugeot-PSA, Renault, Rolls-Royce, Spirit Aerosystems, Total 4

5 Our Products TESTONA Test Design using the Classification Tree Method MERAN Specification of Systems with many Variants MESSINA Platform for virtual integration, model-based simulation, SiL and HiL testing MODENA Infotainment & body specification and test system 5

6 Test Process and Test Activities Specification e.g. Modul-, Integration-, Systemtest Tested (Sub-) System Test Planning Testmanagement, Testdokumentation Test Design Implementation Test Execution Test Evaluation Test Management / Test Documentation 6

7 Test Process and Test Activities Specification e.g. Modul-, Integration-, Systemtest Tested (Sub-) System Test Planning Testmanagement, Testdokumentation Test Design Implementation Test Execution Test Evaluation Test Management / Test Documentation 7

8 Classification Tree Method Black box Test Design Technique Systematic Process Graphical Representation of Test Problem Independent of Test Domain, Test Level and Test Object Tool Support: TESTONA Recommended for Certified Tester 8

9 System under test Example Input domain Privileges Operation Access Method

10 Example Input domain System under test Privileges Operation Access Method Minimum number of test cases = 5 five disjoint classes for the classification Access Method

11 Example Input domain System under test Privileges Operation Access Method Maximum number of test cases 2 * 3 * 5 =

12 Example Input domain System under test Privileges Operation Access Method Logical dependencies Browser NOT delete

13 Classification-Tree Editor TESTONA Syntax-oriented graphical editor to support the classification tree method Automatic verification of test cases against defined dependency rules Automatic test case generation with generation rules Tag approach for annotation of additional information and meta-data Generic and customized tool coupling Statistics 13

14 Prioritization How to identify test relevant aspects (and distinguish from irrelevant stuff)? How to distinguish between good and bad test cases? How to reduce size (=cost) of test suites? Without loosing test quality? 14

15 Example Prioritization 15

16 # Access Method Operation Privileges 1 Browser (with JavaScript) Edit Normal 2 Browser (with JavaScript) Edit Superuser 3 Browser (with JavaScript) Create Normal 4 Native Tool Create Superuser 5 Native Tool Edit Normal 6 Browser (with JavaScript) Delete Normal 7 Native Tool Delete Superuser 8 Browser (no JavaScript) Edit Normal 9 Browser (no JavaScript) Create Superuser 10 Database-Frontend Edit Normal 11 Database-Frontend Create Superuser 12 Browser (no JavaScript) Delete Superuser TESTONA Next Database-Frontend Level 16Classification Tree Method and Delete CombinatorialNormal Test Design

17 Coverage criteria For prioritized pairwise combination Each Used Coverage (EUC) Weight Coverage (WC) 17

18 Coverage criteria Each Used Coverage Weight Coverage Coverage Level Number of test cases 18

19 Optimizing test suites # Access Method Operation Priv. EUC WC 1 Browser (with JavaScript) Edit Normal % weight coverage with one test case 2 Browser (with JavaScript) Edit Superuser Browser (with JavaScript) Create Normal Native Tool Create Superuser Native Tool Edit Normal Browser (with JavaScript) Delete Normal Native Tool Delete Superuser Browser (no JavaScript) Edit Normal Browser (no JavaScript) Create Superuser Database-Frontend Edit Normal Database-Frontend Create Superuser Browser (no JavaScript) Delete Superuser Database-Frontend Delete Normal

20 Optimizing test suites # Access Method Operation Priv. EUC WC 1 Browser (with JavaScript) Edit Normal Browser (with JavaScript) Edit Superuser % weight coverage with only three test cases 3 Browser (with JavaScript) Create Normal Native Tool Create Superuser Native Tool Edit Normal Browser (with JavaScript) Delete Normal Native Tool Delete Superuser Browser (no JavaScript) Edit Normal Browser (no JavaScript) Create Superuser Database-Frontend Edit Normal Database-Frontend Create Superuser Browser (no JavaScript) Delete Superuser Database-Frontend Delete Normal

21 Optimizing test suites # Access Method Operation Priv. EUC WC 1 Browser (with JavaScript) Edit Normal Browser (with JavaScript) Edit Superuser % weight coverage with just seven test cases 3 Browser (with JavaScript) Create Normal Native Tool Create Superuser Native Tool Edit Normal Browser (with JavaScript) Delete Normal Native Tool Delete Superuser Browser (no JavaScript) Edit Normal Browser (no JavaScript) Create Superuser Database-Frontend Edit Normal Database-Frontend Create Superuser Browser (no JavaScript) Delete Superuser Database-Frontend Delete Normal

22 Solution: Weighted Test Cases Reducing number of test cases Based on preferred coverage Interactive preview 22

23 Requirements Traceability DOORS (or other RMTool) Requirement 1 Requirement 2 Requirement 3 Requirement 4 Requirement 5 Requirement 6 Test Case 1 Test Case 2 Test Case 3 Test Case 4 Test Case 5 Test Case 6 TESTONA Coverage of Requirements by one/many Test Cases or Tree Items Bonus: Automatic Tracing of Requirements Changes Linking with Requirements Linked Requirements Non-Linked Requirements 23

24 Requirements Traceability DOORS (or other RMTool) Requirement 1 Requirement 2 Requirement 3 Requirement 4 Requirement 5 Requirement 6 Test Case 1 Test Case 2 Test Case 3 Test Case 4 Test Case 5 Test Case 6 TESTONA Coverage of Requirements by one/many Test Cases or Tree Items Bonus: Automatic Tracing of Requirements Changes Linking with Requirements Linked Requirements Non-Linked Requirements 24

25 MindMap Import 25 TESTONA Test Design using the Classification Tree Method October 2014

26 Excel Import 26

27 Inspector - Checks both the classification tree and test case tree. - The output contains all relevant information about possible problems: - Missing Marks - Incomplete Leaf Node - Unused Leave Class - Missing Connection - Duplicate Test 27

28 Test Completeness Analysis Test Coverage - Analyze existing test suite, Check fulfillment of Coverage Levels (e.g. Minimal, Maximal, Pairwise, ) 28

29 Test result annotation Test result management Monitor test status, e.g. Failed N/A No Run Not Completed Passed 29

30 Test Evaluation Root Cause Analysis - Analyze individual test results to check for problematic system configurations 30

31 TESTONA Tool Integration Matlab Doors testidea Professional Mindmaps MESSINA MERAN 31 TESTONA XML files Command Line Java RMI Eclipse RCP

32 TESTONA Editions, Features, Prices Manual Creation of Classifications Trees Automated Generation of Test Cases Light Express Professional Enterprise X X X X X X X Excel Import X X X Support for DOORS, HP ALM (form. QC) Import Matlab, Autosar Add-On X Node Locked Floating / USB Dongle 3900 On demand Support, Maintenance Not available 400 p.a. X 800 p.a. (Add-Ons 200 p.a.) X On demand 32

33 TESTONA Peter M. Kruse Berner & Mattner Systemtechnik GmbH Gutenbergstr Berlijn Duitsland

34 OPTIMIZE YOUR DEVELOPMENT 34