Test Smarter (with Pair-wise Testing) Presented by: Bill Lewis CEO, Smartware Technologies, Inc. (469)

Transcription

1 Test Smarter (with Pair-wise Testing) Presented by: Bill Lewis CEO, Smartware Technologies, Inc. (469)

2 Agenda Challenges of application development Automated Testing What is orthogonal arrays? Power of pair-wise testing Transforming theory to practice Summary Q&A Copyright 2010 Smartware Technologies, Inc. 2

3 V Model Business Requirements User Requirements Functional & NF Requirements Architecture Project Success UA Testing System Testing Integration Testing Static Testing Verify Design Coding Unit Testing Dynamic Testing 50 % 50 % Validate Copyright 2010 Smartware Technologies, Inc. 3

4 Tractability Matrix A Trace Matrix can be used to track test cases during development and execution Identification of uniquely testable requirements Eliminate wishes, hopes, dreams, and other un-testable statements Eliminate duplicates Assign a unique ID for each requirement and design objective Determine its priority if possible Verification of implementation of all requirements Copyright 2010 Smartware Technologies, Inc. 4

5 Traceability Matrix Example Copyright 2010 Smartware Technologies, Inc. 5

6 Issues Facing Software Testing Organizations Today s software is getting more and more complex with thousands if not millions of different combinations that need to be tested. Humans can t think of all the test scenarios Takes too much time Too error prone Which scenarios do you test? The length of time to test modern software is increasing. Requirements documentation is often ambiguous or inadequate Testing process is poorly defined or nonexistent Copyright 2010 Smartware Technologies, Inc. 6

7 Simple Example: Configuration Testing Copyright 2010 Smartware Technologies, Inc. 7

8 Simple Example (Con t) Suppose you have an application that needs to be tested on several hardware and software configurations. You are told that the application must be able to run on a Windows NT, Windows 2000 and Windows XP machine. It also has to work on a system with a minimum of 128Meg and a max of 512Meg of RAM. Finally, the application must also run on a Pentium II, III, and IV processor and work with Oracle, SQL and Access Databases. Copyright 2010 Smartware Technologies, Inc. 8

9 Simple Example (Con t) Determine test cases for testing the an application on the following: Operating System RAM PC Processor Database Windows NT 128 Meg Pentium II Oracle Windows Meg Pentium III SQL Windows XP 512 Meg Pentium IV Access Copyright 2010 Smartware Technologies, Inc. 9

10 Combinations of Test Cases 3 x 3 x 3 x 3 = 81 Test Cases Copyright 2010 Smartware Technologies, Inc. 10

11 Why Automate? Over time, number of functional features changes/ increase and Number of people and amount of testing time for each new release Remains flat May even decline Test coverage steadily decreases Increases the risk of failure Potential business losses Source: Linda Hayes, Why Certify Copyright 2010 Smartware Technologies, Inc. 11

12 Today s Test Automation Variable Capture/ Replay Tools Latest generation of automated testing tools support variable test data Difference between static and variable capture/ replay With Static the inputs and outputs are fixed With variable inputs and outputs are variable Replacing the captured inputs and expected outputs with variables Variable capture/ replay reduces the risk of not performing regression testing on existing/ new features Copyright 2010 Smartware Technologies, Inc. 12

13 Test Automation Framework Source: Ross Collard Source: Ross Collard Copyright 2010 Smartware Technologies, Inc. 13

14 What s the Missing Link? Answer: scientifically generated test data

15 Simple Example Revisited Determine test cases for testing the an application on the following: Operating System RAM PC Processor Database Windows NT 128 Meg Pentium II Oracle Windows Meg Pentium III SQL Windows XP 512 Meg Pentium IV Access * Constraint: Windows XP cannot run on a Pentium II PC Processor. Copyright 2010 Smartware Technologies, Inc. 15

16 Optimal Test Cases (via Pair-Wise Analysis) Test Case # Operating System RAM PC Processor Database 1 Windows NT 128 Meg Pentium II Oracle 2 Windows NT 256 Meg Pentium III SQL 3 Windows NT 512 Meg Pentium IV Access 4 Windows Meg Pentium III Access 5 Windows Meg Pentium IV Oracle 6 Windows Meg Pentium II SQL 7 Windows XP 128 Meg Pentium IV SQL 8 Windows XP 512 Meg Pentium III Oracle Copyright 2010 Smartware Technologies, Inc. 16

17 Pairwise Definition Pairwise testing an approach to combinatorial testing that executes a pairwise test data set. Pairwise test data set - A set of test cases that covers all combinations of the selected test data values for every pair of a system s input variables. Pairwise testing is also known as 2-way testing; other approaches include 3-way, 4-way,, n-way. Copyright 2010 Smartware Technologies, Inc. 17

18 Secret Sauce (Orthogonal Arrays) Mathematical curiosity discovered by monks in the 1400s Promoted by Dr. Genichi Taguchi in the 1950s for use in statistical test design in manufacturing, agriculture, medicine, etc. Orthogonal Array: Two dimensional array of numbers in which any two columns in the array receive an even distribution of all pairwise combinations of values in the array. Orthogonal arrays are a standard construction used for statistical experiments. Copyright 2010 Smartware Technologies, Inc. 18

19 Example Orthogonal Arrays and Notation L 4 (2 3 ) L 18 ( ) L 9 (3 4 ) Copyright 2010 Smartware Technologies, Inc

20 Why is Pair-wise Testing So Efficient? Example: Insurance Application (GUI) Suppose a bug exists wherein the application produces a garbled record when the spouse's name is empty, but only when the "married" radio button is selected With pair-wise testing we ll find the bug! Dependents (checkbox) Marital Status (radio buttons) Spouse First Name (text field less or equal 20 characters) No Children Single Mary Dependent Children Married Greater than 20 characters Adult Children Divorced Empty Copyright 2010 Smartware Technologies, Inc. 20

21 Exhaustive Testing No Children Single Mary No Children Single Greater than 20 characters No Children Single Empty No Children Married Mary No Children Married Greater than 20 characters No Children Married Empty No Children Divorced Mary 3 x 3 x 3 = 27 tests No Children Divorced Greater than 20 characters No Children Div orced Empty Dependent Children Dependent Children Dependent Children Dependent Children Single Mary Single Greater than 20 characters Single Married Empty Mary Dependent Children Married Greater than 20 characters Copyright 2010 Smartware Technologies, Inc. 21

22 Pair-wise Results (Only 9 test cases are required) When business rules (functional coverage) are also applied to the results, the probability of detecting a defect is dramatically increased. Copyright 2010 Smartware Technologies, Inc. 22

23 Real Results of Pair-wise Testing Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects) Clarification All single-mode defect will be found if every option is tested at least once (unit elements) 2-way or Pair-wise is combinations of 2 items (parameters) that cause a defect 3-way or Tri-wise is combinations of 3 items (parameters) that cause a defect Pair-wise defect detection (pair-wise testing) finds most defects, ex. testing all pairs typically finds 75% of defects Source: Kuhn NASA Deep Space Mission - study showed that 88% of bugs discovered using pair-wise defect detection testing Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, U.S. Food and Drug Administration study showed 98% bugs discovered using pair-wise defect detection Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, The Combinatorial Design Approach to Automatic Test Generation Source: IEEE, Software 1996, Vol. 13, No. 5 Baseline application under study resulted in a reduction in test plan time from 1 month to less than 1 week In several experiments, pair-wise demonstrated good code coverage and defect detection ability Copyright 2010 Smartware Technologies, Inc. 23

24 Managing the Test Combination Explosion

25 Transforming Theory to Practice

26 How does Smart Test work? Copyright 2010 Smartware Technologies, Inc. 26

27 Demonstration Copyright 2010 Smartware Technologies, Inc. 27

28 Invoke Smart Test Generator

29 Smart Test Generated Test Data Copyright 2010 Smartware Technologies, Inc. 29

30 QTP Script Ready to Executed

31 Agenda Challenges of application development Evolution of software testing What is orthogonal arrays? Power of pair-wise testing Transforming theory to practice Summary Q&A Copyright 2010 Smartware Technologies, Inc. 31

32 Questions? Copyright 2010 Smartware Technologies, Inc. 32

33 Contact Information If you want to contact me Bill Lewis, CEO & President Smartware Technologies, Inc. (469) Copyright 2010 Smartware Technologies, Inc. 33