Błaej Pietrzak

Transcription

1 Błaej Pietrzak Fault model Result-oriented testing Test design approach Domain testing model Category-Partition test pattern Polymorphic message test pattern 1

2 Exhaustive testing is usually impossible (intractable)! for (int i = 0; i < n; i++) { if (a.get(i) == b.get(i)) x[i] = x[i] + 100; else x[i] = x[i] / 2; } n Path No Fault Sensitivity the ability to hide faults from test suite Coincidental Correctness the code produces correct results for some input data short scale(short j) { j = j - 1; //should be j = j + 1 j = j / 30000; return j; } Friedman M. A., Voas J.M.: Software Assessment: reliability, safety, testability. New York, John Wiley & Sons 1995 Only , , -1, 0, 29999, generate incorrect results. For 99,9908% of input values the code generates correct results. 2

3 Why are these properties tested with this technique Answer can be based on common sense, experience, assumption, analysis or experiment. Fault Model shows associations and components of the system being tested that have highest probability for faults to occur. Bug hazard is a circumstance that increase the chance of a bug. Conformance Directed Testing to prove conformance with requirements or specificati Tests are representative enough for essential features of the system being tested Every fault makes the system not conformant (Nonspecific fault model) Fault-Directed Testing To prove implementation faults It is based on observation that conformance can be proved for implementation containing bugs. Specific fault model 3

4 Black box testing White box/glass box/structural testing Result-oriented testing Also called behavioral testing, functional testing, data-driven or input/output-driven testing Unconcerned about the internal program structure Test planning- may start when the spec is circulated Acceptance/qualification testing check whether the program is stable enough to be tested Function test and system test check it against spec Beta testing get user feedback Release testing check all things that will go to the customer /manufacturer, e.g. test the ad copy and sales materials Final acceptance testing by the customer 4

5 Logic-driven: examine the internal program structure Part of coding stage Unit and low-level components are often tested by the structural strategies Scope-specific test design (unit, integration, system) and techniques (white/blackbox) are organized into a coherent whole. Design test cases using scope-appropriate responsibility (functionality) -based patterns. Develop efficient test suites that exercise many responsibilities and component interfaces with just a few test cases. If the components are not trusted, choose an integration cycle based on dependencies to control introduction of untrusted parts. Develop and execute the test suite according to this cycle. Determine responsibility coverage by analysis and implementation coverage by instrumentation. Stop testing when the modeled responsibilities and part interfaces have been covered. 5

6 How much code is exercised by tests Statement coverage Every statement is exercised Branch coverage Every branch has been taken in every possible direction (true and false) How can tests achieve full branch coverage void func(int count) { if ((count % 2) == 0) System.out.println("count is even."); for (; count < 5; count++) System.out.println("count " + count); } 6

7 The if statement would need count both even and odd for loop must evaluate at least once and false at least once Two tests: count =4, count=13 long multiply(int arg1, int arg2) { return arg1 + arg2; } arg1 = 0 and arg2 = 0 produce correct result, coverage 100%. Sometimes 100% coverage could not be achieved (i.e. dead code) 7

8 int claims;... if (claims == 0) {... } else if (claims == 1) {... } else if ((claims >= 2) && (claims <= 4)) {... } else if (claims >= 5) {... } Type- safe exclusion - conditions defined for nonbinary decision variable, which fulfills each one of them with only one value. Claims cannot be simultaneously 0 and 3. Some inputs are mutually exclusive Some inputs cannot be produced by the environment Implementation is structured so as to prevent evaluation Often cause of errors 8

9 Number of claims can be simultaneously 0, 1, 2, 3, 4, 5 or more if (iszeroclaims) {... } if (isoneclaim) {... } if (istwotofourclaims) {... } if (isfiveormoreclaims) {... } Reflects incomplete model Example: Algorithm does not define an action for Insured age = 300. It is unknown what will happen. 9

10 Many bugs only found at class scope Cannot fully exercise a class through its clients Fix bugs close to creation The longer you wait the more expensive it is Debugging during system test sucks Reduces schedule risk Improves productivity classes can be test in parallel Don t delegate testing to your clients To verify a class, do we need to test every object of this class Generally, any bug in a class will be present in all objects of that class. Object-specific bugs are possible if an object is clobbered by a runaway pointer or its class contains self-modifying code. These bugs can be revealed by cluster and subsystem scope testing. 10

11 1. Make a preliminary estimate of class under test Plan budget for testing 2. Design and code a test driver For non trivial classes begin with alpha-omega skeleton After the alpha-omega tests pass, add additional tests 3. Select a class scope test pattern 4. Select test design patterns for each method 5. Arrange method test cases According to sequence called for by the class scope pattern 6. Build the test package When all tests pass, evaluate coverage If coverage is insufficient, develop more tests Goal Demonstrate that class under test is ready to test Two approaches Small pop Alpha-Omega cycle 11

12 A Big Bang integration at class level Excersise all untrusted components simultaneously Effective when CUT is small and simple Servers of the class are stable Inherited features are stable Few intraclass dependencies exist Process Develop entire class Write a test driver using any appropriate test pattern Run the test suite Debug as needed 12

13 Alpha-Omega States: Alpha State: object before it is constructed Omega State: object after it has been deleted Alpha-Omega Cycle: Take an object from alpha to omega Send a message to every method at least once Shows that class is ready for more extensive testing No attempt is made to achieve any coverage Order of calling methods: Constructor method Accessor (get) method Boolean (predicate) method Modifier (set) method Iterator method Destructor method Order within steps: private, protected, public 13

14 Purpose: find test values Subset of all possible values Partition the input value set Find best test values Proposed Methods Equivalence Partitioning (classes of values causing same result) Boundary Value Analysis (special class domain boundary) Improved approach: domain testing model Based on fault model Select values based on value types and type domains Solution for complex types (objects) 14

15 15

16 On, In, Off and Out Points On point (pol. punkt brzegowy) lies on a boundary In point (pol. punkt wewntrzny) is within boundary and not on boundary Off point (pol. punkt pozabrzegowy) lies not on a boundary Out point is outside boundary Condition Off point rule On point Off point y >= 1.0 x <= 10 y <= 10.0 x > 0 Open, true y <= x!stack.isfull() Precision

17 Condition Off point rule On point Off point y >= x <= y <= x > 0 Open, true y <= x!stack.isfull() Precision Condition Off point rule On point Off point y >= x <= y <= x > 0 Open, true 0 1 y <= x!stack.isfull() Precision

18 Solve equation constraints must be fulfilled Check central points of independent variable first x > 0, x <= 10 y >= 1.0, y <= 10.0 y = x so x must be x >= 4 and x <= 10 central point is x = 7 on point is y = 7.0 for x = 7 off point is y = for x = 7 Condition Off point rule On point Off point y >= x <= y <= x > 0 Open, true 0 1 y <= x !Stack.isFull() Precision

19 What about complex types (classes) Some classes can be modeled the same as primitive types e.g. Money is a scalar type, its object has one abstract state (a valid amount) and many concrete primitive states (each amount within the legal range). Use state and abstract state invariant An abstract state invariant is a function that evaluates a state invariant expression and returns true or false e.g. isopen(), isclosed() Treat each state invariant as a domain boundary Abstract state on point Smallest possible change in any state variable change produces a state change Abstract state off point Valid state that is not boundary state and differs minimally Abstract state in point Any value combination that places an object of the class in a given abstract state, and is neither an on nor off point of that abstract state. 19

20 State Possible transitions In Point On Point Off Point Empty Loaded 0 () 0 1 Loaded Empty 1<x< Full 1<x< Full Loaded () MAXSTACK = Condition Off point rule On point Off point y >= x <= y <= x > 0 Open, true 0 1 y <= x !Stack.isFull() Precision

21 One on point and one off point for each domain boundary For each relational condition For each nonscalar type For each nonlinear boundaries Special case Equality condition One on point and two off points for each equality condition X == 10: on point is 10, off points are 9 and 11 State invariant One on point and at least one off point for each state invariant Condition for invariant is once true, once false Add expected results and in points for other values Each test case only has one off or on point Select in points for all other values in the test case Avoid to repeat in points (increase chance of finding bugs) Do not repeat identical tests for adjacent sub-domains Adjacent domains may share boundaries. Result Minimal test cases Input variables to excersise boundary conditions For any type of variable types Including abstract complex types (objects) 21

22 Intent Based on input/output analysis Context Appropriate for any method that implements one or more independent functions Constituent responsibilities should be identified and modeled as separate functions This approach is qualitative and may be worked by hand If the method selects one of many possible responses or has many constraints on parameter values, consider using Combinational Function Test patterns 22

23 Fault model Faults are related to value combinations of message parameters and instance variables These faults will result in missing or incorrect method output Strategy 1. Identify the testable functions of the MUT. 2. Identify all input and output parameters of each function. 3. Identify categories for each input parameter. 4. Partition each category into choices. 5. Identify constraints on choices. 6. Generate test cases by enumerating all choice combinations. 7. Develop expected results for each test case using an appropriate oracle 23

24 1. Identify the testable functions of the MUT Method may implement several functions Other functions may be side-effects of the primary function i.e. The current position of List object may be incremented as side-effect of returning the next element A function is considered independently testable if it meets three criteria: It can be invoked by setting message parameters to particular values and sending a message to the method its output can be observed by inspecting the values returned by the method, indirectly by using other class methods, or by using an invasive inspector its output can be differentiated from the output of another function or an incorrect function. 24

25 2. Identify all input and output parameters of each function At method scope, method interface arguments and abstract class state(s) determine the function s response. Class variables, globals, and system environmental variables should be modeled, if they can influence the output or maybe changed by the function call. 3. Identify categories for each input parameter Identify categories for each input parameter Categories do not overlap and must result in distinctly different output. Category is a subset of parameter values Determines a particular behavior or output Whose values are not included in any other category 25

26 4. Partition each category into choices A choice is a specific test value for a category Any reasonable criteria may be used to make choices. Choices for primitive types can be selected by suspicions, past bugs, or the Invariant Boundaries patterns. Should include both legal and illegal values 5. Identify constraints on choices Some choices may be mutually exclusive or inclusive. Practical reasons i.e too big complexity of generating a combination 26

27 6. Generate test cases by enumerating all choice combinations The test cases are identified by generating the cross-product of all the choices. Mutually inclusive or exclusive choices will result in some test cases being dropped from the cross-product set. Entry Criteria Small pop Exit Criteria Every combination is tested once Achieve branch coverage in the MUT The test suite should force each exception at least once 27

28 Consequences Identification of categories and choices is subjective - some bugs may not be revealed Many test cases for even moderately complex methods The size of a Category-Partition test suite is the product of the number of choices, minus the number of combinations excluded by constraints. The Invariant Boundaries pattern may be used to produced smaller test suite whose size is a sum of the number of choices plus one, rather than the product. A superclass Category-partition method test suite may be reused to test an overriding subclass method. Object getnextelement() throws NoPosition, EmptyList; Functions 28

29 Object getnextelement() throws NoPosition, EmptyList; Functions Return next element Object getnextelement() throws NoPosition, EmptyList; Functions Return next element Keep track of last position and wrap from last to first 29

30 Object getnextelement() throws NoPosition, EmptyList; Functions Return next element Keep track of last position and wrap from last to first Throw NoPosition and EmptyList exceptions if appropriate Inputs Outputs 30

31 Inputs Position of the last referenced element List state Outputs Element returned Incremented position pointer Categories and Choices Parameter Position of the last referenced element State of the list Category nth Element Special cases m elements Special cases Choices n=2 n=some x > 2, x < max n = max Undefined First Last, n < max m=some x > 2, x < max Empty Singleton Full (m = max) 31

32 Intent Design a test suite for a client of a polymorphic server that exercise all client bindings to the server Context When a method under test sends messages to overridden methods of a polymorphic server, These messages may be bound to one of several subclasses in the server class hierarchy. High confidence is not warranted for a client of a polymorphic server unless all the message bindings generated by the client are exercised Fault model Client fails to meet all preconditions for all possible bindings Unanticipated binding occurs possibly because of an incorrect construction of a pointer (i.e. Incorrect indexing into an array) Superclass is changed subclasses are rendered inconsistent because of the change 32

33 Account listtransactions() : void debit(value : double) : void account.debit(2.0); SavingsAccount listtransactions() : void debit(value : double) : void Precondition: debit > 500 GICAccount listtransactions() : void debit(value : double) : void Customer.report() { Account[] list = new Account(MAX); int index = savingsindex(); list[index] = getsavingsaccount();... index = gicindex(); list[index] = getgicaccount();... System.out.println("Savings"); index = getgicaccount(); // wrong function! list[index].listtransactions(); } 33

34 Account listtransactions() : void debit(value : double) : void Modify debit so that it takes a negative number as parameter SavingsAccount listtransactions() : void debit(value : double) : void account.debit(-2.0); GICAccount listtransactions() : void debit(value : double) : void Strategy 1. Determine the number of bindings for each message sent to a polymorphic server object 2. Test for all possible bindings 3. Test for run-time binding error 34

35 Entry Criteria Small pop. Minimal feasibility is assured by branch coverage within method Server class should be stable Exit Criteria Each binding with polymorphic server should be tried at least once (achieve branch coverage of extended message flow graph) Consequences A test suites that covers the extended message flow graph will reveal many bugs in the client s usage of server. Does not focus on other kinds of bugs and should be used with an appropriate responsibility-based patterns. Requires analysis of the server class hierarchy to determine all possible bindings. If the server class source or documentation is not available, this task cannot be done. 35

36 Robert V. Binder: Testowanie systemów obiektowych. Modele wzorce i narzdzia. WNT 2003 Code coverage tools for Java: Clover (commercial) JCoverage (commercial, but also GPL license) GroboUtils (MIT license) Hansel (BSD license) Thank You for your attention What is your general impression (1-6) Was it too slow or too fast What important did you learn during the lecture What to improve and how 36