Software Testing part II (white box) Lecturer: Giuseppe Santucci

Software Testing part II (white box) Lecturer: Giuseppe Santucci

4. White box testing White-box (or Glass-box) testing: general characteristics Statement coverage Decision coverage Condition coverage Decision and condition coverage Coverage of a linearly independent set of paths 2

Also called White-box testing: general characteristics Glass-box testing Structural testing (compared to functional) Exploits the control structure, that means the control or flow graph of programs in order to design test cases 3

White-box testing: general characteristics cont.d Allows to test parts of the program With black-box is not possible Allows to cover systematically every part of the program Often errors are found in special cases 4

Types of white-box tests

Statement coverage It is the easiest criterion for a reasonable whitebox test Necessary but not sufficient Goal To identify a set of test cases sufficient to exercise all statements at least once 6

Statement coverage cont.d Java example: A /*A*/ float x = InOut.readFloat(), y = InOut.readFloat(); /*B*/ if (x!= 0) /*C*/ y = y+10; /*D*/ y = y/x; /*E*/ System.out.println(x + ' ' + y); x == 0 B D C x!= 0 The test case {x!= 0; y = any} Covers the statements Control flow graph for the example E Do you see any problem? 7

Statement coverage cont.d Possible problem Even in a case of a program without iterations, the execution of every statement does not guarantee that all possible paths are exercised Possible negative consequence: In previous java example program the error of division by 0 is not identified (statement D) 8

Decision coverage Properly includes statement coverage Vice versa does not hold Focuses on all possible decisions in a program Decisions are in statements if, switch, while, for, do Goal To identify a set of test cases sufficient for guaranteeing that each decision will have value true at least once and value false at least once 9

Java example: Decision coverage cont.d /*A*/ float x = InOut.readFloat(), y = InOut.readFloat(); /*B*/ if (x!= 0) /*C*/ y += 10; /*D*/ y = y/x; /*E*/ System.out.println(x + ' ' + y); A B x!= 0 Test cases 1. {x=20; y=30} 2. {x=0; y=30} x == 0 D C Cover decisions hence statements The second case identifies the error of division by zero in statement D E 10

Decision coverage cont.d Possible problem If decisions are composed of several conditions (AND, OR), decision coverage can be not sufficient Java example: /*A*/ float z = InOut.readFloat(), w = InOut.readFloat(); t = InOut.readFloat(); /*B*/ if (z == 0 w > 0) /*C*/ w = w/z; /*D*/ else w = w + 2/t; /*E*/ System.out.println(z+' '+w+' '+t); What is the problem here? 11

Decision coverage cont.d Test cases: {t = 0; z= 5 ; w= 5} decision TRUE {t = 0; z= 5 ; w= -5} decision FALSE Cover decisions The risk of division by zero in D is identified (thanks to t = 0 that is not due to decision coverage strategy!!!! ) The risk of division by zero in C is not identified (z!=0) We need a criterion that considers the decision s conditions (i.e., the atomic Boolean expression components) (z == 0 w > 0) 12

Condition coverage Does not properly include decision coverage It is not properly included by decision coverage Focus on all possible conditions in a program Conditions are the components of Boolean expressions Goal: To identify a set of test cases sufficient for guaranteeing that every condition (atomic Boolean expression) included in the program s decisions have value true at least once and value false at least once 13

Condition coverage cont.d Java example: /*A*/ float z = InOut.readFloat(), w = InOut.readFloat(); t = InOut.readFloat(); /*B*/ if (z == 0 w > 0) /*C*/ w = w/z; /*D*/ else w = w + 2/t; /*E*/ System.out.println(z+' '+w+' '+t); Test cases {t=0;z=0;w= 5} 1 st condition=t, 2 nd condition=f {t=0;z=5;w= 5} 1 st condition=f, 2 nd condition=t Cover conditions What problem do such case tests make emerge? 14

Condition coverage Test cases {t=0;z=0;w=-5} 1 st condition=t, 2 nd condition=f {t=0;z=5;w= 5} 1 st condition=f, 2 nd condition=t Covers conditions The risk of division by zero in C is identified The risk of division by zero in D is not identified as D is never exercised Decision is always true 15

Decision and condition coverage Properly includes Decision coverage And, as a consequence, statements coverage Condition coverage Goal To identify a set of test cases sufficient for guaranteeing that Each decision is true at least once and false at least once All conditions composing decisions is true at least once and false at least once 16

Decision and condition coverage /*A*/ float z = InOut.readFloat(), w = InOut.readFloat(); t = InOut.readFloat(); /*B*/ if (z == 0 w > 0) /*C*/ w = w/z; /*D*/ else w = w + 2/t; /*E*/ System.out.println(z+' '+w+' '+t); cont.d Test cases {t=0;z=0;w=5} 1 st condition=t, 2 nd condition=t, decision=t {t=0;z=5;w= 5} 1 st condition=f, 2 nd condition=f, decision=f Cover decisions and conditions and identify both division by zero in statements C and D (thanks to t = 0 that is not due to decision and condition coverage strategy!!!! ) 17

Summary of test cases so far Test case t z w cond. 1 cond. 2 decis. Finds error in C? Finds error in D? C1 0 0 5 T T T yes no C2 0 5 5 F T T no no C3 0 0-5 T F T yes no C4 0 5-5 F F F no yes 18

Decision and condition coverage Coverage criterion Associated test cases Finds error in C? Finds error in D? Decision C2 + C4 no yes Condition C2 + C3 yes no Decision and condition C1 + C4, or C2 + C3 + C4 yes yes Which one is the best? 19

Methodological issue There are different combination of test cases that guarantee decision and condition coverage but it is not always trivial to find one We need a simple and effective method for identifying test cases that cover both decisions and conditions 20

Control flow graph s paths coverage Based on Criteria of decision and condition coverage Program s control flow graph Goal To provide a simple method for guaranteeing decision and condition coverage 21

Control flow graph Predicate nodes node D A B C edges while (A) { if (B) { C } else { D } E } F; E F region Example: V(G) = 7 6 + 2 = 3 V(G) = 3 Cyclomatic number V(G) = E N + 2 (E = number of edges, N = number of nodes) V(G) = R (R = number of regions) 22

Control flow graph until Sequence of statements Simple if while case What about the for statement? It must be simulated through a while 23

Control flow graph without conditions /*A*/ float z = InOut.readFloat(), w = InOut.readFloat(); t = InOut.readFloat(); /*B*/ if (z == 0 w > 0) /*C*/ w = w/z; /*D*/ else w = w + 2/t; /*E*/ System.out.println(z+' '+w+' '+t); False D E A B True C The control flow graph should be more detailed 24

Control flow graph with complex if (a b) x; else y; decisions if (a && b) x; else y;...... a==f a a==t a==t a a==f b==f b b==t b==t b b==f y x x y...... 25

Control flow graph for the program /*A*/ float z = InOut.readFloat(), w = InOut.readFloat(); t = InOut.readFloat(); /*B*/ if (z == 0 w > 0) /*C*/ w = w/z; /*D*/ else w = w + 2/t; /*E*/ System.out.println(z+' '+w+' '+t); False D A B True C A!= 0 z <= 0 w > 0 D C == 0 E Control flow graph Without conditions E Control flow graph with conditions 26

Control flow graph, conditions and decisions Boolean values ( true and false ) associated with conditions and decisions have corresponding edges To design a set of test cases such that all edges of the control flow graph are traversed implies condition and decision coverage 27

Paths of the control flow graph The set of test cases corresponds to a set of paths such that every edge is traversed at least once The number of paths sufficient for covering all arcs is always equal or less than the cyclomatic complexity 28

For the previous example /*A*/ float z = InOut.readFloat(), w = InOut.readFloat(); t = InOut.readFloat(); /*B*/ if (z == 0 w > 0) /*C*/ w = w/z; /*D*/ else w = w + 2/t; /*E*/ System.out.println(z+' '+w+' '+t); A!= 0 z <= 0 w > 0 D C == 0 Three test cases C2. {t=0; z=5; w= 5} C3. {t=0; z=0; w= 5} C4. {t=0; z=5; w= 5} Cover all edges, hence decisions and conditions E V=7 6+2=3 The three paths 1. A z w C E (C2) 2. A z C E (C3) 3. A z w D E (C4) 29

How to choose paths Pragmatic rule Experiments show that the the number of errors increases with increasing of the cyclomatic complexity Choose a number of paths equal to the cyclomatic complexity You have a correlation between number of test cases and code complexity Which paths? 30

Linearly independent paths: Basis Path Testing A good criterion for choosing paths is based on the concept of linear independency A maximal set of linearly independent paths is called a basis It is NOT unique A basis contains a number of paths equal to the cyclomatic complexity Intuitively, every path in the control graph can be obtained as a linear combination of paths of a basis By choosing a basis, a certain extent of reliability is guaranteed with respect of combinations of errors that hide each other 31

Example of paths 7 A 1 B 2 3 D C 4 5 E 6 while (A) { if (B) { C } else { D } E } F; Paths. A F. A B D E A F. A B C E A F F Arcs/paths 1 2 3 4 5 6 7 0 0 0 0 0 0 1 1 1 0 1 0 1 1 1 0 1 0 1 1 1 The matrix rank is 3 (the maximum) { } is a basis Note that A B D E A B C E A F covers all edges BUT 32

Remarks Every path corresponds to a binary row vector (0/1) Vice versa does not hold A matrix representing paths has maximum 2 E distinct rows (E = n of edges) McCabe has proved that for each graph G: 1. The matrix rank cannot be greater than V(G) (cyclomatic complexity) 2. There exists always a matrix made of paths with rank equal to V(G) A basis is any set of paths (rows) with maximum rank Note: MacCabe s result is purely topological: /*B*/ if(5>7) /*C*/ i=9; It is not possible to traverse Edge B-C In other cases is necessary to re-iterate the cycles in order to traverse all edges 7 A 1 B 2 3 D C 4 5 E F 6 33

Edges traversable only by iterations /*A*/ int z = ; int t=0; /*B*/ while (t<100) /*C*/ if (t > 20) /*D*/ t=t+90; /*E*/ else t= t+z*z; /*F*/ System.out.println(t); A B F C D Require at least one iteration in order to be traversed {z=10} A B C E K B F {z= 5} A B C E K B C D K B F t>20 D t<100 C E B t<=20 t>=100 So A B F A B C D K B F are not traversable K Fake node F 34

Additional considerations Paths coverage method does not consider explicitly loops iterations In some cases it is necessary to perform iterations in order to traverse all edges It might be interesting to consider some iterations in any case but the number of test cases increases exponentially It is important to decide the type of iterations 35

Pragmatic choices for loops To limit the number of iterations to n It is called loop coverage To execute only certain loops To limit the number of paths to be traversed based on weighted edges and function to be maximized Probability of execution Resources occupancy To limit the number of paths by identifying the paths that define and use program variables (Data Flow testing) Definition of a variable value Use of such value in a test 36

Data Flow Testing B... A int x,y,a,b; B scanf( %d %d,&x, &y); C a=x; D b=y E while (a!=b) F if(a>b) G a=a b; H else b=b a; I printf( %d,a);... a!=b F a>b G C D E a<=b H a==b I For x and y it is not needed to execute the loop (loop coverage = 0) For the definition of a and b the loop has to be iterated once (loop coverage = 1) For the definition and usage of a and b the loop has to be iterated twice (loop coverage = 2) 37

5. Testing strategies Testing and development process Verification vs. Validation The V-model Test levels Test types Maintenance testing 38

Testing and development process 1 The adopted life cycle has a big impact on the testing that is carried out: testing does not exist in isolation Test activities are highly related to software development activities The way testing is organized must fit the development life cycle or it will fail to deliver its benefit If a fully documented software development life cycle is required, the testing must be fully documented 39

Verification vs. Validation In every development life cycle, a part of testing is focused on verification testing and a part is focused on validation testing Verification: Evaluation of a work product, component or system to determine whether it meets the requirements set Answers the question: Is the deliverable built according to the specification? Validation: Evaluation of a work product, component or system to determine whether it meets the user needs and requirements Answers the question: Does the deliverable fit for purpose, e.g., does it provide a solution to the problem? 40

Good practices for testing Good testing practices apply to any life cycle model For every development activity there is a corresponding testing activity Each test level has test objectives specific to that level The analysis and design of tests for a given test level should begin during the corresponding development activity Testers should be involved in reviewing documents as soon as drafts are available in the development cycle 41

V-model The V-model was developed to address some of the problems experienced using the traditional waterfall approach Defects were being found too late in the life cycle, as testing was not started until the end of the project The V-model provides guidance to begin testing as early as possible in the life cycle 42

V-model Activities performed with the user Design Review Planning Testing Requirements Analysis Requirements Review Acceptance test Planning Acceptance Testing System Design Design Review Functional test Planning System Testing Architecture Design Architecture Review Integration test Planning Integration Testing Module Design Module Review Unit test Planning Unit Testing Internal activities Coding 03Test.43

Four test levels 1. Unit testing Searches for defects in and verifies the functioning of software components e.g. modules, programs, objects, classes etc., that are separately testable 2. Integration testing Tests interfaces between components, interactions to different parts of a system such as an operating system, file system and hardware or interfaces between systems 3. System testing Concerned with the behavior of the whole system/product as defined by the scope of a development project or product The main focus of system testing is verification against specified requirements 4. Acceptance testing validation testing with respect to user needs, requirements, and business processes conducted to determine whether or not to accept the system 44

1: Unit testing Also known as component, module, and program testing Searches for defects in, and verifies the functioning of software e.g. modules, programs, objects, classes, etc. that are separately testable Can be done in isolation from the rest of the system Stubs and drivers are used to replace the missing software and simulate the interface between the software components in a simple manner 45

Stub and Driver A stub is called from the software component to be tested A driver calls a component to be tested A A Driver B Stub B 46

1: Unit testing (cont.) Typically occurs with access to the code being tested Can be supported by the development environment such as a unit test framework or debugging tool See for example JUnit (http://www.junit.org/) Usually involves the programmer who wrote the code Defects are fixed as soon as they are found, without formally recording the incidents found 47

Junit (annotation based) 48

Junit assertions 49

Junit test case example @Test public void testclassx_methody(){ ClassX a = new ClassX(); assertequals(4, a.methody(3, 1)); } 50

Test-driven development To prepare and automate test cases before coding Highly iterative Based on cycles of developing test cases, then building and integrating small pieces of code, and executing the component tests until they pass Used in Extreme Programming 51

2: Integration testing Focus: interfaces among components Interactions with different parts of a system an operating system, file system and hardware Interfaces between systems Within the NASA Apollo project, interface errors were the 75% of all errors! 52

Different levels of integration testing Component integration testing Testing the interactions among software components and is done after unit testing System integration testing Tests the interactions between different systems and may be done after system testing In this case, the developing organization may control only one side of the interface, so changes may be destabilizing Worst case: a series of systems that run on different platforms The greater the scope of integration, the more difficult it becomes to isolate failures to a specific interface 53

Two extreme approaches Big-bang integration testing All components or systems are integrated simultaneously After that everything is tested as a whole Advantage: everything is finished before integration testing starts Disadvantage: time-consuming and difficult to trace the cause of failures with this late integration Incremental integration testing All programs are integrated one by one A test is carried out after each step Advantage: defects are found early in a smaller assembly when it is relatively easy to detect the cause Disadvantage: it can be time-consuming since stubs and drivers have to be developed and used in the test 54

Incremental testing approaches Top-down Testing takes place from top to bottom, following the control flow or architectural structure e.g., starting from the GUI or the main menu Components or systems are substituted by stubs Bottom-up Testing takes place from the bottom of the control flow upwards Components or systems are substituted by drivers Functional incremental Integration and testing takes place on the basis of the functions or functionality, as documented in the functional specification 55

Example of top-down approach A B C D Stub E Stub F G Stub H 56

Example of bottom-up approach Driver A Driver B C Driver D E F G H 57

3: System testing Is concerned with the behavior of the whole system/product It may include tests based on Risks and/or requirements specification Business processes Use cases, or other high level descriptions of system behavior Interactions with the operating system and system resources Is most often the final test on behalf of development Should investigate both functional and non-functional requirements of the system System testing requires a controlled test environment with regard to control of the software versions (configuration management) The test environment should correspond to the final target or production environment as much as possible 58

4: Acceptance testing It comes after the system testing Responsibility of the user or customer Answer questions: Can the system be released? What, if any, are the outstanding (business) risks? Has development met its obligations? Executed in a test environment that is representative of the production environment Focused on a validation type of testing It is not necessarily the final level of testing E.g. a large-scale system integration test may come after the acceptance of a system 59

Test types: target of testing Test types are a means of clearly defining the objective of a certain test level for a program or project A test type is focused on a particular test objective For example Testing of a function to be performed by the component or system A non- functional quality characteristic, such as reliability or usability The structure or architecture of the component or system Testing changes, confirming that defects have been fixed (confirmation testing, or re-testing) and looking for unintended changes (regression testing) Depending on its objectives, testing will be organized differently 60

ISO 25010 quality characteristics ISO 25010 (ex 9126) defines eight quality characteristics and the subdivision of each quality characteristic into a number of sub-characteristics 1. Portability 2. Maintainability 3. Security 4. Reliability 5. Functional suitability 6. Performance efficiency 7. Compatibility 8. Usability ISO 25010 61

Functional testing The function of a system (or component) is what it does This is typically described in a requirements specification, a functional specification, or in use cases It is difficult to test against undocumented and implicit requirements some functions that are assumed to be provided Functional tests are based on functions described in documents or understood by the testers Functional tests may be performed at all test levels e.g. test for components may be based on a component specification Functional testing considers the specified behavior and is often also referred to as black-box testing Even if black-box testing includes also non-functional testing 62

Functionality testing based on ISO 25010 Functions are those that satisfy stated or implied needs The objective is to guarantee the existence of a set of functions and their specified properties Functional suitability Completeness Correctness Appropriateness 63

Functionality testing based on ISO 25010 (ex 9126) functional suitability: degree to which a product or system provides functions that meet stated and implied needs when used under specified conditions functional completeness: degree to which the set of functions covers all the specified tasks and user objectives functional correctness: degree to which a product or system provides the correct results with the needed degree of precision functional appropriateness: degree to which the functions facilitate the accomplishment of specified tasks and objectives 64

Non-functional testing Quality characteristics or non- functional attributes of the system How well or how fast something is done We need to measure on a scale of measurement E.g., time to respond Non-functional testing is performed at all test levels 65

Non-functional testing based on ISO 25010 Portability Maintainability Security Reliability Performance efficiency Compatibility Usability 66

Structural testing Is often referred to as white-box or glass-box because we are interested in what is happening inside the box It can occur at any test level It tends to be mostly applied at unit and integration testing and generally is less likely at higher test levels (e.g., acceptance testing) Good tool support to measure code coverage At component level, and to a lesser extent at (component) integration testing Code coverage: the percentage of executable elements, e.g., statements or decision outcomes that have been exercised 67

Confirmation and regression testing Testing of changes If you have made a change to the software, the way it functions, the way it performs (or both) and its structure have changed We look at the specific types of tests relating to changes, even though they may include all of the other test types 68

Confirmation testing (re-testing) A test run on a code that has been changed in order to fix a defect The test is the same that was run before and that causes the defect to emerge The test is executed in order to confirm that the defect has been fixed It is important to ensure that the test is executed in exactly the same way as it was the first time Using the same inputs, data and environment 69

Confirmation testing: question If the test now passes does this mean that the software is now correct? We now know that at least one part of the software is correct Where the defect was The fix may have introduced or uncovered a different defect elsewhere in the software The way to detect these unexpected side-effects of fixes is to do regression testing 70

Regression testing Involves executing (all) test cases that have been executed before The intent is checking that the system has not regressed It does not now have more defects in it as a result of some changes It is common for organizations to have what is usually called a regression test suite or regression test pack They are designed to collectively exercise most functions in a system (but not test each one in detail) It is appropriate to have a regression test suite at every level of testing All of the test cases in a regression test suite would be executed every time a new version of software is produced and this makes them ideal candidates for automation Maintenance of a regression test suite should be carried out so it evolves over time in line with the software 71

Maintenance testing Once deployed, a system is often in service for years or even decades During this time the system and its operational environment is often corrected, changed or extended Testing that is executed during this life cycle phase is called maintenance testing Note that maintenance testing is different from maintainability testing, which defines how easy it is to maintain the system 72

Maintenance testing Usually maintenance testing will consist of two parts 1. Testing the changes 2. Regression tests to show that the rest of the system has not been affected by the maintenance work 73