Well-defined coverage metrics for the glass box test Rainer Schmidberger rainer.schmidberger@informatik.uni-stuttgart.de ISTE (Institute for Software Technology), University of Stuttgart se
Well-defined coverage metrics for the glass box test Agenda: Background and motivation Overview of today s glass box test A closer look at the underlying models and metrics Requirements for a GBT model My approach: A new and precise model for the GBT The Reduced Program Representation (RPR) RPR execution semantics using Petri nets RPR based metric definition The tool CodeCover Overview Test case selective GBT GBT tool support for test case development Conclusion se Well-defined coverage metrics for the glass box test Slide 2 / 22
Glass box test (1) The glass box test (GBT), also known as white box test or structural test, shows which parts of the program under test have, or have not, been executed. This degree of execution is called coverage. green: executed at least by one of the test cases red: not executed yellow: partly executed GBT-results can be used as test completion criterion or as an input for developing test cases. se Well-defined coverage metrics for the glass box test Slide 3 / 22
Glass box test (2) Tools are required, and many GBT tools are available for almost any programming language. Coverage Report: Empirical studies clearly indicate that higher GBT coverage correlates with lower post-release defect density. Standards for safety critical-software require a very high, or even complete coverage (e.g. IEC 61508, DO-178B). So at first glance, the GBT seems to be a wellestablished and mature testing technique se Well-defined coverage metrics for the glass box test Slide 4 / 22
A closer look at the underlying models.. Typically the control flow graph (CFG) is used to build an abstraction model of the original program code. Most popular GBT metrics are defined with respect to the CFG. But the transformation of the real programs into the CFG is ambiguous: Program code void foo() { if(a) { stmt1; while(b) { stmt2; } } } stmt1 while(b) stmt2 Entry if(a) se Well-defined coverage metrics for the glass box test Slide 5 / 22 Exit CFG Are the entry and exit nodes part of the CFG? Does the while statement have a distinct end node? Does the if statement have a distinct end node?
and metrics And even more severe are the missing representation for exception handling, conditional expressions and the short circuit operations of Boolean expressions. GBT tools show different coverage results for the same execution of a given 45-statement reference Java program CodeCover Version: 1.0.2.2 Clover Version: 3.1.0 Emma Version: v2.1.5320 EclEmma Version: 2.2.1 ecobertura Version: 0.9.8 CodePro Version: 7.1.0 Rational Application Developer V. 9.0.0 62,8 % 58,5 % Statementcoverage Branch-/Blockcoverage Branch: 50,0 % Block: 52,2 % Line: 62,0 % Block: 54,0 % Instruction: 56,7 % Line: 63,6 % Instruction: 57,7 % Line: 58,5 % Block: 50,0 % 64,3 % Branch: 50,0 % Line: 67,0 % Block: 60,6 % A (new) reference model for the GBT is required! se Well-defined coverage metrics for the glass box test Slide 6 / 22
Requirements for a new model for the GBT The model forms the basis on which the popular control-flow based metrics as well as the logic and conditional expression based metrics can be defined. The model supports exception handling. There is an easy and precise transformation rule to transform real programs into the model. The model does not depend on any particular programming language. An algorithm implemented in different programming languages should have the same model representation. The model specifies how to place the probes in the program under test that count the execution of the relevant GBT items. se Well-defined coverage metrics for the glass box test Slide 7 / 22
A new and precise model for the GBT 1. Definition of a primitive language RPR(Reduced Program Representation) which abstracts the GBTrelevant aspects of the real programming languages. 2. Definition of the execution semantics using Petri nets. The nets will also include the execution counters that measure the execution of a particular item. 3. On this basis: Precise definition of the popular GBT-metrics. Statement = PrimitiveStatement IfStatement IfStatement = "if" "(" BoolExpression ")" "then" StatementBlock "else" StatementBlock. BoolExpression = Condition CompoundExpression. A exestmts (P, T) t T : exe(a, t) exestmts(p, T) stmtcov(p, T) = stmts(p) se Well-defined coverage metrics for the glass box test Slide 8 / 22 t Normal s CN s CIn t ENormal s N s In t In s E t EAbrupt Exit area Entry area Execution area t Abrupt s CA s A exe(a, t) M(s CN ) > 0
1. The model language RPR Control flow RPR Grammer (control flow) Program = StatementBlock. StatementBlock = ID "{" StatementList "}". StatementList = Statement ( StatementList empty ). Statement = ID ( PrimitiveStatement TerminateStatement WhileStatement IfStatement SwitchStatement TryStatement ) SubExpressions. PrimitiveStatement = "stmt". TerminateStatement = "throw" "return" "break" "continue". IfStatement = "if" "(" BoolExpression ")" "then" StatementBlock "else" StatementBlock. WhileStatement = "while" "(" BoolExpression ")" StatementBlock. Example: Java System.out.print( GBT"); x1 = ((b*(-1) + Math.sqrt(D))/(2*a)); if(n > 20) { n++; return; } while (x > 0) field[x] = x--; RPR S1 stmt [] S2 stmt [] S3 if(... ) then B2 { S4 stmt [] S5 return [] } else B3 { } [] S6 while(... ) B4 { S7 stmt [] } [] The GBT items are attributed with a unique identifier. This ID is necessary to manage coverage information, but is not part of the original code. se Well-defined coverage metrics for the glass box test
1. The model language RPR Expressions RPR Grammer (expressions) Expression = BoolExpression ConditionalExpression. BoolExpression = ID ( Condition CompoundExpression ). Condition = "expr" SubExpressions. CompoundExpression = ( "andthen" "orelse" "and" "or" ) "(" BoolExpression "," BoolExpression ")". ConditionalExpression = ID BoolExpression "?" SubExpressions ":" SubExpressions. SubExpressions = "[" ExpressionList "]". Example: Java A && B f(a B) A = B? 7 : 42; RPR E1 andthen( E2 expr [], E3 expr [] ) E1 expr [ E2 or( E3 expr [], E4 expr []); ] S1 stmt [ E1 E2 expr []? [] : []; ] ExpressionList = Expression ";" ExpressionList empty. RPR defines all GBT-relevant aspects of the real programming languages. se Well-defined coverage metrics for the glass box test
2. GBT model nets primitive items Primitive GBT items such as a primitive statement or a primitive Boolean expression were described as a place-bordered petri (sub)net called GBT model net. The model nets have exactly one distinct input place and one or more distinct output places for normal and abrupt completion. Places with empty post-set count the GBT items execution. The initial marking is exactly one token in the input place. And there are only final markings with exactly one token in one of the output places. Because all model nets are a place-bordered and token-preserving they can be abstracted into sub nets or super places. s CIn t ENormal t Normal s CN s In s N s N s In t In s E Statement s A t EAbrupt Exit area s S Entry area Execution area t Abrupt s CA s A Statement PrimitiveStatement se Well-defined coverage metrics for the glass box test Slide 11 / 22
2. GBT model nets primitive items Primitive GBT items such as a Input place primitive statement or a primitive Boolean expression were described as a place-bordered petri (sub)net called GBT model net. The model nets have exactly one distinct input place and one or more distinct output places for normal and abrupt completion. Places with empty post-set count the GBT items execution. The initial marking is exactly one token in the input place. And there are only final markings with exactly one token in one of s CIn t ENormal t Normal s CN s In s N s N s In t In s E Statement s A t EAbrupt Exit area s S Entry area Execution area t Abrupt s CA s A Statement PrimitiveStatement the output places. Output places Because all model nets are a place-bordered and token-preserving they can be abstracted into sub nets or super places. se Well-defined coverage metrics for the glass box test Slide 12 / 22
2. GBT model nets complex items The complex GBT items such as if statements or compound Boolean expressions contain other GBT items as part of their own structure. They are described as a composition of model nets. RPR provides the composition rules. This embedding technique automatically provides a dominance relationship between the GBT items. The sub net of an Boolean expression BoolExpression StatementBlock [then-block] s In B 1 s N StatementBlock [else-block] s CIn C s CN sa s T s In B 2 s N t Normal s In t In s In s F s N s A s A Entry area t Abrupt Exit area The sub nets of statement blocks Execution area IfStatement s A s CA A = ddom(b) A ddom B the model net of B is (directly ) embedded in the model net of A se Well-defined coverage metrics for the glass box test Slide 13 / 22
3. GBT metric definition Statement coverage: P is a RPR program, stmts(p) is the set of all GBT items corresponding to the RPR statement production. T is a set of test cases. A exestmts (P, T) A stmts(p) t T : exe(a, t) stmtcov(p, T) = exstmts(p, T) stmts(p) se Well-defined coverage metrics for the glass box test Slide 14 / 22
3. GBT metric definition Statement coverage: P is a RPR program, stmts(p) is the set of all GBT items corresponding Additional to the GBT RPR Metrics: statement production. T is a set of test cases. Branch coverage The degree of executed branches A exestmts Block (P, T) coverage A stmts(p) t T : exe(a, t) The degree of executed statement blocks Decision exstmts(p, coverage T) stmtcov(p, T) = Like Branch stmts(p) coverage, but forking expressions are also taken into account Loop coverage, Term coverage and MC/DC se Well-defined coverage metrics for the glass box test Slide 15 / 22
The tool CodeCover CodeCover Perspective Coverage visualization Coverage visualization and coverage report are based on the selected test cases only. Coverage report se www.codecover.org Well-defined coverage metrics for the glass box test Slide 16 / 22
The tool CodeCover CodeCover Perspective Coverage visualization and coverage report are based on the selected test cases only. CodeCover key features: Has frontends for Java, C and Coverage COBOL visualization Is a reference implementation for the described GBT metrics Is Eclipse integrated and provides Ant interfaces Supports the test case selective GBT Eclipse Public Licence (EPL) Coverage report se www.codecover.org Well-defined coverage metrics for the glass box test Slide 17 / 22
Test case selective GBT The test case selective GBT provides evaluations not only for the entire test suite but also for a single test case. It can be used for example for selective regression testing or test case development. Test suite PUT with CodeCoverenhancements GBT log-file Test case ID 4711 Name Precondition Action, Inputs Expected Result Print invoice Customer is selected 1. 2. 201.40 Justus justus.tigris.org JMX- Interface Init Store double gettotal(customer sustomer, double b) { } // normal customer: no discount double discount = 0.0; if(customer.getrevenue() > 2000) { // All major customers get 10% discount discount = 0.1; if(customer.iscommercial()) { // commercial customers have add. 5% discount += 0.05; } } double total = b * (1.0 - discount); return total; Counter values for test case 4711 S1 3 S2 3 B1 1 se Well-defined coverage metrics for the glass box test Slide 18 / 22
Developing new test cases basic concept Tool-based support for the tester: Developing new input data for test cases that increase coverage. How can I find input values in order to execute this part of the program? Idea: Use the test cases as starting point that execute the dominator of the test target! double gettotal(customer customer, double b) { } // normal customer: no discount double discount = 0.0; if(customer.getrevenue() > 2000) { // All major customers get 10% discount discount = 0.1; if(customer.iscommercial()) { // commercial customers have add. 5% discount += 0.05; } } double total = b * (1.0 - discount); return total; The dominator of the test target The test target se Well-defined coverage metrics for the glass box test Slide 19 / 22
Developing new test cases - definitions P is a RPR program, A P is a GBT item the test target, T is a set of test cases for P, and t T a test case. t testcases(a) t T exe(a, t) Select all GBT items A with testcases(a) = 0 testcases(ddom(a)) > 0 - all not executed GBT items with an executed direct dominator - and put them into a list: GBT item A ddom(a) testcases(ddom(a)) Each entry in this list is called test case recommandation and provides systematical support for the tester to develop new test cases that increases coverage. se Well-defined coverage metrics for the glass box test Slide 20 / 22
CodeCovers RecommendationsView www.codecover.org se Well-defined coverage metrics for the glass box test Slide 21 / 22
Conclusion To date, most GBT metrics are defined either intuitively or based on the CFG. But both definitions have severe shortcomings. In the approach described, the popular GBT metrics are precisely defined by using the following model: A notation (RPR) which is applicable to a large class of programming languages. In RPR control flow, expressions and exception handling are well integrated. Model nets that describe the execution semantics of the GBT items in a mathematical sound way. The model net s counters provide a precise specification for the code instrumentation. The model net also provides a dominance relationship between the GBT items. The tool CodeCover is a reference implementation of the presented metrics, and offers a tool based technique that supports the tester in developing new test cases. se Well-defined coverage metrics for the glass box test Slide 22 / 22