Advanced Topics Combining S. M. T

Transcription

1 Advanced Topics Combining S. M. T 1 an empirical study of predicate dependence levels and trends Icse 2003 david binkley mark harman Or How to use slicing to measure testability 2

2 overview evolutionary testing variable dependence empirical study programs results implications 3 DaimlerChrysler approach to test data generation Test Planning the starting point for the study Test Organization Specification Recombination Reinsertion was Test Case Design by Means of EA Selection Program Mutation Evaluation evolutionary test data generation Test Execution Monitoring Test Documentation Test Evaluation 4

3 Target 5 Level 4 Level 3 approximation level control dependence analysis Level 2 Level 1 Target 6

4 Level 4 Level 3 approximation level control dependence analysis Level 2 Level 1 Target local distance calculation if A = B local distance = A - B! fitness = approximation level + local distance 7 search space reduction variable d is not mentioned so we must keep y so c is killed but wait... these are the inputs x, y, and z seem to be killed search space reduction is exponential but suppose we target this predicate of the original 7 variables only 3 matter which variables matter? 8

5 int g1,,gk; how as k many increases? matter? foo(int x1,,int xn) how as n many increases? matter? int a1,,am; if p(a1,x1,a3,g4,x4) if q(a1,x5) 9 an empirical study the question for a typical predicate how much of input space is relevant? why care? how much cohesive easy is do it inputs? to understand affect? the decision? reduce search space implications for slicing slice size for the predicate is closely related comprehension impact analysis measurement 10

6 inputs formal parameters globals in scope du-globals transitively defined or used in the predicate s procedure 11 Name Descriptionreplace Regular expression string replac the programs studied 12

7 data collection modified HRB slicing algorithm implemented using CodeSurfer formals and du-globals become parameters thanks to GrammaTech for CodeSurfer 13 terminology max parameters the maximum number of parameters a predicate could depend upon parameters used the number of parameters a predicate actually depends upon 14

8 result data three results to summarise in one data point max parameters parameters used number of predicates summarised we adopted two diagramatic techniques 15 dependence skyline diagram notice the trend for dependent proportion to drop 8 predicates have max formals of 3 and use 2.2 on average evident savings for prepro replace prepro listed in ascending order of max formals but precise skylines reading give is a lost feeling for size reduction 16

9 dependence bubble chart replace formal parameters prepro angle shows the size reduction these which s predicates max have parameters there bubble x x-axis: y-axis: max depend are at formals s (x,y) max predicates = on parameters of formals average available size used s on used to y them formals trend line 17 over all predicates in all programs 18

10 good news as max formals increases the dependent proportion falls well almost good news for evolutionary testing and also for all the other applications perhaps it is not good for cohesion 19 declining dependent proportion all predicates where max formals < 11 The number of predicates all predicates depended where upon max formals > 10 as a function of max formals available is piecewise linear 20

11 declining dependent proportion max formals < 11 max formals > bad news no such correlation for globals globals could entail untestability using search also bad for other applications is this more evidence that globals are bad 22

12 dependence trends Max formals Max du globals 23 implications formals: as the problem gets worse the solution gets better 24

13 implications globals: large global variable lists present problems hard to generate test data hard to understand high levels of dependence 25 implications cohesion: functions with large numbers of formals may not be so cohesive 26

14 algorithm performance the algorithm used has complexity O(P*S) where P is the number of predicates and S is the size of the procedure PII 450 with 386Mb memory analysis time per procedure = * P * S ms average analysis time 7.6ms per predicate 27 some possible interpretations what do you read into these diagrams? 28

15 flex evolution of du globals version version big bubbles top right big bubbles bottom left profiles sendmail du globals findutils du globals 30

16 conclusions falling formal dependent proportion invariant global dependent proportion analysis is worth performing diagrams may be an aid to understanding evaluation monitoring evolution 31 Testability Transformation Overview or How to use slicing and transformation to improve testing Test Data Generation Evolutionary Test Data Generation The Flag Problem Flag Removal Algorithm Initial Results Testability Transformation Other non meaning-preserving transformations If time permits 32

17 Automatic Test Generation We know that generating good quality test data is hard and knowing what good quality means is hard I do not propose to answer that question today Starting point: structural test adequacy criterion Specifically: that some branch is to be covered and that we are going to use evolutionary testing 33 Distance Based Fitness Level Approximation level Level 3 Relevant branching statements can lead to a miss of the desired target Level 2 Level 1 Target Local distance calculation in in the branching statements with undesired branching Evaluation of predicate in a branching condition in the same manner as described for safety testing, e.g. if A = B Local_Distance = max - ( A - B )! Fitness = Approximation_Level + Local_Distance 34

18 if (A==0)... Suppose we want to make this true Max 10 - ( A-0 ) The Flag Problem flag = A==0; if (flag)... fitness fitness Value of A Value of A Transform program to transform fitness function to transform landscape Better Flag Landscape Tiny plateau of high fitness Large plateau of low fitness 36

19 Informally A transformation is a partial function on programs which preserves meaning,of some kind or other We need to pair the program and test adequacy criterion call this the test pair A testability transformation is a partial function on test pairs such that Testability Transformation Test data which is adequate for the transformed test pair is adequate for the original test pair 38

20 Testability Transformation Paradox We are testing to cover structure but the structure is the problem So we transform the program and this alters the structure So we need to be careful: Are we still testing according to the same criterion? Our transformations will preserve coverage of Statements branches MC/DC Future work: define a semantics to verify this 39 This is not abstract interpretation To preserve if branch (e) skip; coverage: else skip; Cannot be transformed to skip; But the program if (e) x=1; else x=2; Can be transformed to if (e) skip; else skip; 40

21 Flag TT Perhaps Fairly less well well known known Transform the program to remove flags Not always possible but worth doing where possible Our Approach uses Simple transformations Simple amorphous slicing Substitute flag use with definition Brief overview of amorphous slicing Flag Removal Transformation What Suppose initial Suppose we values want n is to of an make n will unsigned achieve flag true integer this? here flag = n < 4; if flag n! (n%2==0) = n; = (n%2==0)?0:(n<4); flag = 0; flag = (n! %2==0)?0:(n! <4); if (a[i]!= 0 && flag) (n! %2==0)?0:(n! <4))... Once we have this We can keep the original flag assignment code Claim: Adding the new flag assignment leaves adequacy criteria invariant 42

22 Simple Flag Removal Algorithm For loop free flag definition code Bush Blossom Slice leaf sequences Convert to conditional assignment Add temporary variables Substitute definition for use 43 Bushing Produces a binary tree p a qqqqqqqqqq q qqqqqqqqqq q b b b rr r r r r r r r r r r rr r r r r r r r r r r r r r r r r c c d cc d d d d c dd ddd c c dd c dd c c dd d d d d d d d dd ccccc dd cccc dd ddddddd d ccccccc ccc c c cc ccccccccccccccc 44

23 Blossoming p Moves all actions to the leaves All internal nodes will be predicates Original predicates may be altered a q q b b r r r r ' b c d c d c d cb b db c d 45 Blossoming is repeated for all internal action nodes Some Now all leaf internal assignments nodes are predicates not to the flag variable p these can be removed using slicing aaa aa a a aa q a q aa ' aa b a a r r''' rr'''' r '' a a a a c d d c d cb db cb db a a a a c d c d cb db c d 46

24 Amorphous slicing gives single assignment at leaves Sometimes But sometimes syntax-preserving Some predicates slicing change slicing does leaves several this too several times during assignments blossoming We assumed freedom from side effects p Fortunately we have a side effect removal transformation q q' b r r'''' r''' r' ' c d bc c bd d ac a ad a cac a ad da c d b b c d Now all leaves are single assignments and they all assign to the flag variable 47 Initial Empirical Analysis Daimler ran their Evolutionary Test Data Generator on both versions Collected coverage information for 6 runs 48

25 Special Values /* date correction for september 1752 */ if(special_days) result = "Day did not exist."; else if (leapflag && is_september && day>13) result = dayname((addmonths(month,year)+ (--day)+firstjanuary(year)+10)%7); else... With flags we never even got up here Remove flags With flags it took longer to get anywhere /* date correction for september 1752 */ if(year==1752 && month==9 && day>=3 && day<= 13) result = "Day did not exist."; else if (year==1752 && month==9 && day>13) result = dayname((addmonths(month,year)+ (--day+firstjanuary(year)+10)%7); else Nothing special flag returnflag = (a==0 b==0 c==0) a>10000 b>10000 c>10000 (c>=a+b) (a>=b+c) (b>=a+c);... Is it hard to find values which make this flag false? true? if (returnflag) return; To have Remove a problem flags we need flag for which few inputs make it true/false There is very little difference Let s look at a bad flag problem if (a==0 b==0 c==0) a>10000 b>10000 c>10000 (c>=a+b) (a>=b+c) (b>=a+c) return; 50

26 Special Value Flag returnflag = (a==99999 b==99999 c==99999);... if (returnflag) return; at less cost There So This than we predicate are seldom this relatively onerandomly has a few smoother values hit this landscape which statement make the flag true... if (a==99999 b==99999 c==99999) return; So we get better coverage 51 Disposable Transformations We generate test data using the transformed program because it is easier then throw away the transformed program Transformation as a means to an end not an end in itself Do the transformations even need to preserve meaning? This is a radically new form of transformation 52

27 Conclusion Test data generation is hard anything which helps is good Test data generation can be impeded by structure so transform the structure We have to be sure to preserve branch coverage but not traditional meaning This suggests a new kind of transformation Testability Transformation 53 References David Binkley and Mark Harman Analysis and Visualization of Predicate Dependence on Formal Parameters and Global Variables. IEEE Transactions on Software Engineering. (journal version of ICSE 2003 paper, to appear) Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, Andre Baresel and Marc Roper. Testability Transformation. IEEE Transactions on Software Engineering. 30(1): 3-16, David Binkley and Mark Harman. An Empirical Study of Predicate Dependence Levels and Trends 25th IEEE/ACM International Conference on Software Engineering (ICSE 2003) May, Portland, Oregon, USA, Pages Electronic copies of these papers are available on my website. 54