Fault Localization Using Tarantula

Transcription

1 Class 20 Fault localization (cont d) Test-data generation Exam review: Nov 3, after class to :30 Responsible for all material up troug Nov 3 (troug test-data generation) Send questions beforeand so all can prepare Exam: Nov 10 Final project presentations: Dec 1, 3; :35-6:5 Assign (see Scedule for links) Problem Set 9 discuss Readings 1 Fault Localization Using Tarantula Wat information does Tarantula use to compute suspicious (and ing) of statements in te program? How is tis information used? Are tere oter ways to compute te suspiciousness using tis information? Wat information oter tan statement coverage could be used for fault localization? Do you tink statement coverage would ave worked for tritype? How could we use fault localization to identify wic canges are most suspicious after a build?

2 Improving Fault-localization Efficiency P P Debug Debug P all pass Are all failing caused by te same fault? Can we associate groups of wit different faults? Can we reduce debugging effort by considering tese groups individually? Can we reduce debugging effort by considering tese groups simultaneously? P i is failure-free Improving Fault-localization Efficiency mid() { int x,y,z,m; 1:read( Enter 3 integers:,x,y,z); 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; 8:else 9: if (x>y) 10: m = z; 11: else if (x>z) 12: m = x; 13:print( Middle number is:, m); } t1 t2 t3 t t5 t6 t t8 t9 3,3,5 1,2,3 3,2,2 5,5,5 1,1, 5,3, 3,2,1 2,1,3 5,,2 5,2,6 Pass/fail Status P P P P P P F F F F

3 Improving Fault-localization Efficiency t1 t2 t3 t t5 t6 t t8 t9 mid() { int x,y,z,m; 1:read( Enter 3 integers:,x,y,z); 2:m = z; 3:if (y<z) : 5: if (x<y) m = y; 6: : else if (x<z) m = y; 8:else 9: 10: if (x>y) m = z; 11: else if (x>z) 12: m = x; 13:print( Middle number is:, m); } Pass/fail Status P P P P P P F F F F 3,3,5 1,2,3 3,2,2 5,5,5 1,1, 5,3, 3,2,1 2,1,3 5,,2 5,2,6 Debugging Process P Debug P Debug P all pass some P Debug P some all pass P j is failure-free P i is failure-free

4 Debugging Process P P Debug Debug P all pass some P Debug P some all pass P some some Debug P all pass Debug P k is P j is failure-free failure-free P i is failure-free Hierarcy of ugs Faults often dominate eac oter Failing test cases are first caused by a set of initial faults Once initial faults are fixed, oter faults manifest temselves Fault 1 Fault 2 Fault 3 Fault Fault 5 Fault 6 Fault Fault 8 Time

5 Debugging Process P P Debug Debug P all pass all Potential benefits: Potential some costs some P P Reduced time to failure-free Overead Debug to partition pass test program cases Less noise in locating eac Multiple debuggers fault (developers) all etter utilization of developer Debug P effort P pass some some Debug P k is P j is failure-free failure-free P i is failure-free Debugging Process P Debug some P Debug P some some Debug Debug P Crucial problem: P Crucial problem: Debug all some P pass P all pass all pass Partitioning into Partitioning into groups of similar beavior groups of similar beavior focus on different faults focus on different faults fault-focusing clusters fault-focusing clusters of test cases P k is P j is failure-free failure-free P i is failure-free

6 Fault-focusing Clusters Overview Test Cases Fault-focusing clusters: Clusters of failing test cases Clusters failing in similar way Eac cluster targeting a different fault Fault-focusing Specialized Test Clusters Suites Test Cases Specialized test suites: Fault-focusing clusters combined wit passing test cases

7 Fault-focusing Specialized Test Clusters Suites Specialized test suites: Fault-focusing clusters combined wit passing test cases Developer Find faults one at a using specialized test suites Developer Fault-focusing Specialized Test Clusters Suites Test Cases Specialized test suites: Fault-focusing clusters combined wit passing test cases Developer 1 Find faults one at a using specialized test suites Find faults at te same time (in parallel) using specialized test suites Developer 2

8 Fault-focusing Clusters test cases execution information Execution Clustering specialized test suites Fault Localization suspiciousness and s Fault-focusing Clusters test cases execution information Execution Clustering specialized test suites Fault Localization suspiciousness and s Clustering by beavior models Dynamic information profiles (branc, metod-metod, ) only Statistical analysis, macine learning generate models for eac execution cluster models Fault-localization for stopping point

9 Clustering eavior Models Models: Most difficult discrete-time problem Markov of clustering cains (DTMCs) is determining from profiles a good (branc, stopping metod, ) criterion? Clustering: iterative wit two most similar according to Sim1 Sim1: sum of absolute difference between matcing Wat is transitions a good stopping in DTMCs point being for compared te clustering for fault-focused clusters? Fault Localization for Stopping Point

10 Fault Localization for Stopping Point AI AU Fault Localization for Stopping Point AI AU

11 Tarantula: Fault Localization mid() { int x,y,z,m; 1:read( Enter 3 integers:,x,y,z); 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; //bug 8:else 9: if (x>y) 10: m = z; //bug 11: else if (x>z) 12: m = x; 13:print( Middle number is:, m); } t1 t2 t3 t t5 t6 t t8 t9 3,3,5 1,2,3 3,2,2 5,5,5 1,1, 5,3, 3,2,1 2,1,3 5,,2 5,2,6 Pass/fail Status P P P P P P F F F F suspiciousness Fault Localization for Stopping Point AI AU

12 Fault-focusing Cluster 1 mid() { int x,y,z,m; 1:read( Enter 3 integers:,x,y,z); 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; //bug 8:else 9: if (x>y) 10: m = z; //bug 11: else if (x>z) 12: m = x; 13:print( Middle number is:, m); } t1 t2 t3 t t5 t6 t t8 t9 3,3,5 1,2,3 3,2,2 5,5,5 1,1, 5,3, 3,2,1 Pass/fail Status P P P P P P F 5,,2 F suspiciousness Fault Localization for Stopping Point AI AU

13 Fault Localization for Stopping Point AI AU Fault-focusing Cluster 2 mid() { int x,y,z,m; 1:read( Enter 3 integers:,x,y,z); 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; //bug 8:else 9: if (x>y) 10: m = z; //bug 11: else if (x>z) 12: m = x; 13:print( Middle number is:, m); } t1 t2 t3 t t5 t6 t t8 t9 3,3,5 1,2,3 3,2,2 5,5,5 1,1, 5,3, Pass/fail Status P P P P P P 2,1,3 F 5,2,6 F suspiciousness

14 Fault Localization for Stopping Point AI AU Fault Localization for Stopping Point AI AU

15 Fault Localization for Stopping Point AI AU Fault Localization for Stopping Point AI AU

16 Fault Localization for Stopping Point AI AU Fault Localization for Stopping Point Composite is similar (above tresold) to bot of its constituents so clustering stops at tis level Result is two clusters: {, }, {, }

17 Fault Localization for Stopping Point Composite is similar (above tresold) to bot of its constituents so clustering stops at tis level Result is two clusters: {, }, {, } Fault-focusing Cluster 1 mid() { int x,y,z,m; 1:read( Enter 3 integers:,x,y,z); 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; //bug 8:else 9: if (x>y) 10: m = z; //bug 11: else if (x>z) 12: m = x; 13:print( Middle number is:, m); } t1 t2 t3 t t5 t6 t t8 t9 3,3,5 1,2,3 3,2,2 5,5,5 1,1, 5,3, 3,2,1 Pass/fail Status P P P P P P F 5,,2 F suspiciousness

18 Fault-focusing Cluster 2 mid() { int x,y,z,m; 1:read( Enter 3 integers:,x,y,z); 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; //bug 8:else 9: if (x>y) 10: m = z; //bug 11: else if (x>z) 12: m = x; 13:print( Middle number is:, m); } t1 t2 t3 t t5 t6 t t8 t9 3,3,5 1,2,3 3,2,2 5,5,5 1,1, 5,3, Pass/fail Status P P P P P P 2,1,3 F 5,2,6 F suspiciousness Visualization of Specialized Test Suites Cluster 1 Cluster 2 mid() { int x,y,z,m; 1:read( Enter 3 integers: 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; 8:else 9: if (x>y) 10: m = z; 11: else if (x>z) 12: m = x; //bug //bug 13:print( Middle number is: } mid() { int x,y,z,m; 1:read( Enter 3 integers: 2:m = z; 3:if (y<z) : if (x<y) 5: m = y; 6: else if (x<z) : m = y; 8:else 9: if (x>y) 10: m = z; 11: else if (x>z) 12: m = x; //bug //bug 13:print( Middle number is: }

19 Empirical Study Variables NSTS: Finding faults using non-specialized test suites STS-S: Finding faults using specialized test suites STS-P: Finding faults using specialized test suites in parallel Measures D: total developer effort FF: total effort to failure-free program Subject SPACE 6000 LOC fault versions; > 1000 derivative versions) Metod For eac of fault versions, debug until failure-free, using Non specialized test suite Specialized test suite bot sequential and parallel D: Total Developer Effort Using specialized test suites based on fault-focusing cluster is less expensive, on average, tan not using specialized test suites enefit olds wen performing fault localization sequentially (one developer) fault localization in parallel (multiple developers)

20 FF: Total Effort to Failure-free Sample Source Sample mean Sample standard deviation 99% confidence interval lower bound 99% confidence interval upper bound FF NSTS FF STS-S FF STS-P Using specialized test suites and performing te fault localization sequential or in parallel can provide significant savings over using non specialized test suites Summary of Results For SPACE Using specialized test suites is usually less expensive tan using non-specialized test suites Total developer effort is reduced for bot sequential and parallel modes Time to a failure-free program is reduced witout negatively affecting te total developer effort

21 Automatic Test Data Generation 1 Test Data Generation Ferguson and Korel described tree categories of test-data generation Wat are tey? 2

22 Test Data Generation Ferguson and Korel described tree categories of test-data generation 1. Random randomly select from universe of inputs 2. Goal-oriented select test data to execute a given entity (e.g., statement, branc, def-use pair) irrespective of te pat taken 3. Pat-oriented select a program pat and generate test data tat will execute tat pat; pat can be selected automatically or selected by te user 3