Scaling Regression Testing to Large Software Systems

Transcription

1 Scaling Regression Testing to Large Software Systems Alessandro Orso Co-authors: Nanjuan Shi, Mary Jean Harrold College of Computing Georgia Institute of Technology Supported in part by National Science Foundation (awards CCR , CCR , CCR , CCR , SBE , and CCR ) and Boeing Commercial Airplane Group.

2 Testing of Evolving Software Program P Test Suite T P T rerun T T T - T rerun P T rerun T - T rerun Time to rerun T Analysis Time Time to rerun T rerun Savings time

3 Previous and Related Work [OOPSLA01] Retest T (coverage info) Ideal solution: two-phase approach P P Class-Level analysis subset of P Stmt-Level Stmt-Level analysis on the subset Analysis T rerun T rerun Time to rerun T Analysis Time Time to rerun T rerun Savings time

4 Previous and Related Work Precise, Retest Expensive Techniques T (coverage info) Ideal solution: two-phase approach P P Class-Level analysis subset of P Stmt-Level Low-Level Stmt-Level analysis on the subset Analysis T rerun T rerun Jboss, web application server, 1 million LOC Time to rerun T Analysis time Time to rerun T rerun time

5 Previous and Related Work Efficient, Imprecise Techniques T (coverage info) P P High-Level Analysis T rerun Time to rerun T Analysis time Time to rerun T rerun time

6 Previous and Related Work Related Work Efficient, Imprecise Techniques P P Analysis time High-Level Analysis T Efficient, less precise (coverage techniques info) White and Leung [CSM92] Chen, Rosenblum, and Vo [ICSE94] Hsia et al. [SMRP97] White and Abdullah [QW97] Ren et al. [OOPSLA04] Expensive, more precise techniques Binkley [TSE97] Rothermel and Harrold [TOSEM97] Vokolos and Frankl [RQSSIS97] Ball [ISSTA 98] Time to rerun Rothermel, T Harrold, and Dedhia [JSTVR00] Harrold et al. [OOPSLA01] Time Bible, to rerun Rothermel, T rerun and Rosenblum [TOSEM01] time T rerun

7 Proposed Solution Two-Phase Technique T (coverage info) P P Class-Level Analysis Subset of P Subset of P Stmt-Level Analysis T rerun Two-phase approach Class-Level analysis subset of P and P Stmt-Level analysis on the subset T rerun

8 Outline Background Technique Empirical Evaluation Conclusion

9 P Motivating Example class A { void foo() { class C extends B { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) {

10 Motivating Example P P class A { class A { void foo() { class C extends B { void foo() { void foo() { class C extends B { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) {

11 Motivating Example P P class A { class A { void foo() { class C extends B { void foo() { void foo() { class C extends B { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) {

12 Motivating Example P P class A { class A { void foo() { class C extends B { void foo() { void foo() { class C extends B { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) {

13 Class-Level Analysis P P void foo() {

14 Class-Level Analysis P P class A { class A { void foo() { class C extends B { void foo() { void foo() { class C extends B {

15 Class-Level Analysis P P class A { class A { void foo() { class C extends B { void foo() { void foo() { class C extends B { void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo();

16 Class-Level Analysis P / P class A { void foo() { void foo() { class C extends B { Interclass Relation Graph (for P an P ) A F void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) { B C D E Inheritance edge Use edge

17 Class-Level Analysis P / P class A { void foo() { void foo() { class C extends B { Interclass Relation Graph (for P an P ) A F void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) { B C D E Inheritance edge Use edge

18 Class-Level Analysis P / P class A { void foo() { void foo() { class C extends B { Interclass Relation Graph (for P an P ) A F void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) { B C D E Inheritance edge Use edge

19 Class-Level Analysis P / P class A { void foo() { void foo() { class C extends B { Interclass Relation Graph (for P an P ) A F void bar() { A ref=null; switch(somevar) { case 1 : ref=new A(); break; case 2 : ref=new B(); break; case 3 : ref=new C(); break; ref.foo(); class E extends D { class F { void bar(d d) { B C D E Inheritance edge Use edge

20 Example: Stmt-Level Analysis Subset of P class A class B { class C void bar() { ; ref.foo(); Subset of P class A class B { void foo() { class C void bar() { ; ref.foo(); G (excerpt) A G (excerpt) A B A.foo() A.foo() ref.foo() C Dangerous Edge ref.foo() C B B.foo()

21 Example: Stmt-Level Analysis Subset of P class A class B { class C void bar() { ; ref.foo(); Subset of P class A class B { void foo() { class C void bar() { ; ref.foo(); G (excerpt) B ref.foo() C A A.foo() Dangerous Edge G (excerpt) Test cases to be rerun: Test cases in T that A.foo() ref.foo() execute the call node B with ref s dynamic type being B or C C A B.foo()

22 Outline Background Technique Empirical Evaluation Conclusion

23 Empirical Setup Tool DejaVOO Subjects Program Versions Classes KLOC Test Cases Retest Time Jaba min Daikon min Jboss 5 2,403 1, min Three empirical studies 1. Effectiveness of Phase 1 2. Precision gain of Phase 2 3. Overall savings in retesting time

24 RQ1: Effectiveness of Phase 1 TwoPhase T (coverage info) P P Class-Level Analysis Subset of P Subset of P Stmt-Level Analysis T rerun

25 RQ1: Effectiveness of Phase 1 Average: 37% Time < 14 seconds

26 RQ3: Overall Savings in retesting Time Time to rerun T Analysis time Time to rerun T rerun Savings time

27 RQ3: Overall Savings in Retesting Time RerunAll TwoPhase

28 RQ3: Overall Savings in Retesting Time RerunAll TwoPhase Savings in Regression Testing Time: TwoPhase vs. RerunAll Jaba:19% Daikon:36% Jboss: 63%

29 Outline Background Technique Empirical Evaluation Conclusion

30 Contribution Extended our existing regression-testing technique Considerable increase in efficiency Same precision First RTS technique that, at the same time: is safe combines coarse and fine-grained analysis handles the features of the Java language Tool (DejaVOO) that implements the technique Empirical results showing the effectiveness of the technique

31 Current and Future Work Further investigate cost-effectiveness of the technique Continuous integration Larger systems Investigate test-suite augmentation Based on changes (program differencing) Based on field-data (impact analysis)

32 Questions?