Integrating Customized Test Requirements with Traditional Requirements in Web Application Testing

Transcription

1 Integrating Customized Test Requirements with Traditional Requirements in Web Application Testing Sreedevi Sampath, Sara Sprenkle Emily Gibson and Lori Pollock University of Delaware July 17, 2006 Workshop on Testing, Analysis and Verification of Web Applications and Services (TAV-WEB)

2 Need for Reliable Web Applications Expedia sells more than $5 million in tickets every week 1 In 1999 KBKids.com $10 off 0 dollars or cents? 2 Huge losses on web site failure Financial services: $6.5 million per hour Credit card sales applications: $2.4 million per hour Media companies: $150,000 per hour Large number of failures during maintenance 4 1. e-business 2.0: Roadmap for Success (2nd Edition) by M. Robinson, D. Tapscott, R. Kalakota More Web sites turn to test tools by Carol Sliwa in CNN.com Web Application Development - Bridging the Gap between QA and Development by Michal Blumenstyk 4. Causes of Failures in Web Applications by Solia Pertet and Priya Narsimhan. December 2005 TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 2/29

3 User-session-based Testing Process Users User 1: register.jsp?name=ss&pass=tst login.jsp?name=ss&pass=tst logout.jsp User-session-based Beta Web Application (v.0.9) Deployment Log User Requests User Sessions Create test cases Web Application Implementation (v.1.0) Test Cases Test Cases Replay Tool Expected Results Actual Results Oracle Pass/ Fail TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware /29

4 Measure Quality of Test Suites Test requirement coverage When to stop testing Which test cases to select How to reduce a test suite Coverage and data flow-based requirements Statement Method Branch Def-use Covering all the statement requirements ensures the statement coverage criterion is satisfied TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 4/29

5 Program Coverage-based Test Requirement Requirement: statement Create reduced suite that covers all statements in code covered by original suite Reduce Original Test Suite (set of test cases) Reduced Test Suite Advantage + Guaranteed program coverage by reduced suite Disadvantage - Expensive to execute the original test suite prior to reduction TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 5/29

6 Measure Quality of Test Suites Test requirement coverage When to stop testing Which test cases to select How to reduce a test suite Coverage and data flow-based requirements We proposed usage-based test requirements Derived from usage-data base, name, seq2, namevalue, seq2name base request User 1: register.jsp?name=ss&pass=tst login.jsp?name=ss&pass=tst logout.jsp name-value pairs TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 6/29

7 Usage-based Test Requirement Requirement: base request Create reduced suite that covers all base requests covered by original suite User 1: register.jsp login.jsp logout.jsp Reduce Original set of user sessions Reduced set of user sessions Advantages + Information present in the user session itself + No need to execute original suite, cost-effective TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 7/29

8 Disadvantage of Usage-based Test Requirements register.jsp with name-value pairs Servlet 1 Servlet 2 How much code are we guaranteed to cover by covering requirement register.jsp? response to user Servlet 4 Servlet register.jsp response to user Servlet 1 TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 8/29

9 Integrate Program-based and Usagebased Requirements We want to achieve a balance between Cost of generating reduced suite and size of reduced suite Program coverage and fault detection effectiveness of reduced suite TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 9/29

10 Our Research: Strategies for Integration Integrate program coverage-based and usage-based test requirements to 1. Compare test suites to identify the better suite 2. Combine test requirements for reduction. Augment existing reduction algorithm TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 10/29

11 Our Research: Strategies for Integration Integrate program coverage-based and usage-based test requirements to 1. Compare test suites to identify the better suite 2. Combine test requirements for reduction. Augment existing reduction algorithm Focus of this talk TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 11/29

12 Usage-based Test Requirement: seqk Example test case < GET login.jsp?name=xxx&pass=yyy, GETshop.jsp?item_no=aaa&book_name=ccc&price=60 > seqk: cover all size k sequences of base requests for k=2, <{GET login.jsp, GET shop.jsp}> TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 12/29

13 Usage-based Test Requirement: name Example test case < GET login.jsp?name=xxx&pass=yyy, GETshop.jsp?item_no=aaa&book_name=ccc&price=60 > name: cover all base requests and names {GET login.jsp?name&pass, GET shop.jsp?item_no&book_name&price} TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 1/29

14 Usage-based Test Requirement: namevalue Example test case < GET login.jsp?name=xxx&pass=yyy, GETshop.jsp?item_no=aaa&book_name=ccc&price=60 > namevalue: cover all base requests and name and value pairs {GET login.jsp?name=xxx&pass=yyy, GET shop.jsp?item_no=aaa&book_name=ccc&price=60} TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 14/29

15 Usage-based Test Requirement: seqkname Example test case < GET login.jsp?name=xxx&pass=yyy, GETshop.jsp?item_no=aaa&book_name=ccc&price=60 > seqkname: cover all size k sequences of base requests and names for k = 2, {<GET login.jsp?name&pass, GET shop.jsp?item_no&book_name&price>} TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 15/29

16 Traditional HGS Augment HGS [Harrold et al.] with usagebased requirements as tie breakers Select next test case to cover the leastcovered requirement Cardinality: number of test cases that cover each requirement Requirements R0 R1 R2 R R4 R5 R6 card T0 T1 Test Cases T2 T T4 T5 TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 16/29

17 Traditional HGS card T0 T1 Test Cases T2 T T4 T5 R0 4 Requirements R1 R2 R R4 R R6 Start with requirement with least cardinality Only T covers R2 TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 17/29

18 Traditional HGS card T0 T1 Test Cases T2 T T4 T5 R0 4 Requirements R1 R2 R R4 R R6 Choosing T covers R2, R4, R5, R6 TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 18/29

19 Traditional HGS card T0 T1 Test Cases T2 T T4 T5 R0 4 Requirements R1 R2 R R4 R R6 R is covered by two test cases, T0 and T4 Choose test case that covers most uncovered requirements Tie betweent0 and T4, Go to next higher cardinality Eventually, select randomly from tied test cases TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 19/29

20 Modified HGS R0 card 4 T0 T1 Test Cases T2 T T4 T5 Usage-based requirement coverage Requirements R1 R2 R R4 R5 R T0 GET login.jsp GET shop.jsp T4 GET login.jsp Select T0 for the reduced suite More effective when large number of ties encountered TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 20/29

21 Case Study Compare reduced suites from Traditional HGS with program coverage-based requirements Traditional HGS with usage-based requirements Modified HGS with program coverage-based requirement () and usage-based tie breakers Metrics Reduced suite size Program coverage Fault detection TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 21/29

22 Subject Application Course Project Manager (CPM) Number of classes Number of s Non-commented LOC Number of seeded faults Number of user sessions Total number of URLs ,52 TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 22/29

23 Case Study: Methodology (1) Users User requests Web Application code Parse requests Logged requests User sessions (original test suite) TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 2/29

24 Case Study: Methodology (2) HGS Program reduce replay Original test suite HGS Usage reduce Modified HGS Prog + Usage reduce Reduced suites replay replay Generate coverage, fault detection reports TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 24/29

25 Results: Reduced Test Suite Size Percent of Original Test Cases S C M stmt brn meth S1 M-s1 N M-n S2 Reduced Suite M-s2 S2N M-s2n NV M-nv Total URLs Requested Percent of Original Test Cases Label in graph M-s1 M-n M-s2 M-s2n M-nv Modified HGS Prog cvg req. Total URLs Requested Tie breaker base request name seq2 seq2name namevalue TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 25/29

26 Results: Program Coverage Effectiveness 5500 Total Number of statements in CPM: 6966 Number of statements covered by original suite: 5468 # of Covered Statements HGS S C M stmt brn meth S1 M-s1 N M-n Reduced Suite S2 M-s2 S2N M-s2n NV M-nv Label in graph M-s1 M-n M-s2 M-s2n M-nv Modified HGS Prog cvg req. HGS-Method Tie breaker base request name seq2 seq2name namevalue TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 26/29

27 Results: Fault Detection Effectiveness 100 Percent Faults Detected S C M stmt brn meth S1 M-s1 N M-n S2 Reduced Suite M-s2 S2N M-s2n NV M-nv Label in graph M-s1 M-n M-s2 M-s2n M-nv HGS-Method Modified HGS Prog cvg req. Tie breaker base request name seq2 seq2name namevalue TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 27/29

28 Results: Summary Fault detection of Modified HGS better than Traditional HGS for same suite size and generation cost Usage-based requirement alone more effective than using Modified HGS with program-based and usage-based tie breakers but test suite size increases Type of usage-based requirement -- no effect on effectiveness of Modified HGS reduced suites TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 28/29

29 Conclusions and Future Work Presented three strategies to integrate usagebased and program-based requirements in paper Experimentally evaluated the strategies Instead of using requirement, use customized requirement alone Combining requirements better than HGS- Modified HGS better than traditional HGS Future Work Extend study to other applications Augment test cases from models of application with usage-based requirements TAV-WEB, July 2006 Sreedevi Sampath. University of Delaware 29/29