Automated Acceptance testing by Developers & Automated Functional Testing by Testers Gowrishankar Sundararajan QA Manager Tata Consultancy Services, Canada
Executive Summary Overview on Traditional Agile Testing Approach Recommended New Agile Approach with Benefits Comparison of Traditional Vs New Proposed Approach Case Study An implementation overview with KPI s 2
Introduction to Agile Introduction Agile Flow Frequent delivery of deployable business value Frequent changes in requirements Close collaboration between entities Prioritization of work is done by the stakeholders on the basis of requirements and focus is then paid on the highest risk issues as the work progresses Customers/business owners make business decisions, developers make technical decisions 3
Test Completion % Planning Traditional Agile An Overview Planning Coding Defect Fix Closure Day 1 Day 2-8 Day 9-13 Day 14-15 Day 1 Planning Test Design QA Plan day wise In a normal 3 weeks Sprint window, below is the QA plan/duration for each testing Phase Day 1 Previous Sprint Handover to UAT/Business and Scope finalization for new Sprint Day 2-8 Test Design phase in parallel with Coding Day 9-13 Test Execution Functional Testing Day 14-15 Test Execution Regression Testing for the current sprint Test Execution 100% Closure Design 80% Execution Closure Week1 Week2 Week3 Sprint Duration 4
Drawbacks/Key Issues faced in Traditional Approach On day 9, when testing team starts with Smoke testing there is always greater chance to identify Blocker defects. Due to blocker defects, a day or two will be lost which makes shorter duration for testing team to complete test execution in 5 days. Issues Comparison Satisfaction Sometimes, even regression for the current sprint release cannot be completed. In Many Cases, regression of prior releases are ignored which will have higher impact in UAT. Finally, test suite preparation of regression & End-End testing involves considerable effort, cost. Due to time constraint, few items needs to be removed from the current sprint. Ultimately, Product team is unhappy with the items not getting delivered which was promised. 5
The Solution New Best in Class Approach Planning Day 1 Day 2-4 Planning Coding Defect Fix Closure Sanity Test Day 5-8 Day 9-11 Day 12-13 Day 14-15 Day1 All possible Automated In-sprint Prior sprint Core Script Combination Script Exec Regression Regression Test Design Test Execution Closure QA Plan day wise Day 2 4 : Testers develop the core automated script and test data for Critical path to satisfy Smoke/Sanity Criteria and hands it to Dev team to run the script before sending the new build to QA Day 5 8 : Testers add scripts to cover all possible test scenarios Day 9 11 : Functional Test Execution using Automation Scripts Day 12 13: In-sprint Defect Resting & Regression Test Execution Day 14 15: Regression for Prior sprints 6
Test Completion % New Best in Class Approach vs. Traditional Approach Comparison Chart Planning Test Completion % Planning Traditional Approach New Approach 80% Sprint Design Sprint Execution Closure Sprint Design Acceptance test Sprint Execution Core Regression Closure Week1 Week2 Week3 Sprint Duration Week1 Week2 Week3 Sprint Duration 7
The Solution New Best in Class Approach Required Change in Approach Automation Early Automation Start automation early in SDLC from smoke/sanity testing rather than waiting for functional testing Smoke/ Sanity Test Testers will no longer perform Smoke/Sanity testing Testing team develops automated smoke/sanity test Scripts Developers executes automated smoke test on new build before handover to QA Functional Functional testing is no longer manual Maximized Automation Scripts reuse for Regression/End-End testing 8
Benefits with New Approach Zero Showstopper Leakage from Dev to SIT to UAT Reduced Major defects leakage from SIT to UAT Defect Leakage Key Benefits Cost Saving Cost saving due to automated smoke, functional testing Automation ROI for regression and end to end testing is much faster Increased coverage by ensuing room for in-sprint regression Ensure coverage for core regression Coverage Customer Satisfaction Deliver what is promised for each print More confidence on delivered product Improved overall customer satisfaction 9
Case Study Testing Support for a Leading Canadian Money Transfer Payment Network Provider Project Information Client Profile and Background No of resources 7 (1 On : 6 Off) Product Usage Money Transfer across FI s Technologies Java, Xml, Batches, Web Canada's largest money transfer Payment network and Gateway provider. The client does the services similar to VISA, PAYPAL etc. and carries out 98% of money transfer across people belonging to same/different FI s with in Canada. Key Features Automation Tool SOAP SONAR Duration Jan 2013 Dec 2015 Sending Money across FI s/credit Unions Requesting Money across FI s/credit Unions Tracking / Listing the transactions Consolidated Reports 10
Case Study Testing Support for a Leading Canadian Money Transfer Payment Network Provider Project Intro Web banking customers of participating FI -> electronic payments to anyone else -> web banking customer of a participating FI. Current functionality one-time, immediate money transfer. Future Release includes Requesting Money, Auto-deposit etc. The system uses an application service provider model (XML s) with an interface to the FI s banking site Scope of Work Current feature is built on Java Technology (Struts/JDBC) -> System heavy and affecting its performance. Rewrite of existing application + new features on Spring/Hibernate feature. No change in the XML Structure that is being used across FI s for Sending Money. Regression Testing of Old Features Functional Testing of New Features Integration Testing of Both Features 11
Case Study Testing Support for a Leading Canadian Money Transfer Payment Network Provider Flow Diagram Send Money Receive Money Customer Send Request Receive Money Customer Info update Banks / Credit Union Gateway Banks / Credit Union Receive Request Fulfill Request Customer Types of Testing Smoke/Acceptance Testing System/Functional Testing System Integration Testing (End to End) Regression Testing Test Automation User Acceptance Testing Beta Testing Performance Testing 12
Work Load Split-Up MODULES USAGE SPLIT % FUNCTIONAL TESTING EFFORT SPLIT % Though our application uses web services only by 50%, actual functional testing effort to test other modules in turn requires to execute web services again. Overall Functional testing effort alone was more than 70% with web services module 13
Case Study Key KPI s to Prove DEFECT Productivity LEAKAGE TO UAT Productivity DEFECT LEAKAGE BY Productivity SPRINT On an average, we had 4 5 Blocker and 24 Major Defects leakage to UAT from SIT before implementing new approach. Later, we only had 5 Major and Max of 1 Blocker defect. From the above graph, it is clear that there is drastic reduction in the number of both Blocker and Major defects from Sprint 9 (after implementing new approach). **First 3 Sprints count is ignored to understand application and taking into consideration as Learning Curve 14
Case Study Key KPI s to Prove REGRESSION Productivity Productivity DEFECTS LEAKAGE TO UAT REGRESSION Productivity DEFECTS IDENTIFIED IN SIT From above graph, it is clear that there is Zero defect leakage (both Blocker & Major) to UAT from Sprint 11. The reason we had it in Sprint 9 & 10 is that regression coverage was done gradually. From the above graph, it is clear that a) Regression Defects are caught in SIT phase. b) Considerable reduction in number of defects leaked from Dev to SIT. 15
Case Study Key KPI s to Prove TEST Productivity EXECUTION Productivity Productivity COVERAGE QA DAYS LOST Productivity QA days lost due to showstopper issues Approach Type Traditional New Days Lost 4 PD s 0 PD s Exit criteria of any testing team is to ensure 100% coverage is covered. But due to Blocker defects, we always try to cover High and Medium Priority test cases. Hence, even some Low priority test cases will result in Major UAT defect. GAPS IN EXISTING SYSTEM After implementing new approach, all the possible negative flows/paths is covered. This helped us in identifying few gaps in existing system, which helped business to fix this issues before rolling out new features. This will in turn save number of tickets opened to support team and their effort in future. 16
How easy to get this approach implemented in other Testing Model Most of the interactions across applications happens thru web services, interfaces/esb. And in recent days, after introduction of No Hassle Component Integration testing this approach is easier to implement and get the benefits out of it. Though we implemented specifically for XML s, same approach can be used for web and other technologies. 17
Questions? 18
Thank You! Contact @ shankarsole@yahoo.co.in