Agile vs Fragile. Susmit Bhattacharya, Solution Architect, Asia Pacific. - The need for Automation in Agile Tricentis GmbH. All Rights Reserved.

Years Months Months Weeks Delivery Cycle Time Weeks Days Perceived Disruption Beyond Continuous Testing Days Hours Hours Minutes Technical Complexity & Rate of Change Today Critical Physical Limit Ever-Growing Challenge 2017 Tricentis GmbH. All Rights Reserved. 2

6+ Big Bang Waterfall 4 Incremental Rational Unified Process 2 Agile Scrum, Kanban 1 Continuous DevOps Requirements Design Implementation Testing Acceptance Deployment Following a Plan Contract Negotiation Copious Documentation Processes & Tools Agile Event Horizon Inception Elaboration Construction Transition Development Operations DevOps Digital Singularity Rigid Rules Flexible Framework Individuals & Interactions Working Software Customer Collaboration Responding to Change Development Response = wall of confusion 2017 Tricentis GmbH. All Rights Reserved. 3

Collaborative Development Continuous Release & Deployment DevOps is no longer a unicorn s discipline. Continuous Integration Dev Ops Continuous Feedback Continuous Testing Continuous Monitoring 81% Dev Shift Left Test Shift Right Ops of enterprises have adopted or plan to adopt DevOps. DevOps Movement 2017 Tricentis GmbH. All Rights Reserved. 4

Budgets won t increase remaining manual tests (exploratory and manual spec-based) 5-20% 60-80% Specification driven automated UI tests 20-40% GUI API automated API tests + service virtualization (key enabler for high automation) 80-95% Current State Target State Testing Future 2017 Tricentis GmbH. All Rights Reserved. 5

Budgets won t increase Optimize Computer-Aided Test Portfolio Design Redundancy-free test case portfolio with maximum risk coverage and fewer tests. Session-Based Exploratory Testing Make your testing intellectually rich. remaining manual tests (exploratory and manual spec-based) automated UI tests Explore Automate Model-Based Test Automation (MBTA) for UI and API Automating the automation framework. GUI API Current State automated API tests + service virtualization (key enabler for high automation) Target State Manage Integrate Orchestrated Service Virtualization Make testing independent. Stateful Test Data Management CT Integration (CI, CD) Distributed Execution Testing Future 2017 Tricentis GmbH. All Rights Reserved. 6

Budgets won t increase Optimize Computer-Aided Test Portfolio Design Redundancy-free test case portfolio with maximum risk coverage and fewer tests. remaining manual tests (exploratory and manual spec-based) automated UI tests GUI API automated API tests + service virtualization (key enabler for high automation) Current State Target State Testing Future 2017 Tricentis GmbH. All Rights Reserved. 7

Have the right test cases. Do the right things. Low Risk Critical Limit The time needed for testing is infinitely larger than the time available. 20% Medium Risk 80% Test Cases High Risk Business Risk Coverage Risk-Based Testing 2017 Tricentis GmbH. All Rights Reserved. 8

defect rate risk coverage [%] M N = 1 M 1 1 1 1 1 1 N weight 100% 80% 60% max 40% 20% 0% critical defects 0 0% 20% 40% 60% 80% test cases 100% Computer-Aided Test Portfolio Optimisation 2017 Tricentis GmbH. All Rights Reserved. 9

defect rate risk coverage M N w 1 w 2 w 3 w 4 w i w N weight w i w i+1 100% 80% 60% max 40% 20% 0% critical defects 0 0% 20% 40% 60% 80% test cases 100% Computer-Aided Test Portfolio Optimisation 2017 Tricentis GmbH. All Rights Reserved. 10

Test Cases 0% 20% 40% 60% 80% 100% Risk Coverage 100% Methodical Test Design 80% 60% Achieve maximum risk coverage with Intuitive a minimal number Test of Design powerful test cases. 40% 20% 0% Risk-Based Testing 2017 Tricentis GmbH. All Rights Reserved. 11

Budgets won t increase Optimize remaining manual tests (exploratory and manual spec-based) automated UI tests Automate Model-Based Test Automation (MBTA) for UI and API Automating the automation framework. GUI API automated API tests + service virtualization (key enabler for high automation) Current State Target State Testing Future 2017 Tricentis GmbH. All Rights Reserved. 12

Example: Commercial select invoice with the car registration number VIC-3456 Keyword Driven Data Driven Custom Hybrid Functionally Decomposed Great for developers who can code...but is even harder to understand for the business team Conclusion: Only for developer & Expensive to maintain Script Based Test Automation Frameworks 2017 Tricentis GmbH. All Rights Reserved. 14

+ Programming Required - No Scripting Record Replay Test Automation Framework Model-Based Test Automation 1 st Generation 2 nd Generation Next Generation 10% Automation Script-Based 30% Automation + 90% Automation - Business Readable + Future of Automated Testing 2017 Tricentis GmbH. All Rights Reserved. 15

Application Pages Technical Name Value Range Make Audi Engine Performance [kw] 200 Year of Construction 2016 Number of Seats 5 Fuel Petrol List Price [$] 35.000 License Plate Number B-CD 123 Usage Private Annual Mileage [mi] 10.000 Scan Application. Next >> Vehicle Data Make - Engine Performance [kw] - Year of Construction 2016;2015;2014;2013 Number of Seats 1;2;3;4;5;6;7;8;9 Fuel Petrol;Diesel;Gas;Other List Price [$] - License Plate Number - Usage Private;Commercial Annual Mileage - Next {CLICK};{RIGHTCLICK} Insurant Data Product Data Start Date 01/03/2016;01/04/2016 Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio Payment Option Yearly;Quarterly;Monthly Next {CLICK} Quote Details Price per Year($);Online Claim;.. Silver; Gold; Platinum; Ultimate sampleapp.tricentis.com Technical Information 2017 Tricentis GmbH. All Rights Reserved. 16

Logical Name Value Action Technical Name Value Range Create Vehicle Insurance Enter Vehicle Data Make Engine Performance [kw] Year of Construction Number of Seats Fuel List Price [$] License Plate Number Audi 200 {Year} 5 Petrol 35.000 B-CD 123 Usage Private Annual Mileage Next 10.000 {CLICK} Enter Insurant Data Enter Product Data Start Date 01/03/2016 Insurance Sum [$] 7Mio Payment Option Yearly Next {CLICK} Verify Quote Details Price List Select Price per Year($) Select Silver 1.535,22 Verify Business Information Vehicle Data Make - Engine Performance [kw] - Year of Construction 2015;2014;2013;2012 Number of Seats 1;2;3;4;5;6;7;8;9 Fuel Petrol;Diesel;Gas;Other List Price [$] - License Plate Number - Usage Private;Commercial Annual Mileage - Next {CLICK};{RIGHTCLICK} Insurant Data Product Data Start Date 01/03/2016;01/04/2016 Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio Payment Option Yearly;Quarterly;Monthly Next {CLICK} Quote Details Price per Year($);Online Claim;.. It s like playing Lego. Silver; Gold; Platinum; Ultimate Technical Information 2017 Tricentis GmbH. All Rights Reserved. 17

Efficiency Gain Sprint Start Time Sprint End 100% +4x Creation +6x Maintenance +20x Execution API Start testing early. Shift left test automation. UI Future test automation must focus on API testing. Degree of Completion 0% API Testing 2017 Tricentis GmbH. All Rights Reserved. 18

Logical Name Create Vehicle Insurance UI Test Case Value Action Browser API Test Case Webservice Enter Vehicle Data Make Audi Enter Vehicle Data Make Audi Engine Performance [kw] 200 Engine Performance [kw] 200 Year of Construction {Year} Year of Construction {Year} Number of Seats 5 Number of Seats 5 Fuel Petrol Fuel Petrol List Price [$] 35.000 List Price [$] 35.000 License Plate Number B-CD 123 License Plate Number B-CD 123 Usage Private Usage Private Annual Mileage 10.000 Annual Mileage 10.000 Next {CLICK} Enter Insurant Data Enter Insurant Data Enter Product Data Start Date 01/03/2016 Enter Product Data Start Date 01/03/2016 Insurance Sum [$] 7Mio Insurance Sum [$] 7Mio Payment Option Yearly Payment Option Yearly Next {CLICK} Verify Quote Details Verify Quote Details Price List Insurance Sum Gross Premium [$] 1.535,22 Select Select Verify Price List Insurance Sum Gross Premium [$] 1.535,22 Select Select Verify Business Information Business Information 2017 Tricentis GmbH. All Rights Reserved. 19

Logical Name UI Test Case Value Create Vehicle Insurance Enter Vehicle Data Make Audi Engine Performance [kw] 200 Year of Construction {Year} Number of Seats 5 Fuel Petrol List Price [$] 35.000 License Plate Number B-CD 123 Usage Private Annual Mileage 10.000 Action Browser API Test Case Enter Vehicle Data Make Audi Engine Performance [kw] 200 Year of Construction {Year} Number of Seats 5 Fuel Petrol List Price [$] 35.000 License Plate Number B-CD 123 Usage Private Annual Mileage 10.000 Next {CLICK} Enter Insurant Data Enter Insurant Data Manual Testers become Automation Specialists Enter Product Data Enter Product Data Start Date 01/03/2016 Start Date 01/03/2016 Insurance Sum [$] 7Mio Insurance Sum [$] 7Mio Payment Option Yearly Payment Option Yearly Next {CLICK} Verify Quote Details Verify Quote Details Price List Select Price List Insurance Sum Select Insurance Sum Gross Premium [$] 1.535,22 Verify Gross Premium [$] 1.535,22 Webservice Select Select Verify Business Information Business Information 2017 Tricentis GmbH. All Rights Reserved. 20

Behaviour-Driven Development (BDD) is a software development process based on Test-Driven Development (TDD) BDD gives a clearer understanding as to what the system should do from the perspective of the developer and the customer. TDD only gives an understanding of what the system should do from a developer perspective. BDD combines techniques and principles of TDD Domain-Driven Design Object-Oriented Analysis & Design TDD 2017 Tricentis vs GmbH. BDD. All Rights Reserved. 21

Open Source (Cucumber, JBehave, Specfolw with Selenium, Watir, etc.) Requires coding! Each GIVEN, WHEN, THEN, AND in the BDD scenario (left) needs a matching function call (right). No support for systematic scenario identification and definition BDD A Working Example. 2017 Tricentis GmbH. All Rights Reserved. 23

BDD 2017 Tricentis using GmbH. All Rights TCD. Reserved. 24

Budgets won t increase Optimize remaining manual tests (exploratory and manual spec-based) Explore Session-Based Exploratory Testing Make your testing intellectually rich. automated UI tests GUI API automated API tests + service virtualization (key enabler for high automation) Current State Target State Testing Future 2017 Tricentis GmbH. All Rights Reserved. 25

I m an automated test case. Scalability Issue Accessibility Issue Reliability Issue Usability Issue Understandability Issue Risks Performance Issue Coherence Issue Security Issue Functional Issue Convenience Issue Testability Issue Stability Issue Automation is doing, what automation does. Automated Testing 2017 Tricentis GmbH. All Rights Reserved. 27

Exploratory Branching The art of deciding on what to test next. New Testing Idea Exploratory Testing Perfect counterpart to specification-based automated & manual testing. Plan as you test! Story-Based Motivating Credible Test Cases vs. Scenarios Pre-specified inputs vs. hypothetical situation Exploratory Testing 2017 Tricentis GmbH. All Rights Reserved. 30

Exploratory Testing Analyze Potential Risks Problem vs. No Problem Mechanical Testing vs. Intelligent Testing Specification Based Testing Testing as artifact creation vs. testing as performance Monitor Known Risks Pass Result vs. Fail Result Main Purpose *That s an illustration. Don t confuse it with reality. 2017 Tricentis GmbH. All Rights Reserved. 31

Agile Testing Law Checked Efficient Automated Testing Exploratory Testing + Analyze Potential Risks Problem vs. No Problem Explored Effective Exploratory Testing Specification Based Testing = Tested Productive Testing R μν 1 2 Rg μν + Λg μν = 8πG c 4 T μν Monitor Known Risks Pass Result vs. Fail Result Agile Equation Of Motion 2017 Tricentis GmbH. All Rights Reserved. 32

Automation is a must. Manual testing is unjustifiable. Exploratory testing is a necessity. *We don t need humans doing something that a machine can do. What we do want the human testers doing is exploratory testing. Core 2017 Tricentis Conclusion GmbH. All Rights Reserved. 33

We are Passionate about Testing