The Future of Testing: Continuous Enterprise Testing

Size: px

Start display at page:

Download "The Future of Testing: Continuous Enterprise Testing"

Abner Colin Peters
5 years ago
Views:

1 The Future of Testing: Continuous Enterprise Testing ANZTB Test Conference, Canberra, 1 June 2018 Thomas Hadorn.

2 Dev Ops

3 Years Months Months Weeks Delivery Cycle Time Weeks Days Perceived Disruption Software Quality Challenge Days Hours Hours Minutes Technical Complexity & Rate of Change Today Critical Physical Limit

4 6+ Big Bang Waterfall Requirements Design Implementation Testing Acceptance Deployment 4 Incremental Rational Unified Process 2 Agile Scrum, Kanban Agile Event Horizon Inception Elaboration Construction Transition Development Operations Rigid Rules Flexible Framework 1 Continuous DevOps Digital Singularity DevOps = wall of confusion

5 Present Manual testing 80% The reliance on manual testing is the top technical challenge in app development World Quality Report, 2015/16 Automated testing 20% UI Load Test automation requires developers AP I 5

6 Present Future Optimize Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Automated testing 20% Manual testing 80%

7 Present Future Manual testing 80% remaining manual tests manual spec-driven and exploratory Optimize Explore Progression testing Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Test the unknown. Session-Based Exploratory Testing t Automated testing 20%

8 Present Future Optimize Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Manual testing Automated testing 80% 20% remaining manual tests manual spec-driven and exploratory Automated testing +85% Explore Progression testing Regression testing Test the unknown. Session-Based Exploratory Testing t t

9 Present Future remaining manual tests manual spec-driven and exploratory Optimize Explore Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Test the unknown. Session-Based Exploratory Testing 80% automated UI tests automated load tests UI and API +85% automated API tests 20% Automate Automating the automation framework for UI and API. Model-based Test Automation (MBTA)

10 Present Future remaining manual tests manual spec-driven and exploratory Optimize Explore Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Test the unknown. Session-Based Exploratory Testing 80% 20% automated UI tests automated load tests UI and API automated API tests + Virtualize data and systems key enabler for high automation +85% Automate Manage Integrate Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Increase the testing window. Test-Driven Service Virtualization Provide matching test data in time. Stateful Test Data Management Automate build to release process. Continuous Integration/Testing/Delivery (CI, CT, CD) Distributed Execution

11 [%] M/N = M N weight 100% risk coverage 80% 60% 40% 20% max defect rate 0% critical defects 0 0% 20% 40% 60% 80% test cases 100%

12 M N w 1 w 2 w 3 w 4 w i w N weight w i w i+1 100% risk coverage 80% 60% 40% 20% max defect rate 0% critical defects 0 0% 20% 40% 60% 80% test cases 100%

13 Test Cases 0% 20% 40% 60% 80% 100% Risk Coverage 100% Methodical Test Design 80% 60% Intuitive Test Design 40% 20% 0% Achieve maximum risk coverage with a minimal number of powerful test cases.

All Combinations Negative Tests Boundary Tests Private provide insurant

14 Get a Quote choose type of vehicle Truck Commercial Car provide vehicle details Bike Auto generate Orthogonal Arrays Linear Expansion Pairwise Tests All Combinations Negative Tests Boundary Tests Private provide insurant details choose insurance cover display premium Purchase no Purchase Purchase - yes yes

15 Present Future remaining manual tests manual spec-driven and exploratory Explore Test the unknown. Session-Based Exploratory Testing 80% automated UI tests automated load tests UI and API automated API tests 20% +85%

16 Testing Continuum Ad-hoc Testing Exploratory Testing Manual Testing Automated Testing Informal Testing Formal Testing

17 Scalability Issue Accessibility Issue Reliability Issue Usability Issue Understandability Issue Risks Performance Issue Coherence Issue Security Issue Functional Issue Stability Issue Convenience Issue Testability Issue

18 I m an automated test case. Scalability Issue Accessibility Issue Reliability Issue Usability Issue Understandability Issue Risks Performance Issue Coherence Issue Security Issue Functional Issue Convenience Issue Testability Issue Stability Issue Automation is doing, what automation does.

19 Risks

20 I m the same test case executed manually. I m an automated test case.

21 Exploratory Branching The art of deciding on what to test next. New Testing Idea Plan as you test Story-Based Motivating Credible Test Cases vs. Scenarios Pre-specified inputs vs. hypothetical situation

22 Exploratory Testing Analyze Potential Risks Problem vs. No Problem Specification Based Testing Monitor Known Risks Pass Result vs. Fail Result *That s an illustration. Don t confuse it with reality.

23 Exploratory Testing «Testing as performance» Formal Testing «Testing as artifact creation» Intelligent Testing Create new test ideas based on what you have learned Analyze Potential Risks Focus on the things you don t know High Information Value Learn something new Low Risk Coverage Hard to scale continuously because it relies on humans Mechanical Testing Process pre-defined data in pre-designed steps Monitor Known Risks Confirm what you already know (measurable things) Low Information Value Repeat what you have already learned High Risk Coverage Easy to scale because it s parallelizable Problem Detector Change Detector

24 Present Future remaining manual tests manual spec-driven and exploratory 80% automated UI tests automated load tests UI and API +85% Automate Automating the automation framework for UI and API. Model-based Test Automation (MBTA) automated API tests 20%

25 Efficiency Gain Sprint Start Time Sprint End 100% +4x Creation +6x Maintenance API Start testing early, shift left test automation Degree of Completion UI +20x Execution 0%

26 Record Replay Test Automation Framework (TAF) Model-Based Test Automation (MBTA) Script-Based B Business Information 1 T Technical Steering Information n B Script-Based (Coded TAF) T Test Data Keywords B Model-Based (automated TAF) T Automation Model Test Coverage Design (MBT) B and/or T change requires re-recording. Manually, coded framework. Automating the automation framework.

27 Systems of Engagement Systems of Record Customer Portals Mainframe Websites & CMS Web Server, API Gateway AppSrv/ESB COTS Social Legacy BI / Analytics Open-Source Model-Based

28 Effective Velocity Script-Based Open Source Script- Based Commercial Model-Based Typical Startup Typical Enterprise QA Staff Programmer Scripter Non Technical SME

29 Present Future remaining manual tests manual spec-driven and exploratory 80% automated UI tests automated load tests UI and API +85% Automate Automating the automation framework for UI and API. Model-based Test Automation (MBTA) automated API tests 20%

Before Cloud Cloud Age Memory Footprint 2011 60% web pages < 50

Testing Complexity 2018 60% web pages > 100 requests/page 80% web pages

30 Before Cloud Cloud Age Memory Footprint % web pages < 50 requests/page 80% web pages < 200KB JS/page Release cycle > weeks Testing Complexity % web pages > 100 requests/page 80% web pages > 200KB JS/page Release cycle < days, minutes, sec Browser-Level Browser-Level Users

31 Cloud economy of scale & hybrid GEOGRAPHIC REGIONS

32 Present Future remaining manual tests manual spec-driven and exploratory 80% automated UI tests automated load tests UI and API automated API tests +85% Manage Increase the testing window. Test-Driven Service Virtualization Provide matching test data in time. Stateful Test Data Management 20%

18 Average # of systems with unrestricted access 96 % of testers have restricted test

33 On average, organizations require access to 33 systems for development or testing. Testing Present Welcome to the tester s hell. 18 Average # of systems with unrestricted access 96 % of testers have restricted test lab access Enterprise system landscapes are alike disease gene networks. *voke, market snapshot report on service virtualization

34 Insurance App API (e.g. Web-Services) Business Logic API (e.g. Web-Services) Instable or not available applications Test Data search & preparation (expensive) 3 rd party or Cloud based services CRM CAR CRM RATING ORDER Decoupled, outsourced Development & coordinated Deployment, multiple Vendors

35 Insurance App API (e.g. Web-Services) Business Logic API (e.g. Web-Services) Virtualisation CRM CAR CRM RATING ORDER Virtualising systems replicates system complexity Virtualising components creates inconsistencies Virtualisation is not a fallback plan

36 Insurance App API (e.g. Web-Services) Business Logic API (e.g. Web-Services) Orchestrated Service Virtualisation CRM CAR CRM RATING ORDER

37 Model-based Test Automation Insurance App Virtualising for automated tests eliminates system complexity API (e.g. Web-Services) Business Logic API (e.g. Web-Services) Virtualising for automated tests solves inconsistencies Test-Driven (Orchestrated) Service Virtualisation CRM CAR CRM RATING ORDER Virtualisation is not a fallback plan, it s the first line of defence

Script-Based Model-Based 80 and 90 Millennia 10 Machine Learning 20

Test Design (MBT) Test-Driven Service Virtualisation Model-Based

Frameworks Selenium + Traditional Commercial Automation

38 Script-Based Model-Based 80 and 90 Millennia 10 Machine Learning 20 R I S K C O V E R A G E 80% 30% 10% Exploratory Testing Model-Based Test Design (MBT) Test-Driven Service Virtualisation Model-Based Cucumber Automation (MBTA) Keyword + Data Driven + Framework Frameworks Selenium + Traditional Commercial Automation Record/Replay Record/Replay Tool Record/Replay Systems of Engagement Systems of Record

39 Present Future remaining manual tests manual spec-driven and exploratory Optimize Explore Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Test the unknown. Session-Based Exploratory Testing 80% 20% automated UI tests automated load tests UI and API automated API tests + Virtualize data and systems key enabler for high automation +85% Automate Manage Integrate Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Increase the testing window. Test-Driven Service Virtualization Provide matching test data in time. Stateful Test Data Management Automate build to release process. Continuous Integration/Testing/Delivery (CI, CT, CD) Distributed Execution

40 Nov 2017 Dec 2015 Aug 2014 Jan 2011 It should be considered by enterprises that have struggled to make test automation work, and by those seeking to support agile continuous automation practices.

41 The Tricentis Tosca X-Factor Optimize Continuous Testing Platform Explore Automate Risk-Based Test Management Model-Based Test Coverage Design Session-Based Exploratory Testing Model-Based Test Automation Browser-Based Load Testing Test-Driven Service Virtualization End-to-End Functional & Load Testing Optimize 2X Risk Coverage 2-3X Release Speed 30-40% Cost Reduction 10X Compared to other testing tools. 41

42 APAC fast-growing customer footprint 42

43 CTMM Focus Area M1 M2 M3 M4 M5 Continuous Testing Maturity Model Most enterprise customers are on level M1 when they decide for Tosca. Explore Optimize Automate Manage Integrate Exploratory Testing Risk Based Prioritization Test Case Design UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization CT Integration (CI, CD) 43

44 ? Thomas Hadorn Managing Director APAC at Tricentis

Agile vs Fragile. Susmit Bhattacharya, Solution Architect, Asia Pacific. - The need for Automation in Agile Tricentis GmbH. All Rights Reserved.

Agile vs Fragile. Susmit Bhattacharya, Solution Architect, Asia Pacific. - The need for Automation in Agile Tricentis GmbH. All Rights Reserved. Agile vs Fragile - The need for Automation in Agile Susmit Bhattacharya, Solution Architect, Asia Pacific 2017 Tricentis GmbH. All Rights Reserved. Years Months Months Weeks Delivery Cycle Time Weeks Days