Shift Happens. The Role of Continuous Testing in DevOps. Alex Martins. CTO Continuous Testing. May 15 th, 2018 CA Technologies.

Transcription

1 Shift Happens The Role of Continuous Testing in DevOps Alex Martins CTO Continuous Testing May 15 th, 2018 CA Technologies February

2 Chris Petterson Account Executive SLED West, CA Technologies 14+ years in software delivery As a former customer, he developed and implemented a quality assurance strategy that was very similar to the concepts of DevOps and Continuous Delivery as we know them today With both a development and quality assurance background, he has seen firsthand the inefficiencies of the traditional software delivery lifecycle (SDLC) He joined CA to help organizations modernize their SDLC and enable them to deliver software projects better, faster, and cheaper

3 WHY do we even talk about Continuous Testing? 3

4 Agile Helps You Move Faster 4x Agile improves success rate of projects over Waterfall 42% Improvement in overall speed-to-production with Agile 65% Reduction in IT costs when DevOps is added to Agile * 4

5 But Moving Fast is Not Enough Bad Software Costs Billions of Dollars 5

6 WHAT holds us back from Continuous Testing? 6

7 Question #1 63 HINT: Biggest bottleneck in DevOps 7

8 QA Is Still a Bottleneck Even in DevOps Shops A 2017 survey found that the majority of delays were occurring at the Test/QA stage of the cycle. Where are the main hold-ups in the software production process? 32% 16% 22% 63% 21% 23% 30% Plan code Pre-test Build Test/QA Release Deploy Review/Improve 29% of DevOps practitioners said that quality was being sacrificed 855b06567fe2/CTG-DevOps-Review-2017.pdf What tends to lose out most in a DevOps environment? Nothing 42% Security 13% Speed 16% Quality 29% 8

9 Question #2 70 HINT: Not automation 9

10 Challenges to Achieving Continuous Delivery & Testing IDEA Requirements 64% of total defect cost originate in the requirements analysis and design phase 1. Development 50% of developers time is spent fixing and finding defects 2? 80% of teams experience delays in development and QA due to unavailable dependencies 3? QA/Testing 70% of all testing is still manual 4! 63% of testers admit they can t test across all the different devices and OS versions 5 50% of time spent looking for test data 6 79% of teams face prohibitive restrictions, time limits or access fees on needed 3rd party services 3 X X X Release 57% are dissatisfied with the time it takes to deploy new features 7 Operations Greater than 50% production outages from configuration management!! Ave. cost of downtime per minute = $4,700 53% percent of visits are abandoned if a mobile site takes more than three seconds to load 1 Hyderabad Business School GITAM University Quality Flaws: Issues and Challenges in Software Development Study conducted by the Judge Business School at Cambridge University January Voke Market Snapshot Service Virtualization January Bloor Report Automated test case generation - September Perfecto Mobile 2014 Benchmark Survey, Why Mobile Apps Fail Forrester Global Modern Service Delivery Benchmark Online Survey - Q BlazeMeter Black Friday study,

11 TESTING REALITY TESTING GOAL TOOLSET TEST PYRAMID TOOLSET TEST PYRAMID UI Tools UI 5% UI Tools 94% API Tools API 45% UI API Tools API 3% Dev Tools Unit 50% Dev Tools Unit 3% *Actual customer example of their testing strategy 11

12 Before: Traditional SDLC Process Testing as a time-boxed event, in a linear process. Customer Ideas APIs PLANNING DEVELOPMENT TESTING OPERATIONS Security Build Business Case Portfolio & Project Prioritization Requirements Definition & Management Code Build Test Case Creation Virtual Service Test Data/Synthetic Data Functional & Performance Testing Service Management Production Applications Monitoring

13 Question #3 Test, document, customer validate, setup releases, stand ups, reviews, etc. HINT: All one person ANSWER: Tasks developers have to do in sprint 13

14 When Shift Happens Developer s tasks in sprint Code Test On the average How many hours in a week do developers actually code? 8 HOURS! Customer Validation Release Automation Videos Monitor Release 14

15 Question #4 Happens. HINT: ANSWER: Shift Happens (Or better yet, Shift Needs to Happen) 15

16 HOW do we achieve Continuous Testing? CA. Confidential. All rights reserved.

17 What Is Continuous Testing? The practice of testing across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected. 17

18 Continuous Feedback & Collaboration Customer Ideas Build Business Case Portfolio & Project Prioritization Planning Requirements Modeling & Management Test Case Creation Code Continuous Integration Development Build Virtual Service Test Data UI User Acceptance Testing API Continuous Delivery Performance Testing Continuous Testing Security Continuous Deployment Operations Service Management APIs Production Applications Security Monitoring Continuous Delivery Director Planning & Requirements Project & Portfolio Planning CA PPM Requirements Modeling Collaboration Development Source Code Repository Continuous Integration Continuous Testing Testing Efficiency Tools Service Virtualization Test Data Management UI Testing Continuous Deployment Deployment Tools Operations AXA APM CA ARD Requirements Management Artifact Repository API Testing Container Orchestration CA Agile Central Performance Testing Security Testing Environment Configuration 18

19 The Goal: Testing Embedded Throughout SDLC Customer Ideas Continuous Feedback & Collaboration APIs PLANNING DEVELOPMENT OPERATIONS Security Build Business Case Portfolio & Project Prioritization Requirements Definition & Management Code Build Service Management Production Applications Test Case Creation Test Data/Synthetic Data Virtual Service TESTING Functional & Performance Testing Monitoring Continuous Testing is the embedding of testing as a fundamental and ongoing aspect of every activity throughout the software delivery cycle

20 CA Solutions Enable Continuous Testing Customer Ideas Continuous Feedback & Collaboration APIs PLANNING DEVELOPMENT OPERATIONS Security Build Business Case Portfolio & Project Prioritization Requirements Definition & Management Code Build Service Management Production Applications Test Case Creation Test Data/Synthetic Data Virtual Service TESTING Functional & Performance Testing Monitoring Agile Requirements Designer Test Data Manager Service Virtualization BlazeMeter APM & UIM Continuous Delivery Director

21 End-to-End Integrated Continuous Testing from CA AUTOMATE THE AUTOMATION! Continuous Delivery Director 21

22 Thank You.