Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland 22 May 2015 Disruptive Technology 2 1
Disruptive Technology 3 Disruptive Technology 4 2
Disruptive Technology 5 CLOUD SOCIAL DISRUPTIVE FORCES MOBILE BIG DATA 3
Biggest Disruptors The Last 5 Years? Agile Offshoring Virtualisation Disruptive Forces DevOps Customer Testing 8 Big Data 4
Startup Corporate 10 You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new. Steve Jobs 5
12 14 Scientific Method for Customer Experiments to Guide Software Development 6
FAIL FAST 15 16 7
3 6 Months 17 1 2 weeks 18 8
You can't just ask customers what they SPEED want and then try to give that to them. By the time you get it built, they'll want something new. is the new black Steve Jobs 19 20 9
21 DEV OPS 10
No Silos 24 CI CD Continuous Integration Continuous Deployment CD 25 Continuous Delivery 11
Rapid Releases Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks Source: http://readwrite.com/2014/01/01/three-reasons-your-startup-needs-devops-or-else 12
Source Control Automated Deployment System Build System Test infrastructure CD Platform Pipeline Management & Visualization Automated Configuration Management (IaC) Artifact/ Package Repository Source: Continuous Delivery at Scale John Esser, Director Engineering Productivity, Ancestry.com ALM Forum Seattle, 2014 Version & Configuration Management Git Flow 13
Version Everything Test Data Static Data Application Code Infrastructure 30 Data Versioning Infrastructure/Platform Versioning 31 14
New Strategies Cloud Computing Duplicate test environments Copy of production Grid based test automation Blue/Green Deployments Resilience testing 32 33 15
Elastic Computing Time Which is better? 34 Release Pipeline Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks 16
Testing Pyramid Right Way SYSTEM Slower UI focused Later in the processes INTEGRATION Focused on linkages Harnesses, Mocks and Stubs UNIT Fast Isolated Aligned with Code 36 Release Pipeline Source: Steve Neely, sneely@rallydev.com @stenvoon http://bit.ly/cdpaper ALM Forum Seattle, 2014 17
Deployment Gating Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks Production Testing Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks 18
Failure Monitoring 40 Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks FAIL FAST 41 19
FAIL FAST MTBF 42 FAIL FAST MTTR 43 20
Customer Testing 45 Testing In The Wild Applause Secured $80m in VC funding 21
Elastic Resourcing Time Which is better? 47 Customer Testing 48 22
A / B Tests E.g. Google 49 Analytics Content Experiments Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks Feature Toggles 50 23
Feature Toggles 51 Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks Feature Toggles Progressive Delivery To Minimise Risk 52 Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks 24
Early Customer Testing 53 Can We Skip Some Pre-Prod Testing? BIG DATA Volume Velocity Variety 55 25
How To Test Prediction 56 Big Data Challenges Staging test data sets Test data privacy Variation of test data sets Production data variability Controllability & Observability Algorithm Evaluation Heuristic Testing Exploratory Testing 57 A/B Testing Customer Sampling & Monitoring 26
New QA Skills Technical Influence Test Environment Provisioning CI / CD Process Experimentation Automation Data Analytics Feature Architecture & Deployment Planning Production Monitoring Canary Releasing 27
Blue Green Deployments Feature Toggles 28
Feature Flippers A / B Tests E.g. Google 64 Analytics Content Experiments Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks 29
Dark Launch 30
68 In 5 Years??? 31
Skill Changes 69 DevOps Teams Embedded, Self Organising, Functional Business Tester Automated Tester DevOps Tester Non-Technical Functional Tester 70 32
Reduced Test Management Coordinate Resourcing Practice Support Consultants Coaches Governance Test Managers Test Leads 71 Crowd Test Coordination Customer Test Coordination Feature Deployment Planning User Experience and Failure Monitoring 72 33
Q&A 73 34