Ninja or Samurai? The Art of War and the Future of Testing Wolfgang Platz, Founder & CPO.
awards founded 2007 locations Austria Germany Switzerland United Kingdom Netherlands USA Australia New Zealand Singapore customers 330+ large enterprise customers Gartner MQ Leader Forrester Wave Strong Performer support TSIA 2014 Staff Excellence 97% Renewal Rates
Dec 2015 Aug 2014 It should be considered by enterprises that have struggled to make Jan 2011 test automation work, and by those seeking to support agile continuous automation practices. 3
The Digital Disruption Businesses must continuously exploit digital technologies to both create new sources of customer value and increase operational agility in service of customers. Forrester, March 2014, The Future Of Business Is Digital
7.4 5.5 4.5 Billion Billion Billion People in the world. People own a mobile phone. People have regular access to a toilet.
World s largest taxi company. World s largest accommodation provider. Their value is all digital. Owns no vehicles. Everyone can be a taxi driver. Owns no real estate. Everyone can be a landlord. Digitalization creates new ways of business operations.
Listening to horses talking in the early 1900s I m worried. Will all these new mechanical muscles make horses unnecessary? No worries. It will make our lives easier. Even if this car thingy takes off, there will be new jobs for horses we can't imagine. Humans need not apply, CPG Grey, 13.08.2014, https://www.youtube.com/watch?v=7pq-s557xqu 7
Hostes [k] Population [k] US, human population vs. number of horses 50.000 45.000 40.000 35.000 250.000 200.000 Since 1915, the number of horses has fallen by 85%. 30.000 150.000 25.000 20.000 100.000 15.000 10.000 50.000 5.000 0 1860s 1880s 1900s 1915 1940s 1960s 1980s 0 Horses [k] Population[k]
As business technology organizations increasingly look to Agile and DevOps, offshore providers' traditional manual testing services model no longer meets the needs of application development and delivery (AD&D) teams. Forrester
1 2 3 Crowd & Exploratory Testing and even Test in Production (TiP) Less about test but more about try Automate - Whenever you can, use API test access. Get Industry Smart Acquire domain knowledge
Change Agile takes it all + - Mode 1 traditional Less Agile Mode 2 Agile More Systems of Innovation Systems of Record - Governance +
Agile Dev meets Operations Dev Ops DevOps is all about Continuous Testing! http://devops.com/2015/02/02/devops-continuous-testing/
Global Software Testing Services Market 32 bnusd 2015 global spend on testing services, 60+% with Indian GSIs 10% CAGR expected growth until 2021* * has been reduced by 5% last year
or will it look more like this? - 50% Traditional manual testing will be down by half. 5x Test automation + Service Virtualization will be 5x. 3x Exploratory + Crowd Testing will be 3x. Sources: Tricentis 14
Today 80% Manual Testing The reliance on manual testing is the top technical challenge in app development. World Quality Report, 2015/16 20% Automated Testing UI API Test automation requires developers.
Is Manual Testing Dying?, Zeshan Ahmad, 30.05.2016, https://www.linkedin.com/pulse/manual-testing-dying-zeshan-ahmad Samurai It won t die, but it will no longer be Samurais doing it. Ninja Ninjas will take over. 16
Ninja Samurai What: scout/explore new territories Find most severe bugs fast How: creative, flexible, highly adaptive to change Self-managed and -structured What: conquer the territory thoroughly Comprehensive coverage Compliance How: efficient, effective Part of a bigger team
Today Future Exploratory Tests Automated UI Tests Ninja 80% Manual Testing API Tests +85% 20% UI Orchestrated Service Virtualization Automated Testing - key enabler for high automation rates - API Samurai 2021
1 Crowd & Exploratory Testing and even Test in Production (TiP) Less about test but more about try
I m an automated test case. Scalability Issue Accessibility Issue Reliability Issue Usability Issue Understandability Issue Risks Performance Issue Coherence Issue Security Issue Functional Issue Stability Issue Convenience Issue Testability Issue
1 2 3 4 5 Risks
Exploratory Branching New testing ideas continually occur during exploratory testing. New Testing Idea Exploratory Testing Perfect counterpart to specification-based automated & manual testing. Risks Test Cases vs. Scenarios Pre-specified inputs vs. hypothetical situation
Exploratory Testing The agile future of manual testing Specification Based Manual testing is unjustifiable. Exploratory Testing is a necessity. Exploratory Testing does not replace spec-based testing since it cannot provide comprehensive risk coverage. http://www.tricentis.com/tosca-community-edition/
The Age of the Customer Delivery Cycle Time 16 Months [M] 10 M 1960s Mainframe 1980s Client Server 6 M here they are 1990s The Web Sys of Record Sys of Engagement 3 M 1 M Week Day Sec 6 x where they need to go here they are 10+ x where they need to go 2010 The Age of the Customer t
Amazon deploys every 11.6 seconds. Velocity 2011, May 2011: Jon Jenkins, "Velocity Culture"
How nature does it In a hyper dynamic environment, nature keeps trying a great number of options at a high pace until it gets to a fit. Modern consumer apps (e.g. retail) use this principle, trying things out in production. Activated T-cell, fishing for viruses 26
Risk = probability x potential damage Very large number of users => very high Atomic damage is not highly business critical => medium/low Systems of Record Limited number of users => low Business critical => very high
Risk = probability x potential damage very high medium/low Incremental user-deployment => reduce risk TiP Test in Production Smaller functional packages => reduce risk Systems of Record low Smaller functional packages => reduce risk very high You have to test! There s no way around it
2 Automate Whenever you can, use API test access. Don t waste time on framework programming. #1 complaint of customers: Our desire for automation cannot be sufficiently resolved by GSIs today. Magic Quadrant for Application Testing Services, Gartner
Efficiency of Test Access Task UI test automation API testing Factor Set-up 100% 25% 4x Maintenance 100% 16% 6x Runtime 100% <1% 100+ x Timing regressive progressive Future Test Automation will focus on API testing.
Today, test automation is a developers discipline.
Logical Name Create Vehicle Insurance Enter Vehicle Data Make Engine Performance [kw] Year of Construction Number of Seats Fuel List Price [$] License Plate Number Usage Annual Mileage Next Enter Insurant Data Enter Product Data Start Date Insurance Sum [$] Value Audi 200 {Year} 5 Petrol 35.000 B-CD 123 Private 10.000 {CLICK} 01/03/2016 7Mio Payment Option Yearly Next {CLICK} Verify Quote Details Price List Insurance Sum Gross Premium [$] 1.535,22 Business Information Model-Based Test Action Browser Easy to read. No programming skills. Test Easy maintenance for UI & API tests. Test Maximum reuse across technologies. Test Robust against changes. Select Select Verify 1 2 3 4 Script-Based Code Synchronization Behavior Code Recovery Handling Code Data Driven Framework Code Keyword Driven Framework Browser Assert.AreEqual("1.535,22", driver.findelement(by.xpath("//table[@id='pricelist']/t body/tr[10]/td[2]/b")).getattribute("value")); Do you see the business information?
Logical Name UI Test Case Value Create Vehicle Insurance Enter Vehicle Data Make Audi Engine Performance [kw] 200 Year of Construction {Year} Number of Seats 5 Fuel Petrol List Price [$] 35.000 License Plate Number B-CD 123 Usage Private Annual Mileage 10.000 Action Browser Logical Name API Test Case Create Vehicle Insurance Enter Vehicle Data Make Engine Performance [kw] Year of Construction Number of Seats Fuel List Price [$] License Plate Number Usage Annual Mileage Value Action Webservice Audi 200 {Year} 5 Petrol 35.000 B-CD 123 Private 10.000 Next {CLICK} Enter Insurant Data Enter Insurant Data Manual Testers become Automation Specialists Enter Product Data Enter Product Data Start Date 01/03/2016 Start Date 01/03/2016 Insurance Sum [$] 7Mio Insurance Sum [$] 7Mio Payment Option Yearly Payment Option Yearly Next {CLICK} Verify Quote Details Verify Quote Details Price List Select Price List Insurance Sum Select Insurance Sum Gross Premium [$] 1.535,22 Verify Gross Premium [$] 1.535,22 Business Information Business Information
A Tester s Day in an Agile Team 09:00 09:30 Ramping up Engineering (some start early) Stand up meeting (15 mins) PO Developers Testers Start of the new Agile Day Daily Business 12:00 Write/Track progress User Stories Create & weight Requirements Write & Review Code Support Test & POs / PMs Provide 3rd Level Support 20% API 25% UI 17:30 Phasing out Engineering (some work long) * including test case maintenance End of the Agile Day
3 Get Industry Smart Acquire domain knowledge #2 complaint of customers: GSIs lack domain knowledge. Gartner
IT Budgets Packaged is on the Rise 2.5% CAGR 2x Growth of Sys of Engagement over Sys of Record 2021 Packaged 15% CAGR Systems of Engagement Systems of Record 36
Acquisition of Domain Knowledge Industry knowledge is increasingly required as a knowledge of Packaged Apps.
1 2 3 Crowd & Exploratory Testing and even Test in Production (TiP) Less about test but more about try Automate - Whenever you can, use API test access. Get Industry Smart Acquire domain knowledge
Thank you for your attention Stay in touch! Host: Wolfgang Platz Email: w.platz@tricentis.comm Follow us on Twitter: @tricentis 39