Introduction Test Automation as a Service (TaaaS) Next Generation Testing: Innovations and Challenges Jonathon Lee Wright Director of Testing jlwright@iee.org @jonathon_wright www.taaas.net (blog) www.automation.org.uk automation.sharepoint.com www.linkedin.com/in/automation Test Automation as a Service (TaaaS) - Slide 1
Introduction Agenda Why the cloud? Automation in the cloud; New opportunities and benefits; New risks; What we are doing now; The future; Live demo. Test Automation as a Service (TaaaS) - Slide 2
Cloud Services Why the cloud? The cloud provides no up-front investment, featuring instant scalability, flexibility and availability; Pay as you use (PAYU), no more expensive vendor licencing costs or un-utilised maintenance agreements; Complex test environments on demand: built to expect & withstand failure ; Established technology avoiding support issues with noncommercial products; Avoid the minefield of complexity requiring internal specialist staff skills following industry best practice; 100% web delivered providing collaboration and mobility. Test Automation as a Service (TaaaS) - Slide 3
New Opportunities New Testing Opportunities Cross Platform and Compatibility - browser, operating system, mobile devices, script runtimes, languages, HTML versions; Market Configuration Testing 10+ Applications (Shop / ShelfReview / Nominations), 10+ Clients (Tesco / Macy s), 10+ Workstreams (Tesco UK / Tesco France), 10+ Languages (en-gb / fr-ca), 10+ Platforms (XP-SP1, SP2 & SP3), 10+ Browsers (I.E 7, 8, 9), 10+ Runtimes (Flash 9.0.44 ~ 10.3.181). Accessibility/Usability/UXD/WAI - due diligence in guaranteeing accessibility to people with disabilities; 88% of the delegate organisations here today failed W3C compliance tests; Globalisation realistic performance & load testing plus compliance with the commercial and civil law of the country; Search Engine Optimisation periodic ranking checks; Penetration, Performance & Load Testing. Test Automation as a Service (TaaaS) - Slide 4
New Risks New Risks (in the cloud) Commercial Ecosystems Fulfilment Systems Test Automation as a Service (TaaaS) - Slide 5 What happens when these Services FAIL? Systemic Failure? (e.g. DNS or Network link) Alternative service solution provider? Unique Services? (such as GPS)
Cloud Automation Taking Automation to the Cloud Customer Systems The Cloud (System Infrastructure) Intranet Site Desktop Application Web Site Web services Virtual lab Persistence Test Suite CMMI/ISO 9000 Rule Store Software Build Process Customer Automation Suite Automation Lifecycle Management (Test Automation as a Service Model) Business Level Keyword Definition Dashboard Test Asset Management Customer Take on Configuration Customer support Test Data Flexible Cost Planning Billing Monitoring/ Operational management Development Support Credentials United Reporting Reporting Maintenance Deployment Test Automation as a Service (TaaaS) - Slide 6
Cloud Automation Why this approach? Approach BPM BPT Keyword Data Modularity Tools Traditional Vendor Tool (Market Leader) Hybrid.NET Development (Visual Studio 2010 + Coded UI APIs) - Limited Support - Full Support Traditional (Vendor Tool) Hybrid (.NET Development) Advantages Constraints Advantages Constraints Vendor Support (maintenance subscription) Object Repository (object-based Record / Replay) Single Technology Stack (SPOF) Tool Vendor Dependency C# Project incompatible with Visual Studio Technology Blocks (Custom Flex Objects/Classes) Multiple Technology Stack (VS2010 / C#.net / Silverlight / Azure / SQL / WCF / CodedUI / Ranorex API / XML / XPATH / TFS) Integration with development (Visual Studio 2010) Integration with deployment (TeamCity) In-house framework development & maintenance effort. Additional skills / knowledge resources Object Repository (limited Record / Replay support) Limited non-technical tester interface Integration with testing (Microsoft Test Manager) Test Automation as a Service (TaaaS) - Slide 7
Login.Process Actions Object Handler Event Handler Cloud Automation What is different with this approach? Design tests before release first day automation testing Create complex test scenarios before the application has even been written; Buy.Tickets B1 B2 C2 C3 B3 B4 B5 A1 Login.Process C1 Check.Tickets C5 Logout.Process BPM D1 C4 D2 My.Account D3 D4 D5 The test language is self-validating and human readable The verbs and nouns in the domain specific language are written in natural language which uses context sensitive validation; Action Object Event Generic Reusable Libraries Verbs Enter Username Text Enter Password Text Nouns Press Login Button Example Business Level Keyword Test Automation as a Service (TaaaS) - Slide 8 Press Login Button
Cloud Automation What is different with this approach? The test definition is not implementation specific Test execution is performed by cloud based technology adapters which act as interpreters for the vendor agnostic and implementation agnostic language; Test Run Generate Environment Under Test (EUT) Select Platform Adapter Select Test Adapter Select Technology Adapter Application Server Database Server End-Client Machine Windows 2008 RC1 Windows 2003 RC2 Windows 7 SP1 Load Testing Performance Functional Coded-UI (Win32) Selenium (ASP.net) Ranorex (Flex 4.0) The definition of a test artefacts is self-maintaining Asset loader uses fuzzy logic matching to identify UI objects; Please enter your password to logon Enter password here logon logoff 1. XPATH Check 2. Regular Expressions 3. Descriptive Programming 4. Fuzzy Logic 5. n/a Artificial Intelligence Please enter your password to login Enter password here login logoff Test Automation as a Service (TaaaS) - Slide 9
Active Passive Cloud Automation Event Based / BDD Testing Automated testing based on trigger events (passive & active): Third party system events such as user story or defect status workflow change; Continuous build and integration or source code gated check-in. Test Automation as a Service (TaaaS) - Slide 10
Cloud Automation Where are we now? Application Lifecycle Management Unified Reporting Model Test Management Tool Team Foundation Server Reporting Dashboard Database Automation Automation Portal (TaaaS) Lab Management Windows Service ObjectHandler.dll EventHandler.dll Development (.NET 4.0) DataHandler.dll Application Programming Interface Application Under Test Test Automation as a Service (TaaaS) - Slide 11 2003/2007/2010
Cloud Automation How it works (in the cloud) End Users Test Automation as a Service (TaaaS) «uses» Run a test TaaaS Portal Cloud Web Endpoint «uses» «uses» Schedule a future test Silverlight Dashboard Test Scheduler Service QA Person View real time test status Dashboard Service Results Service «uses» «uses» Test Assets Repository (MTM / Quality Centre) Asset Scanner Service Business User View historic test results TD-API Adapter Test Runner Environment - A Virtual Machine created by the Framework VM Dispenser Check Development Progress Unified Reporting (Dashboard) TestRunner Test Management Silverlight Dashboard Framework Results Adapter Technology Adapter Platform Adapter Test Data Set Test Adapter View Live Defect Build Automation (Team City / TFS) Client System Test Repository Developer Build Client System Framework Object Adapter Business Level Keywords (BPM/BPT) Object Repository Test Data Sets Results Test Automation as a Service (TaaaS) - Slide 12
Cloud Automation Benefits to the Customer Customer Assets Domain Experts On Demand TaaaS Customer Defined Tests Business Process Model Source code No capital Investment Regression Testing Penetration Testing Software Build Process Test System Test Assets Cost Controlled Performance Testing Accessibility Standards Market Compliance Public facing Web sites Public facing Web Services Use Existing Test Assets Load Testing ISO/CMMI Progress Contribution Desktop applications Intranet Minimal Time Investment Code Standards Search Engine Optimisation Test Automation as a Service (TaaaS) - Slide 13
Cloud Automation Keep it Simple Automation User stories represented by a number of Business Level Keywords Test Automation as a Service (TaaaS) - Slide 14 Save Test (Selectable in the future from the Your Saved Tests List Box) Start Test (As per the custom list of business level keywords)
Cloud Automation TaaaS vs. DIY TaaaS Test Automation as a Service (TaaaS) - Slide 15
The Future Future forecast "Cloudy" 1. Avoid the Hype: Ensure that the critical business flow can be automated followed by complex custom components "avoid automating the simple components first e.g. simple login pages; 2. Don't be tempted: By third generation automation vendors offering so-called "script-less" tools. Be realistic, you need to be able support custom classes and the latest technologies without having to wait on the possibility that it may be supported in the next release; 3. Integration into testing process: Automation is no longer an "us and them" scenario between manual and automated tests; 4. Make the necessary plans: How easy is it to start using the service and how easy is it to move away from the service. Test Automation as a Service (TaaaS) - Slide 16
The Future Something to think about The best aspects of these proven test approaches demonstrate how they have evolved over the past decade, and this echoes some of the changes towards more lean and agile business methodologies. They are in a constant state of evolution just as the underpinning technology evolves over time. my section in Experiences in Test Automation book Dorothy Graham & Mark Fewster 2011 Test Automation as a Service (TaaaS) - Slide 17
Cloud Automation Automation - Demo Test Automation as a Service (TaaaS) - Slide 18