Continuous Testing at Scale

Size: px

Start display at page:

Download "Continuous Testing at Scale"

Valentine Stone
5 years ago
Views:

1 Continuous Testing at Scale TAPOST Conference October 12th 2016, Riga

2 @buzdin Dmitry Buzdin

3 Introduction to Continuous Testing

4 Continuous Delivery Get changes to production in fast and efficient way Deployment happens often and can be performed on demand Code is always in a deployable state

5 Why CD? Frequent release cycle Decreased time to market Decreased delivery risks Commercial product development

6 Cost of Defect Fixing This chart is slightly different in CD Testing is performed as frequently as possible

7 Continuous Delivery Pipeline Every code change needs to be verified Pipeline breaks the delivery process into stages Pipeline stages are not standardised

8 Pipeline Example Commit Build & Unit Test Change Code Review Confidence Deploy to Test Run Tests Deploy to Stage Deploy to Prod

9 What is in Test Phase? Simplest scenario - Write tests in your favoured test framework and execute!

10 Agile Testing Strategy Developers write some integration tests and execute them in the pipeline

11 Have Devs Considered? environment live checks smoke testing functional testing security testing performance testing fault-tolerance testing browser compatibility testing mobile testing test environment preparation test planning test scheduling test reporting testing process traceability

12 Tests Pyramid

13 Another Pyramid Is the right thing built? Is it build right?

14 What tend to happen is Development focused CD

15 Development focused CD sprint 1 sprint 2 sprint 3 Version 1 Version 2 testing 1 Manual Testing testing 2

16 Testers are lagging behind a continuous delivery development train

17 Continuous Testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. OPS Shippable feature DEV TEST

18 Developers Test management Test plans Manual testing Non-functional testing Structured approach Standards compliancy How do you align? Apply TDD No bureaucracy All automated tests Ad-hoc decisions Feature-level testing Testers

19 Is Continuous Testing easy? Not at all! Some advices to follow

20 Story Begins

21 Setting the Context Telecom company had a network management system developed for the last 20 years Time to do a complete rewrite for 4G/5G Continuous Delivery model selected

22 100+ Scrum teams Git Repositories Components Functional Tests Everything Automated 200+ Test Environments

23 Expectations Fast feedback cycle System quality statement few times a day Testing compliant to quality standards Traceability throughout the process Agile development model

24 Traceability is the ability to verify the history, location, or application of an item by means of documented recorded identification. Boring bureaucratic stuff for non-agile environments only, right?

25 JIRA ID Produces ARTIFACT ID Requirement BUILD ID Artefact Build Relates to Covers Executes Test Code SHA ID??? Describes Change Test TEST ID

26 Test ID in Code Every test should have a unique test id Test case related meta-data

27 Test Case Management Relates to a test description in TMS Test ID: TMS-1 Title: Test database connection Requirements: JIRA-124, JIRA-234 Priority: CRITICAL Components: DB Tags: Regression, DB Description:... Test ID: TMS-2 Title: Test login Requirements: JIRA-678 Priority: CRITICAL Components: Authorisation Tags: Regression, UI Description:...

28 Allure Framework Test Results (XML Documents) Run Aggregation Test Cases All popular test frameworks supported Interactive Report

29 Feature coverage by tests

30 Drill-down to test step level

31 Test Run Summary

32 Achievements Achieved a fully automated traceability in a single test run Possibility to extract and store test statistics Traceability is simple given right tools

33 CD Meets Microservices Service A Build Test Deploy Service B Build Test Deploy Service C Build Test Deploy x 100 Teams

34 Service A Service B Service C System Version

35 System Level Testing End 2 End Tests Build Test Deploy New package releases trigger E2E tests

36 Teams are producing testware packages Test Suites Produces Contains Team Testware Package Test Cases

37 Team A Testware 1 Test Schedule Testware packages are combined with help of test schedules Team B Testware 2 Test Executor Team C Testware 3

38 Test Schedule Team A Testware Smoke Testing Environment Preparation Team B Testware Environment Cleanup Consistency Check Team C Testware Schedule items are coming from different teams

39 Tests schedule produces build flow definition, which is given to Jenkins build farm parallel ({ guard { build( run-tests, testware1 ) build( run-tests, testware2 ) } rescue { build( run-tests, cleanup ) } }) Visualisation

40 Tests Test ware Level Reports Aggregation Network Drive Full Execution Report

41 Achievements Tests from multiple teams are combined to a single test run Aggregated reports for all test cases are available

42 How to run tests as quickly as possible to minimise the feedback loop?

43 Tests Quality Police Ensuring tests are efficient (API vs UI, no waits) can run in parallel data-driven reusing other teams test API

44 Distributed Test Executor Cloud based Scale-up/scale-down Health checks Parallelisation

45 MASTER SLAVE SLAVE SLAVE MASTER MASTER MASTER WORKER WORKER WORKER WORKER WORKER WORKER Test Lab

46 Testing Flow Component Component Tests Component Tests Tests Smoke Tests Isolated runs Environment up and running Core use-cases Quick Regression Important use-cases Long Regression Decision is taken if to continue pipeline after every step All use-cases Full Regression

47 Achievements Tests quality is measured and improved Tests execution time is minimised

48 Failed test!= Sometimes

49 Interpreting Results 100% green tests pipeline all the time - not realistic at this scale what is failed test priority? is it a new test? is it a flaky test? was it rerun several times?

50 Test Result Analytics Need something like this, but with more information

51 Internal Test Reporting Solution

52 Achievements Identifying good enough builds Ignoring random faults Historical test results for analysis

53 The Outcome Full testing traceability E2E pipeline automation Continuous testing

54 Lessons Learned

55 There are not too many tools out there to support large scale continuous testing

56 Instead of using monolithic tools - aim for extensible independent services and interoperability standards

57 CD Standards Some well-known standards and platforms: Jenkins - Pipelenes Docker Containers JUnit 5 - Test Runner WebDriver - Selenium HTTP API Allure Reporting - Common for all test tools

58 Next 5 years are going to be about CD and Testing tools interoperability

59 This slide was intentionally left blank

60 Copyright: nisanga / 123RF Stock Photo martinm303 / 123RF Stock Photo dundanim / 123RF Stock Photo enjoyphoto / 123RF Stock Photo kzwwsko / 123RF Stock Photo pedrosala / 123RF Stock Photo saranya2908 / 123RF Stock Photo

How Can Testing Teams Play a Key Role in DevOps Adoption?

June 3, 2016 How Can Testing Teams Play a Key Role in DevOps Adoption? Sujay Honnamane QA Director @sujayh Rameshkumar Bar Sr. Automation Architect @rameshbar 2016 Cognizant Session take away DevOps Overview