Testing in A Large scale agile Development Environment
Content/agenda 1. Introduction 2. Agile on a large scale 3. Testing in a large scale agile environment 4. Conclusion Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 2
Introduction Ericsson: The leading vendor for mobile network infrastructure Tough competition on the telecommunication market: Number of competitors Merging of traditional Circuit Switched Technology and Packet Switched Technology towards all IP Network Motivation: Increase the feedback loops towards design Delivery of high quality products to the customer Being flexible to adapt faster to the market s demand Early and Frequent Delivery Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 3
AGILE ON A LARGE SCALE Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 4
Agile Manifesto for Agile Software Development We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 5
ScRUM IN A NUTSHELL One team collaborating to one feature Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 8
large scale Chief Product Owner: 1 Number of Product owners: 5 Number of Scrum teams: 8 Size of the teams: 6-8 Number of Scrum Masters: 8 Number of people: around 70 Number of locations: 4 Number of Releases: 2 Number of manhours: ~100000 Number of User Stories: ~700 Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 9
LARGE SCALE Our legacy: The Ericsson Mobile Soft Switch 8085042 PLEX statements 20212606 ASA instructions corresponding to approximately 95000000 Intel instructions 2196 SW blocks Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 10
TESTING IN A LARGE SCALE AGILE ENVIRONMENT Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 11
Ways of working Cross functional team of 7 +/- 1 SW developers: Virtual teams Synchronized sprints inside a feature Common Backlog Common ceremonies to keep everybody informed Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 12
Ways of Working Backlog Setup strictly based on functional areas US1 US2 Application Transport US3 US4 Internet Network US5 Team A Team B Team C Team D Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 13
Ways of Working Backlog Setup strictly based on functional areas Risk of dependencies between User Stories Difficulties to test from an end to end perspective More planning needed Prone to generate delays US1 US2 US4 Application Transport Internet Network US3 US5 Team A Team B Team C Team D Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 14
Ways of Working Backlog Setup with end to end focus and functional areas Application US1 US2 Transport US3 US4 US5 Internet Network Team A Team B Team C Team D Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 15
Ways of Working Backlog Setup with end to end focus and functional areas Testing possible from end to end perspective as the User Story is concluded Less planning needed Reduce dependencies due to functional areas Coordination might increase Competence build through training on the job and mix of competences US2 US1 Application Transport Internet US5 US4 US3 Network Team A Team B Team C Team D Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 16
Ways of Working Setup of virtual teams Team A Team B Team C Team D Virtual Test Team LSV team System Architecture Team Scrum of Scrum Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 17
Ways of Working Setup of virtual teams Team A Team B Team C Team D Virtual Test Team LSV team System Architecture Team Scrum of Scrum Overview over the activities with higher priority Testing, Architecture, Software Delivery, Organizational topics Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 18
LLV track LLV track LLV track LLV track Test Overview Feature n Feature 3 Feature 2 Feature 1 LSV 1 2 3 4 5 6 7 8 N N+1 M Note: Simplified view, details on following slides Rebase mandatory Rebase optional LSV Delivery Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 19 Ericsson AB 2009 Ericsson Internal X (X) Date
Supporting the Team Testing activities Automated & Manual Business facing Manual Functional Tests Examples & Integration Story Tests Function Test Tests Unit System Module Tests Q2 Q1 Exploratory Testing Function Usability Testing (User Integration Acceptance Tests) Q3 Q4 Performance & Load Testing Security Verification Testing ility Testing Critique Product Automated Technology facing Tools Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 20 1 : taken from Crispin/Gregory: Agile Testing, Addison Wesley, 2009
Testing activities Q1 Kind of tests are Unit tests testing small pieces of code System Module Tests test that units work together correctly Q2 Kind of tests are Functional & Integration tests positive and deterministic negative tests, if possible to automate User Story Acceptance Tests Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 21
Testing activities Q3 Kind of tests Exploratory Testing feedback into stories Feature Test and Regression Test with Background Traffic (early) Characteristics Q4 Kind of tests Dimensioned Load on LLV/LSV High Load/Overload on LSV (most likely) Parafunctional tests (Performance, stress) Interoperability, Security Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 22
TEST Detailed view Detailed view on Testing Timeframe Team D Team C Feature2 Team B Team A Feature1 LSV 0 Main 0 0 0 0 0 0 0 1 1 1 1 2 1 2 2 3 2 1 1 3 Unit Test Functional Test Integration Test 4 2 3 4 2 1 Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 24
Continuous integration a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily leading to multiple integrations per day. Each integration is verified by an automated build (incl. test) to detect its errors as quickly as possible Martin Fowler Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 25
CI Workflow FCS FIP TF Sprint 1 Sprint 2 Sprint 3 Sprint n GEOFF AutoRT preparation: S1 TCs AutoRT preparation: S2 TCs AutoRT preparation: S3 TCs AutoLT preparation: e2e feature TCs AutoLT preparation: e2e feature TCs CI Server Dump building Dump building Dump building Dump building AutoLT execution (appropriate suite) AutoLT execution (appropriate suite) AutoLT execution (appropriate suite) AutoLT execution (appropriate suite) AutoRT execution S1 TCs AutoRT execution S1+S2 TCs AutoRT execution S1+ +S(n-1) TCs F1 F2 MAx Week MAx Week MAx Week MAx Week F4 Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 26
Test TOOLs & Environment CSCF MSC-S Blade Cluster MSC-S Dual Blade 60 HLR MME M- MGW M- MGW MME BSC RNC RNC BSC Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 28
Test TOOLs & Environment Unit Test: Ericsson proprietary tool testing the interfaces Function Test: Manual: using a simulated environment simulating all nodes but with the possibility to load the real MSC software Automated: using a proprietary tool with an open source framework based on TTCN-3 Regression Test: Based on the automated test cases from the Function Test Running on a Continuous Integration Server (Jenkins) Feature load (moderate/high/overload): Proprietary tool based on TTCN-3 generating load on the real environment Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 29
CONCLUSION/WRAP-UP Challenges Coordination Maintenance of a high reliability Working high quality Legacy Documentation Test tool & environment Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 30
Conclusion Benefits Eliminate waste Early indication of Software Quality Early corrective actions possible Reduce test time enabling earlier release of product Enabling of Continuous Integration as first step towards a possible Continuous Delivery Early customer involvement Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 31
Conclusion Manifesto for Agile Software Development We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Testing in large scale Agile Development Ericsson Internal Ericsson AB 2013 2013-01-28 Page 32