Proactive Shift Left (left, left, left), a practical approach. Szilard Szell, Test Coach, NOKIA 15th November, 2017, Budapest

Transcription

1 Proactive Shift Left (left, left, left), a practical approach Szilard Szell, Test Coach, NOKIA 15th November, 2017, Budapest Nokia Nokia

2 Introduction Szilárd Széll Test Coach, Cloud Transformation Leader, NOKIA Experiences Test Coach in DevOps Center of Excellence in NOKIA with 17 years experience President of the Hungarian Testing Board Program Committee Chair ETSI s UCAAT (2016) Organizational Committee HUSTEF ( ) Education: MSc in IT Management, Central European University MSc in Programming and Mathematics, University of Szeged Certified Scrum Master, ISTQB CTEL-ITP-Full, ISTQB CTAL-TM, CTFL- AT, IREB CPRE, Lean Six Sigma Green Belt Nokia 2017 Szilard Szell Bug Hunting in Practice Public

3 Nokia Nokia internal use

4 Our Agenda NOKIA today the complexity Why to Shift Left the problem How to Shift Left a solution Implement Proactive Shift Left deployment 4 Nokia 2017

5 Nokia described as a house of Testing Cornerstones of Product and Solution Testing and Verification Well defined Testing Processes on Company level Well defined, standardized Requirements as input ISO9001 and TL9000 compliant processes Lean Six Sigma Quality Management Complex, Highly available (99.999%), Robust Products on IT Cloud and Telco HW 5 Nokia 2017

6 Our Agenda NOKIA today the complexity Why to Shift Left the problem How to Shift Left a solution Implement Proactive Shift Left deployment 6 Nokia 2017

7 Well defined Test Areas might result in silo thinking and losing responsibility Test Levels and Areas vs. functional and non-functional testing in NOKIA Functional test Non-functional test Functional test Non-functional test Feature Test Protocol Conformance Test Entity (system) Entity Integration Security Test Operability Test Performance Test Network (system of systems) E2E Feature Verification Network Integration Security Verification Operability Verification Compatibility Testing, On-air Verification Entity Performance Verification Network Performance Verification Interoperability Testing 7 11/28/2017 Nokia 2017

8 Fault analysis of one product shows significant fault leakage Lot of fault Should Have Been Found (SHBF) in module testing and functional testing. How to left shift testing? Value leakage from earlier test phases, % 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Found 0% by: Development Module Test Integration Functional Test System Test Customer SHBF by: System Test Functional Test Integration Module Test Development Specification 8 11/28/2017 Nokia 2017

9 Our Agenda NOKIA today the complexity Why to Shift Left the problem How to Shift Left a solution Implement Proactive Shift Left deployment 9 Nokia 2017

10 The aim of Proactive Shift Left is to reduce effort spent on complex test phases by moving our test focus to lower test levels Network Test PSL Network Test PSL System Test PSL System Test PSL Component Test PSL Component Test PSL Static Testing PSL Static Testing PSL 10 11/28/2017 Nokia 2017

11 Mindset change towards giving feedback early instead of waiting for certification decreases effort Requirement: Draw a RED rectangle, where RED color code is R:192 G:0 B:0 Feedback: PASSED Certified: PASSED Feedback: FAILED THIS IS RED Measured R:192 G:0 B:0 BLUE = NOT RED 11 11/28/2017 Nokia 2017

12 Test Approach based on Proactive Shift Left builds on feedback given early Giving feedback vs. Certify Mindset change From: Product Aspect Specification Review Static Analysis Component Test Integration Test System Test Network Verification Functionality Feedback Feedback Feedback Certify Performance Feedback Certify Reliability To: Product Aspect Specification Review Static Analysis Component Test Certify Integration Test System Test Network Verification Functionality Feedback Feedback Feedback Feedback Feedback Certify Performance Feedback Feedback Feedback Feedback Certify Reliability Feedback Feedback Feedback Feedback Feedback Certify 12 11/28/2017 Nokia 2017

13 Our Agenda NOKIA today the complexity Why to Shift Left the problem How to Shift Left a solution Implement Proactive Shift Left deployment 13 Nokia 2017

14 Workshops resulted in detailed Test Approach focusing on each Quality Characteristic Test Approach using Proactive Shift Left focusing on Performance Quality Aspect Specification Review (Checklist) Static Analysis (Rules) Component Test (Test Cases) Integration Test (Test and Config) System Test (Test and Config) Network Verification (Tests) Performance Feedback: - Perfor-mance Aspects identified - Data structure validation for size - Speed of Algorithm - Expected response times Feedback - Identified coding guidelines for performance - Data structure analysis Feedback - response times during functionalm odul test - CPU and memory usage Feedback - Acceptance test response times on interfaces - Mass interface configuration based on traffic profile Certify - Single test of traffic profile scenarios on simple SUT - Load, Stress testing of traffic profile on target SUT 14 11/28/2017 Nokia 2017

15 Summary o o o o o Proactive Shift Left requires understanding of all Test Phases Silos need to be broken Mindset shift towards feedback giving vs certify product is needed Phased roll-out is possible focusing on different Quality Characteristics Defined Test Approach need to be continuously improved based on leakage info 15 Nokia 2016

16 Thank You! 16 Nokia 2017