Process Improvement for Present and Future. Erik van Veenendaal.

Transcription

1 Test Improvement with TMMi Process Improvement for Present and Future Erik van Veenendaal

2 Erik van Veenendaal www. erikvanveenendaal.nl Founder and major shareholder ImproveQS In testing since 1989 working for many different clients and in many different roles Author TMap, The Little TMMi and many other books and papers Vice-President International Software Testing Qualifications Board (ISTQB) Vice-Chair TMMi Foundation Keynote speaker, e.g. EuroSTAR, STAR-East Winner of the European Testing Excellence Award (2007) Improve Quality Services BV 2

3 Improve Quality Services BV www. improveqs.nl Service organisation in the area of Testing, Requirements Engineering and Quality Management Consultancy, Subcontracting and Training Testing (TMap, TMMi) Test Process Improvement Certification (ISTQB) - incl. Advanced!! Inspections / Reviews Improve Quality Services BV SW Process Improvement Quality Assurance IT-Auditing Requirements Engineering & management (IREB) 3

4 Challenges Increasing size and complexity Amount of software in consumer products doubles every 24 months (Philips) Computable, Sept. 09 Number of requirements for mobile phones doubles every 6 months (Nokia) # Defects hardly decreases Defect density per KLOC is almost constant in the last 10 years (Les Hatton) Deadlines, Competition & Outsourcing Testing often takes 30-40% of project costs Improve Quality Services BV 4

5 Test Improvement Directions People Test Process Improve Quality Services BV 5

6 People Skills via (exchanging) Practical Experiences, Coaching and (formal) Training Test knowledge - Test principles - Techniques - Tools, etc. Domain knowledge - Business process - User characteristics IT knowledge - Scripting - Requirements Requirements (IREB) - Configuration mgt. Soft skills - Communication - Critical mindset - Presentation & reporting Improve Quality Services BV 6

7 Trends in Software Quality CMM(I) Maturity level Design Faults / KLOC (Keene) Delivered Defects / FP (Jones) Shipped Defects / KLOC (Krasner) Relative Defect Density (Williams) Shipped Defects (Rifkin) 5 0,5 0,05 0,5 0, ,14 2,5 0, ,27 3,5 0, ,44 6 0, , ,0 61 Improve Quality Services BV 7

8 Readiness for Test Improvement Some fundamental practices available Requirements documented and managed Project planning exists Configuration management Management needs / supports better testing Quality is important enough A budget for process improvement or reverse quality push..!! Improve Quality Services BV 8

9 TMMi model Test Maturity Model (TMM) Framework Ilinois Institute of Technology - CMM based TMMi Sources CMMI, ISTQB, TMM, IEEE, TPI Starting points Staged representation, CMMI structure Independent or in combination with CMMI TMMi V3.1 released November 2010 A B Improve Quality Services BV 9

10 TMMi process areas by level Helps to set priorities and focus the test improvement process ± 1 year 5 : Optimization - Test process optimization - Quality control - Defect prevention 4 : Management and measurement - Product quality evaluation - Test measurement programme - Advanced reviews 3 : Integration - Test organization - Test training program - Test life cycle and integration - Non-Functional testing - Peer Reviews ± 2 years 2 : Managed - Test policy and strategy - Test planning - Test monitoring and control - Test design and execution - Test environment Improve Quality Services BV 10

11 Certification Requirements defined regarding Assessment requirements (TAMAR) based on ARC (B/C) and ISO Lead Lead-Assessor and Assessor Organizations and persons to be accredited by the TMMi Foundation Register of (lead) assessors Formal TMMi Certification possible Improve Quality Services BV 11

12 Lead Assessor / Assessor Testing Lead assessor (formal assessments) > 5 years Multiple levels & domains ISTQB Advanced Assessor (informal assessments) > 3 years Multiple levels & domains ISTQB Foundation Test Process Improvement TMMi Assessments > 2 years (2 years SPI = 1 year TPI) Training Practical experience Training 20 days practical experience > 1 years Training Practical experience Training 10 days practical experience Improve Quality Services BV 12

13 Level 2 : Managed Test Policy and Strategy Test Planning Test Monitoring and Control Test Design and Execution Test Environment TMMi level 2 Lead time reduction!! Ensures CMMI compliance Verification & Validation Improve Quality Services BV 13

14 . Definition Testing Test Policy 20XX based on business policy to provide visibility regarding the quality and outstanding risks of software products to contribute to the software product quality by finding defects to contribute to customer satifaction regarding the released software products. Mission / Ambition Test targets Organization Customers Employees Test process Test Process Improvement - objectives, performance indicators, model Improve Quality Services BV 14

15 Example Performance Indicator Improvement focus w eek 1-10 w eek w eek Development.percentage of TOTAL UAT.Percentage of TOTAL FAT.percentage of TOTAL Production.Percentage of TOTAL this is what we are doing!! Improve Quality Services BV 15

16 Level 3 : Integration Test Organization Institutionalization Test Training Program Test Life cycle and Integration Non-Functional Testing Peer Reviews 16 req gd dd impl test oper Improve Quality Services BV 16

17 Life cycle Req. Test plan Design Test design Code Test script Execution Test Improve Quality Services BV 17

18 Test Improvements engineering view Testing as a profession Clear responsibilities Test plan(ning) Risk-based testing 15 PRISMA Impact 8 Top Priority Risk Matrix ISR Testing part of development life cycle Early involvement Test design techniques Controlled test process Improve Quality Services BV 18 probability Q/R US/MR ECG S/R Vasc USVW Lic Ptrac Log Srch Impd Merge DB DCMlib Dbkill Vwregr

19 Reported Results Defect Detection Percentage Y1 Y2 Y3 % deviation - spent vs. planned alpha / beta test lead time Y1 Y2 Y3 Y4 Employee satisfaction TMMi 3 TMMi 2 TMMi 1 Improve Quality Services BV 19

20 Getting Started Test policy Management commitment Improvement is a project Resources and effort percentage Maturity of development organization Improve Quality Services BV 20

21 Getting the job done Set both long term and short term goals Use what is available Don t make an external consultant responsible Institutionalize the improvements Change management skills Review against the overall objectives (policy) almost continuously Improve Quality Services BV 21

22 Test Improvement Manifesto Flexibility over Detailed Processes Best Practices over Templates Making TMMi work. Deployment orientation over Process orientation Reviews over Quality Assurance (departments) Business driven over Model driven Improve Quality Services BV 22

23 TMMi Supporting Organizations More information on TMMi available at Improve Quality Services BV 23