Testing: improving quality of software, projects and processes

Transcription

1 Testing: improving quality of software, projects and processes Bert Jagers: Pre-Sales Consultant Application Solutions & Testing 1

2 Company Overview ( Organization: HQ CTG Inc.: Buffalo, New York (Founded in 1966) 26 additional offices in USA Turnover 2008: 353 million $ (= 267 million) Headcount : employees HQ CTG Europe: Diegem, Belgium (Founded in 1976) Offices: Belgium, Luxemburg, UK, Germany Turnover 2008: 59 million Headcount : 550 employees Quality certification ISO 9001, Tick IT Listed on the Nasdaq 2

3 CTG Testing Services Test Improvement Services Collaboration Services Test Mgt Services Enterprise Architecture Services Security Testing Services Performance Testing Services Testing competence Business Alignment Services Business Process Performance Services Test Automation Services 3

4 CTG Testing Market Position Market Leader in Belgium Specialized Test Solution provider for over 10 years Number of test specialists: 160 Preferred test partner Test automation : partnerships with & Active participation International: EuroStar, ICS Test, Professional Tester, Belgium: TI-KVIV, SAI, UAMS, Datanews, Industry Magazine, Smart Business, Network & Telecom, Vacature, etc.

5 Some of our References in Testing Set up of Test Practices / Test Process Improvement Test Management & Implementing / Using Test Tools 5

6 content of the presentation introduction STBox Process People Technology a customer Test Improvement Conclusion 6

7 What is structured testing? CTG definition (ISTQB-2006) The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose, to detect defects, and to advice about risks and quality. (CTG) 7

8 structured testing: why? Deadline Planning Analysis/Design Development Development Test Test Deployment 8

9 structured testing: why? 100!! 50!! Source: Compuware 9

10 structured testing: why? Distribution of Bugs Source: Bender RBT Inc. 10

11 structured testing: why? this is what it should be like Planning Analysis/Design Development Test Deployment Not a phase after development! 11

12 structured testing: why? added value? I ran some tests but I can t remember what happened or how I could repeat those tests was this testing valuable? I ran 100 tests and the system failed 23 times I can t remember how I did it or where the system failed was this test valuable? I found 56 defects so what? the PROCESS of testing only has VALUE if it generates INFORMATION (this is the deliverable) 12

13 structured testing: why? added value? Project Intelligence Risks/Mitigations Critical for management 13

14 V-Model Business Requirements Acceptance Testing Business System Requirements System Integration Testing System Testing Testing Global Design Unit Integration Testing Development Detailed Design Code/Build Unit Testing Test Levels Test Types Regression Testing Retesting White Box Testing Black Box Testing 14

15 Quality Attributes (ISO9126) Accuracy Functionality Suitability Interoperability Security Reliability Maturity Fault Tolerance Recoverability Understandability Useability Learnability Operability Attractiveness Efficiency Time Behavior Resource Utilization Analyzability Maintainability Changeability Stability Testability Adaptability Portability Co-existence Installability Replaceability 15

16 Test Methodology 16

17 why STBox? a Software Testing Method Based on CTG Experience Test Project Preparation For every Iteration/Test Level/Test Type: Box refers to CTG s general project framework Test Management Quality Management STBox is a software testing method that covers 3 dimensions: process people technology 17

18 why STBox? simplify communication, provide consistency, uniformity and continuity ISEB and ISTQB certification programs are used cover only certain aspects (e.g. terminology, templates) STBox provides the glue also aspects where STBox does not follow the standards (e.g. test procedure template, definitions black and white box testing, test basis concept, etc.) 18

19 19

20 the different phases Test Project Preparation For every Iteration/Test Level/Test Type Test Manage ement Quality Management 20

21 STBox in detail 1. Determine Test Scope 2. Determine Test Strategy High-Level Matrix Test Strategy 3. Plan Project Test Project Test Plan Test Project Preparation Test Management FOR EACH ITERATION / TEST LEVEL / TEST TYPE : 4. Determine Detailed Test Strategy Detailed Test Planning FTT Tree Quality Management 16. Staff and Manage Test Team Detailed Test Plan 5. Plan Iteration / Test Level / Test Type Defect s Review Report 22. Review Test Basis 17. Monitor and Adjust Test Plans Additional Test Design 6. Design Tests FTT Tree 18. Follow Up and Report Status Test Status Reports 8. Set Up Test Infrastructure 7. Organize Test Execution Test Execution Schedule Test Procedures Test Cases Checklists Test Scripts 19. Manage Defects 9. Install Test Object 23. Review Test Deliverables and Organize Approval 10. Verify Test Infrastructure & Test Object Test Results Defect s 20. Manage Issues & Changes Test infrastructure or test object needs modification Test repository needs modification 11. Execute Tests Test Results Defect s 21. Facilitate Delivery Test object needs modification Test repository needs modification 12. Summarize Test Execution Test Summary Report Test object needs modification Test repository needs modification 14. Evaluate Test Project Test Evaluation Report 13. Consolidate Test Deliverables 15. Release Test Staff from Test Assignment 21

22 STBox Essentials 22

23 the different phases related to V-Model Project Preparation User Requirements Acc Tst Planning Test Project Preparation System Requirements Sys Int Tst Planning Sys Tst Planning Project Manage ement Global Design Detailed Design Code / Build Test Management Acc Test Build System Int Test Build System Test Build Component Comp Int Tst Planning Comp Tst Comp Planning Int Test Comp Tst Build Build Component Integration System Qu uality Management Quality Management System Integration Deployment Project Closure Acceptance 23

24 Test Project Preparation Test Man nagement For every Iteration/Test Level/Test Type Quality Ma anagement 24

25 test project preparation 1. determine test scope Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TEST MANAGER 1. Determine Test Scope TEST PROJECT PREPARATION 2. Determine High- Level Test Strategy Test Strategy Matrix TEST MANAGER TEST MANAGER 3. Plan Test Project Project Test Plan TO TEST MANAGEMENT TO QUALITY MANAGEMENT TO 8 TO 5 TO 4 25

26 1. determine test scope Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management Meetings, Interviews Context Information Testing Information Checklist Project Plan ACCOUNTABLE RESPONSIBLE Test Manager Inventory and Assess Available Information Define Test Project Scope and Objectives TO 3 TO 2 Project Test Plan CONSULTED Project Manager Stakeholders IT Stakeholders Business 26

27 2. determine high level test strategy Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TEST MANAGER TEST MANAGER TEST MANAGER 27

28 2. determine high level test strategy Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management What to test? / How deep to test? objective: finding the most important defects as soon and as cheap as possible. 28

29 2. determine high level test strategy Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 29

30 Product Risk Management Product Risk = Impact * Likelihood What is the impact on business? What is the likelihood defects will occur? Use risk factors 30

31 Risk factors Impact Business criticality Financial (or other) consequences (or damage) User intensity External visibility Likelihood Complexity New development? (level of reuse) Interrelations (# interfaces) Size Technology Experience 31

32 The Product Risk Matrix 25 I Medium Focus of Development Level Testing II High LIKELIHOO OD 15 5 III Low Focus of System Level Testing IV Medium IMPACT 32

33 The Product Risk Matrix Pair Inspection Statement Coverage 70% 25 Full code Inspection Boundary Value Analysis Statement Coverage 90% I Medium Focus of Development Level Testing II High LIKELIHOO OD 15 Exploratory 5 Testing III Low Focus of System Level Testing IV Medium Statement Coverage 70% IMPACT 33

34 Importance of Risk Management More efficiënt (and cheaper) testing First test in large risk areas And test more thorough there Zekerheid dat het aantal technische risico s tot een minimum gereduceerd is tijdens developer testen. Necessary: Developers know the risks (Technical analyse!) Interesting for regression testing 34

35 2. determine high level test strategy 2.1. identify high level risks and requirements Requirements Open documents Typing new text Editing existing text Printing Saving documents English French Dutch German Test Management Impact Likelihood Priority Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management Product Risks Notepad causes instability of Windows Response times of user actions in Notepad takes too long Notepad for German contains translation errors in menus, pop-up windows, error messages Unexpected crash of Notepad due to unnatural behaviour of user Notepad is not installed correctly together with Windows installation Notepad user interface is not user-friendly and intuitive Notepad allows security exploits (e.g. Access to windows kernel through Notepad) Impact Likelihood Priority Test Project Risks Test team is not experienced in test automation Versions for system testing are not stable enough to proceed testing Delay in development Existing test procedures are not re-used to full extent Test manager leaves the project (sickness, resignation, etc.) Test team is not familiar with the test methodology Impact Likelihood Priority 35

36 FTTs 2. determine high level test strategy 2.3. define risk response Impact Likelihood Priority Risk Response Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Preventive Detective Corrective (Contingency ) Functionality Suitability Open documents Functional Testing H M H Reviewing System requirements Typing new text Functional Testing H L M Editing existing text Functional Testing H L M Printing Hire developer specialized in Functional Testing H H H printing Reviewing System requirements Saving documents Functional Testing H L M English Functional Testing H M H Reviewing System requirements French Functional Testing M L L Dutch Functional Testing M L L German Outsourcing of translations to Functional Testing M H H professional German translator Reviewing System requirements Security Notepad allows security exploits (e.g. Access to windows kernel through Notepad) L L L Security audit on design Security Testing Reliability Fault tolerance Unexpected crash of Notepad due to unnatural behaviour of user H M H Functional Testing Reviewing System requirements Usability Understandability Notepad user interface is not user-friendly and intuitive L L L Usability Testing Efficiency Time Behaviour Response times of user actions in Notepad takes too long M L L Performance Testing Portability Co-existence Notepad causes instability of Windows M L L Portability Testing Installability Notepad is not installed correctly together with Windows installation H M H Installability Testing Reviewing Installation guide Quality Management Test Project Risks Process Existing test procedures are not re-used to full extent Delay in development People Test team is not experienced in test automation Test manager leaves the project (sickness, resignation, etc.) Test team is not familiar with the test methodology Technology Versions for system testing are not stable enough to proceed testing Impact Likelihood Priority L M M H H H H H H L H M H M H M H H Preventive Detective Corrective (Contingency ) Organize meeting between owner of existing test procedures and test manager - Close monitoring of the progress by the project manager - Start with development of highpriority features Assign dedicated test automators and send them to training Assign test lead and involve test lead in test management activities Organize briefing on test methodology Define clear entry criteria for system testing Risk Response Demand overwork and/or engage extra capacity Test lead acts as back-up test manager Send test object back to development team and apply exploratory testing to the unstable test object 36

37 2. determine high level test strategy 2.4. create test strategy matrix ID Priority Info Features To Test Component Testing Component Integration Testing System Testing Test Management System Integration Testing Test Project Preparation For every Iteration/Test Level/Test Type: Acceptance Testing Quality Management H M H M M H L L H FUNCTIONAL TESTING NOTEPAD ENGLISH OPEN SAVE PRINT GENERAL FEATURES TYPING EDITING UNNATURAL BEHAVIOUR NOTEPAD FRENCH NOTEPAD DUTCH NOTEPAD GERMAN L SECURITY TESTING L USABILITY TESTING USER INTERFACE L PERFORMANCE TESTING USER ACTIONS L PORTABILITY TESTING INSTABILITY OF WINDOWS OS INSTALLABILITY TESTING H AUTOMATIC INSTALL TOGETHER WITH WINDOWS 37

38 2. determine high level test strategy 2.5. assign test techniques ID Priority Info Features To Test Component Testing Component Integration Testing System Testing Test Management System Integration Testing Test Project Preparation For every Iteration/Test Level/Test Type: Acceptance Testing Quality Management H M H M M H FUNCTIONAL TESTING NOTEPAD ENGLISH OPEN SAVE PRINT GENERAL FEATURES TYPING EDITING UNNATURAL BEHAVIOUR J-Unit Tst based on decision table Experienced J-Unit developer Tst based on Program Interface Test Program Interface Syntactical Tst Syntactical Tst Part of E2E Test Inspection System Requirements Part of E2E Test J-Unit decision Tst table based Program Test Decision Tables Part of E2E Test on decision table Experienced developer Tbd Interface Test Tbd Syntactical Tst Inspection System Requirements Exploratory Testing Checklist Idiot Proofing Destructive Tst Idiot Proofing Destructive Tst Idiot Proofing Destructive Tst Inspection System Requirements L L H NOTEPAD FRENCH NOTEPAD DUTCH NOTEPAD GERMAN Code Review Code Review Code Review Review translation files Review translation files Exploratory Testing Inspection Translation files Tbd Tbd Exploratory Testing Extra Beta Test German speaking community L SECURITY TESTING Penetration test Penetration test L USABILITY TESTING USER INTERFACE Checklist L PERFORMANCE TESTING USER ACTIONS Manual response time checking Response time checking with tool PORTABILITY TESTING Use list of predefined integration tests Exploratory Testing L INSTABILITY OF WINDOWS OS INSTALLABILITY TESTING H AUTOMATIC INSTALL TOGETHER WITH WINDOWS Code Review Install from network Walkthrough Installation Guide Tbd Install from CD 38

39 3. plan test project Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TEST MANAGER 1. Determine Test Scope TEST PROJECT PREPARATION 2. Determine High- Level Test Strategy Test Strategy Matrix TEST MANAGER TEST MANAGER 3. Plan Test Project Project Test Plan TO TEST MANAGEMENT TO QUALITY MANAGEMENT TO 8 TO 5 TO 4 39

40 3. plan test project Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 40

41 3. plan test project Purpose: the C-words Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management CONSOLIDATION COMMUNICATION COORDINATION CONSENSUS COMMITMENT CLAIM CONTROL (EVALUATION) 41

42 3. plan test project Contents: the W-words Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management WHY WHO WHAT WHEN WHERE WHICH WAY (HOW) WHICH MEANS 42

43 3. plan test project IEEE 829 (1) Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management content 16 points test plan information introduction test Items features to be tested features not to be tested approach item pass / fail criteria suspension and resumption criteria 43

44 3. plan test project IEEE 829 (2) Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management content 16 points test deliverables testing tasks environmental needs responsibilities staffing and training needs schedule risks and contingencies approvals 44

45 Test Project Preparation Test Man nagement For every Iteration/Test Level/Test Type Quality Man nagement 45

46 Test Project Preparation For every Iteration/Test Level/Test detailed test planning Test Management Type: Quality Management 4. determine detailed test strategy TEST MANAGER TEST MANAGER 46

47 Test Project Preparation For every Iteration/Test Level/Test 4. determine detailed test strategy Test Management Type: Quality Management 47

48 Test Project Preparation For every Iteration/Test Level/Test detailed test planning Test Management Type: Quality Management 5. plan iteration / test level / test type TEST MANAGER TEST MANAGER 48

49 Test Project Preparation For every Iteration/Test Level/Test 5. plan iteration/test level/test type Test Management Type: Quality Management 49

50 Test Project Preparation Test Man nagement For every Iteration/Test Level/Test Type Quality Ma anagement 50

51 6. design tests FOR EACH ITERATION / TEST LEVEL / TEST TYPE : FROM 3 FROM 4 FROM 5 Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TEST BUILD Additional Test Design TESTER 6. Design Tests FTT TreeTESTER TEST SUPPORT 8. Set Up Test Infrastructure 7. Organize Test Execution Test Execution Schedule Test Procedures Test Cases Checklists TEST MANAGER TEST SUPPORT 9. Install Test Object Test Scripts FROM 10 FROM 11 FROM 12 TO 8 FROM 10 FROM 11 FROM 12 51

52 Test Project Preparation 6. design tests Test Management For every Iteration/Test Level/Test Type: Quality Management 52

53 Test Project Preparation 6. design tests Test Management For every Iteration/Test Level/Test Type: Quality Management test case a test case is a description of a well considered combination of input data, the processing of that input and the there from resulting predicted output. frame INPUT PROCESS OUTPUT 53

54 test build 7. organize test execution Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TESTER TESTER TEST SUPPORT TEST SUPPORT TEST MANAGER 54

55 test build 8. set up test infrastructure 9. install test object FOR EACH ITERATION / TEST LEVEL / TEST TYPE : FROM 3 FROM 4 FROM 5 Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TESTER Additional Test Design TEST BUILD TEST SUPPORT 6. Design Tests FTT Tree TESTER 8. Set Up Test Infrastructure 7. Organize Test Execution Test Execution Schedule Test Procedures Test Cases Checklists TEST MANAGER TEST SUPPORT 9. Install Test Object Test Scripts FROM 10 FROM 11 FROM 12 TO 8 FROM 10 FROM 11 FROM 12 55

56 Test Project Preparation Test Man nagement For every Iteration/Test Level/Test Type Quality Ma anagement 56

57 test execution 10. Verify test infrastructure & test object Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management FOR EACH ITERATION / TEST LEVEL / TEST TYPE : TO 8 TO 9 FROM 7 FROM 8 FROM 9 TO Verify Test Infrastructure & Test Object Test Results Defect s TEST EECUTION TEST SUPPORT Test infrastructure or test object needs modification Test repository needs modification TESTER 11. Execute Tests Test Results Defect s TESTER Test object needs modification Test repository needs modification Test object needs modification 12. Summarize Test Execution Test Summary Report Test repository needs modification TEST MANAGER TO 14 57

58 test execution 11. execute tests FOR EACH ITERATION / TEST LEVEL / TEST TYPE : TO 8 TO 9 FROM 7 FROM 8 FROM 9 TO 6 Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 10. Verify Test Infrastructure & Test Object Test Results Defect s TEST EECUTION TEST SUPPORT Test infrastructure or test object needs modification Test repository needs modification TESTER 11. Execute Tests Test Results Defect s TESTER Test object needs modification Test repository needs modification 12. Summarize Test Execution Test Summary Report Test object needs modification Test repository needs modification TEST MANAGER TO 14 58

59 11. execute tests record results Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 59

60 11. execute tests record results Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 60

61 11. execute tests record results Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 61

62 test execution 12. create test summary report FOR EACH ITERATION / TEST LEVEL / TEST TYPE : TO 8 TO 9 FROM 7 FROM 8 FROM 9 TO 6 Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 10. Verify Test Infrastructure & Test Object Test Results Defect s TEST EECUTION TEST SUPPORT Test infrastructure or test object needs modification Test repository needs modification TESTER 11. Execute Tests Test Results Defect s TESTER Test object needs modification Test repository needs modification 12. Summarize Test Execution Test Summary Report Test object needs modification Test repository needs modification TEST MANAGER TO 14 62

63 12. create test summary report defects per test hour 3 2,5 2 1,5 Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Severity Desirable Low Medium High Quality Management 1 0,5 0 Week 2 Week 4 Week 6 Week 8 Week 10 63

64 Test Project Preparation Test Man nagement For every Iteration/Test Level/Test Type Quality Ma anagement 64

65 test project closure 13. consolidate test deliverables Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TEST MANAGER TEST MANAGER TEST MANAGER 65

66 test project closure 14. evaluate test project Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TEST MANAGER TEST MANAGER TEST MANAGER 66

67 14. evaluate test project purpose? lessons learned gather historical data consolidated test summary reports transfer test deliverables (reusability) how? according to test plan test process (efficiency) are we doing the things right? test product (effectiveness) are we doing the right things? advice about quality (risks/alternatives) Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 67

68 test project closure 15. release test staff from test assignment Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management TEST MANAGER TEST MANAGER TEST MANAGER 68

69 Test Project Preparation Test Man nagement For every Iteration/Test Level/Test Type Quality Ma anagement 69

70 test management TEST MANAGEMENT FROM 3 Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 19. Manage Defects 16. Staff and Manage Test Team FROM Manage Issues & Changes 17. Monitor and Adjust Test Plans 21. Facilitate Delivery 18. Follow Up and Report Status Test Status Reports TEST MANAGER TO 14 70

71 16. staff & manage test team Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management test budgeting & planning Testing in total project effort 40% 5% 20% 15% Comp & Comp Int Testing System & System Int Testing Acceptance Testing 71

72 But. These ratios are industry standards! HOWEVER: No project is standard -> correction factor needs to be applied - depending on how much you know (risk involved) the ingredients of the correction factor may vary 72

73 Test Budget Influencers Strategy What? How thorough? Productivity How good are my teams? Test Effort Estimate Size How much?? Scope? Expected Quality Test Object? Test Basis? Infra & Tools How fast can we test? Learn from historical Data! 73

74 Correction factor Business Criticality Major Low=2 Medium=4 High=6 Technology Minor Low=1 Medium=2 High=4 Complexity Major Low=2 Medium=4 High=6 Development team Major Low=2 Medium=4 High=6 Test Team Minor Low=1 Medium=2 High=4 20% Test Budget 65 % 74

75 Correction factor Example 1 Questions 1. Business Risk 2. Technology 3. Complexity 4. Developement Team 5. Test Team Priority Answer Factor Comment Major H 6 Some business people are impacted between 30% and 100% Minor Major Major L H M Based on existing and well known technology Very complex, many interfaces to existing applications, high risk for some projects. Experienced development team but problems with quality of analysis documents Minor L 1 Very experienced Test Team. Good business knowledge. Factors 18 Risk Indicator (= Factors/16) 1,125 Industry standard 40 Test Budget 45 % 75

76 16. staff & manage test team test budgeting & planning relation between: time (milestones/deadlines) budget (resources) quality (test coverage) scope Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 76

77 18. follow-up & report status status report Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management purpose: New defects Open defects Defects Solutions Trends & Metrics Resource Tracking Lead Time Tracking Work Product Tracking New Closed Open (cat A + B) Week 1Week 3Week 5Week 7Week 9 Week 11 77

78 18. follow-up & report status Performance Indicators: 1. Define the test performance indicators 2. Align them with: The Test Policy Objectives for Test Process Improvement 3. Collect data in the projects 4. Track and evaluate the Test Performance Indicators 5. Distribute to the stakeholders Goal oriented! 78

79 18. follow-up & report status Product Quality Number of defects found Requirements coverage; reduction of defects Improvement of defect detection percentage (DDP) Efficiency Test lead time; # of test cases automated Test Effort / total development effort Predictability Test lead-time slippage; effort slippage Maturity level 79

80 19. manage defects objectives defect administration follow up status defect found documentation defects found and solution chosen test management Test Management Test Project Preparation For every Iteration/Test Level/Test Type: overview number of open defects per project / per domain / general test repair time: test planning test release management: entry / exit criteria (gates) between test levels regression testing product and process quality defect analysis Quality Management 80

81 19. manage defects defect logging & traceability Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management Defect logging - IEEE defect identification and description id description project date test object reproducible environment status test procedure assigned to test case / test step tester impact information severity of defect impact of defect repair information priority diagnosis information estimated fix time traceability link defects with test cases link test cases with requirements 81

82 19. manage defects defect logging: 10 Golden Rules (Rex Black) Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management 1. structure 2. reproduce 3. isolate 4. generalize 5. compare 6. summarize 7. condense 8. disambiguate 9. neutralize 10. review 82

83 19. manage defects Defect Tracking Flow Test Management Test Project Preparation For every Iteration/Test Level/Test Type: Quality Management New Open Statuses the Role Can Assign Assigned to Fixed Role New Ope en ted Reject Assigne ed to (testers or de evelopers) Postpo oned Close ed Fixe ed Ready for retest Reop pen Test Manager Testers Ready for retest Developers Analysts Postponed Closed Rejected Project Team Decision Forum Representative Reopen 83

84 Test Project Preparation Test Man nagement For every Iteration/Test Level/Test Type Quality Ma anagement 84

85 Test Project Preparation quality management FROM 3 QUALITY MANAGEMENT Test Management For every Iteration/Test Level/Test Type: Quality Management 22. review test basis 23. review test deliverables & organize approval FROM 5 Defect s Review Report 22. Review Test Basis 23. Review Test Deliverables and Organize Approval TO 14 85

86 Reviewing Distribution of Bugs Source: Bender RBT Inc. 86

87 Review Techniques Informal Review Informal technique Walkthrough What are we building? Technical Review How are we building it? Inspection Is everything correctly documented? 87

88 Benefits of reviewing Early defects are often the most important 56% of defects are already created before coding starts Finds other defects then dynamic testing (design mistakes) Development productivity improves Reduced Lead-times Reduced Testing cost & time Less Test Cycles and retesting Improved Communications 88

89 People 89

90 roles and responsibilities 90

91 test organizations function based test organization independent test organization role based test organization test competence center outsourced testing ORGANIZATIONAL MODELS

92 Technology 92

93 technology test infrastructure test environment office environment test infrastructure procedures test tools 93

94 test infrastructure test environment hardware operating system & system software network other applications installed (interfaces) test databases 94

95 test infrastructure test infrastructure procedures organization (ownership, responsibilities, etc.) provision (SLA s regarding response times, availability, etc.) maintenance (upgrades hardware & system software) troubleshooting configuration & change management 95

96 test tools what are tools? tools are no more than pieces of equipment, meant to support the manual as well as the mental work. We use the possibilities of a computer to improve the quality of testing Computer Aided Software Testing: supporting the testing of software by means of a computer a tool is a means and shouldn t be a goal in itself 96

97 test tools CAST IMPROVEMENT OF PRODUCTIVITY tools increase the efficiency and effectiveness!!! 97 structured testing

98 test tools classification of test tools 98

99 Automated Unit Testing Regression Testing purposes Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed. (ISTQB glossary) Every new build Different Tools available 99

100 Test Management Tools Managing your Test Cases/requirements/ Reporting (Defect) Management Different Tools available: 100

101 Test Automation Tools Regression Testing purposes (system (integration) testing) Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed. (ISTQB glossary) Every new build Different Tools available: 101

102 Load and Performance Test Tools Performance Testing Load/Stress Testing Different Tools available: 102

103 Bert Jagers Pre-Sales Consultant Application Solutions & Testing 103