WHY TEST SOFTWARE?...

Transcription

1 2 At a glance 1 PREFACE AT A GLANCE TABLE OF CONTENTS INTRODUCTION WHY TEST SOFTWARE? WHY TEST SOFTWARE? LIMITATIONS OF TESTING ALTERNATIVE TO TESTING PROVING THREE OBJECTIVES OF TESTING SOFTWARE WHAT S SOFTWARE TESTING ABOUT? THE DIFFERENCE BETWEEN A TESTER AND A DEVELOPER THE THREE-HEADED DRAGON OF SOFTWARE DEVELOPMENT THE LANGUAGE OF SOFTWARE TESTING A BUG S LIFE SEVERITY OF BUGS THE KINDS OF BUGS STAGES OF SOFTWARE DEVELOPMENT FOR A TESTER

2 6 7.1 CHALLENGING REQUIREMENTS CHALLENGING SPECIFICATIONS CHALLENGING ARCHITECTURE AND TECHNICAL DESIGN TESTING CODE CHALLENGING DOCUMENTATION RELEASE DIMENSIONS OF SOFTWARE TESTING WHY DIMENSIONS? BY OBJECT BY SOURCE CODE VISIBILITY BY LEVEL BY PLACE BY PURPOSE BY ATTACK SURFACE BY FAULT TYPE QUALITY OF TESTING HUH? CODE COVERAGE SOFTWARE METRICS TEST PLAN REVIEW SEEDING BUGS AND CODE MUTATION TESTABILITY BETA RELEASE/BETA TESTING POST-RELEASE FUN DEPLOYMENT TESTING COMPATIBILITY TESTING SYSTEM TESTING STAGING AND PRODUCTION ENVIRONMENT TESTING SOFTWARE IN A GLOBAL TEAM TECHNICALLY SPEAKING HUMAN-ORIENTED SURVIVING AS A TESTER FUZZY VALUE IN THE SOFTWARE DEVELOPMENT.. 145

3 12.2 COST/VALUE BALANCE RELIGIONS AND CONFESSIONS IEEE ROLE OFFICE POLITICS OUTSOURCING AND JOB SECURITY SEGREGATION FUTURE OF THE TRADE APPENDIX A. TEST CASE APPENDIX B. BIBLIOGRAPHY STANDARDS AND CLASSICS EXTREME TESTING & RELATED OTHER ENDNOTES

4 3 Table of Contents 1 PREFACE 3 2 AT A GLANCE 5 3 TABLE OF CONTENTS 9 4 INTRODUCTION 17 5 WHY TEST SOFTWARE? WHY TEST SOFTWARE? LIMITATIONS OF TESTING The First Law of Software Programming Nothing is ever perfect Requirements Design Requirements context, missed or ignored requirements Overusing formal verification tools without proper qualification Late design changes and code fixes Code Platform Tests Users Monkey 32 9

5 Farmer Hacker The Product Team Richness of possibilities Code paths, state transitions Inputs (including input editing and timing) Data in the environment In essence you cannot try everything ALTERNATIVE TO TESTING PROVING THREE OBJECTIVES OF TESTING SOFTWARE So, why? Objective #1: Quality Bar Objective #2: Acceptance Objective #3: Baseline 41 6 WHAT S SOFTWARE TESTING ABOUT? THE DIFFERENCE BETWEEN A TESTER AND A DEVELOPER A Castle Debugging is not testing Goals Actor Tools Results Super-pessimism Testing that the glass is half empty THE THREE-HEADED DRAGON OF SOFTWARE DEVELOPMENT Software developer Software architect, analyst, project manager Software tester THE LANGUAGE OF SOFTWARE TESTING The bug: what is it? Test case Test plan Bug tracking Version control Test automation A BUG S LIFE 54

6 6.4.1 Bug opened Bug approved Bug fixed Bug verified and closed SEVERITY OF BUGS Know Thine User! THE KINDS OF BUGS Code defect Crash A special kind: Regression Security Performance-related Throughput (performance) Stress Volume Scalability Resources consumption (leaks) Usability Documentation Test issue Bugs in tests and frameworks Mixing up a bug and a feature 70 7 STAGES OF SOFTWARE DEVELOPMENT FOR A TESTER CHALLENGING REQUIREMENTS CHALLENGING SPECIFICATIONS Challenging assumptions Challenging completeness Challenging behavior Challenging usability Challenging testability That s not the end CHALLENGING ARCHITECTURE AND TECHNICAL DESIGN TESTING CODE CHALLENGING DOCUMENTATION RELEASE 83 8 DIMENSIONS OF SOFTWARE TESTING 85 11

7 WHY DIMENSIONS? BY OBJECT Positive testing capabilities Negative testing environment BY SOURCE CODE VISIBILITY Black ( opaque ) box testing White (glass, clear) box testing Gray ( Translucent ) box testing BY LEVEL Unit testing Component testing Integration testing System testing Scenario-based Free-form BY PLACE Product Team Testing (vendor) Customer (user) Side Testing Third Party Testing (certification) BY PURPOSE Functional testing Regression testing Performance related testing Performance (throughput) testing Volume (size) testing Load (stress) testing Scalability testing Resource consumption (including leaks) testing Sizing guidance Conformance testing Standards Legislation Coding policy testing Security testing Functional security testing Security model and STRIDE Security penetration testing Globalization testing 103

8 Encoding Watch your language Formats More calendar fun Units Strings Length of text Making of an error message That s not all Localization testing Cultural differences Platform and dependencies testing Pseudo-testing Usability Reliability Robustness BY ATTACK SURFACE What is an attack surface Input Input GUI Input API Input Files and DB content Environment Environment Resources (storage, memory, etc.) Environment Access rights Environment Network Environment Files and folders Environment Media Environment Output devices Environment Platform User User monkey User farmer User hacker BY FAULT TYPE A word on the fault model Boundary conditions Arithmetic overflows

9 Buffer overflows Race conditions Uninitialized variables Uncaught exceptions Resource allocation That s not all again QUALITY OF TESTING HUH? CODE COVERAGE SOFTWARE METRICS TEST PLAN REVIEW SEEDING BUGS AND CODE MUTATION TESTABILITY BETA RELEASE/BETA TESTING POST-RELEASE FUN DEPLOYMENT TESTING COMPATIBILITY TESTING SYSTEM TESTING STAGING AND PRODUCTION ENVIRONMENT TESTING SOFTWARE IN A GLOBAL TEAM TECHNICALLY SPEAKING HUMAN-ORIENTED Time zone differences Work relations and cultural differences Life in a global org SURVIVING AS A TESTER FUZZY VALUE IN THE SOFTWARE DEVELOPMENT_ COST/VALUE BALANCE RELIGIONS AND CONFESSIONS IEEE ROLE OFFICE POLITICS OUTSOURCING AND JOB SECURITY SEGREGATION FUTURE OF THE TRADE From nobility to peasants 158

10 Crushing craftsmen Assault on software development What s next? APPENDIX A. TEST CASE APPENDIX B. BIBLIOGRAPHY STANDARDS AND CLASSICS EXTREME TESTING & RELATED OTHER ENDNOTES