Carnegie Mellon University Notice

Transcription

1 Carnegie Mellon University Notice This video and all related information and materials ( materials ) are owned by Carnegie Mellon University. These materials are provided on an as-is as available basis without any warranties and solely for your personal viewing and use. You agree that Carnegie Mellon is not liable with respect to any materials received by you as a result of viewing the video, or using referenced websites, and/or for any consequences or the use by you of such materials. By viewing, downloading, and/or using this video and related materials, you agree that you have read and agree to our terms of use ( A Taxonomy of Types 1

2 Copyright 2015 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN AS-IS BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution except as restricted below. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. Carnegie Mellon is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. DM A Taxonomy of Types 2

3 A Taxonomy of Types Software Engineering Institute Carnegie Mellon University Pittsburgh, PA Donald G. Firesmith

4 Topics Relevant Challenges Goals of Presentation What is? Presentation Scope Types Conclusion A Taxonomy of Types 4

5 Relevant Challenges A Taxonomy of Types 5

6 Relevant Challenges Many testers are only aware of a minority of types of testing, let alone know how to perform them. Test managers and developers are aware of even fewer testing types. The test strategies, project test plans, and test sections of system/software development plans tend to identify only a very small number of types of testing (e.g., unit, integration, system, and acceptance testing). Not planned Not performed A Taxonomy of Types 6

7 Goals of the Presentation A Taxonomy of Types 7

8 Goals of the Presentation Make it clear that: There are many different types of testing. is a complex discipline with its own technical jargon. There is a lot of overlap between different classes of testing types. Think multiple classification (object-oriented design) or multiple inheritance (object-oriented programming). Get you to take a look at your: Test strategies and test plans and ask yourselves Are they sufficiently complete? Testers and other testing stakeholders and ask yourselves Do they need additional training in testing types. A Taxonomy of Types 8

9 Polling Question 1 How many different types of testing do you typically use on a project? O 1-5 O 6-10 O O 16+ A Taxonomy of Types 9

10 What is? A Taxonomy of Types 10

11 What is? The execution of an Object Under Test (OUT) under specific preconditions with specific stimuli so that its actual behavior can be compared with its expected or required behavior Preconditions: pretest mode, states, stored data, or external conditions Stimuli: - Calls, commands, and messages (control flows) - Data inputs (data flows) - Trigger events such as state changes and temporal events Actual Behavior: - During Test: Calls, commands, and messages (control flows) Data outputs (data flows) - Postconditions: post-test mode, states, stored data, or external conditions A Taxonomy of Types 11

12 Presentation Scope A Taxonomy of Types 12

13 Presentation Scope Verification and Validation (V&V) Methods Quality Control (QC) Static Legend Dynamic In Scope T&E Test Evaluation Other Analysis Demonstration Inspection Certification Reuse Warantee Static Analysis Dynamic Analysis Desk Checking Inspection Review Walk- Through Peer Review Formal Review A Taxonomy of Types 13

14 The Taxonomy of Types A Taxonomy of Types 14

15 Types of A type of testing is: A specific way to perform testing A class or subclass of testing Much narrower in scope than a testing paradigm There are relationships between the various types of testing. Most testers know: A lot about a few types of testing A little about some additional types of testing Very little about a sizable number of testing types A Taxonomy of Types 15

16 Polling Question 2 Have you ever seen a taxonomy of testing types (i.e., a hierarchical categorization of different ways to test)? O Yes O No O Not Sure A Taxonomy of Types 16

17 Types of What-Based Test Types (What Gets Tested) Object Under Test (OUT)-Based Test Types Domain-Based Test Types 16 Categories of Types Answering the 5W+2H Questions: What? When-Based Test Types (When Occurs) Temporal-Order-Based Test Types Lifecycle-Based Test Types Phase-Based Test Types Built-In-Test (BIT) Types When? Where? Where-Based Test Types (Why Occurs) Organization-Location-Based Test Types Physical-Location-Based Test Types Who? Why? How? How Well? Test Types Who-Based Test Types (Who Does ) Why-Based Test Types (Why Occurs) Collaboration-Based Test Types Organization-Based Test Types Role-Based Test Types Driver-Based Test Types Reason-Based Test Types These supertypes are not disjoint (think multiple inheritance)! How-Based Test Types (How is Performed Automation-Based Test Types Level-of-Scripting-Based Test Types Technique-Based Test Types How-Well-Based Test Types (Quality Verified) Quality-Characteristic-Based Test Types A Taxonomy of Types 17

18 Types of WHAT is Tested A Taxonomy of Types 18

19 What: by Object Under Test (OUT) What-Based OUT-Based Model Hardware Software System Data Center Tool/Environment A Taxonomy of Types 19

20 What: by Object Under Test (OUT) Model What-Based OUT-Based Model Requirements Model Architecture Model Design Model A Taxonomy of Types 20

21 What: by Object Under Test (OUT) Hardware What-Based OUT-Based Hardware Continuity Hardware Stress Highly Accelerated Life (HALT) HW Qualification Power-off A Taxonomy of Types 21

22 What: by Object Under Test (OUT) Software What-Based OUT-Based Software SW Unit [Component] SW Integration SW Application A Taxonomy of Types 22

23 What: by Object Under Test (OUT) System What-Based OUT-Based System Subsystem System Integration System SoS Integration SoS Hardware-in-the-Loop (HIL) Human-in-the-Loop (HIL) Processor-in-the-Loop (PIL) Software-in-the-Loop (SIL) A Taxonomy of Types 23

24 What: by Object Under Test (OUT) Data Center What-Based OUT-Based Data Center Configuration Failover and Restore Integrated System (IST) Network Traffic A Taxonomy of Types 24

25 What: by Object Under Test (OUT) Tool / Environment What-Based OUT-Based Tool/Environment Development Tool Development Environment Test Tool Test Environment A Taxonomy of Types 25

26 What: by Domain What-Based Domain-Based Domain-Independent Domain-Specific Many Types of Domain- Independent Crash Exhaust Emissions Flight Live Fire Sea Trials Wind Tunnel A Taxonomy of Types 26

27 Types of WHEN Occurs A Taxonomy of Types 27

28 When: by Temporal Order When-Based Order-Based Order-By-Direction Order-By-Component Bottom-Up Top-Down Outside-In Feature- Based Layer-Based Subsystem- Based A Taxonomy of Types 28

29 When: by Lifecycle When-Based Lifecycle-Based Waterfall Incremental Continuous (CT) At-The-End V-Model Agile DevOps A Taxonomy of Types 29

30 When: by Phase When-Based Phase-Based Developmental (DT) Acceptance (AT) Operational (OT) Many Types of Developmental Business Acceptance (BAT) Operational Effectiveness Operational Suitability Initial Operational (IOT) Follow-on Operational (FOT) Beware of Synonyms and Almost Synonyms! Customer Acceptance (CAT) Contract(ual) Acceptance (CAT) Factory Acceptance (FAT) Operational Acceptance (OAT) Production Acceptance (PAT) Regulation Acceptance (RAT) Data Center Qualification Operational (QOT) Site Acceptance (SAT) User Acceptance (UAT) A Taxonomy of Types 30

31 When: by BIT Execution Time When-Based Built-In (BIT) Power-Up Built-In (PupBIT) Interrupt-driven Built-In (IBIT) User-initiated Built-In (UBIT) Ongoing Built-In (OBIT) Periodic Built-In (PBIT) Shutdown Built-In (SBIT) Self- via Assertion Checking Self- via Prognostics and Health Management (PHM) Subsystem Self- via Heartbeat A Taxonomy of Types 31

32 Types of WHY is Being Performed A Taxonomy of Types 32

33 Why: by Driver Why-Based Driver-Based Regulatory- Compliance (tests compliance) Needs- Driven (tests why) Requirements-Driven (tests what + how-well) Architecture- Driven (tests how) Design-Driven (tests how) Validates Verifies Verifies Verifies Verifies System Complies with Laws or Regulations System Meets Stakeholder Needs System Meets System Requirements System Conforms to Architecture System Conforms to Design A Taxonomy of Types 33

34 Why: by Reason Why-Based Reason-Based Smoke Reuse Initial Retesting Regression Error Seeding COTS Open Source Legacy Partial Regression Complete Regression A Taxonomy of Types 34

35 Types of WHO Performs A Taxonomy of Types 35

36 Who: by Collaboration Who-Based Collaboration-Based Individual Group Buddy Flash Mob Pair A Taxonomy of Types 36

37 Who: by Organization Who-Based Organization-Based Development Organization Acquisition Organization Independent Test Organization Operations Organization User Organization Prime Contractor Subcontractor COTS Vendor DT Organization OT Organization A Taxonomy of Types 37

38 Polling Question 3 Who performs testing on your projects? Check all that apply. Project-internal Testers Independent Testers Developers Specialty Engineers (e.g., performance, reliability, safety, security, human factors) Quality Engineers Others A Taxonomy of Types 38

39 Who: by Role Who-Based Role-Based Developer Tester Operator User Requirements Engineer Architect Programmer Human Factors Engineer Safety Engineer Alpha Tester Embeded Tester Independent Tester Buddy Pair Database Admin Network Admin Sys Admin Beta Tester User as Tester Closed Beta Open Beta Security Engineer A Taxonomy of Types 39

40 Types of WHERE is Performed A Taxonomy of Types 40

41 Where: by Organizational Location Where-Based Organizational-Location-Based Insourced Outsourced A Taxonomy of Types 41

42 Where: by Physical Location Where-Based Physical-Location-Based Cloud Distributed Local A Taxonomy of Types 42

43 Types of HOW is Performed A Taxonomy of Types 43

44 How: by Level of Automation How-Based Level-of-Automation-Based Manual Automated - Test Script Generation - Test Data Generation - Test Script Execution - Test Report Generation Record-Playback Script-Based Data-Driven Action-Keyword Model-Based A Taxonomy of Types 44

45 How: by Level of Scripting How-Based Level-of-Scripting-Based Scripted Unscripted Exploratory Monkey A Taxonomy of Types 45

46 How: by Technique How-Based Test-Technique-Based Blackbox Graybox Whitebox Patterns- Based Experience- Based Random (Cat on the Keyboard) A Taxonomy of Types 46

47 How: by Technique - Blackbox How-Based Test-Technique-Based Blackbox Cause and Effect Combinatorial End-to-End Requirements Scenario Syntax Classification Tree Decision Table Manual Procedure Risk-Based State-Based User Interface Navigation A Taxonomy of Types 47

48 How: by Technique - Graybox How-Based Test-Technique-Based Graybox Boundary Value Equivalence Class A Taxonomy of Types 48

49 How: by Technique - Whitebox How-Based Test-Technique-Based Whitebox Data Flow Control Flow All Definitions All Uses Branch Condition Statement A Taxonomy of Types 49

50 How: by Technique - Experience- Based How-Based Test-Technique-Based Experience-Based Bug Hunt Error Guessing Exploratory Galumphing A Taxonomy of Types 50

51 How: by Technique - Random How-Based Test-Technique-Based Random (Cat on the Keyboard) Fuzz Monkey Shoe Stuck Key A Taxonomy of Types 51

52 Types of HOW WELL Object Under Test Functions A Taxonomy of Types 52

53 How Well: by Quality Characteristic How-Well-Based Quality-Characteristic-Based by Quality Characteristic Capacity Based on the associated quality characteristic and its associated quality attributes: Uncover related defects Determine level of quality Compatibility Configuration Consistency Correctness Flexibility Functionality Interoperability Performance Reliability Robustness Safety Scalability Security Usability Backwards Compatibility Infrastructure Compatibility Mobile Data Migration Internationalization Personalization A Taxonomy of Types 53

54 How Well: by Quality Capacity How-Well-Based Quality-Characteristic-Based Capacity Load Stress Volume A Taxonomy of Types 54

55 How Well: by Quality Reliability How-Well-Based Quality-Characteristic-Based Reliability Endurance (Stability) Reliability Enhancement Reliability Growth Reliability Mechanism A Taxonomy of Types 55

56 How Well: by Quality Robustness How-Well-Based Quality-Characteristic-Based Robustness Error Tolerance Fault Tolerance Failure Tolerance Environmental Tolerance Communication Error Hardware Error Human Error Software Error System Error Fault Injection Failover and Recovery Acceleration Tolerance Acoustic Tolerance Electromagnetic Compatibility (EMC) Tolerance Pressure/Leakage Tolerance Radiation Tolerance Shock/Drop Tolerance Temperature Tolerance Vacuum Tolerance Pressure Cycling Tolerance Temperature Cycling Tolerance A Taxonomy of Types 56

57 How Well: by Quality Security How-Well-Based Quality-Characteristic-Based Security Access Control Anti-Spoofing Anti-Tamper EMSEC Encryption Infrastructure Penetration Blue Team Red Team A Taxonomy of Types 57

58 How Well: by Quality Usability How-Well-Based Quality-Characteristic-Based Usability A/B Accessibility Alpha Beta Content Usage Closed Beta Open Beta (Flash) Mob A Taxonomy of Types 58

59 Conclusion A Taxonomy of Types 59

60 Conclusion Most systems require quite a few different types of testing. Most testers are not aware of the majority of the different types of testing. If you are not aware that it exists, then you don t know whether you need it. These types of testing can be organized into a taxonomy by the 5W + 2H questions. This taxonomy has several uses: Ensure the test strategy is sufficiently complete with no important type of testing overlooked. Organize testing types to make them and their relationships more understandable. Augment test training materials. Help categorize and understand limitations of testing tools. A Taxonomy of Types 60

61 A Taxonomy of Types 61