How to Improve Test Team Effectiveness Using Test Entities John Kent: john.kent@simplytesting.com
How to Improve Test Team Effectiveness using Test Entities
Discussions at the Test Retreat A work in progress Building Test Management Tools A Test Entity Model could be used for: Software Testing All types of systems testing
The Need for an Entity Model The major constraint on testing is lack of TIME Anything that improves productivity of testers would be welcome To Define Testing and the Language For test tools to be built which implement a real model of testing How can we train testers without a clear picture of testing?
Entity Relationship Modeling Entity Relationship Diagram - ER Diagram Entities Uses rectangle Relationships between entities Uses ellipse symbol Many shown as Song Entities have Attributes Singer Performs Song
Sources Standards BSI 7925 IEEE 829 Training Courses ISEB ISTQB Glossary Syllabus Experience Test Retreat discussions Discussions with testers
Where We Are Now In many organisations tests look like this: Test Script Test Steps Input Expected Output Expected Outcomes Inputs
IEEE 829 Entity Relationship Diagram No Specifications No Test Conditions Relationships ill-defined
Tries to describe all possible relationships not just one way of doing things We will build the model by: 1. Using a possible relationship example then 2. Make statements about the relationships
The Entities Test Entities include: Specifications Test Conditions Test Cases Test Scripts Test Steps Test Execution Schedule Test Results Not included here Defects (I.E. description of a defect) Test Plans Test Strategies Test Coverage Items
Entity: Specification Item An indivisible, testable part of a specification Also called Test Basis Requirement Specification User Reqt Legal Reqt Performance Reqt Security Reqt etc... Design/Functional Specification Attributes Description Risk Priority
Entity: Test Condition Definition ISTQB definition "An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element." 7925 definition not defined Attributes Description Risk Priority A statement of required system behavior under certain conditions A refinement of the specification (from a test perspective) Could be considered to be a specification item
Entity: Test Case Definition ISTQB: A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610] Attributes Described by IEEE 829 Input Expected Output Preconditions Objective Re-use?
Entity: Test Script Definition Also called Test Procedure 7925: A document providing detailed instructions for the execution of one or more test cases ISTQB: Test Procedure A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script. [After IEEE 829] Consists of Test Steps. A sequence of test cases?
Entity: Test Step Definition ISTQB: not defined 7925: not defined But they do exist I ve seen them!
Entity: Test Execution Schedule Test run sequence ISTQB: A scheme for the execution of test procedures. The test procedures are included in the test execution schedule in their context and in the order in which they are to be executed.
Entity: Test Result ISTQB: Result: The consequence/outcome of the execution of a test. It includes outputs to screens, changes to data, reports, and communication messages sent out.. 7925: Actual Outcome: The behavior actually produced when the object is tested under specified conditions.
Relationship: Specifications to Test Conditions Example an Asset Management System Spec: Assets Spec: Add Assets Together the Specifications and Test Conditions can make up a model of the system under test Spec: Delete Assets TCond: Test that cannot delete an Asset when used in an active work order TCond: Test that can delete an Asset when used in an inactive work order TCond: Test that can delete an Asset when used in an inactive job plan A Specification can link to many Test Conditions TCond: Test that cannot delete an Asset when used in an active job plan TCond: Test that can delete an Asset when not used in a job plan or work order
Relationship: Specifications to Test Conditions(Cont) Req Spec: Assets Req Spec: Add Assets A Test condition could link to many specifications Req Spec: Delete Assets TCond: Test that cannot delete an Asset when used in an active work order TCond: Test that can delete an Asset when used in an inactive work order Design Spec: Assets Design Spec: Delete Asset Menu item TCond: Test that can delete an Asset when used in an inactive job plan TCond: Test that cannot delete an Asset when used in an active job plan TCond: Test that can delete an Asset when not used in a job plan or work order
Relationship: Test Conditions to Test Cases TCond: Test that can delete an Asset when used in an inactive job plan The Objective (IEEE 829) attribute in this definition of a Test Case is the linked test conditions A Test Case can link to many Test Conditions TCond: Test that Asset is removed from the asset list when deleted TCase: Pre-condition: An asset linked to an inactive Job Plan Input: Select Delete Asset menu item Expected Outcome: Asset removed from list
Relationship: Test Conditions to Test Cases(cont) TCond: Test that Asset is removed from the asset list when deleted TCase: Pre-condition: An asset not linked to a Work Order or Job Plan Input: Select Delete Asset menu item Expected Outcome: Asset removed from list A Test Condition can link to many Test Cases TCase: Pre-condition: An asset linked to an inactive Work Order Input: Select Delete Asset menu item Expected Outcome: Asset removed from list TCase: Pre-condition: An asset linked to an inactive Job Plan Input: Select Delete Asset menu item Expected Outcome: Asset removed from list
Relationship: Test Scripts to Test Steps A Test Script can be contain many Test Steps A Test Step can be a Test Case in a Test Script A Test Step is not necessarily a Test Case Test Script 1: Asset Tests Test Step: 1 Select an job plan with Status= Active An Asset linked Test Step: 2 Pre-condition: An active job plan Input: Select Inactivate Asset Menu item Expected Outcome: Asset status=inactive Test Step: 3 Pre-condition: An asset linked to inactive JobP Input: Select Delete Asset menu item Expected Outcome: Asset removed from list
Relationship: Test Cases to Test Steps and Scripts A Test Script can consist of many Test Cases Test Script 1: Test Case: 1 Test Case: 2 Test Case: 2 A Test Case may appear in many Test Scripts This is test case re-use Test Step: 3 Test Script 2: Test Case: 7 Test Case: 2 Test Step: 9
Relationship: Test Scripts to Test Execution Schedules Test Script1: Test Script 1 Test Execution Schedule 1 Test Script 7 Test Script 8 Test Script2: Test Execution Schedule 2 Test Script 2 Test Script 1 Test Script 7 A Test Execution Schedule can contain many entire Test Scripts An entire Test Script can be used in many Test Execution Schedules
Relationship: Test Steps to Test Execution Schedules Test Script 1: Test Step: 1 Test Step: 2 Test Step: 3 A Test Execution Schedule can contain many Test Steps Test Execution Schedule 1 Script 1 - Test Step: 1 Script 1 - Test Step: 2 Script 2 - Test Step: 2 Test Script 2: Test Step: 1 Test Step: 2 Test Step: 3 Test Execution Schedule 2 Script 1 - Test Step: 2 Script 1 - Test Step: 3 A Test Step can be used in many Test Execution Schedules Script 2 - Test Step: 2
Relationship: Test Execution Schedules to Test Results Test Execution Schedule Test Results: Run 1 Script 1 - Test Step 1 Test Results: Run 2 Script 1 - Test Step: 2 Test Results: Run 3 Script 2 - Test Step: 2 A Test Execution Schedule can appear in many Test Results
The Full Test Entity Model Specification Item Test Condition Test Case Test Step Test Execution Schedule Test Script Test Results
The Full Test Entity Model Specification Item Test Condition Test Execution Results must be traceable back through all of the relationships to the specifications or requirements Test Case Test Step Test Execution Schedule Test Script Test Results
The major difference between theory and Real World is the Real World contains time and therefore work we do is incomplete It takes time to build the entities It takes time to build the relationships between entities What test entities we build depend on the time we have The level of detail in specifications and tests is dependent upon time Requirements are incomplete Test conditions are also incomplete
Real World Waterfall Model Produce Requirements which are incomplete and slightly wrong Because Requirements are incomplete we need Test Conditions to describe the system we produce a list of incomplete Test Conditions Specify the Syst Design incompletely with errors Code what you can from specs and make up the rest. Bugs always included
Additions to the Full Test Entity Model We could get greater test coverage by producing more test conditions which we could execute directly. Writing TCs gives faster test coverage Specification Item Test Condition Test Case Test Step We have seen that TCs are a form of spec so it would be possible to execute the specifications directly in some test projects Test Execution Schedule Test Script Test Results
Simplified Test Entity Model 1 Specification Item Test Condition Test conditions executed directly using testers knowledge to provide missing steps Test Execution Schedule Test Results
Simplified Test Entity Model 1 Specification Item They may be better off doing this Test Condition This is where many test organisations are Test Step Test Execution Schedule Test Script Test Results
Simplified Test Entity Model 2 Many test teams don t use this bit So they cannot measure test coverage against requirements Specification Item Test Condition Test Script Test Step Entity Model used in Quality Center and Test Director Introduces additional relationships to the Full Model: TCond to Test Script Spec to Test Script Test Execution Schedule May be difficult to implement full model in a database tool lots of many to many relationships Test Results
Simplified Test Entity Model 3 Where the specifications are perfect or Specification Item you don t have enough time Exploratory Testing? Test Execution Schedule Test Results
Simplified Test Entity Model 4 Exploratory Testing may look like this: Specification Item Test Results
The Fuller Test Entity Model Specification Item Test Condition Test Case Test Step Test Execution Schedule Test Script Test Results
Summary
Summary Full Entity Model Not Completely Defined Entities not well defined Relationships not well defined Attributes not defined Risk? Entities left out, Test Coverage, Defects, Test Plans Once we understand the test entity model we may be able to see more/better ways to test In the real world we are likely to use a simplified version of the Entity Model
Summary The most scarce commodity in testing is time Therefore we should create tests and execute them in the most time efficient way we can An Entity Model of Testing can help us do this