STPCon Fall 2013 October 24, 2013

Transcription

1 SESSION #1001 Improved Testing Using a Test Execution Model DOUG HOFFMAN, Software Quality Methods, LLC. Doug.Hoffman@acm.org, Models And Testing A model describes the elements and relationships of some idea or thing Testing is a process of investigation to learn something about quality A model provides a framework for understanding and planning testing A model encourages more complete evaluation of the elements involved in testing Douglas Hoffman Copyright , SQM, LLC. 2 Douglas Hoffman Page 1 of 12

2 Some Models Are Better Than Others To some degree For some purposes All models are wrong, but some are useful. George E. P. Box Doug Hoffman Copyright 2013 SQM, LLC. 3 Software Under Test (SUT) The SUT is the software that we intend to test. The purpose of the test is check one or more characteristics of this software. The SUT may be made up of one or multiple executables, be single or multithreaded, and is frequently nested with other software we need only to indirectly test. Douglas Hoffman Copyright , SQM, LLC. 4 Douglas Hoffman Page 2 of 12

3 Testing Is To Gain Information Running a test can be described in three steps: setup, inputs, and verdict generation: Setupprepares the data and conditions required by the test Inputs are the test exercise Resultsare checked to generate a pass/fail verdict Much more information is available Douglas Hoffman Copyright , SQM, LLC. 5 A Black Box Testing Model Test Inputs Test Results Pre-condition Data Pre-condition Program State Environmental Factors System Under Test Post-condition Data Post-condition Program State Environmental Outcomes Douglas Hoffman Copyright , SQM, LLC. 6 Douglas Hoffman Page 3 of 12

4 Test Inputs Test Inputsare the things done to stimulate the SUT; e.g., input values and mouse clicks for a GUI This is usually what we think of as the test (Most tests are designed based on a simple Input/Process/Output model of computer program execution) Douglas Hoffman Copyright , SQM, LLC. 7 Pre-Condition Data The Pre-Condition Datastate is the set of stored data values at the start of a test The behavior of the SUT is greatly influenced by values inside the program and in dedicated or shared data sets Initial values for internal data UID/password values Database contents Conditioning the data is often part of setup Douglas Hoffman Copyright , SQM, LLC. 8 Douglas Hoffman Page 4 of 12

5 Pre-Condition Program State The Pre-Condition Program Stateis the SUT s internal states when the test begins The behavior of the SUT is highly dependent on the program s internal state Screen display Permissions Internal state machines Program state is not always known or observable Douglas Hoffman Copyright , SQM, LLC. 9 Environmental Factors Environmental Factorsare other outside influences on the SUT behavior The factors fall into three broad categories: Conditions (e.g., user permissions, OS, or hardware configuration) Events (e.g., interrupts, data changes by outside factors, or network load) Sequences of events (e.g., the timing and order of events) Douglas Hoffman Copyright , SQM, LLC. 10 Douglas Hoffman Page 5 of 12

6 Results Resultsare the expected values and events checked to generate the initial pass/fail verdict from running the test The results are selected based on the test These are usually selected to show the positive outcome from the exercise Sometimes they are selected to check for anticipated possible errors Douglas Hoffman Copyright , SQM, LLC. 11 Post-Condition Data The Post-Condition Datastate is the set of stored values when the test completes The SUT has the potential for changing data outside the program in any accessible data sets Intended data Referenced data Unintended data The set of possibilities is infinite, especially considering the possible behavior of the SUT when bugs are present Douglas Hoffman Copyright , SQM, LLC. 12 Douglas Hoffman Page 6 of 12

7 Post-Condition Program State The Post-Condition Program Stateis the internal state after the test completes The program state is frequently delayed from the Inputs and not always observable The behavior of the SUT continues after the last input action from the test Processing time Resource and event requirements Program clean-up Douglas Hoffman Copyright , SQM, LLC. 13 Environmental Outcomes The Environmental Outcomesare influences the SUT manifests in the outside environment The outcomes are either persistent or transient The outcomes fall into two broad categories: Conditions (e.g., user permissions, value changes to OS environment variables, files left around, or memory leaks) Real-time events (e.g., generated interrupts, CPU utilization, or network load) Douglas Hoffman Copyright , SQM, LLC. 14 Douglas Hoffman Page 7 of 12

8 Implications From Influences The test exercise (Inputs) is often the easy part SUT behavior is driven by much more than the setup and input values There are many domains for influences There are an infinite number of possible influences We should monitor or control important influences Douglas Hoffman Copyright , SQM, LLC. 15 More Implications For Influences Not all influences are known or knowable Some influences require test hooks to monitor or control It is impossible to control all the influences Every test run is different because of different influences Non reproducible just means there are unknown influences Douglas Hoffman Copyright , SQM, LLC. 16 Douglas Hoffman Page 8 of 12

9 Implications For Outcomes Positive test results (a pass ) is not conclusive Negative initial results (a fail ) must be investigated before really knowing if a bug has been uncovered SUT [mis]behavior causes much more than just the expected results Not all outcomes are known or knowable Douglas Hoffman Copyright , SQM, LLC. 17 More Implications For Outcomes We should check as much as we can Global outcomes (e.g., files touched and memory leaks) Specific outcomes (e.g., the all of data sets, program state, and CPU/network utilization) Some outcomes are transient and easily missed Some outcomes require test hooks to be visible There are infinite possibilities for outcomes Outcomes from one test become influences for subsequent tests Douglas Hoffman Copyright , SQM, LLC. 18 Douglas Hoffman Page 9 of 12

10 General Things To Do Control what you need to Better focus for the tests with more control Avoids many false alarms Monitor what you can Check generic and test specific elements Include program and system internals if possible Include test specific and general checking Douglas Hoffman Copyright , SQM, LLC. 19 Things To Do About Data Pre-define as much as possible A test should only create data required to get information for the test Limit created data use within the test whenever possible Share data between tests only when that is the point of the two tests Periodically check all the data Douglas Hoffman Copyright , SQM, LLC. 20 Douglas Hoffman Page 10 of 12

11 Things To Do About Program State Control or monitor at the start Control means setting the program state Monitor means logging as much information as practical Monitor for exceptions during testing Consider tests that set or reset the state Douglas Hoffman Copyright , SQM, LLC. 21 Things To Do About The Environment Control or monitor at the start Record invariants Record system state information Monitor during testing Touching things that shouldn t be touched Resource utilization Check for variations after completion Douglas Hoffman Copyright , SQM, LLC. 22 Douglas Hoffman Page 11 of 12

12 Conclusions Testing is improved by gathering more information We need to understand what is being tested Keep in mind that you only see part of the picture Control what you must Monitor what you can Check for unexpected outcomes Douglas Hoffman Copyright , SQM, LLC. 23 Douglas Hoffman Page 12 of 12