L7: Tes(ng. Smoke tes(ng. The test- vee Black- box vs. white- box tes(ng Tes(ng methods. Four levels of tes(ng. Case study

Transcription

1 Smoke tes(ng L7: Tes(ng The test- vee Black- box vs. white- box tes(ng Tes(ng methods Matrix test Step- by- step test Automated test scripts Four levels of tes(ng Debugging Unit tes?ng Integra?on tes?ng Acceptance tes?ng Case study Capstone design 1

2 Mo(va(on Development is accompanied by bugs Catching bugs early saves money The further a bug progresses the more impact it has on the system A bug fix requires all related modules to be retested A bug fix may require redesigning related modules For example PCB design flaw VLSI layout error Subtle coding flaw Tes(ng doesn t remove bugs, it just makes it less likely they exist Capstone design CSE@TAMU 2

3 One approach to tes(ng Smoke tes/ng: Turn on the system and see if it works Magic smoke: It's the blue smoke that makes it work let out the blue smoke and it won't do anything Capstone design 3

4 Instead, tes(ng should proceed with the design process Write tests while designing modules Perform tests while implemen?ng modules The Test- Vee illustrates this process Tes?ng is no just debugging four levels of tes?ng For every stage of development, there is a corresponding test stage Four levels of tes(ng Capstone design CSE@TAMU 4

5 Why develop test cases? Test cases Test cases define exactly what the module must do Tes?ng prevents feature creep, since the development of a module is complete when its test is passed Test cases mo?vate developers by providing immediate feedback Test cases force designers to think about extreme cases Test cases are a form of documenta?on Test cases force the designer to consider the design of the module before building it Capstone design CSE@TAMU 5

6 Proper(es of test cases Accurate: The test should check what it is supposed to and exercise an area of intent Economical: should be performed in a minimal number of steps Limited in complexity: should consist of a moderate # steps (10-15) Repeatable: Should be able to be repeated by another person Appropriate: its complexity should be such that other individuals assigned the tes?ng task can perform it Traceable: The test should verify a specific requirement Self cleaning: The system should return to the pre- test state a]er the test is complete Capstone design CSE@TAMU 6

7 Types of tes(ng Black box Performed with no knowledge of internal organiza?on Assumes one only has access input and outputs Change inputs, then compare outputs to expected values Enumera?ng every possible input/output combina?on is imprac?cal Unfortunately, minimizing the number of test cases requires understanding of the system s internal organiza?on White box Conducted with knowledge of internal organiza?on Test cases are built to target specific internal nodes of the system Might have expecta?on of fault model Create test instance which reveal physical or logical errors Capstone design CSE@TAMU 7

8 Addi(onal concepts Controllability When any node of the system can be set to a desired value Black box has no controllability Observability When any node of the system can be measured Black box has low observability Test stubs A stub is a device used to simulate a subcomponent of a system Why use stubs? The subcomponent may not have been built The risk or cost of damaging the subcomponent are too high Examples A func?on generator for a audio input A prine() instead of a file write DIP switch instead of a bus connec?on Capstone design CSE@TAMU 8

9 Example: tes(ng a BJT amplifier Input simulated with a func?on generator with RC network Output simulated by a resistor whose value is that of typical resis?ve loads which the amplifier will have to drive Capstone design CSE@TAMU 9

10 Debugging process Tips Debugging Observe the problem under different opera?ng condi?ons Form a hypothesis as to what the poten?al problem is Conduct experiments to confirm or eliminate the hypothesized source Repeat un?l the problem is eliminated Start with the simplest and easiest poten?al problems first; why? They are easier to perform You can perform more of them in a given period of?me Go from lowest level to highest of abstrac?on; why? The higher levels cannot operate unless the lower levels are working Capstone design CSE@TAMU 10

11 Types of bugs Bohrbugs Bohrbugs are reliable bugs The error is always in the same place An electron having a definite posi?on Solu?on = set a good trap Heisenbugs Innocuous changes of input yield buggy behavior May not be reproducible They seemingly move around within a system Electron is elusive density func?on Solu?on = think outside the box Example: code has a pointer error that occasionally overwrites the stack Capstone design CSE@TAMU 11

12 Unit tes(ng A complete test of a module s func(onality in isola(on Consists of a set of test cases Each test case establish that a subsystem performs a single unit of func?onality to some specifica?on Should be wrinen with the express intent of uncovering undiscovered defects Example Conver?ng Celsius into Fahrenheit How many test cases do we need? One for each clause (if/else) Boundary condi?ons between the two clauses Extreme values if (16 < input < 32) output = ROM[input-16]; else output = (9* input)/5 + 32; Capstone design CSE@TAMU 12

13 Matrix test Tes(ng methods Best suited to cases where inputs are structurally similar and only differ in their value Example: tes?ng an analog- to- digital converter Each tests varies only in the value of the input, and verifying the output Capstone design 13

14 Step- by- step tests Most effec?ve when the test consists of a sequence of steps A test case is a prescrip?on for genera?ng a test and checking results Format is similar to matrix test with one addi?onal column Example: state diagram for the vending machine in an earlier lecture Capstone design CSE@TAMU 14

15 Automated test script A sequence of commands provided to the unit- under- test (UUT) without user interven?on Outputs are then automa?cally compared against expected outputs Carry a lot of up- front costs but pay dividends with regression tes?ng In regression tes(ng, you retest a module a]er modifying any related part of the system this ensures that no errors were accidentally introduced Matrix, step- by- step and automated test scripts are not limited to unit tes?ng, and can be used for integra?on tes?ng and acceptance tes?ng Capstone design CSE@TAMU 15

16 Integra(on tes(ng Performed axer each subsystems has passed unit tes(ng Integra?on tes?ng checks that the major modules of the system operate correctly together Test cases must be traceable to the high- level design Test cases can be derived by answering the following ques?ons What are the different paths of execu?on through the system? Are all modules exercised at least once during integra?on tes?ng? Have all the interface signals been tested? Have all the interface modes been exercised? Does the system process informa?on at the required rate and met?ming requirements? Capstone design 16

17 Acceptance tes(ng S(pulates the condi(ons under which the customer will accept the system Generally wrinen along with requirements Might be a formal legal document Acceptance tes?ng iden?fies Scope how much of the system is tested? Level how deep will tes?ng be performed? Traceable to engineering requirements The same proper?es of a good requirement (abstract, unambiguous, traceable, verifiable) apply to acceptance test cases A bit of a chicken- and- egg problem as well It is hard to s?pulate the test procedures and results for a system that has not been developed yet Capstone design CSE@TAMU 17

18 Case study: security robot design We will focus on two of the engineering requirements The robot s center must stay within cm of the wall over 90% of the course, while traveling parallel to a wall over a 3 m course The robot s heading should never deviate more than 10 from the wall s axis while traveling parallel to a straight wall over a 3 m course We will develop An acceptance test for the first engineering requirement An integra?on test for the level 1 architecture of the robot A matrix unit test for the digital compass Capstone design CSE@TAMU 18

19 Acceptance test Capstone design 19

20 Level 1 design architecture Capstone design CSE@TAMU 20

21 Func(onal requirements for compass module Capstone design 21

22 Integra(on test #1 Capstone design 22

23 Matrix unit test for digital compass Capstone design 23