So#ware Tes+ng. See also Sommerville, Chapter 8 Amman and Offut, Introduc)on to So,ware Tes)ng, Cambridge University Press, 2008.

Transcription

1 So#ware Tes+ng See also Sommerville, Chapter 8 Amman and Offut, Introduc)on to So,ware Tes)ng, Cambridge University Press, 2008.

2 Tes+ng Most basic form of post- hoc SQA Helps define program func+onality Can be used as documenta+on (c.f. XP) Must ensure program is testable (c.f. PSS docs) Methods become callable Modules get looser coupling

3 Basic Terminology The system under test (SUT) can be of different types (procedural, reac+ve, etc.) A test case consists of a single input for the SUT A verdict is a judgment a#er a test case terminates pass/fail/warning/don t know An oracle is a method to produce verdicts A test suite is a set of test cases How much have we tested? coverage measures!

4 User Requirements Test cases Acceptance tes+ng So#ware Requirements Test cases System tes+ng Architecture Design Test cases Integra+on tes+ng Detailed design & Coding Test cases Unit Tes+ng The V model Integrates design and tes+ng Time 4

5 Structural Tes+ng (Glass or white box) An error must exist along a path If tests don t exercise that path then error can never be observed So iden+fy and exercise each path No loops finitely many paths good! Loops infinitely many paths bad! Loops + branches exponen+al growth in path numbers with loop depth very bad!!!!

6 Structural Tes+ng - Problems What about sins of omission? no path to go down! What about dead code is a path possible? How to avoid redundant tes+ng?

7 Condensa+on Graph Start Assignments1 Cond1 Assignments2 Cond2 Assignments4 Assignments3 Path analysis: 1&3, 1&4, 1&2&3, 1&2&4 etc Halt

8 Structural Tes+ng Requirements These are structural requirements on a test suite They ignore func+onality! Easy to define using graph theory Possible to automate genera+on by constraint solving Easy to measure coverage!

9 Graph Coverage A path is a sequence of nodes n 0,, n k in a (condensa+on) graph G, such that each adjacent node pair, (n i, n i+1 ) forms an edge in G. A test requirement tr(.) is a predicate on paths Node n ( p ) p has node n Edge e (p) p has edge e

10 Graph Coverage Defini5on: Given a set TR of test requirements for a graph criterion C, A test suite T sa+sfies C on graph G if, and only if for every test requirement tr(.) TR, there is at least one path p in G such that tr(p) is true (i.e. p sa+sfies tr(.))

11 Why so Formal? Answer: Some+mes coverage proper+es become very technical to define for efficiency reasons.

12 Structural Coverage Criteria (Amman and Offut, Chapter 2) 2.1. Node Coverage (NC) TR contains each reachable node in G Edge Coverage (EC) TR contains each reachable path of length Edge- Pair Coverage (EPC) TR contains each reachable path of length 2.

13 2.7. Complete Path Coverage (CPC) TR contains all paths in G Specified Path Coverage (SPC) TR contains a set S of test paths, where S is supplied as a parameter. Example. S contains paths that traverse every loop free path p in G and every loop in G both 0 and 1 +mes.

14 Func+onal Tes+ng (Black- box Tes+ng) Black- box = structure of code is invisible + Tests the specifica+on not the code + Insensi+ve to code refactoring - Hard to find test verdicts aka oracle problem - Hard to define coverage - Huge volume of tes+ng user profiles? - Use cases are an excellent source of tests

15 Random Tes+ng Generate input vectors (or sequences) at random, fire into system and observe. Easy or tricky to implement Low level data types e.g. Int easy High level data types e.g. graphs tricky High volume of test cases but is that good structural coverage? Good for low input dimension 1-5? But poor for high input dimension

16 Random Tes+ng Oracle step is difficult to automate without precise requirements Set up and tear down must also be considered Does random distribu+on match expected distribu+on? Are some data combina+ons meaningless? Data interdependencies and constraints! Example consider calendar combina+ons: Year/month/day/day of week 1961/02/29/Wednesday is this legal or not? Can try to filter out bad data but this can be very slow

17 k- wise tes+ng Some components have fixed input size k e.g. a method mymethod(x:string,y:char,z:int) We need test vectors of length k i.e. ( i 1,, i k ) Choose (say) C=10 input values V i = { v i 1,,v i c } for each 1 i k Test suite S k is all combina+ons of V 1,, V k Cartesian Product S = V 1 V k Test suite size S is then C k = 10 k Probably not feasible for S 10,000

18 Example Suppose k = 3, C = 2, then C k = 2 3 = 8 V 1 = { Sat, Sun } V 2 = { A, B } V 3 = { 1, 2 } S 3 = {(Sat, A, 1), (Sun, A, 1), (Sat, B, 1), (Sun, B, 1), (Sat, A, 2), (Sun, A, 2), (Sat, B, 2), (Sun, B, 2) } Clearly S 3 has size 8 In general S k has size C k which is exponen+al in k.

19 n- Wise Tes+ng for n k For 1 i k, let c i be a fixed value from V i An n- wise test is a vector ( i 1,, i k ) such that n elements are chosen from V 1,, V k and the rest are chosen from c 1,, c k. Let S n be the set of all n- wise tests.

20 Example: 1- wise tes+ng Suppose n = 1 and let c 1 = Sat, c 2 = A, c 3 = 1 V 1 = { Sat, Sun } V 2 = { A, B } V 3 = { 1, 2 } S 1 = {(Sat, A, 1), (Sun, A, 1), (Sat, B, 1), (Sat, A, 2) } Clearly S 1 has size k*(c- 1)+1 which is linear in k. So S 1 is small but has limited coverage.

21 Example: 2- wise tes+ng (aka all- pairs or pairwise tes)ng) Suppose n = 2 and c 1 = Sat, c 2 = A, c 3 = 1 V 1 = { Sat, Sun } V 2 = { A, B } V 3 = { 1, 2 } S 2 = {(Sat, A, 1), (Sun, A, 1), (Sat, B, 1), (Sun, B, 1) } (Sat, A, 2), (Sun, A, 2), (Sat, B, 2) } Clearly S 2 has size 7 which is not much smaller than S 3. In general S 2 is much smaller than S k. S 2 grows O(C 2 ), which is much slower than C k!

22 Why Pairwise Tes+ng? Bugs involving interac+ons between three or more parameters are progressively less common[2]. NASA database applica+on. 67 percent of the failures were triggered by only a single parameter value, 93 percent by two- way combina+ons, and 98 percent by three- way combina+ons [13]. 10 UNIX commands. Cohen et al. showed that the pairwise tests gave over 90 percent block coverage [9]. Medical so#ware devices. Only 3 of 109 failure reports indicated that more than two condi+ons were required to cause the failure [14].

23 Why Pairwise Tes+ng Browser and server. More than 70 percent of bugs were detected with two or fewer condi+ons (75 percent for browser and 70 percent for server) and approximately 90 percent of the bugs reported were detected with three or fewer condi+ons (95 percent for browser and 89 percent for server) [13]. User interface so#ware at Telcordia. Studies [8] showed that most field faults were caused by either incorrect single values or by an interac+on of pairs of values. Their code coverage study also indicated that pairwise coverage is sufficient for good code coverage. Established tools, e.g. PICT See

24 Test Cases from Use Cases Instan+ate a scenario with concrete data values, and expected results. Different flows lead to different use cases Sunny day and rainy day scenarios Use graph coverage to measure use case coverage Structured and easy to use Natural focus on most significant use cases Good approach to system and acceptance tes+ng, but may be difficult and unit and integra+on levels

25 UseCaseName: PurchaseTicket Precondi5on: The passenger is standing in front of +cket distributor and has sufficient money to purchase a +cket. Sequence: 1. The passenger selects the number of zones to be travelled, If the passenger presses mul+ple zone buxons, only the last buxon pressed is considered by the distributor. 2. The distributor displays the amount due 3. The passenger inserts money

26 4. The passenger selects a new zone before inser+ng sufficient money, the distributor returns all the coins and bills inserted by the passenger 5. If the passenger inserted more money than the amount due the distributor returns excess change. 6. The distributor issues +cket. 7. The passenger picks up the change and +cket.

27 TestCaseName: PurchaseTicket_SunnyDay Precondi5on: The passenger is standing in front of +cket distributor and has two 5 notes and 3 * 10 Cent coins Sequence: 1. The passenger presses in succession the zone buxons 2, 4, 1 and 2 2. The distributor should display in succession the fares , 0.75 and The passenger inserts a 5 note 4. The distributor returns 3*1 coins, 75Cent and a 2- zone +cket.

28 Postcondi5on The passenger has one 2- zone +cket We should also derive test cases that exercise rainy day scenarios (when something goes wrong) to test robustness.

29 Unit Tes+ng with JUnit Developed by the XP community 2002 Framework for automa+ng the execu+on of unit test for Java classes Write new test cases by subclassing the TestCase class Organise TestCases into TestSuites Automates tes+ng process Built around Command and Composite paxerns

30 Why use Junit? Junit +ghtly integrates development and tes+ng, supports the XP approach Allows you to write code faster while increasing quality (???) Can refactor code without worrying about correctness JUnit is simple. Easy as running the compiler on your code

31 JUnit tests check their own results (oracle step) and provide immediate feedback No manual comparison of expected with actual Simple visual feedback JUnit tests can be composed into a hierarchy of test suites Can run tests for any layer in the hierarchy Wri+ng JUnit tests is inexpensive No harder than wri+ng a method to exercise the code JUnit tests increase the stability of so#ware More tests = more stability

32 Junit tests are developer tests Tests fundamental building blocks of system Tests delivered with code as a cer+fied package Junit tests are wrixen in Java Seamless bond between test and code under test Test code can be refactored into so#ware code and vice- versa Data type compa+bility (float, double etc.) Junit is free

33 Junit Design A TestCase is a Command object A class of test methods subclasses TestCase A TestCase has public testxxx() methods To check expected with actual output invoke assert() method Use setup() and teardown() to prevent side effects between subsequent testxxx() calls

34 Test Run(TestResult) Test testname:string run(testresult) setup() teardown() runtest() TestSuite addtest() Run(TestResult) Test setup() teardown() run(testresult)

35 TestCase objects can be composed into TestSuite hierarchies. Automa+cally invoke all the testxxx() methods in each object A TestSuite is composed of TestCase instances or other TestSuite instances Nest to arbitrary depth Run whole TestSuite with a single pass/fail result Get your own installa+on instruc+ons

36 Wri+ng a Test Case Define a subclass of TestCase Override the setup() method to in+alise object(s) under test Op+onally override the teardown() method to release objects under test Define 1 or more public testxxx() methods hat exercise the object(s) under test and assert expected results

37 Import junit.framework.testcase Public class ShoppingCartTest extends TestCase{ Private ShoppingCart cart; Private Product book1; Protected void setup() { Cart = new ShoppingCart(); Book1 = new Product( mytitle, 50 ); Cart.addItem(book1) }

38 Protected void teardown() { //release objects under test here if necessary } Public void testempty() { Cart.empty(); // empty out cart assertequals(0,cart.getitemcount() ); }

39 Public void testadditem() { Product book2 = new Product( +tle2, 65 ); cart.additem(book2); double expectedbalance = book1.getprice() + book2.getprice(); assertequals(expectedbalance, cart.getbalance ()); assertequals(2, cart.getitemcount() ); }

40 Public void testremoveitem() throws productnotfoundexcep+on { Cart.removeItem(book1); assertequals(1, cart.getitemcount() ); } Public void testremoveitemnotincart() { try{ Product book3 = new Product( +tle3, 10 ); Cart.removeItem(book3); fail( should raise a ProductNotFoundExcep+on ); } Catch(ProductNotFoundExcep+on expected) { //passed the test! } } // of class ShoppingCartTest