TDDD04 Software Testing

Transcription

1 TDDD04 Software Testing Lecture Notes 5 March June 2010 Mohsen Torabzadeh-Tari (presenter), (originator) Department of Computer and Information Science Linköping University, Sweden

2 Integration Testing 2 Literature: Software testing, A craftsman's Approach, Paul C. Jorgensen (Grand Valley State University, Michigan) In September 1999, the Mars Climate Orbiter mission failed after successfully traveling 416 million miles in 41 weeks. It disappeared just as it was to being Orbiting Mars. The fault should have been revealed by integration testing: Lockheed Martin Astronautics used acceleration data in English unit (pounds), while the Jet Propulsion Laboratory did its calculations with metric units (newtons). NASA announced a $50,000 project to discover how this could have happened (Fordahl 1999).

3 Reflection 3 Mention drawbacks of functional decomposition integration What is the break-even point of the SATM example for the Top-Down and Bottom-Up methods in terms of the cost of the stubs compared to drives? (for SATM, stubs 32, drives 10 needed) What is the advantage of Sandwich integration compared to TD and BU integrations? Mention three factors that could have impact on the choice of integration technique.

4 Unit & Integration Testing 4 Objective: to ensure that code implemented the design properly. Code = System Design Specification

5 5 Component code Unit test Tested components Design Specification Integration test Component code Unit test Tested components Integrated modules

6 Stubs and Drivers 6 Stubs: throw-away piece of code that takes place of the actual code. It emulates a called unit Drivers: throw-away code that emulate the procedure calls Component to be tested driver Boundary conditions independent paths interface... stub stub Test cases

7 Levels of Testing 7 Requirements Specification System Testing Preliminary Design Integration Testing Detailed Design Unit Testing Coding

8 Goals/Purpose of Integration Testing 8 presumes previously tested units not system testing (at level of system inputs and outputs) tests functionality "between" unit and system levels basis for test case identification emphasis shifts from how to test to what to test

9 Outline of the Lecture 9 Approaches to Integration Testing 1. Based on Functional Decomposition Top-Down Bottom-up Sandwich Big Bang 2. Based on Call Graph Pair-Wise Neighborhood 3. Based on Paths MM-Paths

10 Example: The SATM System 10 The Simple Automated Teller Machine example here contains an interesting variety of functionality and interactions that typify the client side of client-server systems. The SATM system communicates with bank customers via 15 screens, using a terminal. Customers can select any of three transaction types: deposits, withdrawals, and balance inquiries. These transactions can be done on two types of accounts: checking and savings.

11 Decomposition-Based Integration

12 Functional Decomposition of the SATM System 1 12 A 10 B C D E F Unit Level Unit Name 1 1 SATM System A 1.1 Device Sense & Control D Door Sense & Control Get Door Status Control Door Dispense Cash E Slot Sense & Control WatchCardSlot Get Deposit Slot Status Control Card Roller Control Envelope Roller Read Card Strip Central Bank Comm Get PIN for PAN Get Account Status Post Daily Transactions Table 1: SATM Units and Abbreviated Names Unit Level Unit Nam B 1.3 Terminal Sense & Control Screen Driver Key Sensor C 1.4 Manage Session Validate Card Validate PIN GetPIN F Close Session New Transaction Request Print Receipt Post Transaction Local Manage Transaction Get Transaction Type Get Account Type Report Balance Process Deposit Process Withdrawal

13 Decomposition-Based Integration 13 Top-Down Bottom-up Sandwich Big Bang All these integration orders presume that the units have been separately tested, thus, the goal of decomposition-based integration is to test the interfaces among separately tested units.

14 Big Bang Integration 14 In this view of integration, all the units are compiled together and tested at once. The drawback to this is that when a failure is observed, few clues are available to help isolate the location (s) of the fault.

15 Decomposition-Based Integration Top-Down Integration

16 Top-Down Integration 16 Top Subtree Second Level Subtree Botom Level Subtree

17 Top-Down Integration Student worksheet 17 Top Subtree x stubs y integration sessions*; in each, one component would be actual code, and the other three would be stubs. * a test session is one set of tests for a specific configuration actual code and stubs.

18 Top-Down Integration 18 Top Subtree 4 stubs 5 integration sessions; in each, one component would be actual code, and the other three would be stubs.

19 Top-Down Integration 19 Student worksheet Second Level Subtree x stubs y integration sessions

20 Top-Down Integration 20 Second Level Subtree 15 stubs ( ) 20 integration sessions (15 stubs + 4 roots (subtrees) +1)

21 Top-Down Integration 21 Student worksheet Botom Level Subtree x stubs y integration sessions

22 Top-Down Integration 22 Botom Level Subtree 32 stubs 42 integration sessions (32 stubs + 9 roots (subtrees) + 1 (with all the stubs))

23 23 Would we replace all the stubs at once? If we did, we would have a small bang for units with a high outdegree. If we replace one stub at a time, we retest the main program once for each replaced stub.

24 Decomposition-Based Integration Bottom-Up Integration

25 Bottom-Up Integration 25 Bottom Level Subtree Second Level Subtree Top Subtree

26 Bottom-Up Integration 26 Student worksheet Bottom Level Subtree x drivers y integration sessions

27 Bottom-Up Integration 27 Bottom Level Subtree 5 drivers ( 5 rotts (subtrees)) 17 integration sessions ( )

28 Bottom-Up Integration 28 Student worksheet Second Level Subtree x drivers y integration sessions

29 Bottom-Up Integration 29 Second Level Subtree 9 drivers (9 subtrees) 28 integration sessions ( )

30 Bottom-Up Integration 30 Student worksheet Top Subtree x drivers y integration sessions

31 Bottom-Up Integration 31 Top Subtree 10 drivers (9 subtrees + 1 (main)) 42 integration sessions ( roots (subtrees) + 1 (with all the drives))

32 32 Sessions = nodes leaves + edges For top-down integration: (nodes - 1) stubs are needed For bottom-up integration: (nodes - leaves) drivers are needed

33 33 Student Worksheet For SATM: nodes:? leavs:? edges:? Sessions = nodes leaves + edges For top-down integration: (nodes - 1) stubs are needed For bottom-up integration: (nodes - leaves) drivers are needed

34 34 For STAM: nodes: 33 leavs: 23 edges: 32 Sessions = nodes leaves + edges = 42 sessions For top-down integration: (nodes - 1) stubs are needed 33-1 = 32 stubs For bottom-up integration: (nodes - leaves) drivers are needed = 10 drivers

35 Decomposition-Based Integration Sandwich Integration

36 Sandwich Integration 36

37 Sandwich Integration 37 A combination of top-down and bottom-up integration Less stub and drivers development effort Added difficulty to fault isolation

38 Decomposition-Based Integration Pros & Cons

39 Pros and Cons Functional decomposition-based approaches 39 Whenever a failure is observed, the most recently added unit is suspected. The top-down and bottom-up terms suggest breadth-first traversals of the decomposition tree, but this is not mandatory. One of the most frequent objections to functional decomposition and waterfall development is that both are artificial, and both serve the needs of project management more than the needs of software developers. This holds true also for decomposition-based testing. The whole mechanism is that units are integrated with respect to structure; this presumes that correct behavior follows from individually correct units and correct interfaces. The developing effort for stubs or drivers is one drawback to these approaches, and this is compounded by the retesting effort.

40 Call Graph-Based Integration

41 Call Graph-Based Integration 41 Pair-Wise Neighborhood As software design moves into more detail, the added information lets us refine the functional decomposition tree into a unit calling graph. The unit calling graph is the directed graph in which nodes are program units and edges correspond to program calls.

42 Call Graph of the SATM System

43 Call Graph-Based Integration Pair-Wise Integration

44 Pair-Wise Integration 44 The idea behind pair-wise integration is to eliminate the stub/driver development effort, why not use the actual code? Sounds like big bang! But we restrict a session to only a pair of units in the call graph.

45 Some Pair-wise Integration Sessions

46 Pair-Wise Integration 46 One integration test session for each edge in the call graph (40 for SATM call graph). Not so much reduction in sessions from either topdown or bottom-up (42 sessions). Drastic reduction in stub/driver development.

47 Call Graph-Based Integration Neighborhood Integration

48 Neighborhood Integration 48 The neighborhood of a node in a graph is the set of nodes that are one edge away from the given node. In a directed graph, this includes all the immediate predecessor nodes and all the immediate successor nodes (these correspond to the set of stubs and drivers of the node)

49 Two Neighborhood Integration Sessions

50 Neighborhood Integration 50 SATM Neighborhoods: Node Predecessors Successors , 10, , 15, 6, 8, 2,

51 Neighborhood Integration 51 SATM Neighborhoods: Node Predecessors Successors , 10, , 14, , , , , , 15, 6, 8, 2, , 15, 2, 3, 4, , 24, 26, 27, 25 1 n/a 5, 7, 20, 21, 16, 17, 19, 22

52 Neighborhood Integration 52 Number of integration test sessions = Number of neighborhoods Source node: a node with indegree = 0 Sink node: a node with outdegree = 0 Interior nodes = nodes (source nodes + sink nodes) Neighborhoods = interior nodes + source nodes Neighborhoods = nodes (source nodes + sink nodes) + source nodes Neighborhoods = nodes sink nodes

53 Neighborhood Integration 53 Student Worksheet for SATM Integration test session Nodes:? Source node: a node with indegree =? Sink node: a node with outdegree =? Interior nodes = nodes (source nodes + sink nodes) =? Neighborhoods = interior nodes + source nodes Neighborhoods = nodes (source nodes + sink nodes) + source nodes Neighborhoods = nodes sink nodes =?

54 Neighborhood Integration 54 for SATM Integration test session Nodes: 27 Source node: a node with indegree = 1 Sink node: a node with outdegree = 16 Interior nodes = nodes (source nodes + sink nodes) = 10 Neighborhoods = interior nodes + source nodes Neighborhoods = nodes (source nodes + sink nodes) + source nodes Neighborhoods = nodes sink nodes = = 11 Neighborhood integration yields a drastic reduction in the number of integration test sessions (down to 11 from 40), and it avoids stub and driver development

55 Call Graph-Based Integration Pros &Cons

56 Pros and Cons Call graph-based integration approaches 56 Eliminate the stub/driver development effort. Matches well with developments characterized by build and composition. For example, sequences of neighborhoods can be used to define builds. The biggest drawback is the fault isolation problem, specially for large neighborhood. Resolving a fault in a node means changing the unit s code which in turn means that all previously tested neighborhoods that contain the changed node need to be tested.

57 Path-Based Integration

58 Path-Based Integration 58 Special forms of Paths: MM-Path An MM-Path is an interleaved sequence of module execution paths (MEP) and messages. when a unit executes, some path of source statements is traversed. Suppose that a call goes to another unit along such a path: at that point, control is passed from the calling unit to the called unit, where some other path of source statements is traversed.

59 Example: An MM-Path module A calls module B, which in turn calls module C 59 A B C

60 Extensions to Definitions 60 Definition: A source node in a program is a statement fragment at which program execution begins or resumes. Definition: A sink node in a program is a statement fragment at which program execution terminates. Definition: A module execution path (MEP) is a sequence of statements that begins with a source node and ends with a sink node, with no intervening sink nodes. Definition: A message is a programming language mechanism by which one unit transfers control to another unit.

61 Definitions for Integration Testing 61 Defintion: An MM-Path is an interleaved sequence of module execution paths (MEP) and messages. Definition: Given a set of units, their MM-Path graph is the directed graph in which nodes are module execution paths and edges correspond to messages and returns from one unit to another.

62 Example: An MM-Path module A calls module B, which in turn calls module C Student Worksheet A B C

63 Example (cont.): source nodes and sink nodes Student Worksheet 63 Module A: Source nodes:? Sink nodes:? Module B: Source nodes:? Sink nodes:? Module C: Source nodes:? Sink nodes:?

64 Example (cont.): source nodes and sink nodes 64 Source nodes A 1 Sink nodes

65 Example (cont.): source nodes and sink nodes 65 Source nodes B 1 Sink nodes 2 3 4

66 Example (cont.): source nodes and sink nodes 66 Source nodes C 1 Sink nodes

67 Example (cont.): source nodes and sink nodes 67 Source nodes Sink nodes A B C

68 Example (cont.): source nodes and sink nodes 68 Module A: Source nodes: Nodes 1 and 5 Sink nodes: Nodes 4 and 6 Module B: Source nodes: Nodes 1 and 3 Sink nodes: Nodes 2 and 4 Module C: Source nodes: Node 1 Sink nodes: Node 5

69 Example (cont.) Module Execution Path (MEP) 69 Student Worksheet MEP (A, I) = < 1, 2, 3, 6 > MEP (A, II) =? MEP

70 Example (cont.) module execution path (MEP) 70 MEP (A, I) = < 1, 2, 3, 6 > MEP (A, II) = < 1, 2, 4 > MEP (A, III) = < 5, 6 > MEP (B, I) = < 1, 2 > MEP (B, II) = < 3, 4 > MEP (C, I) = < 1, 2, 4, 5 > MEP (C, II) = < 1, 3, 4, 5 >

71 Example (cont.) module execution path (MEP) 71 A B C MEP(A,II) MEP(B,I) MEP(C,II) MEP(A,I) MEP(A,III) 3 4 MEP(B,II) MEP(C,I) 4 5

72 Example (cont.): MM-Path graph 72 Messages MEP (A, II) MEP (B, I) MEP (A, I) MEP (C, I) MEP (B, II) Returns MEP (A, III) MEP (C, II)

73 Path-Based Integration Pros & Cons

74 Pros and Cons MM-Path based integration approaches 74 MM-Paths are a hybrid of functional and structural testing: functional in the sense that they represent actions with inputs and outputs structural side comes from how they are identified, particularly the MM-Path graph. Path-based integration works equally well for software developed in the traditional waterfall process or with one of the composition-based alternative life cycle models. The most important advantage of path-based integration testing is that it is closely coupled with the actual system behavior, instead of the structural motivations of decomposition and call graph-based integration. But the price: more effort is needed to identify the MM-Paths. This effort is probably offset by the elimination of stub and driver development.

75 75 How many MM-Paths are sufficient? The set of MM-Paths should cover all source-to-sink paths in the set of units.

76 Comparison

77 Number of Integration Testing Sessions Based on Functional Decomposition Top-Down nodes leaves + edges Bottom-Up nodes leaves + edges Sandwich max is number of subtrees 2. Based on Call Graph Pair-wise number of edges Neighborhood nodes sink nodes 3. Based on Paths MM-Paths (application dependent)

78 Effort and Throw-away Code 78 Method Sessions Stubs / Drivers Top-Down stubs Bottom-Up drivers Sandwich?? Big bang 1 none Pair-wise 40 pair dependent Neighborhood 11 neighborhood dependent MM-Path 1 per MM-Path 1 driver per MM-Path

79 Summary: Approaches to Integration Testing Based on Functional Decomposition Top-Down Bottom-up Sandwich Big Bang 2. Based on Call Graph Pair-Wise Neighborhood 3. Based on Paths MM-Paths

80 Reflection 80 Mention drawbacks of functional decomposition integration What is the break-even point of the SATM example for the Top-Down and Bottom-Up methods in terms of the cost of the stubs compared to drives? What is the advantage of Sandwich integration compared to TD and BU integrations? Mention three factors that could have impact on the choice of integration technique.