heory Lecture Plan 2 esting heory Lecture 8 Software Engineering DDC88/DDC93 autumn 28 Department of Computer and Information Science Linköping University, Sweden L - Course Introduction and Overview L2 - Project Management L3 - Requirements L4 - Acceptance esting and Quality actors L5 UML L6 - Design Patterns L7 - System Design and Architecture L8 - esting heory L9 - esting in Practice L - Inspection L - Software Life Cycles and Configuration Management L2 - Software Quality Management L3 - Course Summary, Exam examples, Questions I II V A Software Life-cycle Model Which part will we talk about today? 3 Maintenance Agenda - What will you learn today? 4 Requirements System Design (Architecture, High-level Design) Validate Requirements, Verify Specification Verify System Design Acceptance est (Release ing) (Integration ing of modules) I ing Module Design (Program Design, Detailed Design) Implementation of Units (classes, procedures, functions) Verify Module Design Verify Implementation Unit ing (Integration ing of units) II V Project Management, Software Quality Assurance (SQA), Supporting ools, Education I II V I II V
5 riangle program (simple version) 6 triangle problem is the most widely used example in software ing literature. he program accepts three integers, a, b, and c as input. he three values are interpreted as representing the lengths of sides of a triangle. he program prints a message that states whether the triangle is scalene (oregelbunden), isosceles (likbent) or equilateral (liksidig). On a sheet of paper, write a set Student of Worksheet cases (i.e., specific sets of data) that you feel would adequately this program. est case a b c Expected output 3 3 4 isosceles (likbent) 2???? I II V I II V esting a ballpoint pen 7 8 Does the pen write in the right color, with the right line thickness? Is the logo on the pen according to company standards? Is it safe to chew on the pen? Does the click-mechanism still work after clicks? Does it still write after a car has run over it? What is expected from this pen? Intended use!! Customer Developer Requirements definition Requirements specification unctional requirements Nonfunctional requirements Code = System Design Specification I II V I II V 2
Error, ault, ailure 9 ypes of aults (dep. on org. IBM, HP) Human error (Mistake, Bug) Can lead to ault (Defect, Bug) Can lead to Algorithmic: division by zero Computation & Precision: order of op Documentation: doc - code Stress/Overload: data-str size ( dimensions of tables, size of buffers) Capacity/Boundary: x devices, y parallel tasks, z interrupts iming/coordination: real-time systems hroughout/performance: speed in req. Recovery: power failure Hardware & System Software: modem Standards & Procedure: organizational standard; difficult for programmers to follow each other. ailure I II V I II V A esting Life Cycle 2 Error Requirements Specification ault Design Error ault Putting Bugs IN Development phases Coding Error ault esting Incident inding Bugs esting phase ault Classification ix ault Isolation Error ault Resolutio n Getting Bugs OU I ing I II V I II V 3
Unit & Integration esting 3 4 Objective: to ensure that code implemented the design properly. Component code Unit ested components Design Specification Integration Code = System Design Specification Component code Unit ested components Integrated modules I II V I II V 5 wo ypes of Oracles 6 Input est ailure? Human: an expert that can examine an input and its associated output and determine whether the program delivered the correct output for this particular input. Object Automated: a system capable of performing the above task. Output Oracle I II V I II V 4
Black-box esting 7 Black-box / Closed-box esting 8. Exhaustive ing 2. Equivalence class ing (Equivalence Partitioning) 3. Boundary value analysis 4. Decision table ing incorrect or missing functions interface errors performance error input output I II V I II V. Exhaustive ing 9 2. Equivalence Class esting 2 Definition: ing with every member of the input value space. Input value space: the set of all possible input values to the program. Equivalence Class (EC) ing is a technique used to reduce the number of cases to a manageable level while still maintaining reasonable coverage. Each EC consists of a set of data that is treated the same by the module or that should produce the same result. Any data value within a class is equivalent, in terms of ing, to any other value. I II V I II V 5
Identifying the Equivalence Classes aking each input condition (usually a sentence or phrase in the specification) and partitioning it into two or more groups: Input condition range of values x: -5 Valid equivalence class Invalid equivalence classes 2 Guidelines. If an input condition specifies a range of values; identify one valid EC and two invalid EC. 2. If an input condition specifies the number (e.g., one through 6 owners can be listed for the automobile); identify one valid EC and two invalid EC (- no owners; - more than 6 owners). 3. If an input condition specifies a set of input values and there is reason to believe that each is handled differently by the program; identify a valid EC for each and one invalid EC. 4. If an input condition specifies a must be situation (e.g., first character of the identifier must be a letter); identify one valid EC (it is a letter) and one invalid EC (it is not a letter) 5. If there is any reason to believe that elements in an EC are not handled in an identical manner by the program, split the equivalence class into smaller equivalence classes. 22 I II V I II V Identifying the est Cases 23 Equivalence partitioning 24. Assign a unique number to each EC. 2. Until all valid ECs have been covered by cases, write a new case covering as many of the uncovered valid ECs as possible. 3. Until all invalid ECs have been covered by cases, write a case that cover one, and only one, of the uncovered invalid ECs. Invalid inputs Valid inputs outputs I II V I II V 6
Specification: the program accepts four to eight inputs which are 5 digit integers greater than. 25 3. Boundary Value esting Boundary value ing focuses on the boundaries simply because that is where so many defects hide. he defects can be in the requirements or in the code. 26 he most efficient way of finding such defects, either in the requirements or the code, is through inspection (Software Inspection, Gilb and Graham s book). I II V I II V echnique 27 Boundary value analysis 28. Identify the ECs. 2. Identify the boundaries of each EC. 3. Create cases for each boundary value by choosing one point on the boundary, one point just below the boundary, and one point just above the boundary. Less than Between and 99999 More than 99999 I II V I II V 7
4. Decision able esting 29 echnique 3 Decision tables are an excellent tool to capture certain kinds of system requirements and to document internal system design. hey are used to record complex business rules that a system must implement. In addition, they can serve as a guide to creating cases. he general format of a decision table: Rule Rule 2 Conditions Condition- Condition-2 Condition-m Actions Action- Action-2 Action-n Rule P I II V I II V 3 echnique (cont.) 32 A decision table with don t care entry C C2 Rule Rule 2 Rules 3,4 Rule 5 Rule 6 Rules 7,8 A decision table converted to a case table: est Case est Case 2 est Case P C3 Inputs A Condition- A2 Condition-2 A3 A4 Condition-m Expected Results Action- : don t care entry. he don t care entry has two major interpretations: the condition is irrelevant, or the condition does not apply. Sometimes the n/a symbol for this latter interpretation.. Action-2 Action-n I II V I II V 8
est Cases for the riangle Problem 33 est Cases for the riangle Problem 34 Case ID a b c Expected output 2 3 4 5 6 7 8 9 D 4 2 Not a riangle C: a < b + c? D2 C2: b < a + c? D3 C3: c < a + b? D4 C4: a = b? D5 C5: a = c? D6 C6: b = c? D7 A: Not a triangle A2: Scalene A3: Isosceles A4: Equilateral A5: Impossible D8 D9 D D I II V I II V White-box esting (Glass box ing, Open box ing, Clear box ing, Structural ing) 35 Control low Graphs 36. Control flow ing 2. Data flow ing Process blocks Decision Point Junction Point Sequence If While Until Case I II V I II V 9
37 Code Coverage ( coverage metrics) 38 Definition: Given a program written in an imperative programming language, its program graph is a directed graph in which nodes are statement fragments, and edges represent flow of control (a complete statement is a default statement fragment). Levels of Coverage: Statement/Line/Basic block/segment Coverage Decision (Branch) Coverage Condition Coverage Decision/Condition Coverage Path Coverage I II V I II V Statement Coverage 39 What is Wrong with Line Coverage Steve Cornett (Bullseye ing technology) 4 Begin if ( y >= ) then y = ; abs = y; end; case- (yes): input: y =? expected result:? actual result:? begin y >= no abs = y y = yes Software developers and ers commonly use line coverage because of its simplicity and availability in object code instrumentation technology. Of all the structural coverage criteria, line coverage is the weakest, indicating the fewest number of cases. Bugs can easily occur in the cases that line coverage cannot see. he most significant shortcoming of line coverage is that it fails to measure whether you simple if statements with a false decision outcome. Experts generally recommend to only use line coverage if nothing else is available. Any other measure is better. I II V I II V
Branch Coverage 4 Begin if ( y >= ) then y = ; abs = y; end; begin y >= no y = yes more conditions? abs = y case- (yes): input: y = expected result: actual result: case-2 (no): input: y =? expected result:? actual result:? I II V Condition Coverage 43 Decision/Condition Coverage 44 Begin if ( x < && y > 2) { z = foo (x,y); else z =fie (x,y); } end; no z=fie (x,y) x< && y>2 yes z=foo (x,y) Begin if ( x < && y > 2) { z = foo (x,y); else z =fie (x,y); } end; no z=fie (x,y) x< && y>2 yes z=foo (x,y) case- (,): input: x =?, y =? expected result:? actual result:? case-2 (,): input: x =?, y =? expected result:? actual result:? case- (,,yes): input: x =?, y =? expected result:? actual result:? case-2 (,,no): input: x =?, y =? expected result:? actual result:? I II V I II V
Path Coverage 45 Path with loops 46 A path is a sequence of branches, or conditions. A path corresponds to a case, or a set of inputs. In code coverage ing, branches have more importance than the blocks they connect. b a c d Bugs are often sensitive to branches and conditions. e a c,b,d d e I II V I II V Path Coverage (cont.) 47 Computation of cyclomatic complexity 48 Cyclomatic complexity has a foundation in graph theory and is computed in the following ways: All possible execution paths. Cyclomatic complexity V(G), for a flow graph, G, is defined as: Question: How do we know how many paths to look for? Answer: he computation of cyclomatic complexity V(G) = E N + 2P E: number of edges N: number of nodes P: number of disconnected parts of the graph 2. Cyclomatic complexity V(G), for a flow graph, G, with only binary decisions, is defined as: V(G) = b + b: number of binary decision I II V I II V 2
Examples of Graphs and calculation of McCabe s Complexity Metric 49 A. V(G) = E N + 2P 5 E=, N=, P= V= E=, N=, P= V= E=, N=, P= V= E=, N=, P= V= G E B H D C I J E =? N =? P = V(G) =? 2. V(G) = b + K L b =? M N O P V(G) =? E=, N=, P= V= Q R S I II V I II V 2. Data low esting 5 Define/Reference Anomalies 52 Data flow ing focuses on the points at which variables receive values and the points at which these values are used (or referenced). It detects improper use of data values due to coding errors. Early data flow analyses often centered on a set of faults that are known as define/reference anomalies. A variable that is defined but never used (referenced) A variable that is used but never defined A variable that is defined twice before it is used I II V I II V 3
53 Definitions 54 dd: defined and defined again not invalid but suspicious du: defined and used perfectly correct dk: defined and then killed not invalid but probably a programming error ud: used and defined acceptable uu: used and used again acceptable uk: used and killed acceptable kd: killed and defined acceptable ku: killed and used a serious defect kk: killed and killed probably a programming error. du-path: a definition-use path (du-path) with respect to variable v is a path in PAHS(P) such that, for some v in V, there are defined and usage nodes DE(v, m) and USE(v, n) such that m and n are initial and final nodes of the path. I II V I II V Data low Graphs 55 Hierarchy of data flow coverage metrics 56 define x ~d: the variable does not exist, then it is defined ~u: the variable does not exist, then it is used ~k: the variable does not exist, then it is killed define x All-Paths define x use x use z use y kill z define y use z kill z use x define z Variable x: define x use x use x All-DU-Paths All-Uses All C-Uses/some P-Uses All-Defs All P-Uses/some C-Uses All-P-Uses use y use z All-Edges Branch kill y define z Control flow graph annotated with define-use-kill information for x, y, z All-Nodes Statement I II V I II V 4
57 Integration esting 58 II. op-down 2. Bottom-up 3. Big-bang 4. Sandwich I II V I II V Unit & Integration esting 59 6 Objective: to ensure that code implemented the design properly. Component code Unit ested components Design Specification Integration Code = System Design Specification Component code Unit ested components Integrated modules I II V I II V 5
Components 6 62 driver A Component to be ed Boundary conditions independent paths interface... B C D E G stub stub est cases I II V I II V. op-down 63 2. Bottom-up 64 B A C D B A C D E G E G I II V I II V 6
3. Big-bang 65 4. Sandwich 66 B A C D B A C D E G E G I II V I II V 67 68 V Steps unction ing / hread ing Performance ing Acceptance ing Installation ing ermination Problem I II V I II V 7
69 7 Objective: to ensure that the system does what the customer wants it to do. Customer Developer Component code Unit ested components Design Specification Integration Requirements definition Requirements specification unctional requirements Nonfunctional requirements Component code Unit ested components Integrated modules I II V I II V Integrated modules System functional requirements unction Acceptance Customer requirements spec. I unctioning systems Accepted system Other software requirements II Performance Installation User environment V 7 Verified validated software System In Use! unction ing/hread ing (ing one function at a time) functional requirements A function checks that the integrated system performs its function as specified in the requirement Guidelines use a team independent of the designers and programmers know the expected actions and output both valid and invalid input never modify the system just to make ing easier have stopping criteria I II V 72 8
Cause-and-Effect-Graph ( case generation from req.) 73 Basic cause-effect graph symbols 74 causes: inputs effects: outputs and transformations causes-and-effect graph: boolean graph reflecting causes and effects relationships is a formal language into which a natural language specification is translated a b Identity: if a then b a a b c And: if (a and b) then c b d a b c Or: if (a or b or c) then d Identity: if (not a) then b I II V I II V 75 Sample cause-effect graph 76 Specification: the character in column must be an A or a B. he character in column 2 must be a digit. In this situation, the file update is made. If the first character is incorrect, message 2 is issued. If the second character is not a digit, message 3 is issued. Intermediate node E2 Causes C: character in column is A C2: character in column is B C3: character in column 2 a digit Effects E: update made E2: message 2 is issued E3:message 3 is issued 2 E 3 E3 I II V I II V 9
Decision table for cause-and effect graph 77 Performance esting nonfunctional requirements 78 Cause Cause 2 Cause 3 Effect E est est 2 est 3 est 4 Stress s Volume s Configuration s Compatibility s Regression s Security s iming s Environment s Quality s Recovery s Maintenance s Documentation s Human factors s / usability s Effect E2 Effect E3 I II V I II V Acceptance esting customers, users need 79 Installation esting users site 8 Benchmark : a set of special cases Pilot : everyday working Alpha : at the developer s site, controlled environment Beta : at one or more customer site. Acceptance at developers site installation at users site, otherwise may not be needed!! Parallel : new system in parallel with previous one I II V I II V 2
ermination Problem How decide when to stop ing he main problem for managers! ermination takes place when resources (time & budget) are over found the seeded faults some coverage is reached 8 Summary - What have we learned today? (/2) : esting process I: Black-box esting. Exhaustive ing 2. Equivalence class ing (Equivalence Partitioning) 3. Boundary value analysis 4. Decision table ing White-box esting Control low esting. Statement/Line/Basic block/segment Coverage 2. Decision (Branch) Coverage 3. Condition Coverage 4. Decision/Condition Coverage 5. Path Coverage Data low esting 82 I II V I II V Summary - What have we learned today? (2/2) 83 84 II: (Integration esting) op-down Bottom-up Big-bang Sandwich Do you want to know more about ing? ake the course on Software esting (DDC4). V: I II V I II V 2