B.H. Far

Transcription

1 SENG 521 Software Reliability & Software Quality Chapter 15: Advances Topics Department t of Electrical l & Computer Engineering, i University it of Calgary B.H. Far (far@ucalgary.ca) far@ucalgary.ca 1

2 Section 1 Software Reliability Engineering for Agile Software Development SENG521 (Winter 2008) far@ucalgary.ca 2

3 SRE: Process Requirement & Architecture Design & Implementation Test 1. Define Necessary Reliability SRE Proc 2. Develop Operational Profile 3. Prepare for Test 4. Execute Test 5. Apply Failure Data People involved: Senior management; Test coordinator; Data coordinator; Customer or user 3

4 SRE: Critics Practical implementation of an effective SRE program is a non-trivial task. Occurrence probabilities of operational profile are usually developers best guess. Mechanisms for collection and analysis of data on software product and process must be in place. Fault identification and elimination techniques must be in place. Other organizational abilities such as the use of reviews and inspections, reliability based testing and software process improvement are also necessary for effective SRE. 4

5 Research Questions Does SRE practice have meaningful interpretation in Agile software development? How to apply common SRE practices in Agile software development? 5

6 1. Necessary Reliability How much reliability i i is good enough? 1) Define failure with failure severity classes (FSC) for the product. 2) Set a failure intensity objective (FIO) for each system to be tested. t 3) Find the developed software failure intensity objective. 4) Engineer strategies t to meet the software failure intensity it objective. (Balance among fault prevention, fault- removal, and fault tolerance strategies.) Steps 1-3 are valid for Agile process, too. 6

7 Fault Prevention & Removal Ui Using Ail Agile process requires rather different way of thinking about fault prevention and fault removal. Test Driven Development (TDD): unit tests and acceptance tests are written prior to functioning code. Continuous Integration: regression tests are performed frequently. Pair programming: shifts the activity of code reviews into prevention. Most of fault removal activities (in traditional SRE definition) are moved into fault prevention 7

8 2. Operational Profile Operational lprofile (OP) is a complete set of operations with their probabilities of occurrence The purpose of OP is to provide the developers with the information about how users will use the product being built Needed for better management of development and test resources 8

9 Operational Profile vs. User Story Similarities: i i i Both contain a complete piece of functionality Both are directly related to requirements Both share an emphasis on developing more critical functionality first in order to achieve faster time to market Differences: SRE process has an additional requirement on frequency of use of each operation Collecting usage data by user story (operation in SRE terms) is often missed by teams following agile practices 9

10 How to Find Operations? Supplementary Specifications Glossary Use-Case Realization (identified) Operations Use-Case Model Software Architecture Document Design Elements Classes Subsystems Interfaces Signals & events Use-Case Realization (developed) Control Entity Boundary Analysis Classes 10

11 Case of Agile Development /1 Main source: User stories Example: 11

12 Case of Agile Development /2 Sort stories for each user type Example: BSC clerk operation list: Using blind TDD may lead to tests that do not reflect usage! 12

13 3. Prepare & Execute Test Prepare test cases and test procedures. User stories already contain a list of acceptance tests equivalent to feature (or unit) it)tests t or tests t for verifying i nonfunctional requirements. Execute tests t and collect failure data. 13

14 Prepare & Execute Test SRE: regression tests run after every build involving significant change. Agile process: continuous integration. Checking code and tests into the common repository several times per day. Each time a check-in is made, all regression tests are run, therefore regression tests are run very frequently. 14

15 Preparing Test Differs significantly from common SRE practices: in Agile, tests are not allocated based on usage and/or criticality All testing should be done in conjunction with development, with tests written prior to functionality implementation. i Test definitions are established based on need rather than usage. whose need? 15

16 Number of Test Cases The number of test cases is another point of deviation from SRE which is calculated based on available budget and time. In Agile development a common practice is that the development team will enforce a unit test coverage metric (such as 95% for all developer written code). Every functional user story will receive at least one test case. 16

17 Test Case Selection & Run For performance testing, acceptance tests are repeatedly run by selecting each based on its proportionate usage depicted din the operational profile. Occurrence probabilities indicated in the operational profile can be used as a bias for test selection. 17

18 Test Preparation Summary Deviates significantly from the standard SRE approach. Rather than planning for a certain number of tests and allocating tests and time for tests among defined systems, test driven approach will result in a comprehensive set of tests. Since functionality is not written without tests, every part of the system will be well tested. 18

19 Reporting Tools NUnit for running unit tests CruiseControl.NET for continuous integration CCTray for instant notification LoadRunner for performance testing CASRE for failure analysis 19

20 Tools: NUnit NUnit gives meaningful, real-time feedback to developers for both new tests and for regression test. Users can choose to run only a single test, or the entire suite of unit tests for a given project or a group of projects. 20

21 Tools: CruiseControl.NET CruiseControl.NET examines the common code repository for changes. Finding a change, it compiles the entire solution and runs all tests. 21

22 Tools: CCTray CCTray runs on clients and can instantly notify each developer about the current state of the build status (pass or fail), and upon failure the web interface can be used to determine the exact cause of the failure. 22

23 Tools: LoadRunner LoadRunner can be used for performance testing. It allows for simulation of multiple users, and can execute code or html web queries. It also allows for the spawning of agents to multiple client machines, if necessary, to simulate a real web application. 23

24 Tools: LoadRunner 24

25 Tools: CASRE 1000 Time Between Failure vs. ith Failure Hours ith Failure F il Ti Failure Time 25

26 5. Guided Decision Assessing rather than Measuring reliability Reliability verification procedure is via Certification Test process In certification testing ti we cannot calculate l the exact amount of reliability of the developed product but we can be sure that it has exceeded a certain minimum level of reliability defined by the FIO set during the defining necessary reliability phase. 26

27 Reliability Demo Chart /1 An efficient way of checking whether the FIO ( F) is met or not. It is based on collecting failure data at time points. Vertical axis: failure number (n) Horizontal axis: normalized failure data (Tn), i.e., fil failure time F or failure time / MTTF 27

28 Lessons Learnt /1 Reliability i i growth models of fsre may not suit a test-driven process Test-driven development (TDD) approach and Certification Testing together can be very useful Allocation of tests (number of test and time for system testing) can be better managed by an operational profile Certification Testing will assure minimum level of reliability has been achieved 28

29 Lessons Learnt /2 Regarding test allocation, there seems to be a more natural fit in defining tests on a user scenario basis rather than allocating a number of tests per operation. However we lose the benefit of knowing that all critical i operations are indeed d adequately covered, and we must rely on developer skill rather than the established metrics and statistics of SRE. Operational profile will resolve this 29

30 Lessons Learnt /3 More research: Establishing relationship between test coverage and mean time to failure or failure intensity. Incorporating performance testing into the continuous integration and automated build practices that agile promotes. Classifying failures and ensuring that potential failures are in fact tested. 30

31 Enhancing Agile Define failure severity classes and cover the potential failures in the acceptance tests Defining operational profile and using it to define proper number of tests and system test duration Add mechanisms for tracking performance Add certification testing to ensure minimum required quality is delivered 31

32 Conclusions Agile software development treats quality as cats for the blind! While SRE related concepts and techniques may initially seem incompatible with the test-driven emphasis of Agile methods, we showed that SRE does in fact have value within the Agile process context. 32

33 Section 2 Cleanroom Software Development Department t of Electrical l & Computer Engineering, i University it of Calgary B.H. Far (far@ucalgary.ca) far@ucalgary.ca 33

34 Background Chaos Report [Standish 1995] Based on data representing 8,380 SE projects, only 16.2% of projects met the delivery date, the budget and with all of the specified features and functions. 31% of projects were cancelled before they were completed, 52.7% were delivered with over-budget, over-schedule or with fewer features and functions than specified. Software Productivity Research [Chapman 2000] %60 of the United State s software work force is dedicated to fixing software errors that could have been avoided. In addition, there are only 47 days in a calendar year dedicated to doing development or enhancement of software applications. The rest is spent mainly on fixing the bugs. far@ucalgary.ca 34

35 The First Computer Bug! On September 9 th 1945, Grace Murray Hopper was working on the Harvard University Mark II Aiken Relay Calculator when the machine was experiencing problems. An investigation showed that there was a moth trapped between the points of Relay #70, in Panel F. Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., far@ucalgary.ca 35

36 Research Question Is it possible to build software without any bug in it? Answer: May be. By using cleanroom software development 36

37 Causes for Bugs in Programs The main causes for bugs in programs: Design flaws Coding error Other (including human related error) The first two can be eliminated by formal (e.g. box structure) design verification and automated code generators. Certification testing will take care of the last. 37

38 Cleanroom SE Cleanroom software engineering i (CSE) is an engineering i process for the development of high quality software with certified reliability with the emphasis on design with no defects and test based on software reliability engineering concepts. CSE focuses on defect prevention instead of defect correction, and certification of reliability for the intended environment of use. CSE yields software that is correct by mathematically sound design, and software that is certified by statistically valid testing. CSE represents a paradigm shift from traditional, craft-based SE practices to rigorous, engineering-based practices. far@ucalgary.ca 38

39 CSE: Characteristics Objective: Achieve zero defects with certified reliability Focus: Defect prevention rather than defect correction Process: Incremental (short) development cycles; long product life 39

40 CSE: History 1983: Original idea of Cleanroom came from one of Dr. Harlan Mills published papers 1987: Proposed by Dr. Mills as a SE methodology. The name Cleanroom was borrowed from the electronics industry 1988: Defense Advanced Research Projects Agency (DARPA) Software Technology for Adaptable Reliable Systems (STARS) focus on Cleanroom : Prototyping of Cleanroom Process Guide 1992: A book of CSE published, foundation of CSE : 1993: Army and Air Force Demonstration of Cleanroom Technology : Prototyping of Cleanroom tools 1995: Commercialization of a Cleanroom Certification Tool 1995: Cleanroom and CMM Consistency Review far@ucalgary.ca 40

41 Comparison Craft-Based SE Sequential or chaos development Informal design Unknown reliability Cleanroom SE Incremental development Disciplined engineering specification and design Measured reliability Individual development Individual unit testing Peer reviewed engineering Team correctness verification Informal load or coverage testing Statistical usage testing 41

42 Cleanroom SE: Technologies Development practices are based on mathematical lfunction theory Test practices are based on applied statistics Analysis and design models are based on incremental software model and created using box structure representation A box encapsulates the system (or some aspect of the system) at a specific level of abstraction Correctness verification is applied once the box structure design is complete far@ucalgary.ca 42

43 Cleanroom SE: Technologies Software is tested by defining a set of usage scenarios (i.e., operations or operational modes), determining the probability of use for each scenario (i.e., operational profile), and then defining random tests that conform to the probabilities. Error records are checked. No corrective actions are taken. Only certification test is conducted to check whether errors (i.e., current failure intensity) meet the projected reliability (i.e., failure intensity objective) for the software component. far@ucalgary.ca 43

44 CSE: Processes /1 Cleanroom processes: 1. Management process 2. Specification process 3. Development process 4. Certification process 44

45 CSE: Processes /2 1. Cleanroom Management Process Project planning Project management Performance improvement Engineering change 2. Cleanroom Specification Process Requirements analysis Function specification Usage specification Architecture specification Increment planning 45

46 CSE: Processes /3 3. Cleanroom Development Process Increment design Correctness verification Software reengineering (reuse) 4. Cleanroom Certification Process Usage modeling and test planning Statistical testing and certification 46

47 CSE: Management Process Project tplanning Cleanroom engineering guide Software development plan (incremental) Project Management age e Project record Performance Improvement Performance improvement plan Engineering Change Engineering change log far@ucalgary.ca 47

48 CSE: Specification Process /1 Requirements Analysis Elicitation and analyzes of requirements Define requirements for the software product Obtain agreement with the customer on the requirements Requirements are reconfirmed or clarified throughout the incremental ldevelopment and certification process. Functional Specification Based on the result of Requirements Analysis Specify the complete functional behavior of the software in all possible modes of use Obtain agreement with the customer on the specified function as the basis for software development and certification 48

49 CSE: Specification Process /2 Usage Specification Identify and classify software users, usage scenarios, and environments of use (operational modes) Establish and analyze the probability distribution for software usage models Obtain agreement with the customer on the specified usage as the basis for software certification Architecture t Specification Define the conceptual model, the structural organization, and the execution characteristics of the software Architecture definition is a multi-level activity that spans the life cycle far@ucalgary.ca 49

50 CSE: Specification Process /3 Increment Planning Allocate customer requirements defined in the Function Specification to a series of software increments that satisfy the Software Architecture, Define schedule and resource allocations for increment development and certification Obtain agreement with the customer on the increment plan far@ucalgary.ca 50

51 CSE: Development Process Increment 1 RG BSS FD CV TP CG CI SUT C Increment 2 SE RG BSS FD CV TP CG CI SUT C SE: System Engineering RG: Requirement Gathering BSS: Box structure specification FD: Formal Design CV: Correctness Verification CG: Code Generation CI: Code Inspection SUT: Statistical Use Testing C: Certification Test TP: Test Planning far@ucalgary.ca 51

52 Cleanroom Strategy /1 Requirement gathering (RG) A detailed description of customer level requirements for each hincrement. Box structure specification (BSS) Functional specification using box structure to separate behavior, data and procedures. Formal design (FD) Specifications (black boxes) are refined to become analogous to architectural (state boxes) and procedural (clear boxes) design. far@ucalgary.ca 52

53 Cleanroom Strategy /2 Correctness verification i (CV) A set of correctness verification activities on the design and moves later to code. First level lverification i is via application of a set of correctness questions. Code generation, inspection & verification (CG & CI) The box structure transformed to a programming language. Walkthrough and code inspection techniques are used to ensure semantic conformance with the box structure. far@ucalgary.ca 53

54 Cleanroom Strategy /3 Statistical i test planning (TP) Planning the test based on operational modes, operational profiles and reliability. Statistical use testing (SUT) Creating test case, execute them and collecting error data. Certification (C) Conducting certification test rather than reliability growth to accept/reject developed software components (using reliability demonstration chart, etc). far@ucalgary.ca 54

55 Box Structure /1 Box structures are used to move from an abstract specification to a detailed design providing implementation details far@ucalgary.ca 55

56 Box Structure /2 Black box Specifies the behavior of a system or a part of a system. The system responds to specific stimuli (events) by applying a set of transition rules that map the stimuli to response. State box Encapsulates state data and services (operations). Input to the state box and outputs are represented. Clear box Transition function that are implied by the state box. It contains the procedural design of the state box. far@ucalgary.ca 56

57 Box Structure /3 S R State f: S* R T g12 Black box S R g11 cg1 State T clear box g13 S f: S* R R Black boxes (specifications) State box State boxes (architectural designs) Clear boxes (component designs) far@ucalgary.ca 57

58 Box Structure /4 CB1.1.1 BB1 BB1.1 BB BB1.1.1 SB1.1.1 BB1.1.2 BB1.1.3 CB CB1.1.3 Clear box BB1.n State box Black box 58

59 Correctness Verification Mathematical based techniques are used to verify the correctness of a software increment Examples If a function f is expanded into a sequence g and h, the correctness rule for all input to f is: Does g followed by h do f? If a function f is expanded into a condition if-then-else, the correctness rule for all input to f is: Whenever condition <c> is true does g do f and whenever <c> is fl false does h do f? When function f is refined as a loop, the correctness rule for all input to f is: Is termination i guaranteed? Whenever <c> is true, does g followed by f do f? and whenever <c> is false, does skipping the loop still do f? far@ucalgary.ca 59

60 Black Box Structure /4 If having more than one black box or nested black boxes verify the mapping f f g h Sequential split c g h Parallel l split far@ucalgary.ca 60

61 Black Box Structure /5 If having more than one black box or nested black boxes verify the mapping f f g c g h c Loop split Loop split far@ucalgary.ca 61

62 Advantages of Verification Design verification has the following advantages: Verification is reduced to a finite process Every step of design and every line of code can be verified Near zero defect tlevel lis achieved Scalability is possible Better code (than unit testing) can be generated far@ucalgary.ca 62

63 CSE: Certification Process /1 Usage modeling and test t planning A usage model represents a possible usage scenario of the software Usage model is based on usage specification and is used for testing Similar to the way we defined operations far@ucalgary.ca 63

64 CSE: Certification Process /2 Statistical i Testing and Certification i Testing is conducted in a formal statistical design under experimental control. The software is demonstrated to perform correctly with respect to its specification. Statistically valid estimates of the properties addressed by the certification goals are derived for the software. Management decisions on continuation of testing and certification of the software are based on statistical estimates of software quality. far@ucalgary.ca 64

65 Cleanroom Testing Using statistical i usage concept for testing. Determine a usage probability distribution via the following steps: 1) Analyze the specification to identify a set of stimuli (direct and indirect input variables). 2) Create usage scenarios (operational modes). 3) Assign probability to use of each stimuli (operational profile). 4) Generate test cases for each stimuli according to the usage probability distribution. far@ucalgary.ca 65

66 Certification Test Cleanroom approach DOES NOT emphasize on Unit or integration testing. Bug fixing as a result of test and regression. Certification procedure involves the followings: Create usage scenarios. Specify a usage profile. Generate test cases from the profile. Execute test cases and record failure data. Compute reliability and certify the component or system using reliability demo chart, etc. far@ucalgary.ca 66

67 Reliability Demo Chart An efficient i way of checking whether the failure intensity objective ( F) )is met or not based on collecting failure data at time points. Vertical axis: failure number Horizontal axis: normalized failure data, ie i.e., failure time F far@ucalgary.ca 67

68 Example Automated Teller Machine (ATM) Requirements: The customer has a PIN number and access-card to use the ATM The customer can deposit, withdraw money from the account Transaction involves no bank employee far@ucalgary.ca 68

69 Example: Usage Model <<extends>> Customer <<extends>> Withdraw Deposit Usage percentage: 50% Usage percentage: 50% Card Processor Cash Dispenser Transaction Manager 69

70 Example: Black Boxes Black boxes Card Processor In: ValidCard(cardNum) Out: showmessage(message) g Boolean Cash Dispenser In: enoughcashinmachine(amount) dispensecash(amount) Out: showmessage(message) dispense(amount) Boolean Transaction Manager In: ValidCustomer(cardNum, pin) AmountLimit(amount) EnoughCashInAccount(amount) Out: showmessage(message) Boolean Card Processor Cash Dispenser Transaction Manager far@ucalgary.ca 70

71 State Box: Card Processor /idle /insert card Menu /get pin [false] /send message /notify user Check To cash dispenser /send card verified Notify [true] /call notify 71

72 State Box: Cash Dispenser Menu /get card verified [true] /get amount [false] Check Machine cash [false] [false] Check [true] Check Account [true] /call check account 72

73 Example: Clear Box Spec /1 / insert card // Get customer PIN no ValidCustomer(cardNum, pin) Menu [true] CheckMachine Cash [false] / get pin Check [false] [false] [true] CheckAccount [true] /get amount far@ucalgary.ca 73

74 Example: Clear Box Spec /2 Menu / insert card // Bank returns false // Show message showmessage(mesg); [true] CheckMachine Cash [false] / get pin Check [false] [false] [true] CheckAccount [true] /get amount far@ucalgary.ca 74

75 Example: Clear Box Spec /3 Menu / insert card // Bank returns true // get amount getamount(amount); [true] CheckMachine Cash [false] / get pin Check [false] [false] [true] CheckAccount [true] /get amount far@ucalgary.ca 75

76 Example: Clear Box Spec /4 Menu / insert card // Bank returns false for daily limit // and/or balance // Show message showmessage(mesg); [true] CheckMachine Cash [false] / get pin Check [false] [false] [true] CheckAccount [true] /get amount far@ucalgary.ca 76

77 Example: Clear Box Spec /5 Menu / insert card // Bank returns true for daily limit // and balance Dispenser.enoughCashInAccount(amount) [true] CheckMachine Cash [false] / get pin Check [false] [false] [true] CheckAccount [true] /get amount far@ucalgary.ca 77

78 Example: Clear Box Spec /6 Menu / insert card // Dispenser returns false for // cash level // Show message showmessage(mesg); [true] CheckMachine Cash [false] / get pin Check [false] [false] [true] CheckAccount [true] /get amount far@ucalgary.ca 78

79 Example: Clear Box Spec /7 Menu / insert card // Dispenser returns true for // cash amount Dispenser.dispense(amount); [true] CheckMachine Cash [false] / get pin Check [false] [false] [true] CheckAccount [true] /get amount far@ucalgary.ca 79

80 Example: Testing 50% of tests go to test Withdraw; 50% to test Deposit Test cases are created to test each execution path (or state transition path) A subset of tests are selected for validation SENG521 (Winter 2008) far@ucalgary.ca 80

81 CSE: Team Specification team: Responsible for developing and maintaining the system specification Development team: Responsible for developing and verifying the software The software is not executed during this process Certification team: Responsible for developing a set of statistical tests to exercise the software after development Use reliability growth models to assess reliability far@ucalgary.ca 81

82 CSE: Evaluation Basic features of Cleanroom development that distinguishes it from other SE methodologies are: 1. Formal specification (Box structure) & Correctness verification 2. Statistical certification test 82

83 Evaluation: Formal Spec Advantages: Mathematical and logical foundation for defining requirements accurately with precise notation. Proactive versus reactive approach with regards to requirements validation. Ambiguous, inconsistent and conflicting requirements are caught before the system test. Box structure uses black, state, and clear box and it is a stepwise approach to refine requirements. Usage models define how the software is to be used by the customer. far@ucalgary.ca 83

84 Evaluation: Formal Spec Disadvantages: Requires extra skills and knowledge (e.g. mathematics) Requires substantial effort to fully express the system in formal specification On average Cleanroom projects require 60-80% of the time used in analysis and design Ideal for safety or mission critical systems and not for ordinary commercial development Lacks good enough CASE tools supporting Project specific If time-to-market & conditions are issues, then might not be used 84

85 Evaluation: Incremental Devel Advantages Quick and clean development in Cleanroom Engineering Continuous validation Provides measurable progress Manage higher risk requirements (i.e. prototype). Tracking of requirements Stepwise building functionalities that satisfies stakeholders requirements Allows for fast delivery on important parts Focus on planning and discipline at management level and technical level Statistical testing make the project quality control in proper level Verifiable specifications 85

86 Evaluation: Incremental Devel Disadvantages: Incomplete or conflicting requirements cannot be resolved at the beginning i to determine increments Risk analysis has not been incorporated explicitly Need more care about configuration management Requires extra planning at both the management and technical llevels l Stable requirements for each increment is needed, i.e., cannot adapt quickly to rapidly changing requirements far@ucalgary.ca 86

87 Evaluation: Certification Test Advantages Determines a level of confidence that a software system conforms to a specification Able to statistically evaluate and infer the quality of the software system to meet all requirements Quantitative approach that is verifiable Quantitative data could be recorded and used lt later for benchmarking, etc. far@ucalgary.ca 87

88 Evaluation: Certification Test Disadvantages Testing is derived from a usage model that must be exhaustive in order to select a subset for testing Statistical testing and verification will be more reliable if it is based on the some history data It would be effective if it could be integrated with other testing methods Testing is not suitable for bug-hunting Human residual coding errors may not be addressed far@ucalgary.ca 88

89 Section 3 Case Study SENG521 (Winter 2008) far@ucalgary.ca 89

90 Cleanroom SE: Case Study Cleanroom software development relies on a mathematically sound model of design to ensure that no defects are introduced dinto the software. Cleanroom Software Specification and Design begins with an external view (black box), and is transformed into a state machine view (state box), and is fully developed into a procedure (clear box). far@ucalgary.ca 90

91 Box Structure Box structures map system inputs and the stimulus histories (previous inputs) into outputs. Is Black-Box construct sufficient to represent this? e g Jackson model this? e.g. Jackson model No Box structure inputs history outputs far@ucalgary.ca 91

92 Cleanroom SE: Process /1 1) Define the system requirements 2) Specify and validate the black box Define the system boundary and specify all stimuli and responses Specify the black box mapping rules Validate the black box with owners and users 3) Specify and verify the state box Specify the state data and initial state values Specify the state box transition function Derive the black box behavior of the state box and compare the derived black box for equivalence 92

93 Cleanroom SE: Process /2 4) Design and verify the clear box Design the clear box control structures and operations Embed uses of new and reused black boxes as necessary Derive the state box behavior of the clear box and compare the derived state box to the original state box for equivalence 5) Repeat the process for new black boxes 6) Convert to code 7) Certification test the code 93

94 Requirements Analysis /1 The process begins when one of the following entry criteria is satisfied. Entry 1 The Statement of Work or other initial artifact, such as a statement of allocated system requirements, is available. Entry 2 Changes, including additions and corrections, to the Software Requirements are proposed. Entry 3 A completed increment is ready for customer execution and evaluation. far@ucalgary.ca 94

95 Requirements Analysis /2 Task 1 Define the software requirements. Task 2 Upon completion of each hincrement, reconfirm or clarify requirements through customer evaluation of fthe executable system. far@ucalgary.ca 95

96 Requirements Analysis /3 Verification 1 Review the evolving Software Requirements work product. Verification 2 Validate the Software Requirements work product with the customer and peer organizations. far@ucalgary.ca 96

97 Requirements Analysis Example: Build a simple calculator Detailed ddefinition i i of the calculator l function and what it does must be given and verified with the customer Various formal methods can be used: graph theory, automaton model, etc. far@ucalgary.ca 97

98 Formal Specification Example: Use logic constructs such as is-a (inheritance hierarchy), has-a (association) and such-as as (examples) to formulate the specification. Verify that the total set of sentences form a tree (directed graph with no loop) far@ucalgary.ca 98

99 Black Box Structure /1 Entry #1: first operand (xxx digits) Entry #2: calculation symbol (add, subtract) Entry #3: second operand Entry #4: equal symbol Exit #1: calculation result Box structure inputs history outputs 99

100 Black Box Structure /2 calculator 1 st operand null null calculator Calc symbol 1 st operand null queue Push-down automaton model far@ucalgary.ca 100

101 Black Box Structure /3 calculator l 2 nd operand null 1 st operand Calc sym queue calculator Equal sym 1 st operand Calc sym 2 nd operand queue2 nd operand Calc results far@ucalgary.ca 101

102 Black Box Structure /4 If having more than one black box or nested black boxes verify the mapping f f g h g Sequential split c h Parallel split far@ucalgary.ca 102

103 Black Box Structure /5 If having more than one black box or nested black boxes verify the mapping f f g c g h c Loop split Loop split far@ucalgary.ca 103

104 State Box Structure /1 State transition diagram happy path non-numeric key pressed Error 104

105 State Box Structure /2 Several state tt boxes can be generated td depending on the combination of acceptable (unacceptable) )inputs and dhistoriesi Example: If 1 st operand is non-numeric and calc symbol are typed the next state is error state If 1 st operand is numeric and any other key other than calc symbol is typed the next state is error state etc. far@ucalgary.ca 105

106 State Box Structure /3 State boxes should be generated for all possible combinations of input(s) and history states. The set of state boxes can easily grow beyond control! 106

107 Clear Box Structure /1 107

108 Code Generation Coding will be based on the clear boxes Use of automatic code generation tools is encouraged to reduce the probability of human error 108

109 Cleanroom Testing /1 Cleanroom testing teams must determine a usage probability distribution for the software The operational profile can be used far@ucalgary.ca 109

110 Cleanroom Testing /2 Suppose that tthe inputs to the calculator l program are Input percentage number A1 1 st operand (correct) % A2 1 st operand d(incorrect) % B1 2 nd operand (correct) % B2 2 nd operand (incorrect) % C1 Calculation symbol (correct) % C2 Calculation symbol (incorrect) % D1 Equal symbol (correct) % D2 Equal symbol (incorrect) % far@ucalgary.ca 110

111 Cleanroom Testing /3 We must generate a sequence of usage test tcases that t conform to the usage probability distribution. A series of random numbers are generated between 0 and 99 that corresponds to the probability of stimuli occurrence. For example, the following random number sequences are generated: : A1; D1; B1; B Test case The testing team executes the test cases noted above (and others) and verifies software behavior against the specification for the system. far@ucalgary.ca 111

112 Cleanroom Testing /4 For example for the test case T1: A1; D1; B1; B1 The input sequence is: 1. 1 st operand 2. Equal symbol 3. 2 nd operand 4. 2 nd operand And the output should be: Error far@ucalgary.ca 112

113 Cleanroom Certification The verification and testing techniques lead to certification of software components Certification implies that the reliability can be specified for each component. Each component would have a certified reliability under the usage scenario and testing regime. This information is needed for future use of the components. The certification approach involves five steps: 1U 1. Usage scenario is created. td 2. A usage profile is specified. 3. Test cases are generated from the profile. 4. Tests are executed and failure data are recorded and analyzed. 5. Reliability is computed and certified. far@ucalgary.ca 113

114 CSE: Overall Advantages Suitable for iterative ti and incremental software development. Uses formal specification that defines more accurate, less conflict and complete requirements. Continuous verification of software quality is possible. Software quality can be certified using software reliability engineering method. 114

115 CSE: Overall Disadvantages Cleanroom advocates the use of sequence-based specifications. These are better suited to problems with a high degree of logical interactions. Not suitable for black boxes used in numerical or highly computational applications. Non-functional requirements (real-time, security constraints) and a significant portion of algorithmic requirements are hard to be represented by Box structure. After requirements changes, rework of the boxstructure is a time-consuming process. Statistical test data may be hard to be collected. far@ucalgary.ca 115

116 Conclusions Cleanroom approach his a rigorous approach to software engineering that has emphasis on: Formal specification Mathematical verification of correctness of design Certification of software reliability Cleanroom approach is yet to become a common practice in software development industry because of emphasis on the above three points far@ucalgary.ca 116

117 References Linger, R. and Trammel, C. (1996). Cleanroom Software Engineering Reference Model Version edu/pub/documents/96 pdf Wolack, C. (2001). Taking The Art Out of Software Development An In-Depth Review of Cleanroom Software Engineering. Pressmen and Associates (2000). Cleanroom Engineering Resources

118 One Last Advice Want to impress your customers: use failure intensity + reliability ygrowth methodology! Want to impress your boss (development): use failure density + zero time failure methodology! Want to impress yourself: use target failure intensity + reliability demonstration chart! far@ucalgary.ca 118

119 That is all folks! 119