Testing Safety-Critical Systems

Transcription

1 Content 1. Software Today 2. Safety-related systems 3. Software Testing 4. Software Testing Goals 5. Simulators 6. Statistical Software Testing 7. Software Reliability 8. Conclusion Testing Safety-Critical Systems Presented By : Matrikel-Nr : Date: 21/07/04 SEMINER Advanced Topics in Software Engineering for Safety- Critical Systems Paderborn University / Germany

2 2 Software Today 1 F4 had no digital computer and software (early 70 s) F16A had 50 digital processors and 135 KLOC (late 70 s) F16D had 300 digital processors and 236 KLOC (late 80 s) The B-2 has over 200 digital processors and 5000 KLOC

3 3 Software Today (cont) Boeing 777 could be regarded as a bunch of software with wings 4 million lines of code run it When you start up a BMW 7 series, you activate 20 MB s of computing power The only thing that you can do with an F-22 without software is to take a picture of it 80% of its functionality is achieved by software Software compromised 30% of engineering and manufacturing development costs

4 4 Why is it important? EVENT: On February 25, 1991 / SCUD missile at Saudi Arabia / Radars classified the event as false alarm RESULT:SCUD fell down the city of Dhahran, causing victims and enormous damage REASON: Software bug [2] Software testing is a necessary but expensive process, consuming one-third to one-half the cost of a typical development project [2]

5 Safety-related systems What is a Safety-related system? 2 [SEMINER] Advanced Topics in Software Engineering for Safety-Critical Systems 5 A safety-related system (safety-critical system) in one by which the safety of equipment or plant is assured. in which a failure may result in environmental pollution loss of, or damage to, property injuries to, or illness of, persons loss of human life or lives

6 6 The V Model Requirements Design Code Unit Testing Performed by the programmer on the unit Performed by the development team on the integrated system Performed by the test group to verify system requirements Performed by a dedicated group (with customers) to verify the requirements

7 7 Software Testing What is software testing? 3 Testing is the process of identifying defects, where a defect is any variance between actual and expected results test senarios? Software testing is the process of executing a software system to determine whether it matches its specification and executes in its intended environment.

8 8 Static & Dynamic Testing Static Testing Investigates the characteristics of the system or component without operating it reviews,inspections,walkthroughs Dynamic Testing Involves the execution of the system or component in order to investigate its characteristics. natural working environment or simulation of that environment

9 9 Software Testing Goals Two main goals in testing software: 4 Debug testing Operational testing 1) to achieve adequate quality the objective is to probe the software for defects so that these can be removed [4] 2) to assess existing quality the objective is to gain confidence that the software is reliable. [4]

10 10 Two main goals in testing software(cont) Debug testing Operational testing developer tests to find and correct bugs and improve the delivered software. Instead of actively looking for failures, the tester in this case waits for failures to surface spontaneously In debug testing finding as many bugs as possible, by sampling all situations likely to produce failures (code coverage or specification coverage criteria) concentrating on situations that are considered most likely to do so (stress testing or boundary testing methods)

11 11 Some basic definitons An operation is a major system logical task, which returns control to the system when complete [3] An operational profile is a complete set of operations with their probabilities of occurrence [3] Operation Occurrence Probability Proc. voice call, no pager, ans Proc. voice call, pager, ans Proc. fax call 0.17 Proc. voice call, pager, ans. on page 0.13 Proc. voice call, no pager, no ans Proc. voice call, pager, no ans. on page 0.10 Enter forwardees 0.09 Audit sect. - phone number data base Add subscriber Delete subscriber Recover from hardware failure Total 1

12 12 Operational testing OPERATIONAL PROFILE Operation Occurrence Probability Proc. voice call, no pager, ans Proc. voice call, pager, ans Proc. fax call 0.17 Proc. voice call, pager, ans. on page 0.13 Proc. voice call, no pager, no ans Proc. voice call, pager, no ans. on page 0.10 Enter forwardees 0.09 Audit sect. - phone number data base Add subscriber Delete subscriber Recover from hardware failure Total 1 INPUT GENERATOR INPUT SUT OUTPUT EVALUATOR SUCCESS / FAILURE SUT : SYSTEM UNDER TEST

13 13 Debug testing vs Operational testing redundancy Testing according to the operational profile means accepting a certain redundancy in the test cases. neglecting critical operations It is often argued that testing according to an operational profile means neglecting critical operations that have a low occurrence probability. With bad luck these operations might not be tested at all. input sequences When testing interactive or real time systems the relative frequencies not only of different inputs but also of input sequences have to be known or estimated for building an operational profile program units For program units an operational profile often cannot be determined at all,

14 14 Debug testing vs Operational testing Structural testing (branch covarage, unlikely paths) E.g Adams study of IBM operating systems» 5000 year failures were 33.4 of faults» 1.56 year failurea were 0.4 of faults Spend too much time on rare cases May introduce a higher likehood fault

15 15 Debug testing vs Operational testing Difficulties in estimating operational profile New software New features Feature definition Multiple or unknown user groups

16 16 Simulators 5 A simulator is an abstract model of a system which imitates the functions of a man-made system or a natural process(e.g weather system) [1] The system requirement analysis phase The design and construction phase The test and integration phase Specification Animators Component Environment Simulators Target System Drivers

17 17 Module Interface Structure Software Hardware - Environment

18 18 How good is our simulator? Any simulator is only as good as the mathematical or physical model upon which it is based Assesment of our simulator depends on [1] Variables simulated Which environment variables are included and which are ignored Accuracy of calculations The resolution and accuracy of the calculation used for each variable Timing consideration The effects of calculation speeds and update rates Hard/soft real time systems

19 19 Process of Interface Simulator

20 20 Statistical Software Testing 6 To perform SST it is necessary to : Construct an environment simulation for the software based on the software s operational distribution Conduct statistical inference from observations of the software s performance to produce an estimate of the probability of failure and a measure of the confidence in that estimate

21 Theoretical approaches to statistical software integrity prediction [SEMINER] Advanced Topics in Software Engineering for Safety-Critical Systems 21 Single Version Testing Single unchanging piece of software Time Based Models Estimation is based on observed times between succesive failures Demand Based Models Estimation is based on number of failed demands selected from the operational input distribution Reliability Growth Modelling Succesive releases of software product

22 22 Software Reliability 7 software reliabılity is the probability that a given software program will operate without failure for a specified time in a specified environment. software reliability engineering (sre) is the application of statistical techniques to data collected during system development and operation to specify, predict, estimate, and assess the software reliability of software-based systems. How error free is the software? When should I stop testing?

23 23 Software Reliability Engineering SRE will help you to Satisfy customer needs more precisely Deliver earlier Increase productivity Plan project recources better Currently over 75 models in the Literature Standards already exist, e.g. IEEE and 982.2

24 24 Conclusion 8 Having efficient testing techniques and processes is obligatory. Operational testing is complemantary or alternative Simulators / statistical inference and related models are indispensable SRE is the future

25 25 Thank you for your patience! References [1] Testing Safety-Related Software : a Practical Handbook ; Stewart Gardiner; 1999; Springer. [2] Hard Real-Time Computing Systems - Predictable Scheduling Algorithms and Applications, Giorgio C. Buttazzo, Kluwer Academic Publishers [3] Handbook of Software Reliability Engineering, Michael R. Lyu, Computer Society Press [4] Evaluating Testing Methods by Delivered Reliability, Phyllis G. Frankl et al.