PROPANE: An Environment for Examining the Propagation of Errors in Software Λ

Transcription

1 : An Envirment for Examining the Propagati of Errors in Software Λ Martin Hiller Arshad Jhumka Department of Computer Engineering Chalmers University of Technology Göteborg, Sweden Neeraj Suri ABSTRACT In order produce reliable software, it is important have knowledge how faults errors may affect the software. In particular, designing efficient error detecti mechanisms requires not ly knowledge which types of errors detect but also the effect these errors may have the software as well as how they propagate through the software. This paper presents the Propagati Analysis Envirment () which is a ol for profiling cducting fault injecti experiments software running deskp computers. supports the injecti of both software faults (by mutati of source code) errors (by manipulating variable memory ctents). supports various error types out-of-the-box has support for user-defined error types. For logging, probes are provided for charting the values of variables memory areas as well as for registering events during executi of the system under test. has a flexible design making it useful for development of a wide range of software systems, e.g., embedded software, generic software compents, or user-level deskp applicatis. We show examples of results obtained using how these can guide software developers where software error detecti recovery could increase the reliability of the software system. Keywords Error Propagati Analysis, Software Reliability, Fault Injecti, Software Development Tools. INTRODUCTION In order develop software that functis in a n-harmful manner in the presence of faults errors (as defined in [9]), e requires knowledge of the behavior of the software under these exceptial cditis. In particular, e needs know how faults Λ Supported in part by Volvo Research Foundati (FFP-DCN), NUTEK (P ), Saab, NSF Career CCR 98963, by EC DBench (IST ) Permissi make digital or hard copies of part or all of this work or persal or classroom use is granted without fee provided that copies are Permissi make digital or hard copies of all or part of this work for not persal made or or distributed classroom use for profit is granted or commercial without fee advantage provided that that copies copies are bear not made this notice or distributed the full for profit citati or commercial the first page. advantage To copy otherwise, that copies republish, bear this notice post the servers, full citati or redistribute the first page. To lists, copy requires otherwise, prior specific republish, permissi post /or servers a fee. or redistribute lists, requires prior specific permissi 00 ACM /or $5.00 a fee. ISSTA 00 Rome, Italy Copyright 00 ACM /0/ $5.00. errors propagate affect the executi of software. Knowing propagati pathways may, for instance, be of great help when deciding where place error detecti recovery mechanisms. Learning about error propagati characteristics of a software system requires not ly that e should be able inject errors mir the effect these have system output, but also that e is able mir how these errors are transported through the system. Thus, high observability is required for these activities. Ideally, e should be able observe every individual variable structure in the software. This paper presents the main features of (details are available in [6]), the Propagati Analysis Envirment, which enables the injecti of primarily errors (e.g. erreous variable ctents) but also faults (e.g. source code defects) in software running a deskp computer (currently for Windows NT/000). supports various ways of probing a system, i.e., tracing internal variables events during system operati, as well as ways of injecting software faults errors. can be useful in a number of situatis. For instance, in Compent-Based Software Development (CBSD) generic cfigurable software compents are manufactured assembled form an entire system (inspired by the use of generic hardware compents for building hardware systems). These compents are often ported several different hardware platforms. This limits generalized verificati validati use of ols that focus specific hardware cfiguratis. the other h has no such limitatis as it is does not require any special hardware assistance. Thus, software compents may be verified validated with before porting them various target hardware. This argument will of course also be valid for testing embedded software which in many cases may exist before the hardware platform has been finalized. We emphasize that, through its depicti of error propagati paths, is primarily designed as a software design aid with complementary capability of being used in the evaluati of effectiveness of error hling mechanisms. The remaining paper is structured as follows: In Secti we describe the target system model for which is aimed. Secti 3 describes the ol suite it s main features. An example of actual usage is shown in Secti 4. In Secti 5, we shortly compare some similar ols. Finally, in Secti 6 we summarize this paper.. TARGET SYSTEM MODEL aims at modular software, i.e., discrete software functis interacting deliver the requisite functiality. A module in 8

2 this ctext is a generalized software block having possibly multiple inputs outputs. Modules communicate with each other in some specified way using varied forms of signaling, e.g., shared memory, messaging, parameter passing etc., as pertinent the chosen communicati model. A software block performs computatis using the provided inputs generate the outputs. At the lowest level, such a software block may be a procedure or a functi but could also cceptually be a basic block or particular code fragment within a procedure or functi (at a finer level of software abstracti). A number of such modules cstitute a system they are inter-linked via signals, much like hardware compents a circuit board. Of course, this system may be seen as a larger compent or module in an even larger system. Software cstructed as such is found in numerous systems deskp systems as well as embedded systems. For example, most applicatis ctrolling physical events, e.g. in aumotive systems, are traditially built up as such. Our studies mainly focus software developed for embedded systems in csumer products (high-volume low-producti-cost systems). The envirment is designed with a focus software for single-process user applicatis deskp systems. However, this single process may be multi-threaded. The injecti logging mechanisms are generic are provided in a static C-library, thus allowing for a vast range of applicatis. For example, it has been used in experimentally analyzing the propagati of errors in the software of an embedded ctrol system simulated a Windows-based deskp computer [7, 8]. The requirement for using is that the language used for the source code is able interface with libraries implemented in the C programming language. 3. MAIN FEATURES OF This secti provides an overview of the main features of the ol suite, how it is structured, its proposed usage. 3. Basic system structure is designed run a deskp system csists of a suite of ols, namely: the Setup Crear (PSC), the Campaign Driver (PCD), the Library (PL), the Data Extracr (PDE). An overview is shown in Fig.. The campaign driver invokes the target executable run experiments Setup Crear PSC Campaign Driver PCD Setup Log Target executable Readout Envirment simular User Error Types Library PL Data Extracr PDE Target software User Error Triggers Extracted Figure : An overview of gether with target software envirment simular The PL is used by the target system gain access the probing injecti functiality of is written in the C programming language. The PCD is respsible for hling the actual executi of experiments is in a sense the main administrar of. It has a user interface through which the user can ctrol follow the experiments. The PDE may be used during analysis extract specific from the experiment readout. The PCD the PL are integrated with each other, whereas the PSC the PDE are st-ale compents of. The envirment simular target software are provided by the user. The envirment simular will act as a stimuli generar for the target software may be partially ctrolled by the output generated by the target software (e.g., as in a ctrol loop). The interactis between these two sub parts of the target executable are user-defined. The PSC aids in the creati of setup needed for ctrolling during the executi of FI-experiments. Given informati regarding errors faults, probes, injecti locatis, etc., it will generate the requisite descripti. The PSC will also generate descripti used by the PDE during analysis. For each experiment specified in the descripti, the PCD spawns a new process running an executable file ctaining a complete specificati for cducting e experiment. This executable ctains the PL which performs the actual injecti of errors logging of variables. The executable also has ctain everything necessary run the target system the envirment simular. During the executi of the experiments, log readout are created. The log ctain informati regarding the executi of the experiments, i.e., performance behavior informati, does not ctain any readout gathered from the target software. If the experiment could not be executed successfully for some reas, the log provide hints potential problems. The readout ctain the obtained by the inserted probes the performed injectis are the basis for subsequent error propagati analysis. The envirment simular is designed by the user of the ol, hence it may or may not use descripti may or may not create log /or readout as per user-specified requirements. Also, the format of the read /or written by the envirment simular is userdefined. The PL requires interfacing the envirment simular. However, if an envirment simular exists which does not comply with the interface specificatis, a wrapper layer is warranted which has the interface e side the envirment simular interface the other, acting as a translar between the two compents. Thus, the envirment simular need not necessarily be an integrated part of the target executable. The PDE will extract traces of the various logged variables memory areas can cduct Golden Run Compariss (i.e. comparing system traces obtained during injecti experiments with fault/error free reference traces, so called Golden Runs) detect whether errors have occurred due fault injecti. Informati regarding propagati will be compiled presented. Also, intermediate extracted is sred in special which can subsequently be used in a cusmized analysis ols which may take in account desired experiment specific informati /or aims, such as coverage estimati of error hling mechanisms, failure classificati or other activities which may be target specific. 3. Work process for using The typical work process when using can basically be divided in three main phases, namely: ) Setup, ) Injecti, 3) Analysis (as illustrated in Fig. ). Setup phase: In the Setup phase, descripti are generated the target system is instrumented. The inputs this phase include the original source code of the target software, informati 8

3 Original target software Fault error Usage profile SETUP Descripti Instrumented target software INJECTION Readout Log ANALYSIS Analysis phase: The readout generated in the Injecti phase are analyzed in the Analysis phase evaluate metrics for the target systems. These metrics may include coverage values, propagati informati, etc. One aspect of analysis is compare traces from two different runs with each other, e.g., compare a golden run(i.e. a reference run) with an injecti run. The Data Extracr compiles propagati informati from the readout also generates a set of - ctaining such as detailed results Golden Run Compariss, injecti informati, propagati informati, etc. Results Figure : The basic work process when using. distributi nature of faults /or errors informati about target system usage. The fault error informati is used for determining the fault error sets be injected in the experiments. The usage informati forms the basis for determining the test cases used during the injectis in order provide the target system with a realistic operatial profile. Instrumentati of the target system means adding probes for logging variables, memory areas, events, as well as with high-level software traps for injecting faults/errors the source code. At this point, target instrumentati is still a manual task. However, a ol for aumatic instrumentati is currently being developed will be added the suite. Given basic informati about errors faults, probes, injecti details, etc., PSC generates the required descripti for PCD/PL PDE. The descripti ctain informati which faults are be injected, which errors are be injected at which locatis, which test cases are be used by the envirment simular during the executi of experiments. Injecti phase: During the Injecti phase, the Campaign Driver (PCD) is set up with the descripti generated in the Setup phase. The PCD invokes the target executable as an individual process generates readout ctaining detailed informati the results of the experiments. During the experiment, the specified faults /or errors are injected the specified variables events are logged. Log- are generated recording the actis of the ol itself. Faults are injected when the correspding fault-triggers are activated. Fault injecti at this level means that a faulty piece of code is executed instead of the correct piece. Errors are injected based the built-in error types, or userimplemented error types. Thus, it is possible implement error models which are not originally included in. For example, if some parts of a system work unreliably under extreme temperatures, a user error type could take this in csiderati. Error-triggers are boolean expressis an error is injected when its correspding error-trigger is evaluated true. Errortriggers may be based time, frequency or a probability distributi. In additi the built-in error-triggers, also supports user-implemented error-triggers. As was the case for user error types, a user error-trigger may take in account target specific informati, such as system state or the envirment. In the example with the temperature-induced error type, a correspding errortrigger may evaluate true when the temperature (obtained from the envirment simular) is below a lower threshold or above an upper threshold (or both). 4. EXAMPLE RESULTS GENERATED BY This secti presents example results obtained using. In [7], we used the software of an embedded ctrol system for arresting aircraft short runways (such as aircraft carriers). The system aids incoming aircraft reduce their velocity, eventually bringing them a complete sp. The structure of the software is illustrated in Fig. 3. The numbers shown at the inputs outputs are used for numbering the signals. For instance, PACNT is input # of DIST S, SetValue is output # of CALC. We used the actual software ported it run a Windowsbased computer. The scheduling is slot-based n-preemptive. Thus, from the software viewpoint, there is no difference in running the actual hardware or running a deskp computer. Rotati sensor HW counter Pressure sensor PACNT TIC TCNT 3 ms_slot_nbr CLOCK DIST_S 3 ADC PRES_S IsValue mscnt pulscnt slow_speed spped V_REG i 3 CALC 4 5 SetValue PRES_A OutValue TOC Figure 3: SW structure of the example system. The software is composed of six modules of varying size input/output signal count. CLOCK provides a clock, mscnt, a signal indicating the current executi slot, ms slot nbr. DIST S receives PACNT TIC from sensors are used calculate the distance an aircraft has traveled the runway, pulscnt. It also provides two boolean values, slow speed spped, i.e., if the velocity of the aircraft is below a certain threshold or if it has spped. CALC uses mscnt, pulscnt, slow speed spped calculate SetValue, the preferred value for the system actuars. PRES S reads the value that is actually being applied by the actuars, ADC, provides the signal IsValue. V REG uses SetValue IsValue generate OutValue, the output value the actuars. The modules attempts compensate for the difference between SetValue IsValue. PRES A uses OutValue set the actuar via the hardware register TOC. We injected bit-flip errors in each of the signals (e at a time) mired all the signals. Details the setup further results of this experiment can be found in [7]. During analysis, the PDE extracts vital informati for the assessment of error propagati for each individual experiment run but also for groups of experiments. Due space limitatis, we will in this paper ly show examples of results for groups of ex- 83

4 840 errors PACNT 840 0/0/0 0 0/0/0 pulscnt 69 0/0/0 67 /5/333 i SetValue 307 9/96/ // 8 0/0/0 TOC 4 0/36/ /0/35 5 0/358/ /5/333 ADC 35 7/96/ /96/ /5/5 9 38/44/45 IsValue 0 88/88/88 8 // 9 43/083/ /57/ /08/ /08/ /08/ /08/593 slow_speed 9 /856/80 9 /856/ /890/349 9 /856/80 9 /856/ /96/47 mscnt TCNT TIC ms_slot_nbr OutValue Figure 4: Propagati graph (generated by the dot ol) for errors injected in PACNT. periments. PDE generates ccise informati pertaining the propagati of the injected errors in the system. For each signal that is subjected error injectis, a propagati graph propagati summary will be generated. The PDE sres the propagati graph in two different file formats: i) dot [], ii) GML [4]. As these formats are comm for graph representati, there is a range of applicatis that can be used for plotting manipulating the propagati graphs. In Fig. 4 we can see the propagati graph for errors injected in the PACNT in the example system used in this secti. The graph is generated using the dot ol. The propagati graph illustrates the propagati characteristics of the errors injected in the signals PACNT. The label an arc from e node another tells how many errors propagated alg this arc (p value), the minimum, average maximum propagati times (botm values) for these errors. The graph shows the temporal order between errors in different signals. For example, if we csider the errors detected (during the Golden Run Comparis) in i, we can see that for 0 of them, there were no errors detected earlier in other signals (although errors were detected in pulscnt at the same point in time), whereas for 69 of the detected errors, there were error detected earlier in pulscnt. Using the same example experiment as above, we show the generated propagati summary for errors in PACNT in Table. The summary is obtained by collapsing all ingoing arcs of each node in the propagati graph. Thus, e.g., the summary for i is obtained by adding its two ingoing arcs in the propagati graph, which gives us a tal of 8 errors. The propagati times are obtained from the combined set of propagati times for the errors detected in i. Table : Propagati of errors injected in PACNT. error count is the number of errors detected using Golden Run Comparis the error rate is the same informati normalized. The propagati times are all in millisecds. Signal error count error rate tmin tavg tmax PACNT pulscnt i OutValue SetValue TOC ADC IsValue slow speed mscnt ms slot nbr TCNT TIC In the summary shown in Table we see the number of errors in PACNT that caused errors in other signals (count rate), as well as the minimum, average maximum propagati time for these errors (the rows are ordered according their average propagati time). In this particular example we can see that all of the 840 errors injected in PACNT, propagated pulscnt with an average propagati time of 0 ms. 75 errors made it all the way the output signal TOC with an average propagati time of 65 ms. 84

5 From the software structure shown in Fig. 3 we can see that errors in the signals listed below TOC in Table (except slow speed), must be indirect, since there is no direct path from PACNT. Thus, errors in this signal must have propagated out of the system in the envirment then back in the system again. The results presented give informati how errors propagate through the system, identifying which modules signals that may be in need of special mechanisms for protecti against propagating errors. For example, from the results in Table we see that errors in PACNT mainly propagate through DIST S in CALC using pulscnt. From the propagati graph in Fig. 4 we see that propagati in CALC is fast, whereas propagati out of CALC takes a little lger. Thus, CALC seems delay the propagati of errors. We also see that after CALC, error propagati again is swift. These results would indicate that system reliability could increase if pulscnt were be equipped with with EDMs (error detecti mechanisms) ERMs (error recovery mechanisms), as this would likely break the propagati at an early stage. These examples demstrate s capabilities for generating pertinent informati for propagati analysis. However, the level of detail required may generate very large amounts of raw. In order further analyse this raw (further than de by the PDE) additial actis can be performed reduce the raw in useful informati. We refer the reader [7, 8], where details of actual results, as well as two different analysis frameworks (with different objectives) are described. 5. OTHER TOOLS There are other ols for injecti of errors faults, e.g. DE- PEND [5], Xcepti [], MAFALDA [3], NFTAPE [0]. DE- PEND is aimed at evaluating architectures thus the granularity of the obtained results is at the system level (or node level for distributed systems) thus cannot aid in charting error propagati at the variable level. Xcepti is targeted for evaluati of fault lerance against HW faults its results are also at the system level. Also, Xcepti cnects directly the hardware of a system thus has a tight link the target processor. The aim of MAFALDA is evaluati of the robustness of microkernels investigating the effect of software faults software errors the operati of these kernels. This means that the ol is able inject at the OS-level. is aimed at software at the USER-level, hence it is not suited for these types of investigatis. However, as far as we know, MAFALDA lacks comprehensive logging facilities for examining the propagati of errors in a microkernel. NFTAPE is, in our opini, a very versatile ol which can perform the same investigatis can. NFTAPE, just like, has support for user-defined injecrs as well as userdefined triggers, is capable of observing the target system at the variable level. As both ols have support for user-defined injecrs, both may be extended hle physical fault injecti as well as SWIFI. However, NFTAPE is designed run a LAN, has therefore a separate ctrol host a target node. 6. SUMMARY This paper briefly presents, the Propagati Analysis Envirment, which is a software design-stage profiling ol suite developed for analyzing the propagati effect of errors in software systems. is a deskp envirment ctains support for cducting fault error injectis in target software systems. The ol also provides support for inserting probes in the target system enabling the logging of variables events during injecti experiments. is tally target system independent, i.e., it may be used any target system provided that e can execute it in a deskp envirment. Also, does not require any HW or OS support is easily ported other operating systems (the current versi is available for Windows NT/000-based computers). As is implemented using ANSI C, porting it is mostly just a questi of recompiling for the desired envirment. The injecti capabilities include fault injecti by mutati of source code as well as SWIFI-based injecti of errors. supports user-defined injecrs triggers which makes it capable of supporting other injecti techniques than SWIFI (for example, physical fault injecti). supports observatis down the variable level, i.e., individual variables may be logged during injecti experiments. This enables the detailed examinati of error propagati in software is a valuable help in finding vulnerable software modules /or variables. For analysis, the olkit ctains the Data Extracr, which can perform Golden Run Compariss for each channel created by a variable in the readout. The results will be sred in a text file with a spreadsheet format that is easily imported in other ols for further analysis. The results from the GRC are also compiled show where errors propagate through the system how lg time it takes. The PDE can also extract injecti informati from the readout sre this in separate, create channel logs for each individual channel of each individual experiment if a more detailed analysis or graphical representati is desired. Also, PDE creates propagati graphs summaries which visualize the propagati characteristics of the software system. To demstrate the ol we have shown detailed results from an injecti experiment performed a medium sized embedded ctrol system used for arresting aircraft (similar the cable--hook systems found aircraft carriers). 7. REFERENCES [] Carreira J., et al., Xcepti: A Technique for the Experimental Evaluati of Dependability in Modern Computers, IEEE Trans. Software Eng., Vol. 4, No., pp. 5-36, 998 [] Informati about the ol suite which dot belgs is found at [3] Fabre J.-C., et al., Assessment of COTS Microkernels by Fault Injecti, Int. IFIP Cf. Dependable Computing for Critical Applicatis, 999 [4] Informati about GML related ols is found at [5] Goswami K.K., et al., DEPEND: A Simulati-Based Envirment for System Level Dependability Analysis, IEEE Trans. Comp., Vol. 46, No., pp , 997 [6] Hiller M., A Tool for Examining the Behavior of Faults Errors in Software, TR 00-9, Dept. of CE, Chalmers Univ., (available at [7] Hiller M., et al., An Approach for Analysing the Propagati of Data Errors in Software, Int. Cf. Dependable Systems Networks, pp. 6-70, 00 [8] Jhumka A., et al., Assessing Inter-modular Error Propagati in Distributed Software, Symp. Reliable Distributed Systems, pp. 5-6, 00 [9] Laprie J.-C. (ed.), Dependability: Basic Ccepts Terminology, Dependable Computing Fault-Tolerant Systems series, Vol. 5, Springer-Verlag, 99 [0] Stt D.T., et. al, NFTAPE: A Framework for Assessing Dependability in Distributed Systems with Lightweight Fault Injecrs., Int. Computer Performance Dependability Symposium, pp. 9-00,