Developing Safety-Critical Systems: The Role of Formal Methods and Tools

Transcription

1 Developing Safety-Critical Systems: The Role of Formal Methods and Tools Constance Heitmeyer Center for High Assurance Computer Systems Naval Research Laboratory Washington, DC Abstract In recent years, many formal methods have been proposed to improve the quality of safety-critical software systems. These methods include new specification and modeling languages as well as formal verification techniques, such as model checking and theorem proving. This paper describes numerous ways in which tools supporting formal methods can improve the quality of both software code as well as software specifications and models. However, while promising, formal methods and their support tools are rarely used in software practice. To overcome this problem, I propose several needed improvements, which could lead to more widespread use of formal methods in the development of safety-critical systems and software. Keywords: formal methods, formal specification, formal verification, model checking, theorem proving, software tools, software testing 1 Introduction During the past two decades, many specification and modeling languages have been introduced whose purpose is to precisely describe the required behavior of a software system at a higher level of abstraction than the code. Examples of such languages include the synchronous languages, such as Lustre (Halbwachs 1993); the modeling language Statecharts (Harel 1987); requirements languages such as RSML (Heimdahl & Leveson 1996), and SCR (Heitmeyer, Archer, Bharadwaj & Jeffords 2005); and design languages offered by industry, such as UML and Stateflow (Inc. 1999), a Statecharts variant included in Simulink. Specifications and models expressed in these languages can have important advantages in software development. First, they provide a solid foundation for later development stages i.e., design, implementation, coding, testing, and maintenance. Moreover, because they usually exclude design and implementation details, these specifications and models are more concise than code and, as a result, can serve as an effective medium for customers and developers to communicate precisely about the required behavior. In addition, they allow both manual and automated anal- Copyright c 2005, Australian Computer Society, Inc. This paper will appear in the Proceedings of the 10th Australian Workshop on Safety Related Programmable Systems, August, 2005, Sydney, Australia. This research is supported by the Office of Naval Research. ysis of the required behavior. Analyzing and correcting a software specification is usually cheaper than analyzing and correcting the code itself because the specification is most often smaller thus finding and correcting bugs in a specification is much easier than finding and correcting bugs in code. In recent years, significant advances have also occurred in formal verification. Available since the early 1990s are automated theorem provers, such as PVS (Owre, Shankar & Rushby 1993), as well as model checkers, such as Spin (Holzmann 1997) and SMV (McMillan 1993). These tools may be used to analyze a specification, a model, a design, or even software code for critical properties, such as safety and security properties. Because the analysis performed by the tools may be largely automatic in the case of model checkers, or interactive in the case of theorem provers such as PVS, such tools may be highly cost-effective for verifying (or refuting) that a software or hardware artifact satisfies a given property. In 1997 and 1998, I presented my views on both the promise of formal methods and the barriers to their acceptance (Heitmeyer 1997, Heitmeyer 1998). While confident that formal methods had significant potential for improving the quality of both hardware designs and software, I also believed that the barriers to widespread use of formal languages and tools were formidable most of the tools were simply too hard to use, required mathematically sophisticated users, and would also require, at least in the software domain, significant changes in the process used by industry to develop software. During the intervening eight years, a number of important advances in both hardware design and software development have occurred. First, many formal methods, especially those based on model checking and theorem proving, have begun to play a significant role in industrial hardware design. Second, among software developers, the popularity of modeling languages, such as UML and Stateflow, has increased significantly. In addition, the state of the art in formal verification has advanced dramatically. One of the most important advances in verification has been in decision procedures, algorithms which can decide automatically whether a formula containing Boolean expressions, linear arithmetic, enumerated types, etc., is satisfiable (see, e.g., (Stump, Barrett & Dill 2002, Filliatre, Owre, Ruess & Shankar 2001)). Many decidable theories exist for specifying and reasoning about propositional logic, real and integer arithmetic, restricted forms of arrays, etc. By combining these theories, decision procedures can automatically determine the satisfiability of a logical formula. These powerful decision procedures will significantly enhance the efficiency and power of theorem provers and some model checkers, such as infinite state model

2 checkers (see, e.g., (Bultan, Gerber & Pugh 1997)). Despite these advances, however, formal methods and their support tools are still rarely used in software practice. After briefly reviewing a number of ways in which formally-based tools could help developers significantly improve the quality of both code and software artifacts, this paper describes five areas where improvements are needed in software development. These improvements should not only increase software quality directly but should also encourage the application of existing formal methods in practical software development. 2 On the Role of Tools Many automated techniques and tools have been developed in recent years to improve the quality of software and to decrease the cost of producing quality software. Such tools can play an important role in obtaining high confidence that a software system is correct, i.e., satisfies its requirements. Described below are five different roles that tools can play in improving the quality of both software and software artifacts. The first four help improve the quality of a specification or model. The fifth uses a high-quality specification as the basis for constructing a set of tests useful in checking and debugging software code. See (Heitmeyer 2003) for examples which demonstrate how each class of tools can both improve the quality of real-world software and software artifacts and lower the cost of developing them. 2.1 Demonstrate well-formedness A well-formed specification is syntactically and type correct, has no circular dependencies, and is complete (no required behavior is missing) and consistent (no behavior in the specification is ambiguous). Tools, such as NRL s consistency checker (Heitmeyer, Jeffords & Labaw 1996), can automatically detect wellformedness errors and can also provide feedback useful in determining the cause of an error. 2.2 Discover property violations In analyzing software or a software artifact for a property of interest, a tool, such as a model checker, may uncover a property violation. By analyzing the counterexample returned by the model checker, a developer may trace the problem to either a flaw in the specification or to one or more missing assumptions. Alternatively, the formulation of the property, rather than the specification, may be incorrect. Detecting and correcting such defects can lead to higher quality specifications and to higher quality code. 2.3 Verify critical properties Either a theorem prover or a model checker may be used to verify that a software artifact, such as a requirements specification or a design specification, satisfies a critical property. Verifying that a software artifact satisfies a set of properties can help practitioners develop high confidence that the artifact is correct. 2.4 Validate a specification While a verification tool helps ensure that a specification is right, i.e., correct with respect to certain properties, a validation tool checks that the specification is the right specification, i.e., captures the intended software behavior. By running scenarios through a simulator (see, e.g., (Heitmeyer et al. 2005)), for example, the user can obtain confidence that the system specification neither omits nor incorrectly specifies the required software behavior. 2.5 Automatically construct a test suite Specification-based testing can automatically derive a suite of test cases satisfying some coverage criterion, such as branch coverage, from a formal specification (Gargantini & Heitmeyer 1999). Automated test case generation can be enormously useful to software developers because 1) the cost and time needed to automatically construct tests is much lower than the cost and time needed to manually construct tests, and 2) a suite of test cases mechanically generated from a specification usually checks a wider range of software behaviors than manually generated tests and hence may uncover more software defects. 3 Needed Improvements During the past two years, we have applied formal specification and formal verification techniques to two classes of real-world software. One class consists of software components for space systems under development by NASA these include the Fault Protection Engine (Feather, Fickas & Razermera-Marny 2001) and two modules to be used in the International Space Station the FDIR (Fault Detection, Isolation, and Recovery) module and the Incubator Display module. Another class contains software for a system called CD (Cryptographic Device) II, the second member of a family of systems, which performs cryptographic processing, e.g., decrypts and encrypts data stored on two or more communication channels (Kirby, Archer & Heitmeyer 1999). An essential requirement of CD II is to enforce data separation, i.e., to guarantee that data stored on one channel does not influence nor is influenced by data stored on a different channel. Satisfying this property is critical since data stored on one channel may be classified at a different security level than data stored on another channel. In the NASA projects, the goal has been mostly to construct, simulate and formally analyze a formal requirements specification of a software module. In the CD II task, our group produced 1) a formal Top Level Specification (TLS) in the style of (Landwehr, Heitmeyer & McLean 1984) and 2) a formal statement of the data separation property; translated the TLS and the property into TAME (Archer 2001), a front-end to PVS that represents a system as a state machine; and proved using the TAME strategies and PVS that the TLS satisfies data separation. In the CD II task, we also showed the correspondence between the TLS and security-relevant portions of the CD II code. For the potential of existing formal methods to be realized in software development, our experience in applying formal methods to these systems suggests that new technology and new research are essential. Described below are five areas where progress is needed. 3.1 Techniques for Requirements Capture One critical problem in current software practice is poorly understood and poorly documented requirements. Although some promising approaches, such as scenarios and use cases, have been introduced in industry to help developers understand software requirements, better technology is needed to help software developers construct and analyze scenarios and transform the requirements knowledge obtained from

3 scenarios into precise, consistent, and complete requirements specifications. One of the most difficult problems we encountered in applying formal methods to CD II was obtaining and documenting the behavior of the CD II software that was relevant to data separation. Without this information, formulating a formal Top Level Specification for CD II was impossible. To enhance our understanding of the behavior of the software relevant to data separation and to discover gaps in our understanding of this software, running scenarios through the simulator in the SCR toolset (see (Heitmeyer et al. 2005) for details) proved highly effective. 3.2 Specification and Modeling Languages Most of the specification and modeling languages that have been proposed by researchers have an explicit formal semantics, which makes the meaning of specifications in these languages precise and thus amenable to formal analysis. Although these formal languages have been applied effectively in a few specialized areas, for example, in control systems for nuclear power plants and in avionics systems, they are still not widely used by software developers. While languages, such as UML and Stateflow, are more popular in industry, they lack an explicit formal semantics and often the specifications and models that practitioners produce using these languages include significant design and implement details. Given the lack of formal semantics and the large specifications and models that result from too many details, the opportunity to analyze UML and Stateflow models models using formally-based tools is severely limited. To improve this situation, existing languages either need to be enhanced with features, such as fancy graphical interfaces, that encourage their use by practitioners, or new languages need to be invented. One promising approach is the design of languages for specialized domains. For example, one specification language could be designed to specify and model the required behavior of software in automobiles, while different specification and modeling languages could be designed to specify and model the required behavior of other software, such as avionics software or software in medical systems. As van Wyk and others have observed (Wyk, Heimdahl & Saad 2004), years ago, MathWorks, the company which markets Simulink and Stateflow, recognized the importance of domain-specific languages: the popularity of their development environment, MATLAB, grew when Math- Works began including toolboxes customized for specific applications. Such toolboxes allow developers to construct rapid prototypes of their systems. The benefits of specification and modeling languages with an explicit formal semantics which minimize implementation detail could be enormous. Precise, unambiguous specifications can be analyzed automatically for well-formedness, such as syntax and type correctness, consistency (no unwanted nondeterminism), and completeness (no missing cases); for critical application properties, such as security and safety properties; and for validity (to check that the specification captures the intended behavior). Specifications that are well-formed, correct with respect to critical application properties, and validated using simulation provide a solid foundation both for automatic test generation and for generating correct, efficient source code. 3.3 Specifications and Models Another very serious problem in current software development is the absence of high quality specifications and models. When they exist at all, the specifications and models produced in industry usually include significant design and implementation detail (i.e., are close to the code). The result is large, hard to understand specifications and models, which do not distinguish between the required software behavior and implementation detail. In part, this problem can be solved through education. The attributes of good specifications and models and how to construct them are topics that need more emphasis in software engineering curricula. The problem of poor quality specifications and models can also be ameliorated by improved specification and modeling languages such languages would reduce the opportunity for implementation bias and contain mechanisms that encourage the construction of precise, readable specifications and models that can be checked both manually and mechanically for completeness and consistency. Good examples of high-quality specifications and models would also help practitioners produce better specifications and models. 3.4 Methods for Building Code However, improved languages and improved specifications and models are not enough. In the end, what is needed is correct, efficient code. As noted above, an important benefit of a formal specification is that it provides a solid basis for automatically generating provably correct, efficient code. While some software developers currently use automatic code generators, the code produced by these generators is often inefficient and wasteful of memory. Needed are improved methods for automatic generation of correct, efficient code from specifications. Another promising approach to producing high quality code is to use safe languages, such as Cyclone (Trevor, Morrisett, Grossman, Hicks, Cheney & Wang 2002), a version of C which eliminates code vulnerabilities, such as uninitialized variables and potential sources of buffer overflows and arithmetic exceptions. A third promising approach to constructing high quality code is to encourage programmers to annotate their code with assertions that can be checked by a compiler. Hardware designers routinely include such assertions in their designs. Moreover, some C, C++, and Java programmers routinely use assertions as an aid in both detecting and correcting software bugs. 3.5 Techniques for Code Verification Although improved specifications and automatic code generation from these specifications can help improve the quality of software, in the near future, most source code will still be generated manually. Needed therefore are improved methods for showing that a manually generated program satisfies critical properties. One promising approach is to encourage programmers to annotate their code with assertions that can be checked automatically. While current compilers for C, C++, and Java, support and check code assertions, the set of assertions that can be analyzed is quite limited. Needed are compilers that can not only check simple Boolean equations and inequalities, e.g., x > 0, but more complex assertions, such as priv(p, x) = R (i.e., process P has read privileges for variable x). Such assertions can be translated automatically into logical formulas and then decision procedures can be used to check that the code satisfies these assertions. In addition to helping practitioners document and detect bugs in their code, such assertions may also be used to show that the code satisfies critical properties, such as security and safety properties. A technique which could automatically check code annotated with assertions for a security property,

4 such as data separation or confidentiality, would be extremely useful in developing a secure system. While one approach is to check directly that the code satisfies a specified security property, an alternative is to construct a high-level operational specification of the system s required behavior (e.g., a state machine model), prove mechanically that the high-level specification satisfies the property, and then verify automatically (or semi-automatically) that the code satisfies the specification. The benefit of a state machine specification is that it is closer to the code than a given property, and therefore proving that the code satisfies the specification is easier than proving that the code satisfies some abstract property. In checking CD II for data separation, we followed the latter approach: we constructed a high-level specification which represented CD II s behavior in terms of a state machine, used TAME and PVS to prove that the specification satisfies the data separation property, and then showed that the CD II code, annotated with assertions, satisfies the high-level specification. Manually annotating the code with assertions is highly labor-intensive. Also costly in terms of human effort is the manual process of demonstrating that the code assertions satisfy the high-level specification. More automation of the process we used in verifying the CD II code would have been extremely useful. Such automation would dramatically reduce the human effort needed to construct the assertions and to check the conformance of the assertions with the code and the formal specification. Manual annotation of code with assertions such as pre- and postconditions, while time-consuming, is well-understood, and, as mentioned above, automatic checking of simple code annotations is already supported by many source language compilers. However, generating more complex annotations automatically from code (e.g., constructing loop invariants) is a problem that requires more research. Moreover, using decision procedures to check the conformance of the code with a set of assertions is a promising approach for new research and should help significantly in mechanizing the process for code verification described above. Finally, checking the conformance of a set of validated assertions with a formal specification is also a problem that will require more research. 4 Summary Some improvements described above will require new research in requirements capture, specification and modeling languages, automatic construction and checking of complex code assertions, automatic code generation, and code verification against a set of assertions. Other improvements will require the transfer of existing research results, for example, of safe languages such as Cyclone and of formal techniques, such as model checking, and theorem proving, into software practice. In addition, better educated software developers should have the skills necessary to build high quality specifications and models and to annotate their code with assertions. Finally, existing methods and tools must be easier to use. The result of more powerful, more automated methods, better educated practitioners, and better engineered tools should allow software practitioners to construct software in an environment where tools do the tedious analysis and book-keeping. This will liberate software developers to translate vague notions of the required software behavior into precise, readable specifications that can be transformed either manually or largely automatically into correct, efficient code. Acknowledgments The use of formal methods in the NASA projects is joint work with Ralph Jeffords. The formulation of CD II s Top-Level Specification (TLS), the mechanized proofs in TAME and PVS, and the demonstration of correspondence between the TLS and the annotated code was joint work with Myla Archer and Beth Leonard. John McLean made major contributions in the formulation of the TLS and in the design of our verification process. Section 3.5 on code verification is based on discussions with Michael Colon, Beth Leonard, Myla Archer, and Ramesh Bharadwaj. References Archer, M. (2001), TAME: Using PVS strategies for special-purpose theorem proving, Annals of Mathematics and Artificial Intelligence 29(1-4), Bultan, T., Gerber, R. & Pugh, W. (1997), Symbolic model checking of infinite state systems using Presburger arithmetic, in O. Grumberg, ed., Proceedings of the 9th International Conference on Computer Aided Verification (CAV 1997), Vol of Lecture Notes in Computer Science, Springer, pp Feather, M., Fickas, S. & Razermera-Marny, N. A. (2001), Model-checking for validation of a Fault Protection System, in Proceedings, 6th International Symposium on High Assurance Systems Engineering (HASE 2001), IEEE Computer Society, pp Filliatre, J.-C., Owre, S., Ruess, H. & Shankar, N. (2001), ICS: Integrated Canonization and Solving, in Proceedings, Computer-Aided Verification (CAV 2001), Springer-Verlag, Paris, FR, pp LNCS Gargantini, A. & Heitmeyer, C. (1999), Using model checking to generate tests from requirements specifications, in Proceedings, 7th European Software Engineering Conference and 7th ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE-7), LNCS 1687, Springer-Verlag, Toulouse, FR, pp Halbwachs, N. (1993), Synchronous Programming of Reactive Systems, Kluwer Academic Publishers, Boston, MA. Harel, D. (1987), Statecharts: A visual formalism for complex systems, Science of Computer Programming 8(3), Heimdahl, M. P. E. & Leveson, N. (1996), Completeness and consistency in hierarchical state-based requirements, IEEE Transactions on Software Eng. 22(6), Heitmeyer, C. (1997), Formal methods: A panacea or academic poppycock, in Proceedings, 10th International Conference of Z Users (ZUM 97), Springer-Verlag, Reading, U.K. LNCS 1212 (invited). Heitmeyer, C. (1998), On the need for practical formal methods, in Proceedings, 5th International Symposium on Formal Techniques in Real-Time and Fault-Tolerant Systems (FTRTFT 98), Springer-Verlag, Lyngby, Denmark, pp LNCS 1486 (invited).

5 Heitmeyer, C. (2003), Developing high assurance systems: On the role of software tools, in Proceedings, 22nd Internat. Conf. on Computer Safety, Reliability and Security (SAFECOMP 2003), Edinburgh. (invited). Heitmeyer, C., Archer, M., Bharadwaj, R. & Jeffords, R. (2005), Tools for constructing requirements specifications: The SCR toolset at the age of ten, Computer Systems Science and Engineering 20(1), Heitmeyer, C. L., Jeffords, R. D. & Labaw, B. G. (1996), Automated consistency checking of requirements specifications, ACM Transactions on Software Engineering and Methodology 5(3), Holzmann, G. J. (1997), The model checker Spin, IEEE Transactions on Software Engineering 23(5), Kirby, J., Archer, M. & Heitmeyer, C. (1999), SCR: A practical approach to building a high assurance COMSEC system, in Proceedings, 15th Annual Computer Security Applications Conference (ACSAC 99), IEEE Computer Society, Phoenix, AZ, pp Landwehr, C. E., Heitmeyer, C. L. & McLean, J. (1984), A security model for military message systems, ACM Transactions on Computer Systems 2(3), The Mathworks, Inc. (1999), Stateflow for use with Simulink, User s Guide, Version 2 (Release 11), Natick, MA. McMillan, K. L. (1993), Symbolic Model Checking, Kluwer Academic Pub., Englewood Cliffs, NJ. Owre, S., Shankar, N. & Rushby, J. (1993), User guide for the PVS specification and verification system (Draft), Technical report, Computer Science Lab, SRI Int l, Menlo Park, CA. Stump, A., Barrett, C. & Dill, D. (2002), CVC: A cooperating validity checker, in Proceedings, 14th International Conference on Computer- Aided Verification (CAV 2002). Trevor, J., Morrisett, G., Grossman, D., Hicks, M., Cheney, J. & Wang, Y. (2002), Cyclone: A safe dialect of C, in Proceedings, USENIX Annual Technical Conf., Monterey, CA, pp Wyk, E. V., Heimdahl, M. & Saad, Y. (2004), From models to efficient code: It s all in the middle, in Proceedings, International Workshop on Software Engineering for High Performance Computing System Applications, Edinburgh, Scotland.