An Automatic Test Case Generator for Testing Safety-Critical Software Systems

Transcription

1 An Automatic Test Case Generator for Testing Safety-Critical Software Systems Mehdi Malekzadeh Faculty of Computer Science and IT University of Malaya Kuala Lumpur, Malaysia Raja Noor Ainon Faculty of Computer Science and IT University of Malaya Kuala Lumpur, Malaysia Abstract This paper presents the development of an automatic test case generator (ATCG) for testing safety-critical software systems based on the concepts of specification-based testing. The ATCG receives the specification of system under test in normal specification language form and the causes and effects are automatically extracted and it also will visualize the cause-effect graph specification model. Finally test cases are generated by using cause-effect graph software testing methods in detail combined with Boolean operator techniques. Often in ATCGs too many test cases are created but the results are not always perfect. In testing our ATCG, experimental results showed that quality test cases are produced and that redundant test cases are ignored. Keywords- cause effect graph; requirment specification; safety critical system; test case generator I. INTRODUCTION Electrical systems play a critical part in every aspect of our everyday life, from cooking in kitchens, to working in offices. In many cases this type of electrical systems is combined with embedded software. Malfunction of software of these systems could be disastrous to human life. In fact, the role of software has moved from simple financial or other mathematical software system to controlling and monitoring device which directly impacts human life. Many such important systems exist in application areas such as aerospace, transportation, defense, power-generation, and medical devices. The focus is on the software aspect of these systems. Because the correct operation of these systems depends on software, the possibility of serious damage resulting from a software defect is considerable and growing. As a result, we need some methods to predict and assess the safety and reliability of software. Generally, Software testing is a best method for validating and checking the correctness of those safety critical systems containing software. Therefore the safety critical software must be comprehensively tested, before it is put into practical use. Software testing of critical systems is time consuming and it uses up to half of the overall software cost. Therefore, automated testing is becoming significant in research and industrial communities. Testing has several procedures, including generating test cases, test case execution, comparing expected output with /10/$26.00 C 2010 IEEE actual results, correcting software faults, and managing the whole enterprise [1]. The emphasis of this work is to automate the generation of efficient software test cases. In order to generate test cases, this research has applied test case generation strategies in a black-box testing where testers apply different inputs and compare the output versus specification to validate the correctness of system under the test [2]. The tester does not know about the internal structure of the program and test cases are chosen according to the requirements declared in the specification. In other words, a specification offers an accurate description of the software s basic functionality aspects without more detailed information. Black Box testing consisted of several techniques for generating test cases, including Equivalence Partitioning Partition, Boundary value analysis and Cause-effect graphing. The first two techniques are important for data processing intensive applications while cause-effect graphing (CEG) is normally used for control intensive applications [2]. The focus of this work is on CEG technique for generating test cases. This method is a specification-based technique which allows the tester to create a combination of inputs for creating test cases. The natural language specification is broken down and analyzed, and then causes (inputs) and effects (intermediate results, messages, outputs) and the limitations (constraints) among the causes or effects are identified. The relation between causes and effects are shown as an acyclic Boolean logic network, called Cause-Effect Graph. In other words, a cause-effect graph is a specification model that is translated from natural language specification. The graph is then converted to a limited-entry decision table. Each column in the decision table represents a test case [2, 3]. This study attempts to address the development of an easy-to-use automated test case generator (ATCG). It receives the specification of system in natural language and converts them in good understanding cause effect graph model. A list of test cases is then automatically generated by using this tool. II. RELATED WORKS Initially, CEG had been used as a hardware testing method but later it was customized for software testing. Elmendorf [3] has developed the CEG notation and suggested an algorithm for creating tests from a CEG. Myers 163

2 [2] designed an algorithm to trace CEG methodically and selected an efficient set of test cases. Nursimlu and Probert [3] discussed about the strengths and weaknesses of Myers s approach and they presented possible corrections. They pointed out that one of main problems in Myers s method is that if a CEG has been written in different syntax even though the semantics are the same, a different decision table could be derived. They found methods based on Path Sensitization technique for improving decision table creation. Yokoi and Ohba [4] designed a Test Case Generator tool for generating test cases by using CEG. In this method for generating test cases, the natural-language specification is converted into a textual representation of the CEG. The authors assert that this tool improved the original method by arranging constraint descriptions among specifications and showed that it generates a decreasing number of test cases. In this tool the equivalence partitioning method was adopted in order to decrease the number of test cases. As the creation of a single CEG for a large software system is not practical, Tai et al. [5] suggested a strategy for CEG construction that recognizes most important software specification effects and makes CEG only for these critical effects.they recommended the methods based on the theory of operational profiles and the risk management techniques to rank effects by criticality. They have proposed a strategy for generating test cases by using cause-effect graphs (CEG) in conjunction with the Boolean Operator (BOR) test approach called CEG-BOR. This method concentrates on propositional logic representation of the software specification and also it is useful in finding Boolean Operator faults as well as finding other types of faults. One of the limitations of CEG-BOR method is that every cause is considered as an individual Boolean variable that may cause impracticable test cases to be generated [3]. They suggested combining the BOR method with another method named Meaningful Impact (MI) strategy. This method is based on the detection of missing and/or extra negation operators on individual variables. They called BOR+MI. Paradkar et al. [7] have compared the recent methods. They compared CEG-BOR method with Elmendorf s algorithm and Yokoi and Ohba s method for generating test cases. They mentioned CEG-BOR method has important benefits in compare to other methods. One advantage is that the size of a CEG-BOR test case for an effect node is linear with the number of nodes linked with the effect in the CEG. Desel et al. [8] proposed a testing technique which generates test cases for validation of high-level Petri nets in a systematic way. This method has been obtained from CEG technique. In their approach the association between cause and effects is represented by Petri net. The specifications of information system behavior are modeled into High-level Petri nets. They proposed a systematic method for testing of Petri net models that is named causeeffect-net-concept. It was derived from the concept of cause effect graphing. III. ATCG S STRATEGY Our work involved automating the techniques proposed by Tai et al [5, 6] for generating set of test cases from CEG. We have selected the CEG method as a basic method to develop our tool as this method transforms the specification of the system under test to an accurate and easy to understand model of specification with emphasis on critical aspects of software system without more detailed information. In addition to, deriving tests from the software specification helps to expose ambiguities and/or inconsistencies in the specification. These can then be fixed early in the development cycle at a minimum of cost. Furthermore, CEG techniques allow to the tester to use a combination of inputs [9]. CEG-BOR technique is chosen because it produces the minimum number of test cases to cover 100% of the functional requirements. In addition it is cost-effective method with potential to be automated [7]. Fig. 1 shows the architecture of the proposed tool. There are five main components, including read specification from file, produce cause-effect table, produce Boolean expression, visualize cause-effect graph and generate test cases. Figure 1. Natural language specification Read specification from text file Generate Cause and Effect Table Produce Related Boolean Expression Visualize Cause-Effect Graph Generate Test Cases for an Effect Architecture of the proposed tool. The functionality provided by each component is briefly described below: 164

3 A. Read Specification This component opens a text file consisting of a specification of a critical system that has been written in natural language. The first line of text file is an effect and the remaining lines are compound predicates which are predicates with one or more AND/OR operators. Each compound predicate includes several causes. This component generates tokens, which are input to the generate cause-effect table component. B. Generate Cause-Effect Table This component accepts the tokens generated by previous component, and recognizes each cause based on Boolean Operator (AND/OR) between causes. In addition the first line of text file states the effect. Finally, a causeeffect table will be created. C. Produce Boolean Expression This component produces a Boolean expression by reading causes from a cause-effect table, assigning variable to each cause and combining causes together with related Boolean operators. It replaces the Boolean operator between causes with equivalent mathematical one, i.e. OR to + and AND to *. D. Visualize Cause-Effect Graph The ATCG draws the CEG automatically based on Boolean expression which has been provided by previous component. In order to draw CEG, ATCG converts infix Boolean expression to postfix Boolean expression in the beginning. Because parsing a postfix Boolean expression is much easier than infix one. For example E=a*(b+c) is an infix Boolean expression, the related postfix one will be E=abc+*. In some cases, there are set of constraint for causes that are prepared by tester. Constraints represent external constraints on the system. For causes, valid constraint symbols are E (exclusive), O (one and only one), and I (at least one). The ATCG receives the set of constraints as input and shows them on generated CEG properly. Fig. 2 shows an algorithm which is used by ATCG for visualizing CEG. E. Generate test cases Generate test cases is a most important component in our tool. This component is designed to generate test cases in order to test critical software system. The test cases are generated automatically based on a Boolean expression. Like Visualize Cause-Effect Graph component, the test case component also applies postfix Boolean expression for test case generation. In this research, the BOR testing approach has been used in designing test case generation algorithm. This strategy utilizes cause-effect graphs (CEG) in conjunction with the Boolean Operator (BOR) test strategy [5]. In a CEG, an effect node and its associated portion of the CEG denote a compound predicate in terms of causes. Since a CEG represents a set of compound predicates, the BOR testing strategy can be applied to generate tests. In the BOR testing strategy, each cause node is viewed as a Boolean variable and has "t" or "f" as its value. BOR method produces a minimum BOR constraint set for a compound predicate. Below we describe an adapted version of algorithm BOR in order to produce a minimum BOR test set for a CEG. The nodes in a CEG are visited from cause nodes to effect nodes. A set of test for this node and its associated CEG is created when a node is visited. After the visit of an effect node, a test for the CEG is available. The following are number of definitions that must be understood before we show the test generation algorithm. Let A and B be two sets. A U B denotes the union of A and B, A B the product of A with B, and A the size of A. A B, called the onto from A to B. If both A and B have two or more elements, A B has several possible values and returns any one of them. For example, assume that A = {(a), (b)} and B = {(c), (d)}. Then, A B has two possible sets of elements: {(a,c), (b,d)} and {(a,d),(b,c)}. 1. Convert Infix BooleanExp to Postfix 2. For each character has read from PostfixBooleanExp 3. If the char is an operand 4. Create new node as a cause 5. Else { //the char is operator 6. If two previous nodes are causes { 7. If the recent operator is first operator 8. Create Intermediate node 9. Connect two previous cause nodes to this node 10. If the recent operator as the same as previous intermediate node operator 11. Connect two previous cause nodes to previous Intermediate node}//end if 12. If two previous nodes are intermediate node 13. Find the intermediate node that has same operator as recent operator and connect other intermediate node to this node. 14. If one of two previous nodes is intermediate and other node is cause node { 15. If the operator of intermediate node is the same as recent operator 16. Connect the cause node to intermediate node Else Create new intermediate node} }//End main else Figure 2. An algorithm for visualizing CEG from Boolean expression. 1) BOR algorithm For a cause node N, the minimum BOR test set is {(t), (f)}. Assume that N1 and N2 are nodes in a CEG and that Sl= {(t), (f)} and S2= {(t), (f)} are minimum BOR test sets for N1 and N2 respectively. The following three basic rules explain how to derive a BOR set of test cases for a node with none or both its inputs being intermediate nodes. a) A minimum BOR test set S for an AND node with N1 and N2 as inputs is constructed as follows: S_t=Sl_t S2_t S_f = (S1_f {t2}) U ({tl} S2_f) 165

4 Where tl S1_t, t2 S2_t, and (tl, t2) S_t. b) A minimum BOR test set S' for an OR node with N1 and N2 as inputs is constructed as follows: S'_f=Sl_f S2 _f S'_t = (S1_t {f2}) U ({fl} S2_t) Where fl S1_f, f2 S2_f, and (fl, f2) S'_f. c) A minimum BOR test set S" for a NOT node with N1 as the input is constructed as follows: S"_t = Sl_f S"_f=S1_t IV. CASE STUDY This section explains the application of ATCG Tool for a simplified boiler control and monitoring system. In order to apply ATCG as tool for generating test cases, the informal specification must be converted to formal specification language which is accepted by ATCG. The formal specification are saved in textual file, the first line of text file shows the effect of system under test and remained lines mention to the causes associated with that effect. The specifications for suggested system are shown in Fig. 3. In addition to, Table I shows a cause-effect table for boiler control system and also related CEG based on these critical specifications are represented in Fig. 4. Figure 4. by ATCG. The cause-effect graph for simplified boiler system generated The boiler should be shutdown. [Effect] If the water Level in a boiler is below the 20,000lb. If the water Level in a boiler is above the 120,000lb. If (a water pump has failed OR a pump monitor has failed) AND Steam meter has failed. Figure 3. The specification of simplified boiler system. TABLE II. THE SET OF TEST CASES FOR TESTING SIMPLIFIED BOILER SYSTEM No. a b c d e Result 1 T F F F T T 2 T F T F F T 3 T F F T F T 4 F T F F T T Table I. THE CAUSE-EFFECT TABLE FOR TESTING SIMPLIFIED BOILER SYSTEM No Variable Causes 0 a the water Level in a boiler is below the 20,000lb 1 b the water Level in a boiler is above the120,000lb 2 c a water pump has failed 3 d a pump monitor has failed 4 e Steam meter has failed EFFECT: The boiler should be shutdown. 5 F T T F F T 6 F T F T F T 7 F F T F T T 8 F F F T T T 9 F F F F T F 10 F F T F F F 11 F F F T F F Table II represents the set of test cases that has been produced by ATCG in order to test a simplified boiler control and monitoring system. V. VALIDATION OF TOOL This section discusses the experimental results of ATCG execution in order to evaluate the correctness of produced outputs. The ATCG is tested against several examples, Now, we prove the correctness of list of test cases 166

5 produced by ATCG its associated with CEG in Fig. 4, which denote to the Boolean expression E= a+b+((c+d)*e). Proof of the correctness of related cause-effect Table, Boolean expression and cause-effect Table is simple based on observation. The important issue is proof of the correctness of generated test cases. In order to verify the correctness of test cases we create set of test cases manually by using BOR rules. There are five causes (a, b, c, d) in a CEG and S1= {(t), (f)}, S2= {(t), (f)}, S3= {(t), (f)}, S4= {(t), (f)} and S5= {(t), (f)} are minimum BOR test set for a, b, c, d, e respectively. We have to apply BOR rules as mentioned in BOR algorithm section. By applying rule b for node N2 we have: S'_f(N2)=S3_f S4 _f= {(F,F)} S'_t(N2) = (S3_t {f4}) U ({f3} S4_t) = {(T,F), (F,T)} By applying rule a for node N3 we have: S_t(N3)= S'_t (N2) S5_t = {(T,F,T),(F,T,T)} S_f(N3) = (S'_f {t5}) U ({t_n2} S5_f)= {(F,F,T),(T,F,F),(F,T,F)} By applying rule b for (a+b) we have: S'_f (a+b)=sl_f S2 _f= {(F,F)} S'_t (a+b) = (S1_t {f2}) U ({fl} S2_t) = {(T,F), (F,T)} By applying rule b for N1= (a+b)+n3 we have: S'_f(N1)=S'_f(a+b) S_f(N3)={(F,F,F,F,T), (F,F,T,F,F),( F,F,F,T,F } S'_t(N1) = (S'_t(a+b) {f_n3}) U ({f(a+b)} S_t(N3))={(T,F,F,F,T),(T,F,T,F,F),(T,F,F,T,F), (F,T,F,F,T),(F,T,T,F,F),(F,T,F,T,F),(F,F,T,F,T),(F,F,F,T,T)} silver bullet to address all the requirements. As a suggestion it is better to apply a combination of techniques for generating tests that ensure good coverage and good quality. REFERENCES [1] N. Kobayashi, Design and Evaluation of Automatic Test Generation Strategies for Functional Testing of Software, PH.D: OSAKA, 2002, p. 125 [2] G. J. Myers, C. Sandler, T. Badgett, and T. M. Thomas, The Art of Software Testing, 2nd ed. New York: John Wiley, [3] K. Nursimulu and R. L. Probert, Cause-effect graphing analysis and validation of requirements," In Proceedings of the conference of the Centre for Advanced Studies on Collaborative research, Toronto, Ontario, Canada, 1995, pp [4] S. Yokoi and M. Ohba, "TCG:CEG-based tool and its experiments," In Proc of 13th Software Reliability Symposium, Nara, Japan, Nov 1992, pp [5] K. C. Tai, M. A. Vouk, A. Paradkar, and P. Lu, Evaluation of a Predicate-Based Software Testing Strategy, IBM Systems Journal, vol. 33, pp , [6] A. Paradkar and K. C. Tai, Test Generation for Boolean Expressions, In Proceedings of International Symposium on Software Reliability Engineering'95, Oct,1995,Toulouse, France, pp [7] A. Paradkar, K. C. Tai, and M. A. Vouk, Specification-Based Testing Using Cause-Effect Graph, IBM System Journal, vol. 4, pp , [8] J. Desel, A. Oberweis, and T. Zimmer, A test case generator for the validation of high-level Petri nets, Proceedings of 6th International Conference on Emerging Technologies and Factory Automation (ETFA '97), 9-12 September, 1997, UCLA, Los Angeles,USA, pp [9] E. Kit, Software testing in real world: addison-wesley, The above outcomes prove the correctness of tests produced by ATCG tool for a simplified boiler control and monitoring system. Since, the number of test cases has generated by tool is the same as number of tests produced in above for node N1. Furthermore, the set of produced test cases by ATCG is equivalent with the set of test cases that has been produced manually by applying BOR rules. Testing of critical software system is very important, since the selected testing tool must produce all rational conditions to prove that a failure never occurs in the controlling of critical system. VI. CONCLUSIONS The main objective of this work is development of an easy-to-use automated test case generator tool (ATCG) especially for safety-critical software systems. It receives the specification of system in natural language and converts it to an understanding cause effect graph model. The tester can convert a model to a list of test cases automatically by using this application. It is seen that set of test cases is improved by eliminating redundant test cases. It is clear that this tool has some limitations such as generating test cases just for one effect in each execution. Absolutely, there is no one 167