Evaluation of Embedded Operating System by a Software Method *

Transcription

1 Jan. 2006, Volume 3, No.1 (Serial No.14) Journal of Communication and Computer, ISSN , USA * Junjie Peng 1, Jun Ma 2, Bingrong Hong 3 (1,3 School of Computer Science & Engineering, Harbin Institute of Technology, Harbin , China; 2 Department of Computer, Jiaozuo University, Jiaozuo , China) Abstract: In this paper, software faults are emulated by Software Implemented Fault Injection (SWIFI) to evaluate embedded operating system. The method is based on the fact that the source codes of the operating system are mapped into the code segment in the target memory and it is easy to inject kinds of faults in the memory by SWIFI. Faults are emulated by injecting faults at assembly level instead of source code level. Using the software faults emulated by this method, the dependability of an embedded operating system is studied. The experimental results show that software faults have many effects on the software behavior and dependability, and to increase the dependability of software, exclusion of software faults is very important. Key words: Fault Injection; Dependability; Embedded Operating System; Coverage 1. Introduction In modern society, software has been widely used in many fields and software dependability requirements have become increasingly important. In many fields, especially in onboard aircrafts, nuclear power plant control and radiation therapy machines, etc., high safety and reliability are more critical. To ensure the system safety and reliability, software test is needed, so that faults are excluded from the system and environment as possibly [1]. Software test is the most widely used technique to eliminate the faults and make sure the safety and dependability of software, in which specifications test, * Acknowledgments: This work is supported by the Tenth-five Years Research Program of National Defense (No ). Junjie Peng, PhD candidate; main research fields: embedded real time operating system, dependability, fault injection, etc. Jun Ma, lecturer; main research fields: embedded operating system. Bingrong Hong, professor; main research fields: multi-agent, robots, embedded real time system, fault tolerance and so on. reviews and inspection, formal entry and exit criteria test, functional test, etc. are among the most used methods. All these methods can be classified into two kinds: manual test and automated test. Manual test is that testers can find and correct the faults or errors in the software source codes in time when it is under test. However, many efforts are needed for manual test, especially when software is very big [1]. So automated test is a promising research direction in software testing. The goal of automated test is to minimize the amount of manual work involved in test and gain larger coverage with a large number of test cases. It can decrease the efforts of testers greatly. In this paper, a method using Software Implemented Fault Injection (SWIFI) technique is presented to emulate software faults and evaluate the dependability of software system. Using the method, software faults are emulated at assembly level instead of source code level when the operating system runs. The advantage of the method is that the dependability of the operating system can be evaluated with manual effort being decreased greatly. The structure of this paper is organized as follows: After the introduction in section 1, related research is presented in section 2. In section 3, software faults emulated by SWIFI in embedded system under test are introduced. In section 4, the experimental results are given and analyzed. In section 5, the conclusions are given. 2. Related Work Evaluation of software is a complex thing. It is 79

2 often related with software testing, software reliability modeling, software reliability risk analysis and so on. No matter which technique is considered, specific software faults will be discussed. However, at the different phases, software faults have different forms and characteristic s. So to estimate the reliability of software and predict software faults for risk assessment, the fault history during the different phases should be cons idered. In this paper, software faults are not considered at all kinds of phases, but emulated by SWIFI just in the real execution environment. Through the emulated software faults, the effects of actual software faults on the dependability of the embedded operating system running in the real environment are studied. Fault injection is an effective technique to study the behavior of the target system by faults to determine whether the system behaves properly or not. Several fault injection techniques have been proposed and many fault injection studies have been published. According to the difference of implementation, fault injection methods can be classified into three kinds: Hardware Implemented Fault Injection (HWIFI), Simulation-based Fault Injection (SBFI) and Software Implemented Fault Injection (SWIFI) [2,3]. HWIFI is a method injecting faults directly into the target system hardware by physical means. Though these methods are used, faults injected may be close to the faults in realistic environment and no models are needed to develop, they require specially designed hardware devices for fault injection which are generally dedicated to a specific target system and may damage the target hardware if there are no enough care during the experiment. SBFI is implemented by injecting faults into a simulation model of the target system, which are usually designed with Hardware Description Language. The advantages of these methods are that no other special hardware is needed and the injected faults can be accurately controlled. However, enormous development efforts and much time are required when an accurate simulation model is developed. SWIFI emulates the error state of the target system hardware and software through special programs. It is the most promising method for its low development cost, well flexibility and no damage to the target system, etc. compared with other fault injection methods [3,4]. For its advantages, SWIFI is a widely used method to emulate the faults of computer systems, which is used to evaluate the dependability of the systems. However, most of researches focus on emulating hardware faults by SWIFI, few researches focus on emulating software faults by SWIFI. It is because that the knowledge on the software faults experienced in computer systems is limited, which makes it hard to define meaningful fault sets. 3. Software Faults Emulated by SWIFI 3.1 Fault Types A software fault can be characterized by the change in the codes that is necessary to correct it. To trigger a software fault, trigger conditions must be met, which make the fault exposed. And to make the result convincing, the type represents the faults in the source codes. In ODC, six fault types related to code are proposed [5], which are in assignment, checking, algorithm, timing, interface and function. 3.2 Parameters During the process of fault injection experiment, besides specific fault model, some other parameters are also needed. (1) Location. Location is the place where the fault is injected. The location of fault injection must be in the code segment area in the system, where the source codes of the operating system are mapped. (2) Fault mask. Fault mask is a data that determines which bit or bits of the target data is/are affected. For instance, if the fault mask is 0x2B (high bit low bit) and fault model is bit-flip fault, only bits 2, 4, 6 of the value of the original location are flipped while other bits have no effect. 80

3 (3) Fault duration. Fault duration refers to how long a fault lasts, different faults have different fault duration. For bit-flip fault, only one flip is enough, while for stuck-at faults, fault duration must be considered. When a stuck-at fault is being injecting, the continuous monitoring of the location where fault is injected is needed during the whole process of fault duration. Once the content of the location is used in the fault duration, corresponding fault injection operation works. (4) Trigger condition. Trigger condition is a condition to trigger fault injection process. It includes two kinds of triggering schemes, temporal trigger and spatial trigger. Temporal trigger refers to all kinds of time related events or conditions. Any of these conditions which is met, will trigger fault injection process. For example, injection can be triggered when: (a) a random time since the application begins to run in experiments, (b) a given event occurs. Spatial trigger refers to all kinds of workload or location related states or conditions. For instance, some specific memory units are read /written, the data in tran_ah:test ah,01h jz tr_ah_1 mov ah,80h tr_ah_1:test ah,02h jz tr_ah_2 mov ah,40h tr_ah_2:test ah,04h jz tr_ah_3 mov ah,20h tr_ah_3:test ah,08h jz tr_ah_4 mov ah,10h Fig. 1 The Segment of Source Code some specific address units are operated, or some special instructions are executed and so on, all may trigger fault injection process. Usually, during the actual fault injection experiment, temporal trigger and spatial trigger are combined. A purely spatial or temporal trigger can be viewed as a special case of the combined trigger. 3.3 Emulation of Software Faults To emulate software faults of the embedded operating system at assembly level, bit-flip fault model and stuck-at fault model are used. The target operating system is programmed with assembly language. Fig. 1 presents a segment of the source codes of the embedded operating system. To emulate software faults of the embedded operating system, a commercial system simulator named TRACE32 from Lauterbach GmbH is used, which is used to simulate the architecture of the target CPU. Through the simulator, the tester can determine the location in memory allocated to the target source codes, the addresses of the variables, program branches and so on C F6 C4 01 tran_ah:test ah,01h F jz tr_ah_ B4 80 mov ah,80h EB F6 C4 02 tr_ah_1:test ah,02h jz tr_ah_ B B4 40 mov ah,40h D EB 3A F6 C4 04 tr_ah_2:test ah,04h jz tr_ah_ B4 20 mov ah,20h EB A F6 C4 08 tr_ah_3:test ah,08h D jz tr_ah_ F B4 10 mov ah,10h EB Fig. 2 The Compiled Code Fig. 2 presents the segment codes in compiled file fios.lst, which is the corresponding part in Fig. 1. According to the file fios.lst, the location where faults are injected into can be determined from the simulator. For example, the numbers such as 1893, 1894, 1895, etc. in the fist column in Fig. 2 are the numbers of line. The hexadecimal numbers such as 080C, 080F, 0811, etc. in second column are the offset addresses of the corresponding codes, according to which, the location of the codes in the target system can be attained. Take the second line in Fig. 2 as an example, it shows that the offset address of the line is 080F. This means that if the starting address of code segment in the target system is 5000:0, the codes jz 81

4 tr_ah_1 in line 1894 in fios.lst is stored in the memory address of 5000:080F in the target system. At this moment, if software faults occurre in these codes are needed to emulate, only is fault needed to inject into 5000:080F. From the simulator, it can be seen that the codes in the address of 5000:080F have become jz [0864]. This means that if a fault is injected into 5000:080F, the program will not jump to 5000:0864 as the correct program does, while it may jump to 5000:12B0 as shown in Fig. 3. By this operation, a program branch fault is emulated. Fig. 3 Example for Injecting a Fault in Program In the process of emulation of software faults, several steps are needed to follow. Step 1: Compile the source codes of the target system (fios.asm) with Microsoft macro assembler and get the complied files (fios.lst and fios.bin). Step 2: Load the compiled file fios.bin from the simulator, which connects the target system with serial lines. Step 3: According to the compiled file fios.lst, find the location and addresses where faults will be injected from the simulator. Step 4: Write the location and addresses with other parameters into a configuration file, which is used for fault injection. Step 5: Execute fault injection program and carry out experiment using the script file as fault injection parameters. The code after injection tran_ah: test ah,01h jz [12B0] mov ah,80h jmp [08E6] Should be jz [0864] 4. Experimental Results and Analysis 4.1 Experiment Description The target operating system under test is an embedded operating system that will be used for some space projects in China. It is programmed with assembly language, about 7600 lines of source codes, 188KiloByte in size for source codes, 11.3 KiloByte in size after compilation. It runs on a special micro-computer from CASTC (Chinese Astronautic Science & Technology Corporation), which is used for software test, and nearly has no fault tolerance mechanisms. The micro-computer has 25MHz frequency clock, 1M bytes of RAM memory, 2 RS-422 serial I/O channels, and includes many build-in Error Detection Mechanisms (EDM). And it is connected with TRACE32 system simulator on one end, while the other end is connected with a host computer, which is served by a PC computer. To inject faults for evaluation of the embedded operating system, a SWIFI tool developed by our research group is used [6]. 4.2 Re sults and Analysis Using the method put forward in this paper, times fault injection experiments are carried out in all. All faults are injected randomly when the target system normally runs. All of these faults are classified into three kinds: branches, variables and others. Branches mean that faults are injected to the places where there are jump condition. Variables mean faults are injected into the variables initialization conditions. Others include all other faults injected into the code area except branches and variables. The distribution of the three kinds of faults is presented in table 1. Table 1 Activated Faults and Non-activated Faults Fault Percent (%) Total Activated faults No activated faults Table 1 shows that all faults injected are not activated, that is, 6.46% of faults injected are not activated. This is because that these injections do not cause any fault or error, or are injected into the place where the operating system does not access. Table 3 presents the distribution of failure modes for activated faults, which are classified into four categories: Correct: The operating system normally runs and 82

5 gives correct results when these faults are injected into the target system. Incorrect: The operating system can run normally but give error results when these faults are injected into the target system. Hang: The operating system hangs or enters a dead loop, where users have to stop or reset it after timeout. Crash: The operating system terminates abnormally, and there occurs an exception, which is detected by the system. Among the three kinds of emulated software faults, branches faults have the highest fault percentage. That is because when faults are injected into the branches of the operating system, the operating system will enter a completely wrong running state or path, even a place out of code area, which makes the system more easy to hang or crash than other cases. When faults are injected into variables, there are more incorrect results than other cases. This can be explained that the variables are more sensitive to faults than other parts of the code area when the operating system runs. Table 2 Distribution of Different Faults Fault Percent (%) Branches Variables Others Fig. 4 presents the final results of the experiments. It shows that most of the faults injected into the system cause wrong results. In fact, 79.82% of faults have made the system give wrong results, 5.28% and 1.54% of faults have caused the system to hang and crash respectively. Only 13.36% of faults have no effect on the output of the system. It means that excluding software faults in software is very important to improve the software dependability. Table 3 Failure Modes for Activated Faults Failure modes (%) Correct Incorrect Hang Crash Branches Variables Others Conclusions Percent (%) Correct Incorrect Hang Crash Fig. 4 Final Results of Failure Modes In this paper, a new method using SWIFI technique to emulate software faults is presented. The method emulates software faults by injecting faults into the code area of the target computer system, where the source codes of software are mapped. Using the method put forward, an embedded operating system is evaluated. The experimental results show that software faults have many effects on the software behavior and dependability, and faults in the entrance of the program branches and jumps can more easily cause the operating system to hang and crash. To increase the dependability of software, it is very important to get rid of software faults as possible. References: [1] D. Hamlet, Connecting Test Coverage to Software Dependability, Proc. 5th International Symposium on Software Reliability Engineering, 1994: pp [2] M. C. Hsueh, T. K. Tsai, R. K. Iyer, Fault Injection Techniques and Tools, IEEE Transactions on Computers, 1997, 30(4): pp [3] Y. Y. Yu, A Perspective on the State of Research on Fault Injection Techniques, University of Virginia, USA, Research report (UVA -CSCS-FIT-001), [4] J. Carreira, H. Madeira, J. G. Silva, Xception: Software Fault Injection and Monitoring in Processor Functional Units, IEEE Trans on Software Engineering, 1998, 24(2): pp [5] R. Chillarege, Software Testing Best Practices, IBM research, technical report RC 21457, [6] J. J. Peng, B. R. Hong, C. J. Yuan, Software Implemented Dependability Evaluation System for Evaluation of Onboard System, Proc. 10th Joint International Computer Conference (JICC 2004), Kun Ming, China, 2004: pp (Editors: Brick, Susan, Xiao Wu) 83