A Modeling Framework for Control Fault Tolerant Reactive Systems

Transcription

1 A Modeling Framework for Control Fault Tolerant Reactive Systems Doug Densmore and Shannon Zelinski Department of Electrical Engineering and Computer Sciences University of California, Berkeley December 6, 2001 ABSTRACT This paper describes a class of embedded systems defined as Control Fault Tolerant Reactive Systems (CFTRS). We describe a methodology on which to base a CFTRS. We then provide a modeling system framework based on a refinement of abstraction levels in order to design such systems. Within our framework, fault tolerance is based on three principles: fault detection, isolation, and accommodation. It is these principles which are defined in the context of our modeling system which allow us to reflect the concerns and design considerations of such systems. We then present an example of how the CFTRS model can be adapted to an application. In conclusion, the strengths and weaknesses of our system are discussed. Definitions Control Fault Tolerant Reactive System (CFTRS) - a system that provides fault tolerance of control for a client system. Client System - a system whose external control may be suspected as faulty. 1. INTRODUCTION In embedded systems, a major component is the control logic that determines to a large part the behavior of the system. Unlike other domains, embedded applications have critical constraints such as real-time operation and power considerations. Providing safety and reliability of the system, especially the control, is of utmost importance. This is not always easy, especially if the system receives part of its control externally. This act of providing safety and reliability is called fault tolerance. Other papers [5][3] examine fault tolerance of specific systems, not generic frameworks such as CFTRS. Initially there was a discussion about how abstract to make our design. One approach was to simply compose generic building blocks that could be assembled in order to make a much larger class of problems. The designer would then assemble these building blocks to create a specific system of their choosing. However, this approach was too abstract, lacking in enough formalism and structure to be composed within the scope of this paper. This paper focuses on preserving the safety and reliability of the external control of embedded systems. Control fault tolerance focuses on ensuring that the control logic is able to function in the face of faulty control input. Specifically determine whether the input is in need of correction, and in that event, provide the proper control. This paper defines a domain of systems that provide control fault tolerance and composes a framework to model these systems. This paper is composed as follows. Section 2 considers the domain of the problem and refines the scope of the model. Section 3 discusses our design methodology. Section 4 is a refining of the framework into a more specific model that becomes our design. Section 5 provides a case example of applying the framework. And finally Section 6 concludes with a discussion of the strengths, weaknesses, and potential growth of our framework. 2. DOMAIN OF THE SYSTEM Before attacking the problem, we must define the domain of our framework. The restrictions and assumptions placed on our domain are critical to the direction of our paper. 2.1 Restrictions Fault Tolerance limited to control concerns Fault tolerance in our model is not concerned with data fault tolerance methods such as ECC or parity correction. In addition it is not concerned with component failure or fault tolerance dealing with issues such as circuit or manufacturing failure. We focus exclusively on fault tolerant control issues. Therefore, our system is concerned only with providing fault tolerance for control data delivered to the client system Distinction between requested control and external context data We must provide explicit definitions of both the external control data delivered by the user (requested) and the context data used to indicate if the requested data is faulty. This distinction would be made under the assumptions that it follows the necessary abstractions made in our framework Distinction between inputs and outputs It is important that there be a distinction between what enters the system we are modeling and what goes out. Even in the event that no distinguishable modification has been made, it must still be possible to determine whether the fault accommodation process has occurred or not. This way the client system can be alerted to any tampering of it s control data inputs, even if the replacement inputs are indistinguishable from the originals. 2.2 Assumptions The input is assumed valid The inputs to the system will be valid when received by our system. CFTRS does not provide for the detection of corrupt or invalid input data. This goes for both context data and requested control data described in the previous section. 1

2 2.2.2 The outputs of the system will be fault free. The client will assume that the inputs provided by the CFTRS will be free of faults. Similar to inputs, our outputs are assumed to be valid and free from corruption. Because it is the sole purpose of the CFTRS to replace faulty control instructions to the client system, invalid CFTRS output would imply that either the CFTRS inputs or the system itself are faulty Real Time Constraints Control instruction interpretation is assumed to be digital and therefore periodic at the stage in which the CFTRS provides fault tolerance for the client system. Control fault detection occurs at intervals such that the system can process information prior to the next iteration of the control process. It is assumed that the system will iterate through its process in such a way that inputs are not blocked or dropped and in a timely manner such that needed controls can take effect. The CFTRS must operate very quickly and efficiently as to incur only negligible delay on the client control process. 3. METHODOLOGY Before we design our model, we must address certain concerns in order to establish the correct level of abstraction for this model. The main concern is establishing a framework that is robust enough to cover the important aspects of the design, while at the same time not over constraining the design or tying it to a specific implementation. The key concept for our model is that control data provided to a system can be faulty when based upon external information provided to the system. It is then up to our model to be able to capture both the external control data, the data upon which to put that control into a context, and then provide output that depends upon either the presence or absence of faulty external control data. This framework modeling methodology can be broken into general system concerns, fault tolerance, and calculation. 3.1 System Concerns The first area of the framework to be discussed is the overall guiding concerns of the system. These concepts are implicit in the rest of the design of the system and can be thought of as guiding principles. These concerns are largely similar to those introduced in [1] and [2] Functionality vs. implementation The system must maintain a separation of functionality and implementation. If a specific implementation were to be attached with the model, not only would that implicitly add physical constraints such as bandwidth and protocol, but it would also be restrictive in the designer s view of the system. Both of these aspects would have been restrictive and unnecessary. This also helps to maintain reusability and preserves a portable framework that scales to various systems and environments. The level of abstraction for our model should provide a concrete framework without the need for implementation requirements or even model of computation restrictions Flexible Data Interpretation For the purposes of our framework abstraction level, we desired to make no distinction of external data s use as control or data manipulation (or what we refer to as calculation) information. The interpretation of this data is dependent on the activity in which it is used. This means that the same data can be interpreted as control information by one part of the system and as calculation information by another. To limit data interpretation is not only restrictive but also highly subjective in many systems where no such clear distinction can be made for various scenarios. This is not to say that data of this type in anyway ties control and calculation procedures together. In fact, this behavior is discouraged and not part of our framework Calculation vs. communication By calculation we mean the calculation of any activity whether it is control or data manipulation related. Our use of the word calculation here should not be confused with our general reference of data manipulation activities as calculation activities. Portions of the system where communication is performed and those associated with calculation will remain separated so that communication methods can be designed in the absence of calculation needs and vice versa. This orthogonalization of concerns [1] creates a separation that simplifies the design while at the same time maintaining flexibility. This helps to allow different approaches to communication than traditional methods. For example, with this separation if network communication needs to be performed, protocol issues can be focused on with the assurance that it will not be hindered by calculation, since this stage is ensured to be handled separately by CFTRS. The syntax used in the diagram representations of our system presented in section 4 supports this separation of communication and calculation. Rectangular boxes represent activities in which calculation occurs and arrows represent the communication of outputs from one activity to the inputs of another. 3.2 Fault Tolerance As stated earlier in section 2.1.1, the area of fault tolerance could apply at many levels within our system. However, for the purposes of CFTRS, fault tolerance focuses on the correction of client control input due to contradictory context information. Fault tolerance in general can be separated into three areas that we will discuss and refine to fit our specific application of fault tolerance. These areas are detection, isolation, and accommodation [7] Detection The first step in any fault-tolerant system is fault detection. This mechanism at the most basic level is to simply detect that there is the potential for the need for fault tolerant activities. In other words, detection interprets the context data. This must be sensitive enough to cover the corner cases and worst-case scenarios. For our purposes detection is concerned only with the externally provided control inputs or instructions to the client system. Typically, detection depends on the analysis of sensory data whether the data comes directly from raw sensor measurements or from calculations on existing sensor measurements that model non-existing sensor measurements. No good embedded system can depend on specific sensors. This 2

3 would make the system implementation dependent and not reusable if sensors were to change. Therefore, we suggest that CFTRS detection be dependant on model based sensory data. Since model based calculations can use almost a limitless combination of real sensors, the system will not be dependent on any particular sensor and the separation between functionality and implementation will be maintained. An added attraction of using model based sensory data is that mathematical models can calculate non-measurable characteristics using state estimation, parameter estimation, adaptive filtering, variable threshold logic, statistical decision theory, and pattern recognition [4]. The specific calculation of the model based sensory data is still dependent on implementation and would therefore occur outside of the system. CFTRS need only worry about the information contained within sensory data and not where it came from. Some examples of rudimentary ways to interpret sensory data are redundancy and limit checking. Redundancy is only used to detect faults in the sensory data itself by comparing different calculations of the same data and looking for differences. This does not apply to our system because we are only concerned with finding faults in the control data provided to the client and as stated earlier, all inputs are assumed to be valid. Any validity checks on sensory data should be made outside of the system. Limit checking is based on preset thresholds for sensory data. For example, if the calculated distance to a collision exceeds the preset minimum distance threshold, a fault is detected. Limit checking is the simplest form of fault detection and is very good for enforcing safety tolerances, which is ideal for the CFTRS. However, one must be wary of false positives when limit checking. Measurement noise may filter through sensor calculations that may further obscure the noise causing errant data points to jump over the threshold and cause false positives. One way to protect against false positives is to make sure that a data point remains over the threshold for a number of sampling points before a fault is detected Isolation Isolation provides an expression of determinism in the detection of faults. The process of further diagnosing a detected fault provides a means of not only more intelligently deciding on which calculations to perform but also performing them in an efficient manner that will aid in system performance. This step will add robustness to the minimal fault information provided by the detection mechanism. Isolation provides additional information upon further analysis of external data. This information can include speculated cause of the fault and the severity of the fault. Methods for isolating faults can be similar to those used to detect faults but the sensory information used may be tailored to more specific situations and more than one predetermined threshold may be used to determine severity Accommodation Accommodation is the decision making process that deals with the detection and subsequent isolation of the faulty control. This stage will interpret the fault information provided by isolation in order to decide how to best deal with the fault. This is the aspect of fault tolerance in which the control manipulation can most directly be linked in relation to fault tolerance. Accommodation can determine a fault to be tolerable, conditionally tolerable, or intolerable. The resulting decision is processed and used to manipulate the control instructions under suspicion of corruption. 3.3 Calculation Calculation is where control information is modified depending on the results in fault tolerant area of CFTRS Calculation will progress in stages Calculation will progress in a manner in which various stages allow for manipulation of data due to the influence of the various components of the model. To put it simply, both the initial control and the fault tolerant control have the opportunity to influence behavior at appropriate stages so that the system does not allow the operation of one without the other Calculations will map to deterministic comprehensive behavior This will ensure that for any input to CFTRS, calculation will occur that is not only deterministic but also comprehensive of the various situations that could occur, even if that results in some default action. What this means is that even impossible scenarios have a corresponding calculation result. For example, if input is seen that is not understood by the system, CFTRS would have some behavior that is for these unspecified events. This is important especially since one of our assumptions is that input is valid and we perform no check on this input. 4. REFINING THE FRAMEWORK In the following section we will use our methodology to define the framework model of a CFTRS. We will start at the highest level of abstraction and refine the model until a balance between specification and reusability is achieved. 4.1 Black Box Input-Output System At the highest level of abstraction, we think of the CFTRS in terms of its inputs and outputs only in order to answer the most important question. What does the system do? The CFTRS outputs consist of Internal Instruction and Fault Tolerance signals and the inputs consist of External Instruction and External Sensor signals as seen in figure 1. Figure 1. Black box abstraction of CFTRS Notice that our model puts no restriction the specific source of the inputs or the usage of the outputs by the client system. This is a stand-alone component that can be used depending on the application. Such an application could include a network of CFTRS modules System Inputs System inputs can be broken into External Instruction and External Sensor signals as mentioned above. Each of these areas 3

4 of inputs can contain any number of individual signals depending on the size of the specific application. The External Instruction inputs are the signals externally provided as control inputs for the client system. These signals are intercepted by the CFTRS for analysis of possible faults. These are signals that should be considered default and will require the presence of a conditionally tolerable or intolerable fault to be overruled. The External Sensor inputs provide the CFTRS with information about the client system s environment. This information is then used to detect a possible danger to the client system, check the validity of the External Instruction signals, and ultimately replace the External Instruction signals with Internal Instructions System Outputs The two kinds of system outputs are Internal Instruction and Fault Tolerance signals. Once again the number of signals contained in each group is dependent on the specific application. However, External Instruction and Internal Instruction should be identical in size as one replaces the other to the client system. The Internal Instruction outputs are computed by the CFTRS to replace the External Instruction signals as control inputs for the client system. These may or may not result in an overall change in behavior, meaning they may be identical to External Instruction inputs, but replace them nonetheless. The Fault Tolerance outputs are intended to inform the client system of the severity of any fault and the accommodation decision. This information may be used by the client system to alert the source of the faulty External Instruction inputs to what is going on. In some cases, this information may give the External Instruction source a chance to correct the fault without the CFTRS taking over. After all, the CFTRS is meant only as an override device and we do not want to usurp external control if the problem can be handled externally as originally intended. 4.2 Orthogonalization of System Activities Each activity relies on both External Instruction and Sensor input but the instruction calculation and fault tolerance activities are responsible for only their Internal Instruction and Fault Tolerance outputs respectively. Fault tolerance activities are related to fault detection, isolation, and accommodation as described in the methodology. These activities are responsible for making the major decisions such as determining if there is a fault, how serious it is, what should be done about it. Instruction calculation is the process of calculating the correct Internal Instruction output based on the various decisions made by the fault tolerance activities. This abstraction is shown in figure 2. Figure 2. Fault Tolerance and Instruction Calculation 4.3 Fault Tolerance Decisions and Instruction Calculation Interaction This level of abstraction describes the interaction between Fault Tolerance and Instruction Calculation. We believe this is the lowest level that can be used to describe any CFTRS in our implementation free framework. Further refinements of this model should be made for specific applications. We also believe this is the highest level of abstraction that can sufficiently define the inner workings of a CFTRS without becoming a generic control problem. The fault tolerance activities described above are refined into detection, isolation, and accommodation activities as discussed in the methodology. The instruction calculation activities are refined into suggested instruction and modified instruction activities. The very last steps of fault tolerance and instruction calculation are the only ones that depend on inputs from its orthogonal set of activities. These are accommodation and modified instruction respectively. The inputs and outputs of each of these refined activities are discussed below. Figure 3 shows this final level of abstraction. Figure 3. Final Framework Model Detection Fault detection is dependant on only external sensor input. When sensor input indicates that the client system is in sufficient danger (this is defined depending on the specific application), the external instruction is suspected of being faulty. The output of detection then informs the isolation unit that a fault has been detected. Detection will make no attempt to analyze the fault in any way. The separation of detection from later activities such as isolation helps add modularity to the system representation thus making the system design simpler Isolation Once a fault is detected it must be isolated. A large and complicated CFTRS may be monitoring more than one independent set of external instruction signals. When a fault is detected, any one of those sets of signals could be faulty. The CFTRS must isolate the faulty signals from the rest so no energy is wasted trying to fix what is not broken. A simpler CFTRS may have only one independent set of external control signals to monitor. In this case isolating the fault itself is not necessary for once a fault has been detected the faulty control signals are obvious because there are no others. Isolation is also responsible for determining the possible cause of fault and the severity of a fault as mentioned in our methodology. The External Sensors are required to isolate the location of a fault and determine its severity. Isolation outputs the location and severity of the fault 4

5 to accommodation. It may output more information depending on the application. These outputs may also be included in the Fault Tolerance outputs to the client system. Essentially this is providing meat and meaning to the fault originally flagged in the detection portion Suggested Instruction The outputs for this activity are the suggested Internal Instruction signals calculated assuming that the External Instruction is faulty and must be replaced. The signals are calculated from External Sensor input only because the External Instruction is assumed to be faulty. The Suggested Instruction calculates the quickest and most effective way to avoid any fault presented by the external sensor input. It is assumed that calculation takes up a considerable portion of the CFTRS processor run time. As such it is probably the biggest drain on system resources and power. However, we suggest that the suggested calculation activity run constantly independent of whether a fault was detected or not. If more than one processor is used in implementation, suggested calculation can be run in parallel with the fault detection and isolation processes allowing for more compact scheduling. By the time a fault has been detected and isolated the suggested instruction is ready for comparison in the accommodation activity. Also, running all activities every cycle makes timing more uniform so that the delay that the CFTRS process incurs is constant Accommodation The job of the fault accommodation activity is to output the accommodation decision. Accommodation must decide if the fault is tolerable, conditionally tolerable, or intolerable. A tolerable fault means that the CFTRS believes that the external instruction is sufficient to fix the problem. A conditionally tolerable fault means the external instruction must be modified in order to fix the problem. There may be different degrees of conditionally tolerable faults i.e. different conditions, which are inheritably defined by the application. An intolerable fault means that the external instruction must be completely abandoned and replaced by internal control in order to fix the problem. Accommodation bases its decision on a comparison between the External Instruction, and Suggested Instruction. Therefore, information from isolation calculation is used in this fault tolerance activity. Isolation inputs are used to help accommodation focus on the correct sets of instruction signals. Fault severity information from isolation may also play a big part in the decision. The accommodation decision may be included in the fault tolerance outputs to the client system Modified Instruction Modified instruction is the last phase in the CFTRS process. This activity outputs the final internal instruction set whose tailored calculation is based on the accommodation decision. The final internal instruction could be identical to the external instruction set if the fault is tolerable. Or it could be identical to the suggested instruction if the fault is intolerable. If the fault is found to be conditionally tolerable, modified instruction must use the suggested instruction and external instruction inputs to calculate something in between. In a way the modified instruction acts as a mux, channeling the correct instruction set to the client system. And the accommodation decision is the mux control. Even if the fault was tolerable and the external instruction is identical to the internal instruction, fault tolerance outputs will allow the identification of an instruction that has gone through the CFTRS process. 5. CASE EXAMPLE This section describes the case example of a front-end collision avoidance system using our framework model of the CFTRS. The client system in this case simply interprets acceleration instructions from the users accelerometer and brake (or by any other means) to the vehicle in motion. The CFTRS will intercept these instructions, compare them with environmental data, assess whether they will put the vehicle in danger, and supply alternate instructions if necessary. 5.1 Black Box Description We start with a description of the system at the highest level of abstraction by describing the system inputs and outputs, as should be done with all embedded system designs System Inputs The system inputs are separated into external instruction and external sensors as in the framework. The external instruction is the acceleration instruction from the user. The external sensors will provide information on current distance and velocity System Outputs The system outputs are separated into internal instruction and fault tolerance as in the framework. The internal instruction output is the acceleration instruction delivered to the client system. Again, this instruction may be identical to the external instruction or modified by the CFTRS. The fault tolerance outputs include the accommodation decision and fault severity as determined by fault isolation. For this case the accommodation decision will be only tolerable or intolerable. There is not conditionally tolerable case here because the only course of action when system is faced with the fault of an imminent head on collision is to decelerate. 5.2 Adapting the Final Framework Model Our CFTRS framework is used to model the front collision avoidance system by adapting each generic CFTRS activity to the application. Figure 4 shows how this case example can be represented in our final CFTRS framework model. Each activity is explained below Detection Fault detection in this case is a simple threshold calculation based on the external inputs of velocity and distance. The threshold will be set based upon the desired buffer time to a collision. If at maximum speed, the collision will be reached before the threshold time, a fault is detected. This very simple threshold calculation is given by KVmax > d where K is the time threshold, Vmax is the max velocity of which the vehicle is capable, and d is the distance to a front-end collision Isolation Isolation is responsible for determining the severity of the fault. For this case we have chosen four severity levels. Level 1 is the most severe fault where the actual current velocity will reach the collision before the threshold time. The threshold equation for level 1 is given by KV > d where V is the actual 5

6 current velocity. Level 2 is where the current velocity will reach the collision before twice the threshold time. The threshold equation for level 2 is given by 2KV > d. Level 3 is where a fault has been detected but neither of the previous thresholds have been crossed. Level 4 is the lowest level where there is no fault Suggested Instruction Suggested instruction calculates the suggested acceleration to stay K time away from a forward collision. This suggested acceleration a is given by a = (d K * V) / K 2. Therefore, the preset buffer time K should be chosen such that the maximum deceleration possible will be able to stop the vehicle before crashing Accommodation As mentioned previously, the accommodation decision will be either tolerable or intolerable. A fault is intolerable if the severity level is 1 and the requested acceleration a is greater than the suggested acceleration a. All other situations are tolerable Modified Instruction Because there is no conditionally tolerable situation, modified suggestion act as a simple two case mux where the suggested and requested acceleration are inputs and the accommodation decision is the control. If the fault is tolerable, the external instruction becomes the internal instruction output. If the fault is intolerable, the suggested instruction is used. Figure 4. Front Collision Avoidance System 6. CONCLUSIONS As we have mentioned, the strengths of our approach stem largely from its modular nature. This leads to a high level of reusability, contributes to a compact design, and therefore it is very portable. As mentioned in [6], this modularity should lead to verification gains. The level of abstraction in our model aids in that verification and portability. In CFTRS, all inputs and outputs are relevant and very few restrictions are placed on them. Our design also incorporates the fractal nature of design [1]. There are no implementation restrictions, yet we narrowly define the scope of fault tolerance so that complexity is minimized. Since there is no implementation mapping, hardware/software partitioning is free to be up to the designer. We make several assumptions (section 2.2) that could be thought of as weaknesses in our framework. Also, the same abstraction level that was viewed as a strength could also be a weakness if it turns out to be lacking in enough formalism to express many designs. This model could be expanded to include a discussion of formal models of computation that could be applied to CFTRS. More classes of applications could be shown in our model in order to better explore issues within our framework. It would also be nice to use existing tools to implement our framework and explore case studies and testing issues with our model. 7. REFERENCES [1] Ferrari, A.; Sangiovanni-Vincentelli, A.; System Design: Traditional Concepts and New Paradigms, International Conference on Computer Design, 1999 (ICCD '99), pp [2] Keutzer, K.; et. al.; System-Level Design: Orthogonalization of Concerns and Platform-Based Design., IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol.19, (no.12), IEEE, Dec [3] McLean, D.; Aslam-Mir, S.; Benkhedda, H.; Fault Detection and Control Reconfiguration in Flight Control,, IEEE Colloquium on Fault Diagnosis and Control System Reconfiguration, 1993 pp. 9/1 911 [4] Murphy, S.; Fault Tolerance as Applied to AUVs, Lehigh University, ECE111, Fall [5] Noura, H.: et. al.; Fault-Tolerant Control in Dynamic Systems: Application to a Winding Machine, IEEE Control Systems Magazine, Vol. 20 Issue: 1, Feb. 2000, pp [6] Wong-Toi, H.; Modular Verification of a Fault- Tolerant Active Structure Controller: an example, Proceedings of the 1999 IEEE International Symposium on Computer Aided Control System Design, pp [7] Yang, K.C.H.; Yuh J.; Choi S.K.; Fault-Tolerant System Design of an Autonomous Underwater Vehicle ODIN: an experimental study, International Journal of Systems Science, vol 30, no.9, 1999, pp