Formal Modeling for Verifiable, Efficient Fault Detection and Response

Transcription

1 Formal Modeling for Verifiable, Efficient Fault Detection and Response Meredith Beveridge Lecocke Southwest Research Institute Flight Software Workshop Pasadena, CA 16 Dec 2014

2 Two-Fault Tolerant Systems Two-fault-tolerant avionics (a system that can survive two simultaneous faults) are required for: Manned space programs (personnel safety) Robotic spacecraft (close rendezvous could damage Space Station (ISS) or other critical assets) Reliability requirements addressed with fault detection, isolation, and response (FDIR) Identify all possible faults in system Specify detection and response methods for each 2

3 Vulnerability Points in System Multiple sources of faults Processors Sensors Actuators Radiation (SEUs) Communications links Timing problems Must withstand any two simultaneously (two-fault-tolerant) 3

4 Fault Tolerance Addressed with FMA Process Fault Mode Analysis (FMA) process: Complete view of signals and failures + prioritization of failures = whether and how they should be mitigated (results in new system requirements) Manual Process: Identify all signal outputs from all components Identify all credible ways signals can fail Determine effects and criticality of each failure mode Define coverage method for critical failure modes Inputs: FMEA, ICDs, schematics, hazard analysis Took FDIR/FMA expert and team of engineers 1.5 years 4

5 Manual Process Example FMA spreadsheet with 600+ single faults: all failure modes of all signals, with criticality, detection and response FDIR Dataflow Diagram Board 2 Board output signals analyzed 616 single faults 423 Crit 1A/1B U5 U1 U7 U6 U11 U10 U6 U5 U2 U1 U1 U2 U10 U9 Board 3 U5 U3 U4 U2 U8 U3 U3 U4 U7 U4 U9 U7 U6 U8 5

6 A L AB B D R MR M C H P FMA/FDIR Existing Manual Process Difficulties: 187 signals, 600+ single fault modes did we think of them all? 423 Crit 1A/1B detection and response algorithms must be implemented by hand 423 detection and response algorithms must be verified and validated by hand Criticality Totals 1A B N/A A and 1B Totals per Subsystem

7 A Better Approach Using Formal Modeling Specify the system and its reliability requirements at a higher level of abstraction Verify reliability of system architecture and algorithms on resource-rich desktop PC Automatically generate embedded implementation of detection and response guaranteed to match verified model 7

8 FDIR Using Formal Modeling Specify the system and its reliability requirements at a higher level of abstraction Easier (and faster) to get 30-ish items correct than 600 Knowledge representation language Answer Set Prolog (ASP) facilitates concise, accurate descriptions of: System components, interactions between them States, transitions, faults Goals: how the system should operate, in priority order (i.e. fail safe mode is a last resort) 8

9 FDIR Using Formal Modeling Verify system reliability on resource-rich desktop PC Standard ASP-based reasoning algorithms for planning and diagnosis Automatically verify fault detection and response methods satisfy completeness and rationality properties Completeness = all possible fault sets have been addressed Rationality = best possible outcome is identified for each fault set Execute model against all possible 1- and 2-fault scenarios, verify that path to highest priority goal is found Simplifies and accelerates FDIR analysis, provides confidence in fault tolerant design Model generated 576 fault modes and 161 unique responses (some fault modes use same responses) 9

10 Modeling System Components Our system is specified with following components: Flight computer (fc) 4 control boards 2 on side A (ca1, ca2) 2 on side B (cb1, cb2) 2 motor boards (ma, mb) 2 actuators (aa, ab) Wires: flight computer to all control boards betwixt all control boards control boards to motor boards motor boards to actuator 10

11 Modeling System States System state described with boolean properties: 1. presence of a command a. control board or motor board command at a control board b. motor board or actuator command at a motor board c. actuator command at an actuator 2. presence of sensor data at a control board 3. mode of a control board (inactive, generate only, generate send) 4. mode of a motor board (on, off) 5. health status of motor board (good, bad) 6. health status of the sensors on each side (good, bad) 7. fault state of every component (either faulty or not faulty) 11

12 Modeling System Actions Actions change system state according to the following rules: 1. If the flight computer sends a controller command to a control board, then the control board has the command. 2. If a control board sends a controller command to another control board, then the other control board has the command. 3. If a control board queries its sensors and the sensors are healthy, then it has sensor data. 4. If a control board disregards sensor data, then it no longer has sensor data. 5. If a control board sends its sensor data to another control board, then the other control board has the sensor data. 6. If a control board generates a motor board command, then it has a motor board command. 7. If a control board sends a motor board command, then the motor board has the command. 8. If a control board sends the shutdown command to a motor board, then the motor board is off. 9. If a control board changes mode to inactive, generate only or generate send, then the control board is in the new mode. 10. If a control board is in inactive mode, then it may not generate or send a command. 11. If a control board is in generate only mode, then it may generate, but not send a command. 12. If a control board is in generate send mode, then it may generate and send a command. 13. Only one control board on a side can be in generate send mode. 14. A control board must have both a controller command and sensor data in order to generate a motor board command. 15. If a motor board generates an actuator command, then it has an actuator command. 16. If a motor board sends an actuator command, then the actuator has the command. 17. A motor board must be on in order to generate or send an actuator command. 18. A motor board must have a motor board command in order to generate an actuator command. 19. A component may not send information (commands or sensor data) it does not have. 12

13 Modeling System Faults Conditions that prevent an action from having its usual effect: 1. If a wire between components is faulty, then information may not pass over the wire. 2. A faulty control board will not be able to successfully generate a command, send a command, or query its sensors. 3. A faulty motor board will not be able to successfully generate or send an actuator command. 4. A faulty control board, motor board, or actuator will not be able to successfully receive any commands. 13

14 Modeling System Goals Required goals (prescribe system behavior): 1. Unhealthy motor boards must be immediately shutdown. 2. Controller boards must immediately disregard sensor data from unhealthy sensors. Prioritized goals (ordering of acceptable outcomes): 1. Move the arm with two actuators (cooperative mode). 2. Move the arm with one actuator (degraded mode). 3. Secure the system in a safe stationary position (failsafe mode). 14

15 Modeling System Outputs Output: Plans are generated for every possible fault set Example: no faults 1. flight computer sends a controller command to all control boards 2. Control boards ca1 and cb1 change mode to generate send, and ca2 and cb2 change mode to generate only. 3. All control boards query sensors and then generate a motor board command. 4. Control boards ca1 and cb1 send motor board commands. 5. Both motor boards generate and then send an actuator command. Example: faulty motor board ma 1. flight computer sends a controller command to all control boards 2. Control boards cb1 change mode to generate send, and cb2 change mode to generate only. The control boards on the A side perform no action and remain in inactive mode. 3. Control boards cb1 and cb2 query sensors and then generate a motor board command. 4. Control board cb1 sends motor board commands. 5. Motor board mb generates and then sends an actuator command. 15

16 Inpu t: A1 Inpu t: A1.A Inpu t: A1.B Inpu t: A2 Inpu t: A2.A A A activ e B B Acti ve A A Acti ve, A B Stan dby Outp ut: A1 Outp ut: A2 Outp ut: A stat e singl e chan nel Outp ut: A stat e activ e activ e stan dby Acti ve, singl e chan nel Auto-Generating Reconfiguration Table Automatically generate embedded implementation of detection and response Guaranteed to match the verified model Efficient lookup table: small memory/processing footprint No complicated if/else, case/switch, state machine structures Easily updated: update the model, regenerate the table, download to memory Input: A1 Input: A1.A Input: A1.B Input: A2 Input: A2.A Output : A1 Output : A2 Output: A state Output: A state A A active active B B Active active A A Active, single standby A B Standby Active, single 16

17 Executing Reconfiguration Table Simple boolean algorithm: Flight software matches observed state to line in Fault Detection table to determine fault source Flight software matches fault set to Fault Response table to determine required actions Simplifies development, execution, and verification Input: A1 Input: A1.A Input: A1.B Input: A2 Input: A2.A Output : A1 Output : A2 Output: A state Output: A state A A active active B B Active active A A Active, single standby A B Standby Active, single 17

18 Fault Response Example 1. No faults detected; executing plan for fault set zero 2. Detect a sensor mismatch 3. Consult Fault Detection table to identify potential Fault Detection Table fault sources a. faulty(wire(ca2,b1)) b. faulty(ca2) 4. Test two possible fault sources to further diagnose: a. test(wire(ca2,b1)) passes b. test(ca2) fails 5. Update fault set: ca2 faulty 6. Execute plan corresponding to new fault set Fault Response Table Av Ca1 Ca2 A2 A1 plan Norm A2!=A1 A2!=B1 A2!=B2 Wire(a2,b1) ca Fault ResponseTable Av Ca1 Ca2 A2 A1 plan Norm Use ca Use ca

19 Conclusions Both approaches susceptible to human or, but formal modeling lowers the risk by working at higher level of abstraction: Better communication between team members Better design Catch ors earlier in development process Fewer things to keep straight and get right Model-based reasoning can proceed long before hardware ready Increased confidence in specification is passed on to autogenerated implementation Verification of full coverage on resource-rich desktop much easier than ad hoc testing on target platform Easy to update when changes made 19

20 Comparison to Manual Approach Formal modeling improves reliability of analysis Auto-generation from model improves reliability of implementation Formal modeling reduces design, analysis, and testing time Formal modeling and auto-generation is sustainable throughout lifecycle (update model, validate, generate tables, download to target) 20

21 References [1] Balduccini, M., and Gelfond, M. "Model-Based Reasoning for Complex Flight Systems," Proc. (Amer. Inst. of Aeronautics and Astronautics), 2005 [2] Balduccini, M., Gelfond, M., and Nogueira, M. "Answer Set Based Design of Knowledge Systems." Annals of Mathematics and Artificial Intelligence, 2006, pp [3] Boykin, J. and Thibodeau, J. "Evolution of Shuttle Avionics Redundancy Management/Fault Tolerance," Space Shuttle Technical Conference, NASA Johnson Space Center, June 28-30, 1983 [4] Bozzano, M., et al. "The COMPASS Approach: Correctness, Modeling and Performability of Aerospace Systems," Proc. 28th Int. Conf. on Computer Safety, Reliability and Security (SAFECOMP 2009). pages Volume 5775 of LNCS. Springer, 2009 [5] Briere, D. and Traverse, P. Airbus A320/A330/A340 electrical flight controls: A family of fault tolerant systems, International Symposium on Fault-Tolerant Computing, Toulouse, France, June 1993, pp [6] Conquet, E. et al. "Formal Model Driven Engineering for Space Onboard Software," Embedded Real Time Software and Systems (ERTS2012) [7] Fraser, D.C. and Felleman, P.G. "Digital Fly-by-Wire: Computer Lead the Way," Astronaut. Aeronaut., 12, (July-August 1974) [8] Gelfond, M., and Lifshitz, V., Classical Negation in Logic Programs and Disjunctive Databases. New Generation Computing, vol. 9, pp ,

22 Thanks for your attention! Any questions?

23 Southwest Research Institute Mission Statement Benefiting government, industry and the public through innovative science and technology Founded in 1947 as an Independent, Nonprofit, Applied Engineering and Physical Sciences Research and Development Institution Broad Technological & Scientific Base 1200 Acre Campus in San Antonio, Texas Over $500M in annual revenue (FY12) roughly 50:50 government and industrial ~$7M in Internal Research (FY12) Over 3000 employees Over 990 patents and 35 R&D 100 awards 2.2 million ft 2 of Laboratory & Office Space 23