Boeing Certification Techniques for Advanced Flight Critical Systems Challenge Problem Integration (CerTA FCS CPI) Briefing at the

Transcription

1 Boeing Certification Techniques for Advanced Flight Critical Systems Challenge Problem Integration (CerTA FCS CPI) Briefing at the AFRL Safe & Secure Systems & Software Symposium (S5) BOEING is a trademark of Boeing Management Company. 03 June 2009 Dayton, Ohio Approved for Public Release. Case Number: 88ABW

2 Outline CerTA FCS CPI Program Impact Program Approach and Program Tasks Measures of Merit and Key Performance Parameters Initial Effectiveness Analysis V&V Technologies Exercised on Program Technology Description Impact Summary 2

3 CerTA FCS CPI in Boeing Context HALE (High-Altitude Long Endurance) ScanEagle Boeing Development Cycle Flight Test / OA Day-in-the-Life i ca tio n nit e fi ali da tio na nd V er if - GVT - SMI h ic Ve lin de Mo le EOM ΣF=ma Design: Implement Requirements g Adaptive Control Margins (UIUC) Unit Testing -MATRIXx -DTE Code Generation Analysis - Stability - Performance - FQ s - Failures at ion,v Modeling and Simulation te gr d an ion H/W and S/W Integration & Test (HIL, SIL) In sd Vehicle Integration Testing Rqmts & ICDs SpecSafe Safety Analysis (EDAptive) ScanEagle Compressed Carriage System Verification Testing (SVT) V&V Managers (Boeing) Configuration Trade Studies nt me ire Flight Surfaces Deployed qu Re Integrator Phantom Ray CodeHawk Code Properties (Kestrel Technology) MORC Code EDICT Fault Generation (Boeing) Tolerance (WW Tech) X-48B 3

4 Closer Look at Boeing Development Cycle Day - in - the - Life Functional Qualification Test Flight Test / OA Configuration Management Configuration Trade Studies Problem Report Tracking Requirements Database System Verification Testing (SVT) Vehicle Integration Testing Rqmts & ICDs Modeling and Simulation Autocode -- **************************************************************** ************/ #include < stdio.h > #include < math.h > #include " sa_sys.h " #include " sa_defn.h " #include " sa_types.h " #include " sa_math.h " #include " sa_user.h " #include " sa_utils.h " #include " sa_time.h " #include " sa_fuzzy.h " /*** System Data ***/ Model - Based Design S y s t e m S y s t e m A B S y s t e m S y s t e m C D S y s t e m E F H/W and S/W Integration & Test (HIL, SIL) EOM Σ F=ma Design: Implement Requirements Unit Testing Code Analysis Generation - Stability - Performance - FQ s - Failures 4

5 Program Approach Boeing Program Experience Innovative CerTA FCS V&V Technologies The Boeing Team Expertise The Boeing CerTA FCS CPI Program Task 1 : Challenge Problem Definition Identify desired characteristics Evaluate candidates Acquire project metrics Review with AFRL Task 2: V&V Tech Identification and Integration Identify the key innovations Characterize their properties Integrate into CerTA- FCS-CPI V&V process Review with AFRL Task 3 : Develop Measures of Merit Analyze CP & tech innovations Collect extensive Boeing experience Select MoMs & KPPs Review with AFRL Challenge Set of MoMs & Innovative Problem KPPs to Quantify V&V Process Definition Improvements Task 4 : Initial Effectiveness Analysis Option - Task 5 : Challenge Problem Demonstration Apply historical data Apply CerTA FCS & experience with the CPI V&V process to CP technologies Assess in terms of MoMs & KPPs Review with AFRL Assess in terms of MoMs & KPPs Demonstrate for AFRL Transitions to Effectiveness of Improvements of V&V Innovations to Certification Emerging Programs Task 1 Challenge Problem Definition Task 2 V&V Technology Identification and Integration Task 3 Develop MoMs / KPPs Task 4 Initial Effectiveness Analysis 5

6 The Challenge DO-178B the gold standard of SW verification Based on defined processes and thorough testing We can t test our systems safe! Systems growing exponentially in size and complexity The test space is growing/exploding even faster The time has come for new approaches to assure system safety & security Software for Dependable Systems: Sufficient Evidence Also Internationally recognized 6

7 Solutions Identified Safety / Assurance Cases Analysis performed earlier in development process Formal Methods, Verification Conditions, other math techniques Mitigate some later testing Automation of the Development Process Certification aspects can be particularly labor intensive Evolutionary Humans will stay in the loop Today we automate a lot more than we used to 7

8 MOMs and KPPs Measures of Merit (MOMs) represent the top level goals and objectives of a program, platform, or process improvement effort. Time/Cost Assurance Expertise (to use/apply) Enhanced Vehicle Capability (Increases in software complexity providing advanced functionality) Key Performance Parameters represent the benefits of individual features, aspects, steps or technologies Benefit of investment in a development process step such as resulting in lower development cycle cost / lifecycle cost 8

9 The Big Picture Experimentation Parameter Analysis Tool Analysis Output KPP MOM 9

10 Initial Effectiveness Analysis Approach Evaluation of the MOMs and KPPs for each technology Benefits relative to a UAV development As quantifiable as possible Estimating cost / schedule impact of inserting tool into process Consideration of benefits (and potential costs) Cycle time reduction Defect (and re-work) avoidance Increased assurance and confidence Expertise needed to use tool 10

11 V&V Technologies Undergoing Experimentation Shown in Boeing Development Cycle context Day-in-the-Life Flight Test / OA Requirements Definition and Vehicle Modeling Configuration Trade Studies Rqmts & ICDs SpecSafe Safety Analysis (EDAptive) Modeling and Simulation EOM ΣF=ma Design: Implement Requirements Adaptive Control Margins (UIUC) V&V Managers (Boeing) Unit Testing -MATRIXx -DTE Code Generation Analysis - Stability - Performance - FQ s - Failures EDICT Fault Tolerance (WW Tech) H/W and S/W Integration & Test (HIL, SIL) MORC Code Generation (Boeing) System Verification Testing (SVT) Vehicle Integration Testing - GVT - SMI CodeHawk Code Properties (Kestrel Technology) Integration, Validation and Verification 11

12 Technologies and Application to a Challenge Problem V&V Technologies CodeHawk (Kestrel Technology) Stability and robustness margins (UIUC) SpecSafe (EDAptive Computing) MORC (Boeing) EDICT (WW Technology Group) V&V Managers (Boeing) Challenge Problem Aspects/Artifacts C source code Control laws with adaptation Matlab design portion of inner loop control design Considerations: properties to prove MORC converts control system models directly into object code. AADL model build from UAV requirements, design artifacts, including both H/W and S/W V&V artifacts, such as requirements, traceabilityrelated annotation of Matlab models, autogenerated source code. 12

13 CodeHawk Kestrel Technology Abstract Interpretation applied to source code Input: C source code Output: Errors, Warnings, Safe indications of each location in the source code relevant to the property Approach Parse code Produce conservative overapproximation of the reachable state space Check property against state space CodeHawk is actually a checker generator Define property Generate checker Automatically check property 13

14 Illustration of Abstract Interpretation M = 0 M M i = 0 i i i < 10? stop M 20,000 M = M M = M ,000 i++ 10 i 14

15 MOMs and Kestrel Technology CodeHawk Time/Cost without theoretically proving properties of the source code, additional testing and peer review would be required, and additional errors may escape the software coding phase leading to rework Assurance formal methods techniques provide more state space coverage than current methods Expertise (to use/apply) want to bring this to the level of the normal Software Engineer CerTA FCS CPI is helping us explore this dimension 15

16 Stability / Robustness Margins for Adaptive Systems University of Illinois at Urbana- Champaign (UIUC) Reference Model + - Adaptive Control Guidance Baseline Autopilot + + Airframe 16

17 Stability / Robustness Margins for Adaptive Systems V&V Challenge Verify control laws satisfy required robustness margins Challenge Problem Manifestation Adaptive control laws with non-linear behavior are difficult to verify V&V Technology Theoretically justified Adaptive Control Margins & L1 adaptive controller applied to challenge problem flight controller 17

18 MOMs and UIUC Adaptive Control Margins Time/Cost without theoretically justified robustness margins analysis, V&V of adaptive control would necessitate a very large set of simulation runs and analysis of results given today s processes Assurance robustness margins analysis aim to provide more state space coverage than current methods (i.e., numerous simulation runs) Expertise (to use/apply) want to bring this to the level of the normal Controls Engineer CerTA FCS CPI is helping with this transition Enhanced Vehicle Capability robustness margins techniques are aimed at enabling the fielding of flight platforms that benefit from adaptive control Better performance at challenging flight envelope conditions Better performance in various CG and outer mold line configurations (e.g., different load-outs) Better disturbance rejection Better performance in the face of vehicle and subsystem degradation and damage 18

19 SpecSafe EDAptive Computing, Inc. Theorem Proving applied to Control Models Input: Simulink diagram Output: Proven properties Approach Translate to internal form (Rosetta/Syscape) Automatically generate properties Prove automatically (PVS) Additional capability: Generate run-time monitors 19

20 SpecSafe Tool Flow Functional Test Vectors INPUT, Simulink Model FCS Specification Design (Rosetta) Equivalency Checking FCS Specification Design Model / Specification Conversion Automated Assistance PVS Proof Script Theorem Proving Proven FCS Specification Design Assertion Generation Run-Time Assertions Run-Time Assertions Conversion Output 1, Proven Lemmas FCS Specification with Run-Time Assertions Output 2, Simulink Model with Runtime Assertions STANDARD FLOW 1. A Simulink design is converted to Syscape 2. Equivalency Checking is performed: a. Structural Verification b. Boundary Verification c. Dynamic Verification 3. Syscape model is converted to PVS lemmas 4. The lemmas are Proven 5. Safety Properties are proven 6. Assertions may be translated to Simulink run-time monitors. 20

21 MOMs and EDAptive SpecSafe Time/Cost without theoretically proving properties of the controller, additional human-intensive simulationbased validation of the model would be required, and additional errors may escape the controller design phase leading to rework Assurance formal methods techniques provide more state space coverage than current methods Expertise (to use/apply) want to bring this to the level of the normal Controls Engineer CerTA FCS CPI is helping us explore this dimension 21

22 MORC (Model to Object Reversible Compiler) Boeing Go from model to verifiable assembly code Input: Simulink/MATRIXx Diagram Output: Assembly code Approach Generate code that preserves the structure of the model for easier traceability (and reversibility) Generate efficient embedded code Generate safe code Reduces V&V cost of safety critical systems (Level A) Guarantees object code matches the model Eliminates need for analysis of object code to show traceability Facilitates the reuse of test cases developed in simulation phase When used in conjunction with a COTS code generator, provides an independent thread of verification Additional capability: the generated code compiles to more efficient object code and boosts real-time performance 22

23 MORC (Model to Object Reversible Compiler) Traditional Autocoder Approach Simulation Test Vectors Built-In Autocoder C/Ada Code Compiler/ Linker Simulation Test Results Model Embedded Compiler/ Linker Simulation on host Target Test Vectors Target processor Target Test Results 23

24 MORC (Model to Object Reversible Compiler) MORC Approach Test Vectors Built-In Autocoder C/Ada Code Compiler/ Linker Test Results =? Independent Model-to- Object Autocoder Model Assembly code for target Assembler/ Linker Simulation on host =? Model Reverser Target processor Test Results 24

25 MOMs and Boeing MORC Code Generation Time/Cost Translation from model-based design directly to object code skips the steps of source code generation, source code peer review, source code configuration management, and compilation Assurance Structure of generated code simplifies mapping of model to code, increasing confidence in the correctness of generated code Expertise (to use/apply) want to bring this to the level of the normal Software Engineer CerTA FCS CPI is helping us explore this dimension Enhanced Vehicle Capability Generated code is more efficient (smaller size, faster execution) 25

26 EDICT Fault Tolerance Analysis WW Technology Group SafeSuite EDICT Tool Suite Domain Tools Error Security Handling Safety SafeSpec Project Help Management Views Model Editors Analysis Core Tools SafePlace Team OSATE System Composer SOC System Analyzer Inter-Tool Interfaces V&V Challenge Verify that fault containment requirements are satisfied Measurement Effort to analyze the model, Effort to incorporate any identified changes Platform Run-Time Resource Services Eclipse Environment SW Component Library 26

27 EDICT Fault Tolerance Analysis 27

28 MOMs and WW Technology Group EDICT Time/Cost Reduced cost through earlier detection of fault containment errors and consequent avoided rework Assurance Automated analysis increases confidence that all cases are covered Expertise (to use/apply) want to bring this to the level of the normal Systems Engineer / Architect CerTA FCS CPI is helping us explore this dimension 28

29 V&V Managers (VVMs) Boeing Certification Artifacts created throughout process Manual production very expensive E.g., every G&C requirement is implemented in the modelbased design VVMs automate this collection (n-1) Dev. Step VVM (n-1) Meta-data about implementation n th Dev. Step Executes on VVM n Results Certification Artifacts 29

30 MOMs and Boeing V&V Managers Time/Cost Reduced cost through Automation of V&V data collection Automation of V&V data package preparation V&V efficiencies with standardization of evidence packages Assurance Higher certification authority confidence in collected, assessed, and presented certification data Expertise (to use/apply) will enhance the effectiveness of Software and Quality Engineers 30

31 Summary CerTA FCS CPI has allowed for relevant experimentation Promising V&V technologies On a real UAV challenge problem design (Requirements, Model-Based Design, Autocode and Hand Code) The tools/technologies we are working with are showing promise of benefits to the development process for flight critical software subject to airworthiness certification Our evaluations are being fed back to the V&V tool development teams Scaling Usability Relevance to our domain Effectiveness analysis providing an initial look at value 31

32 Copyright Boeing. All rights reserved. 32