Don t Be the Developer Whose Rocket Crashes on Lift off LDRA Ltd

Size: px

Start display at page:

Download "Don t Be the Developer Whose Rocket Crashes on Lift off LDRA Ltd"

Daniel Berry
6 years ago
Views:

2 Don t Be the Developer Whose Rocket Crashes on Lift off 2015 LDRA Ltd

Cost of Software Defects Consider the European Space Agency s Ariane 5 flight 501 on Tuesday, June 4 1996 Due to an error in the software design

3 Cost of Software Defects Consider the European Space Agency s Ariane 5 flight 501 on Tuesday, June Due to an error in the software design (inadequate protection from integer overflow), the rocket veered off its flight path 37 seconds after launch and was destroyed by its automated selfdestruct system

Rapid Adoption of Function Safety Standards Lockheed Martin Orion Functional safety standards adoption in space Commercial space industry, DOD, and others that are not normally required

4 Rapid Adoption of Function Safety Standards Lockheed Martin Orion Functional safety standards adoption in space Commercial space industry, DOD, and others that are not normally required to meet avionics standards are rapidly adopting Embedded systems across industries Adopting best practices incrementally Standards compliance is becoming fundamental to winning business

Tests) Data/control flow and coupling Meeting coding standards Traceability of requirements

5 Functional Safety Software Challenges Objectives and project documents Verification evidence and audit trail Integration with targets Tool qualification Structural coverage (Functional and unit Tests) Data/control flow and coupling Meeting coding standards Traceability of requirements through code and test Reduce cost of compliance Manage distributed team Reduce time to compliance and market

6 Leading Safety Critical Standards Avionics Industrial DO 178B (First published 1992) / DO 178C IEC (First published 1998, Updated 2010) Railway Nuclear Automotive Medical Process CENELEC EN (First published 2001) IEC (First published 2001) ISO (Published 2011) IEC (First published 2006) IEC (First published 2003)

7 Requirements Traceability Across Software Levels

8 Linking Requirements to Design Requirements Design Implementation Unit Test System Test Deployment Requirements Design Simulink

9 Linking Low Level Requirements to Design and Implementation Requirements Design Implementation Unit Test System Test Deployment Source Code Simulink

10 Typical Decomposition Within DOORS

Traceability Throughout The Lifecycle Traceability from high level system requirements through lower level requirements, code, and tests ensures

11 Traceability Throughout The Lifecycle Traceability from high level system requirements through lower level requirements, code, and tests ensures that requirements are properly implemented and verified Essential for reliability and ensures security requirements are properly implemented and verified

12 Returning Data Back To DOORS

13 Automating Coding Standards Adherence Requirements Design Implementation Unit Test System Test Deployment C/C++ Java ADA High Quality Software MISRA C / C++ CERT C / CERT JAVA CWE High Integrity C++ HIS IPA/SEC C JSF++ AV Netrino C

14 Another Example: Code Result uint16_t a; uint16_t b; uint32_t c; uint32_t x; x = a + b + c; Depends on the size of int used by the compiler. If it is 16 bit, then there may be loss of granularity for a + b

15 Array Bounds Exceeded Code Result int32_t a[ 10 ]; uint32_t i; for ( i = 0; i < 20; ++i ) { a[ i ] = 0; } Depending on the runtime environment (OS, etc), this will result in an exception or overwrite unrelated memory.

16 Automation of Unit / Low level Testing Requirements Design Implementation Unit Test System Test Deployment LLR_0001 LLR_ LLR_000n LLT_0001 LLT_ LLT_000m

17 Developing, Executing, and Reviewing Tests

18 Robustness Testing at the Unit Level

19 The Requirement, Test Case and Results Visibility into requirements and test data at the point of test creation

20 Aggregating Coverage from High and Low Level Testing Requirements Design Implementation Unit Test System Test Deployment HLR_0001 HLR_ HLR_000n HLT_0001 HLT_ HLT_000m LLR_0001 LLR_ LLR_000n LLT_0001 LLT_ LLT_000m

21 What is Structural Coverage? Measurement of Test Effectiveness How effectively did tests exercise code? Exercised, entry points, statements, branches, compound conditionals, execution paths Systems requirement reliability levels up with one defect per 10 9 operating hours Metric that helps determine when a system is adequately tested Structural Coverage is Often Mandated DO 178B/C DO 278(A) for Commercial/Defense avionics and ground systems IEC for industrial controls ISO for automotive IEC for medical devices EN for rail Company based standards (in house)

22 Types of Coverage Depending on the SIL or DAL level and functional safety standard being followed, coverage requirements and required methodology varies Statement Coverage Branch Decision Coverage Modified Condition / Decision Coverage (MC/DC) Data Coupling and Control Coupling Coverage Object Code Coverage Linear Code Sequence And Jump Coverage Test Path (LCSAJ)

23 Visualising Structural Coverage Statement and Branch Coverage as viewed within a Flowgraph

adherence DO 178C Level A High order language coverage as

24 Assembler Code Coverage Traceability of requirements through source code and object code Requirement for adherence DO 178C Level A High order language coverage as well as coverage at the Assembly level to show traceability to object code

Data and Control Coupling Coverage DO 178C section 6.4.

data and control coupling between code components Control coupling coverage is ensuring

25 Data and Control Coupling Coverage DO 178C section c states: Analysis to confirm that the requirements based testing has exercised the data and control coupling between code components Control coupling coverage is ensuring that every invocation of a function has been exercised Data coupling coverage is ensuring that we have exercised every access to the data

From the Simulation to the Target Model Test

Modeling tools Reusing model tests Model

Coverage Application Behaviour And Code Coverage

Computer Test Cases Reused Test Cases Pass

26 From the Simulation to the Target Model Test Execution In Simulation Simulation Test Cases Modeling tools Reusing model tests Model Coverage/Code Coverage Model Behaviour And Model Coverage Application Behaviour And Code Coverage Generated Code Executed on Host Test Cases Host Computer Test Cases Reused Test Cases Pass Coverage Requirements Met Generated Code Executed on Target Test Cases Target H/W

27 Integrations Required Across the Lifecycle Requirements Modeling Tools RTM Compilers Processors Source Languages & Host Platforms IDE s Communication Protocols RTOS Version Control

28 Characteristics of Secure Systems Dependable Executes predictably Operates correctly Trustworthiness Minimal vulnerabilities that can be exploited No malicious logic Survivability Resists or tolerates known and novel attacks Quick recovery Dependability Trustworthiness Survivability Secure Software

29 Standards/Organizations Common Criteria IEC ISO IEC IEEE ISO/IEC Common Criteria for Information Technology Security Evaluation Industrial Network and System Security Information Security Management Information Security for Power System Control Operations (Smart Grids) Systems and Software Engineering. Replaced MIL STD 498 Security Techniques Message Authentication Codes

30 CERT Standards/Organizations Computer Emergency Response Team (CERT C and CERT Java) CWE/CVE OWASP DIACAP NIST Common Criteria Common Weakness Enumeration Open Web Application Security Project Defense Information Assurance Certification and Accreditation Process Information Assurance DOD National Institute of Standards and Technology Risk Management Framework Information Technology Security Evaluation

Thinking Safety vs Security Does the application perform reliably as designed?

31 Understanding Quality and Security Software Security Software Quality Reliability Processes Security Processes Safety Critical Security Critical Thinking Safety vs Security Does the application perform reliably as designed? Is the application resistant to external attacks Failing Functionality vs Attack Surface

defects Common programming errors Follow

32 CERT Computer Emergency Response CERT Analysed more than 1000 Vulnerabilities Team Project through Carnegie Mellon University Addresses Vulnerability of S/W that is being developed S/W already deployed Primary Cause Easily avoided defects Common programming errors Follow the Standard Uniform set of rules and guidelines Determined by project and organisation Programmer s familiarity or preference doesn't matter

33 Automating Secure Standards Adherence

34 CWE Structural Coverage

35 Utilizing Continuous Integration Agility in high reliability software Popularity of continuous integration in the functional safety area Adoption of Agile development practices Finding practical ways of merging traditional requirements driven development with Agile practices

36 Improving Workflow with IDE Integrations Unified development and verification environment More efficient and easier to implement within a process Utilizes existing building/execution framework Reduces process implementation cost and developer time

37 Application Lifecycle Requirements Design Implementation Unit Test System Test Deployment

38 Questions & Answers

39 For Further Information LDRA Software Technology LDRA Limited

Structural Coverage Analysis for Safety-Critical Code - Who Cares? 2015 LDRA Ltd 1

Structural Coverage Analysis for Safety-Critical Code - Who Cares? 2015 LDRA Ltd 1 What is Structural Coverage? Measurement of Test Effectiveness How effectively did tests exercise code? Exercised, entry