Don t Be the Developer Whose Rocket Crashes on Lift off 2015 LDRA Ltd
Cost of Software Defects Consider the European Space Agency s Ariane 5 flight 501 on Tuesday, June 4 1996 Due to an error in the software design (inadequate protection from integer overflow), the rocket veered off its flight path 37 seconds after launch and was destroyed by its automated selfdestruct system
Rapid Adoption of Function Safety Standards Lockheed Martin Orion Functional safety standards adoption in space Commercial space industry, DOD, and others that are not normally required to meet avionics standards are rapidly adopting Embedded systems across industries Adopting best practices incrementally Standards compliance is becoming fundamental to winning business
Functional Safety Software Challenges Objectives and project documents Verification evidence and audit trail Integration with targets Tool qualification Structural coverage (Functional and unit Tests) Data/control flow and coupling Meeting coding standards Traceability of requirements through code and test Reduce cost of compliance Manage distributed team Reduce time to compliance and market
Leading Safety Critical Standards Avionics Industrial DO 178B (First published 1992) / DO 178C IEC 61508 (First published 1998, Updated 2010) Railway Nuclear Automotive Medical Process CENELEC EN 50128 (First published 2001) IEC 61513 (First published 2001) ISO 26262 (Published 2011) IEC 62304 (First published 2006) IEC 61511 (First published 2003)
Requirements Traceability Across Software Levels
Linking Requirements to Design Requirements Design Implementation Unit Test System Test Deployment Requirements Design Simulink
Linking Low Level Requirements to Design and Implementation Requirements Design Implementation Unit Test System Test Deployment Source Code Simulink
Typical Decomposition Within DOORS
Traceability Throughout The Lifecycle Traceability from high level system requirements through lower level requirements, code, and tests ensures that requirements are properly implemented and verified Essential for reliability and ensures security requirements are properly implemented and verified
Returning Data Back To DOORS
Automating Coding Standards Adherence Requirements Design Implementation Unit Test System Test Deployment C/C++ Java ADA High Quality Software MISRA C / C++ CERT C / CERT JAVA CWE High Integrity C++ HIS IPA/SEC C JSF++ AV Netrino C
Another Example: Code Result uint16_t a; uint16_t b; uint32_t c; uint32_t x; x = a + b + c; Depends on the size of int used by the compiler. If it is 16 bit, then there may be loss of granularity for a + b
Array Bounds Exceeded Code Result int32_t a[ 10 ]; uint32_t i; for ( i = 0; i < 20; ++i ) { a[ i ] = 0; } Depending on the runtime environment (OS, etc), this will result in an exception or overwrite unrelated memory.
Automation of Unit / Low level Testing Requirements Design Implementation Unit Test System Test Deployment LLR_0001 LLR_0002.. LLR_000n LLT_0001 LLT_0001.. LLT_000m
Developing, Executing, and Reviewing Tests
Robustness Testing at the Unit Level
The Requirement, Test Case and Results Visibility into requirements and test data at the point of test creation
Aggregating Coverage from High and Low Level Testing Requirements Design Implementation Unit Test System Test Deployment HLR_0001 HLR_0002.. HLR_000n HLT_0001 HLT_0001.. HLT_000m LLR_0001 LLR_0002.. LLR_000n LLT_0001 LLT_0001.. LLT_000m
What is Structural Coverage? Measurement of Test Effectiveness How effectively did tests exercise code? Exercised, entry points, statements, branches, compound conditionals, execution paths Systems requirement reliability levels up with one defect per 10 9 operating hours Metric that helps determine when a system is adequately tested Structural Coverage is Often Mandated DO 178B/C DO 278(A) for Commercial/Defense avionics and ground systems IEC 61508 for industrial controls ISO 26262 for automotive IEC 62304 for medical devices EN 50128 for rail Company based standards (in house)
Types of Coverage Depending on the SIL or DAL level and functional safety standard being followed, coverage requirements and required methodology varies Statement Coverage Branch Decision Coverage Modified Condition / Decision Coverage (MC/DC) Data Coupling and Control Coupling Coverage Object Code Coverage Linear Code Sequence And Jump Coverage Test Path (LCSAJ)
Visualising Structural Coverage Statement and Branch Coverage as viewed within a Flowgraph
Assembler Code Coverage Traceability of requirements through source code and object code Requirement for adherence DO 178C Level A High order language coverage as well as coverage at the Assembly level to show traceability to object code
Data and Control Coupling Coverage DO 178C section 6.4.4.2 c states: Analysis to confirm that the requirements based testing has exercised the data and control coupling between code components Control coupling coverage is ensuring that every invocation of a function has been exercised Data coupling coverage is ensuring that we have exercised every access to the data
From the Simulation to the Target Model Test Execution In Simulation Simulation Test Cases Modeling tools Reusing model tests Model Coverage/Code Coverage Model Behaviour And Model Coverage Application Behaviour And Code Coverage Generated Code Executed on Host Test Cases Host Computer Test Cases Reused Test Cases Pass Coverage Requirements Met Generated Code Executed on Target Test Cases Target H/W
Integrations Required Across the Lifecycle Requirements Modeling Tools RTM Compilers Processors Source Languages & Host Platforms IDE s Communication Protocols RTOS Version Control
Characteristics of Secure Systems Dependable Executes predictably Operates correctly Trustworthiness Minimal vulnerabilities that can be exploited No malicious logic Survivability Resists or tolerates known and novel attacks Quick recovery Dependability Trustworthiness Survivability Secure Software
Standards/Organizations Common Criteria IEC 62443 ISO 27001 IEC 62351 IEEE 12207 ISO/IEC 9797 1 Common Criteria for Information Technology Security Evaluation Industrial Network and System Security Information Security Management Information Security for Power System Control Operations (Smart Grids) Systems and Software Engineering. Replaced MIL STD 498 Security Techniques Message Authentication Codes
CERT Standards/Organizations Computer Emergency Response Team (CERT C and CERT Java) CWE/CVE OWASP DIACAP NIST Common Criteria Common Weakness Enumeration Open Web Application Security Project Defense Information Assurance Certification and Accreditation Process Information Assurance DOD National Institute of Standards and Technology Risk Management Framework Information Technology Security Evaluation
Understanding Quality and Security Software Security Software Quality Reliability Processes Security Processes Safety Critical Security Critical Thinking Safety vs Security Does the application perform reliably as designed? Is the application resistant to external attacks Failing Functionality vs Attack Surface
CERT Computer Emergency Response CERT Analysed more than 1000 Vulnerabilities Team Project through Carnegie Mellon University Addresses Vulnerability of S/W that is being developed S/W already deployed Primary Cause Easily avoided defects Common programming errors Follow the Standard Uniform set of rules and guidelines Determined by project and organisation Programmer s familiarity or preference doesn't matter
Automating Secure Standards Adherence
CWE Structural Coverage
Utilizing Continuous Integration Agility in high reliability software Popularity of continuous integration in the functional safety area Adoption of Agile development practices Finding practical ways of merging traditional requirements driven development with Agile practices
Improving Workflow with IDE Integrations Unified development and verification environment More efficient and easier to implement within a process Utilizes existing building/execution framework Reduces process implementation cost and developer time
Application Lifecycle Requirements Design Implementation Unit Test System Test Deployment
Questions & Answers
For Further Information www.ldra.com info@ldra.com @ldra_technology LDRA Software Technology LDRA Limited