NSWCDD-PN-18-00055 t s just software Or t s all software and it s the new normal John Seel, Ph.D. Distinguished Engineer for Warfare s Software 540-653-4443 John.seel@navy.mil
Thoughts about software We ve moved from a model of it s just software to a model where software is the key ingredient in many systems Software sustainment requires domain expertise and software expertise Software sustainment is about: Continuous engineering and technical debt Fixing errors and problems with existing software New capabilities, new hardware, new cyber challenges There is no reset in software sustainment, only changes from the baseline Software sustainment triggers: The complete systems engineering process Re-design or re-factoring of existing code Potentially Recertification 2 2
Software Reliant Warfare s Challenges Number Software Size, Reliance, Complexity, and Cyber Threats Cost, Schedule, & Technical Challenges overnment in-house applied Software Expertise * Verified in documented performance result reports of DOD software system acquisition Time Challenges include ever increasing: - Demand for faster and cheaper development and delivery of systems to meet emergent needs & threats - Pressure to cut corners and not utilize rigorous and disciplined data-driven bestpractices/processes - Cyber threats and associated information and software assurance requirements - and -of- (SOS) complexity and inter-dependencies - Rapid evolution of software technologies and methodologies A single wrong character in the thousands to millions of lines-of-code can result in a mission critical failure 3 3
Engineer-n Quality, Safety, Security, and Resiliency OAL: Define, develop and deliver high quality, safe secure and resilient systems nformation Assurance (A) compliance (patches) does NOT equal cyber security Utilize processes, tools, and technology to facilitate vulnerability/threat detection, isolation, and recovery Define Assurance and Resiliency Requirements Architect/Design resilient defense-in-depth systems Protect-Detect-solate-Endure-Recover -Limit interfaces to external systems - dentify and harden (firewall) critical control points (interfaces) - Add processing to Detect cyber intrusions - solate and limit consequences (protect mission critical components)- - Endure through the Cyber intrusion to successfully complete the mission - Return the system to a trusted state Utilize multiple-tools to remove vulnerabilities during development Mission & Stakeholder Analysis Concept eneration Requirements Definition. SYSTEM ANALYSS & CONTROL User Centered, Risk Focused, and Total-Cost based Engineering Analysis and Assessment: Modeling & Simulation Architecture nterface Definition Configuration Management Specialty Engineering: Anti-Tamper, Environmental, Human s ntegration nformation Assurance, Logistics, Safety, Training, etc. Verification, Validation, & Certification ntegration &Test nformation Assurance Patches only Decommission address& Dispose Known threats Train, Operate,& Support ---------------------------- Deploy/nstall Cannot test in system & software security & resiliency Define and utilize metrics and tools to quantify sw vulnerability, survivability, resiliency, and risk Subsystem/nterface Design Subsystem ntegration/test Utilize data-driven best engineering practices nclude independent Audits to ensure best practices Build and Unit Test (Hardware/Software) Assurance : The level of confidence that system functions as intended and is free of vulnerabilities, either intentionally or unintentionally designed or inserted as part of the software throughout the lifecycle. Application of technologies and processes to achieve a required level of confidence that systems and services function in the intended manner, are free from accidental or intentional vulnerabilities, provide security capabilities appropriate to the threat environment, and recover from intrusions and failures. 4
Common Software Oversight and ntegration Challenges Software Development Responsibility % DEFNE Req s ARCHTECT & Software DEVELOP & Software Key: : NSWCDD SMEs : ndustry Organizations NTERATE & TEST A common software (sw) system acquisition approach is for the government to over-rely on private industry for the sw architecure, design, and coding activities overnment interaction with the sw development effort is limited to reactive (versus proactive) technical reviews nd Engineering Technical Reviews (SETRs) E.. SRR, SFR, PDR, CDR, TRR, This approach frequently results* in: - Vendor-Lock & Proprietary Software - Cost & Schedule Overruns - Poor Quality - Non Modular Open s Architecure (MOSA) - ntegration ssues Limited insight and understanding of the sw development organization(s) development approach, processes, methodologies, toolsets, and if best-practices are being utilized Limited insight or understanding of private industries cost/schedule/quality Basis-of-Estimates (BOEs) - Failure to validate estimates against actual historical performance for that specific contractor Lack of leading-indicators (metrics) for sw development cost, schedule, technical performance and quality Lack of regular data-driven performance/risk management reviews that include gov t software experts Limited insight and understanding of SW arch, design, interfaces (SETRs occur too late to address issues) * As documented in numerous studies/reports from the ov t Accounting Office (AO), Defense Science Board (DSB), Carnegie Mellon University Software Engineering nstitute (CMU SE), and others 5 5
Software Success Teaming: ndustry & overnment in-house expertise overnment and ndustry SW development Teaming Approach: ov t engineers responsible not just for oversight but also for actual development (subset of sw architecture, design and coding) Benefits: Provides PMs with business and technical advantages Provides Program Managers with access to in-house sw experts Facilitates controlling cost: government is not solely reliant on industry expertise Enables transitioning sw between development organizations if req d due to performance issues Provides industry with a true technical peer to help negotiate fair cost, schedule, and technical approach Teaming With ndustry Software Development Responsibility % DEFNE Req s ARCHTECT & Software DEVELOP & Software Hands-On Development NTERATE & TEST Key: : NSWCDD SMEs : ndustry Organizations ov t n-house Applied Expertise Pipeline Responsibility & Complexity ov t hands-on sw development at all levels of complexity is required to maintain expertise pipeline& be a technical peer with industry Mission and s-of-s Level Level Component Level nd Sub-Components Level Time / Experience ov t Software Experts team with ndustry SW Experts to Define, Design, Develop, and Deliver Software s 6 6 6
Keys to Software Project Oversight and ntegration Success* Proposals that include Cost, Schedule and Quality estimation Basis-Of-Estimates AND historical data that demonstrates the contractor can execute within the estimates and goals Data-driven continuous improvement ; and historical data/demonstration of this capability Contracts that specify and require: Documented Mature Data-Driven Best-Practice Based SW Development Processes Software Assurance and reliability requirements Modular Open s Architecure (MOSA) ov t ownership rights of Source Code and ALL other supporting documentation, scripts, problem reports, & quality metrics, etc. ncluding all libraries, environments, etc required to compile and regenerate the executables Frequent structured data-driven communication, including metrics and leading indicators for cost, schedule Utilization of Formal best-practice based management of Requirements, nterfaces, Risk and Configurations Adequate Cost, Schedule and Resources to develop and/ or utilize model-based-system-engineering, automated testing, models/simulations, data-extraction and other non-operationally-delivered sw products Not the complete set of best practices; just a selected subset of proven techniques 7 7
overnment and ndustry SW Development Teaming Consistent Success Rapid Development Tactical Development Strategic Development Detect-Track-Engage s: Lasers, Precision Munitions, uns, Lethal & Non-Lethal Tomahawk Weapon Control Surface Warfare Mission Module (Littoral Combat Ship) Submarine Ballistic Missile Mission Planning and Fire Control s 10+ years 30+ years 50+ years ov t in-house developers teaming with industry developers has been successfully utilized for some of the Navy s most mission critical systems for many decades December 2016 SW Quality assessment - Data from 10 programs,15 systems, 177 different sw components, and 110 deliveries between 2012-2016 - Wide range of systems: Missile, un, Laser Weapon/Fire Control & Mission Planning systems - Wide range of sw size: 9 Million Source-Lines-of-Code (SLOC) to several-thousand SLOC - Wide range of sw development approaches: Nuclear Certified, DOD 5000 Waterfall, Agile, Rapid Consistent high quality software delivered on schedule and within budget - Operationally successful and proven results (e.g. Tomahawk Weapon Control, Battle Management ) - Well over 90% of projects delivered software with ZERO open mission critical defects - Well over 90% of systems reported ZERO mission critical defects after operational delivery 8 8
Technical Excellence, Rigor, and Discipline MATTERS We develop some of the most lethal and dangerous systems in the history of mankind These are our friends & colleagues husbands & wives mothers & fathers sons & daughters brothers & sisters THER LVES DEPEND ON US We must deliver safe, reliable, secure, and high quality systems 9