Planning and Executing an Integration Test Strategy for a Complex Aerospace System Mr. Derrick Hinton Principal Deputy Director, Test Resource Management Center Complex Aerospace Systems Exchange September 11, 2012 Pasadena, CA 1
TRMC Organization Office of the Secretary of Defense Secretary of Defense Honorable Leon Panetta Deputy Secretary of Defense Honorable Ashton Carter Director, Operational Test & Evaluation Honorable J. Michael Gilmore Under Secretary of Defense for Acquisition, Technology & Logistics Honorable Frank Kendall Principal Deputy, vacant Assistant Secretary of Defense for Acquisition (ASD(A)) Honorable Mrs. Katrina McFarland Assistant Secretary of Defense for Research & Engineering Honorable Zachary Lemnios Director, Test Resource Management Center Mr Ed Greer Deputy Assistant Secretary of Defense for Systems Engineering Mr Stephen Welby Deputy Assistant Secretary of Defense for Developmental Test & Evaluation Mr Ed Greer Serves concurrently as Director, TRMC Section 231 of the 2003 NDAA established TRMC as a DoD Field Activity 2
TRMC Organization TRMC Organizational Structure DASD(DT&E) Dir, TRMC Deputy Director, Corporate Operations Deputy Director, T&E Range Oversight PD, TRMC Deputy Director, Technology Development T&E/S&T Program Agency Range Oversight Army Range Oversight Air Force Range Oversight Navy Range Oversight Deputy Director, Test Capabilities Development CTEIP Strategic Planning Coordinator International Facilities Coordinator T&E Budget Cert Coordinator Deputy Director, Interoperability JMETC Program Cyberspace, Interoperability & Distributed Testing 3
TRMC Background Need for Test Range Oversight Concern about managing the test infrastructure 1999 and 2000 studies. Recognition that the right range infrastructure resources enable and support adequate T&E of weapon systems. Need to assess duplicative test capabilities as essential or unnecessary costs. These identified concerns and needs resulted in the 2002 legislation: Bob Stump NDAA for FY03 (PL107-314) Sec 231: Establish a DoD Test Resource Management Center (TRMC) as a Field Activity under USD(AT&L) that will: Complete and maintain a Strategic Plan for T&E Resources Review and provide oversight of proposed DoD T&E budgets Certify Service and Defense Agency T&E proposed budgets Administer: Central Test and Evaluation Investment Program (CTEIP) Test and Evaluation/Science and Technology (T&E/S&T) Codified in Title 10 U.S. Code 196 4
TRMC Background Additional Responsibilities JMETC The 2005 Program Decision Memorandum II directed the stand-up of the Joint Mission Environment Test Capability (JMETC) Program under the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (OUSD [AT&L]) to provide the test infrastructure necessary for conducting Joint distributed test events USD(AT&L) assigned responsibility for execution to the Director, TRMC The JMETC Program Office stood up in October 2006 PDM II Nov 30, 2005 Cyber Effective October 1, 2012, I designate the Test Resource Management Center (TRMC) as the responsible organization for the National Cyber Range (NCR), as part of the TRMC mission to ensure adequate and available range capabilities for the Department. -- Mr. Frank Kendall USD(AT&L) Memorandum dated August 28, 2012 Subject: National Cyber Range Management Transition USD(AT&L) Memo Aug 28, 2012 5
Our Cyberspace T&E Problem Cyberspace vulnerabilities exist in DoD weapons systems Inadequate environments for conducting cyberspace T&E and experimentation Building a federated cyberspace environment is costly and time consuming Participant stove pipes delay integration of cyberspace capabilities DoD Needs A persistent, operationally realistic cyberspace environment with: Interoperability Standards that break down stove pipes Rapidly integrated Live, Virtual, & Constructive capabilities Common test & experimentation tools and applications 6
Decision Authority Development/Design Engineering Specialty Engineering Cyberspace T&E within the Acquisition Process Input Requirements Definitions (JCIDS) Operational Need Measures of Effectiveness/Suitability Output to Next Phase of Development or Lifecycle Support Transition Identify Cyberspace Measures Requirements Analysis System Measures of Performance Validation OT&E Validation against actual Cyber Threats Decomposition & Definition Cyberspace Vulnerability Assessments Component Level Testing to the chipset level Identify Logical cyberspace Analysis System and sub-system Dependencies Design Solution Detailed Design Allocated Functions & Performance Requirements Component/Interface Definition Element Design Criteria Implementation Test System of Systems Verification Mitigation of Cyberspace Threats Integration Confirmation Audits Realization & Assessment DT&E Verification in Realistic Environment Integrated Component Testing Tech Data & Training Pubs Component Characterization Hardware Fabrication Software Creation/Coding V Chart Source: DAU Acquisition Community Connection 7
Tenets of Good Government and Industry Systems Integration Testing Derives from Mission-Focused Requirements & Test Plans Realistic Mission Test Environment Combines a Tiered Management Structure & Accountability Utilizes Agile Development Concepts Leverages Existing Government and Industry Test Capabilities Employs Interoperability Standards that Enable Rapidly integrated Live, Virtual, & Constructive Capabilities Shares common Test & Experimentation Tools and Applications Unifies approach in conducting both Cyberspace and Traditional Testing Applies Effective Knowledge Management Techniques & Structures 8
TRMC Mission is to ensure all DoD Test Infrastructure is operated, improved, and sustained to meet current & future DoD requirements Enterprise Approach Benefits: Mitigates duplicative, inefficient, and stovepipe technical solutions Fosters standards & common test procedures improving data quality Provides confidence in out-year availability to acquisition programs Defines cost model for usage Optimizes modernization/investment plans to corporate priorities Enhances reusability across DoD DoD Acquisition Infrastructure Enterprise System Integration Laboratories Navy SILs DISA SILs AF SILs Navy ISTFs Army SILs AF ISTFs Installed System Test Facilities Hardware-in-the-Loop Test Laboratories DISA HWILs Army HWILs Army Ranges AF HWILs Realistic Mission Test Environment TENA-enabled Interoperability AF Ranges Navy HWILs Navy Ranges Open-Air Test Ranges Persistently connected via JMETC and Joint IO Range Industry HWILs Industry Labs Industry SILs Industry ISTFs Industry Test Facilities TENA: Test and Training Enabling Architecture Enterprise Approach mitigates duplication, improves efficiency and reusability, and optimizes long-range improvements and modernizations across the Department 9
Keys to an Effective Knowledge Management Approach Primary Goal of T&E: Discern Knowledge from Basic Measurement Data Trusted processes across government and industry that identify acquisition problems sooner rather than later Result: Develops mutual trust in testing products Accessibility of knowledge & data to legitimate users Result: Efficiencies gained through reducing the amount of testing required to obtain knowledge Discoverability of knowledge & data obtained over time Result: Enables data scientists to leverage data mining and analytical tools that rapidly answer questions and discover unknown unknowns Availability of knowledge through common tools & technologies Example: Cloud storage and services Result: Efficiencies gained from readily available & shared technology, hardware, software, and processes 10
BACKUP 11
TRMC Responsibilities Investment Programs: T&E/S&T Test & Evaluation Science & Technology Program Program Goal: To exploit new technologies and expedite their transition from the laboratory to the T&E community Program Philosophy Work closely with test capability developers (e.g., CTEIP projects, Service Range I&M projects) to mature and reduce risk in new test technologies Leverage heavily from ongoing efforts in industry, academia, and the DoD S&T community (DARPA, ONR, ARL, AFRL, etc.) Current T&E/S&T Test Technology Areas Cyberspace Test Directed Energy Test Electronic Warfare Test High Speed Systems Test Technology Development Net-Centric Systems Test Unmanned & Autonomous Systems Test Advanced Instrumentation Systems Technology Spectrum Efficient Technology 12
TRMC Responsibilities Investment Programs: T&E/S&T Mission: Develop Technologies Required to Test Future Warfighting Capabilities Established in FY02 Joint DDR&E / DOT&E Initiative Transitioned to TRMC in FY05 RDT&E Budget Activity 3 funds Purpose High Risk / High Payoff R&D for Testing FY 2012 80 Active Projects Foster technology transition to major DoD test ranges Risk reduction for test capabilities developments Annual Broad Agency Announcements (BAAs) Academia Industry Government Laboratories Tri-Service working groups Validate requirements Evaluate proposals Facilitate technology transition Central Oversight Distributed Execution High Speed Systems 13 Active Projects Current Test Technology Areas Unmanned & Autonomous Systems 3 Active Projects Spectrum Efficiency 14 Active Projects Advanced Instrumentation 8 Active Projects Directed Energy 17 Active Projects Cyberspace 2 Active Projects Electronic Warfare 16 Active Projects Net-Centric Systems 7 Active Projects Shaping Technology into Tomorrow s T&E Capabilities 13
TRMC Responsibilities Investment Programs: CTEIP Central Test and Evaluation Investment Program Test Capability Development Develops high priority test capabilities that resolve Joint / multi-service requirements shortfalls Promotes common test infrastructure and standard test methodologies Minimizes duplicative engineering development and life cycle support costs Partnership with Services and Agencies Service T&E Executives prioritize needs OSD centrally manages funding Service field activities lead development 14
TRMC Responsibilities Investment Programs: CTEIP Mission: Develop or Improve Major Test Capabilities that have Multi-Service Utility Initiated DEPSECDEF 9 November 1988 Established in FY91 by Congress RDT&E Budget Activity 4 funds Purpose Have multi-service utility Be developmental Be non-procurement FY2012 43 Active Projects T&E Master Plan (TEMP) References MILSATCOM Atmospheric Scintillation Simulator AEHF survivability and effectiveness testing in realistic atmospheric environments Ground Mounted Seeker Simulation Advanced open-air SAM simulator to support IDECM Block IV effectiveness testing Prec Tgt Sig - Performance Mover (PTS-RPM) Radar reflective/mobile targets for Gray Eagle UAS testing Threat Systems Project (TSP) 1-2 year requirement horizon EMD of target capabilities Address shortfalls in threat systems representation Coordinated with DOT&E $3-5M/year Joint Improvement & Modernization (JIM) 3-5 year requirement horizon EMD of Major Test Capabilities Must address joint requirements Services & Agencies budget for O&M over Life-Cycle of delivered capabilities $110-120M/year Resource Enhancement Project (REP) 1-2 year requirement horizon EMD of instrumentation needed to address an emergent requirement Must address OT shortfalls Coordinated with DOT&E $18-20M/year 11 TSP, 19 JIM, 13 REP = 43 Projects 15
TRMC Responsibilities Investment Programs: JMETC Joint Mission Environment Test Capability Provide a persistent and robust infrastructure to integrate Live, Virtual, and Constructive systems for test and evaluation in a Joint Systems-of-Systems environment Persistent connectivity, integration software, tools, reuse repository, technical expertise Supports distributed testing across the acquisition life-cycle Address SoS interoperability issues early Generate adequate numbers of systems under test Create adequate supporting cast of systems Link the right threat types, fidelity, and density for live testing JMETC Users Group Focus on tech requirements & solutions relevant to current & future distributed testing needs: Identify infrastructure requirements, use cases, and opportunities to collaborate Identify, investigate, and resolve issues Interoperability Discuss available solutions, tools, and techniques; share lessons learned For the Schedule and Location of next quarterly meeting: WWW.JMETC.ORG FY2012 70+ Distributed Event Activities 16
TRMC Responsibilities Investment Programs: JMETC 17