BY ORDER OF THE ELECTRONIC SYSTEMS CENTER/ ESC/ENI INSTRUCTION ENGINEERING INTEGRATION (ESC/ENI) CHIEF Date: 8 August, 2011

Transcription

1 BY ORDER OF THE ELECTRONIC SYSTEMS CENTER/ ESC/ENI INSTRUCTION ENGINEERING INTEGRATION (ESC/ENI) CHIEF Date: 8 August, 2011 Test and Evaluation CAPABILITIES-BASED TEST AND EVALUATION COMPLIANCE WITH THIS PUBLICATION IS MANDATORY NOTICE: This publication is available digitally on the ESC/ENI SharePoint website: OPR: ESC/ENIT (Mr. Bloise Stubblefield) Certified by: ESC/ENI (Mr. Eugene Wright) Pages: 24 Distribution: F This instruction provides guidance and procedures for implementing an Integrated Developmental Test and Evaluation (IDT&E) program within Electronic Systems Center (ESC)/Engineering Integration (ENI) and is based on Air Force Instruction (AFI) , Capabilities-Based Test and Evaluation, and Air Force Materiel Command Instruction (AFMCI) , Test Management. It defines responsibilities for IDT&E programs that deliver Defense Business Systems (DBS) operational capability and or infrastructure to the war fighter. SUMMARY OF CHANGES This instruction is substantially revised and must be completely reviewed. This document supersedes and replaces 554 ELSWI , Capabilities-Based Test and Evaluation. 1. Scope. This instruction applies to all DBS and services for which ESC/ENIT is appointed as the Responsible Test Organization (RTO). The Chief, Engineering Integration Test Branch (ESC/ENIT), or designated representative, is the sole waiver authority for policies in this instruction. Refer to attachment 1 for a list of references and attachment 2 for a list of acronyms. 2. Responsibilities. 2.1 Chief, Test Branch (CTB). This instruction establishes the ESC/ENIT CTB to act as the overarching authority for IDT&E policy, training, and resources within ESC/ENIT. The CTB is responsible for advising the Air Force Program Executive Office (AFPEO) and Program Managers (PMs) on issues concerning IDT&E policies, procedures, and execution. The CTB acts as the liaison between the ESC Center Test Authority (CTA) and AFPEO organizations. The Chief, Engineering Integration (ESC/ENI), will appoint a primary and an alternate CTB. The CTB will: Be proficient in the performance of IDT&E and trained to a minimum of Acquisition Professional Development Program (APDP) Program Management

2 2 ESC/ENII August, 2011 Level III, Test and Evaluation Level III, and Information Technology Level III. The CTB will be certified based on their assigned acquisition coded position. The CTB will possess expert knowledge of: Applicable directives, instructions, manuals, standards, and techniques for management of all phases of IDT&E IDT&E strategy, planning and execution documentation development and review IDT&E execution, management, and reporting requirements Advise AFPEO and PMs on IDT&E policy and related issues Represent ESC/ENI IDT&E issues to HQ ESC, HQ AFMC, HQ USAF/TE, MAJCOMs, Deputy Assistant Secretary of Defense (DASD), Developmental Test and Evaluation (DT&E), Director, Operational Test & Evaluation (DOT&E), and other external test agencies Participate in IDT&E improvement working groups and test management councils, as-required Foster test partnerships and establish the level of participation needed for IDT&E test events with the Air Force Operational Test and Evaluation Center (AFOTEC), other Operational Test Organizations (OTO), the Joint Interoperability Test Command (JITC), Participating Test Organizations (PTOs), and other external test organizations Assign Test Directors (TD) to maintain insight into all IDT&E activities being conducted for each program and notify respective PMs of their assigned TD Advise Integrated Test Teams (ITTs) on RTO recommendations Develop and maintain appropriate IDT&E policies, processes, procedures, guides, templates, checklists, and other miscellaneous documentation. These artifacts will be reviewed and updated annually Ensure adherence with established IDT&E policies, processes and procedures Ensure IDT&E activities are conducted in operationally representative environments to conduct low risk, low cost IDT&E for each program for which the ESC/ENIT is appointed the RTO Develop and implement a standard IDT&E metrics program to capture and report relevant measurements. These decision-quality metrics will be used for assessing the maturity and quality of a program s IDT&E activities.

3 ESC/ENII August, Implement a standard set of tools and procedures to effectively manage, execute, and report IDT&E activities Manage the professional development of the human resources assigned within ESC/ENIT (APDP training, policy and process training, Air Force Institute of Technology (AFIT) courses, etc.) Establish and maintain a standard lessons learned program to capture and report relevant lessons learned during IDT&E planning and execution. 2.2 Test Director (TD). The TDs, assigned by the CTB, are responsible for directing the planning, execution, and overall management of IDT&E activities. TDs will be identified to support each program for which ESC/ENIT is appointed as the RTO. TDs will: Be proficient in the performance of IDT&E and trained to a minimum of APDP Program Management Level II, Test and Evaluation Level II, and Information Technology Level II. The TDs will be certified based on their assigned acquisition coded position. The TDs will possess expert knowledge of: Applicable directives, instructions, manuals, standards, and techniques for management of all phases of IDT&E The ITT and Test Integrated Process/Product Team (TIPT) establishment, management, and operation The Test & Evaluation (T&E) strategy, planning, and execution documentation development and review T&E execution, management, and reporting requirements Configuration control and baseline management procedures Conducting Test Readiness Reviews (TRR) Represent the CTB in all IDT&E activities in support of assigned programs. Provide oversight and assistance for all test execution events Ensure adequate RTO resources are allocated to support all IDT&E activities Coordinate on all baseline schedules and change actions that affect IDT&E activities and resources Promote ITT collaboration and ensure efficiencies of test activities across PMs, supporting engineers, system integrators, and all other participating agencies Advise AFPEO and PMs on issues related to test infrastructure requirements and test resource availability.

4 4 ESC/ENII August, Participate in program milestone and major review meetings for assigned programs. See section 4 for additional information. 2.3 Test Manager (TM). TMs, assigned by the TD, will be identified to support each program for which ESC/ENIT is appointed as the RTO. TMs will: Be proficient in the performance of T&E and trained to a minimum of APDP Program Management Level I, Test and Evaluation Level I, and Information Technology Level II. The TMs will be certified based on their assigned acquisition coded position Represent the CTB and TD by participating in all ITTs and TIPTs for assigned programs Prepare a Statement of Capability (SOC) that details the PMO and RTO support agreement. The SOC should include all identified RTO resources, capabilities, and funding required for each program. All SOCs will be submitted to the CTB and the PM for approval Develop or assist ITT members in preparing appropriate IDT&E documentation including test strategies, integrated test plans, integrated test descriptions, test scenarios, cases, and scripts and other test-related documentation Coordinate on all baseline schedules and change actions that affect IDT&E activities and resources Participate in all developer or system integrator testing. Assist PMs in making technically informed, objective assessments of developer or system integrator testing Receive all test artifacts and release packages delivered for formal government IDT&E and place them under configuration control Manage independent government IDT&E in accordance with approved test plans and descriptions. Serve as the focal point for resolution of test issues identified during IDT&E activities Coordinate all PTO participation for independent government IDT&E Ensure a common IDT&E database is established using the standard tools identified in paragraph Ensure all team members are trained in the use of the tools prior to execution of any IDT&E event Assist the Program Test Manager (PTM) with reporting, validating, and prioritizing Problem Reports (PRs) identified during IDT&E.

5 ESC/ENII August, Recommend corrective actions and recycle release packages back to the PM for rework when warranted Provide IDT&E test results and the final Integrated Test Report (ITR) to the PM and other stakeholders in support of program milestones and decision reviews Assist the PM and ITT in preparing for TRRs and Full Deployment Decision (FDD) meetings Assist the PM and ITT in certifying programs for dedicated operational test readiness in accordance with AFMAN , Certification of System Readiness for Dedication Operational Testing Assist the PM and ITT in transferring PRs identified, but not corrected prior to operational deployment, into the appropriate Deficiency Report (DR) management tool Participate in all program milestone and major review meetings for assigned programs. See section 4 for additional information. 2.4 Program Managers (PM). In addition to the responsibilities listed in AFI , chapter 3, and AFMCI , PMs will: Ensure appropriate T&E personnel participate in all test-related peer reviews and program milestone/review meetings as prescribed in the SEP. See section 4 for additional information Fund all IDT&E hardware, software, and activities from program inception through IDT&E activities and may also be required to fund portions of the Operational Test and Evaluation (OT&E) activities and related costs. Refer to AFI , volume 1, chapter 14, Budget Guidance and Procedures and the Systems Engineering Process (SEP) for more details Coordinate all baseline schedules and change actions with the TD (through the assigned TM) that affect IDT&E activities and resources Ensure developers/system integrators adhere to the IDT&E process and requirements as identified in this instruction. The assigned TM will assist the PM in preparing appropriate IDT&E contractual language. 2.5 Program Test Manager (PTM). The PTM, assigned by the PM, will: Represent the PM, when directed, in all test-related activities for assigned programs.

6 6 ESC/ENII August, Facilitate collaboration between all test participants and the PM Identify and coordinate life cycle test support and resource requirements with the developer or system integrator, CTB (through the TD and TM), and other external test organizations Develop or assist test team members in preparing appropriate IDT&E documentation including test strategies, plans, descriptions, scenarios, cases, scripts, and reports Participate in developer/system integrator IDT&E testing and monitor all other phases of T&E Ensure system changes are analyzed to adequately assess the level of regression testing required Establish a Watch Item Tracker (WIT)/ Deficiency Review Board (DRB) to ensure test teams report, validate, and prioritize deficiencies discovered during the test and evaluation process in accordance with T.O D-54, chapter Assist the PM in certifying program readiness in accordance with AFMAN Assist the PM in preparing for TRRs, including Operational Test Readiness Review (OTRR), and FDD meetings. 2.6 Evaluators. Evaluators are members of the test team that physically perform T&E of systems and review/prepare test-related documentation Lead Test Evaluators (LTEs) ensure all pre-test conditions are met prior to beginning any test activities. LTEs perform or observe all IDT&E tests in accordance with appropriate test plans and descriptions, and document the results (pass/fail/pr) in the common T&E database. LTEs also assist the TM in performance of their duties including preparation of test reports. In some cases, the TM serves as the LTE Test Evaluators (TEs) work with the TMs/LTEs and the project test team to assess testable requirements, prepare test data and scripts, and execute or observe IDT&E tests in accordance with appropriate test plans and descriptions. TEs document the results (pass/fail/pr) in the common T&E database. TEs also assist the TM/LTE in performance of their duties including preparation of test reports Functional Evaluators (FEs) may consist of any subject matter expert assisting with test event execution. This includes customers, users, or any other evaluator that has knowledge of the system and its requirements.

7 ESC/ENII August, Performance Evaluators (PEs) conduct application, database, operating system, and network performance evaluations on systems during IDT&E, in accordance with appropriate test plans and descriptions. The PE documents the results of all tests in the common T&E database and formal test report Information Assurance Evaluators (IAEs) plan, develop, and execute technical security evaluations for systems in accordance with appropriate test plans and descriptions. The IAEs also analyze network security scans, recommend corrective actions for vulnerabilities discovered, and assist the TM in preparing test reports Documentation Evaluators (DE) edit, review, and document PRs related to software documentation for systems. The DEs recycle documentation for correction of PRs prior to release to operational locations. DEs also assist the TM in preparing test reports. 3. Integrated Developmental Test & Evaluation (IDT&E) Process. IDT&E is conducted to evaluate design, quality, performance, functionality, security, interoperability, supportability, suitability, usability, and maturity of a system or capability in an operationally relevant environment. The IDT&E employs integrated testing to the maximum extent possible. Integrated testing is a concept for structuring T&E to more effectively support the requirements and acquisition process by integrating testing stakeholders, techniques, and procedures. Integrated testing assists in managing the correct amount of testing and managing risk. Integrated testing minimizes the transitions between contractor, developmental, operational, and information assurance testing by implementing integrated test principles, techniques, and procedures. Key stakeholders from multiple disciplines must integrate their efforts, produce efficient schedules, eliminate stovepipes, share information in open T&E databases, identify problems early, engage contractors to fix deficiencies sooner, and ensure systems are ready to enter dedicated operational testing with a high probability of success. The IDT&E includes contractor and government testing and is conducted over the life cycle of a system. IDT&E supports the acquisition of new systems or enhanced capabilities of existing systems before fielding decisions. IDT&E integrates contractor and government test activities during DT&E. It also integrates the operational test community into DT&E activities so that DT&E test results can be used to satisfy operational test objectives. Refer to attachment 3 for a flowchart that depicts the IDT&E process. The IDT&E is composed of three integrated phases: Initial Integrated Test Design (IITD) Component Validation and Integration (CV&I) Qualification Test and Evaluation (QT&E)

8 8 ESC/ENII August, Initial Integrated Test Design Phase (IITD). IITD is the planning phase of the IDT&E process. IITD planning will begin early in the acquisition life cycle and will be an integrated effort consisting of both contractor and all government resources and activities. PMs will ensure T&E personnel are involvement early in the system s life cycle starting with developing the definition of requirements, acquisition strategy development, program tailoring activities, and development of program schedules. Test planning will be conducted throughout a program s life cycle. Refer to attachment 4 for a flowchart that depicts the IITD phase. IITD phase activities include: The ITT is formed as a cross-functional, chartered team of empowered representatives/organizations that assists the acquisition community with all aspects of T&E planning and execution. The ITT will ensure development of the Test and Evaluation Strategy (TES), Test and Evaluation Master Plan (TEMP), Integrated Test Plan (ITP), and other T&E related documentation If AFOTEC declines participation, the ITT should determine if MAJCOM OT is required The RTO will co-chair the ITT as the senior test agency when there is no operational test organization participating The ITT will assist the PM in creating integrated master schedules TIPTs will be formed by the ITT to conduct specific T&E planning tasks or work defined test issues T&E personnel will participate in all program activities that support the development of requirements, configurations, or blueprinting activities. T&E personnel will assist the PM by ensuring that requirements/configurations are complete, clear, concise, and testable and can be validated against documented capabilities Requirements are first identified by an operational need. These requirements, developed by the functional community, take the form of operational, functional capability, and technical requirements in such documents as the Continuity of Operations (CONOPS), Capabilities Description Document (CDD), Business Case, and System Requirements Document (SRD). Operational requirements will take the form of business processes, Key Performance Parameters (KPPs), and Key System Attributes (KSA). Measure of Suitability (MOS) and Measure of Effectiveness (MOE) will be used to measure operational requirements The RTO, PMO, and functional community collaborate to perform a Capabilities Risk-Based Assessment by using the risk tab located in the Requirements Module of Hewlett-Packard (HP ) Quality Center. This assessment will provide PMs the ability to access/evaluate the risk to the

9 ESC/ENII August, program if enough time has not been incorporated into the schedule to fully test the capability. The idea is to test the higher level capability or functionality versus testing each individual requirement. By testing the higher level capability, it inherently compresses some of the testing time to a more manageable level. This type of information is vital to the PM s decisionmaking process. The assessment will identify the risk associated with each business process by: Business Criticality - determined by the functional community Probability of Failure - determined by the PM engineering team along with the developer NOTE: There is a manual method identified in the Capabilities Risk-Based Assessment Guide for those that do not have access to HP Quality Center The TES is the initial integrated T&E strategy for the program that describes how operational capability requirements will be tested and evaluated in support of the acquisition strategy. The TES may be the first version of the TEMP or incorporated into the Life Cycle Management Plan (LCMP). Under direction of the ITT, each program will prepare and coordinate a TES. The CTB (through the assigned TD and TM) will coordinate on all TES documents prior to approval. Refer to attachment 5 for a pictorial sample of a test strategy which may appear in the TEMP/LCMP and ITP. The strategy should identify the integrated DT&E and OT&E Test Phases, the environments, and the planned test segments The TEMP documents the overall structure and objectives of the T&E program. It provides a framework within which to generate detailed T&E plans and it documents schedule and resource implications associated with the T&E program. The TEMP identifies the necessary developmental and operational test activities. It relates program schedule, test management strategy and structure, and required resources to: Critical operational issues Critical technical parameters Objectives and thresholds documented in the requirements document Milestone decision points Under the direction of the ITT, Acquisition Category (ACAT) I and all Office of the Secretary of Defense (OSD) oversight programs (those on either the DOT&E or Acquisition Oversight Lists) will prepare and coordinate a TEMP. If a TEMP is not prepared, document the test strategy in the LCMP. When a TEMP is not required, all pertinent test information from the TEMP should be incorporated in the LCMP. The CTB (through the assigned TD and TM) will coordinate on all TEMPs prior to approval.

10 10 ESC/ENII August, 2011 NOTE: Tailored acquisition strategies may call for different T&E planning artifacts; however, they will contain the same type of information as the TES and TEMP The ITP establishes the scope of the IDT&E activities of the program/family of systems. The ITP integrates developer and government test approaches and activities into a single overarching plan covering all IDT&E events. Under the direction of the ITT, each program will prepare and coordinate an ITP. The ITP for all ACAT I programs will be approved by the PM and CTB (coordinated through the assigned TD and TM). All other ITPs will be approved by the PM and TD The Integrated Test Description (ITD) establishes the scope of the IDT&E activities of the program for a SPECIFIC release. It documents the specific activities to be conducted for all IDT&E test phases and test segments. Under direction of the ITT, each program will prepare and coordinate an ITD for each specific release. The ITD establishes the scope of the IDT&E activities for a specific release. It integrates developer/integrator and government test approaches and activities into a single set of descriptions Test scenarios, cases, and scripts guide T&E activities by defining step-bystep instructions to effectively evaluate the requirements of a release. Under the direction of the ITT, each program will prepare representative set test scenarios, cases, and scripts to facilitate disciplined T&E of their program. These test scenarios, cases, and scripts will be maintained in HP Quality Center Under the direction of the ITT and in accordance with TO 00-35D-54, USAF Deficiency Reporting, Investigation, and Resolution, each program will establish and implement a WIT/DRB to manage PRs and deficiencies identified during T&E activities. Periodic WIT/DRB meetings are conducted during test execution to determine the severity, root cause, and ownership/responsibilities of PRs generated during T&E activities ESC/ENI employs the following standard test tools to manage and execute IDT&E activities HP Quality Center is a web-based system for automated software testing and management. Programs will utilize HP Quality Center to: Document testable program requirements Build test scenarios, test cases and test scripts and link them to program requirements (requirements traceability and test coverage) Execute automated test scripts

11 ESC/ENII August, Standardize WIT/PR management Gather enterprise-wide test-related metrics HP Load Runner measures the performance of every tier, server, and component of a system. When applicable, programs will utilize HP Load Runner to test mission applications with end-user loads versus response times to determine system bottlenecks HP Quick Test Professional provides functional test and regression test automation capabilities. This automated testing solution deploys the concept of keyword-driven testing to simplify test creation and maintenance. When applicable, programs will utilize HP Quick Test Professional, to the best extent possible, to automate regression testing of mission applications HP Service Oriented Architecture (SOA) Service Test is a SOA testing solution that simplifies and accelerates the automated functional testing of SOA services. It reduces the time required to test SOA services and helps ensure that these services meet a program s business requirements prior to deployment. When applicable, programs will utilize HP SOA Test, to the best extent possible, to test SOA based applications. 3.2 Component Validation and Integration (CV&I). CV&I, led by the developing agency or system integrator with government participation/oversight, demonstrates that each individual component and the assembled components are developed in accordance with the approved design and functions properly to meet specified requirements. CV&I may be conducted in iterations as components are completed and integrated. Refer to attachment 6 for a flowchart that depicts the CV&I phase. CV&I includes the following test segments: Individual Component Validation (ICV) validates each individual component is developed and operates in accordance with specified requirements and approved designs Component Integration Test (CIT) validates completed components can be integrated into a complete system in accordance with specified requirements and approved designs Data Management (DM) is the process to Extract, Transform, and Load (ETL) data from one system for use in another, usually for the purpose of application interoperability or system modernization. DM may consist of Data Migration, Data Conversion, and or Data Validation Requirements Operability Test (ROT) validates the integrated system functions properly and meets specified requirements and approved designs.

12 12 ESC/ENII August, Performance Evaluation Test (PET) evaluates the performance of the integrated system by employing techniques which may include bandwidth analysis, load testing, and stress testing to ensure the system performs in accordance with specified requirements and approved designs Information Assurance Evaluation (IAE) evaluates information-related risks to a system. It includes assessments of: Network security policies Identification and authentication Access controls Auditing Confidentiality, integrity, and availability of data and their delivery systems The IAE may include application of Security Technical Implementation Guides (STIGs), Security Readiness Review (SRR) Scans, Information Assurance Control Validation, and Application Software Assurance Center of Excellence (ASACOE) application code scans System Integration Test (SIT) validates the integration of a system into an operationally-relevant environment (installation, removal, and backup and recovery procedures). CV&I SIT is conducted by the developing activity in an operationally-relevant environment, observed by the government User Evaluation Test (UET) is typically ad-hoc testing conducted by end users of the system. UET is conducted to offer an early look at the maturity of the system and to evaluate how well the system meets mission requirements. NOTE: Not all CV&I activities will be accomplished in an operationally representative environment. Regression testing will be conducted throughout all phases of IDT&E. It validates existing capabilities and functionality is not diminished or damaged by changes or enhancements introduced to a system. Regression testing also includes break-fix testing which verifies corrections implemented functions to meet specified requirements. 3.3 Qualification Test and Evaluation (QT&E). QT&E, led by the RTO, validates the product integrates into its intended environment, meets specified requirements in accordance with the approved design, meets performance standards, and validates the information assurance controls employed by the system meet Department of Defense (DoD) standards and policies. QT&E is performed in a government-provided and managed operationally-relevant environment. The purpose of QT&E is to validate the product integrates into its intended environment, meets both functional and nonfunctional requirements, meets performance standards, and validates the information assurance controls employed by the system meet DoD standards and policies.

13 ESC/ENII August, The QT&E also supports Limited Deployment (LD) test activities in the actual production environment. QT&E can also be supported by end-users (with oversight by the RTO/PTO) to provide end-user input related to system maturity and its ability to meet operational mission requirements. Refer to attachment 7 for a flowchart that depicts the QT&E phase. QT&E includes the following test segments: SIT validates the integration of a system into an operationally-relevant environment (installation, removal, and backup and recovery procedures) DM is the process ETL data from one system for use in another, usually for the purpose of application interoperability or system modernization. DM may consist of Data Migration, Data Conversion, and or Data Validation System Operability Evaluation (SOE) is traceable scenario and or script driven end-to-end qualification testing of a system that validates the integrated system operates in accordance with specified requirements and approved designs. SOE should also include Regression testing, which validates that existing capabilities/functionality is neither diminished nor damaged by changes or enhancements introduced to a system PET evaluates the performance of the integrated system by employing techniques such as bandwidth analysis, load testing, and stress testing. PET during QT&E may be tailored based on the results of similar tests conducted during CV&I IAE evaluates information-related risks to a system. It includes assessments of: Network security policies Identification and authentication Access controls Auditing Confidentiality, integrity, and availability of data and their delivery systems The IAE may include application STIGs, SRR Scans, Information Assurance Control Validation, and ASACOE application code scans UET is typically ad-hoc testing conducted by end users of the system. UET is conducted to offer an early look at the maturity of the system and to evaluate how well the system meets mission requirements LD begins when the Functional Sponsor and the Milestone Decision Authority (MDA) approves fielding the capability into an operational environment for:

14 14 ESC/ENII August, 2011 System Acceptance Test (SAT), conducted during the QT&E Test Phase as part of Limited Deployment, obtains confirmation that a system meets requirements. The end users or subject matter experts provide such confirmation after they conduct a period of trial or acceptance test Initial Operational Test and Evaluation (IOT&E) of the implementation and use of a major release at one or more selected operational sites It provides the opportunity to observe the initial implementation and use of the system under actual operating conditions prior to the FDD. NOTE: Regression testing will be conducted throughout all phases of IDT&E. It validates existing capabilities and functionality is not diminished or damaged by changes or enhancements introduced to a system. Regression testing also includes break-fix testing that verifies corrections implemented functions to meet specified requirements. 4. Program Reviews and Milestones. The involvement of T&E personnel in major program reviews and milestone decisions are critical to program success. During all phases of a program, T&E personnel will ensure test strategy, planning, and execution activities supporting the program are organized and accurately reported during reviews and milestone decisions. 4.1 Program Reviews. PMs will ensure the TD, TM, and appropriate evaluators are invited to participate in the following program reviews: Program Management Reviews (PMRs), Technical Management Reviews (TMRs), Initial Technical Reviews (ITRs), Alternative System Reviews (ASRs), SRRs, System Functional Reviews (SFRs), Technology Readiness Assessments (TRAs), Preliminary Design Reviews (PDRs), Critical Design Reviews (CDRs), and DRBs. 4.2 Sufficiency Reviews. Sufficiency Reviews are conducted as: CV&I assessment prior to TRR I to determine the sufficiency of CV&I test activities, provide an engineering go/no-go recommendation and determine readiness to conduct TRR I QT&E midpoint assessment of IDT&E effectiveness used to determine the readiness for movement into OT&E, Limited Deployment, or production. The RTO and PTM will assist the PM in preparing for this review 4.3 Test Readiness Reviews (TRR) Test Readiness Review I (TRR I). TRR I is the formal review, conducted by the PM, signifying the CV&I portion of IDT&E is complete and recommends the system move into the QT&E portion of IDT&E testing. The results of the TRR I will demonstrate that each individual component and the assembled components are developed or configured in accordance with the approved design and function properly to meet specified requirements. The PTM and RTO will assist the PM in preparing for and conducting TRR I.

15 ESC/ENII August, Test Readiness Review II (TRR II). TRR II is the formal review, conducted by the PM, signifying the main phase of QT&E is complete and that the release is ready to move into Limited Deployment test activities. The PTM and RTO will assist the PM in preparing for and conducting TRR II Operational Test Readiness Review (OTRR). OTRR is a multi-disciplined product and process assessment to ensure that the production configuration system can proceed into IOT&E with a high probability of success. The PTM and RTO will assist the PM in preparing for the OTRR. 4.4 Full Deployment Decision (FDD). FDD is the final decision made by the Milestone Decision Authority (MDA) authorizing an increment of the program to deploy for full operational use. The PTM and RTO will assist the PM in preparing for FDD. 4.5 Milestone Reviews. The PTM and RTO will assist the PM in preparing for all Milestone Reviews. X Eugene S. Wright III EUGENE S. WRIGHT, III, GS-15, DAF Chief, Engineering Integration Division

16 16 ESC/ENII August, 2011 References Attachment 1 Glossary of References and Supporting Information AFI , Acquisition and Sustainment Life Cycle Management AFI , Volume 1, Budget Guidance and Procedures AFI , Capabilities Based Test and Evaluation AFI , Major Range and Test Facility Base (MRTFB) Test and Evaluation Resource Planning AFMAN , Certification of System Readiness for Dedicated Operational Test and Evaluation AFMCI , Test Management CJCSI G, Joint Capabilities Integration and Development System CJCSI E, Interoperability and Supportability of Information Technology and National Security Systems DoDD , The Defense Acquisition System DoDI , Operation of the Defense Acquisition System Defense Acquisition Guidebook (DAG) TO 00-35D-54, USAF Deficiency Reporting, Investigation, and Resolution Systems Engineering Process for Defense Business Systems Capabilities Risk Based Test Assessment Guide

17 ESC/ENII August, Attachment 2 ACAT AFI AFIT AFMAN AFMC AFMCI AFOTEC AFPEO APDP ASACOE ASR CDD CDR CIT CONOPS CTA CTB CV&I DAG DASD DBS DE DISA DM DoD DOT&E DR DRB DT&E ELSWI ENI ENIT ESC ETL FDD FE HP HQ IAE ICV IDT&E IITD Abbreviations and Acronyms Acquisition Category Air Force Instruction Air Force Institute of Technology Air Force Manual Air Force Materiel Command Air Force Materiel Command Instruction Air Force Operational Test and Evaluation Center Air Force Program Executive Office Acquisition Professional Development Program Application Software Assurance Center of Excellence Alternative Systems Review Capabilities Description Document Critical Design Review Component Integration Testing Concept of Operations Center Test Authority Chief, Test Branch Component Validation & Integration Defense Acquisition Guide Deputy Assistant Secretary of Defense Defense Business Systems Documentation Evaluators Defense Information System Agency Data Management Department of Defense Director, Operational Test & Evaluation Deficiency Report Deficiency Review Board Developmental Test and Evaluation Electronic Systems Wing Instruction Engineering Integration Engineering Integration Test Branch Electronic Systems Center Extract, Transform, and Load Full Deployment Decision Functional Evaluator Hewlett-Packard Headquarters Information Assurance Evaluation / Information Assurance Evaluator Individual Component Validation Integrated Developmental Test and Evaluation Initial Integrated Test Design

18 18 ESC/ENII August, 2011 IOT&E ITD ITP ITR ITR ITT JITC KPP KSA LCMP LD LTE MAJCOM MDA MOE MOS OSD OTO OTRR OT&E PDR PE PET PM PMR PR PTM PTO QT&E ROT RTO SAT SEP SFR SIT SOA SOC SOE SRD SRR STIG T&E TD TE TEMP TES Initial Operational Test and Evaluation Integrated Test Description Integrated Test Plan Initial Technical Review Integrated Test Report Integrated Test Team Joint Interoperability Test Command Key Performance Parameter Key System Attribute Life Cycle Management Plan Limited Deployment Lead Test Evaluator Major Command Milestone Decision Authority Measure of Effectiveness Measures of Suitability Office of the Secretary of Defense Operational Test Organization Operational Test Readiness Review Operational Test and Evaluation Preliminary Design Review Performance Evaluator Performance Evaluation Testing Program Manager Program Management Review Problem Report Program Test Manager Participating Test Organization Qualification Test & Evaluation Requirements Operability Testing Responsible Test Organization System Acceptance Test Systems Engineering Process System Functional Review System Integration Testing Service Oriented Architecture Statement of Capability System Operability Evaluation System Requirements Document System Requirements Review / Security Readiness Review Security Technical Implementation Guide Test and Evaluation Test Director Test Evaluator Test and Evaluation Master Plan Test and Evaluation Strategy

19 ESC/ENII August, TIPT TM TMR TRA TRR I/II UET USAF WIT Test Integrated Process/Product Team Test Manager Technical Management Review Technology Readiness Assessment Test Readiness Review I/II User Evaluation Testing United States Air Force Watch Item Tracker

20 20 ESC/ENII August, 2011 Attachment 3 Integrated Developmental Test and Evaluation (IDT&E) Overview

21 ESC/ENII August, Attachment 4 IDT&E Initial Integrated Test Design (IITD) Phase

22 22 ESC/ENII August, 2011 Attachment 5 Pictorial Sample Strategy for Test

23 ESC/ENII August, Attachment 6 IDT&E Component Validation & Integration (CV&I) Phase DT&E Component Validation and Integration (CV&I) Phase

24 24 ESC/ENII August, 2011 Attachment 7 IDT&E Qualification Test and Evaluation (QT&E) Phase