6/18/ ACC / TSA Security Capabilities Workshop THANK YOU TO OUR SPONSORS. Third Party Testing Program Overview.

Transcription

1 2015 ACC / TSA Security Capabilities Workshop June 16-18, 2015 #SecurityCapabilities THANK YOU TO OUR SPONSORS 2015 ACC/TSA Security Capabilities Workshop June Arlington, VA #SecurityCapabilities Third Party Testing Program Overview June 18, 2015 WARNING: This document is (FOUO). It contains information that may be exempt from public release under the Freedom of Information Act (5 U.S.C. 552). It is to be controlled, stored, handled, transmitted, distributed, and disposed of in accordance with DHS policy relating to FOUO information and is not to be released to the public or other personnel who do not have a valid need-to-know without prior approval of an authorized DHS official Transportation Security Administration 2 1

2 Third Party Testing (TPT) Program Introduction The Transportation Security Administration (TSA) TPT Program provides a standardized approach to testing Transportation Security Equipment (TSE) in response to procurement opportunities. This approach will help streamline the acquisition process by requiring vendors to provide more mature products up-front as well as save cost and time required to test, fix, and re-test during Qualification Testing (QT) and Operational Testing (OT). Key Benefits Savings in cost, time, and resources o More mature systems entering QT and OT result in a reduced cycle of test, fix, and re-test Transparency between government and vendors o Transparency leads to trust, reliability, consistency, and improves the quality of the Test and Evaluation (T&E) process 3 Cost-Benefit Analysis From TSE that exhibited system deficiencies requiring TPT, the TSA selected three systems for a cost-benefit analysis. Each TSE had resulted in multiple rounds of retests, extending the T&E timeline and increasing cost. Test Timeline Cost TSE 1 TSE 2 39 month delay between planned ARB and actual ARB 19 month delay between planned ARB and actual ARB $1.08M spent on retests $1.21M spent on retests TSE 3 59 month delay between planned ARB and projected ARB $1.22M spent on retests By addressing system deficiencies in TPT to avoid multiple rounds of retests, the TSA believes the T&E timeline can be significantly reduced. 4 TPT Program: High Level View With the TPT Program in place, the TSA will be able to approve TPT organizations that will then be available to independently test vendors systems before they enter the formal T&E process. TSA OSC T&E TPT Organization Vendor Approve TPT Organizations Maintain list of approved TPT organizations Conduct QT & OT TPT Procedures/ Admin Conformance to TSA Procedures Approve TPT Compliance to ISO/IEC and TSA specific requirements Conduct testing of product technologies per approved scope Generate test report Product Funding Test Results Offer Checked Baggage and Checkpoint products Meet TSA functional and operational requirements Test Results 5 2

3 TPT Program: Phases & Timeline FY15 Pre-Launch Finalize requirements and documentation in preparation of Go-Live TPT Program FY16 Launch Initial Go-Live and continuous monitoring FY16 Sustaining On-going improvement based on lessons learned Key Activities Publish TSA Test Method Publish TPT Program & Procedures Approve TPT Organizations Updated TSA Test Continuous monitoring Method based on and incorporation of feedback lessons learned Enhanced Program Activities Gather Feedback & Incorporate Lessons Learned Outcomes Initial List of Approved TPT Organizations On-going program improvements On-going program improvements 6 TPT Conformity Assessment Process The TPT Program requires vendors to use Third Party Test Organizations (TPTs) to independently test the TSE meets TSA requirements. Approval Conformity assessment process approves TPTs based on: Independence and quality management in accordance with ISO/IEC Technical competence of critical TSE operational areas per technology Key Benefits The benefits of leveraging the formal TPT approval process include: Consistency in testing resulting in more robust products Ensuring TPT independence from vendors Establishing an approved list of TPTs for vendors to use 7 TPT Approval Organizations applying to be TSA approved TPTs must submit an application, as well as quality system documentation conformant to the requirements of ISO/IEC and TSA specific management and technical requirements. Management Organization Management System Document Control Review of Requests, Tenders, Contracts Subcontracting Service to the Customer Complaints Improvement Corrective Action Preventive Action Control of Records Internal Audits Management Review Technical Personnel Accommodations and Environmental Conditions Test Methods and Method Validation Equipment Measurement Traceability Sampling Handling of Test Items Assuring Quality of Test Results Reporting the Results 8 3

4 TPT Approval Process Review Application & Gather Materials TPT Applicant reviews application & requirements; makes a business decision to apply TPT Applicant gathers required artifacts and completes application Submit Application TPT Applicant submits application Review by TSA TSA Review Team reviews the application and requested materials for adherence to TPT requirements Approval Decision TSA provides an approval decision to the TPT Applicant based on application review List of Approved TPT Organizations for Use by Vendors 9 Current T&E Process PMO ORD FRD Vendor Requirement Substantiation QDP Development Third Party Testing Level C Iterative test/fix cycle once formal T&E starts; increased schedule and cost OSC T&E OEM QDP Qualification Testing Operational Testing System Evaluation Report * ** Document Formal T&E Process Begins 10 Future T&E Process PMO Vendor ORD FRD Requirement Substantiation QDP Development Standard Test Method Consistent, repeatable testing processes Standard Test Reports Third Party Testing Level C Use of TPT will assist to streamline QT & OT timelines OSC T&E OEM QDP Qualificati on Testing Operational Testing System Evaluation Report * ** Document Formal T&E Process Begins 11 4

5 Appendix 12 Third Party Testing Strategy Highlights The TSA requires vendors entering the T&E process use TPT to support the development of their Qualification Data Package (QDP) Level C requirements, resulting in a more mature TSE and a higher quality QDP The TSA TPT Strategy limits a TSE to one (1) Significant Failure during QT and/or OT *Removal from testing relies on participation from the TSA Office of Security Operations in scoring failures and their determination and rationale associated with the operational impact The TSA reserves the right to put the TSE through QT or OT in it s entirety based on the assessment of the TPT effort and results The TSA also reserves the right to determine what information is shared with the OEM and TPT agent 13 National Conformity Assessment Hierarchy for Testing Approver ISO/IEC & TSA TSA OSC T&E TPT Organizations ISO/IEC TSA Specific Checked Baggage & Checkpoint Technology TSA Functional 14 5

6 TPT Process for QDP Development prior to QT/OT 15 TPT Process for Significant Failures during QT/OT 16 TPT Program: Key Activities & Timeline The TSA will develop TPT requirements and documentation, collaborating with industry partners in the process. Test Method Development Identify Critical Operational Areas per Technology Develop test scenario template Define draft test scenarios per technology based on current QT plans TSA is focusing on technologies in the process of developing new QMPs TPT Ensure TPT competency requirements map to critical op areas Update TPT requirements as needed Finalize TPT Application Process Develop TPT Application Form (including specific scope of approval) Develop TPT Test Report template Publish draft TPT Procedures for industry review Stakeholder Engagement Plan for stakeholder involvement to gain feedback on TPT requirements & program Define agenda for industry day - includes OEM & potential TPTs Hold industry day 17 6

7 TPT Program: Key Activities & Timeline (Continued ) After finalizing all necessary documentation, the TSA will launch its TPT program. Launch Preparation Finalize TPT Procedures & Test Method based on industry feedback Prepare communication channels for launch - website, box, etc. Finalize forms and TPT letters (approve, need more info, deny) Publish final TPT Procedure document and Test Method Initial List Launch Announce start of TPT application process Approve TPT organizations Post approvals on website Announce approved TPT list to stakeholders Continuous Monitoring Continue approval of TPT organizations, as needed Monitor TPT organizations Discuss possible transition to private-sector Accreditation Body 18 TPT Program: Major Accomplishments Developed TPT Strategy Identified critical operational areas per technology for TPT test method development and TPT organization scope definition Developed test scenario template Initiated drafting test scenarios for EMD Initiated drafting test scenarios for ETD 19 Cybersecurity in Test & Evaluation James S. Wells Deputy Director, Cyberspace & HSE Programs Office of Test & Evaluation 7

8 Problem Statement Insufficient T&E information regarding a system s cybersecurity posture is available to support major acquisition decisions. Networked information technology is a major component of most major DHS acquisition programs As a result, our adversaries have unprecedented access to our data and the ability to disrupt our operations Current T&E policies and practices do not adequately incorporate cybersecurity considerations in order to inform acquisition decisions 21 Current Parallel Processes Programs already plan and conduct cybersecurity activities IAW the Risk Management Framework HOWEVER Cybersecurity and T&E communities do not routinely coordinate and synchronize activities separate plans and separate reports to separate decision makers AND Operational T&E does not include realistic, threatrepresentative cyber attacks 22 Current Parallel Processes 0 1 2A 3 Need Analyze/ Select 2B Obtain 2C Produce/Deploy/Support Test & Evaluation Input to Operational Develop T&E Strategy Refine T&E Strategy Conduct OTEP Developmental T&E Conduct Operational T&E OTER OTER Risk Categorize Management System Framework Select Controls Implement Controls SAP SAR ATO Assess Authorize POAM SAR Controls Operation POAM Monitor Controls Systems Engineering Life Cycle SPR SER PPR SDR CDR IRR PRR OTRR ORR PIR PDR Solution Planning Integration Operations & Design Development Implementation Engineering Definition and Test Maintenance Disposal 23 8

9 Cybersecurity-Informed Acquisition Decisions Is there a sound plan to collect adequate cybersecurity data to inform future production & deployment decisions? Is the system sufficiently cyber secure to enter initial production/deployment? Is the system sufficiently cyber secure to enter full production/deployment? 0 1 2A 3 Need Analyze/ Select 2B Obtain 2C Produce/Deploy/Support Input to Test & Develop T&E Operational Refine T&E Strategy Evaluation Strategy Define Cybersecurity Threats & Add Cybersecurity T&E Environment Strategy to based on Identify RMF Planning Cybersecurity Conduct OTEP Developmental T&E Improve Fidelity of Cybersecurity DT&E and Synchronization with RMF Conduct Operational T&E OTER OTER Add Cybersecurity to OT&E Risk Categorize Management System Framework Select Controls Implement Controls SAP SAR ATO Assess Authorize POAM SAR Controls Operation POAM Monitor Controls Systems Engineering Life Cycle SPR SER PPR SDR CDR IRR PRR OTRR ORR PIR PDR Solution Planning Integration Operations & Design Development Implementation Engineering Definition and Test Maintenance Disposal 24 Draft DOT&E Policy Programs will include cybersecurity in s Threat description, evaluation framework, integrated T&E objectives & resources OTAs will include cybersecurity in test plans, test concept briefs, and evaluation reports Realistic threat portrayal to determine mission effects DOT&E will include cybersecurity in s Effectiveness, Suitability, Interoperability, & Cybersecurity 25 Current Activities Iterative coordination with DHS OCIO Initial discussions with Components & programs Inventorying possible cybersecurity T&E assets Coordinating with several programs as pilots Investigating process for program threat assessments with DHS I&A 26 9

10 Next Steps Coordinate and publish initial DOT&E cybersecurity policy memo Start integrating cybersecurity into s Start including cybersecurity in OT&E plans, reports, and DOT&E s Continue coordination with OCIO, DHS I&A, and Components Coordinate with Joint Council Continue discussion with red teams for possible recurring acquisition program support Continue coordination with pilot programs ACC / TSA Security Capabilities Workshop June 16-18, 2015 #SecurityCapabilities 10

11 THANK YOU TO OUR SPONSORS 2015 ACC/TSA Security Capabilities Workshop June Arlington, VA #SecurityCapabilities 11