Cybersecurity Test and Evaluation Alex Hoover Test Area Manager Cyberspace & Homeland Security Enterprise Programs 202-254-5615 alex.hoover@hq.dhs.gov Office of Test & Evaluation Science and Technology Directorate
Agenda Policy Practice Threat Assessment COI/MOE/MOP 2
Procedures for Cybersecurity OT&E Purpose Improve operational resilience of network-enabled capabilities and inform major acquisition decisions. Applicability Acquisition programs subject to DOT&E oversight will incorporate these procedures into all future TEMPs and OT&E Plans. Programs will include cybersecurity in TEMPs Mission context, threat description, stakeholders, evaluation framework, integrated T&E, and resources OTAs will include cybersecurity in OT&E concepts, plans, & reports Realistic threat portrayal to determine mission effects DOT&E will include cybersecurity in LOAs Effectiveness, Suitability, and Cybersecurity 3
Cybersecurity-Informed Acquisition Is the capability sufficiently cyber secure to enter initial production/deployment? Test & Evaluation NEED ANALYZE/ SELECT Refine T&E Strategy OBTAIN 1 2 2B 2C 3 Input to Operational Requirements Develop T&E Strategy TEMP TEMP Conduct Developmental T&E LOA OTEP TEMP OTER PRODUCE/ DEPLOY/ SUPPORT LOA LOA Conduct Operational T&E OTER 4
Lifecycle Cybersecurity T&E Activities NEED JRC ANALYZE/ SELECT OBTAIN 1 2 2B 2C LRIP 3 DT&E IOT&E PRODUCE/ DEPLOY/ SUPPORT FOT&E ST&E ST&E Cybersecurity Requirements MNS ORD CONOPS Threat Assessment Intent Denial Disruption Modification Exfiltration Pivot Attack Surface Local Adjacent Network Kill Chain Tactics Exploits T&E Attack Surface Local Adjacent Network Kill Chain Tactics Exploits Security T&E Blue & Red Team Assessments Realistic threats in cyber domain Rigorous T&E is essential to close the gap between authorities to operate and operating securely 5 5
Program-Specific Threat Assessment Clearly define the threat(s) to the system and corresponding missions Threat assessment should answer the following general questions: Which threat actors may target the missions that the system supports? What is their intent? What do they view as the critical terrain to accomplish their intent? What are their capabilities in terms of knowledge, tools, and operations? What are their most likely and most dangerous attack vectors based upon their intent and capabilities? DHS does not have an existing process or office of primary responsibility for program-specific threat assessments Use Requests for Information thru Component and Department intelligence offices DOT&E working long-term solution with DUSM, DHS I&A, and JRC 6
Intent Denial Blocking completion of mission tasks. Degradation Decreasing the speed, quality, or other performance characteristics for mission tasks. Manipulation Altering the information available to decision makers. Exfiltration Gaining information about mission details to be exploited against other assets. Pivot Using access to one system/network to gain access to a partner system/network. 7
Operations Tools Knowledge Capabilities General Systems Target Network and Systems Target Operations Hardware Software Infrastructure Planning Procedures Minimal Limited Moderate Advanced Home market hardware, Common hardware, firmware, and Custom hardware, embedded Classified systems, platforms, and networks and, generalpurpose defensive devices. Enterprise systems, and less common software. Cross-domain devices, languages. Basic network and OS. Industry data network/protocols, specialized cryptography and associated hardware. user OS and applications. protocols. 0-day exploits of less firmware. Biometric-based 0-day exploits of restricted government Public cryptography/ common/more vulnerable software, authentication. 0-day exploits of systems and industrial control systems. authentication. Public exploits custom software. more common/less vulnerable of known vulnerabilities. software. Information found from commonly available open sources or from external reconnaissance of target organization. Information found from commonly available open sources or from external reconnaissance of target organization. Inexpensive home market hardware. Freeware and inexpensive commercial tools. Access through publically available infrastructure. Opportunistic actions, no planning. No demonstrated stealth, nonattribution or efficient use of resources Knowledge of network and system specifications and type/configuration of host-based defenses equivalent to an authorized user in the target environment. Knowledge from more specialized literature or equivalent to prior experience with target operations, including key information or supporting systems. Hard-ware, clusters, costing $10,000s or dozens of man hours. Commercial software. Direct control of leveraged public infrastructure. Intent and short-range plans formed on-the-fly as needed. Countermeasures for common defensive systems. Non-attribution. Efficiency in use of resources consistent with intent. Knowledge of network and system specifications and type/configuration of networked defenses equivalent to an authorized administrator in the target environment. Knowledge equivalent to substantial prior experience with target operations, including work flow and sub-task objectives. Hardware costing $100,000s or hundreds of man hours. Custom software, polymorphic malware, rootkits. Covert remote access tools and loggers. Organizes one or more operations with specific target systems and associated effects on target organization Advanced and custom nonattribution tools. Efficiency in use of resources consistent with intent Knowledge of network and system specifications and defenses equivalent to an authorized domain administrator in the target environment. Knowledge of current target operations equivalent to an experienced authorized operator. Custom hardware costing $1,000,000s or thousands of man hours. Custom software, firmware-resident malware. Covert close access. Organizes multiple operations against separate targets, synchronizing timing, accesses, and planned second-order effects High degree of control of defensive infrastructure. Non-attribution, false flag operations. Efficiency in use of resources consistent with intent Persistence Intermittent, directed activity. Gradual, low level passive operations. Repeated active operations. 24/7 monitoring and control of offensive capabilities. 8 8
Possible Evaluation Questions Critical Operational Issue Is this capability resilient to cyber attack? Measures of Cybersecurity How resilient is this mission to DOS attack of this capability? How resilient are the tasks to cyber degradation? How resilient are the procedures to data manipulation? How resilient is the mission to data exfiltration of the key cyber terrain? How well does this system protect against attack from/to interfaced capabilities? 9
Sample Cybersecurity Evaluation Structure Cybersecurity Is this capability resilient to cyber attack? Denial of Service (Mission Impact) - Probability of Occurrence - Repeatability - Duration - Attack Resources Understand Collective Impact on Mission/Task Accomplishment Degradation of Service (Task Impact) - Probability of Occurrence - Degree of Degradation - Duration - Attack Resources - Repeatability - Defend Resources - Probability of Detection Data Manipulation (Task Impact) Effectiveness Suitability Cybersecurity - Probability of Occurrence - Degree of Manipulation - Duration - Attack Resources - Repeatability - Defend Resources - Probability of Detection Measure 1 Measure 1 Measure 1 Data Exfiltration (Enterprise Impact) Measure 2 Measure 3 Measure 2 Measure 3 Measure 2 Measure 3 - Probability of Occurrence - Significance of Exfiltration - Duration - Attack Resources - Repeatability - Defend Resources - Probability of Detection External Pivoting (Enterprise Impact) - Probability of Occurrence - Probability of Detection - Duration - Attack Resources - Repeatability - Defend Resources 10
Rules of Engagement Purpose Support evaluation of... Threat Assessment Actors Intent Capabilities (FIS, Terrorist, Criminal, Activist, Mercenary, Hackers) (Historical, Projected, Surrogates) Strategic Goals Definition of Capability Under Test System Boundary Included Systems Excluded Systems Mission Impacts (DOS, DEG, DMAN, EXFIL, PIVOT) Operational Objectives Targeted Data (leads to...) Deliberate Targeted Systems (leads to...) Deliberate Targeted Networks (leads to...) Deliberate 80 / Exploratory 20 Targeted Interfaces (leads to...) Deliberate 50 / Exploratory 50 Relevant Vulnerabilities Deliberate 20 / Exploratory 80 11
Rules of Engagement (cont d) Tactical Plan Schedule Operational Objective, Capability (Surrogate) Start, End Initial Access TTP by Scheduled Event (planned and contingency) OCO Actions Limits of Action Prohibited Actions Termination Conditions/Notification DCO Posture Active Events Events the DCO will carry out Stop Events Events where the DCO will report detection to the red team and the event will stop Passthrough Actions Events where the DCO will report detection and the event will proceed Data Handling 12
13