Q48: I noticed an amendment to the ASED BAA, what has changed? Q48: The due date for proposals has been extended from November 9 to November 28.

Transcription

1 HR001117S0050 Active Social Engineering Defense (ASED) Frequently Asked Questions As of October 20, 2017 Q50: We would like to know if there is interest in, or if our bid would be considered compliant, if we include in our bid a fully functional virtual test environment to be used for interim evaluation and testing. A50: The purpose of the test range is to supply real people and real people s responses to measure human in the loop system performance for TA1+TA2 and for TA3. If a proposed test range facilitates ASED testing, it would be considered. Q49: TA3 states, "TA3 systems and teams will be required to deploy on a U.S. Governmentprovided test range for continuous evaluation as described below in Section I.C. TA3 teams must work with the U.S. Government team to manage the deployment of their systems and the evaluation of bi annual improvements of TA1 TA2 systems throughout the program. Is the government test range the only range the ASED program will use or is the government open to additional testing resources? A49: The Government provided range is the only deployment option for TA3 proposals. Q48: I noticed an amendment to the ASED BAA, what has changed? Q48: The due date for proposals has been extended from November 9 to November 28. Q47: Per page 7 of the BAA, TA1 teams will support multiple purpose driven communication channels. The BAA states To make using multiple channels practical, TA1 systems must automatically route communications appropriately. Will TA1 teams be provided with programmatic access to an existing system in use at the government provided testbed? Q47: Yes, access will be given to system accounts of volunteers. Q46: Will the users ( senders) select the proper channel, or is TA1 routing supposed to be automatic? A45: TA1 routing should be automatic when possible. TA1 systems can make use of the human to help route but this should be rare. Q44: Will the purpose driven channels integrate SMS and , or be single mechanism? A44: The purpose driven channels integrate SMS and .

2 Q43: One way of detecting a link/attachment based phishing attack is to pre test the payload. Is the government providing testing capabilities as one of the limited resources described on p9 of the BAA (e.g. sandboxed virtual machines )? A43: No, it is up to performers to create such resources. Q42: Is research in this type of payload testing in scope for the program? Or will all teams be provided with a standard suite? A42: No, but usage of existing payload testing tools is within scope to help perform late stage detection. Q41: If a performer has pre existing capabilities in this domain, can they be deployed in the government testbed? A41: Yes, but it is subject to IT verification. Q40: On page 5, the BAA mentions that the test range may include an organization s phone system. Is the program covering voice communications in addition to text? A40: No, but it will cover SMS. Q39: Figure 4, page 11, shows evaluations/integrations at months 6 and 12, though month 18 shows the first integrated toolkit or preliminary system. Are the evaluations that start at month 6 and 12 of the individual TA1 (or TA2) components in isolation? Or is integration of TA1 and TA2 components happening as early as month 6? A39: Integration starts early, but targets only apply at the end of each phase. Q38: Regarding data generated in the test range, is that to be made available for training learning algorithms? If so, is any available before month 6? When do you expect such data to be available? A38: Data generated in the test range will be made available between evaluations and after each evaluation event. Q37: Will data from the test range always remain in the test range, or can it be made available to participants outside of the test range for development? A37: It will be available for participants under an NDA and after cleaning. Q36: Will the test range allow continual monitoring by TA1 and TA2 performers remotely? Or only at the test range? A36: In general, only on site or after export procedures. Subsets of the test range data, namely social media, may be available from outside the range. Q35: What is the CONOPs for ASED participant systems during evaluations? A35: To detect potential attacks and generate as much identifying information about attackers as possible.

3 Q34: Is the input to TA1 and TA2 systems limited solely to what is accessible to the accounts and alter egos created for performers on the test range, or will performers be able to run additional services and appliances? A34: Solely the data and metadata from the range. Outside data (outside of attached metadata) won t be available. Q33: Will performers have control over the systems that run the bots, including access to network traffic? In promiscuous mode? A33: Performers may configure systems prior evaluation exercises. Access will not be authorized during the exercise. Q32: Will accounts created for performers include administrative level privileges that will allow bots to create, delete, and control multiple user accounts? A32: Admin privileges will be available to setup bots and infrastructure. Q31: Some information that will be relevant to defending social engineering attacks are already captured with existing network and system monitoring tools (geolocation, timestamped netflows, spam filters, etc.). Will these services be running within the test bed and will performers have access to them? A31: Logs from commercial firewalls and internal spam filters will be available. Outside of enterprise" data will not be available. Q30: Can TA1/TA2 performers install and integrate network security monitoring tools? A30: Not beyond what already exists. Q29: Can TA1/TA2 performers install software on user endpoint systems and impact the user interface for the witting and savvy subjects? A29: Yes. Q28: Will the volunteer subject pool also create traffic in and amongst themselves that will be needed to be distinguished from attack traffic, or does the non attack traffic come from a different source? A28: The volunteer subject pool will create traffic that performers will have access to. Q27: Will consumable resources such as honeypots and bitcoin wallets be provided as part of the testbed or are performers responsible for their creation and management? If they are provided, will performers also be able to create additional and novel consumable resources accessible to agents outside the enterprise network as well as domain names resolvable through DNS? A27: It is the responsibility of the TA1/TA2 teams to procure these.

4 Q26: How much access would teams have to the testbed between the evaluations, e.g., for development, training, and assessment purposes? A26: Data will be collected during this time, but performers won t have access to these systems. Q25: Can an applicant institution submit more than one proposal under a given technical area? A25: Yes. Q24: What is the anticipated start date for the Active Social Engineering Defense (ASED) program? A24: We anticipate a program start date of May Q23: In addition to , text messaging was mentioned as a media type of interest. Is it necessary to have SMS (or imessage) traffic in the testbed? A23: The desired end result will encompass more communication channels than just . Q22: Would the testbed need to host software (e.g., bots) written by people who are not from the test bed creator? A22: Yes, the testbed needs to host other performer software. Q21: Is there a set of required information about the users that the systems are trying to protect, or do we get to decide that? A21: Specific information will be defined during the evaluations but in general privileged system information, PII, etc. will be the class of information that we ask TA1/TA2 systems to protect. Q20: What are desirable levels of scale for the number of users in the testbed? A20: The testbed will operate on hundreds thousands of users. The ultimate goal is to be able to deploy the defenses on a large DISA scale network. Q19: Can an applicant institution submit more than one proposal under a given technical area? A19: Yes. Q18: Is there any exclusion to performance across technical areas? A18: Yes, teams that perform TA3 may not perform on TA1 or TA2. However, teams can perform on both TA1 and TA2. Q17: Can different groups within the same company perform on TA1/2 and TA3. A17: No. As stated in the BAA, while proposers may submit proposals for all three technical areas, proposers selected for TA3 cannot be selected for any portion of the other two technical areas, whether as a prime, subcontractor, or in any other capacity from an organizational to an individual level. This is to avoid OCI situations between the

5 technical areas and to ensure objective test and evaluation results. The decision as to which proposal to consider for award is at the discretion of the Government. Q16: Will bots have information about human targets they are protecting? A16: Possibility, but they would protect this information and have human in the loop access. Q15: Is there interest in non digital communications? A15: Yes. Q14: Will the government provide a corpus of attacks for detection? A14: The government will provide attacks for training purposes and in TA3. Q13: What is the temporal length of attacks? A13: It will vary on the attack, but it may be days and/or weeks. Q12: What is the frequency of attacks? A12: The frequency of attack will vary. Q11: Will DARPA provide the IRB? A11: DARPA does not provide an IRB. The range provider will have IRB certifications for conducting program level experiments. Internal experimentation involving human subjects by performers outside of program sponsored events will require additional IRB approvals by the performer. Q10: What will be the application/deployment scenarios? Will it be centralized or distributed? A10: A large scale, centralized, synthetic testing environment will be used on the scale of DISA. Q9: What are the target communications channels? A9: , text, social media and others. Q8: Is securing PII part of the research objective? A8: Yes. Q7: Will TA3 provide training data for machine learning in TA1 and TA2? A7: Yes. Q6: Do wireless communications and cybersecurity issues apply? A6: No. Q5: Are models of trust within scope? A5: Yes.

6 Q4: What are the elements required to identify an attacker for TA2? How will they be measured? A4: The goal is to discover the identity of the attacker. During experiments, attackers will have known dossiers. Performers will be measured based on how much of the dossier they can discover. Q3: Is it in scope for TA1 bots to elicit information from users? A3: Yes. Q2: Can I build my bots as browser based extensions? A2: No. Q1: Is fundamental R&D on algorithms for human in the loop machine assisted investigations within scope? A1: Yes.