Valkyrie Detectors Tutorial

Transcription

1 Valkyrie Detectors Tutorial

2 Contents 1. Valkyrie Academia Precise Detectors Precise Detector Types Supported File Types Custom Precise Detector Implementation Detector Language Templates Method Definitions Static Python Detector Static C/C++ Detector Dynamic Python Detector Dynamic C/C++ Detector Hybrid Python Detector Hybrid C/C++ Detector Submission to Valkyrie Academia Constraints Template Detector Files Python Detector Template C++ Detector Template Example Precise Detectors Example Python Detector Example C++ Detector Appendix A: Dynamic Analysis Behaviors Data Appendix B: Dynamic Analysis Dropped Files Data Appendix C: Dynamic Analysis Network Data... 8 ALL RIGHTS RESERVED. 2 / 8

3 1. Valkyrie Academia Valkyrie Academia [1] is an extended version of original Valkyrie [2] service for academicians and community researching malware analysis and different detection techniques. Valkyrie is a cloud based file analysis and classification service, designed with modular analysis services architecture (See Figure- 1), conducting several static and dynamic file analysis. Valkyrie Academia offers researches to integrate their own implemented detection techniques in to the analysis flow and run for each file analysis as Precise Detectors. This document explains, what is a Precise Detector, types of precise detectors, how to implement and submit a new detector, best practices for implementation and constraints. Figure 1 - Analysis Flow 2. Precise Detectors Precise Detector is an implementation of a particular malware detection technique that runs on each file analysis in Valkyrie. Valkyrie provides different data to each detector according to its type, and provides a template class in Python and C++ languages. Following section describes different types of precise detectors, data provided to each one and supported file types in latest Valkyrie Academia version Precise Detector Types There are 3 different detector types that researches can implement and add in to Valkyrie Academia analysis flow, namely: Static Type: This type of detector operates on file's data. Valkyrie provides file's full path that to this detector. Method to be implemented for these type of detectors. Dynamic Type: This type of detector operates on data generated by Dynamic Analysis Hybrid Type: This type of detector uses both data generated by Dynamic Analysis and file's data ALL RIGHTS RESERVED. 3 / 8

4 Each detector type fits in to proper place in analysis flow and is executed when required input data is generated by the system. Implementation details of each type are explained in next sections Supported File Types In current version, only detectors for PE 32/64 file types are supported in Valkyrie Academia. 3. Custom Precise Detector Implementation 3.1. Detector Language There are two different programming language supported by the system, Python / C++: Using python template and implementing related method Using C++ template and implementing related method 3.2. Templates In order to implement own detection technique as precise detector, a researcher must implement one of the related methods in provided detector.py template script or detector.cpp template, as explained in Section 3.3. Also template folders are provided to researchers for both of the language types in compressed format. Once archive file extracted, you will find a main folder called GPD_python/GPD_c_cpp that has template files/folders to be filled by detector owners. Content of this GPD_python/GPD_c_cpp folders are as follows: Python Detector (Provided Archive File: GPD_python.zip) data (Folder): Detector owner may put any data file that detector needs in this folder. detector_codes (Folder): This folder holds all of the custom python scripts and/or sub-folders used by researchers. python_requirements.txt: This is the Python's Virtual Environment dependencies. We will install these via pip and run detector.py using this Virtual Environment. os_requirements.txt: This is the additional Operating System (Ubuntu 16.04) dependencies. We will install these via sudo rights. readme.txt: This is information file that Detector owner provides detector related data. C/C++.so Detector (Provided Archive File: GPD_c_cpp.zip) data (Folder): Detector owner may put any data file that detector needs in this folder. os_requirements.txt: This is the additional Operating System (Ubuntu 16.04) dependencies. We will install these via sudo rights. readme.txt: This is information file that Detector owner provides detector related data Method Definitions Static Python Detector def on_static(self, sha1, filename, sample_filepath): :param sha1: SHA1 of the file, String :param filename: filename seen on client, String :param sample_filepath: full file path that you can read binary data using open, String :return: you must call self.form_json() and provide related parameters ALL RIGHTS RESERVED. 4 / 8

5 ############## IMPLEMENT THIS METHOD DETECTOR TYPE IS STATIC ################ # If your detector type is Static, we will call this method in each analysis ##################################################################### Static C/C++ Detector char * on_static (char *sha1, char *filename, char *filepath) { /* :param sha1: SHA1 of the file, char * :param filename: filename seen on client, char * :param filepath: full file path that you can read binary data using open, char * :return: you must return a string with following format: "{\"verdict\":1, \"sha1\":\"<sha1>\",\"reason\":\"detected the XXX Virus!\"}" */ /* IMPLEMENT THIS METHOD DETECTOR TYPE IS STATIC*/ /* If your detector type is Static, we will call this method in each analysis*/ } Dynamic Python Detector def on_dynamic(self, sha1, filename, all_behaviors_filepath, dropped_files_filepath, network_activity_filepath): :param sha1: SHA1 of the file, String :param filename: filename seen on client, String :param all_behaviors_ filepath: full file path of the behaviors collected in dynamic analysis of the file that you can read using open, String (See: Appendix A) :param dropped_files_filepath: full file path of dropped files information during the analysis that you can read using open, String (See: Appendix B) :param network_activity_filepath: full file path of the network activity of the file that you can read using open, String (See: Appendix C) :return: you must call self.form_json() and provide related parameters ############## IMPLEMENT THIS METHOD DETECTOR TYPE IS DYNAMIC ################ # If your detector type is Dynamic, we will call this method in each analysis when we perform our Dynamic analysis # and generate required data for your detector ##################################################################### Dynamic C/C++ Detector char * on_dynamic char * sha1, char * filename, char * all_behaviors_ filepath, char * dropped_files_filepath, char * network_activity_filepath) { ALL RIGHTS RESERVED. 5 / 8

6 /* :param sha1: SHA1 of the file, char * :param filename: filename seen on client, char * :param all_behaviors_ filepath: full file path of the behavior summary of the file that you can read using open, char * (See: Appendix A) :param dropped_files_filepath: full file path of dropped files information during the analysis that you can read using open, char * (See: Appendix B) :param network_activity_filepath: full file path of the network activity of the file that you can read using open, char * (See: Appendix C) :return: you must return a string with following format: "{\"verdict\":1, \"sha1\":\"<sha1>\",\"reason\":\"detected the XXX Virus!\"}" */ /* IMPLEMENT THIS METHOD DETECTOR TYPE IS DYNAMIC */ /* If your detector type is Dynamic, we will call this method in each analysis when we perform our Dynamic analysis and generate required data for your detector*/ } Hybrid Python Detector def on_hybrid(self, sha1, filename, sample_filepath, all_behaviors_ filepath, dropped_files_filepath, network_activity_filepath): :param sha1: SHA1 of the file, String :param filename: filename seen on client, String :param sample_filepath: full file path that you can read binary data using open, String :param all_behaviors_ filepath: full file path of the behaviors collected in dynamic analysis of the file that you can read using open, String (See: Appendix A) :param dropped_files_filepath: full file path of dropped files information during the analysis that you can read using open, String (See: Appendix B) :param network_activity_filepath: full file path of the network activity of the file that you can read using open, String (See: Appendix C) :return: you must call self.form_json() and provide related parameters ############## IMPLEMENT THIS METHOD DETECTOR TYPE IS DYNAMIC ################ # If your detector type is Dynamic, we will call this method in each analysis when we perform our Dynamic analysis # and generate required data for your detector ##################################################################### Hybrid C/C++ Detector char * on_hybrid char * sha1, char * filename, char *filepath, char * all_behaviors_ filepath, char * dropped_files_filepath, char * network_activity_filepath) { /* :param sha1: SHA1 of the file, char * :param filename: filename seen on client, char * :param filepath: full file path that you can read binary data using open, char * ALL RIGHTS RESERVED. 6 / 8

7 :param all_behaviors_ filepath: full file path of the behavior summary of the file that you can read using open, char * (See: Appendix A) :param dropped_files_filepath: full file path of dropped files information during the analysis that you can read using open, char * (See: Appendix B) :param network_activity_filepath: full file path of the network activity of the file that you can read using open, char * (See: Appendix C) :return: you must return a string with following format: "{\"verdict\":1, \"sha1\":\"<sha1>\",\"reason\":\"detected the XXX Virus!\"}" */ /* IMPLEMENT THIS METHOD DETECTOR TYPE IS DYNAMIC */ /* If your detector type is Dynamic, we will call this method in each analysis when we perform our Dynamic analysis and generate required data for your detector*/ } 3.4. Submission to Valkyrie Academia If detector owner decides to write a detector with python, the following steps should be followed: 1. Related detector type method should be implemented in template detector.py file in detector_codes directory. If there are any extra python script they should also be in detector_codes directory and imports should be in either in same directory or satisfied by requirements. 2. python_requirements.txt and os_requirements.txt files should be filled if necessary. 3. Fields in readme.txt should be filled. 4. Any extra data files, models, etc. should be put in data folder. 5. These files and folders should be zipped with the name GPD_python and be sent to the Valkyrie Academia team. If detector owner decides to write a detector with C/C++, the following steps should be followed: 1. Related detector type method should be implemented in detector.cpp file in detector_codes directory. If there are any extra C++ codes as dependencies, they should also be in detector_codes directory and imports should be in either in same directory or satisfied by requirements. 2. os_requirements.txt files should be filled if necessary. 3. Fields in readme.txt should be filled. 4. Any extra data files, models, etc. should be put in data folder. 5. These files and folders should be zipped with the name GPD_c_cpp and be sent to the Valkyrie Academia team. 4. Constraints Following constraints apply for each detector: No write / execute permission for detector will be given. data folder max. size: 250 MB. max. size of other files and folders except data: 50 MB ALL RIGHTS RESERVED. 7 / 8

8 Each detector has 30 seconds time limit to process given data and provide an output. 5. Template Detector Files 5.1. Python Detector Template See detector.py for python detector template. This is the file mostly researches will use to implement their techniques C++ Detector Template See detector.cpp for C++ detector template. This is the file mostly researches will use to implement their techniques. 6. Example Precise Detectors 6.1. Example Python Detector See static_python_detector_example.py for a dummy python detector written by Valkyrie Academia team to given an insight about python detectors Example C++ Detector See static_cpp_detector_example.cpp for a dummy C++ detector written by Valkyrie Academia team to given an insight about C++ detectors. 7. Appendix A: Dynamic Analysis Behaviors Data See behavior.json file and its content for this dynamic analysis data given to dynamic and hybrid type detectors. 8. Appendix B: Dynamic Analysis Dropped Files Data See dropped.json file and its content for this dynamic analysis data given to dynamic and hybrid type detectors. 9. Appendix C: Dynamic Analysis Network Data See network.json file and its content for this dynamic analysis data given to dynamic and hybrid type detectors. [1] [2] ALL RIGHTS RESERVED. 8 / 8