Using Code Bloat to Obfuscate Evolved Network

Transcription

1 Using Code Bloat to Obfuscate Evolved Network Traffic Patrick LaRoche, Nur Zincir-Heywood, and Malcolm I. Heywood Faculty of Computer Science, Dalhousie University Halifax, Nova Scotia, Canada Abstract. In this work, we investigate the ability of genetic programming techniques to evolve valid network patterns, while avoiding detectability by obfuscating the intent of the traffic. In order to validate our system s capabilities, we choose to evolve a port scan attack while running the packets through an Intrusion Detection System (IDS). In turn, the evolutionary process uses feedback such that it minimizes the alarms raised while port scanning across a network range. Results build off of previous work allow us to further analyze and understand what the role of introns, code bloat, play in the systems ability to reduce the detectability of it malicious behaviour. 1 Introduction Vulnerability testing, a mechanism in which one attempts to discover weaknesses in a security mechanism, is done both on the part of the attackers and the protectors. Fuzzing, a form of vulnerability testing where one explores the boundaries and variations of a systems designed input parameters, allows the fuzzer to discover the weaknesses of the targeted system. In this case, the aim of the fuzzer is to analyze the vulnerability and attempt to remedy the situation before it becomes exploited. Such an exploration is inherently limited by the breadth and depth of the expert s knowledge in the system. In this work, we apply paged based Linear Genetic Programming (GP) [1, 2] techniques in order to discover new variants of known vulnerabilities. By harnessing the natural exploratory nature of GP techniques, our work aims to start with base domain knowledge, and allow the GP to explore outside the realm of what an expert might conceive. In the work described here, we examine how GP learns to take advantage of introns (code bloat) to hide its desired network behaviour, reducing detectability. Also of importance, we look at the competition during evolution between attacking and remaining undetectable. By employing GP techniques, we are allowing for the discovery of mechanisms in which one can obfuscate the real intent and avoid detection. Previous work has shown the validity of evolutionary techniques by exploring mimicry attacks at the system call level against host based IDSs [3]. The mimicry aspect is that the actual attack consists of not only the core sequence,

2 2 Patrick LaRoche, Nur Zincir-Heywood, and Malcolm I. Heywood but also functionality surrounding it, which is used to obfuscate that actual attack. In [4], authors implemented a preliminary work towards port scans. In this work, we explore attacks at the network level that require the generation of network packets. As a validation of our technique, we have chosen to mimic a port scan attack against a block of hosts and ports, which are protected by the Snort IDS [5]. We focus on analyzing how GP achieves the attack, not the framework of the attack itself, which was analyzed in [4]. To this end, the proposed system evolves proper TCP/IP packets, including header values, such that it not only succeeds in creating port scan attacks, but also remain stealthy and evades a well know IDS, Snort [5]. The rest of this paper is organized as follows; Section 2 and 3 describe background information on both the networking and evolutionary methods, respectively. Section 4 details the goals and the design of the experiments and presents their results. Finally, conclusions are drawn and future work is discussed in Section 5. 2 Background This paper explores the use of code bloat for vulnerability testing. To achieve this, we employe GP in order to evolve malicious network behaviour by evolving TCP/IP network packets. The resulting network traffic is then used to perform fuzzing at the transport level. This allows us to investigate the behaviour of the individual solutions, comparing to known attack behaviours. 2.1 Port Scans The prevalence of port scanning, a method many employ to ascertain their next victims [6] continues to be on the rise [7]. Due to this increase, and the direct correlation with network attacks [8], many detection mechanisms have been developed. The basic detection mechanism for port scanning logs the number of packets X sent to Y host ports in time T. This establishes a threshold that has a temporal dependency, counting the number of events in a specific time frame. Snort, which is an open source IDS, implements such a mechanism. One can classify port scans into 4 generic models, in all cases the attacker is attempting to detect whether a port is open, closed, or managed. In our work we allow the GP to evolve solutions that fit into any one of these models. A high level summary of these models is given below: 1. Basic - The attacker scans a series of ports sequentially, attempting a full TCP connection. 2. Random - The attacker behaves in the same manner as the basic attack, except randomizing the port numbers scanned. 3. Stealth SYN- Instead of trying to establish a full TCP connection, the attacker sends a TCP packet with the SYN flag set. This attack is more difficult to detect due to not initiating a full TCP connection, it simply starts one.

3 Using Code Bloat to Obfuscate Network Traffic 3 4. NULL, FIN, or Xmas scans- Similar to the stealth SYN scan, this attack does not initiate a TCP connection, sends packets with any combination of flags set, looking for RST packet to be returned (or not). 2.2 TCP/IP Packets The intent of a port scan is to determine whether a port is open, closed or filtered. As such, we have selected the following TCP/IP features in order to control source and destination addresses and ports, as well as what control bits are set (also known as flags). All remaining values are calculated and implemented in a manner such that the packet fits requirements of the corresponding RFCs, 791 and 793. We use; IP Addresses, Port # (both source and destination) and Control Bits. It is the above feature set, that GP evolves in creating packets. This provides the appropriate tool set in order to create useful packets for port scanning, but does not exclude the ability to create non-sensical packets as well. We recognize that some combinations of these features would result in a packet that would not make it to its destination, for this work we still consider this a valid packet. 2.3 Code Bloat Code bloat, the rapid growth of Genetic Programs, without relationship to fitness, is well documented in literature [9,10]. Much work has been done on solving this growth issue [11]. Typically code bloat, in the form of introns or repeated sequences of operations that do not contribute to the overall solution, are considered to be extraneous and trimmed out either during the evolutionary process, or after the final generation, as they result in a solution that performs the desired task, but in an inefficient manner. However, in the realm of malicious network traffic, one could argue that the ability to run code that slows down the processing of the solution individual, might in fact not be a negative, but a positive quality. In our work, we focus on packets with temporal malicious behaviour, so introns that lead to additional computational time could perhaps be a benefit for avoiding detection by obfuscating the actual intent. 3 Evolutionary Model Using the knowledge gained from past works [3, 12], authors have developed a system that allows the GP to create valid TCP/IP packets while responding to feedback from Snort [4]. As such, a multi-objective evolutionary model is defined to evolve individual solutions that create valid network packets, send said packets such that a port scan is completed, and avoid detection. These objectives are defined as follows: Solutions determine which ports are available on a range of hosts Solutions raise a minimal number of Snort Alarms

4 4 Patrick LaRoche, Nur Zincir-Heywood, and Malcolm I. Heywood These two competing objectives are then summed into a scalar value. Thus the fitness evaluation not only assists us to rank the evolved attacks relative to the core attack but also provides feedback in terms of Snort s reaction. 3.1 Instruction Set In selecting an instruction set, we must provide the mechanisms in which our system will be successful in recreating the core attack, but also be able to do so in a way that remains undetected. In our work, the attack consists of a series of packets. As described in Section 2, we are concerned with specific fields within a TCP/IP packet, as such we define an instruction set that allows our model to create valid TCP/IP packets, table 1 lists this set. Table 1. Instruction Set for GP Function Parameter 1 Parameter 2 set flags Decimal 0-15 N/A set ips Source IP Destination IP set ports Source Port Destination Port send packet N/A N/A An example of creating a packet would be to call each function call once, in any order, with the last function being send packet, lines 1-4 in Fig 3.1. Each function call is considered a tick, thus we do not give the model an explicit NOP (No OPeration) or wait function in which to use up time. However, it is plausible that a series of functions not involving a send packet could exist to use up ticks (obfuscation), effectively adding a temporal component. An example of a code bloat scenario, Fig. 1, lines 5 and 6 show how the second call overrides the first before sending the packet in line 7. If the intent of the individual was to simply send the packet, then the first system call would be trimmed from the individual. However, in our case, we wish to harness these introns such that they allow the individual to either use up time (by using up processor cycles without sending a packet) or send multiple packets to non target hosts, shown in lines This is a good example of how code bloat obfuscates the intent of the attacker (for this example, assume is the target, i.e the victim). 3.2 Fitness Evaluation In order to assign a fitness metric to our individuals performance, we need to define what a successful port scan is, as well as incorporating feedback from the

5 Figure 1 Example Sequence 1: set flags(0) 2: set ips( , ) 3: set ports(80,22) 4: send packet() 5: set ports(1024,20) 6: set ports(22,31) 7: send packet() 8: set ips( , ) 9: send packet() 10: set ips( , ) 11: send packet() 12: set ips( , ) 13: send packet() Using Code Bloat to Obfuscate Network Traffic 5 detector in order to minimize alarms raised. For the purpose of this work, we consider the scan to be successful if a specific destination host:port pair is found to be open during the scanning process. The specific details of which port and which host varies on the experiment, as described in Section 4. In order to provide useful feedback to the fitness evaluation we have developed a metric that defines how close IP addresses are to each other. Our assumption when assigning this metric is that one considers the proximity of two IP addresses based on what class network they are on, and how similar those networks are to each other. A IPv4 address consists of 4 octets expressed in decimal with period as the delimiter between them. This describes a 32 bit address space, allowing for 4, 294, 967, 296(2 32 ) possible network address. For our testing we evaluate the closeness by looking from the left most octet to the right and measuring the difference between each octet, favouring closeness in the left most octet (which represents the network address) versus the right most octet (which represents host address). Eq. 1 describes this, where the distance is between two IP addresses, A.B.C.D and a.b.c.d. The result of this distance metric is that GP is encouraged to find IP addresses that have open ports and are on the same network. We then use this metric in computing the fitness of an individual, that is to say that an individual is rewarded for finding targets close to actual targets. This encourages exploration of similar networks, allowing for eventual discovery of targets. DistanceIP = ((A a) 4) + ((B b) 3) + ((C c) 2) + (D d) (1) The second part of our fitness function, the feedback from the detector - Snort, is provided in real time as the solutions are evolving. The feedback is offered in the form of an alarm count, where the worst case scenario is the number of alarms greater than or equal to the number of packets sent during the scan. Similarly, the best case is zero alarms being triggered during the execution of an individual solution.

6 6 Patrick LaRoche, Nur Zincir-Heywood, and Malcolm I. Heywood Thus, fitness function is then described as the combination of two metrics (Eq. 1 and 2), with each section being described as a percentage. In this case, the first objective is maximizing the percentage of packets sent that correspond to a targeted open host:port pair, whereas the second objective is the inverse of the percentage of alarms raised by all packets sent, as shown in Eq. 2. F it = ( ) #T argetip : P orthit T otalt argetip : P ort + #Alarms 1 T otalp acketssent Where # Target IP: Port Hit describes the number of target host:port pairs detected and Total Target IP: Port is the total number of open host:port pairs that were available for detection during the training process. # Alarms is the number of alarms triggered by Snort during the execution of the individual, and Total Packets Sent represents the total number of complete packets sent. Note that the size of the individual is not the same as the number of packets sent, as it requires at least four function calls to send one packet. (2) 3.3 Evolutionary Model In order to achieve our goals we have implemented a page based linear GP. The basis for such a representation is a (virtual) register machine [1, 2, 13]. Such a machine consists of a predefined set of general purpose registers, support for the execution of an instruction set and the ability to supply input features describing the state of the environment at any given time step. Thus, individuals in the population of possible solutions are defined using a fixed length representation, with each individual representing a linearly sequential set of instruction calls. Selection of individuals between each generation is performed using a steady state tournament over 4 individuals. Search operators are applied to all in the tournament and the resulting two best performers replace the worst two. The search operators employed in this work are: Crossover, single point: A single page at a location (chosen with uniform probability) in one parent is interchanged with the page located at the same position in other parent. Swap Selector: Two instructions are selected in an individual and swapped, individual length remains fixed. Instruction-wise Mutation: For each instruction, test for application of mutation operator. If mutation is to be applied, another instruction from the complete instruction set is chosen with uniform probability and used as replacement. In this work, we have employed instruction-wise mutation. With the linear annealing schedule, the probability of mutation linearly decreases over the generation count, reducing it to zero for the last iteration. Table 2 lists the specific evolutionary model parameters used during our experiments.

7 Using Code Bloat to Obfuscate Network Traffic 7 Table 2. Parameters for Evolutionary Model Parameter Value Parameter Value Population 1000 Mutation 0.5 with linear decay Page Count 100 Swap 0.5 Page Size 6 Crossover 0.9 Tournament Size 4 Stop Criteria Tournaments 4 Experiments and Results In these experiments, we explore the relationship between the available search space and the size of the target list, where the search space consists of all possible IP addresses while the target list varies in size, building off of the previous work s success. The experiments consist of three separate target lists, only used as a validation to the fitness of an individual, versus the entire IPv4 range. The first experiment consists of 10 targets, the second 100 and the final experiment being 1000 hosts. All three experiments are run 30 times using a different initial seed. Thus, we run 90 experiments in total. Experiments are performed such that there is a machine originating the attacks (running the GP) and a machine acting as a monitor for traffic generated by the attack machine. Snort is is deployed on the monitor, seeing all the traffic. Snort was configured such that all packets received are monitored and inspected for malicious behaviour. As such this testbed mimics the attacker and the targeted machines in a controlled manner, allowing us to examine alarms triggered solely by our work. It should be noted here that we use the default Snort configuration, and set pre-processor specific variables for the most robust port scan detection possible. Specifically, the flow-portscan pre-processor was enabled, and the sfportscan preprocessor was set to the highest scan level available. To assign a metric value to the performance of our system we rank our fitness out of 100 (perfect). A perfect solution would be a solution that sent a packet to each target machine without triggering any alarms. Also of importance, in order to realize the usage of code bloat towards obfuscating network traffic. This is measured by comparing the number of function calls (total size) versus the number of packets sent to the network, thus showing what percentage of an individual results in a packet being sent. We list the best solution as well as the average of each target list in Table 3. Also of interest is the tradeoff between the two competing metrics we use in our fitness evaluation, namely the metric of being able to find targets (open ports on hosts) versus staying stealthy. Moreover, how this develops over the evolutionary process is also important to understand the effect of code bloat in our experiments. Fig 2 presents the two components (finding targets, remaining stealthy) of the fitness for all experiments. By examining the fitness evaluation of each objective (component of the fitness function) separately, we can see how they evolve over each generation. Thus, Fig. 2 shows that the exploratory target objective evolves slowly at a linear rate, meanwhile the stealthy objective

8 8 Patrick LaRoche, Nur Zincir-Heywood, and Malcolm I. Heywood Table 3. Results 10 Targets 100 Targets 1000 Targets Best Solution s Fitness Best Solution s Length Best Solution s Packets Mean Population Solution Fitness Mean Population Solution Length Mean Population Packets Send evolves exponentially. This difference in learning rates has led to two further experiments where we modified the fitness function and re-ran the 100 Target experiments again. In this case, the fitness function was altered such that in one set of experiments once the IDS fitness component reached 90% we started tracking the target fitness (results shown as as Alt #2). Alt #1 shows the other additional experiment in which the Target fitness was weighted more (2 to 3) then the IDS fitness to encourage more improvement on the seemingly more difficult objective. Average results of each experiment at a given generation count are shown in Fig 2. Fig. 2. Fitness over Generation Count In both additional experiments we see an improvement in the rate in which the target fitness is learnt, with the alternative version #1 showing an actual non linear improvement after approximately generations. Further work in this area needs to be performed, but some promise is shown. 4.1 Analysis The system implemented in this work needed to concur two problems concurrently, scan a number of unknown targets to find a small subset that had available

9 Using Code Bloat to Obfuscate Network Traffic 9 services while also remaining stealthy in its activity. The system used feedback from the IDS (Snort) as well as distance metrics to determine how close it was to finding the available targets. In analyzing our results the system does indeed learn to achieve this by achieving the best solution fitness values of 91.57%, 87.39% and 85.47% in the three sets of experiments, Table 3. In looking at the size of the solutions as well as the breakdown of the fitness metrics, we see that the system is learning which problem is easier to solve: (i) finding all the targets, or (ii) avoiding detection. In all three sets of experiments, we can see that the avoiding of detection is the simpler problem to solve, Fig 2. The apparent difficulty of the problem can also be seen in the exponential increase in fitness with regard to detection avoidance versus the slow linear growth of the fitness associated with finding the targets. We explored this further by implementing two additional sets of experiments where we modified the relationship between the two objectives, first by artificially capping the IDS fitness at 90% until the Target Fitness improves, then by weighting the Target Fitness more then the IDS fitness. In both experiments, we show that we can further improve our results to best solution fitness values of 92.8% and 93.2% respectively, Fig 2. As stated earlier, the underlying motivation of our work is to examine the GP s ability to obfuscate it s core network intent using code bloat. To this end, we observe that the GP focuses on sending many packets, to many sources, while using a smaller amount of genes to change other packet header values. This is also an appropriate method for avoiding detection, as the GP is sending many packets, to many sources, raising no flags in Snort (as they are not being sent all to a specific IP range or port addresses, Snort does not flag this as an attack). 5 Conclusion and Future Work We have demonstrated the ability for GP to generate network traffic, while adding complexity to the problem by increasing the search space to real-life scenarios. To achieve this, we modified the fitness evaluation of [4] by adding a distance metric between targeted IP addresses, as well as allowing the GP to explore the complete IP address space. We performed 90 experiments with the new fitness function, where the malicious user would locate a small, medium, and large number of targets during the scanning process while remaining stealthy in its activity. Then, the ability for GP is explored to learn how it obfuscates the network traffic through the use of code bloat. The resulting solutions did not use code bloat in methods we had expected. Instead, solutions were focusing on maximizing packets sent by an individual to a large set of targets, probably due to the larger search space required in these experiments. However, the solutions remain stealthy in their activities. On one hand it is not surprising that finding 10 target hosts out of the full IPv4 range is difficult, but on the other hand we did not expect the stealthy component to be relatively easier to solve. In future work, we plan to further explore this apparent disparity in order to improve our system while also applying it to further network activities.

10 10 Patrick LaRoche, Nur Zincir-Heywood, and Malcolm I. Heywood 6 Acknowledgment This research is sponsored by the granting agencies NSERC and MITACS as well as the industrial partners TARA Inc. and SwissComm Innovations SA. References 1. Heywood, M.I., Zincir-Heywood, A.N.: Dynamic page based crossover in linear genetic programming. IEEE Transactions on Systems, Man, and Cybernetics: Part B - Cybernetics 32(3) (2002) Nordin, P.: A compiling genetic programming system that directly manipulates the machine code. In Kinnear, Jr., K.E., ed.: Advances in Genetic Programming. MIT Press (1994) Kayacik, H.G., Heywood, M.I., Zincir-Heywood, A.N.: Evolving buffer overflow attacks with detector feedback. In: Applications of Evolutionary Computing, EvoWorkshops2007: EvoCOMNET, EvoFIN, EvoIASP, EvoInteraction, Evo- MUSART, EvoSTOC, EvoTransLog. Volume 4448 of LNCS., Valencia, Spain, EvoStar, Springer Verlag (11-13 April 2007) Patrick LaRoche, Nur Zincir-Heywood, M.H.: Evolving tcp/ip packets: A case study of port scans. In: Second IEEE Symposium on Computational Intelligence for Security and Defense Applications. (2009) 5. Snort.org: Snort ids (March 2009) 6. Ashfaq, A.B., Robert, M.J., Mumtaz, A., Ali, M.Q., Sajjad, A., Khayam, S.A.: A comparative evaluation of anomaly detectors under portscan attacks. In Lippmann, R., Kirda, E., Trachtenberg, A., eds.: RAID. Volume 5230 of Lecture Notes in Computer Science., Springer (2008) Symantec: Global Internet Security Threat Report trends for July December 07. Volume XIII. Symantec (2008) 8. Panjwani, S., Tan, S., Jarrin, K.M.: An experimental evaluation to determine if port scans are precursors to an attack. In: DSN 05: Proceedings of the 2005 International Conference on Dependable Systems and Networks, Washington, DC, USA, IEEE Computer Society (2005) Soule, T., Professor, M., Foster, J.A., Foster, J.A., Alves-foss, J., Frenzel, J.F., Frincke, D., Jacobsen, R.T., Shreeve, J.M.: Code growth in genetic programming (1998) 10. Koza, J.R.: Genetic programming as a means for programming computers by natural selection. Stat. Comput. (UK 4 (1994) Nordin, P., Francone, F., Banzhaf, W.: Explicitly defined introns and destructive crossover in genetic programming (1995) 12. Kayacik, H.G., Heywood, M., Zincir-Heywood, N.: On evolving buffer overflow attacks using genetic programming. In: GECCO 06: Proceedings of the 8th annual conference on Genetic and evolutionary computation, New York, NY, USA, ACM (2006) Huelsbergen, L.: Toward simulated evolution of machine language iteration. In Koza, J.R., Goldberg, D.E., Fogel, D.B., Riolo, R.L., eds.: Genetic Programming 1996: Proceedings of the First Annual Conference, Stanford University, CA, USA, MIT Press ( )