Detecting Denial of Service Intrusion Detection Aamir Islam Dept. of Computer Science, University of Central Punjab, Lahore, Pakistan.

Detecting Denial of Service Intrusion Detection Aamir Islam Dept. of Computer Science, University of Central Punjab, Lahore, Pakistan. aamir.islam@pcit.ucp.edu.pk Abstract Denial of Service (DoS) attack is an attack on a target system by a nasty system or user to render the normal services offered by the target system unavailable to legitimate users. DoS attacks are quite common and almost all servers are bound to be under a DoS attack at some time or the other. The most common and easiest to execute DoS attack is known as SYN Flooding. The mechanisms of detecting SYN Flooding DoS attacks come in vast variety. It ranges from very trivial techniques like netstat to more advanced mechanisms like signature based and anomaly based. Under signature based detection, an approach of detecting DoS attacks using time dependant deterministic finite automata [1] is discussed. Under anomaly based detection, a detection mechanism is discussed which is based on TCP SYN FIN (RST) pairs [2] and uses Cumulative Sum (CUSUM) method [3]. Finally a novel approach of making target system strong enough to be not effected badly by DoS attacks is presented. I. INTRODUCTION The SYN flooding attack abuse TCP/IP's three way handshake protocol. The three way handshake protocol is: 1. Client sends a Synchronization (SYN) Packet to the remote host. 2. Host replies with a Synchronization / Acknowledgement Packet (SYN/ACK) to the client. 3. Client replies with an ACK, acknowledging the packet sent by the host earlier. Only when the above 3 steps are completed, a complete TCP/IP connection is established between the source and the destination, known as a fully established connection. In a SYN flooding attack, several SYN packets are bombarded on to the server. These SYN packets have a fictitious source IP Address. When the target system receives these SYN Packets it responds to each one of them with a SYN / ACK packet. The target system now waits for an ACK message to come from the fictitious IP address. Since, the fictitious IP does not actually exist, the target system never receives the ACK packet. It therefore queues up all these requests until it receives an ACK message. The requests are not removed until the target system gets an ACK message, which results in consumption of target system s valuable resources such as memory. Thus the target system is unable to cater to requests for information made by legitimate users. In TCP/IP protocol, after a certain time has passed, time out takes place which discards the connection requests queued up by the target system and frees the memory. However, in a SYN flooding attack, the attacker keeps on sending connection requests from spoofed addresses at a rate faster than the time out of the earlier connection requests. As a result, even though due to time out the queued up connection requests are being discarded, the memory of the target system does not get freed up. In this paper, I present DoS attack detection mechanisms. Starting with the trivial approach, the more sophisticated Intrusion Detection approaches are discussed which are based on either Signature based or Anomaly based detection. The rest of the paper is organized as follows. In Section II, I present the Intrusion Detection Mechanisms. In Section III, the performance of the

Intrusion Detection Mechanisms identified in section II is discussed. In Section III, a novel approach of making a target system less vulnerable to DoS attacks is discussed. Finally in Section IV, I present some concluding remarks. 1. A Trivial Approach II. DoS Intrusion Detection In response to the request of client who wishes to establish connection, the target system sent the SYN ACK packet to the client and waits for an ACK from the client. The existing connection is said to be a half open connection and the host is said to be in the state of SYN_RECEIVED. It is very simple to detect whether the system is under SYN floods are not using this state. Merely type in the following Netstat command at the prompt: C:\windows>netstat Active Connections Proto Local Address Foreign Address State TCP aamir 201.xx.34.23 SYN_RECEIVED TCP aamir 197.xx.21.31 SYN_RECEIVED TCP aamir 1.xx.91.66 SYN_RECEIVED TCP aamir 151.xx.45.0 SYN_RECEIVED TCP aamir 187.xx.71.98 ESTABLISHED TCP aamir 197.xx.11.41 SYN_RECEIVED TCP aamir *:* SYN_RECEIVED TCP aamir *:* ESTABLISHED. On execution, if the above command shows a lot of connections in the SYN_RECEIVED state, it is highly probable that system is under a SYN Flood Attack. As can be seen in above output, some connections are in the Established state. They represent legitimate connections, which remain unaffected even after the SYN flood attack on the target system. 2. A Signature Based Approach A signature based approach relies on its database of attack signatures to identify a DoS attack. If the signature in database matches with the signature of live traffic, system is under Dos attack. In this case, alarm is triggered. But if signature of attack is not in the database, this approach will fail to detect the attack. The Time Dependant Deterministic Finite Automata (TDDFA) is based on this approach. Before discussing the architecture of TDDFA, I will be discussing some basics of TDDFA and how it can be used to represent DoS attacks. Finally conclusion is drawn by using the results of [1]. 2.1 Basics Deterministic Finite Automata (DFA) is an abstract model of a computer. They are used to identify regular languages. DFA have finite number of states represented by circles. A transition between two states occurs on consuming an input and is represented by unidirectional arrows. Final state(s), represented by doubled circle, is reached when entire input string is accepted. Figure 1 shows an example DFA which accepts input of the alphabet {a,b} containing ab. b a,b a a b q 0 q 1 q 2 Figure 1: An Example DFA

TDDFA is analogous to DFA except it considers the time interval between inputs in recognizing member of language. An example of TDDFA is shown in figure 2 which recognize the pattern a,b<4 i.e. b must occur within 4 seconds of a. q 0 b a a b<4 q 1 q 2 b a The very nature of TTDFA makes it an ideal choice in representing DoS attacks and recognizing them. DoS attacks are packets that make a system inoperable. TDDFA states can represent incremental conditions of a system as it reaches a state of intrusion and arcs represents packets. 2.2 TDDFA Architecture Figure 2: An Example TDDFA TDDFA consists of four modules: (1) Data Filtration Unit, (2) Event Token Generator, (3) TDDFA Transversal Unit and (4) TDDFA Provider. TDDFA interfaces with external components, Local Area Network (LAN), Stored Network traffic data and a Client machine. Local Area Network (LAN) and Stored Network traffic data are the sources for generating live traffic data and traffic data in offline mode respectively. Figure 3 represents the entire architecture. Data Filtration Unit (DFU) filters the relevant data from network packet and leaves the remaining. Data parsed by DFU contains following fields: Packet type, Source and Destination IP addresses, Destination Port, More Fragment flag, Timestamp, SYN and ACK flags, Echo request and Echo reply. Event Token Generator (ETG) translates the DFU text into special tokens. One DFU text can generate one to many corresponding tokens. Tokens are string of one or more ASCII characters and combine to create a language used by TDDFA to detect DoS attack. Table 1 below highlights some tokens that compose a language for TDDFA. Token(s) e S F Definition used for UDP storm attack recognition Packet s SYN flag is checked Packet s MF flag is checked Table 1: Sample ETG tokens TDDFA Transversal Unit (TTU) is the attack detection unit which detects various DoS attacks such as Land, SYN Flood, Ping Flood, Smurf, Teardrop and UDP storm. ETG exhibits indication of probable DoS attack in the form of tokens and then TTU reads the token and identify that whether the host is under attack or not. ETG tokens served as input characters to traverse TTDFA and if TTU detect that TTDFA has reached the final state, it alerts about the attack.

LAN TDDFA Provider Client DFU ETG TTU Stored NW Data TTDFA Provider will supply TDDFA with all other transitions that are required because client specify only the attack signature, i.e. states and transitions that lead to attack only. It also provides TTU with user defined TDDFA. 2.2 Performance Evaluation Figure 3: TDDFA Architecture For carrying out tests to evaluate TDDFA, data is taken from [4]. The third and fourth columns of table 2 represent the timestamp when the attack is recognized by the dataset and TDDFA respectively. The only discrepancy in data lies in SYN Flooding attacks in week 4 and 6 of 1998 which are shaded. This indicates the possible weakness of TDDFA model not to symbolize most variations of attack signatures. Dataset Attack MIT Time TDDFA Time SYN Flood 11:55:38 08:50:15 Ping Flood 20:11:31 20:11:31 Teardrop 23:15:10 23:15:10 Teardrop 08:15:02 08:15:02 Smurf 12:53:15 12:53:15 Smurf 15:33:28 15:33:28 SYN Flood 17:27:07 17:27:07 Smurf 18:00:15 18:00:17 Ping Flood 13:04:56 13:04:56 Land 17:53:49 17:53:49 Teardrop 08:32:12 08:32:12 SYN Flood 09:31:52 No Smurf 19:12:37 19:16:27 Ping Flood 08:50:15 08:50:15 Land 15:57:15 15:57:15 SYN Flood 11:04:16 11:04:16 Land 15:47:15 15:47:15 Ping Flood 09:18:15 09:18:15 SYN Flood 11:20:15 11:20:15 Table 2: Test Results

3. An Anomaly Based Approach An Anomaly based approach is based on the profile of normal network traffic which is created during the tuning period. Once the profile is created, anything detected outside the profile is reckoned as a threat and is usually brought to the attention of a system administrator by triggering an alarm. This would include any event, state, content, or behavior that is considered to be abnormal by a pre defined standard. Anything that deviates from this baseline of normal behavior will be flagged and logged as anomalous. 3.1 Basics A Flooding Detection System (FDS) is an anomaly based mechanism which detects the SYN flooding attacks. The simplicity of FDS lies in statelessness, not dependent on time and site, and low computation over head. FDS uses the beginning (SYN TCP packet) and end (FIN TCP packet) for flooding detection. As shown in the figure 4, one SYN (SYN/ACK) packet results in a one FIN (FIN) packet in the normal condition. Also the reset (RST TCP packet) generates the FIN packet. The RST TCP packet are generated for two reasons i.e. one RST passive is transmitted in response to the arrival of packet at the closed port and second RST active is transmitted to abort the TCP connection. Each RST active is associated with the SYN while each RST passive is not associated with the SYN which causes the violation in the SYN-FIN pairs. So the three types of SYN pairs are considered in FDS i.e. (SYN, FIN), (SYN/ACK, FIN) and (SYN, RST active ). FDS cannot differentiate between active and passive RST and considers 3 out of 4 packets as an active RST. Client Socket, Connect (blocks) (active open) SYN_SENT ESTABLISHED Connect returns write read (blocks) read returns SYN j, mss SYN K, ack j+1, mss ack Data Data (reply) + ack of ack of Server Socket, bind, listen (passive open) accept( blocks) SYN_RCVD ESTABLISHED accept returns read (blocks) read returns write read (blocks) Close (active close) FIN_WAIT_1 FIN_WAIT_2 TIME_WAIT FIN ack FIN ack CLOSE_WAIT (passive close) read returns 0 Close LAST_ACK CLOSED Figure 4: TCP States

The FDS is deployed at the leaf router i.e. first-mile leaf router where the traffic is coming from intranet to internet (outbound interface) or last-mile leaf router where the traffic is coming from internet to intranet (inbound interface). Leaf router at the same time can be a first-mile leaf router and last-mile leaf router depending upon the direction of the flow of traffic on the network as shown in figure 5. FDS are installed at each leaf router in case if first-mile leaf router is different from the last-mile leaf router and they coordinate with each other via shared memory [2]. Figure 5: FDS at leaf router 3.2 Attack Detection A multi-layer IPSec protocols has been proposed in which only trusted routers are allowed to access the Transport layer header. By accessing the transport layer header we can easily differentiate between the control packets and data packets as shown in figure 6. Three variables are introduced to count the number of SYN (SYN, SYN/ACK), FIN and RST at routers (both inbound and outbound), which stores the number of SYN, FIN and RST packet during the every observation period. The strong positive correlation between the number of SYN and FIN packets offers the clear indication for SYN flooding [2]. No Protocol is TCP? Yes Non TCP No Fragment Offset is 0? Yes No TCP header in payload No Get IP header length. Compute offset of Code BITs No Is SYN/ FIN/RST Flag On? Yes TCP data SYN/FIN/ RST Figure 6: Packet Classification Mechanism at Leaf Router

3.3 Statistical Analysis. Let t 0 be observation time to collect the number of the SYN, FIN (RST) packet. The sampling time of FIN (RST) is t d later then that of SYN. The t 0 is chosen to 20 seconds and t d to 10 seconds. Under the normal condition the difference between the number of SYN and FIN (RST) remains very small. Under the SYN flooding attack, it was observed that the number of SYN request receives at the rate of 500 SYN per sec to 14000 SYN per sec. So in order to shut down the victim server attacker has to generate the 300,000 SYN packets in around 10 minutes. While during this time the number of FIN remain unchanged so the difference between the SYN and FIN (RST) will increase significantly. So the larger difference of SYN and FIN (RST) indicates the SYN flooding attack [2]. 3.4 Cumulative Sum (CuSum) Algorithm The detection algorithm is based on the sequential change point detection [3]. Let n be the difference in the number of the SYN s and the corresponding FIN s (RST). n is dependent upon the time, site and the access pattern. In order to normalize n by average number of FIN S (RST) during the sampling period t 0 F (n) = F (n-1) + (1- ) FIN (RST) (n) where n is the discrete time and lies between 0 to 1. Define the mean as X n = n / F (n) Now the X n is independent upon the time, site and the access pattern. In order to model X n apply the non parametric cumulative sum (CUSUM) method. Chose parameter a as an upper bound on the mean value and define X n X n = X n a The X n is small negative mean during the normal condition and large positive mean during the attack. During an attack, the increase in the mean of X n can be lower-bounded by h. Our change detection is based on the observation that h > c. Define another term S k and y n as S k yn = k i = 1 X i = Sn min 1 k n S k Let D n be the decision time equal to 0 at normal condition and 1 at the attack condition. Then D n (y n ) = 0 if yn <= N 1 if yn > N The two design parameters are: a the upper bound and N flooding threshold.

3.5 Performance Evaluation The FDS is tested on the various network traces under the normal condition as discussed in [2], which clearly indicates the synchronization between the SYN and FIN (RST). There are two fundamental performance measures for the sequential change point detection: The time duration with no false alarm reported when there is no attack. The detection delay after attack starts Under the normal condition CUSUM algorithm is applied on the traces with the flooding threshold of 0.6 for the first mile, flooding threshold of 1 for the last mile and of a = 1 shows y n = 0 as shown in figure 6 (a) of [2]. In most of the cases y n is much smaller then N. So, no false alarms are reported. Under the attack condition, the flooding traffic is induced at the rate of 500 SYN / sec showing the cumulative sum y n exceeds the flooding threshold 1, thus reporting the false alarm in 20 sec as shown in figure 7 (c) of [2]. III. Minimizing SYN flooding effect Assuming that rather then detecting the denial of service attack we equip the machine with some mechanism to minimize the effect of the denial of service attack. In other words, we want to make the machine strong enough that in case DoS attacks occur, minimum performance deterioration occurs. If any machine is under the denial of service attack then the queue is filled up with the incoming SYN requests from malicious user as well as legitimate user. This will continue until the queue is filled up. Assume Q is the length of the queue and T is timeout before the TCP start dequeuing the SYN requests. Then Q / T (request/sec) is the maximum rate at which the machine can cope with the denial of service attack. If this ratio (Q/T) is greater then the rate of incoming SYN requests, minimum effect is on the machine. In other words, Q/T > R where R is the rate of incoming SYN requests, this can be written as Where > 1 Q / T = R I propose the following algorithm to minimize the effect on machine under DoS attack: 1. Find Ratio Q/T using the well known DOS attack data sets. 2. Find the most appropriate time of receiving SYN requests acknowledgement using the history of the normal connections. 3. Using the above time as a timeout (T) in ratio Q/T will give us the optimal queue length. 4. Set the queue length of service running at the machine equal to the optimal queue length.

IV. Conclusions Instruction Detection Mechanisms are still in infancy. The results of TDDFA are quite pleasing when compared with already calculated attack detection time based on datasets in [4]. The table 2 shows the results. However, lack of SYN Flood attack detection in two instances is of concern. The most probable reason for this could be inability of TDDFA model to detect variations on attack signatures. Also, since TDDFA technique is signature based, it inherits all the flaws of signature based approach. The most significant being the ability to detect attacks of known signatures only. But when compared with statistical anomaly based algorithms like adaptive threshold [3], the rate of raising false alarms is considerably low which results in higher performance. The inability of TTDFA approach to detect SYN flooding attacks and shortcoming in signature based Intrusion detection leads me to investigate anomaly based Intrusion detection. The scheme discussed under this category is FDS. The distinguishing features of FDS are that it does not undermine the end to end TCP performance because it doesn t intercepts the TCP traffic between client and the server and does not maintain any state for each TCP connection. Also, it does not require any IP trace back to detect the source of the attack inside the stub network due to proximity of first mile FDS to the flooding sources. This result in substantial lowering of overhead incurred in tracing the source. The detection time also reduces under the high intensity attacks. REFERENCES [1] Joel W. Branch, Alan Bivens, Chi Yu Chan, Taek Kyeun Lee and Boleslaw K. Szymanski. Denial of service Intrusion Detection Using Time Dependant Deterministic Finite Automata. 2002 [2] H. Wang, D. Zhang and K.G. Shin. Detecting SYN flooding attacks. In Proc. Of IEEE INFOCOM 02, 2002. [3] Vasilios A. Siris and Fotini Papagalou. Application of Anomaly Detection Algorithms for Detecting SYN Flooding Attacks. 2003. [4] MIT Lincoln Laboratory. DARPA Intrusion Detection Evaluation. http://www.ll.mit.edu/ist/ideval/, 1999.