An Analysis of Botnet Attack for SMTP Server using Software Define Network

Size: px

Start display at page:

Download "An Analysis of Botnet Attack for SMTP Server using Software Define Network"

Kelley York
6 years ago
Views:

1 RHUL workshop on February 29 An Analysis of Botnet Attack for SMTP Server using Software Define Network Mohd Zafran (PhD Candidate) & Koji Okamura Graduate School of Information Science and Electrical Engineering Kyushu University Kyushu University, Japan 15/4/2016 1

2 Introduction (Problem Statement & Research Proposal) Related Works Methodology Experiment Setup / Simulation /Result Kyushu University, Japan 15/4/2016 2

3 What is Software Define Network?

Distributed Control Previous/Current Network Centralized Control Current/Future Network SDN Controller Switch Switch Switch Switch Control plane Control

4 Distributed Control Previous/Current Network Centralized Control Current/Future Network SDN Controller Switch Switch Switch Switch Control plane Control plane Data Plane Data Plane Data Plane Data Plane Switch Switch Switch OpenFlow Switch Control plane Control plane Data Plane Data Plane Data Plane Data Plane

5 A distributed denial-of-service (DDoS) Botnet attack on SMTP server Botnet Botnet Mail transfer Agent Fig. 1 Botnet Attack using syn flood attack technique scenario 15/4/2016 Kyushu University, Japan 5

6 Problem Statement Botnet attack will consume all resource such as cpu, network and storage. These attack also term as Distributed Denial of Services (Ddos) attacks as the flood traffic comes from many machines, and is not a single flow on the network.when an attack target host upstreams network bandwith,these attack also named as bandwith attack The bigger network bandwidth, different IDS and IPS capacity need to be use Kyushu University, Japan 15/4/ Fig. 2 Intrusion Detection System & Intrusion Prevention System

7 Introduction: 1.0 The proposed approach By using SDN Technology at multi domain, SDN Control can detect the spam botnet flow before the botnet arrive to destination ip. Existing spam filtering database such as spamhaus and spamcop, can be integrate by develop new app at SDN CTRL layer to retrieve the information about spam botnet source blacklisted IP and feed new information about botnet IP source blacklisted. By having the information on botnet blacklisting source IP. The early mitigation on botnet can be done. Flows can be specified using any or a combination the following ten tuples, match fields:in Port, VLAN-ID, Source MAC, Destination MAC, Ethernet Type, Source IP, Destination IP, Protocol, Source Port, Destination Port By using 10 tuples field be use to create a new algorithm to detect the flow of botnet. Kyushu University, Japan 15/4/2016 7

8 1.1 Botnet attack scenario SDN Domain A SDN Domain Controlller Spam Haus Server SDN Domain Controlller WAN WAN SDN Domain Controlller SDN Domain B SMTP server A SDN Domain C WAN SMTP server B Fig. 3 Botnet attack from two domain Kyushu University, Japan 15/4/ SMTP server C

9 2.0 Related Works 1.Method to detect the Botnet attack to smtp server : An approach detecting a flooding Attacks Based on Entropy measurement of Multiple Protocols 2.Method to communicate between Multi Domain using SDN platform: DISCO: Distributed Multi-domain SDN Controllers 3. Study of spam characteristics on network layer : Study of spam characteristics on network layer A large-scale empirical analysis of spam detection through network characteristics in a stand-alone enterprise Kyushu University, Japan 15/4/2016 9

10 Related Works: 2.1 Several protocol protocol SMTP (Simple Mail Transfer Protocol) POP3 (Post Office Protocol Version 3) IMAP (Internet Message Access Protocol) File Server File Server Fig 4. SMTP message flows Kyushu University, Japan 15/4/

11 2.1.1 Recap on SMTP Protocol SMTP Server Connection Establishment 1.. (Client) --> [SYN] >(Server) 2.. (Client) <-- [SYN/ACK] <--(Server) 3.. (Client) --> [ACK] >(Server) Connection Termination 1.. (Client) --> ACK/FIN ---->(Server) 2.. (Client) <-- ACK < (Server) 3.. (Client) <-- ACK/FIN <----(Server) 4.. (Client) --> ACK >(Server) Fig 5. SMTP message flows Fig 6. TCP flows Kyushu University, Japan

12 2.2 Objective 1.Design the mechanism of SDN Multi Domain for detecting the Botnet Attack based on attack on smtp server 2.Performance Analysis to detect the Botnet Attack that attack on smtp server 3. Comparization Analysis study with other related works Kyushu University, Japan 15/4/

13 Methodology: 3.0 Design mechanism of SDN Every Domain SDN Controller Sending information about flow count /flow size and packet size Specific on port number & Destination IP to Main SDN controller SMTP Server Main SDN Controller SpamHaus server Feed information to spamhaus Decision for identify botnet attack 15/4/2016 Install the domain with blacklist ip Kyushu University, Japan 17

3.1 The flowchart mechanism of SDN New flow entry coming at Domain

Main SDN Controller SpamHaus server Send flow entry match

Domain Permit the flow message and forward the packet to next node

Algorithm Kyushu University, Japan Drop the next packet from the

14 3.1 The flowchart mechanism of SDN New flow entry coming at Domain R1,R2, R3 Check src ip (blacklist) yes Drop packet No SMTP Server Main SDN Controller SpamHaus server Send flow entry match information (TCP /UDP 25/110) DST IP to SDN controller in every Domain Permit the flow message and forward the packet to next node NO Controller check the Botnet Attacks Based on Decision Tree Algorithm Kyushu University, Japan Drop the next packet from the same ip src flow message update information blacklist ip to spamhaus server Yes 18 15/4/2016

15 3.2 Retrieve flow information before arrive at targeted Domain Time stamp Flow entry Ip src Ip dst SDN Domain A SDN Domain A Controlller Time stamp Flow exit Ip src Ip dst Spam Haus Server SDN Domain B Controlller WAN WAN SDN Domain C Controlller SDN Domain B SMTP server A SDN Domain C WAN SMTP server B Fig. 7 Botnet attack from domain A Kyushu University, Japan 15/4/ SMTP server C

3.3 Early Botnet Attack detection close to smtp server attack target on multi Domain using SDN technology Scenario: Assume that there 1 protocols serving for

16 3.3 Early Botnet Attack detection close to smtp server attack target on multi Domain using SDN technology Scenario: Assume that there 1 protocols serving for smtp Server are monitored at 4 different periods, where the time-period series is listed as : Fig. 8 Botnet attack from domain A Kyushu University, Japan 15/4/

17 3.4.1 Related works on study characteristics smtp flow and packet on smtp flood or syn flood on smtp server :T. Ouyang, S. Ray, M. Allman, and M. Rabinovich, A largescale empirical analysis of spam detection through network characteristics in a stand-alone enterprise, Comput. Networks, vol. 59, pp , 2014 Network Layer Application Layer Content blind techniques Fig.9 The proses flow to filter the spam from network layer until application layer Kyushu University, Japan 15/4/

18 3.4.2 SMTP Network Traffic analysis technic :T. Ouyang, S. Ray, M. Allman, and M. Rabinovich, A largescale empirical analysis of spam detection through network characteristics in a stand-alone enterprise, Comput. Networks, vol. 59, pp , 2014 Dataset May 2009 to April 2011 BRO Spamflow,Bro and p0f Network traffic characteristics Decision Tree Algorithm using Weka tool Packet & Flow features Fig. 10 Process SMTP network traffic analysis technic Kyushu University, Japan 15/4/

19 3.4.3 Related works: T. Ouyang, S. Ray, M. Allman, and M. Rabinovich, A large-scale empirical analysis of spam detection through network characteristics in a stand-alone enterprise, Comput. Networks, vol. 59, pp , 2014 Machine Learning Algorithm Decision trees (using Weka Tool) Create Root nodes (fins_local) Create Decision Nodes 3whs GeoDistance Create Leaf Nodes Ham Spam Fig. 11 Decision trees process Fig. 12 Fragment of tree using packet + flow features Kyushu University, Japan 15/4/

20 3.5 Decision Tree Algorithm Rtt_C_S <= 0.03s T F 3whs<=0.045 RTO_s_c<= 2.2s T F T F Ham fgnr_ttl<=98 Ham Spam T F Ham Spam Symbols Ham = Legitimate Spam = Spam Rtt_C_S = RTT packet in Switch Flow Table Client<-> Server 3whs = Flow duration between the arrival of SYN from Client and Flow Duration Ack of Syn/ACK by Server Fngr_ttl = time to live packet client, if more 98 will be windows platform RTO_s_c = Retransmission timeout from server to client in second Fig. 13 Fragment of tree using packet + flow features Kyushu University, Japan 15/4/

21 3.5.1 PSEUDOCODE If dst port= 25 Then Forward to controller Packet_in Flow count go to module 1 Else drop the packet Module 1 (RTT Client between Server) Module 2 (3 way hand shake flow count and time) If rtt client <-> server between two switch t>= s Then go to module 2 Else go to module 3 If flow count packet_in = 2,same src ip same dist ip,time arrival for 2 nd flow <= for between client <-> server Then install the flow in flow table, forward the next packet Else go to module 4 Fig. 14 Pseudocode using decision tree algorithm Kyushu University, Japan 15/4/

22 3.5.2 PSEUDOCODE Module 3 (RTO_s_c) If RTO from server less than 2.2 second Then install the flow in the flow table and forward the next packet Else blacklist the ip source send information to spamhaus Module 4 TTL feature If ip ttl <= 96 Then install the flow in the flow table and forward the next packet Else blacklist the ip source send information to spamhaus Fig. 14 Pseudocode using decision tree algorithm Kyushu University, Japan 15/4/

23 3.5.3 RTT (module 1) Time record started after packet out (server -> client) SDN Domain B SDN Domain B Controlller Packet_in First time, Start flow count=1 WAN SDN Domain A SDN Domain A Controlller SMTP server A Time stamp Flow entry Packet out Time stamp Packet_in 2 nd Time Flow count =2 1 RTT complete Client<-> Server WAN SDN Domain C Spam Haus Server SDN Domain C Controlller WAN SMTP server B Fig. 15 Roundtrip time calculation in Openflow Kyushu University, Japan 15/4/ SMTP server C

24 way handshake time (module 2)Time record started after packet out (server -> client) SDN Domain B SDN Domain B Controlller Packet_in First time, Start flow count=1 WAN SDN Domain A SDN Domain A Controlller SMTP server A Time stamp Flow entry Packet out Time stamp Packet_in 2 nd Time Flow count =2 3whs complete Client<-> Server WAN SDN Domain C Spam Haus Server SDN Domain C Controlller WAN SMTP server B Sym: Syn Syn-Ack Ack Fig way handshake time calculation in Openflow Kyushu University, Japan 15/4/ SMTP server C

25 3.5.5 Module 4 : TTL (hop limit) feature (Recap) Most of botnet came from windows platform Kyushu University, Japan 15/4/

26 4.0 Experiment setup SDN Domain A SDN Domain Controlller Spam Haus Server SDN Domain Controlller WAN WAN SDN Domain Controlller SDN Domain B SMTP server A SDN Domain C WAN SMTP server B Fig. 17 Proposed Experiment setup Kyushu University, Japan 15/4/ SMTP server C

27 4.1 Simulation setup using Mininet Wireshark & Tcpreplay internet Fig. 18 Simulation setup using Mininet Kyushu University, Japan 15/4/

28 4.1.1 Parameter Dataset internet traffic from University New Brunswick (UNB) Canada Day Date Description Size (GB) Saturday 12/6/2010 Sunday 13/6/2010 Monday 14/6/2010 Tuesday 15/6/2010 Wednesday 16/6/2010 Normal Activity. No malicious activity Infiltrating the network from inside + Normal Activity HTTP Denial of Service + Normal Activity Distributed Denial of Service using an IRC Botnet Normal Activity. No malicious activity Table 1. Dataset internet traffic parameter Kyushu University, Japan 15/4/

29 4.1.2 Parameter Dataset Botnet García, S. (2013). Malware Capture Facility Project. CVUT University. Dataset CTU-Malware-Capture-Botnet-1. Retrieved February 03, 2013, from Botnet name Type Portion of flows in dataset Neris IRC (12%) Rbot IRC (22%) Virut HTTP 1638 (0.94 %) NSIS P2P 4336 (2.48%) SMTP Spam P2P (6.48%) Zeus P2P 31 (0.01%) Zeus control (C & C) P2P 20 (0.01%) Table 2: Distribution of botnet types in the training dataset Botnet name Type Portion of flows in dataset Neris IRC (5.67%) Rbot IRC 83 (0.018%) Menti IRC 2878(0.62%) Sogou HTTP 89 (0.019%) Murlo IRC 4881 (1.06%) Virut HTTP (12.80%) NSIS P2P 757 (0.165%) Zeus P2P 502 (0.109%) SMTP Spam P2P (4.72%) UDP Storm P2P (9.63%) Tbot IRC 1296 (0.283%) Zero Access P2P 1011 (0.221%) Weasel P2P (9.25%) Smoke Bot P2P 78 (0.017%) Zeus Control(C& P2P 31 (0.006%) C) ISCX IRC bot P2P 1816 (0.387%) Table 3: Distribution of botnet types in the test dataset Kyushu University, Japan 15/4/

30 4.1.2 Parameter Dataset Botnet Type IP Neris RBot Menti Sogou Murlo Virut IRCbot and black hole Black hole Black hole TBot , , , Weasel Botmaster IP: Bot IP: Zeus(zeus sample 1 and 2 and 3, , , bin_zeus) , Osx_trojan Zero access (zero access 1 and 2) , Smoke bot Type IRC > > > > > > > > > > > > > > > Table 5: List of malicious IPs IP Table 4: List of malicious IPs Kyushu University, Japan 15/4/

31 4.2 Result Performance Analysis SMTP traffic & Botnet attacks Kyushu University, Japan 15/4/

32 4.2.1 Analysis SYN Flood Attack on smtp server using Botnet traffic database Fig. 19 Flow graph botnet for syn flood Kyushu University, Japan 15/4/

33 4.2.2 SMTP Packet analysis on RTT & RTO Packets Time (12 Jun 2010) ---- RTT Packet ---- RTO Packet Fig. 20 Total number of packets per second smtp traffic on 12 jun 2010 Kyushu University, Japan 15/4/

34 4.2.3 SMTP Packet analysis on RTT & RTO Packets Time (13 Jun 2010) ---- RTT Packet ---- RTO Packet Fig. 21 Total number of packets per second smtp traffic on 13 jun 2010 Kyushu University, Japan 15/4/

35 4.2.4 SMTP Packet analysis on RTT & RTO Packets Time (14 Jun 2010) Kyushu University, Japan ---- RTT Packet ---- RTO Packet Fig. 22 Total number of packets per second smtp traffic on 14 jun /4/

36 4.2.5 SMTP Packet analysis on RTT & RTO Packets Time (15 Jun 2010) ---- RTT Packet ---- RTO Packet Fig. 23 Total number of packets per second smtp traffic on 15 jun 2010 Kyushu University, Japan 15/4/

37 4.2.6 SMTP Packet analysis on RTT & RTO Packets Time (16 Jun 2010) ---- RTT Packet ---- RTO Packet Fig. 24 Total number of packets per second smtp traffic on 16 jun 2010 Kyushu University, Japan 15/4/

38 4.2.7 Botnet Training (SMTP Packet analysis on RTT & RTO) Packets Time (3 Feb 2013) ---- RTT Packet ---- RTO Packet Fig. 25 Total number of packets per second smtp traffic on 3 Feb 2013 with botnet training SMTP Spam p2p Attacks Kyushu University, Japan 15/4/

39 4.2.8 Botnet Test (SMTP Packet analysis on RTT & RTO) Packets Time (3 Feb 2013) ---- RTT Packet ---- RTO Packet Fig. 26 Total number of packets per second smtp traffic on 3 Feb 2013 with botnet test SMTP Spam p2p Attacks Kyushu University, Japan 15/4/

40 Time (s) Analysis on SMTP Packet characteristic 10 RTT/RTO/3WHS 1 Jun-12 Jun-13 Jun-14 Jun-15 Jun-16 Botnet Testing Botnet Training Dataset RTT RTO RTO2 3WHS Fig. 27 Max roundtrip time and retransmission time out for7 internet dataset DATASET RTT (s) RTO (s) RTO2 (s) 3WHS (s) Jun Jun Jun Jun Jun Botnet Testing Botnet Training Table 6 Max roundtrip time and retransmission time out for7 internet dataset Kyushu University, Japan 15/4/

41 Hop Limit Time to live packet client 140 TTL Jun-12 Jun-13 Jun-14 Jun-15 Jun-16 Botnet Testing Botnet Training Internet Traffic Dataset (SMTP) DATASET TTL Jun Jun Jun Jun Jun Botnet Testing 128 Botnet Training 128 TTL Fig. 28 Average TTL for packet for 7 internet traffic dataset Table 7 Average TTL for packet for 7 internet traffic dataset Kyushu University, Japan 15/4/

42 Conclusion By using Decision Three Algorithm we can study the Botnet attacks at early stage before arrive to target SMTP Server Most of botnet attacks come from windows based platform This approach only valid within under multi domain SDN controller environment. RTT and RTO are related to the Botnet attacks smtp server. These research also can be focus on other protocol such as http Kyushu University, Japan 15/4/

43 Reference [1] T. Ouyang, S. Ray, M. Allman, and M. Rabinovich, A large-scale empirical analysis of spam detection through network characteristics in a stand-alone enterprise, Comput. Networks, vol. 59, pp , [2] D. Rana, N. Garg, and S. Chamoli, A Study and Detection of TCP SYN Flood Attacks with IP spoofing and its Mitigations, Int. J., vol. 3, no. August, pp , [3] H. Chen, C. Mao, and S. Tseng, An Approach for Detecting a Flooding Attack Based on Entropy Measurement of Multiple Protocols, vol. 18, no. 1, pp , [4] K. Phemius, M. Bouet, and J. Leguay, DISCO: Distributed multi-domain SDN controllers, IEEE/IFIP NOMS IEEE/IFIP Netw. Oper. Manag. Symp. Manag. a Softw. Defin. World, [5] S. Scott-Hayward, G. O Callaghan, and S. Sezer, SDN security: A survey, SDN4FNS Work. Softw. Defin. Networks Futur. Networks Serv., [6] S. Lim, J. Ha, H. Kim, Y. Kim, and S. Yang, A SDN-Oriented DDoS Blocking Scheme for Botnet-Based Attacks, pp , [7] T. Xingl, Z. Xiongl, and D. Huangl, SDNIPS: Enabling Software-Defined Networking Based Intrusion Prevention System in Clouds 1, pp , [8] M. Vizv and J. Vykopal, Future of DDoS Attacks Mitigation in Software Defined Networks. [9] T. Sochor, Overview of SPAM Elimination and its Efficiency, in Research Challenges in Information Science (RCIS), 2014 IEEE Eighth International Conference on, 2014, pp [10] P. Lin, P. Lin, P. Chiou, and C. Liu, Detecting Spamming Activities by Network Monitoring with Bloom Filters, pp , /4/

44 END Thank You Kyushu University, Japan 15/4/

Cybersecurity Threat Mitigation using SDN

Cybersecurity Threat Mitigation using SDN Mohd Zafran (PhD Candidate) & Koji Okamura Graduate School of Information Science and Electrical Engineering Kyushu University Kyushu University, Japan 29/9/2017