P2P Botnet Detection Using Min-Vertex Cover

Transcription

1 1176 JOURNAL OF NETWORKS, VOL. 7, NO. 8, AUGUST 2012 P2P Botnet Detection Using Min-Vertex Cover Lei Xu College of computer, Nanjing University of Posts and Telecommunications, Nanjing, China 1, 2 XiaoLong Xu and 1 Yue Zhuo 1 College of computer, Nanjing University of Posts and Telecommunications, Nanjing, China 2 State Key Laboratory of Information Security (Institute of Software, Chinese Academy of Sciences), Beijing , China xuxl@njupt.edu.cn, michellezhuo26@gmail.com Abstract P2P botnet is one of the most critical threats to the current Internet security. In this paper, we propose a session-based analysis and minimum vertex cover mining detecting model for core nodes of P2P botnets. This model is focused on solving the core nodes of botnet and has a good performance when the characteristics of botnet are changed to avoid detecting. The simulation experiments reveal that when the session detection rate is at 50% and session falsepositive rate is at 29%, the core node detection rate still remains 98.9%, and the false-positive rate of core node is only 4.87%. Index Terms Network security, Botnet detection, Session analysis, Minimum vertex cove, Core nodes I. INTRODUCTION Botnets are the networks that attacker, for malicious purposes, controls masses of hosts by spreading bots, and then takes use of the command and control channels to communicate[1]. Botnets are not analogous to worms and viruses that cause harm to the computers directly, but to provide a platform for the attacker, who can use this platform for DDOS, spam and some other attacks. Early botnets were mainly based on the IRC protocol and HTTP protocol. With the P2P technology extensively applied in the network, there appeared P2P-based botnets. P2P botnets have many merits compared to centralized botnets (Based on IRC or HTTP protocol), they no longer requires a specific central server, hence the botnets avoid the problem of single point failure and they behave more flexible due to their command and control channels, which make it difficult to track for the attacker. Currently, Researchers proposed several new P2P botnet models, which correct some shortcomings of early botnets and become more difficult to detect and defend. Zhang et al. summed up the shortcomings of of existing botnet models, and proposed some approaches to strengthen the botnet [2]. Wang et al. introduced a new hybrid P2P botnet, in which the bots are divided into two roles, Client bots and Server bots. Using data encryption and personalized service ports, the botnet has a strong robustness and invisibility [3]. Of course, there are many researches are focused on the detection and prevention of botnets. Against the early centralized botnet, many approaches have been proposed [4~8]. Wang et al. and Goebel et al. proposed detecting methods for IRC botnets by the use of similarity of nickname of bots. Gu et al. presented BotSniffer, which can capture spatial-temporal correlation in network traffic and utilize their statistical algorithms to detect botnets. In real-world experiments, BotSniffer shows a high accuracy and has a very low false-positive rate. These methods needn t priori knowledge, and meet the needs of real-time detection. But they aren t appropriate for P2P botnet detection. Recently, network traffic and behavior analysis are popular approaches for P2P botnet detection [9~12]. Liao et al. proposed a P2P botnet detection method based on traffic monitoring and data mining technology [9], and pointed out the difference between behaviors of botnet traffic and those of normal network traffic. Hossein et al. posed a general P2P botnet detection framework [10], which monitored a group of hosts to find out the malicious network traffic and behaviors. However, these methods have some defects. For example, if the characteristics of botnet change, the success rate of detection will sharply decline. Yun et al. put forward a detection measure based on hop distance [13], which compares the communication hop distance with the IP block detection threshold from start point to end point to judge whether they are botnets. This method is merely applied to detect P2P botnet which based on spam attack. As P2P botnet traffic shows a power-law distribution [19] feature, most of the traffic of botnets is passed though several important nodes, we call them core nodes. Hence, the core nodes can represent the entire botnet s behavior performance to a great extent. Additionally, the connectivity of a botnet bears so much on its core nodes that it becomes isolated parts with losing of core nodes. In this sense, if a defender of botnets can find out the core nodes of botnet and then monitor them, he can detect and defend botnets in effect. In this paper, we present a novel P2P botnet detection model that combines session-based analysis doi: /jnw

2 JOURNAL OF NETWORKS, VOL. 7, NO. 8, AUGUST network traffic Session analysis module TCP process Induce character Core node calculation Construct undirected graph Calculate min vertex cover white list filter Analyze suspicious degree UDP process Induce character core nodes table white list filter Analyze suspicious degree session table Figure 1. Session-based analysis and minimum vertex cover mining P2P botnet detection model and minimum vertex cover theory, this model only analyzes network header information, regardless of extra network data load, and takes use of minimum vertex cover to probe the core nodes of botnet. In addition, it maintains a high success rate even if the attacker changes the features of botnets to counterattack. The rest of this paper is organized as follows. Section II describes our P2P botnet detection model including network session analysis and core nodes excavating. Section III gives simulation experiments about our algorithm to solve core nodes of botnet. In Section IV, we draw a conclusion about our paper. II. BOTNET DETECTION METHOD A. Detection and Defense Model We present a detection model with session-based analysis and the minimum vertex cover theory. In Fig. 1, the model can be run on the network gateway, utilize the SPAN technology[20] to gain network traffic, then network session analysis module conducts statistics and analysis of the traffic characteristics. The analysis results will be stored into the session table in the database. And then core node calculation module, via session table, structures a simple undirected graph which represents the botnet. Using the algorithm mentioned below, we calculate the core nodes of network, and save them into the core nodes table. Furthermore, the content of core nodes table will back feed to network session analysis module, in order to induct and adjust the attributes of botnets. B. Network Session Analysis 1) Session Table P2P botnet traffic has some different characters in comparison with normal Internet traffic [9][14]. Such as, packets of P2P botnets are mainly in small size (data portion less than 100 bytes). Accordingly, we structure a session table to count and analyze the properties of network communications. The following shows the process of establishing a session table. In a period of time, we regard 32-bits-integer IP address as the identifier and compile statistics for certain features of network traffic. Besides the IP addresses and characteristics parameters, the session table also has a suspicious degree attribute which describes the possibility of the session to be a bot-session. In Table I, we established a session table including two sessions with parameter of small packet rate. TABLE I. Address 1 Address 2 A SESSION TABLE WITH SMALL PACKET RATE Small packet rate Suspicious degree ) Suspicious Degree Analysis In a session table which includes X1, X 2,..., X n parameters, we analyze these parameters, then determine whether the session is a bot or not. Definition 1 Bot-Session Possibility: in session table, the probability that the current session is the bot session is Bot-Session Possibility, its mathematical expression is: Bot ( s) P( s is bot session X1 x1, X2 x 2,..., Xn xn ) (1) s represents current session, X1, X2,..., X n represents characteristic parameters, x1, x2,..., xn is the statistic of the characteristic parameters corresponding to current session. By the use of Bayes theorem, B ot(s) can be rewritten as: PX ( 1 x1, X2 x2,..., Xn xns is bot session) Ps ( is bot session ) Bot () s PX ( 1 x1, X2 x2,..., Xn xn ) (2)

3 1178 JOURNAL OF NETWORKS, VOL. 7, NO. 8, AUGUST 2012 Because of Ps ( is bot session )and PX (1 x1, X2 x2,..., Xn xn)is constant value for a period of time, and assume that X 1, X 2,..., Xn parameters are independent random variables, then B ot(s) can be further adapted as : Bot ( s) C P( X x s is bot session )... P( X x 1 1 n n s is bot session) (3) In order to quantify the bot-like probability of current session, take the logarithmic processing for B ot(s) : n log( Bot ( s)) log( C) log( P( Xi xi s is bot session)) (4) i 1 If the bot-sessions confidence interval is Ai,then gives: 1, xi Ai log( PX ( i xi s is bot session)) (5) 0, xi Ai We provide the definition of Bot-Session Suspicious Degree: Definition 2 Bot-Session Suspicious Degree: in session table, a definite value that represents the session bot-like degree is Bot-Session Suspicious, its mathematical expression is: n EST() s log( P( Xi xi s is bot session)) (6) i 1 Set the threshold value m(1 m n 1), if EST ( s) m, then regard the session as normal, or as a bot-session. 3) Character Induction As P2P botnet traffic shows a power-law distribution feature, hence the core nodes can take the gauge of entire botnet s behavior performance. Taking into consideration the problem that interference of normal network traffic and changes of botnet characteristics will bring to the potential increase in false-positive rate of botnet detection, we apply botnet core nodes to estimate the botnet traffic characteristics for further induction and adjustment. We cam use some methods to get the interval for determining botnets. For example, if mean value of certain botnet traffic characteristic parameters Xi( i 1,2,..., n) which come from core nodes table approximately meet normal distribution, we can calculate the average value X and variance S 2 of Xi, and select (0 1), then confidence interval Ai is : S S ( X t ( m 1), X t ( m 1)) m 1 m 1 (7) 2 2 We use A i as the range during inducting characteristics parameter, when sessions carry on suspicious degree analysis, if the relevant statistical parameters falls within A i, the session s suspicious degree will correspondingly increase. C. Solve Core Nodes With Minimum Vertex Cover According to the session table, we create a simple undirected graph G. In the graph, both sides of session is transformed to vertex v, and session is converted to edge e, the weight of edge we ( ) represents the bot-session suspicious degree. Therefore the task of seeking for the core nodes can be converted to the problem of solving the minimum vertex cover of G. Definition 3 Minimum Vertex Cover: Give F V( G), if every edge of the graph G has at least one vertex that belongs to set F, then set F is one of the vertex covers of G. If F has min sum of vertex weight, then F is the minimum vertex cover of G. Solving minimum vertex cover [15] is a NP-complete problem, but scholars put forward many approximate algorithms [16~18]. Balaji et al. introduced SRA algorithm, its worst time complexity is Omn ( 2 ). We take advantage of SRA algorithm and put forward solving algorithm of botnet core nodes: Give that minimum vertex cover is Vc, core nodes set is Core, select rate is. Step1 Build simple undirected graph G ( V, E), set the value of edge weight we ( ) with session suspicious degree, and set Vc. Step2 Calculate each vertex s weight wv ( ), wv ( ) w(e), v0 V. ( vv, 0 ) E Step3 Calculate each vertex s support degree s( v ), s( v) w( v0), v0 v ( vv, 0 ) E. Step4 Calculate each vertex s sum degree sum( v ), sum( v) w( v) s( v). Step5 Traverse vertex set V,look for vertex v : sum( v) max{ sum( v1), sum( v2),..., sum( vn)}, vi V then add v to Vc and delete the edges relevant to v. Step6 Repeat Step 2 to Step 5, until E. Step7 Sort Vc according to the added time order, then the set of core nodes of botnet is: Core { vi vi Vc AND i Vc }. III. SIMULATION EXPERIMENTS AND RESULTS ANALYSIS A. Experiments Design Simulation on a PC implements 2000-nodes network, which contains 200 bots and randomly select 10% bots to be core nodes, the distribution of bots refer to the botnet model in [3]. Whereas some other P2P applications will disturb session analysis, we layout 5 P2P application networks that draw on the P2P application networks traffic traits in [21]. P2P botnet detection picks individual

4 JOURNAL OF NETWORKS, VOL. 7, NO. 8, AUGUST Figure 2. the relationship between session detection rate and core nodes detection rate Figure 4. the distribution of false-positives in core nodes set Figure 3. the relationship between detection false -positive rate and core nodes false-positive rate Figure 5. the influence of select rate on core nodes detection feature parameter, and preset detection rate and falsepositive rate, and then we will test if the algorithm in section II is effective. B. Experiments Data Analysis In the case that session analysis detection rate and false-positive rate is known, the success rate of core nodes mining of proposed algorithm is in Fig. 2. We can see even though the session detection rate is not fine, the algorithm works well. However, Fig. 3 indicates that, since minimum vertex cover all edges of graph, including the edges belonged to mistake detection result and pseudo bot-like sessions have a lower connectivity than bot-sessions, the solved core nodes have a higher falsepositive. We found that, if core nodes are displayed by added time sort according to our algorithm, the top 20% nodes have a very low false-positive as shown in Fig. 4. Accordingly, we choose the session detection rate as 60% and session false-positive rate as 25%, then via change the minimum vertex cover select rate to observe the core nodes detection rate and false-positive rate in Fig. 5. It reveals that the detection rate of core nodes always Figure 6. core nodes detection under 20% select rate remain high and false-positive rate almost performs a liner growth with select rate when select rate is below 20%. Finally, balancing the relationship between detection rate and false-positive rate, we set the select rate as 20% and examine the detections of core nodes with various session detections in Fig. 6. The result shows that the false-positive is much lower than that of session detection

5 1180 JOURNAL OF NETWORKS, VOL. 7, NO. 8, AUGUST 2012 and core nodes detection rate decrease slightly compared to that of 100% select rate. Particularly, when the session was at 50% detection rate and 29% false-positive rate, the core nodes detection rate reached 98.9%, while the falsepositive rate was 4.87%. IV. CONCLUSION As core nodes play a significant role in botnets running and current P2P botnet detection methods still remain some deficiencies, we presents a detection model based on session analysis and minimum vertex cover theory. The system can excavate the core nodes of botnet while there is no guarantee on detection rate and false-positive rate of bots. Furthermore it can use the core nodes to induce the features of botnet, which are feedback to system to increase detection efficiency at next time. The algorithm for solving the core nodes in the simulation results shows a finer performance in face of poor session detection rate and false-positive rate. ACKNOWLEDGEMENT The subject is sponsored by the National Key Basic Research Program (973 Program) of China (No. 2011CB302903), the National Natural Science Foundation of China (No ), the Specialized Research Fund for the Doctoral Program of Higher Education (No , No ), the China Postdoctoral Science Foundation Funded Project (No.2011M500095), the Natural Science Foundation of Jiangsu Province (No. BK , No. BK ), the Jiangsu Postdoctoral Science Foundation Funded Project (No C), the Science and Technology Support Program of Jiangsu Province (No. BE )and the Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions(No.yx002001). REFERENCES [1] Jianwei Zhuge, Xinhui Han, Yinglin Zhou, Zhuyuan Ye and Wei Zou, Research and development of botnets, Journal of Software, vol. 19, Mar. 2008, pp [2] Zonghu Zhang, Ando Ruo and Kadobayashi Youki, Hardening botnet by a rational botmaster, 4 th International Conference on Information Security and Cryptology, [3] Ping Wang, Sherri Sparks and Cliff C. Zou, An advanced hybrid peer-to-peer botnet, IEEE Transactions on Dependable and Secure Computing, vol. 7, 2010, pp [4] Wei Wang, Binxing Fang and Cui Xiang, IRC botnet detection based on host behavior, Chinese Journal of Computers, vol. 32, Oct. 2009, pp [5] Guofei Gu, Junjie Zhang and Wenke Lee, BotSniffer:Detecting Botnet Command and Control Channels innetwork Traffic, 15 th Annual Network and Distributed System Security Symposium (NDSS 2008), [6] Wei Wang, Binxing Fang, Zhaoxin Zhang and Chao Li, A novel approach to detect IRC-based botnets, International Conference on Networks Security, Wireless Communications and Trusted Computing (NSWCTC 2009), Apr [7] Jan Goebel and Thorsten Holz, Rishi: Identify Bot Contaminated Hosts by IRC Nickname Evaluation, 1 st Conference on Hot Topics in Understanding Botnets(HotBots 2007),CA USA, [8] Wei Lu and A A. Ghorbani, Botnets Detection Based on IRC-Community, IEEE Global Telecommunications Conference (GLOBECOM 2008), [9] Wenhwa Liao and Chiaching Chang, Peer to Peer detection data mining scheme, International Conference on Internet Technology and Applications (ITAP 2010), Agu [10] Zeidanloo Hossein Rouhani, Manaf Azizah Bt, Vahdani Payam, Tabatabaei Farzaneh and Zamani Mazdak, Botnet detection based on traffic monitoring, International Conference on Networking and Information Technology (ICNIT 2010), June [11] Dan Liu, Yichao Li, Yue Hu and Zongwen Liang, A P2Pbotnet detection model and algorithms based on network streams analysis, 2010 International Conference on Future Information Technology and Management Engineering (FITME 2010), Oct [12] Chang Su and Thomas E. Daniels, P2P botnet detection using behavio clustering & statistical tests, ACM Conference on Computer and Communications Security (AISec 09), Nov [13] Sanqki Yun, Byunqseung Kim, Saewoong Bahk and Hyoqon Kim, Identifying IP blocks with spamming bots by spatial distribution, IEICE Transactions on Communications, vol. E93-B, Aug. 2010, pp [14] Alexander K. Seewald and Wilfried N. Gansterer, On the detection and identification of botnets, Computers and Security, vol. 29, Feb. 2010, pp [15] M. R. Garey, D. S. Johnson and L. Stockmeyer, Some simplified NP-complete problems, 6 th annual ACM symposium on Theory of computing(stoc 74), New York, USA, [16] S. Balaji, V. Swaminathan and K. Kannan, An effective algorithm for minimum weighted vertex cover problem, International Journal of Computational and Mathematical Sciences, vol. 4, Winter 2010, pp [17] Mingyu Xiao, Jianer Chen and Xuli Han, Improvement on vertex cover and independent set problem for low-degree graphs, Chinese Journal of Computers, vol. 28, Feb. 2005, pp [18] Mingyu Xiao, A note on vertex cover in graphs with maximum degree 3, Lecture Notes in Computer Science, vol. 6196, 2010, pp [19] Duanbin Chen, Ying Wan, Junwei Tian and Yan Fu, A P2P botnet control strategy based on social network analysis, Jisuanji Kexue, vol.6, 2009, pp [20] SPAN, lyst6500/ios/12.2sxf/native/configuration/guide/span.html #wp [21] Marios Iliofotou, Hyunchul Kim, Michalis Faloutsos, Michael Mitzenmacher, Prashanth Pappu and George Varghese, Graption: A graph-based P2P traffic classification framework for the internet backbone, Computer Networks, vol. 55, June. 2011, pp

6 JOURNAL OF NETWORKS, VOL. 7, NO. 8, AUGUST Lei Xu is an undergraduate in the college of computer at Nanjing University of Posts and Telecommunications. His research interests include computer nework and information security. Xiaolong Xu graduated from Nanjing University of Posts and Telecommunications (NUPT), China, in He received the M.E. degree from NUPT in 2002 and the ph.d degree from NUPT, China, in He is currently an asociate profesor at College of Computer, NUPT. His research interests include computer software, distributed computing, senor networks and information security. Yue Zhuo is an undergraduate in the college of computer at Nanjing University of Posts and Telecommunications. His research interest is information security.