CHAPTER 7 Normalization of Dataset

Transcription

1 Introduction CHAPTER Introduction Objective of this chapter is to address dataset normalization. From our detailed literature review and also from our previous experiments of [9], we found following questions which must be answered for better normalization: 1. Why do we need to normalize the data or dataset? 2. Which features of the data or dataset should be normalized? 3. How do we normalize the selected features? To answer the first question, we did detailed literature review of dataset normalization. We observed that normalization reduces the size of the dataset and more importantly reduces the response time of detection engine by a large extent [9]. Initial portion of this chapter revolves around first question. To answer the second question, we did detailed analysis of the dataset which is supposed to be used in our research model. On the basis of this analysis, we proposed a list of features that should be normalized. This has been addressed at middle portion of the chapter. Later portion of the chapter addresses the third question in which, dataset normalization model is discussed. Normalization model presented in this chapter can also be used to normalize the real-time network traffic also. More detail about dataset normalization is available in our previous work of [9]. 7.2 Dataset Normalization Dataset Normalization or data normalization is a process or technique in which data is transferred in a form which is more suitable to the data processing engine. Such transformation includes: Data Cleaning, Data Encoding, Data Scaling and Data Reduction [9]. 7.3 Advantages of Dataset Normalization As per our detailed literature review and our implementation which is available in [9], we found the following advantages of dataset normalization: Reduces size of the dataset. 57

2 CHAPTER 7 Requires less hardware resources. Processing is very fast. 7.4 Disadvantages of Dataset Normalization As per our detailed literature review and our implementation which is available in [9], we found the following disadvantages of dataset normalization: More overhead if used for the dataset having less number of rows. More overhead if used for the dataset with less number of features. If proper care is not taken, it may loss the internal structure of the data which reduces the accuracy. 7.5 Need for the Data Normalization in the Proposed Model First phase of implementation of our research model uses KDD CUP 1999 dataset to train BPNN. This KDD CUP 1999 Data set has 41 features and has training records. In the systems like above, where large dataset is used, performance largely depends upon the number of the inputs and their quality [9]. If these inputs are not in normalized form then performance of BPNN will be degraded [8]. Our detailed literature review also shows that usage of normalization decreases the dataset size and also reduces the processing time by a large extent. To use such benefits, we normalized data in our research model. For data normalization, authors of [1] have suggested following key points: 1. Text to Numeric Conversion: Attributes which are in text format, must be converted to the numeric values. 2. Scaling: Attributes having very high range, must be scaled to [-1, 1] or [0, 1] for optimal performance. 3. Size Reduction: Without reducing the number of records, the size of dataset should be reduced. 7.6 Analysis of KDD CUP 1999 Dataset To find the answer for the question what to be normalized, we did the detailed analysis of KDD CUP 1999 dataset. During our analysis, we found following important observations: Text Attributes: In KDD CUP 1999 dataset, attributes like protocol_type, service, flag, and classification of attack are in text format. BPNN is faster in case of numeric values as compare to text values [1]. So, for better performance, these text values must be encoded with numeric values. 58

3 The Model for Dataset Normalization Un-scaled Attributes: In KDD CUP 1999 dataset, attribute number 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17,18,19 23, 24, 32, 33 and 42 are not properly scaled. BPNN can perform better with the small values of inputs, typically around [-1, 1] or [0, 1] [1]. So these un-scaled attributes must be scaled into small range. Size of Dataset: Size of KDD CUP 1999 dataset is very large. If we can reduce the size, without reducing the number of rows and most importantly without loss of data, then memory requirement can be minimized. Results of our above analysis show that if this KDD CUP 1999 dataset which is in unnormalized form is directly used for training and testing purposes, training time and response time will be high which is not acceptable for any real time intrusion detection system. So before using this dataset, we suggest to normalize it. 7.7 The Model for Dataset Normalization KDD/ NSL Dataset Encoding Dataset Scaling Dataset Lossless Size Reduction Checking Dataset Normalized Dataset FIGURE7.1: KDD CUP 1999 and NSL Dataset Normalization Model for Anomaly Detection using BPNN [9] Figure 7.1 shows the model for the normalization of KDD CUP 1999 dataset for anomaly 59

4 CHAPTER 7 detection using BPNN. The model has encoding, scaling, lossless size reduction and checking modules. The model works as follows Encoding Dataset KDD CUP 1999 dataset contains various features which are in text format. BPNN can work faster with numeric values, as compared to text values [1]. For better performance, this sub module encodes text features of KDD CUP 1999 dataset to the numeric values Scaling Dataset After encoding, all the attributes of the dataset is in numeric form, but not in the same scale. BPNN performs faster with input values in [0, 1] range [1]. This sub module scales the numeric values to [0, 1]. Topmost care should be taken while scaling the values as it may lead to loss of information Lossless Size Reduction Scaled dataset is passed to size reduction unit, which replaces 0.00 and 1.00 with 0 and 1 respectively. This replacement reduces the size of the dataset. Since, we are replacing the 0.00 and 1.00 to 0 and 1 respectively, its lossless size reduction. Due to the lossless size reduction technique used, the false alarm rate will not be affected. This reduced dataset is known as pre-final dataset Checking Dataset Dataset which is generated after size reduction is pre-final dataset. This pre-final dataset is given to the checking module. This checking module compares the number of records with the original dataset. If pre-final dataset and original dataset has the same number of records, then final normalized KDD CUP 1999 dataset is ready to use Data Stream To test the BPNN based anomaly detection system on the real network traffic, data streams can be used. The model can also be used to normalize data streams. When the model is used to normalize the data stream, checking sub module of the model is not required. 7.8 Implementation of the Model Following are the details of the implementation of the data normalization model on KDD CUP 1999 dataset Encoding Dataset In KDD CUP 1999 dataset, attributes like protocol_type, service, flag, and classification of attack are in text format. To convert them in numeric values, unique numeric code had been assigned to each possible value of the given attribute. Table 7.1 shows the list of 60

5 Implementation of the Model attributes with their text and corresponding numeric code, which we assigned during implementation. TABLE 7.1: Encoding of Text Attributes to Numeric Value for KDD CUP 1999 Dataset Attribute Protocol type Service value Flag value Attribute Value With Their Numeric Value tcp=1,udp=2,icmp=3 private=1 ftp_data=2 eco_i=3 telnet=4 http=5 smtp=6 ftp=7 ldap=8 pop_3=9 courier=10 discard=11 ecr_i=12 imap4=13 domain_u=14 mtp=15 systat=16 iso_tsap=17 other=18 csnet_ns= 19 finger=20 uucp=21 whois =22 netbios_ns=23 link=24 Z39_50=25 sunrpc=26 auth=27 netbios_dgm=28 uucp_path=29 vmnet=30 domain=31 name=32 pop_2=33 http_443=34 urp_i=35 login=36 gopher=37 exec=38 time=39 remote_job=40 ssh=41 kshell=42 sql_net=43 shell=44 hostnames=45 echo=46 daytime=47 pm_dump=48 IRC=49 netstat=50 ctf=51 nntp=52 netbios_ssn=53 tim_i=54 supdup=55 bgp=56 nnsp=57 rje=58 printer=59 efs=60 X11=61 ntp_u=62 klogin=63 tftp_u=64 red_i=65 urh_i=66 http_8001=67 aol=68 http_2784=69 harvest=70 REJ=1 SF=2 RSTO=3 S0=4 RSTR=5 SH=6 S3=7 S2=8 S1=9 RSTOS0=10 OTH=11 Classification of attack neptune=1 normal=2 saint=3 mscan=4 guess_passwd=5 smurf=6 apache2=7 satan=8 buffer_overflow=9 back=10 warezmaster=11 snmpgetattack=12 processtable=13 pod=14 httptunnel=15 nmap=16 ps=17 snmpguess=18 ipsweep=19 mailbomb=20 portsweep=21 multihop=22 named=23 sendmail=24 loadmodule=25 xterm=26 worm=27 teardrop=28 rootkit=29 xlock=30 perl=31 land=32 xsnoop=33 sqlattack=34 ftp_write=35 imap=36 udpstorm=37 phf=38 warezclient=39 spy=40. During our experiments, we observed 24% reduction in size. This reduction in size minimizes the memory requirement during execution of BPNN for anomaly detection Scaling Dataset As per [1], if the input values of the BPNN is in the range of [-1, 1], the performance of the system will be better. To scale the values, authors of [3] [4] and [5] had used (7.1), while authors of [6] had used (7.2). 61

6 CHAPTER 7 (X-Xmin) X'= Starting Value.... (7.1) (Xmax-Xmin) (X-Xmin) X'= (7.2) (Xmax-Xmin) Where X is normalized value and X is original value TABLE 7.2: Attribute and Their Maximum Values for KDD CUP 1999 Dataset Attribute No. Max. Value Attribute No. Max. Value Attribute No. Max. Value E E After completion of encoding step, all attributes has various maximum values. Mapping of attribute to their maximum value has been shown in the Table 7.2. From the table, it can be seen that attribute number 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 19, 23, 24, 32, 33 and 42 has other than 0 and 1 as maximum value. For better performance of BPNN in anomaly detection, these attributes must be scaled using either (7.1) or (7.2). If starting value is taken as 0, then (7.1) and (7.2) both will become same. During our implementation, as all the features of the dataset has starting values as 0, we used (7.2) to scale the dataset. 62

7 Implementation of the Model Lossless Size Reduction The dataset which has been scaled by the model has many fields with 0.00 and 1.00 as the values. If these fields are converted to 0 and 1 value respectively, then size of the dataset will be reduced without loss of information. Reduction in the size will lead to the lower memory requirement during training as well as testing. To check whether replacement of 0.00 and 1.00 will not have any effect on performance, we performed two set of experiments on 3 Variable XOR problem with learning rate=0.1, standard error =0.01, initial weights =0.5. In the Set I, we took 0 and 1 values as inputs, while in Set II, 0.00 and 1.00 values had been taken as input, and other parameters as constants. For each set, we performed 1000 experiments. Table 7.3 shows the result of our experiments. TABLE 7.3: Comparison of Total Learning Time Between Set I (Inputs with 0 and 1 values) and Set II (Inputs with 0.00 and 1.00 values). The Difference is Negligible. Set I: Total Learning Time for 1000 Experiments (In Seconds) Set II: Total Learning Time for 1000 Experiments (In Seconds) % of difference As per the Table 7.3, % of the difference in total learning time for 1000 experiments for both the set is This difference is mainly due to runtime environment difference caused by various background processes. These experiments suggest that, in 3 Variable XOR problem, there isn t any performance effect due to the replacement of 0 and 1. So, our experiment suggests that to reduce the size of dataset, we can replace all 0.00 and 1.00 values with 0 and 1 respectively Checking Dataset As per the model, after encoding, scaling and size reduction, pre-final dataset is generated. During the generation of this pre-final dataset, if single value is missed or added or corrupted then it can damage the entire dataset. To overcome this, pre-final dataset is compared with original dataset by number of rows. If both have same number of rows then dataset is undamaged and can be treated as final normalized dataset. If this normalization model is used for real-time network traffic then checking sub modules is not required. 63

8 CHAPTER References 1. Siddhartha Bhattacharyya, University of Illinois Chicago at UIC, Class Notes of IDS: 572 -Data Mining for Business, Fall, October, Nsl-kdd dataset for network-based intrusion detection systems. Available on: March Poojitha, G., K. N. Kumar, and P. J. Reddy. "Intrusion Detection using Artificial Neural Network." In Computing Communication and Networking Technologies (ICCCNT), 2010 International Conference on, pp IEEE, Ganesh Kumar, P., and D. Devaraj. "Network intrusion detection using hybrid neural networks." In Signal Processing, Communications and Networking, ICSCN'07. International Conference on, pp IEEE, Kaewarsa, Suriya. "Classification of power quality disturbances using S-transform based artificial neural networks." In Intelligent Computing and Intelligent Systems, ICIS IEEE International Conference on, vol. 1, pp IEEE, Jiang, Jiefeng, Jing Zhang, Gege Yang, Dapeng Zhang, and Lianjun Zhang. "Application of back propagation neural network in the classification of high resolution remote sensing image: Take remote sensing image of beijing for instance." In Geoinformatics, th International Conference on, pp IEEE, Tavallaee, Mahbod, Natalia Stakhanova, and Ali Akbar Ghorbani. "Toward credible evaluation of anomaly-based intrusion-detection methods." Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 40, no. 5 (2010): Wang, Wei, Xiangliang Zhang, Sylvain Gombault, and Svein J. Knapskog. "Attribute normalization in network intrusion detection." In Pervasive Systems, Algorithms, and Networks (ISPAN), th International Symposium on, pp IEEE, Bhavin Shah, Bhushan H. Trivedi, Data Set Normalization : For Anomaly Detection Using Back Propagation Neural Network, IEEE - International Conference on Research and Development Prospectus on Engineering and Technology (ICRDPET),