Network redundancy flow research work include: network data fetching platform, data. Keywords: Network; redundant; flow analysis

Transcription

1 Analysis of Redundant Flow in Network 1 Yan Liu, 2 Yulong Yang 1, 2 Southwest University of Science and Technology ABSTRACT It is information society nowadays, with the rapidly growing popularity of the Internet, more and more people surf the Internet every day. Audio, video and hot web site become more and more popular, resulting in the sharply increase of redundant flow, a great waste of bandwidth, and consumedly reduce the quality of network service. Therefore, people are trying their best to seek a method to solve the problem of insufficient bandwidth. Also, it is very important to prove the existence of redundant flow in the network and study the next-generation Internet architecture. This article firstly uses network sniffer to capture massive data in the network, then statistically analyses the data and finally confirms the presence of a large amount of redundant traffic in the network. Keywords: Network; redundant; flow analysis 1. BACKGROUND According to the experimental results, there are a lot of redundant traffic between two node in the Interne, redundant traffic is refers to the network between the two nodes of the same content repeated more than once. These redundant transmission increases the burden of network traffic, reduces the quality of network service. Especially now the network resources have become more and more rich, and the audio and voide information gradually become the mainstream of the network which challenge the Internet a lot. if we do nothing about it, after a few years the Internet may breakdown due to the bandwidth bottleneck and congestion. Not only the audio and video information, but also popular site visits is ten thousands of times of a common site visits. Popular audio and video, and web page file produced by the redundant traffic are sharply increasing, greatly influenced the quality of the network. So people are trying to seek better ways to meet the needs of sharing network resources, reduce the network load. Researching network redundancy flow characteristics can help analyze the behavior of network user behavior a lot and provide personalized service. In the process of the reformation of the Internet, information storage structure is haven been used to solve the crisis of Internet bandwidth. So-called seeding save structure (Broad - Storage), is a fusion of CDN (Content Delivery Network and Content Delivery Network) and a wide range of Storage resources of the Network structure which P2P traffic often has. It will transport via satellite broadcast and content directly to the national urban and rural areas, and the place distribution of storage is closest to the user. Use the forms of "broadcast and storage" to build storage of the network structure [1]. With the expanding of network size, network traffic characteristics research has become one of the major direction of network performance analysis. Studying of network traffic characteristics can help us to predict network performance, QoS, SLA service applications, network measurement and the basis for application with necessary support. Because now most of the interconnect network are using the TCP/IP protocol, the research of IP packet has been one of the major ways in the study of network traffic. As the set of IP packets, IP flows can not only show the characteristics of a message, but also show the user preference from the perspective of higher levels, so there are quite a lot of literatures analyzing the protocol behavior and user behavior by IP flow analysis to get how it can impact the network load and performance. 2. WEB LOG ANALYSIS We have observed the HTTP accesses which the users searched in the southwest university of science and technology campus network in May Southwest university of science and technology has more than twenty thousand student. The observation was carried out from May 9 to 15 which had 1.15 million visits and involved more than twenty thousand sites, and the heats of the visits are very different. Most popular sites are requested more than hundreds of thousands of times, the coldest only accept a visit. Sort by heat: top 10 sites (need) attract times (43%), 100 sites attract times (72%) befo re the visit, the first 1000 sites attract 1.04 million times (90%). China has millions of websites and 1000 web sites accounted for only about one over one thousand. Visible, most Internet users interest has focused on a few sites. This model is based on the network log server, and can reflect the user community's interest, the user interest distribution. But this method does not analyze the network data traffic, especially the data of these main interest distribution of the flow which is the mainstream of the network communication, and especially the main interest of data is the mainstream of network data communication. The mainstream of the huge amounts of data is duplicate data in the network transmission which takes up a large number of backbone network bandwidth. So we need study the characteristics of the data, then adopt a new mode of transmission, in order to improve the performance of network. If a content is only a single site, and many users in the same local area network (LAN) all need to enjoy the content, then the web server to the LAN routers between the trunk network redundancy inevitably brought traffic. Network redundancy flow research work include: network data fetching platform, data 606

2 processing system. 3. NETWORK DATA CAPTURE PLATFORM 3.1 Principles of Network Data Packet Capture In shared media Ethernet, information is transmitted in plaintext. Ethernet uses a multi-channel access protocol (CSMA / CD), so that Ethernet is a broadcast LAN, i.e. when an adapter to transmit one, all adapters on the LAN can receive the frame. NIC has four operating modes: broadcast mode, multicast transmission mode, direct mode, promiscuous mode. Because the card in the default mode of operation is broadcast mode or direct mode, in this mode the card can only receive frames sent to their own, in order to capture the same collision domain transmit all packets will need to be set to promiscuous NIC mode. Capture technology uses WinPcap-based packet capture technology. The library of this technology works in the bottom of network analysis system modules, it can obtain the most complete and real data from the lowest layer of the network. Winpcap packet capture technology can monitor the communications of the network in realtime, obtaining a copy of the required packages, and does not affect the normal transceiver communication, to ensure the normal use and production. Through data collection module developed on WinPcap can achieve network monitoring and data collection in the case of not affecting the current network load and network node machine system configuration. The data frame captured by WinPcap is Ethernet data frames generated from the encapsulation of the transport layer, network layer and the data link layer in fact, so making it possible to further resolve the data frame [2]. 3.2 Intermediate Node Network Packet Capture System Description It can capture the entire packet by the approach to traffic data collection using probes, the amount of information is the most complete, including Layer 2 to Layer 7 information [3].Parallel probe adds beam splitter to the link monitoring to receive the optical signal separated by the beam splitter, to catch packets. Parallel probe will not increase the network latency, jitter and packet loss, does not affect the throughput of the network, and will not be an additional point of failure to the network, even if the probe fails, normal communications are not interrupted. You can also copy the targeted traffic to the acquisition device by port mirroring function on the router. Network data collection test bed network topology system is shown in Figure 2.We build network data collection test platform in a Network Information Center node of a university, and analyze the reliability and efficiency of capture system. Fig 2: Hierarchical semantic web data extraction test bed network topology diagram 3.3 Intermediate Node Network Packet Capture System Efficiency Analysis Packet access platforms are used in schools on the central node of the network packet capture. The reliability and efficiency of the system are particularly prominent when the data captured is a mass Reliability Put the date obtained from the hierarchical semantic web data extraction test platform as the source for analyzing, with IRIS, Sniffer, Wire shark and other specialized packet capture and analysis tools widely used at home and abroad for analysis. Finally we can get basically the same situation to capture, the accuracy of credibility judgment and quality judgment of information on the hierarchical semantic network data extraction test platform rate up to 95%. Fig 1: Intermediate node network packet capture process Efficiency Test environment is designed as follows to verify the system's efficiency. We use a switch with management function, and connect it to the capture system, then open the port and the switch starts counting at the same time. After some time, we stop capturing packages and close the port at the same time. Switch port stops counting network packet. Next to do is reading the value of the 607

3 switch port and access to the packet size of the file. Packets capture system efficiency verification platform comparison of experimental data is shown in Table 1. Seen, the average efficiency of the experiment is 99.77%.Analysis of experimental error is mainly produced in the switch using a web form to capture. That is in the page refreshes time, the switch still counting state, resulting in errors. Table 1: Packet capture system efficiency comparison table of verification platform experimental data S.No Switch data(bytes) Packet access platform(bytes) Efficie ncy 1 326,719, ,763, % 2 504,933, ,542, % 3 685,327, ,570, % 4 832,011, ,213, % 5 1,266,361, ,579,115,20 6 1,261,920,2 31 1,576,971, % 99.86% 3.4 Kernel File Operations All measuring instruments need to capture packet from network firstly, then analyze. It must improve the capability of measuring instruments and reduce even avoid packet loss in order to get an accurate and reliable analysis result. The cost of software measuring instrument is lower than hardware and it is very convenient to use and maintain. The usually process of this kind of instrument is capture, filtration, then transmitting the needed packet to user layer to store or deal later. This process involves coping packets from operating system kernel to user layer and saving them to files or databases, so it is very important to improve the measuring instrument s capability. This article introduces one method --use the system kernel to deal the data directly since the operating system kernel has a high priority and fast speed. This method not only reduces the system s calling times but also avoids the course sending the data to user layer to deal. Fig 3: Example System Services As picture 3 shows, the Write File functions (at Kernel32.dll) execute system service, and passé service number of Nt Write File, then sinks into kernel, after completing necessary preparatory work by calling Nt Write File function. The system service processor calls the real Nt Write File function to deal I/O request, and completes the real work of writing file, finally returns the result from kernel model to user model application program. Therefore, from the angle of operating system, it should be able to reduce the cost coming from data copy and system call by the method that directly completes file operations at kernel mode, in order to improve the program s capability [4]. 4. PACKET ANALYSIS 4.1 The Method of Identification TCP and UDP Packet from Source File Firstly, determine that the packet contains frame part certainly. Then, check that the hexadecimal of the frame type field of the Ethernet s frame header section is 0080 (Internet Protocol IP) or not. After the data frame, the IP message header is followed. The content which has 9 bytes interval between the IP message header and the Ethernet s type filed referred just now is protocol field in IP message header. If the decimal form of this value is equal to 11 then it is UDP protocol, and if it is equal to 06 then it is TCP protocol. 4.2 The Framework of Hypertext Transfer Protocol (HTTP) Nowadays, WWW mainly uses HTTP/1.0 and HTTP/1.1. HTTP works on client/server model. A browser is a client and an application program that receives connection and returns information to request is web server. The Internet s web server monitors the client s request on the 80th port of TCP [5]. When a browser downloads a web page file from the server, it is needed to shake hands by three times with the 80th port of the server in order to build connection, then send request message to the server by the connection, and finally the server responds to it. 608

4 HTTP Request packet GET /index.shtml HTTP/1.1\r\n Host: Connection: Keep-Alive\r\n User-Agent: Mozilla/5.0\r\n Accept-Language: zh-cn\r\n \r\n The first line of this HTTP request message is called request line, and the request line is made up of three fields: method field, URI field, HTTP edition field. Method field can be GET, POST, HEAD, PUT and DELETE. The browser will use GET method when requests an object [6]. A few lines after the first line are called header line. Header line consists of header field and its value, separated by one space. When the header field is Host, its value is the target host. HTTP Response packets HTTP/ OK\r\n Date: Sun, 21 Apr :11:31 GMT\r\n Content-Type: image/gif\r\n Content-Length: 43\r\n Last-Modified: Wed,19 Sep :33:50 GMT\r\n Connection: close\r\n Accept-Ranges: bytes\r\n \r\n (data) The response message can be divided into three parts: status line, header line and entity body. The entity body is the main part of the message. 4.3 Capture Data Analysis Process It is needed to analyze the resource s location firstly, in other words, to analyze the packet s URL, then further analyze the resource s visit time and data size according to the URL in order to realize the statistical analysis of the redundant flow in network. The captured packet stores in heap file, so it is needed to pick up them one by one from the heap file, and analyze the IP address, port number, URL, serial number and other information of the packet, and then store it back to database. Sort the captured information by IP address, port number and serial number. Then, find out this single resource s data size according to IP address, port number and confirmation number of the packet that has URL. Finally, we can obtain the visit time of all resources, redundant data and the proportion in total resources according to the database packet cluster analysis. field. If the two fields exist at least one, then load it into string container, and connect the URI field at the back of the container, thus we can get the URL. Then regard the URL associated client IP (source IP), the server IP (destination IP), the source port number and the serial number (the confirmation number in TCP header) as the information of this packet, and store it into the MYSQL picketing table. b. If the packet s source port number is 80, then store the information of server IP (source IP), client IP (destination IP), destination port number, serial number and length into picketing table. c. Sort all the data in picketing table by client IP, server IP, port number and serial number. The serial number that is an URL column and is not an empty row in picketing table is the serial number of the packet that responds to client as resource from server. Analyze the data size that responds to the URL from server according to the serial number, and store the related URL into packet URL table. d. Do grouping cluster analysis for the packet URL according to URL, then we can analyze the repeated visit time of the resource and the data size of the network redundant transmission. 5. THE ANALYSIS OF THE EXPERIMENTAL RESULT We can get entire data from any node of network of the school by suing the network data collection test platform that uses the RSPAN of the CISC06509 router in the computer center which is shown below. We capture data from the Port 80 from one building of the school for research, and we totally get 48.2G data from 9:00 to 17:00 from the port. By analyzing the data using the method above, we find out the result as follow: The specific process of packet analysis is as following: a. We read the packet from heap file firstly. If the destination port number of this packet is 80 and the request line is GET method, We obtain the URL from this packet. URL is made up of Host and URI (resource location), and URI begins with the "GET\0x20" and ends with one '\0x20', and Host begins witch the "Host:\0x20" and ends with '\r'. It will find "Refer:\0x20" instead if the request line does not have "Host:\0x20" Fig 4: The analyzing result of the network package There are only million different Independent resource requests among the million we have found out. The repeat rate of the resource request is as high as 46.74% all most a half. The same resource access repetition rates up to a maximum of 186,100 times, 609

5 the relative are also followed by 185,700 times. Another group of independent resources are from 184,000 different pages. The highest Web is 496 times, the next is 391 times. This means that there are a very high percentage of repeat request events in the process of resource access for this similar campus network of the particular network topology. Nearly half of network transmission bandwidth belongs to repeated occupation which will inevitably result in a tremendous waste of resources. 6. CONCLUSION The experiment is carried out on the network data collection test platform, and the high efficiency and the high reliability of the platfrom ensure the completion of the data we captured which reduces the experiment error. According to the statistics of the data information confirmed, there are a number of redundant traffic in the network especially in hot web. If we can reduce the presence of these redundant traffic, Internet performance will be greatly ascend. The system only analyses redundancy situation of the flow of the website, when it faces to the flow of the audio and the video will be worse. For a more comprehensive analysis of network traffic redundancy situation, a comprehensive analysis of various resources is needed. REFERENCES [1] Broadcast memory structures to save Internet bandwidth crisis, fw/web html,may [2] Xiaowen Chen, Wenfei Hu, Yingmin He, WinPcap bypass IP blocking Research and implementation [Finance and Management] January [3] Bing Ni, Yunqing Chen, P2P traffic monitoring technology research and existing network test "Journal of Jilin University (Information Science Edition)" May [4] Fang Liu, A method of improving the efficiency of network measurement-kernel file operation and performance analysis, February [5] TaiMing Ren, TCP/IP Protocol and Network Programming, March [6] James F.Kurose, Keith W.Ross, Computer Networking: A Top-Down Approach,