Distributed Object Storage toward Storage and Usage of Packet Data in a High-speed Network

Transcription

1 Distributed Object Storage toward Storage and Usage of Packet Data in a High-speed Network Masahisa Tamura, Ken Iizawa, Munenori Maeda, Jun Kato, Tatsuo Kumano, Yuji Nomura, Toshihiro Ozawa Fujitsu Laboratories Ltd. Kawasaki, Japan {masahisa, iizawa.ken, maeda.munenori, jun.kato, kumano_tatsuo, nomura.yuji, t.ozawa}@jp.fujitsu.com Abstract As the amount of data flowing in the network increases and as the number of applications having complex traffic patterns increases as well, packet data analysis is becoming all the more important for improving the quality of communications and facilitating troubleshooting. In particular, demand for the storage and analysis of individual items of packet data is increasing in addition to analysis based on statistical information to enable a detailed diagnosis of phenomena that have occurred on the network and at terminals. This paper proposes distributed object storage that can achieve both highthroughput storage of packet data and high-speed session searching. Specifically, we propose and evaluate a system for storing small-size data such as packet data with high throughput while deleting old data and conducting searches, and compare the storage performance of the proposed system with that of other distributed storage systems, and present the results of evaluating search performance for typical search patterns. Keywords packet capture; distributed object storage I. INTRODUCTION Recent years have seen increasing amounts of data flowing in the network as well as the appearance of problems caused by complex traffic patterns not envisioned at the network design stage. Packet data analysis has consequently been growing in importance as a means of solving these problems. In particular, there is a strong demand for techniques that go beyond analysis based on statistical information and sampled data by enabling the storing and analyzing of all packet data to obtain a detailed diagnosis of phenomena occurring on the network. Furthermore, it has been shown that some targeted attacks are skillfully mounted by making small, step-by-step preparations over the long term, so there is also a need to store data over an extended period of time so that long-term traffic patterns can also be analyzed. The storage of large amounts of packet data and the analysis of that data in parallel is required to analyze and communications quality and security problems. Such analysis often involves the extraction of sessions or calls included in groups of packet data, so functions that search for required sessions or calls must have high-performance characteristics. However, high-throughput storage and high-speed searching have a tradeoff relationship, so achieving both requires some type of improvement to be taken with regard to data structure. To store large amounts of data, the development of distributed storage technology for forming a large-capacity storage system in combination with general-purpose servers (commodity hardware) has been accelerating in conjunction with data storage technologies on the cloud. This type of storage has the advantages on the capacity and data management. While the packet data stored can be useful at the time of a problem analysis, it is difficult to estimate just how long packet data should be stored, but distributed storage, which has excellent scale-out capabilities, can be initially implemented on a small scale and can be added the capacities if necessary. Additionally, packet data incorporates metadata such as time stamps and source addresses, which can be used to narrow down packet data to a targeted group of packets during the analysis process. As a result, object storage, which features a search function based on metadata, can reduce data management costs by enabling the uniform management of packet data and metadata. However, distributed object storages on the cloud are typically developed for a large size data and not designed for small size data while searching the sequence of data such as a session. So, some types of improvement need to be taken to utilize distributed object storage for packet data storage and session searching. In this paper, we propose distributed object storage technology for achieving both high-throughput storage of packet data and high-speed session searching. We propose, in particular, a system that can store small-size data such as packet data with high throughput while deleting old data and conducting searches. We also compare the storage performance of the proposed system with that of other distributed storage systems and present the results of evaluating search performance for typical search patterns. II. ISSUES The following issues must be addressed in the development of distributed object storage toward the storage and usage of packet data flowing in a high-speed network. A. High-throughput storage of small- and medium-size data Object storage typically targets relatively large-size data (several tens of MB and larger), small- and medium-size data (up to several MB) is generally not the focus of this type of storage technology. However, achieving both storage and analysis capabilities in a packet-data storage and usage system Copyright IEICE - Asia-Pacific Network Operation and Management Symposium (APNOMS) 2014

2 requires high-throughput storage on the level of network throughput specifically for small- and medium-size data. Here, an optimal balance must be achieved between analysis and storage needs due to the tradeoff relationship between these functions. That is, if data size is too large, data becomes difficult to use at the time of analysis, and if data size is too small, acceptable storage performance cannot be extracted. Moreover, for the same total storage capacity, a smaller object size means a greater number of objects over the entire system, which might become bottlenecks and degrade searching performance. B. High-speed data deletion In object storage, the focus is on data storage and the deletion of data is usually not a matter of importance. The storage of packet data, however, requires that new data be stored while deleting old data once storage fills up. This datadeletion operation therefore requires a level of performance the same as that of storage. module is to cache the hash-space allocation table that indicates which server an object is stored on (or which server the object should be sent to at the time of initial writing) by querying the storage servers beforehand and to update the allocation table if access should later fail by querying the servers again. Here, to save index information for search purposes, the concept of a bucket is created for objects as shown by the conceptual diagram of Figure 1. The relationship between buckets and objects is such that multiple objects exist under a bucket similar to the way that files exist under a directory. In addition, a bucket will consolidate the metadata used for searching from all of the metadata associated with its group of objects and will likewise save that metadata in index form. Buckets will also be distributed among storage servers using name-based consistent hashing the same as done for objects and will also be saved in redundant form among multiple storage servers. C. Real-time analysis maintaining storage performance In packet data analysis, it is essential that no packets be lost. This means that data needed for analysis must be searched for, retrieved, and read at high speed while maintaining the required level of storage performance. We propose the distributed object storage for highthroughput packet data as a total system for resolving issues A to C above. III. BASIC DESIGN OF DISTRIBUTED OBJECT STORAGE FOR HIGH-THROUGHPUT PACKET DATA Objects can be made redundant among storage servers by having a capture machine directly transmit the same object to multiple storage servers. This process, however, can generate transmission bottlenecks in the network. Additionally, while the storage of packet data that comes one after another generates a continuous load in the storage servers, analysis is performed in an ad hoc manner. Data access should therefore incorporate some form of load distribution at the time of analysis. Furthermore, to store a large volume of small- and medium-size objects, a very large number of objects will have to be retained across the entire system, but the centralized management of those object-storage locations could lead to bottlenecks that degrade storage and search performance. We developed new distributed object storage consisting of a proxy module and storage servers mounting hard disk drives (HDDs). Here, an item that combines the data itself and various types of metadata is called an object. It distributes objects among multiple storage servers using name-based consistent hashing and saves objects in redundant form among those servers. Each storage server in the system writes objects to its HDDs and controls interfacing with other storage servers. It also determines the area in hash space allocated to it through adjustments made with other storage servers. Access to the system is achieved via the proxy module. The role of this Figure 1: Conceptual diagram of buckets and objects ( # of replicas = 1 ) Redundant objects (called replicas ) spread across multiple servers each have a role with a ranking attached (primary, secondary, tertiary, etc.). When writing is performed by a proxy module as shown in Figure 2, an object will be transferred along a chain in the order of primary, secondary, tertiary, etc., and when the end point of the chain is reached, responses will be returned in reverse order. Furthermore, on receiving an object write request, a server will transfer that object to the next server and will transfer the associated metadata to the bucket that it belongs to. Here, the processing for adding information to indexes in a receiving bucket will be performed in order along a chain the same as object-write processing. For the sake of balancing load, reading and searching can be performed from any redundant object.

3 IV. Figure 2: Writing and index creation ( # of replicas = 3 ) HIGH-THROUGHPUT STORAGE AND HIGH-SPEED SEARCHING SYSTEM A. Data structure The type of distributed object storage that we aim for requires that small-size packet data be stored at highthroughput and that typical forms of searching be performed at high speeds. Here, if we were to aim only for high-throughput storage of small-size data, we could consider a scheme that repeatedly buffers data in memory and writes that data sequentially to HDDs. This scheme is actually used in some packet capture systems. When also performing analysis, however, this scheme would mean that the packets targeted for reading which compose a session or a call would be stored at discrete locations within this time series of stored packet data. As a result, a large volume of random reads would occur in the analysis process that would not only hinder high-speed analysis but also affect storage performance. Search keys typically used in packet data analysis are time and source address. Among these, searching on the basis of the source-address key is optimized by grouping and writing packets in appropriate ranges of source addresses (called source groups) instead of writing them in time-series order. In this way, the search range can be limited when searching for a specific source address and high-speed searching can be performed. Additionally, prior to grouping by source address, packet groups whose packets share the same values for the 5- tuple set of metadata (destination/source addresses, destination/ source port numbers, protocol number) are extracted as sessions and the metadata of those extracted sessions is saved in units of sessions. This approach can greatly reduce metadata capacity compared to the case of saving metadata in units of packets. Session metadata includes such items as session start/end times and session size in addition to the above 5-tuple items. Object structure oriented to packet data storing and searching is shown in Figure 3. Each object consists of a data section and a metadata section. The data section is of fixed length and consists of packet data and headers for analysis use. Packet data are grouped in units of sessions and ordered in time-series form within a session. Grouping and storing packet data in units of sessions in this way means that reading only has to be performed at a specific location instead of having to perform discrete reads across all items of stored packet data. This approach makes for high-speed analysis. Additionally, given that the size of the area used for buffering packets is finite, sessions that continue for a relatively long time can be spread across multiple objects. In packet-data analysis, it is generally assumed that the packet data targeted for analysis is narrowed down by a number of key-based searches. If each key-based search accesses to the HDDs, that can significantly affect packet-data storage performance. Our system, however, consolidates metadata in units of sessions thereby reducing the volume of metadata significantly. This means that the index used for searching can be placed in memory enabling searches to be executed without affecting storage performance. Once the object which includes the packets targeted is found, then it is read out for analysis. Since the bigger the object is, the more sessions are included in it and not related packets are included, smaller object size is better for reading out and reduces overhead at the time of analysis. Object metadata are consolidated and indexed in units of buckets. As a method for optimizing the search process using time and source address, the system determines these buckets in terms of the earliest start time (called object start time) from among the sessions included in an object and in terms of source group. In other words, indexes are divided along the two axes of time and source address. Here, search processing and index creation in units of buckets can be performed in parallel, which means that using a high number of buckets can lead to improved search performance. On the other hand, a very detailed division of indexes can increase the number of searches needed for analysis thereby degrading the overall performance of analysis processing. An optimal number of buckets used must therefore be set. Figure 3: Object example B. Data allocation The storage performance of an HDD is at its highest when data is written sequentially. When packet data begins to be

4 stored, maximum throughput can be achieved by writing new objects sequentially. A problem here is that new data will have to be written while deleting old data once the HDD has reached its full capacity. As a result, the HDD will have to be accessed to update HDD area management information in conjunction with the deletion of old data, and this, in turn, will impede the sequential writing of data and affect storage performance. In response to this problem, HDD area management information is stored in memory in bitmap form. Here, setting the data section of objects to a fixed length simplifies bitmap management. As for the metadata section, the metadata of multiple objects are grouped into a size close to the fixed length of the data section before recording the data on an HDD. When writing objects, HDD areas are allocated in the direction of increasing addresses regardless of the distribution of free areas, and when the end of the HDD is reached, area allocation begins again from the top of the HDD. C. Search method As described above, an index is divided into object start time and source group. This approach enables the search target to be narrowed down when search conditions includes time and source address. In conducting a search, the system begins by extracting the group of buckets targeted by the search. It then submits search queries in parallel against those extracted buckets, merges the results returned from the queries, and generates overall search results. Table 2: Storage-server/search-server specifications Model Fujitsu PRIMERGY RX100 S6 CPU Intel Xeon E Memory 16GB HDD Seagate Constellation2 SATA 1TB 4 with RAID0 RAID Card LSI SAS1064e B. Storage-server HDD performance Each server was given a RAID0 configuration using 4 SATA HDDs and a RAID card. For this RAID0 volume, we measured the performance of sequential read/write and random read/write for a range of I/O sizes. Specifically, we measured sequential performance using IOzone [6] and measured random performance by issuing random accesses across all HDD areas using asynchronous direct I/O. As shown in Figure 4, sequential write performance was steady at about 365 MB/s regardless of I/O size. Assuming that all storage servers making up distributed storage were running in an ideal state in which all writing to HDDs is sequential, then the network performance limit of 10 Gbps could be reached by 4 servers for a replica number of 1 and by 10 servers for a replica number of 3. In contrast, random write performance varied by I/O size. These results show that performance improves as I/O size increases. In the following evaluation, we used a data size of 4 MB. V. EVALUATION With the aim of evaluating the performance of our distributed object storage in the storage and use of packet data, we compared it with other distributed storage systems (Apache HDFS [7], OpenStack Swift [8]) and evaluated its search performance using typical search patterns. A. Experimental environment We performed this evaluation in an experimental system interconnecting one capture machine, 30 storage servers, and one search server in a 10 Gbps network. For the capture machine, we used a server having 2 sockets of an 8-core/16-thread Intel Xeon E processor and 256 GB of main memory (Table 1), and for each storage server, we used a server having 1 socket of a 4-core/8-thread Intel Xeon E processor, 16 GB of main memory, and 4 1TB SATA HDDs (Table 2). Additionally, for the search server, we used the same type of server as the storage server. We also used a Cisco Nexus 5548 switch to interconnect these servers in a 10 Gbps Ethernet system. Each server ran a 64-bit version of CentOS 6.4 as OS. Table 1: Capture machine specifications Model Fujitsu PRIMERGY RX300 S7 CPU Intel Xeon E Memory 256GB Figure 4: Storage-server HDD performance ( HDD 4 with RAID0 ) C. Storage performance comparison Given packet-data storage under steady-state conditions in which new data is being written while old data is being deleted, we compared storage performance with that of open-source distributed storage Apache HDFS (ver. CDH4.3.1) and OpenStack Swift (ver ). We used the capture machine described above as a load machine, and on this machine, we ran a proxy module for our system and Swift measurements and NameNode for HDFS measurements.

5 Our system creates multiple buckets according to source addresses to limit the search range as described earlier. To emulate this feature, we created 256 buckets beforehand covering all storage servers and repeatedly wrote 4 MB objects selecting buckets at random. We also measured storage performance when deleting old objects in parallel. Specifically, we used the following procedure: (1) begin object writing, (2) after 1800 s, begin deleting any objects for which a time period of 60 s or longer has elapsed since their creation, and (3) calculate average storage performance s after beginning deletion. Storage performance of 10Gbps with triple replication, for instance, consumed 225GB of storage. Storage performance is compared in Figure 5 for a replica number of 3. It can be seen from these results that our system achieves a level of storage performance near the theoretical value and reaches its performance limit of 10 Gbps using 16 storage servers. In contrast, the storage performance of HDFS and Swift for 16 storage servers is about one half and one third, respectively, that of our system. Figure 6: Comparison of storage performance ( # of replicas = 1, object size = 4MB ) HDFS achieved its limit in storage performance at about one half that of our system, but this was due to a CPU bottleneck at NameNode. When writing, HDFS determines the write destination after issuing a query to NameNode, which means that the number of queries sent to NameNode increases as object size becomes smaller. Taking that into consideration, Figure 7 shows results when making measurements for an object size of 64 MB. In this case, HDFS surpassed its performance limit under an object size of 4 MB and achieved a performance limit of 10 Gbps the same as that of our system. Figure 5: Comparison of storage performance ( # of replicas = 3, object size = 4MB ) Using a method different from that of our system and HDFS, Swift creates replicas by having the proxy module write them directly to multiple storage servers. Consequently, for a replica number of 3, the limit in storage performance that can be achieved from one load machine by Swift is one third that of others. With this in mind, storage performance is compared in Figure 6 for a replica number of 1. These results show that our system achieved a storage performance near the theoretical value and reached its performance limit of 10 Gbps using only 3 storage servers. HDFS, meanwhile, achieved the same storage performance as that for a replica number of 3 with 3 storage servers. Swift, however, with replica number set to 1, showed an improvement in its upper performance limit compared to that for a replica number of 3, but the storage performance so reached was only about half that of our system. Figure 7: Comparison of storage performance ( # of replicas = 3, object size = 64MB ) D. Search performance We next measured search performance during actual capturing and storing of packet data. The traffic targeted for capturing was created using Agilent s N2X test platform. Here, given one TCP session for each of 256 source groups (for a total of 256 sessions), traffic flowing at a total rate of 10 Gbps in the network was captured at the capture machine and stored in the system. Then, after a certain amount of time, search processing began and response time was measured. Specifically, at 600 s after the start of packet data capturing, a search was performed for sessions included in the one-second

6 interval 600 to 599 s earlier. This search was performed 10 times each for the case of specifying a source address (SA) and the case of not specifying a SA and average response time was computed. Storage performance of 10Gbps with triple replication, for instance, consumed 225GB of storage. Measurement results for 16 storage servers and a replica number of 3 while varying packet length in the range of bytes are shown in Figure 8. The left axis represents storage performance and the right axis represents search performance. Since the processing required for session extraction increases as packet length becomes smaller, a bottleneck occurred at the capture-machine CPU for a packet length of 256 bytes resulting in dropped packets during capture processing. This reduced the amount of packet data stored. However, for a packet length of 1024 bytes, it became possible to perform capture processing without dropping packets resulting in a storage load of 10 Gbps. At packet lengths of 1024 and 1280 bytes corresponding to a storage load of 10 Gbps, performing a session search while specifying SA enabled search results to be retrieved in msec but doing the same while specifying no SA resulted in a retrieval time of msec. high-speed network. Finally, Deri et al. [5] proposed a system for extracting a level of performance near 10 Gbps by making buffer processing more efficient in general-purpose servers mounting solid state drives (SSDs), but the use of SSDs here means a high-cost system for storing data over the long term. Our proposed technique represents a new approach in that it attempts to achieve distributed object storage realizing both high-throughput storage and high-speed searching of packet data using general-purpose servers mounting HDDs. VII. CONCLUSION We proposed a distributed object storage system for achieving a packet capture system that can provide a high level of performance for both storage and searching. Our system consolidates packet data in units of sessions and groups them according to source address into fixed lengths of small/medium size. It also generates objects combining packet data and the metadata of that session, creates a distribution of those objects using constant hashing and writes them to servers in a chainlike sequence for redundant storing, and creates indexes divided into the two metadata axes of object start time and source address. In comparing our system with Apache HDFS and OpenStack Swift, we showed that the proposed system achieves a level of storage performance about 2 and 4 times that of HDFS and Swift, respectively, for an object size of 4 MB and writing to 16 storage servers. Furthermore, for a system capturing and storing traffic on an order of 10 Gbps, we showed that session searching could be performed in about milliseconds when specifying a source address and in about milliseconds when not specifying a source address. Looking forward, we plan to evaluate the proposed system even further with the aim of enhancing the storage and use of packet data flowing in high-speed networks. We will also examine distributed processing on the storage-server side using the metadata attached to objects. Figure 8: Search performance during data storing VI. RELATED RESEARCH Morariu et al. [3] proposed a packet capture system using P2P-type distributed storage with a distributed hash table. They showed that this system could achieve high-speed searching by selecting a storage-destination server using, for example, packet source address and applying a data structure specialized for searching within the server. Lee et al. [2] proposed a method using Hadoop while Fusco et al. [4] proposed a method capable of high-speed packet searching using bitmap indexes, but neither method focused on storage performance. In addition to the above, Tsukahara et al. [1] proposed a method for storing packet data on the capture machine and improving packet storage performance by tuning system parameters in general-purpose servers mounting HDDs. It would be difficult to say, however, that this method achieves sufficient performance for storing packet data flowing in a REFERENCES [1] Tsukahara, Yasuhito, Takashi Tomine, and Kazunori Sugiura. The Tuning Method of Packet Capturing System to Put Analyzing the Network Traffic Data with General Computer into Reality, Information Processing Society of Japan (IPSJ) Technical Report, Computer Security Group (CSEC), 2010-CSEC-48(7), pp. 1-7, (in Japanese) [2] Lee, Yeonhee, Wonchul Kang, and Youngseok Lee. A hadoop-based packet trace processing tool. Traffic Monitoring and Analysis. Springer Berlin Heidelberg, [3] Morariu, Cristian, Thierry Kramis, and Burkhard Stiller. DIPStorage: distributed storage of IP flow records. Local and Metropolitan Area Networks, LANMAN th IEEE Workshop on. IEEE, [4] Fusco, Francesco, et al. pcapindex: an index for network packet traces with legacy compatibility. ACM SIGCOMM Computer Communication Review 42.1 (2012): [5] Deri, Luca, Alfredo Cardigliano, and Francesco Fusco. 10 Gbit line rate packet-to-disk using n2disk. Computer Communications Workshops (INFOCOM WKSHPS), 2013 IEEE Conference on. IEEE, [6] IOzone homepage: [7] Apache Hadoop homepage: [8] OpenStack homepage: