Byte Index Chunking Approach for Data Compression

Transcription

1 Ider Lkhagvasuren 1, Jung Min So 1, Jeong Gun Lee 1, Chuck Yoo 2, Young Woong Ko 1 1 Dept. of Computer Engineering, Hallym University Chuncheon, Korea {Ider555, jso, jeonggun.lee, yuko}@hallym.ac.kr 2 Dept. of Computer Science and Engineering, Korea University, Seoul, Korea hxy@korea.ac.kr Abstract. This paper presents an algorithm and structure for a deduplication method which can be efficiently define identical data between files existing different machines with high rate and performing it within rapid time. The algorithm identifies to some part of the destination file, and only sends those parts which cannot be matched in this way. The fundamental aspects of reaching faster and accurately looking up result is that data are expressed as fixed-size block chunks and indexed by its anchor byte values in Index-table. Index-table is a 256x256 sized table structure; indexing the edge chunk byte values are used as their cell row and column numbers. To check commonality between two chunks by their SHA1 hash is fundamental to get stable data deduplication. Keywords: Deduplication, Cloud storage, Chunk, Index-table, Anchor Byte 1 Introduction With the explosion of data such as text, audio, video, image, and the proliferation of the data centers, the regulatory back-up data is the pending issue to be resolved. With usage of data deduplication mechanism, the required storage capacity can be reduced and the deduplication scheme is adapted to various storage systems. There are lots of data processing system [1][2], In Content-defined Chunking[3], each block size is partitioned by anchoring based on their data patterns. This scheme can prevent the data shifting problem of the Static Chunking approach. One of the well-known Content-defined Chunking algorithms is LBFS [4], a network File system designed for low bandwidth networks. However, content-defined data deduplication approach can achieve high deduplication ratio, but requires too much time to perform deduplication process in comparison to the other data deduplication approaches. Static 1 This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(mest) (No ), and this research was supported by the MKE, Korea and NHN. under IT/SW Creative research program supervised by the NIPA (NIPA-2012) IST 2013, ASTL Vol. 23, pp , 2013 SERSC

2 Proceedings, The 2nd International Conference on Information Science and Technology chunking [4] is the fastest algorithm among the others for detecting duplicated blocks but the performance is not acceptable with boundary shifting problem. The primary goal of this work is to develop simple fast and high rate data duplication ratio result approach. The key idea is to start lookup process with predict data from destination file which are highly probable to identical to source file in rapid time. Then verify these data are duplicated by sha1 hash commonality checking. If duplicate probable chunk is not verified to duplicated, we start to lookup process again for only this chunk to determine different suspicious chunk is stored in this chunk or not. Then we continue verifying process as previous steps. 2. Design In this work, we implemented a deduplication server, employing source-based approach[5] with a refined and improved touch. We transfer Index-table (size: 256*256) from the server to the client in order to perform lookup process and find high probability of the duplicated data and number of duplicated data blocks. Indextable is a 256x256 sized table structure; keeping chunk numbers and the chunk s edge chunk byte values are used as their cell row and column numbers. Server Client Take file hash DBMS File Find duplicate probable chunks. Send Index matrix Index table Send duplicated probable chunks Data deduplication Ensure suspicious chunk to duplicate Chunking update Block I/O Block I/O Local disk Storage Fig. 1 Proposed System architecture overview: Byte Index based approach First of all, we create Index-table of source file in the server; send it to client and using it to start whole file lookup to define high identical probable data from destination file in client side. Result we can found parts of data which are be very high probability to be duplicated in very fast and got a scheme as shown in Fig

3 First step, we start to byte shifting lookup and aim to find the chunks that are expected to be duplicated (highly probable duplicate chunk) using Index table by only read their boundary byte values in this algorithm. For improving our search results to be more accurate, we not only do search a single chunk, but we also aim to seek adjacent double chunks for per offset in the modified file. Adjacent chunks, we find in modified file, is not only has the same length with the any adjacent chunks in the server, but also both these chunks store same bytes of values at the position where boundaries of each chunk, that s why we call this chunk in modified file as Highly probable duplicate chunk ( Fig. 2). Fig. 2 Duplicated chunk look up process overview. Probability to be a duplicate is the only one from (4,294,967,296) occasions. Nevertheless, Highly probable duplicate chunk is possibly seen to be duplicated but we confirm whether they are duplicated or not by their SHA1 value. Chunking process is implemented on the server. We divide the source file stream into the fixed sized length chunks and calculate its hashes. The step of chunking process is similar to the fixed sized chunking. At the next step is bit difference from fixed sized chunking, in which we also consider values of edge two boundary bytes for each chunk and we call them Anchor bytes Fig. 3.Also we numbering each chunk in source file in the server. By anchor bytes with chunk hash and its chunk number (chunk-index), we create Meta data and then save them into data storage in the server. Fig.3 Overview of chunking with anchor points 319

4 Proceedings, The 2nd International Conference on Information Science and Technology After the chunking process, we then create [256,256] amount of index table. For every chunk, we need to set a value to the convenient cell of Index-table. For a chunk, first anchor point (left edge byte) value of byte represents the horizontal direction index and a last anchor point (right edge byte) value of byte is a reference to the vertical direction of the Index-table. (Fig. 4) Fig. 4 Overview of filling Index-table table Figure 4 shows how Index-table is filled with reference points of the chunk. When we create metadata (with chunk index, chunk hash and chunk anchor points (value of edge bytes)) list, we save the each chunk index to the Index-table at the location where metadata chunk s anchor point (edge bytes) values. 3. Performance evaluation Our experiments were conducted on identical dual-core1.9ghz Celeron processor, each with 2GBytes of RAM and a 7,200RPM, 8.9ms Seagate ST AIDE drive. We examined the behavior of proposed system s deduplication ratio result and performance time consumption separately with comparing Content-defined break point chunking, and Fixed-size chunking approach with performing results. Experiments are based on using a file with 20% random modification from 1110 MB file and measured when block size are 16KB, 32KB, 64KB, 128KB, 256KB, 512KB and 1MB in each case. As can be seen Figure 5, Content-based chunking shows best performance for detecting duplicated region of data. But as described previous section, the overhead of Contend-based chunking is very high. The proposed scheme shows high performance deduplication capability comparable to Content-based chunking. Our scheme consumes very small sized metadata and can be processed faster than Content-based chunking. The performance of Fixed-size chunking is not good because it suffers from boundary shift problem. 320

5 Fig 5 Deduplication ratio result of chunking based approaches. 4. Conclusion In this paper, we introduced an algorithm and structure for a deduplication method which can be efficiently used in a storage system. The algorithm identifies to some part of the destination file, and only sends those parts which cannot be matched in this way. The fundamental aspects of reaching faster and accurately looking up result is that data are expressed as fixed-size block chunks and indexed by its anchor byte values in Index-table. Index-table is a 256x256 sized table structure; indexing the edge chunk byte values are used as their cell row and column numbers. Performance evaluation result shows that the proposed approach is very useful compared to Content-based chunking and Fixed-sized chunking. References 1. JY. Oh and HJ. Kouh, A study on aes extension for large-scale data. The Journal of IWIT. 9, 6 pp (2009) 2. KY. Lee, MJ. Lim, JJ. Kim, KH. Kim and JL. Kim, Design and implementation of a data management system for mobile spatio-temporal query. The Journal of IWIT. 11,1 pp (2011) 3. K. Eshghi and H. Tang, A framework for analyzing and improving content-based chunking algorithms. Hewlett-Packard Labs Technical Report TR. 30(2005) 4. A. Muthitacharoen, B. Chen and D. Mazieres, A low-bandwidth network file system. ACM SIGOPS Operating Systems Review. 35, 5 pp (2001) 5. S. Quinlan and S. Dorward. Venti: A New Approach to Archival Storage, In Proceedings of the FAST 2002 Conference on File and Storage Technologies,