Two Levels Hashing Function in Consistent Hashing Algorithm

Transcription

1 1 Yihua Lan, 2Xiaopu Ma, 3Yong Zhang, 4Haozheng Ren, 5Chao Yin, 6Huaifei Hu School of Computer Engineering, Huaihai Institute of Technology, Lianyungang, China, 2 School of Computer and Information Technology, Nanyang Normal University, Nanyang, China, 3 School of Computer Engineering, Huaihai Institute of Technology, Lianyungang, China, 4 School of Computer Engineering, Huaihai Institute of Technology, Lianyungang, China, 5 School of Computer Science, Huazhong University of Science and Technology, Wuhan, P.R.China, *6,Corresponding Author School of Biological Science and Medical Engineering, South-entral University For Nationalities, Wuhan, P.R.China, 1 Abstract This article focused on consisting hashing algorithm. Traditional hashing function in consistent hashing is linear hashing, and linear hashing needs more virtual nodes to guarantee even data distribution. We designed a new hashing method two levels hashing. This new hashing method uses two hashing functions to eliminate the count of virtual nodes. First level hashing function transfer raw data to more even distributed data, and the second level hashing uses linear hashing method. After two levels hashing, data can distribute more plain than only linear hashing method used, so less virtual nodes can used. Our experimental results show that two levels hashing function can do better than current linear hashing function. Keywords: Hash Buckets, Consistent Hash, Data Distribution 1. Introduction In the circumstance of distributed systems, especially in cloud computing and cloud storage, data distribution is always in a dilemma: efficiency and balanced distribution [1-2]. Considering the efficiency, data need to be stored in the storage devices which as near as better, so less network band rate usage and less latency will occur. But for the device living time, all storages should be balanced access, and the data also should be distributed balanced. Xianhui Li et al. [3] proposed a new data distribution algorithm named consistent hashing to solve the problem of data balanced distribution. Consistent hashing can distribute data in a real number circle range, and for N storage devices, this range divided into N sub-ranges averagely. Every storage device was looked as a hash bucket, and each hash bucket occupies a sub-range. Consistent hashing algorithm has many virtues such as: distribute data evenly [4], flexible in adding more storage devices [5]. But for a linear hashing function, all the data cannot distribute evenly in a real number circle range. Normally, some sub-ranges have intensive data and some sub-ranges have little data. To solve this problem, the concept of virtual node was introduced [6]. Each hashing bucket is divided into M virtual nodes, and totally, N storage devices has M N virtual nodes. Experientially, M N must larger than 200. Too many virtual nodes affect the efficiency of the storage system. In this paper, we proposed a new hashing algorithm to decrease the number of virtual nodes. Our hashing algorithm uses two levels hashing function to map data into a real number circle range, and this method can map data more evenly than one level linear hashing function, so less virtual nodes are introduced. 2. Consistent hashing algorithm and linear hashing function Consistent hashing algorithm is widely used in distributed systems, such as Dynamo [7], Cassandra [8] and DHTbd [9]. International Journal of Digital Content Technology and its Applications(JDCTA) Volume6,Number20,November 2012 doi: /jdcta.vol6.issue

2 With a glance of consistent hashing algorithm, consistent hashing uses circle hash space to replace linear hash space in common hash algorithm. Suppose the algorithm uses 64 bits to form a circle hash space, then the circle space from 0 through 264-1, each object i is hashed as a number in the space, and each storage node X is hashed another number in the space, as Figure 1. Consistent hashing algorithm shows. Objects and nodes use the same hashing function, like equation (1) and equation (2) shows. Hash(object i ) key i (1) Hash(node X ) key X (2) Figure 1. 0~264-1 form a circle hash space and data objects and nodes are hashed into the same space Each node forms a bucket, and each data object stores in the node which is the first node when data object rotate clockwise, as figure 2. Data store in node shows. Figure 2. Data objects store in the node which is the first node in clockwise direction This simple algorithm has the fatal problem: data objects perhaps not distribute evenly in all the arcs which two nodes adjacent. This will cause different node stores different number of data objects. To solve this problem, virtual node is introduced. A real node is split into M virtual nodes, and all the M virtual nodes spread on the circle evenly. In Figure 3. A real node is split into 2 virtual nodes, two objects are stored in two virtual nodes: v2 and v4, these two virtual nodes point to the real node n1 and n2. 466

3 Figure 3. A real node is split into 2 virtual nodes, and data objects store in virtual nodes, mapping into real nodes. Virtual nodes can solve the problem of uneven distribution, as Figure 4. Data objects store in virtual nodes illustrates. In Figure 4(a). without virtual nodes, data object 1 is stored in node A, while data object 2, data object 3 and data object 4 are stored in node C, two nodes stores no equal number of data objects. In Figure4 (b). with virtual nodes, data object 1 is stored in virtual node A2, data object 2 is stored in virtual node A1, data object 3 is stored in virtual node C1 and data object 4 is stored in virtual node C2. Each virtual node stores 1 data object, and two virtual nodes mapped in a real node. So each real node stores 2 data objects. Data objects are distributed evenly in all real nodes. (a) Without virtual nodes (b) With virtual nodes Figure 4. Data objects store in virtual nodes, with virtual nodes, data distribute evenly in two real nodes Generally, for the purpose of even distribution, the circle should have 200~300 virtual nodes. And the hashing function is linear hashing function. Linear hashing functions are widely used in different hashing algorithm. For a key(k) of a data, common linear hashing function is use mod operate about K, that is: V K mod N (3) In equation (3), N buckets in the system and V is the bucket number which data K stored. In real storage system, consistent hashing algorithm has two versions: FNV-1 and FNV-1a, Sheepdog uses FNV-1a. The algorithm of FNV-1a is described as Algorithm 1. FNV-1a algorithm. 467

4 Algorithm 1. FNV-1a algorithm Require: offset_basis: a constant base number FNV_prime: a large prime Output: Hash: hashing value of a key(data object or node) 1. Hash = offset_basis; 2. for each key to be hashed 3. { 4. Hash = Hash xor key; 5. Hash = Hash * FNV_prime; 6. } 7. return Hash; Different real system can chose different offset_basis and FNV_prime, for Sheepdog, the two parameter is offset_basis=0xcbf29ce ull and FNV_prime= xb3. So the expression of FNV-1a in Sheepdog is linear, that means the data distribution on the circle will be the same with data trace. If we want to eliminate the number of virtual nodes, we should smooth the data distribution and let it be more even. 3. Two levels hashing function Two levels hashing function can smooth the data trace to be more even. C.Raczy et al. [10] proved all the data distribution can be transferred into Gaussian Distribution. For Gaussian distribution, center data is density and other zone is sparse. We can use first level hashing function to separate density data, and the second level hashing function uses linear hashing function to map data area into circle area First level hashing function of data objects The most intuitionistic method of transfer Gaussian Distributed data into even data is use reverse function of Gaussian Distribution. But the reverse function is not easy to be calculated, so the alternative is use another function. Tangent function has the similar distribution of Gaussian Distribution, though we can use the reverse function of tangent function to transfer data into even distribution. There are some problems to use reverse function of tangent. The first one is which is the center of data? There are two solutions: the first one is adjusting the data center according to the history, and another is set data center manual. The first solution requires 1 day data history at least, and marks these data with a flag. These data use only linear hashing function. The other data can use two levels hashing function to map. When read or write data, if I/O found the flag is true, linear hashing function is used, while the flag is false, two levels hashing function are used. The other solution requires administrator set the data center, and the administrator should be sophisticated. Right data center will achieve the purpose and wrong data center can only cause data distribution more badly. The second problem the reverse function of tangent s value area is π/2 through π/2, and linear hashing function uses these values as input parameters, so well defined linear hashing function should be chosen. The third problem is deal with fraction. Reverse function of tangent will cause fraction, but in the circle of consistent hashing algorithm, integers are required. The strategy of transfer fractions to integers is cut the number less than certain number. It has no effect when data set is large enough. We use data s address A as the key, and A is from 0 through the whole capacity of storage, suppose it is T. We choose C as the data center, and use a zoom factor εto adjust address range, so the key is: key A C (4) The first level hashing function can calculate the result Hash1: 468

5 Hash1 arctan(key ) (5) 3.2. Second level hashing function of data objects The second level hashing function is linear hashing function. It clips a parameter from π/2 through π/2 to a field [0, 2R-1]. As section 3.1illustrated, the parameters should cut the number less than a certain number S, for more accuracy, the certain number should more less. For example, the certain number S is , and if the parameter is π/4, it will be For easy calculation, S should be 10-I, and I is a integer larger than 0. The easiest linear hashing function is as equation (3) shows, so we can use this equation as our second level hashing function. Hash (Hash1 / 2) 10I mod 2R (6) 3.3. Virtual nodes hashing Two levels hashing function also need virtual nodes, but less than linear hashing. Virtual nodes distribute evenly on the circle area. We can use another simple way to distribute them. Suppose there are N real nodes, and every real nodes has M virtual nodes, so every virtual node has an extent of 2R/(M N). For real node i, its virtual node j, the hash value is: HashNode j N i 2R M N (7) For R = 160, N = 4, M = 8, the circle will be as Figure 5. Virtual nodes distribution on the ring shows. Figure 5. Virtual nodes distribution on the ring 4. Experiments and results We choose different data trace to show that our method will be less virtual nodes than traditional hashing function in consistent hashing algorithm Traces used in the article All the trace used in the article is block level trace: TPC-C is a database trace, which collected by the Performance Evaluation laboratory at Brigham Young University [11]. MSR-Cambridge is a serial 469

6 of traces provided by Microsoft researchers [12]. MSR-Cambridge has many volume-based sub-traces, we choose Hm, which hardware monitoring server, volume 0; MDS, which is Media server, volume 0 and prxy, which is Firewall/web proxy server, volume 0. Table 1. Traces summarize shows the characters of different traces. Trace name Lowest data Address TPC-C HM Prxy MDS Table 1. Traces summarize Highest data Address Total request count Following 4 figures illustrate requested data address distributions of four traces. Four numbers in the legend is the address interval of sample. For example, TPC-C has 200 requests have the same data address between 1200 and 1300, then the Request Count is 200 when Data Address is 1200, and this counterpart legend is 100. From the figures, we can see all data are distributed with Gaussian Distribution, and we can use two levels hashing function to relocate all the data. First thing to do is confirm the data center. We can manual set data center as Table 2. Data center of traces. Here all the data centers are proper data centers. Trace name TPC-C HM Prxy MDS Table 2. Data center of Traces Data center address Figure 6. Data distribution of TPC-C Figure 7. Data distribution of HM 470

7 Figure 8. Data distribution of Prxy Figure 9. Data distribution of MDS 4.2. Compare of two levels hashing function and linear hashing function We use our two levels hashing function compared with Sheepdog s FNV-1a hashing function. The first level hashing function used here according to equation (4) and (5), the second level hashing function we used here accord equation (6). Virtual nodes hashing we used here all according equation (7). Although Sheepdog use different virtual nodes hashing function with equation (7), but for comparative, we also use equation (7) here to guarantee all the virtual nodes are distributed evenly on the circle. For testing the virtual node number, we choose 4 real nodes, that is node0, node1, node2 and node3. Each real node has the same count of virtual nodes. In the experiments, we use VN = 8, VN = 20 and VN = 50 to compare different virtual nodes count. 471

8 Figure 10. Data distribution in real nodes with the trace of TPC-C Figure 11. Data distribution in real nodes with the trace of HM Figure 12. Data distribution in real nodes with the trace of Prxy 472

9 Figure 13. Data distribution in real nodes with the trace of MDS The upper 4 figures show that at the same virtual nodes count, two levels hashing can achieve more even data distribution than linear hashing. In another words, with the same distribution, two level hashing need less virtual nodes than linear hashing Parameters effects on experimental results Parameters can affect experimental results. First parameter affect two hashing methods is circle range, here is identified by R. If R is large enough, all the data addresses will less than 2R, and mod operation will not affect the result, so more virtual node can achieve even distribution. In equation (4), we use a zoom factorεto adjust address range, this is very important. If key larger than 100, all the value of arctan(key) will close to π/2, so the data will fall in a single virtual node, that is to say, most data will be stored in a real node, this is not the purpose of our design. And badly, almost all the data address is larger than C+100. We should use this zoom factor to adjust the data address. To choose this zoom factor, we should know the data address edge of a trace. For example, in Figure 6. Data distribution of TPC-C, the address can be seen as the edge of intensive requests and sparse requests, and the data center of TPC-C is , the difference of edge and center is , so the εcan be set as 1/ The third parameter can be seen in equation (6). I also can be seen as a zoom factor to enlarge data address. For the value of Hash1 is from -π/2 through π/2, so concentrated data address will cause the problem that all the data fall in the same virtual node. Proper selected I can enlarge data address to proper range. Generally speaking, I cannot be too small, but larger I affect little on the results. We choose I=5 or 6 in experiments. 5. Conclusions From the experimental results, we can see two levels hashing can obviously decrease the count of virtual nodes in consistent hashing. Less virtual nodes bring the benefit of efficiency, but two levels hashing should be more calculation. Function arctan(x) is a time consuming operation. Time consuming is for CPU, but for storage devices, more even cause more living time. 6. References [1] Anmin Ding, "On the MGIS-Oriented Design Web Service", JDCTA: International Journal of Digital Content Technology and its Applications, Vol. 5, No. 6, pp. 8 ~ 15, 2011 [2] Xiujuan LI Tao CAI Shiguang JU, "The Hash Postfix Table Based Metadata Management Algorithm for Mass Storage System", JDCTA: International Journal of Digital Content Technology and its Applications, Vol. 4, No. 9, pp. 97 ~ 105,

10 [3] Xianhui Li, Cuihua Ren, Menglong Yue, A Distributed Real-time Database Index Algorithm Based on B+ Tree and Consistent Hashing, In International Conference on Advances in Engineering 2011, Volume 24, pp , 2011 [4] Zhenbin Yan, Wenzheng Li, Research of a scheduling and load balancing scheme based on large-scale distributed systems, in Software Engineering and Service Science (ICSESS), 2012 IEEE 3rd International Conference, pp 22-24, June 2012 [5] Bong Jun Ko, Vasileios Pappas, Ramya Raghavendra, Yang Song, Raheleh B. Dilmaghani, Kangwon Lee, Dinesh Verma, An information-centric architecture for data center networks, in ICN '12 Proceedings of the second edition of the ICN workshop on Information-centric networking, pp 79-84, 2012 [6] Thorsten Schutt, Florian Schintke, Alexander Reinefeld, Structured Overlay without Consistent Hashing: Empirical Results, in the Sixth IEEE International Symposium of Cluster Computing and the Grid, CCGRID 06, 2006 [7] Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall and Werner Vogels, Dynamo: Amazon s Highly Available Key-value Store, In Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, pp , 2007 [8] Avinash Lakshman, Prashant Malik, Cassandra-a decentralized structured storage system, in ACM SIGOPS Operating systems review, Volume 44, issue 2, pp , 2010 [9] George Parisis, George Xylomenos and Theodore Apostolopoulos, DHTbd: A Reliable Blockbased Storage System for High Performance Clusters, In Proceedings of IEEE/ACM CCGRID,2011 [10] C. Raczy, G.Tan and J. Yu, A Sort-Based DDM Matching Algorithm for HLA, Journal of ACM Transactions on Modeling and Computer Simulation(TOMACS) Volume 15, Issue 1, pp , January 2005 [11] TPC-C block i/o trace. Performance Evaluation Laboratory, Brigham Young University, [12] D. Narayanan, A. Donnelly, and A. Rowstron, Write off-loading: Practical Power Management for Enterprise Storage, ACM Transactions on Storage, vol. 4, pp. 1 23, Nov