An Ad Hoc Adaptive Hashing Technique for Non-Uniformly Distributed IP Address Lookup in Computer Networks

Transcription

1 An Ad Hoc Adaptive Hashing Technique for Non-Uniforly Distributed IP Address Lookup in Coputer Networks Christopher Martinez Departent of Electrical and Coputer Engineering The University of Texas at San Antonio San Antonio, TX , USA and Wei-Ming Lin Departent of Electrical and Coputer Engineering The University of Texas at San Antonio San Antonio, TX , USA ABSTRACT Hashing algoriths have been widely adopted for fast address look-up, which involves a search through a database to find a record associated with a given key. Hashing algoriths transfors a key into a hash value hoping that the hashing renders the database a unifor distribution with respect to the hash value. The closer to unifor hash values, the less search tie required for a query. When the database is key-wise uniforly distributed, any regular hashing algorith (bit-extraction, bit-group XOR, etc.) leads to a statistically perfect unifor hash distribution. When the database has keys with a non-unifor distribution, perforance of regular hashing algoriths becoes far fro desirable. This paper ais at designing a hashing algorith to achieve the highest probability in leading to a uniforly distributed hash result fro non-unifor distributed database. An analytical pre-process on the original database is perfored to extract critical inforation that significantly benefits the design of a better hashing algorith. This process includes sorting the bits of the key to prioritize the use of the in the hashing sequence. Such an ad hoc hash design is critical to adapting to all real-tie situations when there exists a changing database with an irregular non-unifor distribution. Keywords: Hashing, Coputer Networks, Address Lookup, Packet Matching, Internet, Intrusion Detection, Network Security. INTRODUCTION The coputer and counication networks counity has seen extensive advanceent in its research and coercial fields in the past few decades. To further increase the speed of networks takes ore than just physical advances in transission speed through a given ediu. One hindrance that all network coponents, such as routers, firewalls, intrusion detection, and others, have faced and suffered fro it ore as the network size grows is fro the required search and lookup process through a large address space. Fast address lookup or identification atching has becoe critical to the feasibility of any odern applications. In a general for, this proble involves a search process through a large database to find a record (or records) associated with a given key. One odern exaple is in that the routers in wide-area networks have to look through a large database, a routing table, for a forwarding link that atches the given destination address. Another exaple that calls for iinent attention these days is in the area of internet security, in which intrusion detection deands rapid evaluation of client requests. In this, rules are established to allow the intrusion detection syste to check for wrong-doing. Usually, packet headers need to be atched quickly in real tie with rule database. Such a atching process usually is carried out through a hashing process to reduce the otherwise potentially excessively long search tie. There have been any schees developed for IP address lookup proble using a hash function. A coplete survey and coplexity analysis on IP address lookup algoriths has been provided in []. A perforance coparison of traditional XOR folding, bit extraction, CRC-based hash functions is given in [4]. Although ost of the regular hash functions, such as the siple XOR folding and bit extraction, are relatively inexpensive to ipleent in software and hardware, their perforance tends to be far fro desirable. CRC-based hash functions are proved to be excellent eans but are ore coplex to copute and ipleent. Soe schees are hardware based that achieve an iproveent in IP look up by aintaining a subset of routing table in a faster cache eory [7,9], while others are software based which iprove their search per- ISSN: SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME - NUMBER

2 forance ainly through efficient data structures [,3]. Waldvogel et al. [4] proposed an address look up schee based on a binary search of hash table, requiring an extra update process in a look up table. Other hashing algoriths have also been widely adopted to provide for the address look-up process [,3,,6]. All hashing algoriths inevitably suffer fro unpredictable coplexities involving conflicts aong the data with the sae hash result. A search for atching a given query could end up with a sequential search through the nuber of axial conflicts in the database. This ay result in a long search process tie that exceeds the tie liitation iposed by design specifications. When records in the database are key-wise uniforly distributed, any regular hashing algorith would easily lead to the sae probabilistically expected optial perforance in ters of search tie required. On the other hand, if these records are instead not uniforly distributed, then even different regular hash functions would lead to different expected perforance. This paper is aied at the establishent of a ethodology following which a hashing algorith can be designed to achieve the highest probability in leading to a uniforly distributed hash result fro a non-unifor database. An analytical pre-process on the original database is first perfored to extract critical inforation that would significantly benefit the design of a better hashing algorith. This process includes sorting on the bits of the key to prioritize the use of the in the XOR hashing sequence, or a siple bit extraction, or even a cobination of both. Significant iproveent fro our siulation results has been obtained on randoly generated data set as well as real data set. Hashing for Address Look-up H hash cells H subtables H xxx : rule H : hashing operator Rule Set Figure : A Hashing Exaple Basically, hashing is a process that allows the search to go through a statistically saller nuber of steps than a siple sequential straightforward search would have perfored. A hash function, usually a atheatical one, aps a nuber with a large value range into another nuber with a saller range. For exaple, a siplified one as shown in Figure, a database of eight given records are to be atched against for any incoing record. Due to the large size of each record and potentially large nuber of records in the database under real situations, searching through the whole database one at a tie could be erely ipractical. One ay choose to use a portion of the record (or its entirety), as a key, to hash into the target value (a three-bit value as shown in this exaple) using the hash function (operator) H. Therefore, a database of eight records are now grouped into bins of records according to their corresponding hash results. With this, any incoing record would go through the sae hashing to identify the one bin it would need to search through, instead of the whole database. Perfect hashing would guarantee that every bin contains exactly one record, which leads to a search process of exactly one atching to take place. A hash function is considered better than the other if it leads to a saller expected nuber of atching steps required. Note that, if the records in the database follow a unifor distribution with respect to value of the key, any regular hash function, e.g. non-overlapped XOR, bit extraction, etc., would siply lead to a unifor distribution of these records onto the bins. That is, it should always lead to best expected perforance probabilistically in search tie required. The goal of this paper is to develop a universal hashing ethodology applicable to all non-uniforly distributed data sets.. AN HOC ADAPTIVE HASHING FOR DATA SETS WITH AN ARBITRARY pdf One proble that ay arise in the lifetie of the database is that, as the nuber of entries grows and/or the entries are constantly updated, the original hashing algorith previously designed ay becoe obsolete and leads to a large aount of collisions. To deal with such a situation when there exists a changing database with an irregular non-unifor pdf, it would be preferable to develop a hashing algorith based on an ad hoc design that can adapt to the expanding or changing database. To base a hashing algorith on an ad hoc design, a paraeter ust be used to instruct the algorith to adapt over tie. The database is defined as consisting of M = entries with each entry having n bits in length. In order to render the best (unifor) distribution in the final hashed data set, all the bits in the final hashing function H should deonstrate a distribution as probabilistically rando as possible, i.e. evenly distributed between s and s. An optial H will have each of its bits deonstrate even distribution of s and s, and thus leading to the highest probability in reaching the best hashing. The ad hoc paraeter used for adaptation in this paper is identified to be the binary distribution of each bit in the n-bit entry key length. For bit position i, d i is defined as the absolute difference between the nuber of s and s in that bit vector across the data set. Once the d value is found for each bit vector as shown in Figure, they are then sorted into a SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME - NUMBER ISSN: 69-44

3 non-decreasing order as shown in Figure. Note that, Bit Vector V Bit Position Bit Position DB Entry DB Entry # # # # # # #3 #3 #4 #4 # # #6 #6 #7 #7 d = d = Figure : Database with d Values Found Database Sorted By d while sorting is perfored on the d values, no actual rearrangeent of the database is needed; instead, an array of the sorted bit indices is used as the hash function. Assue the bit sequence before sorting is (b n,b n,...,b ) and the bit sequence afterwards is (s n,s n,...,s ). A bit vector V i with d i =indicate that there are even nuber of s and s; while, a d = M reflects that all the bits in the vector have the sae value. Translated to the distribution fro hashing, a bit of d =gives an even hashing distribution (i.e. evenly divided) aong the entire hash space while a d = M hashing will result in hashing to only one half of the hash space. Intuitively, using the bits with saller d values for hashing would lead to a probabilistically better hash distribution, i.e. less conflict in the final apping. Ideally, if one can identify (or through XORing to obtain) bits with all their d values equal to, it should lead to the best potential perforance, assuing no correlation aong the bit vectors. Siulation Setup Our siulation is devoted to finding the proposed technique on several practical perforance indicators. A syste is assued to have a database of entries each of which contains a key of n bits to be used for hashing into bits, and thus each such entry is then apped to a location in a eory of locations (hashed bins) for atching. Three different perforance easureents (indicators) are used to copare different hashing functions: () Nuber of Epty Bins (NEB), () the Average axiu Search Length (ASL), and (3) the Maxiu Search Length (MSL). An illustrating exaple of these indicators is given in Figure 3 showing the nuber of rules apped (hashed) onto each of the bins. NEB tells how any locations in eory that no rule is apped into. ASL is the average axiu nuber of atching steps needed for any given record to atch. MSL denotes the largest nuber of rules that are apped into any location in the eory. For all of the perforance easureents, the saller the value the better the perforance is. The optial perforance value will be for both ASL and MSL, and for NEB, which all indicate that the keys have been evenly distributed aong the eory locations. Data sets 3 Hashed Bins Maxiu Searech Length (MSL) = 3 Average Maxiu Search Length (ASL) = Nuber of Epty Bins (NEM) = Figure 3: Three Perforance Indicators are generated either randoly or fro the real IP addresses collected. Sorting-Based Bit Extraction Hashing Bit extraction is a coonly adopted hashing process for its siplicity. Iediate benefit fro the sorting process is, when eploying siple bit extraction for hashing, significant perforance gain can be easily achieved copared to the regular (rando) bit extraction process. After the sorting, the first bits which have the sallest d values becoe obvious candidates for bit extraction. That is, the final hash keys are (s n,s n,...,s n ). Fro probabilistic point of view, bit extraction using these bits would lead to the best expected perforance aong all bit extraction hashing choices. Figure 4 gives the perforance coparison between the proposed bit extraction hashing and the rando bit extraction approach (i.e. using the first bits before sorting), when n =3and is set to various values. Two data sets are used for siulation, perforance gain (%) NEB MSL perforance gain (%) NEB MSL 8 9 Figure 4: Perforance Iproveent fro Bit-Extraction Hashing on Rando Data Set Real IP Data Set one being a data set randoly generated (in ) such that the d value for each bit position is uniforly distributed, and the other being the real network traffic subnet ID data set (in ). This shows that a very significant iproveent argin is obtained by using this siple bit-extraction approach. A close-to-% reduction in MSL is achieved, thus drastically reducing the search tie required by alost one half. Hybrid XOR Hashing XOR operator has been widely used for hashing and known to be an excellent operator in enhancing rando- ISSN: SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME - NUMBER 3

4 ness in distribution. It also possesses a nice characteristic allowing for analytical perforance analysis and thus better algorith design. XOR-folding is a coonly used hashing technique by siply folding the n-bit key into - bit hash result through a siple process XORing every n key bits into a final hash bit. In soe cases, siple extraction ay outperfor XOR hashing, while in other cases outcoe goes the opposite. A cobination of both would very likely outperfor both if the inforation fro the sorted sequence is properly analyzed and utilized. All-key hashing techniques, i.e. all the bits in the key are used in XORing as in all known XORing hashing techniques, have the benefit in its siplicity, thus easy to ipleent; however, they tend to over-generalize the data set without fully exploiting the d values fro the sorted result. To precisely quantify the benefit in using two bits for XORing with their d values being d i and d j, a forula can be derived to find the expected resultant d for the new bit after XORing, denoted as d [di d j]: d [di d j] = x j k= x C i x M 4k k CM x i j k Cx M j where x i = M di,x j = M dj, =x i x j () Note that the ter C xi x j k CM xi k /Cx M j indicates the probability for the resultant vector to have a d value equal to M 4k. In order not to degrade the hash perforance, every intended XOR operation to be taken between two bits s i and s j after sorting, with d i d j, should lead to a d [di d j] value such that d [di d j] <d i d[di dj] d i Figure : Spectru of d [di d j] Values 3 d j assuing no correlation exists in between bit vectors V i and V j ; otherwise, siply retaining (extracting) the bit s i for hashing would yield better perforance than using XORing of the two bits s i and s j. A coplete spectru of d [di d j] for all possible integral values of d i and d j for M =3is given in Figure. Figure 6 displays, for each d i, the cutoff d j ( d i ) value that would lead to d [di d j] <d i, i.e., the sallest d j that can still be a candidate to be XORed with d i for better perforance. For exaple, when M = 64, ad = will never achieve any ore iproveent when XORing with any d value, i.e. its cutoff is d j =64. On the other hand, a d =6, with its cutoff equal to d j =36, is allowed to XORed with a bit with its d value at least equal 36. Note that, for hashing a database, ultiple bits (i.e. ore than two bits) can be XORed together. That is, when ore than two bits are to be XORed, their final d value would require soe additional process. It can be shown that the process in deterining the d value aong three bits is a process with the property of associativity, that is, d [d[di d j ] d k ] = d [di d [dj d k ]] () Cutoff dj Value d i M=6 M=3 M=64 M=8 M=6 M= M=4 Figure 6: Cutoff Points for Each Integral d i for Different M Note that, d [di d j] (or d [dj d k ]) ay not be an integer anyore, thus the forula in Equation is no longer applicable. We have chosen a siple interpolation/extrapolation 4 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME - NUMBER ISSN: 69-44

5 approach to approxiate the result when non-integral d values are involved when using the forula in Equation. After the bit vectors are sorted with their d values, our proposed hybrid hashing technique is to adopt a folding hash approach in finding suitable candidate bits to XOR. During each folding cycle, at ost ore bits fro the right end of the sorted bit sequence are XORed with the current -bit interediate hash result with their current d values denoted as (D,D,...D ). Initial hash result is siply an extraction of the first bits of the sorted sequence, thus, ( D, D,... D ) ( G, G,... G j i ( d n, d n,... d n ) ({ s n },{ s n },...{ s n }) (D,D,...D )=(d n,d n,...,d n ) Groups of bits hashed so far are recorded as (G,G,...,G ), and it is initialized to the first bits: (G,G,...,G )=({s n }, {s n },...,{s n }) N cutoff d[ D i ds ] j < Di? D i Y d[ ] D i ds j Essentially, in each folding cycle, each interediate hashing bit-group result with an interediate d value equal to D i, starting with the one with the sallest d value, is XORed with the rightost bit s j with a d value of d sj if d [Di d sj ] <D i ; otherwise it has reached its cutoff point. Folding continues until no ore bits are left for XORing. Once cutoff point is reached for an interediate hash bit, it is autoatically excluded for further XORing in subsequence folding cycles since the sequence is already a sorted one. The coplete algorith is shown in Figure 7. For exaple, given a database with 6 entries with each entry bits long, the database key will be hashed into four bits. An exaple is given in Figure 8. In this exaple, the d values for the bit vectors are {,, 4, 4, 6, 6, 8, 8, 8,,, 4} after sorting. Initial hash result is siply (D 3,D,D,D ) = (,, 4, 4) and the bit-grouping is (G 3,G,G,G ) = ({s }, {s }, {s 9 }, {s 8 }). In the first folding cycle, bit s is XORed with bit s since d 4 =.7 <, while for bit s, the next candidate, bit s has a d value of already passing the cutoff point of d s =. Bits s 9 and s 8 subsequently take their corresponding XOR candidates, bit s and s, to wrap up the first folding cycle with (D 3,D,D,D ) = (.7,, 3.36, 3.36) and (G 3,G,G,G )=({s,s }, {s }, {s 9,s }, {s 8,s }). In the second folding cycle, D 3 runs into its cutoff fro d s3 =8, and therefore bit s 3 is XORed with D and bit s 4 is XORed with D, ended with (D 3,D,D,D ) = (.7,,.99,.99) and (G 3,G,G,G ) = ({s,s }, {s }, {s 9,s,s 3 }, {s 8,s,s 4 }). In the third folding cycle, all D s have their cutoff et and thus ending the process. Note that, this greedy algorith requires a very econoic pre-processing tie coplexity of O(n) in ters of the nuber of bits to visit, while an optial algorith using an exhaustive search would require a tie coplexity of O(( n ) ) by allowing all the n bits to potentially contribute to each of the hash bits. end N N G i G i { s j } j = j+; i = i j < n? Y i <? Y restart a folding cycle Figure 7: The Algorith to Deterine a Hybrid XOR Hashing Function Bit Position s s s 9 s 8 s 7 s 6 s s 4 s 3 s s s d = : XOR : cutoff Figure 8: An Exaple of the Hybrid XOR Process ISSN: SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME - NUMBER

6 Figure 9 shows the perforance iproveent fro using the proposed Hybrid XOR approach over the straightforward all-key XOR on rando data set, and Figure shows the perforance iproveent on a real IP data set. Figure gives breakdowns of MSL iproveent on perforance gain (%) perforance gain (%) n=3 n=4 n= n=3 n=4 n=48 Figure 9: Perforance Iproveent in MSL and ASL with Hybrid XOR on Rando Data Set three different types of IP addresses. Very respectable gains are reached in both results. Perforance iproveent is very significant when is sall, that is, when ore folding is allowed. When becoes larger, the proposed ad hoc algorith gradually loses its advantage until passes a threshold value. This is ost likely due to the fact that, on MSL, the original XOR-folding approach is incapable of liit the search length (the axial nuber of data apped to the sae bin) as well as the proposed technique when is increased. Fro, type-a IP addresses lead to ore iproveent with the proposed technique than the other two types. This could be due to the scenario that ore subnet addresses in this type are ore aggregated (i.e. ibalanced) than the other two types, 3. CONCLUSION The wide applicability of the proposed ethodology is clearly deonstrated by our preliinary application results. General applications include: () IP Address Lookup () Intrusion Detection Systes (3) General Database Query (4) String Matching. The success of this developent will benefit any ore application engineers, scientists and syste designers/analysts. There are still any potential extensions along this line of research, including: perforance gain (%) perforance gain (%) MSL ASL Type-A Type-B Type-C Figure : Siulation Results of Iproveent Using Real Work Loads with n = 3: Aggregate Results Breakdown on Types Precise calculation of d values: we plan to extend our analytical work to find an even ore reliable approxiation of the d value calculation when the coposing d values are not integers. Effect of d values and their cobinations on perforance: Exact effect on perforance fro a hashing with a group of final d values still reains unsolved. For exaple, for =(or even larger ), between two final hashing results, (D,D ) and (D,D ), if D <D but D >D, how to deterine on the better one requires ore a coplex analysis and the analysis becoes harder when becoes larger. A thorough understanding of this is critical to the finding of the best hashing function. Correlation aong bit vectors: when there exists correlation aong bit vectors as ost real data sets exhibit, the best hashing produced by relying solely on the d values ay turn out to be far fro the best. How to coe up with a standard and reliable calibration echanis to quantify the correlation between the two bit vectors (c ij ) and derive a function F (c ij,d i,d j ) to decide whether or not the intended XOR is beneficial. Effect of bit re-using: Once bit(s) are re-used, correlation between bit positions sharing the bit(s) arises. How to decide on whether the ensuing correlation is 6 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME - NUMBER ISSN: 69-44

7 too big an investent to copensate fro the gain through using the isleadingly saller d values reains a challenge for this project. 4. REFERENCES [] Ole Able and Donald E. Knuth, Ordered Hash Tables. Chapter 7 in D.E.Knuth Selected Papers on Analysis of Algoriths, CSLI Stanford CA,,. [] A. Broder and M. Mitzenacher, Using Multiple Hash Functions to Iprove IP Lookups, IEEE IN- FOCOM,. [3] Sang-Hun Chung, J. Sungkee, Hyunsoo Yoon, Jung- Wan Cho, A Fast and Updatable IP Address Lookup Schee, International Conference on Coputer Networks and Mobile Coputing,. [4] Raj Jain, A Coparison of Hashing Schees for Address Lookup in Coputer Networks, IEEE Transactions on Counications,, Vol. 4, No., Oct 99. [] Donald E. Knuth, The Art of Coputer Prograing, Vol 3: Sorting and Searching. nd Ed., Addison-Wesley, Reading MA, 998. [6] Christopher Martinez, Wei-Ming Lin and Parial Patel, Optial XOR Hashing for A Linearly Distributed Address Lookup in Coputer Networks, Syposiu on Architectures for Networking and Counications Systes, Oct.,, Princeton, New Jersey [7] Andreas Moestedt, Peter Sjodin, IP Address Lookup in Hardware for High-speed Routing, Proc. IEEE Hot Interconnects 6 syposiu, Stanford, California, pp.3-39, August 998. [8] P. Newan, G. Minshall, T. Lyon, and L. Hutson, IP Switching and Gigabit Routers, IEEE Counications Magazine, January 997, p.64-p.69. [9] Xiaojun Nie, David J. Wilson, Jeroe Cornet, Gerard Da, Yiqiang Zhoa, P Address Lookup Using A dynaic Hash Functio, IEEE Electrical and Coputer Engineering, Canadian Conference, Page(s) 646-6, May -4,. [] Stefan Nilsson and Gunnar Karlsson, IP Address Lookup Using LC-Tries, IEEE Journal on Selected Areas in Counications, pp. 83-9, June 999. [] D. Pao, C. Liu, L. Yeung and K.S. Chan, Efficient Hardware Architecture for Fast IP Address Lookup, IEEE INFOCOM,. [] M.A. Ruiz-Sanchez, E.W. Biersack, and W. Dabbous, Survey and Taxonoy of IP Address Lookup Algoriths, IEEE Network, Vol., pp.8-3, Mar./Apr.. [3] V. Srinivasan and G. Varghese, Faster Ip Lookups Using Controlled Prefix Expansion, Proceedings of SIGMETRICS 98, pp. -, Madison, 998. [4] M. Waldvogel, G. Varghese, J. Turner and B. Plattner, Scalable High Speed Ip Routing Lookups, in Proc. ACM SIGCOMM 97, pp. -3, Sept [] G.B. White and M.I. Huson, A Peer-based Hardware Protocol for Intrusion Detection Systes, Military Counications Conference (MILCOM) 996. [6] P.A. Yilaz, A. Belenkiy, N. Uzun, N. Gogate and M. Toy, A Trie-based Algorith for IP Lookup Proble, Global Telecounications Conference (GLOBECOM). [7] Daxiao Yu, Brandon Sith, and Belle Wei, Forwarding Engine for Fast Routing Lookups and Updates, Global Telecounications Conference,, Globeco 99, p.6-p.64. ISSN: SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME - NUMBER 7