ADDRESS LOOKUP SOLUTIONS FOR GIGABIT SWITCH/ROUTER

Transcription

1 ADDRESS LOOKUP SOLUTIONS FOR GIGABIT SWITCH/ROUTER E. Filippi, V. Innocenti and V. Vercellone CSELT (Centro Studi e Laboratori Telecomunicazioni) Via Reiss Romoli 274 Torino, ITALY ABSTRACT The Internet continuous growth and its widespread applications require adequate solutions for scaling routing performance. A number of approaches have been recently proposed by different vendors to prepare for the next generation Internet, and new router architectures, capable of processing millions of packets per second, are starting to appear. The efficient processing of IP addresses in such high performance internetworking equipment is one of the most challenging problems to solve. This paper focuses on the available technological solutions for the implementation of the address lookup unit in new generation IP switches and routers. Address lookup algorithms are discussed, and two fast, cheap and flexible hardware-assisted implementations are proposed and evaluated. INTRODUCTION In the last few years routing technology, pushed by the extraordinary success of the Internet, has been forced to evolve towards definitely higher capacity and per-port packet processing speed. On one hand the traditional, shared-medium based interconnection fabric, that represented a major performance bottleneck, has been generally replaced by a true fast packet switch fabric allowing to linearly scale the aggregate packet transfer rate between input and output ports of the router. Modern switch cores usually adopt a crossbar architecture (commercial examples are the Ascend GFR TM, the Lucent Cajun TM Switch and the Cisco router), or a shared memory architecture (like the Bay Networks Accelar TM 1000 and the Packet Engines Summit48 TM switch-router). This, of course, is not sufficient to implement a full gigabit switch-router, because routing implies a specific processing of incoming packets that requires manipulation on a number of relevant fields of the packet header. In conventional router architectures, a centralized CPU performed this packet-by-packet processing. The performance bottleneck, generated by the need to send each packet to the CPU and back to the interface modules, has been overcome by migrating layer 3 packet-by-packet processing tasks from centralized software implementations towards distributed hardware implementations (fig.1). Modern line cards are highly specialized embedded systems that include a large packet buffer (up to 128 Mbytes) and a powerful microprocessor, plus dedicated circuits implementing critical functions such as buffer management, direct memory access and IP forwarding table lookup. 1 N Line card Line card FE FE Fast packet switch core Route processor Fig. 1: general architecture of a Switch-Router The latter is the single most time-consuming operation that must be performed in layer 3 forwarding today: given a destination address, routing information must be retrieved in a large forwarding table consisting of a dictionary of initial address substrings (i.e. address prefixes). The problem is to find the longest initial substring of the destination address that is included in the forwarding database (longest prefix match). IP longest prefix match is made slow and complex by the fact that IP prefixes have variable length and can be prefixes of each other. Most fast retrieval algorithms and data structures (like hashing) need major modifications to meet these requirements. In addition, the forwarding database is very large (up to some tens of thousand entries) and must be dynamically updated. In the following section, the different IP address lookup strategies adopted in the nodes available today are discussed, and state-of-the-art algorithms compared. IMPLEMENTATION ISSUES In the today Internet, more than 0.5 millions of address lookups are required per Gigabit of traffic. The ability to achieve this forwarding speed is due, on one side, to recent improvements in longest address matching algorithms and, on the other, to the remarkable and continuous progress of silicon integration technologies, that have made possible either software or hardware assisted efficient IP route tables lookups. Conventional software implementations usually adopt a cache-based approach: a small forwarding sub-table containing only the most recent (or frequent) destination addresses is tried first, then the large full address prefix table is searched only if the cache lookup fails. Although cache based approaches are largely used, the achievable improvement in performance is extremely dependent on

2 the highly unpredictable dynamic of network traffic. This is an undesirable behavior, considering that the Internet maturity and its commercial orientation require the ability to manage and predict performance. Thus, modern high-end routers avoid caching and implement the full route table lookups on each line card. In the last two years, research efforts have been focused on new lookup algorithms optimized for fast software implementations. On the other side, due to both cost and performance reasons, special hardware blocks are becoming widely used to implement full IP route lookups. Both custom proprietary ASICs and standard devices (like CAMs) can be utilized for this purpose. The main problem with CAMs is that today they are one order of magnitude smaller and slower then RAMs: as a result, a tree-walking ASIC (implementing a search algorithm on a tree data structure stored in a conventional RAM) may be a viable alternative. This is the solution adopted by the majority of recent commercial routing switches. The main issue with hardware implementations is that they run the risk of being quickly made obsolete by either software technologies running on faster hardware, or changes in the forwarding decision specifications (e.g QoS, filtering), or heavy changes in the traffic profile. In this paper we propose some enhancements to the well-known PATRICIA [2] and TRIE [1] retrieval schemes which make them particularly attractive for hardware assisted IP lookups. CONVENTIONAL IP LOOKUP ALGORITHMS Conventional IP address matching implementations are based on the PATRICIA algorithm, which is essentially a compressed binary tree where only bit positions in the input key which discriminate between paths are considered when traversing the tree. An example is the Radix Tree algorithm in the NetBSD kernel. As described in [2], support for longest matching prefixes is provided by means of a recursive backtracking mechanism, which worsen the search time upper bound to N 2, where N is the maximum length of the stored prefixes. A PATRICIA with a non-recursive backtracking mechanism called dynamic prefix (DP) trie is presented in[3]. In a DP-trie, the search algorithm starts from the root, moves downward to a leaf and then backtracks only once to the first node with a matching prefix. The search time upper bound is proportional to 2N, and maintaining the data structure is very easy (random insertion and deletion operations are possible with a cost comparable to those of searching operations). RECENT IP LOOKUP ALGORITHMS PATRICIA (as well as any other search scheme based on binary prefix trees) requires O(N) memory references, where N is the maximum prefix length. With current memory and microprocessor technologies, the lookup time for both software and hardware implementations falls in the range of the microseconds. Hence, significant performance improvements can only be achieved with multiple-bit-at-a-time search algorithms. Some interesting algorithms have recently been proposed, which have demonstrated the feasibility of fast software IP lookups, and will be quickly described in the following. Binary search on hash tables: the algorithm described in [4] is based on binary search on a series of hash tables organized by prefix length. Due to the longest match problem, markers have to be added in the hash tables to ensure that a search operation requires at most log(n) hash lookups, where N is the address length. A number of refinements to the basic scheme are also proposed. No details are given on the hashing scheme used (which should be a perfect scheme if high performance is required): this may be a critical point especially for long and/or frequent prefix lengths. Also, building and maintaining the whole data structure is slow and complex (the paper suggests to batch a number of changes and do a complete build of the search structure). Degenmark s algorithm: another interesting algorithm is proposed in [5]. Here the forwarding table is represented as a tree with three levels. Prefixes are restructured in order to obtain a complete tree in which each prefix uniquely defines an interval of addresses, then the lookup problem is considered as a interval set membership problem, for which very compact data structure are known. Each sub-tree requires up to four memory accesses and about fifty instructions to be searched. The typical number of memory accesses required in a search operation is 8, the maximum 12. The resulting forwarding table size is around Kbytes for the largest tried routing table (containing 33K entries), small enough to almost entirely fit into the secondary cache of actual microprocessors with great benefits for the overall performance. As for the previous algorithm, insertions and deletions require a complete build of the search structure. Modified tries: after this work was done, a number of other works proposing compressed/multibit TRIEs has appeared in the literature [6,7,8]. The main similarities and differences with the search schemes proposed here will be signalled in the following section. THE PROPOSED IMPLEMENTATIONS Our interest is focused on IP lookup techniques well suited for fast and low cost implementation of the forwarding engines that forward packets at each line card (fig. 1). The requirements that we identified are briefly listed in the following: 1. The algorithm should be implementable with simple control logic and low cost standard memories. The algorithm data structure should be regular enough to allow easy memory allocation. 2. The algorithm should be general enough to efficiently support both longest matching and exact matching on the same hardware. Its performance should not be the result of tuning on a specific IP route table or prefix distribution.

3 3. Incremental, local updates should be allowed. 4. Performance must be predictable. Both a low average and a low standard deviation are required for the search time. In other words, we were interested in implementing a low cost hardware module emulating a configurable ternary/binary CAM, with the additional capability of solving multiple matches (normal CAM usually can only detect multiple matches). A survey of existing algorithms (see the previous sections) showed that none of them meets all our requirements: for example, [4,5] assume quasi-static forwarding database; [5,6,7] use some form of in place optimization; efficient memory allocation is hard in [5,6], performance of [3] is not adequate. THE hwdp-trie First of all, we decided to optimize the DP-trie proposed in [3], exploiting the natural concurrent behavior of a hardware implementation. In our implementation: Backtracking is avoided, resulting in a better worstcase bound (N instead of 2N, where N is the address length). The additional comparisons required during the forwarding phase have no impact on hardware implementations. Only one half (one cell) of a basic data row must be retrieved at each step. This means that our implementation requires one half of the memory throughput required, for instance, by [3]. Like in the DP-trie case, the maximum storage complexity is directly proportional to the number of entries, insert and delete operations have strictly local effects, and the resulting data structure is invariant to the particular sequence of insert/delete operations. Each row in the data structure (fig. 2) is composed of two cells addressed by a single bit of the key and containing: a next row pointer, the next significant bit of the key (skip), a target information and the address prefix stored in the cell. Cell 0 Cell 1 prow next_row skip target mask dest next_row skip target mask Prefix cell 0 Fig. 2: basic row of the data structure dest Prefix cell 1 The search algorithm immediately stops when a key comparison fails. This is particularly useful if the default route is present. The pseudo-code for the search algorithm is reported in fig.3 (where is the prefix operator and << the left shift operator). To add or delete an entry, a search is performed in order to find the correct insert/delete point, then a corresponding row is added/deleted. This second step involves updating only one row, plus possibly its parent and children, requiring a fixed time not depending on the size of the table. Hence, the add/delete operation complexity is the same as for a search operation. dest dest_input prow 0 /* search always begin on the first row */ target DEFAULT_TARGET while bit 0 if Patricia[prow][dest<n-1>].dest dest_input then bit bit - Patricia[prow][dest<n-1>].skip dest dest << Patricia[prow][dest<n-1>].skip prow Patricia[prow][dest<n-1>].next_row if Patricia[prow][dest<n-1>].target NULL then target Patricia[prow][dest<n-1>].target else bit 0 Fig.3: search algorithm for the hwdp-trie Search performance can be improved by using a frontend hash table (built with the L most significant bits of the input key) giving direct access to a row in the level L+1 of the hwdp-trie. This requires some prefix expansion up to level L, and an auxiliary table is required to recover the older less-specific targets when an overlapping prefix is deleted; alternatively, the first L levels of the hwdp-trie can be kept in memory and used for maintenance purposes only. When direct access to level L+1 is used, the resulting search time upper bound is O(N-L). As L increases, the amount of memory needed to store the hash table increases, while the amount of memory needed to store the residual trie levels decreases: then, an optimal value for L exists which maximizes performance without any additional memory cost. If L is small enough, memory usage and prefix expansion management are not a major concern and the identity hash function can be used (the L-MSBs of the key directly address a table containing a pointer to the correct row). Otherwise, a more sophisticated perfect hashing function has to be used to map the L-bit strings into a smaller set of compressed addresses. One-level perfect minimal hashing tables are hard to find. Instead, in the following section a multilevel perfect hash table based on a modified multibit trie (the Longest Match Trie) is proposed. The LM-trie itself is a good standalone lookup algorithm for large and dense routing tables (or sub-tables). The LM-TRIE In a basic non-binary trie, the N-bit addresses in the forwarding database are partitioned into a number of sections, each of length K bits, and each section is used to address a different level of a 2 k -ary search tree. The tree is stored as a matrix, where each row represents a node and contains exactly 2 k cells containing pointers to subsequent rows (that represent valid children of that node). The lookup time has an upper bound proportional to N/K and can be reduced by increasing K, which in turns increases the storage complexity. The time-space tradeoff is extensively analyzed in [1]. Memory usage for a specific table may be reduced if a variable number

4 of bits are used to address each level or node in the trie, as suggested in [6]. Multibit tries (K>1) require some extensions if they must be used with variable length overlapping prefixes (as it is the case for IP lookups). A systematic prefix expansion is required to deal with prefixes that are not multiple of K and some mechanism must be added to deal with multiple targets on the same path. Both problems may be solved a priori by strongly restructuring the prefix set before building the forwarding table, as proposed for example in [5,7] and partially in [6]. In this case (named leaf pushing by [6]) only the leaves of the tree can store target information, simplifying the lookup algorithms at the cost of slow, complex table updates. In our implementation 1, we choose to avoid leaf pushing and we use an original strategy to store the target information necessary for both the use and the maintenance of the forwarding table. The data structure: like in the basic tries, the data structure is a two-dimensional matrix. Each row is composed of 2 K cells (addressed by K bit of the input key), plus an additional cell containing target information that is common to all the cells in the row. Each cell contains two flags and either a next row pointer or target information (not both), depending on the value of the two flags: - if the go flag is set, then it contains a next row pointer; - if the target flag is set and the go flag is zero, then it contains target information; - if both flags are set, then it contains a next row pointer, and the associated target information is stored in the additional cell of the row addressed by the next row pointer. Note that when this structure is used under exact match conditions, additional cells don t need to be allocated (the two flags cannot be both set in the same cell). Target data includes: a next hop pointer (the field value), a mask field that (used to identify the length of the corresponding prefix during insertion/deletion operations) and a backup next hop value t_p (used to quickly recover older less-specific targets when a prefix is deleted). In fact, from a logical point of view, a row in a K-bit trie represents a chunk of length K in the equivalent binary tree: then, up to 2 2K -2 different prefixes may be stored into it. Among them, 2 K correspond to leaves of the binary sub-tree and cannot be overlapped by any longer prefixes into the same row. The next hop information of the remaining 2 K -2 prefixes may be associated to the 2 K cell with a simple function that guarantees that: 1. each prefix is assigned to a different cell in the row 2. each prefix is assigned to a cell for which it is a valid target. This implies that only cells carrying target information need to store a backup next hop value t_p. If the number of rows is much larger than the number of distinct next hops, then there is enough space in each cell to store both the primary next hop and the backup next hop. Otherwise, a separate data structure may be used to store t_p (and/or mask). As will be shown below, this methodology for storing data needed to support deletion is more efficient (in terms of memory requirements and deletion speed) than using auxiliary 1- bit tries. The number of rows that are required to store a given routing table is highly dependent on the prefix distribution. According to [1], the maximum number of rows required to store 2 E prefixes of length N with a K- bit trie is (assuming N>E>K): E K K E N E 2 1 R u = 2 + K K K 2 1 This bound is very pessimistic. Anyway, it may be useful to dimension the size of the basic cell when designing a custom ASIC. For example, 3 bytes are enough to store pointers for any Ipv4 table with more than one million prefixes and up to 512 different next hops (if K>2). Search: The search always starts in row 0, follows a path along the trie (under the guidance of the next-row pointers and of the input key bits) collecting the encountered targets, and immediately stops when go=0. A search operation requires up to (N/K + 1) memory references. The pseudo-code for the search algorithm is reported in fig. 4. next_row = 0; target = NULL; target_pnt = 0; continue = TRUE; while ( continue ) index = route_dest >> ( N - K ); continue = Trie[prow, index].go; if ( continue ) if (Trie[prow, index].target) target_pnt = Trie[prow, index].data; prow = Trie[prow, index].data; route_dest = route_dest << K; else if (Trie[prow, index].target) target = Trie[prow, index].data; endwhile if ( target== NULL) target= Aux [ target_pnt ].data; Fig. 4: the search algorithm for the lm-trie Insertion: to add a new prefix, a search is performed in order to find the correct insertion point. If the insert point falls into an existing row, then all the cells of the row covered by that prefix must be checked and (if needed) updated. Otherwise, if the search ended before the end of the prefix, a pointer to a new row must be set and all the cells of the new row covered by that prefix 1 Patent pending

5 must be initialized. An insertion operation requires at most (N/K+1+2 K ) memory references. Deletion: to delete a prefix, a search is performed in order to find the correct deletion point, then the corresponding target must be removed from all the cells of the row covered by that prefix and a substitute target searched by looking at up to K t_p fields. If a substitute target is not found, and the whole row is empty, it must be deleted. If a list containing the number of active cells in each row is maintained, a deletion operation requires to access at most (N/K + K + 2 K ) cells. PERFORMANCE EVALUATION First, we evaluated the required number of memory references for our lookup schemes, which is a good implementation-independent performance index, which is proportional to the performance of a hardware search engine.. The average number of accesses is computed under the (unrealistic) assumption of uniform probability of accessing each routing entry. This may lead to a pessimistic estimation, because in real traffic mixes more aggregate (shorter) prefixes are likely to be more frequent. We used four different real IPv4 and IPv6 tables. Stella (523 prefixes 2 ) is the IPv4 routing table of the main node of our company Intranet. CseltIP6 (140 prefixes) is the IPv6 routing table of the Cselt 6BONE node. MaeWest (16,249 prefixes) and MaeEast (39,930 prefixes) tables are taken from a public database ( which is commonly used as a benchmark. Table 1 reports the average and maximum number of memory references for the various proposed schemes and for different routing tables. Type of Stella Mae-West Mae-East trie Avg Size Avg Size Avg Size 0-hwdp hwdp hwdp hwdp bit-lm bit-lm Table 1: average number of memory references and memory size (in Kbytes) for different tries It can be seen that the average number of memory references in the hwdp-trie with L=0 tends to increase linearly as a function of log(m), where M is the number of entries in the database (thus approaching binary search behavior). This is also true for the small Cselt IPv6 table: despite the fact that the upper bound in Ipv6 tables is 128, the effective average and maximum number of memory references we achieved with a 0- hwdp-trie are, respectively, 10.2 and 16. This is of course due to the particular distribution of the entries in that Ipv6 table. Anyway, Patricia trees and related algorithms could behave better than expected under 2 At the time we sampled it (the number of entries in an IP routing table is floating over time). IPv6, provided that the utilized prefixes (or clusters of prefixes) will be sufficiently randomly distributed. As one can expect, the effectiveness of flattening the root of trie (L>0) and of using multi-bit tries increases with the size of the routing table. It is also interesting to observe that for L>0 up to a critical value the overall memory complexity decreases. The 4-bit LM-trie seems to provide the best performance/size ratio for both small and large tables. Another advantage of the LM-tries is their deterministic behavior: the search time is almost independent of the table size, and the difference between the worst case and the average performance is significantly smaller than with hwdp-tries. As K increases, the memory increases too, but the search time decreases and allows the use of slower, cheaper memory. As an example, a 8-bit LM-trie with up to 1.5 Mbytes of inexpensive 50ns RAM, requires in the worst case 250ns for a lookup with all the real routing tables we tried. If a 20ns RAM is used, lookup time is guaranteed to be less than 100ns for any incoming packet. Despite the fact that the hwdp-trie and the lm-trie have been optimized for general-purpose, hardware-assisted lookups, they have shown a very good behavior also with software implementations. Table 2 and 3 report the average lookup time we obtained by running the code of the hwdp-trie and the lm-trie (compiled by using gcc) on a 300 MHz Sparc Ultra. Lookup times have been estimated with the gprof tool. Twenty million lookups have been performed for each experiment. Table 2 reports the average lookup time we obtained for some hwdp-trie. 0-hwdp 8-hwdp 12-hwdp 16-hwdp Stella Mae-West Mae-East Table 2: average search time (in µs) of the hwdp-trie with different values for L on a 300 MHz Ultra Sparc When L=0, the search performance of the hwdp-trie is, of course, similar to that of the classical Patricia, but it requires less memory. If a front-end array is used, the lookup time is good enough for most applications, with a minimal updating complexity and moderate memory complexity (see table 1). 4bit-lm 8bit-lm search insert del search insert del Stella MaeWest MaeEast Ipv Table 3: average search, insert and delete time (in µs) for the 4-bit and 8-bit lm-trie on a 300 MHz UltraSparc On the other side, the lm-tries provide really good and low-cost performance. In all our experiments on Ipv4

6 tables, the worst case lookup time never exceeded 600ns when K=4 and 300ns when K=8. Comparison with other schemes: it is very difficult to compare software implementations, unless a common platform is defined. Anyway, in table 4 we try to summarize and normalize the results presented in the literature for the main algorithms to compare them with our schemes. Reported data are purely indicative. They are referred to a 300 MHz generic microprocessor and to the Mae-East routing table. Data for the Radix Trie and the Binary search on hash tables are taken from [4] and [6]. Data for the Variable-stride trie are taken from [6]. Data for the Degenmark s algorithm are taken from [5]. Data for the LC-trie are taken from [7]. The number of average memory references may be assumed as a performance index for a HW implementation. It is important to note that the comparison is limited to the search performance. It does not take into account other parameters, like insertion/deletion performance, algorithmic complexity, flexibility. Memory references per lookup (average) Memory size (KBytes) Mpps (300 MHz µproc) Radix trie Binary search on hash table Notes (1) 4 (1) supposing minimal perfect hash Degenmark LC-trie level trie w/ controlled expansion (2)(3) 2-6 (2) 16-hwdp bit-lmtrie (2) Depending on the chosen type (3) auxiliary memory excluded Table 4: approximate comparison of different algorithms for the MaeEast table. Mpps are calculated assuming a software implementation running on a 300 MHz processor. FURTHER REMARKS AND CONCLUSIONS Forwarding speed over 1 Mpps per line card (forwarding engine) can now be easily achieved due to both recent improvements in the longest address matching algorithms and to the remarkable and continuous progress of silicon integration technologies. Even if recent works on IP routing lookups algorithms demonstrate the feasibility of IP lookups in software at gigabit speed, hardware-assisted distributed lookups may scale better and provide a better cost/performance ratio. We suggested two lookup schemes based on tries which are well suited for low-cost, highly reusable hardware implementations and, at the same time, can provide both very fast address lookups and fast, incremental table updates. In particular, the LM-trie is general and flexible enough to implement a broad class of different packet classifications or filtering. The search procedure is not aware of the size and semantic of the search key (as well as of the semantic of the target information), which are relevant only for the insertion/deletion procedures. Thus, the search algorithm may be easily implemented with a simple and fast FSM accessing a memory maintained by flexible software algorithms running on a microprocessor. Different types of addresses may share the same tree-walking ASIC. Additional fields (e.g. for QoS differentiation or firewall filtering) may be attached to the destination address forming a set of extended flow identifier (the classification filters) without any change in the search ASIC. In this case the next hop pointers returned by the search algorithm must be seen as a (possibly large) set of compressed identifiers. REFERENCES [1] Pei e C. Zukowski: Putting Routing Tables in Silicon, IEEE Network Magazine, Jan 1992 [2] Wright, W.R. Stevens: TCP/IP illustrated, vol.2 - The implementation, Addison Wesley Publishing Co., 1995 [3] Doeringer et al., "Routing on longest-matching prefix" IEEE trans. on Networking, vol.4, no.1, February [4] Waldvogel et al. Scalable high speed IP routing Lookups, Proceedings of ACM SIGCOMM, September [5] Degenmark et al.: Small forwarding tables for fast routing lookups, Proceedings of ACM SIGCOMM, September [6] Srinivasan, Varghese, Faster IP lookups using controlled prefix expansion, Proceedings of ACM SIGMETRICS, May [7] Nilsson, Karlsson, Fast address lookup for Internet Routers, Proceedings of IFIP Workshop on Broadband Communication, April [8] Gupta, Lin, McKeown, Routing lookups in hardware at memory access speeds, Proceeding of IEEE INFOCOM, April 1998.