Design and Analysis of a Robust Pipelined Memory System

Transcription

1 Design and Analysis of a obust Pipelined Memory System Hao ang 1 Haiquan (Chuk) Zhao 2 Bill Lin 1 Jun (Jim) Xu 2 1 Department of Eletrial and Computer Engineering, University of California, San Diego {wanghao,billlin}@usd.edu 2 College of Computing, Georgia Institute of Tehnology {hz,jx}@.gateh.edu Abstrat Many network proessing appliations require wirespeed aess to large data strutures or a large amount of flowlevel data, but the apaity of SAMs is woefully inadequate in many ases. In this paper, we analyze a robust pipelined memory arhiteture that an emulate an ideal SAM by guaranteeing with very high probability that the output sequene produed by the pipelined memory arhiteture is the same as the one produed by an ideal SAM under the same sequene of memory read and write operations, exept time-shifted by a fixed pipeline delay of Δ. The design is based on the interleaving of DAM banks together with the use of a reservation table that serves in part as a data ahe. In ontrast to prior interleaved memory solutions, our design is robust even under adversarial memory aess patterns, whih we demonstrate through a rigorous worstase theoretial analysis using a ombination of onvex ordering and large deviation theory. I. INTODUCTION Modern Internet routers often need to manage and move a large amount of paket- and flow-level data. Therefore, it is essential for the memory system of a router to be able to support both read and write aesses to suh data at link speeds. As link speeds ontinue to inrease, router designers are onstantly grappling with the unfortunate tradeoffs between the speed and ost of SAM and DAM. hile fitting all suh data into SAM (aess lateny typially between 5 to 15ns) is fast enough for the highest link speeds, the huge amount of SAM needed renders suh an implementation prohibitively expensive, as hundreds of megabytes or even gigabytes of storage may be needed. On the other hand, although DAM provides inexpensive bulk storage, the prevailing view is that DAM (aess lateny typially between 50 and 100ns) is too slow for providing wirespeed updates. For example, on an 40 Gb/s OC-768 link, a new paket an arrive every 8 ns, and the orresponding read or write operation to the data struture need to be ompleted within this time frame. In this work, we do away with this unfortunate tradeoff entirely, by proposing a memory system that is almost as fast as SAM, for the purpose of managing and moving paket and flow data, and almost as affordable as DAM. A. Motivation To motivate our problem, we list here two massive data strutures that need to be maintained and/or moved inside high-speed network routers. Network flow state Maintaining preise flow state information at an Internet router is essential for many network seurity appliations, suh as stateful firewalls, intrusion and virus detetion through deep paket inspetion, and network traffi anomaly analysis. For example, a stateful firewall must keep trak of the urrent onnetion state of eah flow. For signature-based deep paket inspetion appliations suh as intrusion detetion and virus detetion, per-flow state information is kept to enode partial signatures (e.g., urrent state of the finite state automata) that have been reognized. For network traffi anomaly analysis, various statistis and flow information, suh as paket and byte ounts, soure and destination address and port information, flow onnetion setup and termination times, routing and peering information, et, are reorded for eah flow during a monitoring period for later offline analysis. Indeed, the design of very large flow tables that an support fast lookups and updates has been a fundamental researh problem in network seurity. High-speed paket buffers The implementation of paket buffers at an Internet router is another essential appliation that requires wirespeed aess to large amounts of storage. Historially, the size of paket buffers have been inreasing with link speeds. To takle this problem, several designs of paket buffers based on hybrid SAM/DAM arhitetures or memory interleaved DAM arhitetures have been proposed [1], [2], [3], [4]. Although these designs are both effetive and pratial, these solutions are problem-speifi. Ideally, we would like a general memory arhiteture that is appliable to both this paket buffering problem as well as the various network flow state implementation problems. B. Our Approah In this work, we design a DAM-based memory arhiteture that an emulate a fast massive SAM module by exploiting memory interleaving. In this arhiteture, multiple DAM banks are running in parallel to ahieve the throughput of a single SAM. Our arhiteture provides fixed delay for all the read requests while ensuring the orretness of the output results of the read requests. More speifially, if a memory read operation is issued at time t, its results will be available at output exatly at time t+δ, where Δ is a fixed pipeline delay. This way, a proessor an issue a new memory operation every

2 2 yle, with deterministi ompletion time Δ yles later. No interrupt mehanism is required to indiate when a read data is ready or a write operation has ompleted for the proessor that issued the requests. ith a fixed pipeline delay, a proessor an (statially) shedule other instrutions during this time. Although this Δ is muh larger than the SAM aess lateny, router tasks an often be properly implemented to work around this drawbak thanks to the deterministi nature of this delay. For example, in paket sheduling algorithms (for provisioning Quality of Servie), the flow table needs to maintain for eah flow a timestamp that indiates whether the flow is sending faster or slower than its fair share, whih in turn determines the priorities of the pakets belonging to this flow that are urrently waiting in the queue. Using weighted fair queueing [5], when a new paket pkt arrives at time t, a read request will be issued to obtain its flow state suh as the virtual finish time of the last paket in the same flow. At time t +Δ, when the result omes bak, the virtual finish time of pkt (a per-flow variable) and the system virtual time (a global variable) an both be preisely omputed/updated beause they depend only on paket arrivals happening before or at time t, all of whih are retrieved at time t +Δ. In other words, given any time t, all flow state information needed for sheduling all pakets that arrive before or at time t are available for omputation at time t +Δ. Although all pakets are delayed by Δ using our design, we will show that this Δ is usually in the order of tens of miroseonds, whih is three orders of magnitude shorter than end-to-end propagation delays (tens of milliseonds aross the US). hile multiple DAM banks an potentially provide the neessary raw bandwidth that high-speed routers need, traditional memory interleaving solutions [6], [7] do not provide onsistent throughput reliably beause ertain memory aess patterns, espeially those that an oneivably be generated by an adversary, an easily get around the load balaning apabilities of interleaving shemes and overload a single DAM bank. Another solution proposed in [8] is a virtuallypipelined memory arhiteture that aims to mimi the behavior of a single SAM bank using multiple DAM and SAM banks. All the operations that an be stored in the buffer are provided with fixed delay using a yli buffer. However, this arhiteture assumes perfet randomization in the arrival requests to the memory banks. Under adversarial arrivals, the buffer will overflow and start dropping requests, whih greatly degrades the system performane. To guard against adversarial aess patterns, memory loations are randomly distributed aross multiple memory banks so that a near-perfet balaning of memory aess loads an be provably ahieved, even under arbitrary (inluding adversarial) memory aess patterns. This random distribution is ahieved by means of a random address permutation funtion. Note that an adversary an oneivably overload a memory bank by sending traffi that would trigger the aess of the same memory loation beause they will neessarily be mapped to the same memory bank. However, this ase an be easily handled with the help of a reservation table (whih serves in part as a data ahe) in our memory arhiteture. ith a reservation table of C entries, we an ensure that repetitive memory requests to the same memory address within a time window C will only result in at most two memory aesses in the worst-ase, with one read request followed by one write request. All the other requests will only be stored in the reservation table to provide a fixed delay. Another key ontribution of this paper is a mathematial one we prove that index randomization ombined with a reasonably sized reservation table an handle with overwhelming probability arbitrary (inluding adversarial) memory request patterns without having overload situations as refleted by long queueing delays (to be made preise in Setion V). This result is a worst-ase large deviation theorem in nature [9] beause it establishes a bound on the largest (worst ase) value among the tail probabilities of having long queueing delays under all admissible (inluding adversarial) memory aess patterns. Our methodology for proving this hallenging mathematial result is a novel ombination of onvex ordering and (traditional) large deviation theory. C. Outline of Paper The rest of the paper is organized as follows. Setion II defines the notion of SAM emulation. Setion III desribes the basi memory arhiteture. Setion IV extends our proposed memory arhiteture providing robustness against adversarial aess patterns. Setion V provides a rigorous analysis on the performane of our arhiteture in the worst-ase. Setion VI presents an evaluation of our proposed arhiteture. Finally, Setion VII onludes the paper. II. IDEAL SAM EMULATION In this paper, we aim to design a DAM-based memory arhiteture that an emulate a fast SAM by exploiting memory interleaving. Suppose that the SAM-to-DAM random aess lateny ratio is μ (e.g. μ = 4ns/40ns = 1/10, where the aess latenies of SAM and DAM are assumed to be 4 ns and 40 ns respetively). e use B>1/μ DAM banks and randomly distribute the memory requests aross these DAM banks so that when the loads of these memory banks are perfetly balaned, the load fator of any DAM 1 bank is Bμ < 1. For simpliity, we assume that an ideal SAM an omplete a read or write operation in the same yle that the operation is issued. On the other hand, it takes a DAM bank 1/μ yles to finish a read or write operation. A SAM emulation that mimis the behavior of an ideal SAM is defined as follows Definition 1 (Emulation) A memory system is said to emulate an ideal SAM if under the same input sequene of reads and writes, it produes exatly the same output sequenes of read results, exept time-shifted by some fixed delay Δ. More speifially, our emulation guarantees the following semantis Fixed pipeline delay If a read operation is issued at time t to an emulated SAM, the data is available from the

3 3 memory ontroller at exatly time h = t +Δ (instead of the same yle), where h is the ompletion time. Cohereny The read operations output the same results as an ideal SAM system, exept for a fixed time-shift. Fig. 1 illustrates the onept of SAM emulation. In Fig. 1, a series of six read and write aesses to the same memory loation are initiated at times 0, 1, 2,..., 5 respetively. If the memory is SAM, then the read aesses at times 0, 4, 5 should return values a,, respetively. ith our SAM emulation, exatly the same values a,, will be returned, albeit at times Δ, 4+Δ, 5+Δ respetively. input operations random address permutation request buffers DAM banks reservation table B data out time time op addr data b 2 op addr data b Fig. 1. a 0 a 0 Ideal SAM time SAM Emulation time data out 5 4 data out SAM Emulation 3 2 a Δ+5 Δ+4 Δ+3 Δ+2 Δ+1 Note the definition of emulation does not require a speifi ompletion time for a write operation beause nothing is returned from the memory system to the issuing proessor. As to the read operations, the emulation definition only requires the output sequene (i.e. sequene of read results) to be the same, exept time-shifted by a fixed Δ yles. The definition does not require the snapshot of the memory ontents to be the same. Therefore, the ontents in the DAM banks do not neessarily have to be the same as in an ideal SAM bank at all times. For example, as we shall see later in our extended memory arhiteture, with the two bak-tobak write operations in Fig. 1, only the data of the seond write operation is required to update the memory to provide ohereny, whereas all write operations orrespond to atual memory updates in an ideal SAM. III. THE BASIC MEMOY ACHITECTUE The basi memory arhiteture is shown in Fig. 2. It onsists of a reservation table, a set of DAM banks with a orresponding set of request buffers, and a random address permutation funtion. A. Arhiteture eservation Table The reservation table with Δ entries is implemented in SAM to provide for a fixed delay of Δ for all inoming operations. For eah operation arriving at time t, an entry (a row in Fig. 2) is reated in the reservation table at loation (t +Δ) modδ. Eah reservation table entry onsists of three parts a 1-bit field to speify the operation (read or write), the memory 1 a 0 Δ Fig. 2. Basi Memory Arhiteture address, and the data field. The data field stores the data to be written to or to be retrieved from the memory. DAM Banks and equest Buffers There are B>1/μ DAM banks in the system to math the SAM throughput. Also, there are B request buffers, one for eah DAM bank. Memory operations to the same DAM bank are queued at the same request buffer. Eah request buffer entry is just a pointer to the orresponding reservation table entry, whih ontains the information about the memory operation. Given the random permutation of memory addresses, we an statistially size the request buffers to ensure an extremely low probability of overflow. e defer to Setion V to disuss about this analysis. Let K be the size of a request buffer. Then the B request buffers an be implemented using multiple memory hannels. In eah hannel the request buffers are servied in a round-robin order and different hannels are servied onurrently. For eah hannel there is a dediated link to the reservation table. For example, with μ = 1/16 and B = 32 the request buffers an be implemented on two hannels of 16 banks on eah hannel. Therefore, eah request buffer takes 1/μ yles to be servied. A memory operation queued at a request buffer would take at most Δ=K/μ yles to omplete. e set the fixed pipeline delay in yles for the external memory interfae to this Δ. andom Address Permutation Funtion The goal of the random address permutation funtion is to randomly distribute memory loations so that memory operations to different memory loations are uniformly spread over B DAM banks with equal probability 1/B. Unless otherwise noted, when referring to a memory address, we will be referring to the address after random permutation. Note that if the inoming operations aess the same memory loation, then they will still generate entries to the same request buffer. B. Operations For a read operation issued at time t, its ompletion time is h = t +Δ. A reservation table entry is reated at loation h mod Δ. The data field is initially pending. A DAM read

4 4 operation gets inserted into the orresponding request buffer. hen a read operation at the head of a request buffer is servied, whih may be earlier than its ompletion time h, the orresponding data field of its reservation table entry gets updated. In this way, the reservation table effetively serves as a reorder buffer. At the ompletion time h of the read operation, data from orresponding reservation table entry gets opied to the output. The reservation table entry is removed after Δ yles. For a write operation issued at time t, a reservation table entry is reated at loation h mod Δ, where h = t+δ.also, a DAM write operation is inserted into the orresponding request buffer. hen the write operation at the head of a request buffer is servied, whih may be earlier than its ompletion time h, the write data is updated to the DAM address. Its reservation table entry is removed Δ yles after its arrival time t. By sizing the reservation table to have Δ entries, the lifetime of a read or write reservation entry is guaranteed to be longer than the time it takes for the DAM read or write to our respetively. C. Adversarial Aess Patterns Even though a random address permutation is applied, the memory loads to the DAM banks may not be balaned due to some adversarial aess patterns as follows. First, many appliations require the lookup of global variables. epeated lookups to a global variable will trigger repeated operations to the same DAM bank loation regardless of the random address permutation implemented. Seond, although attakers annot know how memory addresses are mapped to DAM banks, they an still trigger repeated operations to the same DAM bank by issuing memory operations to the same memory loations. Due to these adversarial aess patterns, the number of pending operations in a request buffer may grow indefinitely. To mitigate these situations, our extended memory arhiteture in Setion IV effetively utilizes the reservation table in SAM as a ahe to eliminate unneessary DAM operations queued in the request buffers. IV. THE EXTENDED MEMOY ACHITECTUE Fig. 3 depits our proposed extended memory arhiteture. As with the basi arhiteture, there is a reservation table, a set of DAM banks, and a orresponding set of request buffers. For eah memory operation, its ompletion time is still exatly Δ yles away from the perspetive of the issuing proessor. However, in ontrast to the basi memory arhiteture, we do not neessarily generate a new DAM operation to the orresponding request buffer for every inoming memory operation. In partiular, we an avoid generating new DAM operations in many ases by using the reservation table effetively as a data ahe. e will desribe in more details in our operation merging rules later in this setion. At the high-level, our goal is to merge operations that are issued to the same memory loation within a window of C yles. This is ahieved by extending the basi arhiteture input operations random address permutation Fig. 3. request buffers MI table (CAM) reservation table C DAM banks -link -link -link -link M table (CAM) Extended Memory Arhiteture in the following ways. First, the size of the reservation table an be set to any C Δ entries to ath mergeable operations that are issued at most C yles apart. Seond, eah reservation table entry is expanded to ontain the following information Pending Status p A memory read operation is said to be pending with p =1if there is a orresponding entry in the request buffer, or if it is linked to an operation that is still pending. Otherwise, the read operation is nonpending with p =0. The memory write operations in the reservation table are set as pending with p =1. -link Pending read operations to the same memory address are linked together into a linked list using the -link field, with the earliest/latest read operations at the head/tail of the list respetively. In addition to the additional fields in the reservation table, two lookup tables are added to the extended arhiteture a Most eently Issued (MI) lookup table and a Most eent rite (M) lookup table. Both of these tables are implemented using Content-Addressable-Memories (CAM) to enable diret lookup. The MI table keeps trak of the most reently issued operation (either read or write) to a memory address. hen a new read operation arrives, an MI lookup is performed to find the most reent operation in the reservation table. The M table keeps trak of the most reent write operation to a memory address. hen a write operation in the reservation table is removed, an M lookup is performed to hek if there is another more reent write operation to the same memory address. For an MI lookup, if there is a mathed entry, it returns a pointer to the row in the reservation table that ontains the most reent operation to the same memory address. Similarly, an M lookup returns a pointer to the row in the reservation table orresponding to the most reent write operation for a given memory address. In the extended arhiteture, eah entry in the request buffers needs a data field for write operations. For read operations in the request buffers, a request buffer entry serves as a pointer to the orresponding reservation table entry. But for write operations in the request buffers, a request buffer p p p p B C C data out

5 5 entry stores the atual data to be written to a DAM bank. A. eservation Table Management A reservation table entry gets freed when the operation in the entry stays in the reservation table for C yles. For an operation arrived at time t, its reservation table entry will be freed at time g = t + C. To ahieve this, we maintain two pointers pointer h as a ompletion time pointer, and pointer g as a garbage olletion time pointer. hen the urrent time is g, then the orresponding reservation table entry is freed (meaning the entry is leared). On a MI (or M) lookup, if it returns a reservation table entry with different address (addr), then we also remove this MI (or M) entry. B. Memory Operations hen a read operation () arrives, a new entry in the reservation table is reated for. Then we have the following three ases Case 1 If the MI lookup fails, whih means there is no previous operation to the same memory address in the reservation table, we then generate a new read operation to the orresponding request buffer. A new MI entry is reated for this operation also, and its pending status p is set to 1. hen the atual DAM read operation finishes, the data field of in the reservation table entry is updated, and p is reset to 0. Case 2 The most reent operation to the same memory address returned by the MI lookup is a pending read operation. In this ase, we follow these steps 1) e reate a -link (shown as the dashed line in Fig. 4) from this most reent read operation to and set its pending status p to 1. e do not reate a new read operation in the request buffers. 2) The MI entry is updated with the loation of in the reservation table. Essentially, a linked list of read operations to the same address is formed, with head/tail orresponding to the earliest/latest read, respetively. At the ompletion time of a read operation, we opy the read data to the next read operation by following its -link, and we reset p to 0. Therefore, only the first read operation in the linked list generates an atual read operation in the request buffer. The remaining read operations simply opy the data from the earlier read operation in the linked list order. An example of the management of -links for read operations is shown in Fig. 4. pending Fig. 4. -Link most reent request -Link -link for read operation new Case 3 The most reent operation to the same address returned by the MI lookup is a write operation or a non-pending read operation. In this ase, we opy the data from this most reent write or non-pending read operation to the new loation. In this way, is guaranteed to return the most reent write or non-pending read data. The MI entry is updated by pointing to this new operation. hen a read operation reahes its fixed delay Δ and departs from the system, its data is sent to the external data out. The orresponding reservation table entry is removed after C yles. Also, if there is an entry in MI that points to the reservation table entry, then this MI entry is removed also. hen a new write operation () arrives, a new entry in the reservation table is reated for. Also, we have the following three ases Case 1 If the MI lookup fails, we reate a new write operation into the orresponding request buffer, and we reate a new M and MI entry for this new operation. Case 2 If the MI lookup returns an entry in the reservation table, then we reate an entry in the reservation table for. Also, an M lookup is performed. If the M lookup returns an entry in the reservation table, then the orresponding entries in the MI and M are updated by pointing to the new operation in the reservation table. Otherwise, if the M lookup does not find an entry, then the orresponding entry in MI is updated to and a new entry is reated in M for. A write operation is removed from the reservation table after C yles. An M lookup is performed to hek if there is a more reent write operation in the reservation table. If there is suh a write operation (i.e. the M lookup returns a different memory address), then no request buffer is reated for. Otherwise, there is no other write operation in the reservation table, and the M lookup returns the address of operation. Then a new entry in the request buffer is generated for operation to update the orresponding DAM bank loation. Also, we delete the entry in M pointing to the operation. C. Operation Merging ules Based on the desriptions of the read and write operations above, effetively we are performing the following merging rules to avoid introduing unneessary DAM operations. The operations arrive in the order from right to left on the left-hand-side of the arrows. The atual memory operation(s) generated in the request buffers are shown on the right-handside of the arrows. 1) The operation opies data diretly from the operation stored in the reservation table, thus it is not inserted into the request buffers. 2) The earlier (right) operation does not generate new entry in the request buffers. 3) The latter (left) operation is linked to the earlier operation by its -link. No entry is reated in the request buffer for the latter operation.

6 6 4) Both the operation and the operation have to aess the DAM banks. They an not be merged. Entries in the request buffer needs to be reated for the and operations. For the last rule,, even though the operations annot be merged, the latter inoming operation to the same memory address an be merged. Consider the following two ases (input sequene from right to left) () () The last two () merged to () using the rule. () () The last two () merged to () using the rule. ith the above merging rules, data ohereny is maintained. Sine the reservation table has already provided fixed delay Δ for all the read operations, to show data ohereny, we just need to prove that all the read operations will output the orret data. e will prove this by showing in the following that the relative ordering of the read and write operations are preserved in our arhiteture, whih means that the read operations output the same data as in an SAM system. In partiular, we will fous on a write operation and show that operation merging rules will not affet the read operations before or after the write operation. Let s onsider the following ases 1) A newly arrived write operation will not affet the data retrieved by the read operations before it, sine the write operation will not be inserted into the request buffer until C yles later, by whih time all the read operations before have already finished. 2) For read operations arriving after a write operation in the reservation table, with no other writes in between, they will read the orret data diretly from. 3) For read operations arriving after a write operation in the reservation table, with other writes in between, they will read the orret data from the most reent write operation. 4) A write operation is removed from the reservation table C yles after it arrives. Upon removal, if there is another most reent write operation in the reservation table, the newly arrived read operations will read the orret data from. Therefore, the removal of will not affet the data of the future read operations. 5) Upon the removal of a write operation in the reservation table, if there is no other more reent write operation, but there is a most reent read operation, then an entry is generated in the request buffer for to update the data in the orresponding DAM bank. All future inoming read operations will read the orret data from diretly or from the DAM bank. Sine is already in the request buffer, future inoming read operations will generate entries in the request buffer only after, and thus they will read the orret data from the memory banks. 6) Upon the removal of a write operation in the reservation table if there is no other operations aessing the same DAM loation in the reservation table, then an entry is generated in the request buffer for to update the memory banks. All the future read operations will only generate entry in the request buffer after, thus they will be able to retrieve the orret data. In summary, the read operations always return the orret data with a fixed delay C, and therefore the system is data oherent. Further, using the proposed merging rules, we an ensure the following results There an be only one write operation in the request buffer every C yles to a partiular memory address. A write operation is generated in the request buffer only when there is a write operation in the reservation table for C yles and there is no more reent write operation in the reservation table during this period of time. So a write operation is generated in the request buffer at most one every C yles. There an be at most one read operation in a request buffer every C yles to a partiular address. hen there are more than one arriving read operations to the same address within C yles, we are guaranteed that all exept the first read are merged using -link. There an be at most one read operation followed by one write operation in a request buffer every C yles to a partiular address. If there is a read operation following a write operation, then the read operation an get data diretly from the write operation. Therefore, no new entry will be generated in the request buffer for the read operation. V. ANALYSIS In this setion, we prove the main theoretial result of this paper, whih bounds the probability that any of the request buffers will overflow for all (inluding any adversarial) sequenes of read/write operations. As explained earlier, this worst-ase large deviation result is proved using a novel ombination of onvex ordering and (traditional) large deviation theory. The main idea of our proof is as follows. Given any sequene of read/write operations (to the DAM banks) over a time period [s, t] (viewed as a parameter setting), we are able to obtain a tight stohasti bound of the number of arrivals of read/write operations to the request buffer of a DAM bank during [s, t] using various tail bound tehniques. Sine our sheme has to work with all possible sequenes, our bound learly has to be the worst ase (i.e. the maximum) stohasti bound over all of them. However, the spae of all suh sequenes is so large that enumeration over all of them is omputationally prohibitive and low omplexity optimization proedures in finding the worst ase does not seem to exist. Fortunately, we disover that the aforementioned number of arrivals under all these sequenes are dominated by that under a partiular (i.e., worst-ase) sequene, in the onvex order (not in the stohasti order). Sine e θx is a onvex funtion, we are able to upper-bound the moment generating funtions (MGF) of the number of arrivals under all other sequenes

7 7 by that under the worst-ase sequene. However even this worst-ase MGF is prohibitively expensive to ompute. e solve this problem through upper-bounding this MGF by a omputationally friendly formula, then applying Chernoff tehnique to it. The rest of this setion is organized as follows. In Setion V-A, we desribe the overall struture of the tail bound problem, whih shows that the overall overflow event D over time period [0,n] is the union of a set of the overflow events D s,t, 0 s<t n, whih leads to a union bound. In Setion V-B, we show how to bound the probability of eah individual event D s,t using the aforementioned tehnique of ombining onvex ordering with large deviation. A. Union Bound The First Step In this setion we bound the probability of overflowing request buffer Q of a partiular DAM bank. Let D0,n be the event that one or more operations are dropped beause Q is full during time interval [0,n] (in units of yles). This bound will be established as a funtion of system parameters K, B, μ, and C. eall that K is the size of the request buffer, B is the number of DAM banks, μ is the SAM-to-DAM random aess lateny ratio, C is the size of the ahe. In the following, we shall fix n and will therefore shorten D 0,n to D. Note that Pr[ D] is the overflow probability for just one out of B suh request buffers. The overall overflow probability an be bounded by B Pr[ D] (union bound). e first show that Pr[ D] is bounded by the summation of probabilities Pr[D s,t ], 0 s t n, that is, Pr[ D] Pr[D s,t ]. (1) 0 s t n Here D s,t, 0 s<t n, represents the event that the number of arrivals during the time interval [s, t] is larger than the maximum possible number of departures in the queue, by more than the queue size K. Formally letting X s,t denote the number of read/write operations (to the DAM bank) generated during time interval [s, t], then we have D s,t {ω ΩX s,t μ(t s) >K}. Here we will say a few words about the impliit probability spae Ω, whih is the set of all permutations on {1,..., N}, where N is the number of distint memory addresses. Sine we are onsidering the worst ase bound, we assume the maximum number of read/write operations that an be generated to request buffers during time interval [s, t]. Letτ = t s, and let the maximum number be τ. hen τ C, sine eah request an result in at most one read and one write bak, we have τ =2τ. hen τ > C, there an be at most C write baks that annot be aounted for by one of the writes among the τ requests, so τ = τ + C. Combining both we get τ = τ + min(τ,c) (2) e assume that the operations to the request buffers are generated following arbitrary pattern, with the only restrition that read requests to the same address an not repeat within C yles, and write requests to the same address an not repeat within C yles. This is due to the smoothing effet of the reservation table, i.e. repetitions within C yles would be absorbed by the reservation table. Given an arbitrary sequene of operations satisfying the above restrition, then eah instane ω Ω gives us an arrival sequene to the DAM request buffers. The inequality (1) is a diret onsequene (through the union bound) of the following lemma, whih states that if the event D happens, at least one of the events {Ds,t } 0 s<t n must happen. Lemma 1 D = 0 s t n D s,t e omit its proof here as it is straightforward from elementary queueing theory and is idential to that of [10, Lemma 1]. B. Bounding Individual Pr[D s,t ] In this subsetion we find the worst-ase read/write operation sequene for deriving tail bounds for individual Pr[D s,t ] terms. The probability Pr[D s,t ] is learly a (random) funtion of the sequene of read/write operations (viewed as parameters) during the interval [s, t]. As mentioned before, it is not possible to enumerate over all possible parameter settings (i.e., sequenes) to find the worst-ase Pr[D s,t ] bound. Fortunately, onvex ordering omes to our resue by allowing us to analytially bound the MGF of X s,t under all parameter settings by that under a worst-ase setting. For simpliity, in this setion we will drop the subsripts of X s,t and use X instead. 1) Mathematial Preliminaries In the following, we first desribe the standard Chernoff tehnique for obtaining sharp tail bounds from the MGF of a random variable (in this ase X) 1. Pr[D s,t ] = Pr[X>K+ μτ] =Pr[e Xθ >e (K+μτ)θ ] E[eXθ ] e. (K+μτ)θ where θ>0is any onstant, and the last step is due to Markov inequality. Here τ is defined as t s. Sine this is true for all θ, wehave Pr[D s,t ] min θ>0 E[e Xθ ]. (3) e (K+μτ)θ Then, we aim to bound the moment generating funtion E[e Xθ ] by finding the worst-ase sequene. Note that we resort to onvex ordering, beause stohasti order, whih is the onventional tehnique to establish ordering between random variables and is stronger than onvex order, does not hold here, as we will show shortly. Sine onvex ordering tehniques are needed to establish the bound, we present the definition of onvex funtion and onvex ordering here Definition 2 (Convex funtion) A real funtion f is alled onvex, iff(αx +(1 α)y) αf(x)+(1 α)f(y) for all x and y and all 0 <α<1. 1 This tehnique was apparently first used by Bernstein.

8 8 Definition 3 (Convex order [11, 1.5.1]) Let X and Y be random variables with finite means. Then we say that X is less than Y in onvex order (written X x Y ), if E[f(X)] E[f(Y )] holds for all real onvex funtions f suh that the expetations exist. 2) orst-case Parameter Setting In this setion, we speify the worst-ase parameter setting (in the sense of onvex ordering) and prove it is indeed the worst-ase. Let m i, 1 i N be the total number of read and write operations generated to the request buffers for the i th address during time interval [s, t]. So N i=1 m i = τ, where τ is defined in (2). e have X = N i=1 m ix i, where X i is the indiator random variable for whether the i th address is mapped to the DAM bank. e have E[X i ]= 1 B.Butthe X i s are not independent sine we are doing permutation on the addresses. As explained earlier, among operations generated to request buffers, read requests to the same address an not repeat within C yles, and write requests to the same address an not repeat within C yles. Therefore none of the ounts m 1,..., m N an exeed 2T, where T = τ C. One an ahieve m i = 2T by issuing pairs of a read operation followed by a write operation for the i th address every C yles. 2 Moreover, let q 1 = τ (T 1)C, then at most q 1 addresses ould have ount 2T. e all any vetor m = {m 1,..., m N } a valid splitting pattern of τ if the following are satisfied 0 m i 2T, N i=1 m i = τ, {i m i =2T } q 1.LetMbe the set of all valid splitting patterns. 3 Let X m = N i=1 m ix i. Let q 2 = (τ 2Tq 1 )/(2T 1), r = τ 2Tq 1 (2T 1)q 2. Let m be suh a splitting pattern m 1 =... = m q1 =2T, m q1+1 =... = m q1+q 2 =2T 1, m q1+q 2+1 = r, and the rest of m i are 0. e omit the proof of the following theorem sine it is similar to that of [10, Theorem 2]. Theorem 1 m is the worst ase splitting pattern in terms of onvex ordering, i.e. X m x X m, m M. emark Note that stohasti order does not hold here, sine E[X m ]=E[X m ]=τ /B. For stohasti order to hold between two random variables of different distributions, their expetations must differ [11, Theorem 1.2.9]. Unfortunately, it is in general not possible to apply the Chernoff bound diretly to the MGF of X m,asx m is the sum of dependent random variables and is very expensive to ompute. Our solution is to find a way to upper-bound E[e X m θ ] by a more omputationally friendly formula. For this purpose we use a lemma by Hoeffding [12, Theorem 4], whih bounds the outome of sampling without replaement by that of sampling with replaement in the onvex order. Then we apply the Chernoff tehnique to obtain the following bound. e omit the proof here sine it is similar to those of [10, Theorem 3] and [10, Theorem 4]. 2 Stritly speaking it is every C +1yles. 3 Not all valid splitting pattern may have a plausible read/write sequene mathing it, but this does not affet out bound. Theorem 2 ( 1 B For τ C, Pr[D s,t ] min e2θ +(1 1 B ))τ ; θ>0 e (K+μτ)θ For τ>c, e (q1e2tθ +q 2e (2T 1)θ +e rθ q 1 q 2 1)/B Pr[D s,t ] min. θ>0 e (K+μτ)θ e an see that the bound is translation invariant, i.e. it only depends on τ = t s. Therefore, the omputation ost of (1) is O(n) instead of O(n 2 ), where n is the length of the total time interval. In onlusion, we have established the bound for the overflow probability under any read or write sequenes. It an be omputed though O(n) number of numerial minimizations for one-dimensional funtions expressed in Theorem 2. VI. EVALUATION In this setion, we present evaluation results for our proposed extended memory arhiteture desribed in Setion IV. In partiular, we used parameters derived from two realworld Internet traffi traes for our evaluations. The traes that we used were olleted at different loations in the Internet, namely University of Southern California (USC) and University of North Carolina (UNC), respetively. The trae from USC was olleted at their Los Nettos traing faility on February 2, 2004, and the trae from UNC was olleted on a 1 Gbps aess link onneting the ampus to the rest of the Internet on April 24, The trae from USC has million pakets and around 8.6 million flows, and the trae segment from UNC has million pakets and around 13.5 million flows. To support suffiient data storage for both traes, we set the number of addresses in the DAM banks to be N =16million. Overflow Probability Bound C=6000 C=7000 C=8000 C= Queue Length K Fig. 5. Overflow probability bound as a funtion of request buffer size K with µ =1/10 and B =32. In Fig. 5, the overflow probability bounds with different reservation table sizes C as a funtion of request buffer sizes K are presented, where μ = 1/10 and B = 32. As K inreases, the overflow probability bound dereases. ith

9 9 C 8000 we an ahieve an overflow probability bound of starting from a request buffer of size K = 180. Overflow Probability Bound B=32 B=34 B=36 B= equest Buffer Size K Fig. 6. Overflow probability bound as a funtion of number of memory banks B with µ =1/10 and C = In Fig. 6, the system overflow probability bounds with different numbers of memory banks B as a funtion of queue length K are presented, where μ =1/10 and C = It an be seen from this figure that given the same K as B inreases, the overflow probability bound dereases. Now let s onsider the size of the reservation table. For eah entry in a reservation table, one bit is for op to distinguish read and write operations. ith N =16millions entries in the DAM banks, addr of size log 2 N =24bits is suffiient to address every memory loation. The size of -link is log 2 C =13bits, with C = Moreover the size of the pending status p is 1 bit. Let the data size be 8 bytes or 64 bits. Altogether the total size of eah entry in the reservation table is 103 bits. For a reservation table with C = 8000 entries, its total size is about 101 KB, whih an be easily implemented in SAM. For the MI and M tables, only pointers are stored to enable fast searhing on the reservation table entries. Eah entry in the MI or M table is of size log 2 N =24bits, where N is the number of addresses in the DAM banks. There an be at most C entries in the MI or M, where C is the size of the reservation table. ith C = 8000, the total size of MI or M is about 24 KB, whih an be easily implement in CAMs. In the request buffers, eah entry is a pointer to an entry in the reservation table plus a data field for write operation. An entry in the request buffers is of size log 2 C.ForC = 8000, the size of the pointer in the request buffer is 13 bits. Let the size of the data field be 8 bytes. Let B =32and K = 180 to provide with overflow probability, the total size of the request buffers is only about 55 KB. It is worth noting that our evaluations are based on the assumption of worst ase senarios where the requests to the same memory loations are repeated every C yles. For real-world traffi the assumption above is far too pessimisti. e expet that muh smaller request buffers (K) and muh smaller reservation table size (C) will be suffiient for most real-world Internet traffi, whih will result in muh smaller delay Δ, where Δ=K/μ. VII. CONCLUSION e proposed a memory arhiteture for high-end Internet routers that an effetively maintain wirespeed read/write operations by exploiting advaned arhiteture features that are readily available in modern ommodity DAM arhitetures. In partiular, we presented an extended memory arhiteture that an harness the performane of modern ommodity DAM offerings by interleaving memory operations to multiple memory banks. In ontrast to prior interleaved memory solutions, our design is robust to adversarial memory aess patterns, suh as repetitive read/write operations to the same memory loation, by using only a small amount of SAM and CAM. e presented a rigorous theoretial analysis on the performane of our proposed arhiteture in the worstase using a novel ombination of onvex ordering and large deviation theory. Our arhiteture supports arbitrary read and write patterns at wirespeed of 40 Gb/s or beyond. Aknowledgement This work is supported in part by ollaborative NSF grants CNS and CNS , funded under the Amerian eovery and einvestment At of 2009 (Publi Law 111-5), and NSF grant CNS EFEENCES [1] S. Iyer and N. Mkeown, Designing buffers for router line ards, Stanford University, Teh. ep. T02-HPNG , Mar [2] J. Gara, J. Corbal, L. Cerd, and M. Valero, Design and implementation of high-performane memory systems for future paket buffers, in International Symposium on Miroarhiteture (MICO), [3] G. Shrimali and N. MKeown, Building paket buffers using interleaved memories, in orkshop on High Performane Swithing and outing (HPS), May [4] S. Kumar, P. Crowley, and J. Turner, Design of randomized multihannel paket storage for high performane routers, in Symposium on High Performane Interonnets (HOTI), [5] A. K. Parekh and. G. Gallager, A generalized proessor sharing approah to flow ontrol in integrated servies networks the singlenode ase, IEEE/ACM Trans. Networking, vol. 1, no. 3, pp , [6] B.. au, Pseudo-randomly interleaved memory, in Pro. 18th Annual International Symposium on Computer Arhiteture, [7]. Lin, S. K. einhardt, and D. Burger, eduing DAM latenies with an integrated memory hierarhy design, in Pro. of IEEE HPCA, ashington, DC, USA, 2001, p [8] B. Agrawal and T. Sherwood, Virtually pipelined network memory, in Pro. 39th Annual IEEE/ACM International Symposium on Miroarhiteture (MICO). ashington, DC, USA IEEE Computer Soiety, 2006, pp [9] C. Pandit and S. Meyn, orst-ase large-deviation asymptotis with appliation to queueing and information theory, Stohasti Proesses and their Appliations, vol. 116, no. 5, pp , [10] H. Zhao, H. ang, B. Lin, and J. Xu, Design and performane analysis of a dram-based statistis ounter array arhiteture, in ACM/IEEE Symposium on Arhitetures for Networking and Communiations Systems (ANCS), [11] A. Muller and D. Stoyan, Comparison Methods for Stohasti Models and isks. iley, [12]. Hoeffding, Probability inequalities for sums of bounded random variables, Journal of the Amerian Statistial Assoiation, vol. 58, no. 301, pp , 1963.