FREQUENCY ESTIMATION OF INTERNET PACKET STREAMS WITH LIMITED SPACE: UPPER AND LOWER BOUNDS

Transcription

1 FREQUENCY ESTIMATION OF INTERNET PACKET STREAMS WITH LIMITED SPACE: UPPER AND LOWER BOUNDS Prosejit Bose Evagelos Kraakis Pat Mori Yihui Tag School of Computer Sciece, Carleto Uiversity {jit,kraakis,mori,y ABSTRACT. We cosider the problem of approximatig the frequecy of frequetly occurig elemets i a stream of legth usig oly a memory of size m. 1 Itroductio We cosider the problem of processig a data stream x 1,..., x of packet classes i oe pass. This models the process of gatherig statistics o Iteret packet streams usig a memory that is small relative to the umber of classes or categories of packets. More formally, we cosider packet coutig algorithms that process the stream x 1,..., x oe item at a time. A packet coutig algorithm has a memory of fixed-size ad has access to m iteger couters, each of which ca be labelled with a packet class. If a couter is labelled with some packet class a the we say that couter is moitorig a. While processig a item, the algorithm may modify its memory, perform equality tests o packet classes, icremet or decremet couters ad chage the labels of couters. However, other tha comparig packet classes ad storig them as couter labels, the algorithm may ot do ay other computatios o or storage of packet classes. After the algorithm completes, the couter value for a packet class a is the value of the couter moitorig a. If o couter is moitorig a the the couter value for a is defied to be zero. The problem of accurately maitaiig frequecy statistics i a data stream has applicatios i Iteret routers ad gateways, which must hadle cotiuous streams of data that are much too large to store ad postprocess later. As a example, to implemet fairess policies oe might like to esure that o user (IP address) of a router or gateway uses more tha 1% of the total available badwidth. Keepig track of the idividual usage statistics would require (at least) oe couter per user ad there may be tes of thousads of users. However, the results i this paper imply that, usig oly 99 couters, we ca idetify a set of users, all of whom are usig more tha 1% of the available badwidth ad which cotais every user usig more tha 2% of the badwidth. If more accuracy is required, we could use 990 couters, ad the threshold values become 1% ad 1.1%, respectively. Motivated maily by applicatios like the oe described above, there is a growig body of literature o algorithms for processig data streams [1, 2, 3, 5, 6, 8, 9, 10, 11]. A early work, particularly relevat to the curret paper, is the work of Fischer ad Salzberg [7] who showed that, usig oe couter ad makig oe pass through a data stream, it is possible to determie a class a such that, This work was partly fuded by the Natural Scieces ad Egieerig Research Coucil of Caada. 1

2 if ay elemet occurs more tha /2 times, the it is a. Demaie et al. [4] showed that Fischer ad Salzberg s algorithm geeralizes to a algorithm which they call FREQUENT. The FREQUENT algorithm uses m couters ad determies a set of m cadidates that cotai all elemets that occur more tha /(m + 1) times. Although ot explicitly metioed, it is implicit i their proof that, whe FREQUENT termiates, the couter value c a for a packet class a that occurs a times obeys c a a c a + m + 1. Other work o the particular problem of estimatig frequecies i packet streams icludes the work of Fag et al. [6] who propose heuristics to compute all values above a certai threshold. Charikar et al. [2] give algorithms for computig the top k cadidates uder the Zipf distributio. Esta ad Varghese [5] attempt to idetify a set of packet classes that are likely to cotai the most frequetly occurig packet classes, ad give probabilistic estimates of the expected cout value i terms of a user selected threshold. I this paper we are cocered with the accuracy of packet coutig algorithms. We say that a packet coutig algorithm is k-accurate if, for ay class a that appears a times, the algorithm termiates with a couter value c a for a that satisfies c a a c a + k. (1) Therefore, the FREQUENT algorithm is /(m + 1)-accurate. I geeral, o algorithm is better tha /(m + 1) accurate sice, if m + 1 classes each occur /(m + 1) times the oe of those classes will have a couter value of 0 whe the algorithm termiates. However, this argumet breaks dow whe we cosider the case whe some particular packet class a occurs a α times, for some α > 1/(m+1). I this case, it may be possible for the algorithm to report the umber of occureces of a (ad other elemets) more accurately. We explore this relatioship betwee accuracy ad α. Our results are outlied i the ext paragraph. I Sectio 2 we show that the FREQUENT algorithm of Demaie et al. is (1 α)/m-accurate, where α is the umber of times the most frequetly occurig packet class appears i the stream. I Sectio 3 we give a lower-boud of (1 α)/(m + 1) o the accuracy of ay determiistic packet coutig algorithm ad a lower boud of (1 α)ω(/m) o the accuracy of ay radomized packet coutig algorithm. This latter result solves a ope problem posed by Demai et al. [4] about whether radomized packet coutig algorithms ca be more accurate tha determiistic oes. I Sectio 4 we summarize ad coclude with ope problems. 2 The FREQUENT Algorithm The FREQUENT algorithm of Demaie et al. [4] uses m couters. Whe processig stream item x i, the followig rules are applied i order: 1. If there is a couter moitorig class x i the icremet that couter, otherwise 2. if some couter is equal to 0 the set that couter to 1 ad have it moitor class x i, otherwise 2

3 3. decremet all couters by 1. A ice way to visualize this algorithm is to imagie a set of m buckets that hold colored balls. Whe a ew ball arrives we either place it i the bucket that cotais balls of the same color (Case 1), place it i a empty bucket (Case 2) or discard oe ball from every bucket as well as the ew ball (Case 3). To aalyze the accuracy of this algorithm, we first provide a rough upper-boud o the accuracy ad the use this upper-boud to bootstrap a better aalysis. Let d be the total umber of times Case 3 of the algorithm occurs. No couter is ever less tha 0, ad each of Case 1 ad Case 2 icremets exactly oe couter. Therefore, if C is the total sum of all couters whe the algorithm termiates, the C = (m + 1)d 0, so that d m + 1. It follows immediately that for ay class a that occurs a times, the couter moitorig a has a value of at least c a a d a m + 1. Suppose that a = α for some α 1/(m + 1). Now we ca repeat the above argumet, sice we have just show that C = (m + 1)d α m + 1, so that (1 α) d m (m + 1) 2. ad the value of c a satisfies ( ) (1 α) c a α d α m (m + 1) 2. I geeral, we ca repeat the above argumet k times to show that c a α k i=1 (1 α) (m + 1) i (m + 1) k+1. I particular, as k, we obtai c a α (1 α)/m. Now, sice c a is clearly ever greater tha a, we have (1 α) c a a c a +, m so the output c a is (1 α)/m-accurate. Fially, we observe that the above aalysis gives a upper-boud o d, ad this gives a upper boud o the accuracy of the couter value for a. However, the upper boud o d also gives a upper boud o the accuracy of ay couter, ot just the couter for a. This implies our first result. Theorem 1. For ay stream i which some elemet occurs at least α times, the FREQUENT algorithm is (1 α)/m-accurate. 3

4 abcd yabcd y abcd yaaaaaa a abcd y }{{} abcd }{{ y } abcd y }{{} zzzzzz }{{ z } m + 1 m + 1 m + 1 α Figure 1: The adversary s two streams. 3 Lower-Bouds o Accuracy I this sectio we give lower bouds o the accuracy of determiistic ad radomized packet coutig algorithms. 3.1 A Determiistic Lower-Boud Here we give a lower boud for determiistic packet coutig algorithms by usig a adversary argumet. Our adversary builds two distict streams that the algorithm caot distiguish betwee. Our adversary uses m + 2 packet classes ad builds its streams i two parts (see Figure 1). The first part of both streams is of legth (1 α) ad cosists of the first m + 1 packet classes each occurig the same umber of times, so that each class occurs (1 α)/(m+1) times. At this poit the two streams diverge. I the first stream, the adversary adds α occureces of the uique packet class a of the m + 1 first classes that is ot beig moitored by the algorithm after processig the first part of the stream. I the secod stream, the adversary adds α occureces of the uique packet class z that does ot appear i the first part of the stream. Observe that, sice either a or z is stored i ay of the algorithm s couters after processig the first part of the stream, the oly iformatio the algorithm obtais by readig the last elemet of the stream is that it is ot beig moitored. Therefore, sice the algorithm is determiistic, its couter value c a for a o the first stream will be equal to its couter value c z for z o the secod stream. However, i the first stream a occurs a = (1 α)/(m + 1) + α times ad i the secod stream, z occurs z = α times. I order to be accurate at all (refer to (1)) the algorithm must termiate with a couter value c a = c z z. But i this case, the algorithm is ot better tha (1 α)/(m + 1)-accurate for the first stream. Theorem 2. For ay determiistic algorithm, there exists a stream i which some symbol a occurs a α times, but the algorithm reports a value c a such that c a > a or c a a (1 α)/(m + 1). 3.2 A Radomized Lower-Boud Next we give a lower boud for radomized algorithms. We do this by providig a probability distributio o iput streams such that the expected accuracy of ay determiistic algorithm o this distributio is at least (1 α)c/m. Sice ay radomized algorithm is just a probability distributio o determiistic algorithms, the lower-boud therefore holds for radomized algorithms as well. 1 The distributio we 1 Techically, this is a applicatio of Yao s Priciple [12]. 4

5 use is a probabilistic versio of our determiistic costructio. Our distributio uses two costats 1 < c 1 < that will be specified later. Each stream of our distributio is a two part data stream made up of m packet classes. The first part of all streams is idetical. As before, it is of legth (1 α), ad it cosists of the first c 1 m packet classes each occurig a equal umber of times, so that each class occurs (1 α)/c 1 m times. For the secod part of the sequece, we select a packet class uiformly at radom from all m classes ad make that class occur α times. Let a be the packet class chose to make up the secod part of the sequece. Immediately after the first part of the sequece has bee processed by the algorithm, there are three cases to cosider: 1. The algorithm has a couter that is moitorig a. Sice the algorithm has oly m couters, this happes with probability at most p 1 m m = 1, ad the umber of occureces of a is 1 = (1 α)/c 1 m + α. 2. The algorithm does ot have a couter moitorig a ad a comes from the first c 1 m packet classes. This happes with probability at least p 2 (c 1 1)m m = c 1 1, ad the umber of occureces of a is also 2 = (1 α)/c 1 m + α. 3. The class a does ot come from the first c 1 m packet classes (so the algorithm is ot moitorig a). This happes with probability p 3 = 1 c 1, ad the umber of occureces of a is 3 = α. Let c a be the value output by the algorithm for class a. Sice we are provig a lower-boud, we ca assume that i Case 1, the algorithm aswers with perfect accuracy, i.e., c a = (1 α)/c 1 m + α. However, if the algorithm is ot moitorig class a (Cases 2 ad 3) the it caot distiguish betwee Cases 2 ad 3. Sice the algorithm is determiistic, if must output the same couter value c a i both cases. Therefore, the expected error made by the algorithm is at least E [ c a a ] p p 2 c a 2 + p 3 c a 3 ( ) p 2 (1 α) c a c 1 m + α + p 3 c a α ( ) = p 2 (1 α) x a + p 3 x a c 1 m c ( ) ( 1 1 (1 α) x a + 1 c ) 1 x a, c 1 m where x a = c a α. Settig c 1 = 1 + 2/2, = 1 + 2, ad simplifyig we obtai ( ( ) 2 (1 α) E [ c a a ] 2(1 + 2) x a (1 + + x a ) 2/2)m 5

6 2 2(1 + 2) ( (1 α) (1 + 2/2)m ) (1 α)/m Theorem 3. For ay radomized algorithm, there exists a stream i which some symbol a occurs a α times, but the algorithm has a couter value c a such that E a c a (1 α)/m. We observe that the proof of Theorem 3 exteds to a slightly more powerful model i which the packet coutig algorithm is allowed to periodically output class/value pairs of the form (a, c a ) whose meaig is a has occured c a times ad the couter value for a is cosidered to be the last such value output. A similar model is used by Demaie et al. [4] to study probabilistic packet streams. To see that the lower-boud carries over, observe that the last such pair (a, c a ) is either output before the secod part of the stream begis, or after. I the latter case, the argumet above shows that E [ a c a ] = Ω((1 α)/m). I the former case, the algorithm outputs the value c a without havig see the fial α occureces of a. A argumet similar to the oe above shows that, i this case, there is a packet class a such that E [ a c a ] = Ω(α). 4 Coclusios We have studied the problem of approximatig the frequecy of items i a data stream usig a fixed umber, m, of couters. We have show that whe some data item a occurs α times i a stream of legth, the the FREQUENT algorithm of Demaie et al. [4] is (1 α)/m-accurate. This is early optimal for a determiistic algorithm sice we have show that o determiistic algorithm is better tha (1 α)/(m + 1)-accurate. Fially, we have show that radomized algorithms ca ot be sigificatly more accurate sice ay radomized algorithm has a expected accuracy of at least (1 α)ω(/m). The mai ope problem left by our research is that of determiig if the costat factor i the accuracy of the FREQUENT algorithm ca improved by somehow itroducig radomizatio. It may well be the case that ruig FREQUENT o a radom sample of the origial iput stream is eough to foil a adversary ad improve its accuracy. Refereces [1] N. Alo, Y. Matias, ad M. Szegedy. The space complexity of approximatig the frequecy momets. I Proceedigs of the 28th ACM Symposium o the Theory of Computig (STOCS 96), pages 20 29, [2] M. Charikar, K. Che, ad M. Farach-Colto. Fidig frequet items i data streams. I Proceedigs of the 19th Iteratioal Colloquium o Automata, Laguages ad Programmig, pages , [3] M. Datar, A. Giois, P. Idyk, ad R. Motwai. Maitaiig stream statistics over slidig widows. I Proceedigs of the 13th Aual ACM-SIAM Symposium o Discrete Algorithms (SODA 2002), pages , [4] E. D. Demaie, A. López-Ortiz, ad J. I. Muro. Frequecy estimatio of iteret packet streams with limited space. I Proceedigs of the 10th Aual Europea Symposium o Algorithms (ESA 2002), pages ,

7 [5] C. Esta ad G. Varghese. New directios i traffic measuremet ad accoutig. I Proceedigs of the ACM SIGCOMM Iteret Measuremet Workshop, [6] M. Fag, S. Shivakumar, H. Garcia-Molia, R. Motwai, ad J. Ullma. Computig iceberg queries efficietly. I Proceedigs of the 24th Iteratioal Coferece o Very Large Databases, pages , [7] M. J. Fischer ad S. L. Salzberg. Fidig a majority amog votes: Solutio to problem 81-5 (Joural of Algorithms, jue 1981). Joural of Algorithms, 3(4): , [8] P. Gupta ad N. McKeow. Packet classificatio o multiple fields. I Proceedigs of ACM SIGCOMM, pages , [9] P. J. Haas, J. F. Naughto, S. Sehadri, ad L. Stokes. Samples-based estimatio of the umber of distict values of a attribute. I Proceedigs of the 21st Iteratioal Coferece o Very Large Databases (VLDB 95), pages , [10] P. Idyk. Stable distributios, pseudoradom geerators, embeddigs, ad data stream computatios. I Proceedigs of the 41st Aual IEEE Symposium o Foudatios of Computer Sciece (FOCS 2000), pages , [11] P. Idyk, S. Guha, M. Muthukrisha, ad M. Strauss. Histogrammig data streams with fast peritem processig. I Proceedigs of the 19th Iteratioal Colloquium o Automata, Laguages ad Programmig, pages , [12] A. C. Yao. Probabilistic computatios: Towards a uified measure of complexity. I Proceedigs of the 18th Aual Symposium o Foudatios of Computer Sciece (FOCS 77), pages ,