Self-Tuning, Bandwidth-Aware Monitoring for Dynamic Data Streams

Transcription

1 Self-Tunng, Bandwdth-Aware Montorng for Dynamc Data Streams Navendu Jan, Praveen Yalagandula, Mke Dahln, Yn Zhang Mcrosoft Research HP Labs The Unversty of Texas at Austn Abstract We present, a self-tunng, bandwdth-aware montorng system that maxmzes result precson of contnuous aggregate queres over dynamc data streams. Whle pror approaches mnmze bandwdth cost under fxed precson constrants, they may stll overload a montorng system durng traffc bursts. To facltate practcal deployment of montorng systems, therefore bounds the worst-case bandwdth cost for overload reslence. The prmary challenge for s how to dynamcally select updates at each node to maxmze query precson whle keepng per-node montorng bandwdth below a specfed budget. To address ths challenge, s herarchcal algorthm (1) allocates bandwdth budgets n a near-optmal manner to maxmze global precson and (2) selftunes bandwdth settngs to mprove precson under dynamc workloads. Our prototype mplementaton of provdes key solutons to (a) prortze pendng updates for mult-attrbute queres, (b) buld bounded fan-n, load-aware aggregaton trees to mprove accuracy, and (c) combne temporal batchng wth arthmetc flterng to reduce load and to quantfy result staleness. Our evaluaton usng smulatons and a network montorng applcaton shows that ncurs low overheads, mproves accuracy by up to an order of magntude compared to unform bandwdth allocaton, and performs close to the optmal algorthm under modest bandwdth budgets. I. INTRODUCTION Dstrbuted stream processng systems [1] [3] must provde hgh performance and hgh fdelty for query processng as they grow n scale and complexty. In these systems, data streams are often bursty where nput rates may unexpectedly ncrease [4] [6]. Examples nclude network traffc montorng, dentfyng dstrbuted denal-of-servce (DDoS) attacks on the Internet, fnancal stocks montorng, web clck stream analyss, and event-drven montorng n sensor networks. Therefore, t s desrable for these systems to bound the montorng load whle stll provdng useful accuracy guarantees on the query results. Exstng technques [7] [12] am to address ths problem by mnmzng the montorng cost whle promsng an a pror error bound (e.g., ± 1%) on query precson. Unfortunately, although these technques effectvely reduce load, they are unsutable n dynamc, hgh-volume data stream envronments for three reasons. (1) Settng precson a pror requres workload knowledge: Choosng error bounds a pror s unntutve when workloads are not known n advance or may change unpredctably over tme e.g., should the error be 1% or 3%? Conversely, t may be easer to set the montorng budget e.g., a system admnstrator s wllng to pay.1% of network bandwdth for montorng. (2) Bad precson settng hurts performance: A bad choce of the error bound may sgnfcantly degrade the qualty of a query result when the error bound s too large [13] or ncur a hgh communcaton and processng cost when the error bound s too small. (3) Bursty traffc mposes unacceptable overheads: Even wth reasonable error bounds, montorng systems may stll rsk overload under bursty and often unpredctable traffc condtons as we llustrate wth an example below. To address these challenges, we propose that for many montorng systems, the rght approach s to fx the bandwdth cost whle optmzng query precson. By boundng the worst case bandwdth cost, montorng systems acheve three key benefts. Frst, they avod need for a pror workload knowledge or keepng up wth workload changes to optmally set precson bounds. Second, they can maxmze precson durng perods of low workload and provde a graceful degradaton under hgh workloads. Thrd, they avod rsk of overload durng traffc bursts and stll be able to delver results wth useful accuracy. We motvate these benefts usng an example. Motvatng Example: We present a smple example to llustrate the challenges for scalable montorng under bursty workloads. We smulate a set of 1 data sources connected to a centralzed montor wth an ncomng bandwdth lmt of messages per second. The nput workload dstrbuton s modeled based on the standard exponental dstrbuton wth a parameter λ, and upon each arrval, the value of attrbute a at data source s updated accordng to random walk model n whch the value ether ncreases or decreases by an amount sampled unformally from [., 1.]. Fgure 1 shows the load-error tradeoff for a sngle data source. As expected, on ncreasng the error budget, arthmetc flterng [9], [1] quckly decreases the load as a majorty updates get fltered. To quantfy the montorng cost, we use λ=1 and compute a SUM aggregate under a baselne error budget of 2 and ts correspondng expected load of. (Fgure 1), effectvely settng the total expected cost for montorng 1 data sources at messages per second. Fgure 2 shows the nduced message load at the central montor under a fxed error budget of 2. We observe that under peak data arrvals, the system sometmes ncurs message costs 4x hgher than the expected cost of messages per second. Ths devaton s due to bursty

2 Expected Messages per second 1 Load-Error Tradeoff Error Budget Messages per second Load (Fxed Error = 2) Expected Load Tme (seconds, *1) Dvergence Dvergence (Fxed BW=) Expected Dvergence Tme (seconds, *1) Fg. 1. Expected outgong message load vs. error budget for a sngle attrbute at a data source under a random walk workload. Fg. 2. Montorng system nduces overload under bursty workloads to bound result error. The system overload s up to 4x over expected load. Fg. 3. Montorng system causes hgh naccuracy under bursty workloads to bound load. The error s up to x hgher over expected dvergence. posson arrvals the probablty of 2x, 4x, and 8x overload s.18,.1, and 9e-6, respectvely, and worse for many other dstrbutons. Conversely, Fgure 3 shows the dvergence between data source attrbutes and ther cached values at the central montor under a fxed load of. In ths case, the system may provde hghly naccurate results havng up to x more error than the expected error of 2. As a result, exstng montorng technques are neffectve under bursty workloads they rsk overload, loss of accuracy, or both. Our Contrbutons: Our goal s to bound the bandwdth cost whle stll delverng query results wth useful accuracy. To acheve ths goal, provdes three key technques: (1) Maxmze precson under fxed bandwdth budget: formulates a bandwdth-aware optmzaton problem whose goal s to maxmze the result precson of an aggregate query n a herarchcal aggregaton tree subject to gven bandwdth constrants. Ths model provdes a closed-form, near-optmal soluton to self-tune bandwdth settngs for achevng hgh accuracy usng only local and aggregated nformaton at each node n the tree. Further, provdes a graceful degradaton n query precson as the avalable bandwdth decreases. Our expermental results show that self-tunng bandwdth settngs mproves accuracy by an order of magntude over unform bandwdth allocaton and performs close to the optmal algorthm under modest bandwdth budgets. (2) Scalablty va multple aggregaton trees: bulds on recent work that uses dstrbuted hash tables (DHTs) to construct scalable, load-balanced forests of self-organzng aggregaton trees [14] [16]. Scalablty to tens of thousands of nodes and mllons of attrbutes s acheved by mappng dfferent attrbutes to dfferent trees. For each tree n ths forest of aggregaton trees, s self-tunng algorthm adjusts bandwdth settngs to acheve hgh precson under dynamc workloads where the estmate of bandwdth vs. precson trade-offs, and hence the optmal dstrbuton, can change over tme. (3) Hgh performance by combnng arthmetc flterng and temporal batchng: ntegrates temporal batchng wth arthmetc flterng to reduce the montorng load and to quantfy staleness of query results. For hghvolume workloads, t s advantageous for nodes to batch multple updates that arrve close n tme and send a sngle combned update. In an aggregaton tree, ths temporal batchng allows leaf sensors to condense a seres of updates nto a perodc report and allows nternal nodes to combne updates from dfferent subtrees before transmttng them further. Further, t bounds the delay (e.g., up to 3 seconds) from when an update occurs at a leaf untl t s reported at the root. We have mplemented a prototype of on PRISM [17], a scalable aggregaton system bult on top of FreePastry [18]. Our prototype provdes two key optmzatons. Frst, t constructs bounded fan-n, bandwdth-aware aggregaton trees to mprove accuracy and to quckly detect anomales n heterogeneous envronments. Recent studes [7], [1], [19] [21] suggest that only a few attrbutes (e.g., network flows) generate a sgnfcant fracton of the total traffc n many montorng applcatons. Thus, for fast anomaly detecton, an aggregaton tree should be able to quckly route mportant updates towards the root such that no nternal node becomes a bottleneck due to ether hgh n-degree or low bandwdth. Second, for mult-attrbute queres, provdes a refresh schedule that selects and prortzes attrbutes for refreshng n order to mnmze the error n query results. We evaluate through extensve smulatons and a real network montorng applcaton of detectng dstrbuted heavy htters. Experence wth ths applcaton llustrates the mproved precson and scalablty benefts: for a gven montorng budget, s adaptvty can sgnfcantly mprove the query precson whle montorng a large number of attrbutes. Compared to unform bandwdth allocaton, mproves accuracy by up to an order of magntude and provdes accuracy wthn 27% of the optmal algorthm under modest bandwdth budgets. Example Queres: For concreteness, we lst several real-world applcaton queres to llustrate the types of queres was desgned to support. Q 1 Identfy the heavy htter network flows that send the hghest traffc n aggregate across all network endponts.

3 Q 2 Q 3 Q 4 Fnd the top-k ports across all nodes that have been heavly scanned n the recent past, possbly ndcatng worm actvty. Montor anomaly condtons e.g., SUM(nodes sensng fre) threshold, MAX(chemcal concentraton) n a physcal sensor network. Montor the top-k popular web objects n a wde-area content dstrbuton network such as Akama. All these aggregate queres requre processng a large number of rapd update streams n lmted bandwdth/battery-lfe envronments, and all can beneft from. In Secton VI, we show results for Q 1 usng a network montorng applcaton we have mplemented. In summary, ths paper makes the followng contrbutons. We dentfy the key lmtatons of prevous fx error, mnmze load technques, and we address them by boundng the worst-case load whle stll provdng query results wth useful accuracy. We descrbe a practcal, self-tunng, bandwdth-aware montorng system that adapts bandwdth budgets to maxmze precson of contnuous aggregate queres under hgh-volume, dynamc workloads. Our mplementaton provdes key optmzatons to mprove accuracy and to quckly detect anomales n heterogeneous envronments. Our evaluaton demonstrates that provdes a key substrate for scalable montorng: t provdes hgh accuracy n dynamc envronments and performs close to the optmal algorthm under modest bandwdth budgets. The rest of ths paper s organzed as follows. Secton II dscusses related work. Secton III descrbes PRISM [17], a scalable DHT-based aggregaton system, and precsonperformance tradeoffs that underle. Secton IV descrbes the adaptve algorthm that self-tunes bandwdth settngs to mprove result accuracy. Secton V and VI present the mplementaton and expermental evaluaton of. Fnally, Secton VII hghlghts our conclusons. II. RELATED WORK bulds upon pror fx error, mnmze load technques [7] [12], but t departs n three sgnfcant ways drven by our focus on achevng overload reslence for practcal, scalable montorng. Frst, reformulates the key optmzaton problem, whch we beleve s an mportant contrbuton. Whle pror approaches mnmze cost under fxed precson constrants, bursty, hgh-volume workloads may stll overload a montorng system as dscussed n Secton I. In fact, t s precsely durng these abnormal events that a montorng system needs to avod overload but stll delver results wth useful accuracy n real-tme. therefore bounds the worst-case system cost to provde overload reslence. Second, the techncal advances to solve ths new problem are sgnfcant. Whle uses Chebyshev nequalty [1] to capture the load vs. error trade-off, t faces new constrants of lmted bandwdth budgets both at a parent and each of ts chldren n a herarchcal aggregaton tree. Gven these constrants, self-tunes bandwdth settngs n a nearoptmal manner to acheve hgh accuracy under dynamc workloads. Thrd, s prototype provdes key solutons to prortze updates, buld bandwdth-aware trees, and combne temporal batchng wth arthmetc flterng. Recently, load sheddng technques [], [22] have been proposed to handle overload condtons. The key dea s to carefully drop some tuples to reduce processng load but at the expense of reducng query result accuracy. These approaches handle load spkes assumng ether the CPU [], [22] or man memory [23] to be the key resource bottleneck. In comparson, bandwdth s the prmary resource constrant n. Best-effort cache synchronzaton technques [24], [2] am to mnmze the dvergence between source data objects and cached copes n a one-level tree. In these technques, each object s refreshed ndvdually. In comparson, performs herarchcal query processng for scalablty and corelates updates to the same attrbute at dfferent data sources to maxmze the accuracy of an aggregate query result. To our knowledge, there has not been pror work on maxmzng precson of aggregate queres n herarchcal trees under lmted bandwdth. Babcock and Olston [26] focus on effcently computng top-k aggregate values gven fxed error n a one-level tree but do not consder how to maxmze precson of top-k results under fxed bandwdth n herarchcal trees. Slbersten et al. propose a samplng-based approach at randomly chosen tme steps for computng top-k queres n sensor networks [27]. Ther approach focuses on returnng the k nodes wth the hghest sensor readngs. In comparson, focuses on large-scale aggregate queres such as dstrbuted heavy htters whch requre computng a global aggregate value for each of the tens of thousands to mllons of attrbute across all the nodes n the network. Sketches are small-space data structures that provde approxmate answers to aggregate queres [28]. These technques, however, requre error bounds to be set a pror to provde the approxmaton guarantees. III. POINT OF DEPARTURE extends PRISM [17] whch embodes two key abstractons for scalable montorng: aggregaton and DHTbased aggregaton. then ntroduces controlled tradeoffs between precson bounds and montorng bandwdth. A. DHT-based Herarchcal Aggregaton Aggregaton s a fundamental abstracton for scalable montorng [12], [14] [16], [2] because t allows applcatons to access summary vews of global nformaton and detaled vews of rare events and nearby nformaton. s aggregaton abstracton defnes a tree spannng all nodes n the system. As Fgure 4 llustrates, n PRISM each physcal node s a leaf, and each subtree represents a logcal group of nodes. An nternal non-leaf node, whch we call a vrtual node, s smulated by a physcal leaf node of

4 37 Vrtual Nodes (Internal Aggregaton Ponts) L Physcal Nodes (Leaf Sensors) Fg. 4. The aggregaton tree for key n an eght node system. Also shown are the aggregate values for a smple SUM() aggregaton functon. the subtree rooted at the vrtual node. Fgure 4 llustrates the computaton of a smple SUM aggregate. leverages DHTs [16], [29], [3] to construct a forest of aggregaton trees and maps dfferent attrbutes to dfferent trees for scalablty. DHT systems assgn a long (e.g., 16 bts), random ID to each node and defne a routng algorthm to send a request for ID to a node root such that the unon of paths from all nodes forms a tree DHTtree rooted at the node root. By aggregatng an attrbute wth ID = hash(attrbute) along the aggregaton tree correspondng to DHTtree, dfferent attrbutes are load balanced across dfferent trees. Ths approach can provde aggregaton that scales to a large number of nodes and attrbutes [14], [1]. B. Query Result Approxmaton quantfes the precson of query results n terms of numerc error between the reported result and the actual value. We defne the numerc approxmaton of a query result usng Arthmetc Imprecson [9] [11], [31]. Arthmetc mprecson (AI) determnstcally bounds the numerc dfference between a reported value of an attrbute and ts true value. For example, an AI of 1% ensures that the reported value ether underestmates or overestmates the true value by at most 1%. When applcatons do not need exact answers [9] [12], arthmetc mprecson reduces load by flterng updates that le wthn the AI error range of the last reported value. Next, we descrbe how uses the AI mechansm to enforce the numerc error bounds whle reducng bandwdth load. Mechansm: We frst descrbe the basc mechansm for enforcng AI for each aggregaton subtree n the system. To enforce AI, each aggregaton subtree T for an attrbute has an error budget δ T that defnes the maxmum naccuracy of any result the subtree wll report to ts parent for that attrbute. The root of each subtree dvdes ths error budget among tself δ self and ts chldren δ c (wth δ T δ self + c chldren δ c), and the chldren recursvely do the same. Note that δ c determnes the chld s error flter to cull as many updates as possble before sendng them to the parent, and δ self s useful for applyng addtonal flterng after combnng all updates receved from the chldren. Here we present the AI mechansm for the SUM aggregate snce t s lkely to be common n network montorng [9], [1] and fnancal applcatons [32]; other standard aggregaton functons (e.g., MAX, MIN, AVG, etc.) are smlar and defned precsely n a techncal report [17]. L3 L2 L1 Ths arrangement reduces system load by flterng small updates that fall wthn the range of values cached by a subtree s parent. In partcular, after a node A wth error budget δ T reports a range [V mn, V max ] for an attrbute value to ts parent (where V max V mn + δ T ), f the node A receves an update from a chld c, the node A can skp updatng ts parent as long as t can ensure that the true value of the attrbute for the subtree les between V mn and V max,.e., f V mn c chldren V c mn V max c chldren V c max where [Vmn c, V max] c denote the latest update receved from c. mantans per-attrbute δ T values so that dfferent attrbutes wth dfferent error requrements and dfferent update patterns can use dfferent δ budgets n dfferent subtrees. C. Case-study Applcaton To gude the system development of and to drve our performance evaluaton, we have bult a casestudy applcaton usng the PRISM aggregaton framework: a dstrbuted heavy htter detecton servce. Dstrbuted Heavy Htters (DHH) detecton s mportant for both detectng network traffc anomales such as DDoS attacks, botnet attacks, and flash crowds as well as for accountng and bandwdth provsonng [19]. We defne heavy htters as enttes that account for a sgnfcant proporton of the total actvty measured n terms of number of packets, bytes, connectons, etc. [19] n a dstrbuted system for example, the 1 IPs that account for the most ncomng traffc n the last 1 mnutes [19]. To answer ths dstrbuted query, the key challenge s scalablty for aggregatng per-flow statstcs for tens of thousands to mllons of concurrent flows across all the network endponts n real-tme. For example, a subset of the Ablene [33] traces used n our experments ncludes 8 thousand flows that send about 2 mllon updates per hour. To scalably compute the global heavy htters lst, we chan two aggregatons where the results from the frst feed nto the second. Frst, PRISM calculates the total ncomng traffc for each destnaton address from all nodes n the system usng SUM as the aggregaton functon and hash(hh-step1, destip) as the key. For example, tuple (H = hash(hh-step1, ), 9 KB) at the root of the aggregaton tree T H ndcates that a total of 9 KB of data was receved for across all vantage ponts n the network durng the last tme wndow. In the second step, we feed these aggregated total bandwdths for each destnaton IP nto a SELECT- TOP-1 aggregaton wth key hash(hh-step2, TOP-1) to dentfy the TOP-1 heavy htters among all flows. In Secton VI we show how s self-tunng algorthm adapts bandwdth settngs to montor a large number of attrbutes and provdes hgh result accuracy by flterng majorty of mce flows [19] (attrbutes wth low frequency) whle prortzng updates for the heavy-htter flows; we expect typcal montorng applcatons to have a large number of mce flows but only a few heavy htters [7], [1], [19] [21]. (1)

5 IV. DESIGN In ths secton we present the desgn and descrbe our self-tunng algorthm that adapts bandwdth settngs at each node to maxmze query precson. A. System Model We focus on dstrbuted stream processng envronments wth a large number of data sources that perform n-network aggregaton to compute contnuous aggregate queres over ncomng data streams. The bandwdth resources for query processng may be lmted at a number of ponts n the network. In partcular, a node j s outgong bandwdth may be constraned (Bj O ), a node j s ncomng bandwdth may be constraned (Bj I ), or both. Note that constranng the ncomng bandwdth bounds (a) the control traffc overhead for montorng and (b) the CPU processng load for computng the aggregaton functon across ncomng data nputs. Fnally, these bandwdth capactes may vary among nodes n heterogeneous envronments and wth tme f bandwdth s shared wth other applcatons. At any tme, each attrbute s numerc value s bounded by an AI error δ. If δ s small, then updates may frequently drve an attrbute s value out of ts last reported range [V mn, V max ], forcng the system to send messages to update the range. A system can, however, reduce ts bandwdth requrements by ncreasng δ. Thus, f a herarchcal aggregaton system has bandwdth constrants, t should determne a δ value at each aggregaton pont that meets the bandwdth constrants nto and out of that pont, and t should select these δ values so as to mnmze the total AI for the attrbute. In partcular, rather than splttng each node s ncomng bandwdth evenly among ts chldren, the system should attempt to assgn bandwdth to where t wll do the most good by reducng the resultng mprecson. In the rest of ths secton, we descrbe the algorthm for mnmzng mprecson whle meetng bandwdth constrants n four steps. Frst, we descrbe a smplfed algorthm for a one-level aggregaton tree and statc workloads. For each leaf node n the system, ths algorthm calculates an deal error settng δ and correspondng expected bandwdth consumpton b such that (1) each leaf node s outbound bandwdth (.e., rate of updates sent to the root) s at most ts outgong bandwdth budget, (2) the root node s ncomng bandwdth (.e., sum of update rates nbound from chldren) s at most ts ncomng bandwdth budget, and (3) the sum of the δ s s mnmzed gven the frst two constrants. Second, we descrbe how to handle dynamc workloads where the estmate of AI error δ vs. bandwdth trade-offs, and hence the optmal dstrbuton, can change over tme. A key challenge here s throttlng the rate at whch the system redstrbutes bandwdth budgets across nodes snce such redstrbuton also ncurs bandwdth costs. Thrd, we generalze the algorthm to handle mult-level aggregaton trees. Symbol Meanng B O outgong bandwdth constrant for node B I ncomng bandwdth constrant for node u nput update rate at node σ standard devaton of node s nput workload chld() all chldren of node δ node s AI error settng b node s outgong bandwdth load δ opt node s optmal AI error settng node s optmal outgong bandwdth load b opt TABLE I SUMMARY OF KEY NOTATIONS. Fnally, we dscuss how our mplementaton copes wth varablty. In partcular, sets the per-node δs so that the average bandwdth meets a target. However, spkes of update load for an attrbute or concdent updates for multple attrbutes could cause nstantaneous bandwdth to exceed the target. To avod such nstantaneous overload, therefore prortzes pendng updates based on the mpact they wll have on ther aggregate values and drans them to the network at the target rate, smlar to broadcast schedulng [34]. B. One-Level Tree Quantfy Bandwdth vs. AI Precson Tradeoff: To estmate the optmal dstrbuton of load budgets among dfferent nodes, we use a smple load vs. error tradeoff model based on Chebyshev nequalty [1]. Ths model quantfes query precson that can be acheved under a gven bandwdth budget. Let X be a random varable wth fnte expectaton µ and varance σ 2. Chebyshev s nequalty states that for any k, P r( X µ kσ) 1 k 2 (2) For AI flterng, the term kσ represents the error budget δ for node. Substtutng for k n Equaton 2 gves: P r( X µ δ ) σ2 δ 2 (3) Ths equaton mples that f the error budget s smaller than the standard devaton (.e., δ σ and k 1), then δ s unlkely to flter many data updates. In ths case, Equaton 3 provdes only a weak bound on the message cost: the probablty that each ncomng update wll trgger an outgong message s bounded by 1. However, f δ kσ for any k 1, the fracton of unfltered updates s probablstcally bounded by σ 2. In general, gven the nput update rate u δ 2 for node wth error budget δ, the expected message cost for node per unt tme s: { σ 2 } M = MIN u, δ 2 u (4) We use the expected message cost M to estmate node s optmal outgong bandwdth b opt. Estmate Bandwdth Settngs under Fxed Load: To estmate the optmal load dstrbuton and settngs of AI error δs at each node n a one-level tree rooted at node r, we formulate an optmzaton problem of mnmzng the total

6 error for a SUM aggregate at root r under gven bandwdth budgets Br I (at the root) and B O (at chld ): MIN chld(r), b opt s.t. chld(r) δ opt chld(r) σ 2 u (δ opt ) 2 BI r = σ2 u (δ opt ) 2 mn{bo, u } where b opt denotes the estmated optmal settng of outgong bandwdth of node to meet the global objectve of mnmzng the total error chld(r) δ opt () subject to two constrants: (1) node s outgong bandwdth budget ( b opt mn{b O, u } ) ( and (2) ) ncomng bandwdth budget at root r b opt Br I. chld(r) Enforcng Incomng Bandwdth Constrant: To solve Equaton (), we frst relax the outgong bandwdth constrants (.e.,, b opt mn{b O, u }) but enforce the ncomng bandwdth constrant. Later, we provde a soluton when the outgong bandwdth constrants are also enforced. Usng Lagrangan Multplers, the above formulaton yelds a closed-form and computatonally nexpensve soluton [17]: 3 σ 2 c u c δ opt c chld(r) = σ 3 2 u (6) B I r whch provdes a closed-form formula for settng b opt b opt = σ2 u ) = 2 BI r (δ opt δ 2 = BI r N : 3 σ 2 u (7) 3 σ 2 c u c c chld(r) As a smple example, f u = σ = 1, then each node sets the same δ for the attrbute as δ = N, and the bandwdth Br I budget for node wll be 1.e., gven a total bandwdth budget and unform workload dstrbuton across nodes, each node gets a unform share of the bandwdth to update each attrbute. Note that to set b opt (Equaton (7)), each node needs to 3 know σ 2 c u c and root r s ncomng bandwdth c chld(r) budget B I r ; computes a smple SUM aggregate across chldren nputs to obtan ths nformaton. As a smple optmzaton, these messages are pggy-backed on data updates. Enforcng Outgong Bandwdth Constrants: Note that the above load assgnment assumes that the outgong bandwdth constrant b opt c mn{bc O, u c } holds for every chld c. If for a node, the above soluton doesn t satsfy b opt mn{b O, u }, then we need to set b opt = mn{b O, u } to satsfy ts bandwdth constrant. Ths stuaton may arse n heterogeneous envronments where a subset of nodes may experence hgher nput loads (e.g., DDoS attacks) or may become severely resource-constraned (e.g., sensor networks wth low power devces). Settng b opt = mn{b O, u } can free up part of r s ncomng capacty Br I, whch can be reassgned to other chldren to ncrease ther bandwdth budget thereby mprovng the overall accuracy. therefore apples an teratve algorthm that n each teraton determnes all saturated chldren C sat at each step, fxes ther load budgets and error settngs, and recomputes Equatons (6), (7) for all the remanng chldren (assumng chld set C sat s absent). A chld j s labeled saturated f b opt j Bj O.e., ts avalable outgong bandwdth s at most the estmated optmal settng of ts outgong bandwdth. Note that for our DHT-based aggregaton trees, the fan-n for a node s typcally 16 (.e., a 4-bt correcton per hop) so the teratve algorthm runs n constant tme (at most 16 tmes). C. Self-Tunng Bandwdth Settngs The above soluton s derved assumng that σ and u are gven and reman constant. In practce, σ and u may change over tme dependng on the workload characterstcs. therefore self-tunes bandwdth settngs to mprove accuracy. Cost-Beneft Throttlng: We apply cost-beneft throttlng to dynamcally adapt the bandwdth settngs. Specfcally, after computng the new bandwdth budgets, a node computes a charge metrc for each attrbute a whch estmates the reducton n error ganed by refreshng a: charge a = (T curr T a lastsent) D a (8) where T curr s the current tme, TlastSent a s the last tme an attrbute a s update was sent, and D a denotes the devaton between a s current value at that node and ts last reported range [V mn, V max ] to the parent. For example, f a s AI error range cached at the parent s [1, 2] and a new update wth value 11 arrves at a chld node, we expand the range to [1,11] at the chld to nclude the new value settng D a =1. Notce that an attrbute s charge wll be large f () there s a large error mbalance (.e., D a s large), or () there s a longlastng mbalance (e.g., T curr TlastSent a s large). Further, only usng D a may hurt precson f the attrbute has a repeated behavor of quckly dvergng after the last refresh. Therefore, the temporal term (T curr TlastSent a ) prortzes attrbutes who are lkely to agan dverge slowly after beng refreshed thereby gvng a long-term precson beneft. Snce redstrbuton also consumes bandwdth budgets, we only send messages to readjust the bandwdth settngs when dong so s lkely to reduce the tme-averaged error for those attrbutes by at least a threshold τ (.e., f charge a > τ). Further, to ensure the nvarant that a node does not exceed ts bandwdth whle sendng updates for multple attrbutes, we present a smple prortzaton technque n Secton IV-E. D. Mult-Level Trees To scale to a large number of nodes, we extend our basc algorthm for a one-level tree to a dstrbuted algorthm for a mult-level aggregaton herarchy. Note that n a mult-level tree, leaf nodes use AI error δ to flter sensor updates and nternal nodes retan a local AI error, δ self, to help prevent

7 updates receved from ther chldren from beng propagated further up the tree [1], [3]. In an aggregaton tree, each nternal node apples the selftunng algorthm smlar to the one-level tree case: the nternal node s a local root for each of ts mmedate chldren, and the bandwdth targets Bp I for parent p and Bc O for each chld c are nput as constrants n Equaton (). The key dfference s that for chldren who are nternal nodes, we use ther AI error δ self n ths optmzaton framework. To estmate the optmal bandwdth and AI error settngs for each chld, the parent node p tracks ts ncomng update rate.e., the aggregate number of messages sent by all ts chldren per tme unt (u p ) and the standard devaton (σ p ) of updates receved from ts chldren. Note that u c, σ c reports are accumulated by chld c untl they can be pggy-backed on an update message sent to ts parent. Gven ths nformaton, a parent estmates for each chld c, the optmal outgong bandwdth b opt c and the AI error that mproves the total precson n the aggregate value δ opt c(self) computed across the chldren gven bandwdth constrants 1. E. Prortzng Pendng Updates Fnally, we descrbe how our mplementaton addresses the challenge posed by varablty n bandwdth targets. In partcular, we want to avod stuatons where a node meets ts average bandwdth target but may temporarly exceed nstantaneous bandwdth target due to spkes of update load for an attrbute, concdent updates for multple attrbutes, or a sudden ncrease n avalable bandwdth. Each node needs to send messages to the root to mnmze the query error. Therefore, we need to prortze sendng those messages that beneft precson the most. A nave refreshng algorthm that updates attrbutes n a round-robn fashon could easly spend network messages that do not reduce error whle consumng valuable bandwdth resource. Lmtng sendng of non-useful updates s a partcular concern for applcatons lke DHH that montor a large number of attrbutes, only a few of whch are actve enough to be worth optmzng. To address ths problem, our mplementaton provdes a prorty heap of all pendng updates that need to be sent ordered by ther prorty. Usng ths heap, prortzes pendng updates based on the mpact they wll have on ther aggregate values and drans them to the network at a target rate. Specfcally, at each tme step, a node keeps removng the maxmum prorty attrbute from the heap and sendng t to the correspondng parent untl the node s nstantaneous target bandwdth s reached. Note that to compare prortes across dfferent attrbutes that have dfferent value ranges across dfferent queres, we normalze an attrbute a s refresh prorty (Equaton (8)) by dvdng the devaton D a by the standard devaton σ a of that attrbute. 1 Note that fndng a globally optmal soluton s too expensve n ths envronment as t requres perfect knowledge of (possbly varyng) (1) per node bandwdth settngs, (2) network topology, and (3) update rate and varance of nput workload at all tmes. Therefore, adapts bandwdth n a best-effort, self-tunng manner to acheve hgh precson n herarchcal trees and our experments show that ths approach works well n practce. level 4 level 3 level 2 level 1 level Event level 4 level 3 level 2 level 1 level Event TI TI (a) Send synchronzed updates every TI seconds. TI/4 TI/2 3TI/4 TI (b) Send unsynchronzed updates every TI/ l seconds. Next TI nterval starts here Fg.. For a gven TI bound, ppelned delays wth synchronzed clocks (a) allows nodes to send less frequently than unppelned delays wthout synchronzed clocks (b). V. IMPLEMENTATION In ths secton, we descrbe the prototype mplementaton of. Frst, we present a key performance optmzaton of temporal batchng of updates to reduce load and to quantfy staleness of query results. Second, we descrbe how to buld bandwdth-aware aggregaton trees to mprove accuracy n heterogeneous network envronments. Then, we descrbe how handles falures and reconfguratons. Fnally, we dscuss how to mprove precson for dfferent aggregates. A. Temporal Imprecson ntegrates temporal mprecson wth arthmetc flterng to provde staleness guarantees on query results and to reduce the montorng load. Temporal Imprecson (TI) bounds the delay from when an event/update occurs untl t s reported [8], [9], [11], [36]. A temporal mprecson of TI (e.g., TI = 3 seconds) guarantees that every event that occurred TI or more seconds ago s reflected n the reported result; events younger than TI may or may not be reflected. In, each attrbute has a TI nterval durng whch ts updates are batched nto a combned message, checked f the combned update drves the aggregate value out of the last reported AI range, and then pushed nto the prorty queue to be sent to the parent. Temporal mprecson benefts montorng applcatons n two ways. Frst, t accounts for nherent network and processng delays n the system; gven a worst-case per-hop cost hop max even mmedate propagaton provdes a temporal guarantee no better than l hop max where l s the maxmum number of hops from any leaf to the root of an aggregaton tree. Second, explctly exposng TI allows to reduce load by usng temporal batchng: a set of updates at a leaf sensor are condensed nto a perodc report or a set of updates that arrve at an nternal node over a tme nterval are combned nto a sngle message before beng sent further up the tree [17]. Ths temporal batchng mproves scalablty by reducng processng and network load as we show usng experments on a network montorng applcaton n Secton VI.

8 mplements TI usng a smple mechansm of havng each node send updates to ts parent once per TI/l seconds smlar to TAG [8] as shown n Fgure (b). Further, to maxmze the possblty of batchng updates, when clocks are synchronzed 2, ppelnes delays across tree levels so that each node sends once every (TI ) seconds wth each level s sendng tme staggered so that the updates from level arrve just before level + 1 can send (Fgure (a)). The term accounts for the worst-case per hop delays and maxmum clock skew; detals are n the extended techncal report [17]. B. Bandwdth-Aware Tree Constructon As descrbed n Secton III, leverages DHTs [16], [29], [3], [38] to construct a forest of aggregaton trees and maps dfferent attrbutes to dfferent trees [14] [16], [39] for scalablty and load balancng. then uses these trees to perform n-network aggregaton. constructs bounded fan-n, bandwdth-aware aggregaton trees to mprove result accuracy and quckly detect anomales n heterogeneous envronments. Recent studes [7], [1], [19] [21] suggest that only a few attrbutes (e.g., elephant flows [19]) generate a sgnfcant fracton of the total traffc n many montorng applcatons. Thus, to provde fast anomaly detecton, an aggregaton tree should quckly route the updates of elephant flows towards the root such that no nternal node becomes a processng/communcaton bottleneck due to ether hgh n-degree or low bandwdth. DHTs provde dfferent degrees of flexblty n choosng neghbors and next-hop paths n buldng aggregaton trees [4]. Many DHT mplementatons [18], [3] use proxmty (usually round-trp latency) n the underlyng network topology to select neghbors n the DHT overlay. Ths neghbor selecton n turn determnes the parent-chld relatonshps n the aggregaton tree. However, n a heterogeneous envronment where dfferent nodes have dfferent bandwdth budgets, an aggregaton tree formed solely based on RTTs may degrade the qualty precson of the query result for two reasons. Frst, a node may not have suffcent outgong bandwdth to send updates up n the tree even though ts underlyng tree may be suffcently well-provsoned. In such an envronment, ths node becomes a bottleneck as the updates sent by the underlyng subtree go wasted. Second, a resource-lmted parent may not be able to process the aggregate outgong update rate of all ts chldren. Thus, a practcal technque for buldng trees would be to bound the number of chldren at each nternal node and use both latences and bandwdth constrants to select the best parent at each tree level. To mprove accuracy and to quckly dentfy anomales, bulds DHT-based aggregaton trees as follows: Bound the fan-n (.e., number of chldren) at each parent node. In our mplementaton, a chld node selects ts parent such that the fan-n at the parent s at most 16.e., each parent has maxmum up to 16 chldren. 2 Algorthms n the lterature can acheve clock synchronzaton among nodes to wthn one mllsecond [37]. Use both network latency and avalable bandwdth capacty as the proxmty metrc for selectng parent nodes. orders DHT neghbors of a node such that they have the hghest ncomng bandwdth capacty and have network latency below a specfed threshold. Thus, nodes close n proxmty and havng hgh bandwdth capactes are hghly lkely to be selected as parent nodes. Fnally, note that n some envronments, t mght be useful to select nodes wth low bandwdth as parents e.g., f the nput workload at the leaf nodes comes from an ndependent unform dstrbuton, then a node closer to the root s expected to receve very few updates snce an aggregate (e.g., SUM) s lkely to become more stable gong towards the root. However, n practce, real workloads (1) are often non-unform wth few attrbutes generatng a sgnfcant fracton of the total traffc [19] and (2) exhbt both temporal and spatal skewness wth nput rates unexpectedly ncreasng over tme and across nodes. We quantfy the effectveness of constructng bounded fan-n, bandwdth-aware aggregaton trees n Secton VI. C. Robustness Falures and reconfguratons are common n dstrbuted systems. As a result, a query mght return a stale answer when nodes whose nputs are needed to compute the aggregate result become unreachable. Furthermore, n a large-scale montorng system, such falures can nteract badly wth our technques for provdng scalablty herarchy, arthmetc flterng, and temporal batchng. E.g., f a subtree s slent over an nterval, t s dffcult to dstngush between two cases: (a) the subtree has sent no updates because the nputs have not sgnfcantly changed or (b) the nputs have sgnfcantly changed but the subtree s unable to transmt ts report. As a result, reported results can devate arbtrarly from the truth. Addressng ths problem of node falures and network dsruptons n large-scale montorng systems s beyond the scope of ths paper. In a separate paper [41], we descrbe how a new consstency metrc called Network Imprecson (NI) safeguards the accuracy of query results n the face of falures, network dsruptons, and system reconfguratons. D. Dscusson Improvng precson for dfferent aggregates: focuses on the SUM aggregate snce t s lkely to be common n network montorng [9], [1] and fnancal applcatons [32]; maxmzng precson for the AVG aggregate s smlar to SUM. For MAX, MIN aggregates, f the AI error budget s fxed, then the best error assgnment s to gve equal AI error budget to all the leaf nodes. However, snce bandwdth budget s lmted n practce, each node may set ts precson dfferently to meet ts avalable bandwdth. An ntutve soluton to compute global MAX, MIN values s to smply broadcast them to each node, and a node sends an update only f t changes the global aggregate. However, ths approach may lmt scalablty n large-scale data stream systems that montor a large number of dynamc attrbutes. For the TOP-K aggregate, acheves hgh accuracy by prortzng updates based on the

9 Fg. 6. provdes hgher precson benefts as skewness n a workload ncreases. The fgures show the average error vs. load budget for three dfferent skewness settngs (a) 2:8%, (b) :%, and (c) 9:1%. hghest aggregate values and the result devaton D a. Our experments n Secton VI show that ths approach works well n practce. We plan to develop mechansms for other aggregates such as quantles n future work. Boundng Query Result Dvergence: ams to delver results wth useful accuracy under lmted bandwdth resources. Some applcatons, however, may requre guaranteed upper bounds on the dvergence of query results e.g., settng a threshold on the total traffc sent by a dstrbuted experment n PlanetLab [42]. To accommodate such cases, allows an attrbute to be tagged as a hard precson lmt attrbute, and t uses STAR [1] that adapts gven AI error budgets to mnmze bandwdth cost for such attrbutes. VI. EXPERIMENTAL EVALUATION In ths secton, we present the precson and scalablty results of an expermental study of. Frst, we use smulatons to evaluate the mprovement n query result accuracy due to s adaptve bandwdth settngs. Second, we quantfy the scalablty and accuracy acheved by for the DHH applcaton n a network montorng mplementaton. To perform ths experment, we have mplemented a prototype of usng the PRISM aggregaton system [17] on top of FreePastry [29]. We use Ablene [33] netflow traces and perform the evaluaton on 12 node nstances mapped to 3 machnes n the department Condor cluster. Fnally, we nvestgate the precson benefts of constructng bandwdthaware aggregaton trees usng our prototype. A. Smulaton Experments In ths secton, we quantfy the result accuracy acheved by compared to unform bandwdth allocaton and an dealzed optmal algorthm. Frst, we assess the effectveness of adaptve bandwdth settngs n mprovng result precson as skewness n a workload ncreases. Second, we analyze the effect of fluctuatng bandwdth. Fnally, we evaluate for dfferent workloads. In all experments, all actve sensors are at the leaf nodes of an aggregaton tree. Each sensor generates a data value every tme unt (round) for two sets of synthetc workloads for 1, rounds: (1) a random walk pattern n whch the value ether ncreases or decreases by an amount sampled unformally from [., 1.], and (2) a Gaussan dstrbuton wth standard devaton 1 and mean. We smulate m {1, 1} data sources each havng n {1, 1} attrbutes n one-level and mult-level trees, and under fxed and fluctuatng bandwdth. All attrbutes have equal weghts, messages have the same sze, and each message uses one unt of bandwdth. Evaluatng Update Rate Skewness: Frst, we evaluate the precson benefts of as skewness n a workload ncreases. We compare t wth (1) the optmal algorthm under the dealzed and unrealstc model of perfect global knowledge of each attrbute s dvergence at each data source and (2) a unform allocaton polcy where the ncomng bandwdth capacty B at a parent s allocated equally ( B C ) among ts chld set C. Although ths smple polcy s correct (the total ncomng load from the chldren s guaranteed to never exceed the ncomng load of ther parent), t s not generally the best polcy as we show n our experments. We frst perform a smple experment for a one-level tree and later show the results for herarchcal topologes. Fgure 6 shows the query precson acheved by for m=1 and n=1 under the followng skewness settngs: (a) 2:8%, (b) :%, and (c) 9:1%. For example, the 2:8% skewness represents that a randomly selected 2% attrbutes are updated wth probablty.1 whle the remanng ones are updated consstently every round under the random walk model. In all subsequent graphs n ths secton, the x-axs denotes the bandwdth budget as a fracton of the total cost m.n of refreshng all the attrbutes across all nodes; the y-axs shows the resultng average error. For 2:8% skewness, snce only a small fracton of attrbutes are stable, can only reclam up to 2% load budget from stable attrbutes sources and dstrbute t to dynamc sources to reduce ther error. For small bandwdth budgets, mproves accuracy by up to 3% compared to unform allocaton. The optmal algorthm mproves accuracy by 27% over. As the load budget ncreases, converges to the optmal soluton. mproves error by 4% over unform allocaton under 2% load budget and by more than an order of magntude under suffcently large budgets. For the : case, can reclam % of the total load budget compared to unform allocaton and gve t to unstable sources. In ths case, reduces error

10 Fg. 7. mproves accuracy even under fluctuatng bandwdth. The maxmum bandwdth varaton n the fgure s (a) 1%, (b) 2%, and (c) 3%. by up to % over unform polcy at 4% bandwdth and acheves accuracy close to the optmal soluton. Fnally, for 9% skewness, acheves the accuracy of the optmal algorthm even under 2% fracton of the total bandwdth and mproves accuracy by more than an order of magntude over unform allocaton. We observed qualtatvely smlar results for other m and n settngs. Note that the advantages of depend on the workload skewness. We expect that for systems montorng a large numbers of attrbutes (e.g., DHH), some attrbutes (e.g., elephants) wll have a hgh varablty n data values and update rates so they gan only a modest advantage n accuracy from, whle other attrbutes (e.g., mce) wll have large ratos and hence, a query (e.g., top-k) wll gan large advantages snce t only needs to provde hgh accuracy for the top-k flows. Effect of Fluctuatng Bandwdth: Next, we evaluate the effectveness of under fluctuatng bandwdth. We vary the ncomng bandwdth over tme followng a sne wave pattern and set the maxmum rate of bandwdth change to 1%, 2%, and 3% for m=1 nodes each havng n=1 attrbutes. We use update rate skewness of % as descrbed above. From Fgure 7, we observe that under 1% varaton, provdes % reducton n error over unform allocaton at 4% bandwdth fracton. As we ncrease the bandwdth fluctuaton from 1% to 3%, reduces error by about 7% under 4% bandwdth fracton, and more than an order of magntude for larger fractons. In all cases, acheves accuracy close to the optmal algorthm. Evaluatng Dfferent Workloads: Fnally, we evaluate under dfferent confguratons by varyng nput workload, standard devaton (step szes), and update frequency at each node. The workload data dstrbuton s generated from a random walk pattern and Gaussan models. For standard devaton/step-sze, 7% nodes have unform parameters as prevously descrbed; the remanng 3% nodes have these parameters set proportonal to rank (.e., wth localty) or randomly assgned (.e., no localty) from the range [., 1]. Fgure 8 shows the correspondng results for dfferent settngs of nput workload and standard devaton for m=1 and n=1. The update frequency s set to.7 skewness as descrbed prevously. We make three key observatons. Frst, Fg. 8. Precson benefts of vs. optmal algorthm and unform allocaton for dfferent {workload, step szes/standard devaton} confguratons: (a) random walk, rank, (b) random walk, random, (c) Gaussan, rank, and (d) Gaussan, random. s error s close to the optmum algorthm under the rank based assgnment as shown n Fgure 8(a),(c). Second, under random assgnment, s accuracy benefts are reduced snce updates generated from wthn the same subtree are lkely uncorrelated. In both cases, as bandwdth ncreases, s error approaches the optmal. Thrd, because stepszes are based on node rank, prortzes attrbutes havng hgher step-szes and update rates, and apples costbeneft throttlng to ensure that the precson benefts exceed costs. The unform polcy, however, does not make such a dstncton equally favorng all attrbutes that need to be refreshed, thereby neffectve n mprovng precson. Fnally, under lmted bandwdth, refreshng mce attrbutes wth small step szes does not sgnfcantly reduce the result error for queres such as top-k heavy htters but consumes valuable bandwdth resources. For all these confguratons, mproves accuracy by up to an order of magntude over unform allocaton. The optmal approach mproves accuracy by 2% over. Overall, across all confguratons n Secton VI-A, reduces naccuracy by up to an order of magntude compared to unform allocaton and s wthn 27% of the optmal algorthm under modest bandwdth fracton.