An Algebraic Approach to Adaptive Scalable Overlay Network Monitoring

Transcription

1 An Algebraic Approach to Adaptive Scalable Overlay Network Monitoring ABSTRACT Overlay network monitoring enable ditributed Internet application to detect and recover from path outage and period of degraded performance within econd. For an overlay network with n end hot, exiting ytem either require O(n 2 ) meaurement, and thu lack calability, or can only etimate the latency but not congetion/failure. Our earlier extended abtract [] briefly propoe an algebraic approach that elect and monitor k linearly independent path that can fully decribe all the O(n 2 ) path. The lo rate (thi can alo be extended to latency) of thee path can be ued to etimate the lo rate of all other path. Our cheme only aume knowledge of the underlying IP topology, with link dynamically varying between loy and normal. In thi paper, we improve, implement and extenively evaluate uch a monitoring ytem. We further make the following contribution: i) calability analyi which how that for reaonably large n (e.g., ), k i only in the range of O(n log n), ii) efficient adaptation algorithm for topology change, uch a the addition/removal of end hot and interconnection, iii) meaurement load balancing cheme for better calability, iv) topology meaurement error handling for better accuracy. Both imulation and Internet experiment demontrate we achieve high path lo rate etimation accuracy while adapting to topology change within econd and handling topology error. We can alo continuouly update the lo rate etimate online. For example, in the Internet experiment, the average update time i.6 econd for all 255 path, the abolute error of lo rate etimation i.27 and the average error factor i... INTRODUCTION The rigidity of the Internet architecture make it extremely difficult to deploy innovative diruptive technologie in the core. Thi ha led to extenive reearch into overlay and peer-to-peer ytem, uch a overlay routing and location, application-level multicat, and peer-to-peer file haring. Thee ytem flexibly chooe their communication path and target, thu can benefit from end-to-end network ditance etimation (e.g., latency and lo rate). Accurate lo rate monitoring can detect path outage and period of degraded performance within econd. It can both facilitate ditributed ytem management (uch a a virtual private network (VPN) or a content ditribution network), and help build adaptive overlay application. For intance, exiting treaming media ytem treat the underlying network a a bet-effort black box, thu adaptation are limited and can only be performed at the tranmiion end-point. In contrat, the monitoring ervice can enable thee ytem to bypa faulty or low link through overlay routing for continuou media tranmiion. Thu it i deirable to have a calable overlay lo rate monitoring ytem which i accurate and incrementally deployable. However, exiting network ditance etimation ytem are inufficient for thi end. They can be grouped into two categorie baed on the targeted metric: general metric [2] and latency only [3, 4, 5, 6]. Sytem in the former category can meaure any metric, but require O(n 2 ) meaurement where n i the number of end hot, and thu lack calability. On the other hand, the latency etimation ytem are calable, but cannot provide accurate congetion/failure detection (ee Sec. 2). We formulate the targeted problem a follow: conider an overlay network of n end hot; we define a path to be a routing path between a pair of end hot, and a link to be an IP link between router. A path i a concatenation of link. There are O(n 2 ) path among the n end hot, and we wih to elect a minimal ubet of path to monitor o that the lo rate and latencie of all other path can be inferred. In an earlier extended abtract [], we ketched the idea of a tomography-baed overlay monitoring ytem (TOM ) in which we electively monitor a bai et of k path. Any end-to-end path can be written a a unique linear combination of path in the bai et. Conequently, by monitoring lo rate for the path in the bai et, we infer lo rate for all end-to-end path. Thi can alo be extended to other additive metric, uch a latency. The end-to-end path lo rate can be computed even when the path contain unidentifiable link for which lo rate cannot be computed. In [], we only briefly introduce the baic formulation and model. Many quetion remain a follow: How calable i the ytem? In other word, how will k grow a a function of n? It i our conjecture that for reaonably large n (ay ), k = O(n log n) []. In an overlay network, end hot frequently join/leave the overlay and routing change can happen from time to time, how to adapt to thee efficiently without etting up the ytem from cratch? How to ditribute the meaurement load evenly among the end hot to improve the calability? How to handle topology meaurement error for better accuracy? How doe TOM perform under variou topologie and lo condition a well a the real Internet? To addre thee iue, in thi paper, we firt improve TOM in everal direction to enhance it accuracy, calability, and make it dynamically adaptive to network change. Then we implement and extenively evaluate the ytem. In particular, we make the following contribution. To demontrate it calability, we how that k doe grow a O(n log n) through extenive linear regreion tet on both ynthetic and real topologie. To adapt to changing network topology, efficient algorithm are deigned o that for each path addition and deletion, it only cot O(k 2 ) time intead of O(n 2 k 2 ) to completely reinitialize. To improve the calability and accuracy, we propoe cheme for meaurement load balancing and topology meaurement error handling.

2 We evaluate TOM through extenive imulation, a well a Internet experiment to validate the imulation. Both imulation and PlanetLab experiment reult how that we achieve high accuracy when etimating path lo rate with O(n log n) meaurement. For the PlanetLab experiment, the average abolute error of lo rate etimation i only.27, and the average error factor i., although about % of the path have no or incomplete routing information. The average etup (monitoring path election) time i.75 econd, and the online update of the lo rate for all 255 path take only.6 econd. In addition, we adapt to topology change within econd without acrificing accuracy. The meaurement load balancing reduce the load variation and the maximum v. mean load ratio ignificantly, by up to a factor of 7.3. The ret of the paper i organized a follow. We urvey related work in Sec. 2, decribe our model and baic tatic algorithm in Sec. 3, and evaluate it calability in Sec. 4. we extend the algorithm to adapt to topology change in Sec. 5, and to handle overloading and topology meaurement error in Sec. 6. The methodology and reult of imulation are decribed in Sec. 7, and thoe of Internet experiment are preented in Sec. 8. Finally, we conclude in Sec RELATED WORK There i a lot of prior work on calable end-to-end latency etimation. They are either clutering-baed [5, 6] or coordinate-baed [3, 4]. Clutering-baed ytem cluter end hot baed on their network proximity or latency imilarity under normal condition, then chooe the centroid of each cluter a the monitor. But a monitor and other member of the ame cluter often take different route to remote hot. So the monitor cannot detect congetion for it member. Similarly, the coordinate aigned to each end hot in the coordinate-baed approache cannot embed any congetion/failure information. Network tomography ha been well tudied ([7] provide a good urvey). Mot tomography ytem aume limited meaurement are available (often in a multicat tree-like tructure), and try to infer link characteritic [8, 9] or hared congetion [] in the middle of the network. However, the problem i under-contrained: there exit unidentifiable link [8] with propertie that cannot be uniquely determined. In contrat, we are not concerned about the characteritic of individual link, and we do not retrict the path we meaure. Shavitt et al. alo ue algebraic tool to compute ditance that are not explicitly meaured []. Given certain Tracer tation deployed and ome direct meaurement among the Tracer, they earch for path or path egment whoe lo rate can be inferred from thee meaurement. Thu their focu i not on Tracer/path election. Neither do they examine the topology meaurement error or the topology change problem. Recently, Ozmutlu et al. elected a minimal ubet of path to cover all link for monitoring, auming link-bylink latency i available via end-to-end meaurement [2]. Their approach ha the following limitation: ) Traceroute cannot give accurate link-by-link latency. Many router in the Internet hide their identitie. 2) It i not applicable for lo rate, becaue it i difficult to etimate link-by-link lo rate from end-to-end meaurement. Many application, like treaming media and multiplayer gaming, are more en- Overlay Network Operation Center End hot Figure : Architecture of a TOM ytem. itive to lo rate than latency. 3) It aume tatic routing path and doe not conider topology change. 3. THE ALGEBRAIC MODEL AND BASIC ALGORITHMS In thi ection, we briefly decribe the algebraic model of TOM and it baic tatic algorithm. 3. Algebraic Model and Intuition Suppoe there are n end hot that belong to a ingle or confederated overlay network(). They cooperate to hare an overlay monitoring ervice, and are intrumented by a central authority (e.g., an overlay network operation center (ONOC)) to meaure the routing topology and path lo rate a needed. For implicity, we uually aume ymmetric routing and undirected link in thi paper. However, our technique work without change for aymmetric routing, a ued in the PlanetLab experiment. Fig. how a ample overlay network with four link and four end hot; ix poible path connect the end hot. The end hot meaure the topology and report to the ONOC, which elect four path and intrument two of the end hot to meaure the lo rate of thoe path. The end hot periodically report the meaured lo rate to the ONOC. Then the ONOC infer the lo rate of every link, and conequently the lo rate of the other two path. Application can query the ONOC for the lo rate of any path, or they can et up trigger to receive alert when the lo rate of path of interet exceed a certain threhold. Symbol Meaning M total number of node N number of end hot n number of end hot on the overlay r = O(n 2 ) number of end-to-end path # of IP link that the overlay pan on t number of identifiable link G {, } r original path matrix Ḡ {, } k reduced path matrix k rank of G l i lo rate on ith link p i lo rate on ith meaurement path x i log( l i) b i log( p i) v vector in {, } (repreent path) p lo rate along a path N (G) null pace of G R(G T ) row(path) pace of G (== range(g T )) Table : Table of notation A part of the future work, we will invetigate technique to ditribute the work of the central authority.

3 The Internet topology can be much more complicated than the example in Fig.. Here we introduce the algebraic model that work with any topology. Suppoe an overlay network pan IP link. We repreent a path by a column vector v {, }, where the jth entry v j i one if link j i part of the path, and zero otherwie. Suppoe link j drop packet with probability l j; then any path repreented by v ha it lo rate p a p = j= ( l j) v j () We take logarithm on both ide of (). Then by defining a column vector x with element x j = log ( l j), and writing v T a the tranpoe of the row vector v, we can rewrite () a follow: log ( p) = j= v j log ( l j) = j= v jx j = v T x (2) There are r = O(n 2 ) path in the overlay network, thu r linear equation of the form (2). Putting them together, we form a rectangular matrix G {, } r to repreent thee path. Each row of G repreent a path in the network: G ij = when path i contain link j, and G ij = otherwie. Let p i be the end-to-end lo rate of the ith path, and let b r be a column vector with element b i = log ( p i). Then we write the r equation in form (2) a Gx = b (3) Normally, the number of path r i much larger than the number of link. Thu equation (3) can be viually repreented a Fig. 2(a). Thi ugget that if we can elect path to monitor, and then olve the link lo rate variable x, the lo rate of other path can be inferred from (3). r = r = k k (a) Gx = b (b) ḠxG = b Figure 2: Matrix ize repreentation. However, in general, G i rank deficient: i.e., k = rank(g) and k <. If G i rank deficient, we will be unable to determine the lo rate of ome link from (3). Thee link are alo called unidentifiable in network tomography literature [8]. A b 2 x 2 (,,) G = b 3 (,-,) D b x b null pace 2 3 C = (unmeaured) x G x2 b2 B x3 b3 x 3 Figure 3: Sample overlay network. row(path) pace (meaured) We illutrate how rank deficiency can occur in Fig. 3. There are three end hot (A, B and C) on the overlay, three link (, 2 and 3) and three path between the end hot. We can not uniquely olve x and x 2 becaue link and 2 alway appear together. We know their um, but not their difference. When viewing the geometry of the linear ytem in Fig. 3 with each variable x i a a dimenion, there i a et of vector α T which are not defined by (3). Thi et of vector i the null pace of G (N (G)). Meanwhile, there i an orthogonal row(path) pace of G (R(G T )), which for thi example i a plane determined by x + x 2 and x 3. Unlike the null pace, any vector on the row pace can be uniquely determined by (3). To eparate the identifiable and unidentifiable component of x, we decompoe x a x = x G + x N, where x G R(G T ) i it projection on the row pace and and x N N (G) i it projection on the null pace (i.e., Gx N = ). The decompoition of [x x 2 x 3] T for the ample overlay i hown below. x G = (x + x2) 2 x N = + x 3 (x x2) 2 = b /2 b /2 b 2 Thu the vector x G can be uniquely identified, and contain all the information we can know from (3) and the path meaurement. Next, we examine x G cloely for the intuition. From (4), there are two component in x G: x + x 2 and x 3. Thi ugget we can tranform link and 2 to a ingle virtual link with an identifiable lo rate x +x 2 a in Fig. 4. Baically, the virtual link are the minimal path egment whoe lo rate can be uniquely identified. Any end-to-end path can be repreented a an unique linear combination of the virtual link. In other word, the lo rate of the virtual link are ufficient to decribe all the path lo rate. A more intereting example i hown in the bottom figure of Fig. 4. There are four path, but they are linearly dependent, o rank(g) = 3, and none of the link lo rate are computable. We can ue any three out of the four path a virtual link, and the other one can be linearly repreented by the virtual link. For example, path 4 can be decribed a virtual link G = Rank(G)=3 A b 2 b 3 D 2 b 2 3 C B 4 Virtualization Real link (olid) and all of the overlay path (dotted) travering them Virtual link Figure 4: Virtualization of ample overlay path. Since the dimenion of R(G T ) i k, the minimum number of virtual link which can fully decribe R(G T ) i alo k. x G i a linear combination of the vector repreenting the virtual link. Since virtual link are identifiable, x G i alo computable. From x G, we can compute the lo rate of all end-to-end path a we can do with virtual link. Becaue x G lie in the k-dimenional pace R(G T ), only k independent equation of the r equation in (3) are needed to (4) (5)

4 uniquely identify x G. We meaure thee k path to compute x G. Then ince b = Gx = Gx G + Gx N = Gx G, we can compute all element of b from x G, and thu obtain the lo rate of all other path. For example, in Fig. 3, we only need to meaure b and b 2 to compute x G, from which we can calculate b 3. Next, we will give more detailed algorithm. 3.2 Baic Algorithm There are two tep for the baic algorithm. Firt, we elect a bai et of k path to monitor. Such election only need to be done once for etup. Then, baed on continuou monitoring of the elected path, we calculate and update the lo rate of all other path Meaurement Path Selection To elect k linearly independent path from G, we ue tandard rank-revealing decompoition technique [3], and obtain a reduced ytem: Ḡx G = b (6) where Ḡ k and b k conit of k row of G and b, repectively. The equation i illutrated in Fig. 2(b) (compared with Gx = b). A hown below, our algorithm i a variant of the QR decompoition with column pivoting [3, p.223]. It incrementally build a decompoition ḠT = QR, where Q k i a matrix with orthonormal column and R k k i upper triangular. procedure SelectPath(G) for every row(path) v in G do 2 ˆR2 = R T Ḡv T = Q T v T 3 ˆR22 = v 2 ˆR if ˆR 22 then 5 Select v a a meaurement path R 6 Update R = 2 and ˆR22 Ḡ = Ḡ v end end Algorithm : Path (row) election algorithm Note that the Ḡ elected may not be unique - it depend on the order of path canning in Algorithm. Take the ample overlay network in Fig. 3 for example, if the order of path can in tep of Algorithm i AB, AC, BC, Ḡ =. If the order i AB, BC, AC, Ḡ =. Thee bai et are equivalent on decribing all end-to-end path G. In Sec. 6, we make ue of it for ditributed load balancing. In general, the G matrix i very pare; that i, there are only a few nonzero per row. We leverage that for peedup. We further ue optimized routine from the LAPACK library [4] to implement Algorithm o that it inpect everal row at a time. The complexity of Algorithm i O(rk 2 ) and the contant in the bound i modet (ee the running time reult in Sec. 7.4 and 8.2). The memory cot i roughly k 2 /2 ingle-preciion floating point number for toring the R factor. Notice that the path election only need to be executed once for initial etup Path Lo Rate Calculation To compute the path lo rate, we mut find a olution to the underdetermined linear ytem ḠxG = b. The vector b come from meaurement of the path. Zhang et al. report that path lo rate remain operationally table in the time cale of an hour [5], o thee meaurement need not be taken imultaneouly. Given meaured value for b, we can compute one olution uing the QR decompoition we contructed during meaurement path election [3, 6]. We chooe the unique olution x G with minimum poible norm by impoing the contraint that x G = ḠT y where y = R R T b. Once we have xg, we can compute b = Gx G, and from there infer the lo rate of the unmeaured path. The complexity for thi tep i only O(k 2 ). Thu we can update lo rate etimate online, a verified in Sec. 7.4 and SCALABILITY ANALYSIS Such an overlay monitoring ytem i calable only when the ize of the bai et, k, grow relatively lowly a a function of n. Given that the Internet ha moderate hierarchical tructure [7], it i our conjecture that k = O(n log n) []. In thi ection, we will how through linear regreion on both ynthetic and real topologie that k i indeed bounded by O(n log n) for reaonably large n (e.g, ). We experiment with three type of BRITE router-level topologie - Barabai-Albert, Waxman and hierarchical model - a well a with a real router topology with 284,85 node [8]. For hierarchical topologie, BRITE firt generate an autonomou ytem (AS) level topology with a Barabai-Albert model or a Waxman model. Then for each AS, BRITE generate the router-level topologie with another Barabai- Albert model or Waxman model. So there are four type of poible topologie. We how three of them except the Barabai-Albert model for both AS and router level which ha very imilar trend to the Waxman model for both AS and router level. We randomly elect end hot which have the leat degree, to form an overlay network. We tet by linear regreion of k on O(n), O(n log n), O(n.25 ), O(n.5 ), and O(n.75 ). A hown in Fig. 5, reult for each type of topology are averaged over three run with different topologie for ynthetic one and with different random et of end hot for the real one. We find that for Barabai-Albert, Waxman and real topologie, O(n) regreion ha the leat reidual error - actually k even grow lower than O(n). The hierarchical model have higher k, and mot of them have O(n log n) a the bet fit. Conervatively peaking, we have k = O(n log n). Note that uch trend till hold when each end hot i in a different acce network. One extreme cae i the tar topology - each end hot i connected to the ame center router via it own acce network. In uch a topology, there are only n link. Thu k = O(n). Only topologie with very dene connectivity, like a full clique, have k = O(n 2 ). Thoe topologie have little link haring among the end-toend path. 5. DYNAMIC ALGORITHMS FOR TOPOL- OGY CHANGES During normal operation, new link may appear or diappear, routing path between end hot may change, and hot may enter or exit the overlay network. Thee change may caue row or column to be added to or removed from

5 8 2 Rank of path pace (k) original meaurement regreion on n regreion on nlogn regreino on n.25 regreion on n.5 regreion on n.75 Rank of path pace (k) original meaurement regreion on n regreion on nlogn regreion on n.25 regreion on n.5 regreion on n Number of end hot on the overlay (n) 9 Barabai-Albert model of 2K node Number of end hot on the overlay (n) Waxman model of 2K node 8 7 original meaurement regreion on n regreion on nlogn regreion on n.25 regreion on n.5 regreion on n original meaurement regreion on n regreion on nlogn regreion on n.25 regreion on n.5 regreion on n Rank of path pace (k) 5 4 Rank of path pace (k) Number of end hot on the overlay (n) Hierarchical model of 2K node (AS-level: Barabai-Albert and router level: Waxman) Number of end hot on the overlay (n) Hierarchical model of 2K node (AS-level: Waxman and router level: Waxman) 6 7 x original meaurement regreion on n regreion on nlogn regreion on n.25 regreion on n.5 regreion on n original meaurement regreion on n regreion on nlogn regreion on n.25 regreion on n.5 regreion on n.75 Rank of path pace (k) 3 2 Rank of path pace (k) Number of end hot on the overlay (n) Hierarchical model of 2K node (AS-level: Waxman and router level: Barabai) Number of end hot on the overlay (n) A real topology of 284K router Figure 5: Regreion of k in variou function of n under different router-level topologie.

6 G, or entrie in G may change. In thi ection, we deign efficient algorithm to incrementally adapt to thee change. 5. Path Addition and Deletion The baic building block for topology update are path addition and deletion. We have already handled path addition in Algorithm ; adding a path v during an update i no different than adding a path v during the initial can of G. In both cae, we decide whether to add v to Ḡ and update R. To delete a path that i not in Ḡ i trivial; we jut remove it from G. But to remove a path from Ḡ i more complicated. We need to update R; thi can be done in O(k 2 ) time by tandard algorithm (ee e.g. Algorithm 3.4 in [9, p.338]). In general, we may then need to replace the deleted path with another meaurement path. Finding a replacement path, or deciding that no uch path i needed, can be done by re-canning the row of G a in Algorithm ; however, thi would take time O(rk 2 ). procedure DeletePath(v, G, Ḡ, R) if deleted path v i meaured then 2 j = index of v in Ḡ 3 y = ḠT R R T e j 4 Remove v from G and Ḡ 5 Update R (Algorithm 3.4 in [9, p.338]) 6 r = Gy 7 if i uch that r i then 8 Add the ith path from G to Ḡ (Algorithm, tep 2-6) end end 9 ele Remove v from G Algorithm 2: Path deletion algorithm We now decribe Algorithm 2 to delete a path v more efficiently. Suppoe v correpond to the ith row in Ḡ and the jth row in G, we define Ḡ (k ) a the meaurement path matrix after deleting the ith row, and G (r ) a the path matrix after removing the jth row. By deleting v from Ḡ, we reduce the dimenion of Ḡ from k to k. Intuitively, our algorithm work in the following two tep.. Find a vector y that only decribe the direction removed by deleting the ith row of Ḡ. 2. Tet if the path pace of G i orthogonal to that direction, i.e., find whether there i any path p G that ha a non-zero component on that direction. If not, no replacement path i needed. Otherwie, replace v with any of uch path p, and update the QR decompoition. Next, we decribe how each tep i implemented. To find y which i in the path pace of Ḡ but not of Ḡ, we olve the linear ytem Ḡy = ei, where ei i the vector of all zero except for a one in entry i. Thi ytem i imilar to the linear ytem we olved to find x G, and one olution i y = ḠT R R T e i. Once we have computed y, we compute r = G y, where G i the updated G matrix. Becaue we choe y to make Ḡ y =, all the element of r correponding to elected row are zero. Path uch that r j are guaranteed to be independent of Ḡ, ince if row j of G could be written a w T Ḡ for ome w, then r j would be w T Ḡ y =. If all element of r are zero, then y i a null vector for all of G ; in thi cae, the dimenion k of the row pace of G i k, and we do not need to replace the deleted meaurement path. Otherwie, we can find any j uch that r j and add the jth path to Ḡ to replace the deleted path. Take the overlay network in Fig. 3 for example, uppoe Ḡ i compoed of the path AB and BC, i.e., Ḡ =. Then we delete path BC, Ḡ = [ ] T and G =. Applying Algorithm 2, we have y = [ ] T and r = [ ] T. Thu the econd path in G, AC, hould be added to Ḡ. If we view uch path deletion with the geometry of the linear ytem, the path pace of G remain a a plane in Fig. 3, but Ḡ only ha one dimenion of the path pace left, o we need to add AC to Ḡ. When deleting a path ued in Ḡ, the factor R can be updated in O(k 2 ) time. To find a replacement row, we need to compute a pare matrix-vector product involving G, which take O(n 2 (average path length)) operation. Since mot routing path are hort, the dominant cot will typically be the update of R. So the complexity of Algorithm 2 i O(k 2 ). 5.2 End hot Join/Leave the Overlay To add an end hot h, we ue Algorithm to can all the new path from h, for a cot of O(nk 2 ). However, to delete an end hot h, to imply ue Algorithm 2 to delete each path involving h i not efficient. Thee path are often dependent to each other. Thu deleting one path form Ḡ often caue adding another to-be-removed path into Ḡ. To avoid that, we remove all thee path from G firt, then ue the updated G in Algorithm 2 to remove each h-related path in Ḡ. Each path in Ḡ can be removed in O(k2 ) time; the total cot of end hot deletion i then at mot O(nk 2 ). 5.3 Routing Change In the network, routing change or link failure can affect multiple path in G. Mot Internet path remain table over day [2]. So we can incrementally meaure the topology to detect change. Each end hot meaure the path to all other end hot daily, and for each end hot, uch meaurement load can be evenly ditributed throughout the day. For each link, we keep a lit of the path that travere it. If any path i reported a changed for certain link(), we will examine all other path that go through thoe link() becaue it i highly likely that thoe path can change their route a well. We ue Algorithm and 2 to incrementally incorporate each path change. 6. LOAD BALANCING AND TOPOLOGY ERROR HANDLING To further improve the calability and accuracy, we need to have good load balancing and handle topology meaurement error, a dicued in thi ection. 6. Meaurement Load Balancing To avoid overloading any ingle node or it acce link, we evenly ditribute the meaurement among the end hot. We randomly reorder the path in G before canning them for election in Algorithm. Since each path ha equal probability to be elected for monitoring, the meaurement load on each end hot i imilar. Note any bai et generated from Algorithm i ufficient to decribe all path G. Thu

7 the load balancing ha no effect on the lo rate etimation accuracy. 6.2 Handling Topology Meaurement Error Becaue our goal i to etimate the end-to-end path lo rate intead of any interior link lo rate, we can handle certain topology meaurement inaccuracie, uch a incomplete routing information and poor router alia reolution. For completely untraceable path, we add a direct link between the ource and the detination. In our ytem, thee path will become elected path for monitoring. For path that only give partial route, we add link from where the normal route become unavailable (e.g., elf loop or diplaying * * * in traceroute), to where the normal route reume or to the detination if uch anomalie perit to the end. For intance, if the meaured route i (rc, ip, * * *, ip 2, det), the path i compoed of three link: (rc ip ), (ip, ip 2), and (ip 2, det). By treating the untraceable path (egment) a a normal link, the reulting topology i equivalent to the one with complete routing information for calculating the path lo rate. For topologie with router aliae which may preent one phyical link a everal link, we do not really have to reolve thee aliae. At wort, our failure to recognize the link a the ame will reult in a few more path meaurement becaue the rank of G i higher. For thee link, their correponding entrie in x G will be aigned imilar value becaue they are really a ingle link. Thu the path lo rate etimation accuracy i not affected, a verified by Internet experiment in Sec. 8. In addition, our ytem i robut to meaurement node failure and node change by providing bound on the etimated lo rate. 7. EVALUATION In thi ection, we preent our evaluation metric, imulation methodology and imulation reult. 7. Metric The metric include path lo rate etimation accuracy, variation of meaurement load among the end hot, and peed of etup, update, and topology change adaptation. To compare the inferred lo rate ˆp with real lo rate p, we analyze both abolute error and error factor. The abolute error i p ˆp. We adopt the error factor F ε(p, ˆp) defined in [8] a follow: p(ε) F ε(p, ˆp) = max ˆp(ε), p(ε) ˆp(ε) (7) where p(ε) = max(ε, p) and ˆp(ε) = max(ε, ˆp). Thu, p and ˆp are treated a no le than ε, and then the error factor i the maximum ratio, upward or downward, by which they differ. We ue the default value ε =. a in [8]. If the etimation i perfect, the error factor i one. Furthermore, we claify a path to be loy if it lo rate exceed 5%, which i the threhold between tolerable lo and eriou lo a defined in [5]. We report the true number of loy path, the percentage of real loy path identified (coverage) and the fale poitive rate, all averaged over five run of experiment for each configuration. There are two type of meaurement load: ) ending probe, and 2) receiving probe and computing lo rate. The load reflect the CPU and uplink/downlink bandwidth conumption. For each end hot h, it meaurement load i linearly proportional to, and thu denoted by the number of monitored path with h a ender/receiver. Then we compute it variation acro end hot in term of the coefficient of variation (CV) and the maximum v. mean ratio (MMR), for ending load and receiving load eparately. The CV of a ditribution x, defined a below, i a tandard metric for meauring inequality of x, while the MMR check if there i any ingle node whoe load i ignificantly higher than the average load. CV (x) = tandard deviation(x) mean(x) The imulation only conider undirected link, o for each monitored path, we randomly elect one end hot a ender and the other a receiver. Thi i applied to all imulation with or without load balancing. 7.2 Simulation Methodology We conider the following dimenion for imulation. (8) Topology type: three type of ynthetic topologie from BRITE (ee Sec. 7.3) and a real router-level topology from [8]. All the hierarchical model have imilar reult, we jut ue Barabai-Albert at the AS level and Waxman at the router level a the repreentative. Topology ize: the number of node range from to 2 2. Note that the node count include both internal node (i.e., router) and end hot. Fraction of end hot on the overlay network: we define end hot to be the node with the leat degree. Then we randomly chooe from % to 5% of end hot to be on the overlay network. Thi give u peimitic reult becaue other ditribution of end hot will probably have more haring of the routing path among them. We prune the graph to remove the node and link that are not referenced by any path on the overlay network. Link lo rate ditribution: 9% of the link are claified a good and the ret a bad. We ue two different model for aigning lo rate to link a in [9]. In the firt model (LLRD ), the lo rate for good link i elected uniformly at random in the -% range and that for bad link i choen in the 5-% range. In the econd model (LLRD 2), the lo rate range for good and bad link are -% and -% repectively. Given pace limitation, mot reult are under model LLRD except for Sec Lo model: After aigning each link a lo rate, we ue either a Bernoulli or a Gilbert model to imulate the lo procee at each link. For a Bernoulli model, each packet travering a link i dropped at independently fixed probability a the lo rate of the link. For a Gilbert model, the link fluctuate between a good tate (no packet dropped) and a bad tate (all packet dropped). According to Paxon oberved meaurement of Internet [2], the probability of remaining in bad tate i et to be 35% a in [9]. Thu, the Gilbert model i more likely to generate burty loe than the 2 2 i the larget topology we can imulate on a.5ghz Pentium 4 machine with 52M memory.

8 9 9 9 Cumulative percentage (%) node, 5 end hot node, end hot 2 5 node, end hot 5 node, 3 end hot 2 node, end hot 2 node, 5 end hot Abolute error Cumulative percentage (%) node, 5 end hot 2 node, end hot 5 node, end hot 5 node, 3 end hot 2 node, end hot 2 node, 5 end hot Abolute error Cumulative percentage (%) end hot end hot 2 end hot Abolute error Cumulative percentage (%) node, 5 end hot 2 node, end hot 5 node, end hot 5 node, 3 end hot 2 node, end hot 2 node, 5 end hot Relative error factor Cumulative percentage (%) node, 5 end hot 2 node, end hot 5 node, end hot 5 node, 3 end hot 2 node, end hot 2 node, 5 end hot Relative error factor Cumulative percentage (%) end hot end hot 2 end hot Relative error factor BRITE Barabai-Albert topology BRITE Hierarchical topology Real topology of 284K router Figure 6: Cumulative ditribution of abolute error (top) and error factor (bottom) under Gilbert lo model for variou topologie. # of # of end hot # of # of link rank MPR loy path (Bernoulli) loy path (Gilbert) node total OL(n) path(r) original AP (k) (k/r) real coverage FP real coverage FP % %.3% 437.%.2% % % 2.% %.2% % % 2.% %.% % % 4.% %.3% % % 3.4% %.6% % % 5.5% %.4% # of # of end hot # of # of link rank MPR loy path (Bernoulli) loy path (Gilbert) node total OL(n) path(r) original AP (k) (k/r) real coverage FP real coverage FP % %.% %.4% % % 4.6% %.5% % % 3.9% %.4% % % 7.% %.% % % 2.3% %.4% % % 5.7% %.5% # of # of end hot # of # of link rank MPR loy path (Bernoulli) loy path (Gilbert) node total OL(n) path(r) original AP (k) (k/r) real coverage FP real coverage FP % % 2.% %.5% % %.6% %.3% % %.6% %.2% % %.8% %.% % %.2% 483.%.% % %.3% %.% Table 2: Simulation reult for three type of BRITE router topologie: Barabai-Albert (top), Waxman (middle) and hierarchical model (bottom). OL give the number of end hot on the overlay network. AP how the number of link after pruning (i.e., remove the node and link that are not on the overlay path). MPR i the monitored path ratio between our approach and the pair-wie cheme. FP i the fale poitive rate.

9 Bernoulli model. The other tate tranition probabilitie are elected o that the average lo rate matche the lo rate aigned to the link. We repeat our experiment five time for each imulation configuration unle denoted otherwie, where each repetition ha a new topology and new lo rate aignment. The path lo rate i imulated baed on the tranmiion of packet. Uing the lo rate of elected path a input, we compute x G, then the lo rate of all other path. 7.3 Reult for Different Topologie For all topologie in Sec. 7.2, we achieve high lo rate etimation accuracy. Reult for the Bernoulli and the Gilbert model are imilar. Since the Gilbert lo model i more realitic, we plot the cumulative ditribution function (CDF) of abolute error and error factor with the Gilbert model in Fig. 6. For all the configuration, the abolute error are le than.8 and the error factor are le than.8. Waxman topologie have imilar reult, and we omit them in the interet of pace. The loy path inference reult are hown in Table 2. Notice that k i much maller than the number of IP link that the overlay network pan, which mean that there are many IP link whoe lo rate are unidentifiable. Although different topologie have imilar aymptotic regreion trend for k a O(n log n), they have different contant. For an overlay network with given number of end hot, the more IP link it pan on, the bigger k i. We found that Waxman topologie have the larget k among all ynthetic topologie. For all configuration, the loy path coverage i more than 96% and the fale poitive ratio i le than 8%. Many of the fale poitive and fale negative are caued by mall etimation error for path with lo rate near the 5% threhold. We alo tet our algorithm in the 284,85-node real routerlevel topology from [8]. There are 65,8 end hot router and 86,683 link. We get the ame trend of reult a illutrated in Fig. 6 and Table 3. The CDF include all the path etimate, including the monitored path for which we know the real lo rate. Given the ame number of end hot, the rank in the real topology are higher than thoe of the ynthetic one. But a we find in Sec. 4, the growth of k i till in the range of O(n). 7.4 Reult for Different Link Lo Rate Ditribution and Running Time We have alo run all the imulation above with model LLRD 2. The lo rate etimation i a bit le accurate than it i under LLRD, but we till find over 95% of the loy path with a fale poitive rate under %. Given pace limitation, we only how the loy path inference with the Barabai-Albert topology model and the Gilbert lo model in Table 4. The running time for LLRD and LLRD 2 are imilar, a in Table 4. All peed reult in thi paper are baed on a.5 GHz Pentium 4 machine with 52M memory. Note that it take about 2 minute to etup (elect the meaurement path) for an overlay of 5 end hot, but only everal econd for an overlay of ize. The update (lo rate calculation) time i mall for all cae, only 4.3 econd for 24,75 path. Thu it i feaible to update online. 7.5 Reult for Meaurement Load Balancing We examine the meaurement load ditribution for both ynthetic and real topologie, and the reult are hown in # of # of end hot loy path (Gilbert) peed (econd) node total overlay real coverage FP etup update %.% % 3.% % 3.5% %.4% %.% % 4.6% Table 4: Simulation reult with model LLRD 2. Ue the ame Barabai-Albert topologie a in Table 2. Refer to Table 2 for tatitic like rank. FP i the fale poitive rate. Table 5. Given the pace contraint, we only how the reult for Barabai-Albert and hierarchical model. Our load balancing cheme reduce CV and MMR ubtantially for all cae, and epecially for MMR. For intance, a 5-node overlay on a 2-node network of Barabai-Albert model ha it MMR reduced by 7.3 time. We further plot the hitogram of meaurement load ditribution by putting the load value of each node into equally paced bin, and counting the number of node in each bin a y-axi. The x-axi denote the center of each bin, a illutrated in Fig. 7. With load balancing, the hitogram roughly follow the normal ditribution. In contrat, the hitogram without load balancing i cloe to an exponential ditribution. Note that the y-axi in thi plot i logarithmic: an empty bar mean that the bin contain one member, and. mean the bin i empty. Number of end hot in each bin Amount of meaurement (average for each bin) 22. (a) with load balancing 24.7 Number of end hot in each bin Amount of meaurement (average for each bin) 6 3 (b) without load balancing Figure 7: Hitogram of the meaurement load ditribution (a ender) for an overlay of 3 end hot on a 5-node Barabai-Albert topology. 7.6 Reult for Topology Change We tudy two common cenario in P2P and overlay network: end hot joining and leaving a well a routing change. Again, the Bernoulli and the Gilbert model have imilar reult, thu we only how thoe of the Gilbert model End hot join/leave # of end # of rank loy path hot path real coverage FP %.2% %.% (45) (99) (837) (623) %.2% (5) (225) (997) (795) Table 6: Simulation reult for adding end hot on a real router topology. FP i the fale poitive rate. Denoted a +added value (total value). For the real router topology, we tart with an overlay network of 4 random end hot. Then we randomly add an end hot to join the overlay, and repeat the proce until the ize 46

10 # of end hot # of # of link rank MPR loy path (Bernoulli) loy path (Gilbert) on overlay (n) path(r) after pruning (k) (k/r) real coverage FP real coverage FP % %.9% 92.%.2% % %.9% %.3% % % 3.% %.4% Table 3: Simulation reult for a real router topology. MPR and FP are defined the ame a in Table 2. # of OL Barabai-Albert model hierarchical model node ize CV MMR CV MMR (n) ender recver ender recver ender recver ender recver LB NLB LB NLB LB NLB LB NLB LB NLB LB NLB LB NLB LB NLB Table 5: Meaurement load (a ender or receiver) ditribution for variou BRITE topologie. OL Size i the number of end hot on overlay. LB mean with load balancing, and NLB mean without load balancing. # of end # of rank loy path hot path real coverage FP %.2% %.2% (55) (485) (5.7) (97.3) %.% (5) (225) (995.) (89.7) Table 7: Simulation reult for deleting end hot on a real router topology. FP i the fale poitive rate. Denoted a -reduced value (total value). of the overlay reache 45 and 5. Averaged over three run, the reult in Table 6 how that there i no obviou accuracy degradation caued by accumulated numerical error. The average running time for adding a path i 25 mec, and for adding a node,.8 econd. Notice that we add a block of path together to peedup adding node (Sec. 3.2). Similarly, for removing end hot, we tart with an overlay network of 6 random end hot, then randomly elect an end hot to delete from the overlay, and repeat the proce until the ize of the overlay i reduced to 55 and 5. Again, the accumulated numerical error i negligible a hown in Table 7. A hown in Sec. 5, deleting a path in Ḡ i much more complicated than adding a path. With the ame machine, the average time for deleting a path i 445 mec, and for deleting a node, 6.9 econd. We note that the current implementation i not optimized: we can peed up node deletion by proceing everal path imultaneouly, and we can peed up path addition and deletion with iterative method uch a CGNE or GMRES [22]. Since the time to add/delete a path i O(k 2 ), and to add/delete a node i O(nk 2 ), we expect our updating cheme to be ubtantially fater than the O(n 2 k 2 ) cot of re-initialization for larger n Routing change We form an overlay network with 5 random end hot on the real router topology. Then we imulate topology change by randomly chooing a link that i on ome path of the overlay and removing of uch a link will not caue diconnection for any pair of overlay end hot. Then we aume that the link i broken, and re-route the affected path(). Algorithm in Sec. 5 incrementally incorporate each path change. Averaged over three run, reult in Table 8 how that we adapt quickly, and till have accurate path lo rate etimation. # of path affected 4.7 # of monitored path affected 36.3 # of unique node affected 4.7 # of real loy path (before/after) 76./784. coverage (before/after) 99.8%/99.8% fale poitive rate (before/after).2%/.% average running time 7.3 econd Table 8: Simulation reult for removing a link from a real router topology. 8. INTERNET EXPERIMENTS We implemented our ytem on the PlanetLab [23] tetbed. In thi ection, we preent the implementation reult. 8. Methodology We chooe 5 PlanetLab hot, each from a different organization a hown in Table 9. All the international PlanetLab hot are univeritie. Area and Domain # of hot.edu 33.org 3 US (4).net 2.gov.u France Sweden Europe Denmark (6) Germany UK 2 International () Aia (2) Taiwan Hong Kong Canada 2 Autralia Table 9: Ditribution of elected PlanetLab hot. Firt, we meaure the topology among thee ite by imultaneouly running traceroute to find the path from each hot to all other. Each hot ave it detination IP addree for ending meaurement packet later. Then we meaure the lo rate between every pair of hot. Our meaurement conit of 3 trial, each of which lat 3 mec.

11 During a trial, each hot end a 4-byte UDP packet 3 to every other hot. Uually the hot will finih ending before the 3 mec trial i finihed. For each path, the receiver count the number of packet received out of 3 to calculate the lo rate. To prevent any hot from receiving too many packet imultaneouly, each hot end packet to other hot in a different random order. Furthermore, any ingle hot ue a different permutation in each trial o that each detination ha equal opportunity to be ent later in each trial. Thi i becaue when ending packet in a batch, the packet ent later are more likely to be dropped. Such random permutation are pre-generated by each hot. To enure that all hot in the network take meaurement at the ame time, we et up ender and receiver daemon, then ue a wellconnected erver to broadcat a START command. Will the probing traffic itelf caue loe? We performed enitivity analyi on ending frequency a hown in Fig. 8. All experiment were executed between am-3am PDT June 24, 23, when mot network are free. The traffic rate from or to each hot i (5 ) ending freq 4 byte/ec. The number of loy path doe not change much when the ending rate varie, except when the ending rate i over 2.8Mbp, ince many erver can not utain that ending rate. We chooe a 3 mec ending interval to balance quick lo rate tatitic collection with moderate bandwidth conumption. Number of loy path Bandwidth conumption (Kbp) Sending frequency (number of trial per econd) Figure 8: Senitivity tet of ending frequency Note that the experiment above ue O(n 2 ) meaurement o that we can compare the real lo rate with our inferred lo rate. In fact, our technique only require O(n log n) meaurement. Thu, given good load balancing, each hot only need to end to O(log n) hot. In fact, we achieve imilar CV and MMR for meaurement load ditribution a in the imulation. Even for an overlay network of 4 end hot on the 284K-node real topology ued before, k = If we reduce the meaurement frequency to one trial per econd, the traffic conumption for each hot i 8668/4 4 byte/ec = 4.9Kbp, which i typically le than 5% of the bandwidth of today broadband Internet link. We can ue adaptive meaurement technique in [2] to further reduce the overhead. 8.2 Reult From June 24 to June 27, 23, we ran the experiment time, motly during peak hour 9am - 6pm PDT. Each experiment generate = 765K UDP packet, totaling 76.5M packet for all experiment. We run the lo rate meaurement three to four time every hour, and run the pair-wie traceroute every two hour. Acro the 3 2-byte IP header + 8-byte UDP header + 2-byte data on equence number and ending time. run, the average number of elected monitoring path (Ḡ) i 87.9, about one third of total number of end-to-end path, 255. Table how the lo rate ditribution on all the path of the run. About 96% of the path are nonloy. Among the loy path, mot of the lo rate are le than.5. Though we try to chooe table node for experiment, about 25% of the loy path have % loe and are likely caued by node failure or other reachability problem a dicued in Sec lo [, loy path [.5,.] (4.%) rate.5) [.5,.) [.,.3) [.3,.5) [.5,.). % 95.9% 5.2% 3.% 23.9% 4.3% 25.6% Table : Lo rate ditribution: loy v. non-loy and the ub-percentage of loy path Accuracy and peed When identifying the loy path (lo rate >.5), the average coverage i 95.6% and the average fale poitive rate i 2.75%. Fig. 9 how the CDF for the coverage and the fale poitive rate. Notice that 4 run have % coverage and 9 run have coverage over 85%. 58 run have no fale poitive and 9 run have fale poitive rate le than %. Cumulative Percentage (%) Fale poitive rate (%) Coverage of loy path Fale poitive rate 9 85 Coverage of loy path (%) Figure 9: Cumulative percentage of the coverage and the fale poitive rate for loy path inference in the experiment. A in the imulation, many of the fale poitive and fale negative are caued by the 5% threhold boundary effect. The average abolute error acro the run i only.27 for all path, and.58 for loy path. We pick the run with the wort accuracy in coverage (69.2%), and plot the CDF of abolute error and error factor in Fig.. Since we only ue 3 packet to meaure the lo rate, the lo rate preciion granularity i.33, o we ue ε =.5 for error factor calculation. The average error factor i only. for all path. Cumulative Percentage (%) Error factor of lo rate etimation Abolute error Error factor Abolute error of lo rate etimation Figure : Cumulative percentage of the abolute error and error factor for the experiment with the wort accuracy in coverage. 8 75