Locating Internet Bottlenecks: Algorithms, Measurements, and Implications

Transcription

1 Locaing Inerne Bolenecks: Algorihms, Measuremens, and Implicaions Ningning Hu Li (Erran) Li Zhuoing Morley Mao Carnegie Mellon Universiy Bell Laboraories Universiy of Michigan Peer Seenkise Carnegie Mellon Universiy Jia Wang AT&T Labs Research ABSTRACT The abiliy o locae nework bolenecks along end-o-end pahs on he Inerne is of grea ineres o boh nework operaors and researchers. For example, knowing where boleneck links are, nework operaors can apply raffic engineering eiher a he inerdomain or inradomain level o improve rouing. Exising ools eiher fail o idenify he locaion of bolenecks, or generae a large amoun of probing packes. In addiion, hey ofen reuire access o boh end poins. In his paper we presen Pahneck, a ool ha allows end users o efficienly and accuraely locae he boleneck link on an Inerne pah. Pahneck is based on a novel probing echniue called Recursive Packe Train (RPT) and does no reuire access o he desinaion. We evaluae Pahneck using wide area Inerne experimens and race-driven emulaion. In addiion, we presen he resuls of an exensive sudy on bolenecks in he Inerne using carefully seleced, geographically diverse probing sources and desinaions. We found ha Pahneck can successfully deec bolenecks for almos 8 of he Inerne pahs we probed. We also repor our success in using he boleneck locaion and bandwidh bounds provided by Pahneck o infer bolenecks and o avoid bolenecks in mulihoming and overlay rouing. Caegories and Subjec Descripors C.2.3 Compuer-Communicaion Neworks]: Nework Operaions Nework Monioring General Terms Algorihms, Measuremen, Experimenaion Keywords Acive probing, packe rain, boleneck locaion, available bandwidh. INTRODUCTION Permission o make digial or hard copies of all or par of his work for personal or classroom use is graned wihou fee provided ha copies are no made or disribued for profi or commercial advanage and ha copies bear his noice and he full ciaion on he firs page. To copy oherwise, o republish, o pos on servers or o redisribue o liss, reuires prior specific permission and/or a fee. SIGCOMM 4, Aug. 3 Sep. 3, 24, Porland, Oregon, USA. Copyrigh 24 ACM /4/8...$5.. The abiliy o locae nework bolenecks along Inerne pahs is very useful for boh end users and Inerne Service Providers (ISPs). End users can use i o esimae he performance of he nework pah o a given desinaion, while an ISP can use i o uickly locae nework problems, or o guide raffic engineering eiher a he inerdomain or inradomain level. Unforunaely, i is very hard o idenify he locaion of bolenecks unless one has access o link load informaion for all he relevan links. This is a problem, especially for regular users, because he design of he Inerne does no provide explici suppor for end users o gain informaion abou he nework inernals. Exising acive bandwidh probing ools also fall shor. Typically hey focus on end-o-end performance 2, 8, 26, 3, 36], while providing no locaion informaion for he boleneck. Some ools do measure hop-by-hop performance, ], bu heir measuremen overhead is ofen very high. In his paper, we presen an acive probing ool Pahneck based on a novel probing echniue called Recursive Packe Train (RPT). I allows end users o efficienly and accuraely locae boleneck links on he Inerne. The key idea is o combine measuremen packes and load packes in a single probing packe rain. Load packes emulae he behavior of regular daa raffic while measuremen packes rigger rouer responses o obain he measuremens. RPT relies on he fac ha load packes inerleave wih compeing raffic on he links along he pah, hus changing he lengh of he packe rain. By measuring he changes using he measuremen packes, he posiion of congesed links can be inferred. Two imporan properies of RPT are ha i has low overhead and does no reuire access o he desinaion. Euipped wih Pahneck, we conduced exensive measuremens on he Inerne among carefully seleced, geographically diverse probing sources and desinaions o sudy he diversiy and sabiliy of bolenecks on he Inerne. We found ha, conrary o he common assumpion ha mos bolenecks are edge or peering links, for cerain probing sources, up o 4 of he boleneck locaions are wihin an AS. In erms of sabiliy, we found ha iner-as bolenecks are more sable han inra-as bolenecks, while AS-level bolenecks are more sable han rouer-level bolenecks. We also show how we can use boleneck locaion informaion and rough bounds for he per-link available bandwidh o successfully infer he boleneck locaions for 54 of he pahs for which we have enough measuremen daa. Finally, using Pahneck resuls from a diverse se of probing sources o randomly seleced desinaions, we found ha over half of all he overlay rouing aemps improve boleneck available bandwidh. The uiliy of mulihoming in improving available bandwidh is over 78.

2 F This paper is organized as follows. We firs describe he Pahneck design in Secion 2 and hen validae he ool in Secion 3. Using Pahneck, we probed a large number of Inerne desinaions o obain several differen daa ses. We use his daa o sudy he properies of Inerne bolenecks in Secion 4, o infer boleneck locaions on he Inerne in Secion 5, and o sudy he implicaions for overlay rouing and mulihoming in Secion 6. We discuss relaed work in Secion 7. In Secion 8 we summarize and discuss fuure work. 2. DESIGN OF PATHNECK Our goal is o develop a ligh-weigh, single-end-conrol boleneck deecion ool. In his secion, we firs provide some background on measuring available bandwidh and hen describe he concep of Recursive Packe Trains and he algorihms used by Pahneck. 2. Measuring Available Bandwidh In his paper, we define he boleneck link of a nework pah as he link wih he smalles available bandwidh, i.e., he link ha deermines he end-o-end hroughpu on he pah. The available bandwidh in his paper refers o he residual bandwidh, which is formally defined in 2, 8]. Informally, we define a choke link as any link ha has a lower available bandwidh han he parial pah from he source o ha link. The upsream rouer for he choke link is called he choke poin or choke rouer. The formal definiion of choke link and choke poin is as follows. Le us assume an endo-end pah from source o desinaion hrough rouers. Link has available bandwidh! "$#&('*)+. Using his noaion, we define he se of choke links as:,-/./2(3$ * : ";# ' )<=;?>@BAC$) D DE! GF and he corresponding se of choke poins (or choke rouers) are,-/./2 HI *4J 5K6 L5NM,-/./2 3O";# =P'Q) Clearly, choke links will have less available bandwidh as hey ge closer o he desinaion, so he las choke link on he pah will be he boleneck link or he primary choke link. We will call he second o las choke link he secondary choke link, and he hird o las one he eriary choke link, ec. Le us now review some earlier work on available bandwidh esimaion. A number of projecs have developed ools ha esimae he available bandwidh along a nework pah 2, 8, 26, 3, 36, 3]. This is ypically done by sending a probing packe rain along he pah and by measuring how compeing raffic along he pah affecs he lengh of he packe rain (or he gaps beween he probing packes). Inuiively, when he packe rain raverses a link where he available bandwidh is less han he ransmission rae of he rain, he lengh of he rain, i.e., he ime inerval beween he head and ail packes in he rain, will increase. This increase can be caused by higher packe ransmission imes (on low capaciy links), or by inerleaving wih he background raffic (heavily loaded links). When he packe rain raverses a link where he available bandwidh is higher han he packe rain ransmission rae, he rain lengh should say he same since here should be lile or no ueuing a ha link. By sending a seuence of rains wih differen raes, i is possible o esimae he available bandwidh on he boleneck link; deails can be found in 8, 26]. Using he above definiion, he links ha increase he lengh of he packe rain correspond o he choke links since hey represen he links wih he lowes available bandwidh on he parial pah raveled by he rain so far. measuremen packes measuremen packes B 5B load packes 6 packes TTL Figure : Recursive Packe Train (RPT). 3 packes Unforunaely, curren echniues only esimae end-o-end available bandwidh since hey can only measure he rain lengh a he desinaion. In order o idenify he boleneck locaion, we need o measure he rain lengh on each link. This informaion can be obained wih a novel packe rain design, called a Recursive Packe Train, as we describe nex. 2.2 Recursive Packe Train Figure shows an example of a Recursive Packe Train (RPT); every box is a UDP packe and he number in he box is is TTL value. The probing packe rain is composed of wo ypes of packes: measuremen packes and load packes. Measuremen packes are sandard raceroue packes, i.e., hey are 6 bye UDP packes wih properly filled-in payload fields. The figure shows 3 measuremen packes a each end of he packe rain, which allows us o measure nework pahs wih up o 3 hops; more measuremen packes should be used for longer pahs. The TTL values of he measuremen packes change linearly, as shown in he figure. Load packes are used o generae a packe rain wih a measurable lengh. As wih he IGI/PTR ool 8], load packes should be large packes ha represen an average raffic load. We use 5 bye packes as suggesed in 8]. The number of load packes in he packe rain deermines he amoun of background raffic ha he rain can inerac wih, so i pays off o use a fairly long rain. In our experimen, we se i empirically in he range of 3 o. Auomaically configuring he number of probing packes is fuure work. The probing source sends he RPT packes in a back-o-back fashion. When hey arrive a he firs rouer, he firs and he las packes of he rain expire, since heir TTL values are. As a resul, he packes are dropped and he rouer sends wo ICMP packes back o he source 7]. The oher packes in he rain are forwarded o he nex rouer, afer heir TTL values are decremened. Due o he way he TTL values are se in he RPT, he above process is repeaed on each subseuen rouer. The name recursive is used o highligh he repeiive naure of his process. A he source, we can use he ime gap beween he wo ICMP packes from each rouer o esimae he packe rain lengh on he incoming link of ha rouer. The reason is ha he ICMP packes are generaed when he head and ail packes of he rain are dropped. Noe ha he measuremen packes are much smaller han he oal lengh of he rain, so he change in packe rain lengh due o he loss of measuremen packes can be negleced. For example, in our defaul configuraion, each measuremen packe accouns for only.2 he packe rain lengh. We will call he ime difference beween he arrival a he source of he wo ICMP packes from he same rouer he packe gap. 2.3 Pahneck The Inference Tool RPT allows us o esimae he probing packe rain lengh on each link along a pah. We use he gap seuences obained from a se of probing packe rains o idenify he locaion of he boleneck link. Pahneck deecs he boleneck link in hree seps: Sep : Labeling of gap seuences. For each probing rain, Pah-

3 A ) R c c ) i ) A ) A ) ^ i k k k ^ i p ^ i A V ) gap value gap value hill poin valley poin hop coun Figure 2: Hill and valley poins. sep changes sep 7 hop coun Figure 3: Maching he gap seuence o a sep funcion. neck labels he rouers where he gap value increases significanly as candidae choke poins. Sep 2: Averaging across gap seuences. Rouers ha are freuenly labeled as candidae choke poins by he probing rains in he se are idenified as acual choke poins. Sep 3: Ranking choke poins. Pahneck ranks he choke poins wih respec o heir packe rain ransmission rae. In he remainder of his secion, we describe in deail he algorihms used in each of he hree seps Labeling of Gap Seuences Under ideal circumsances, gap values only increase (if he available bandwidh on a link is no sufficien o susain he rae of he incoming packe rain) or say he same (if he link has enough bandwidh for he incoming packe rain), bu i should never decrease. In realiy, he bursiness of compeing raffic and reverse pah effecs add noise o he gap seuence, so we preprocess he daa before idenifying candidae choke poins. We firs remove any daa for rouers from which we did no receive boh ICMP packes. If we miss daa for over half he rouers, we discard he enire seuence. We hen fix he hill and valley poins where he gap value decreases in he gap seuence (Figure 2). A hill poin is defined as R in a hree-poin group (R J RTS ) wih gap values saisfying A 'UA WV AX V AY'ZA S. A valley poin is defined in a similar way wih S. Since in boh cases, he decrease is shor-erm A (one sample), we assume i is caused by noise and we replace wih he closes neighboring gap value. We now describe he core par of he labeling algorihm. The idea is o mach he gap seuence o a sep funcion (Figure 3), where each sep corresponds o a candidae choke poin. Given a gap seuence wih ]\ gap values, we wan o idenify he sep funcion ha is he bes fi, where bes is defined as he sep funcion for which he sum of absolue difference beween he gap seuence and he sep funcion across all he poins is minimal. We reuire he sep funcion o have clearly defined seps, i.e., all seps mus be larger han a hreshold (^_`\`R ) o filer ou measuremen noise. Cab@Jc We use ^d\ bcj)fe ^ (go^ ) as he hreshold. This value is relaively small compared wih possible sources of error (o be discussed in Secion 2.4), bu we wan o be conservaive in idenifying candidae choke poins. We use he following dynamic programming algorihm o idenify he sep funcion. Assume we have a gap subseuence be- ween hop and hop dh`a E >iyafj *# : ( ), and le us define k ml E eu 5no 5p 7 sr;b, and he disance sum of he subseuence as ^_ ^v Cwj k l E 5no 6 >iyafj ` kt 5X6 c. Le R7_ j ` k denoe he minimal sum of he disance sums for he segmens beween hops and (including hops and ), given ha here are a mos seps. The key observaion is ha, given he opimal spliing of a subseuence, he spliing of any shorer inernal subseuence delimied by wo exising spliing poins mus c be an opimal spliing for his inernal subseuence. Therefore, R7_ j ` hk can be recursively defined as he follows: R7_ j ` k x eu ^_ ^v CWj ` k?"zyq{# Ca )O4dc R7_ j } c R7_~ j ` k F V "zyq{# R7_`~ j ` k Ca)4dc RK_ j `=f r c RK_ j =r : {#Q= ' "N# ' } koƒ 6 j `=T k7w j =ˆr : } k 6 ^_s\r F j `Š=T Here k denoes he las sep value of he opimal sep = funcion fiing he gap subseuence beween and wih a mos seps, and j =zrz KŒ k denoes he firs sep value of =<r he opimal sep funcion fiing he gap subseuence beween and wih a mos }W seps. The algorihm begins wih U" and hen ieraively improves he soluion by exploring larger c values of c. Every ime R7_`~ j ` k is used o assign he value for R7_ j ` hk, a new spliing {Ž j ` = poin is creaed. The spliing poin is recorded in a se k, which is he se of opimal spliing poins for he subseuence beween <Ž j and"x using a mos spliing poins. The algorihm reurns ]\ ]\ k as he se of opimal spliing poins for he. enire gap seuence. The ime complexiy of his algorihm is ]\ )+ d, which is accepable considering he small value of ]\ on he Inerne. Since our goal is o deec he primary choke poin, our implemenaion only reurns he op hree choke poins wih he larges hree seps. If he algorihm does no find a valid spliing poin, i.e., <Ž j "X \ ]\ as he candidae choke poin.?, i simply reurns he source Averaging Across Gap Seuences To filer ou effecs caused by bursy raffic on he forward and reverse pahs, Pahneck uses resuls from muliple probing rains (e.g., 6 o probing rains) o compue confidence informaion for each candidae choke poin. To avoid confusion, we will use he erm probing for a single RPT run and he erm probing se for a group of probings (generally probings). The oucome of Pahneck is he summary resul for a probing se. For he opimal spliing of a gap seuence, le he seuence of sep values be ^ "/# # *, where is he oal number of candidae <s#z #Q * choke poins. The confidence for a candidae choke poin is compued as bšc)+ Inuiively, he confidence denoes he percenage of available bandwidh change implied by he gap value change. For he special case where he source is reurned as he candidae choke poin, we se is confidence value o. Nex, e each candidae choke poin in he probing se we calculae _s\ as he freuency wih which he candidae choke poin appears in he probing se bšc)+ š"x wih. Finally, we selec hose choke poins wih _`\ "X œ. Therefore, he final choke poins for a pah are he candidaes ha appear wih high confidence in a leas half of he probings in he probing se. In

4 A Secion 3.4, we uanify he sensiiviy of Pahneck o hese parameers Ranking Choke Poins For each pah, we rank he choke poins based on heir average gap value in he probing se. The packe rain ransmission rae is _^ A p, where _^ is he oal size for all he packes in he rain and is he gap value. Tha is, he larger he gap value, he more he packe rain was sreched ou by he link, suggesing a lower available bandwidh on he corresponding link. As a resul, we idenify he choke poin wih he larges gap value as he boleneck of he pah. Noe ha since we canno conrol he packe rain srucure a each hop, he RPT does no acually measure he available bandwidh on each link, so in some cases, Pahneck could selec he wrong choke poin as he boleneck. For example, on a pah where he rue boleneck is early in he pah, he rae of he packe rain leaving he boleneck can be higher han he available bandwidh on he boleneck link. As a resul, a downsream link wih slighly higher available bandwidh could also be idenified as a choke poin and our ranking algorihm will misakenly selec i as he boleneck. Noe ha our mehod of calculaing he packe rain ransmission rae is similar o ha used by cprobe 3]. The difference is ha cprobe esimaes available bandwidh, while Pahneck esimaes he locaion of he boleneck link. Esimaing available bandwidh in fac reuires careful conrol of he iner-packe gap for he rain 26, 8] which neiher ool provides. While Pahneck does no measure available bandwidh, we can use he average per-hop gap values o provide a rough upper or lower bound for he available bandwidh of each link. We consider hree cases: Case : For a choke link, i.e., is gap increases, we know ha he available bandwidh is less han he packe rain rae. Tha is, he rae compued above is an upper bound for he available bandwidh on he link. Case 2: For links ha mainain heir gap relaive o he previous link, he available bandwidh is higher han he packe rain rae, and we use as a lower bound for he link available bandwidh. Case 3: Some links may see a decrease in gap value. This decrease is probably due o emporary ueuing caused by raffic bursiness, and according o he packe rain model discussed in 8], we canno say anyhing abou he available bandwidh. Considering ha he daa is noisy and ha link available bandwidh is a dynamic propery, hese bounds should be viewed as very rough esimaes. We provide a more deailed analysis for he bandwidh bounds on he boleneck link in Secion Pahneck Properies Pahneck mees he design goals we idenified earlier in his secion. Pahneck does no need cooperaion of he desinaion, so i can be widely used by regular users. Pahneck also has low overhead. Each measuremen ypically uses 6 o probing rains of 3 o load packes each. This is a very low overhead compared o exising ools such as pahchar ] and BFind ]. Finally, Pahneck is fas. For each probing rain, i akes abou one roundrip ime o ge he resul. However, o make sure we receive all he reurned ICMP packes, Pahneck generally wais for 3 seconds he longes roundrip ime we have observed on he Inerne afer sending ou he probing rain, and hen exis. Even in his case, a single probing akes less han 5 seconds. In addiion, since each packe rain probes all links, we ge a consisen se of measuremens. This, for example, allows Pahneck o idenify muliple choke poins and rank hem. Noe however ha Pahneck is biased owards early choke poins once a choke poin has increased he lengh of he packe rain, Pahneck may no longer be able o see downsream links wih higher or slighly lower available bandwidh. A number of facors could influence he accuracy of Pahneck. Firs, we have o consider he ICMP packe generaion ime on rouers. This ime is differen for differen rouers and possibly for differen packes on he same rouer. As a resul, he measured gap value for a rouer will no exacly mach he packe rain lengh a ha rouer. Forunaely, measuremens in 6] and ] show ha he ICMP packe generaion ime is prey small; in mos cases i is beween go^ and 5gO^. We will see laer ha over 5 of he gap changes of deeced choke poins in our measuremens are larger han 5gO^. Therefore, while large differences in ICMP generaion ime can affec individual probings, hey are unlikely o significanly affec Pahneck boleneck resuls. Second, as ICMP packes ravel o he source, hey may experience ueueing delay caused by reverse pah raffic. Since his delay can be differen for differen packes, i is a source of measuremen error. We are no aware of any work ha has uanified reverse pah effecs. In our algorihm, we ry o reduce he impac of his facor by filering ou he measuremen ouliers. Noe ha if we had access o he desinaion, we migh be able o esimae he impac of reverse pah ueueing. Third, packe loss can reduce Pahneck s effeciveness. Load packe loss can affec RPT s abiliy o inerleave wih background raffic hus possibly affecing he correcness of he resul. Los measuremen packes are deeced by los gap measuremens. Noe ha i is unlikely ha Pahneck would lose significan numbers of load packes wihou a similar loss of measuremen packes. Considering he low probabiliy of packe loss in general 23], we do no believe packe loss will affec Pahneck resuls. Fourh, muli-pah rouing, which is someimes used for load balancing, could also affec Pahneck. If a rouer forwards packes in he packe rain o differen nex-hop rouers, he gap measuremens will become invalid. Pahneck can usually deec such cases by checking he source IP address of he ICMP responses. In our measuremens, we do no use he gap values in such cases. Pahneck also has some deploymen limiaions. Firs, we discovered ha nework firewalls ofen only forward 6 bye UDP packes ha sricly conform o he packe payload forma used by sandard Unix raceroue implemenaion, while hey drop any oher UDP probing packes, including he load packes in our RPT. If he sender is behind such a firewall, Pahneck will no work. Similarly, if he desinaion is behind a firewall, no measuremens for links behind he firewall can be obained by Pahneck. Second, even wihou any firewalls, Pahneck may no be able o measure he packe rain lengh on he las link, because he ICMP packes sen by he desinaion hos canno be used. In heory, he desinaion should generae a desinaion por unreachable ICMP message for each packe in he rain. However, due o ICMP rae limiing, he desinaion nework sysem will ypically only generae ICMP packes for some of he probing packes, which ofen does no include he ail packe. Even if an ICMP packe is generaed for boh he head and ail packes, he accumulaed ICMP generaion ime for he whole packe rain makes he reurned inerval worhless. Of course, if we have he cooperaion of he desinaion, we can ge a valid gap measuremen for he las hop by using a valid por number, hus avoiding he ICMP responses for he load packes. 3. VALIDATION We use boh Inerne pahs and he Emulab esbed 3] o evaluae Pahneck. Inerne experimens are necessary o sudy Pahneck

5 ª ª Table : Bolenecks deeced on Abilene pahs. Probe ž ŸdB Boleneck AS pah desinaion (Uah/CMU) rouer IP ( - L o ) calren2.7/ princeon.64/ sox.62/ ogig.7/ (Uah) (CMU) L < is boleneck rouer s AS#, f is is pos-hop rouer s AS#. calren = princeon = sox = ogig = wih realisic background raffic, while he Emulab esbed provides a fully conrolled environmen ha allows us o evaluae Pahneck wih known raffic loads. Besides he deecion accuracy, we also examine he accuracy of he Pahneck bandwidh bounds and he sensiiviy of Pahneck o is configuraion parameers. Our validaion does no sudy he impac of he ICMP generaion ime. 3. Inerne Validaion For a horough evaluaion of Pahneck on Inerne pahs, we would need o know he acual available bandwidh on all he links of a nework pah. This informaion is impossible o obain for mos operaional neworks. The Abilene backbone, however, publishes is backbone opology and raffic load (5-minue SNMP saisics) ], so we decided o probe Abilene pahs. The experimen is carried ou as follows. We used wo sources: a hos a he Universiy of Uah and a hos a Carnegie Mellon Universiy. Based on Abilene s backbone opology, we chose 22 probing desinaions for each probing source. We make sure ha each of he major rouers on he Abilene backbone is included in a leas one probing pah. From each probing source, we probed every desinaion imes, wih a 2-second inerval beween wo consecuive probings. To avoid inerference, he experimens conduced a Uah and a CMU bcj)o «"h were run a differen imes. Using and _s\ "X œ, we only deeced 5 nonfirs-hop boleneck links on he Abilene pahs (Table ). This is no surprising since Abilene pahs are known o be over-provisioned, and we seleced pahs wih many hops inside he Abilene core. The _s\ values for he probes originaing from Uah and CMU are very similar, possibly because hey observed similar congesion condiions. By examining he IP addresses, we found ha in 3 of he 4 cases ( is he excepion), boh he Uah and CMU based probings are passing hrough he same boleneck link close o he desinaion; an explanaion is ha hese bolenecks are very sable, possibly because hey are consrained by link capaciy. Unforunaely, all hree bolenecks are ouside Abilene, so we do no have he load daa. For he pah o he boleneck links appear o be wo differen peering links going o AS46. For he pah from CMU o he ougoing link of he boleneck rouer is an OC-3 link. Based on he link capaciies and SNMP daa, we are sure ha he OC-3 link is indeed he boleneck. We do no have he SNMP daa for he Uah links, so we canno validae he resuls for he pah from Uah o Tesbed Validaion We use he Emulab esbed o sudy he deailed properies of A meaningful sudy of he ICMP impac reuires access o differen ypes of rouers wih real raffic load, bu we do no have access o such faciliies. 3M.5ms 5M M.ms 2 X 8M 7M.4ms 3 5M.4ms 4 4ms 5 Y 2ms 6 4ms 7 4ms 8 3M ms Figure 4: Tesbed configuraion. Table 2: The esbed validaion experimens # Trace Commens 5 2 ligh-race on all Capaciy-deermined Mbps exponenial-load on, ligh-race oherwise heavy-race on, ligh-race oherwise heavy-race on, ligh-race oherwise exponenial-load on boh direcions boleneck Load-deermined boleneck Two-boleneck case Two-boleneck case The impac of reverse raffic Pahneck. Since Pahneck is a pah-oriened measuremen ool, we use a linear opology (Figure 4). Nodes and are he probing source and desinaion, while nodes -8 are inermediae rouers. The link delays are roughly se based on a raceroue measuremen from a CMU hos o The link capaciies are configured using he Dummyne 2] package. The capaciies for links and depend on he scenarios. Noe ha all he esbed nodes are PCs, no rouers, so heir properies such as he ICMP generaion ime are differen from hose of rouers. As a resul, he esbed experimens do no consider some of he rouer relaed facors. The dashed arrows in Figure 4 represen background raffic. The background raffic is generaed based on wo real packe races, called ligh-race and heavy-race. The ligh-race is a sampled race (using prefix filers on he source and desinaion IP addresses) colleced in fron of a corporae nework. The raffic load varies from around 5Kbps o 6Mbps, wih a median load of 2Mbps. The heavy-race is a sampled race from an ougoing link of a daa cener conneced o a ier- ISP. The raffic load varies from 4Mbps o 36Mbps, wih a median load of 8Mbps. We also use a simple UDP raffic generaor whose insananeous load follows an exponenial disribuion. We will refer o he load from his generaor as exponenial-load. By assigning differen races o differen links, we can se up differen evaluaion scenarios. Since all he background raffic flows used in he esbed evaluaion are very bursy, hey resul in very challenging scenarios. Table 2 liss he configuraions of five scenarios ha allow us o analyze all he imporan properies of Pahneck. For each scenario, we use Pahneck o send probing rains. Since hese scenario are used for validaion, we only use he resuls for which we received all ICMP packes, so he percenage of valid probing is lower han usual. During he probings, we colleced deailed load daa on each of he rouers allowing us o compare he probing resuls wih he acual link load. We look a Pahneck performance for boh probing ses (i.e., resul for consecuive probings as repored bšc)+ " by Pahneck) and individual probings. For probing ses, we use and _s\ "X œ o idenify choke poins. The real background raffic load is compued as he average load for he inerval ha includes he probes, which is around 6 seconds. For individual probings, we only C bcj)+ I " use for filering, and he load is compued using a 2 ^ packe race cenered around he probing packes, i.e., we use he insananeous load.

6 change cap of link Y: 2 3Mbps, wih no load 75 change load on link Y (5Mbps): 2 2Mbps CDF.5 gap value (us) 6 55 gap value (us) all wrong bandwidh difference (Mbps) hop ID hop ID Figure 6: Cumulaive disribuion of bandwidh difference in experimen 3. Figure 5: Comparing he gap seuences for capaciy (lef) and load-deermined (righ) bolenecks Experimen Capaciy-deermined Boleneck In his experimen, we se he capaciies of and o 5Mbps and 2Mbps, and use ligh-race on all he links; he saring imes wihin he race are randomly seleced. All probings deec hop 6 (i.e., link ) as he boleneck. All oher candidae choke poins are " bšc)+ ' filered ou because of a low confidence value (i.e., ). Obviously, he deecion resuls for he probing ses are also accurae. This experimen represens he easies scenario for Pahneck, i.e., he boleneck is deermined by he link capaciy, and he background raffic is no heavy enough o affec he boleneck locaion. This is however an imporan scenario on he Inerne. A large fracion of he Inerne pahs fall ino his caegory because only a limied number of link capaciies are widely used and he capaciy differences end o be large Experimen 2 Load-deermined Boleneck Besides capaciy, he oher facor ha affecs he boleneck posiion is he link load. In his experimen, we se he capaciies of boh and o 5Mbps. We use he 35Mbps exponenial-load on and he ligh-race on oher links, so he difference in raffic load on and deermines he boleneck. Ou of probings, 23 had o be discarded due o ICMP packe loss. Using he remaining 77 cases, he probing ses always correcly idenify as he boleneck link. Of he individual probings, 6 probings correcly deec as he op choke link, 2 probings pick link ± ² ³Y (i.e., he link is deeced as he secondary choke link. 6 probings miss he real boleneck. In summary, he accuracy for individual probings is 8.6. afer ) as he op choke link and Comparing he Impac of Capaciy and Load To beer undersand he impac of link capaciy and load in deermining he boleneck, we conduced wo ses of simplified experimens using configuraions similar o hose used in experimens and 2. Figure 5 shows he gap measuremens as a funcion of he hop coun (µ axis). In he lef figure, we fix he capaciy of o 5Mbps and change he capaciy of from 2Mbps o 3Mbps wih a sep size of Mbps; no background raffic is added on any link. In he righ figure, we se he capaciies of boh and o 5Mbps. We apply differen CBR loads o (changing from 2Mbps o 2Mbps) while here is no load on he oher links. For each configuraion, we execued probings. The wo figures plo he median gap value for each hop; for mos poins, he 3-7 percenile inerval is under 2g^. In boh configuraions, he boleneck available bandwidh changes in exacly he same way, i.e., i increases from 2Mbps o 3Mbps. However, he gap seuences are uie differen. The gap increases in he lef figure are regular and mach he capaciy changes, since he lengh of he packe rain is sricly se by he link capaciy. In he righ figure, he gaps a he desinaion are less regular and smaller. Specifically, hey do no reflec he available bandwidh on he link (i.e., he packe rain rae exceeds he available bandwidh). The reason is ha he back-o-back probing packes compee un-fairly wih he background raffic and hey can miss some of he background raffic ha should be capured. This observaion is consisen wih he principle behind TOPP 26] and IGI/PTR 8], which saes ha he probing rae should be se properly o accuraely measure he available bandwidh. This explains why Pahneck s packe rain rae a he desinaion provides only an upper bound on he available bandwidh. Figure 5 shows ha he upper bound will be igher for capaciy-deermined bolenecks han for load-deermined bolenecks. The fac ha he gap changes in he righ figure are less regular han ha in he lef figure also confirms ha capaciy-deermined bolenecks are easier o deec han load-deermined bolenecks Experimens 3 & 4 Two Bolenecks In hese wo experimens, we se he capaciies of boh and o 2Mbps, so we have wo low capaciy links and he boleneck locaion will be deermined by load. In experimen 3, we use he heavy-race for and he ligh-race for oher links. The probing se resuls are always correc, i.e., is deeced as he boleneck. When we look a he 86 valid individual probings, we find ha is he real boleneck in 7 cases; in each case Pahneck successfully idenifies as he only choke link, and hus he boleneck. In he remaining 7 cases, is he real boleneck. Pahneck correcly idenifies in 65 probings. In he oher 4 probings, Pahneck idenifies as he only choke link, i.e., Pahneck missed he real boleneck link. The raw packe races show ha in hese 4 incorrec cases, he bandwidh difference beween and is very small. This is confirmed by Figure 6, which shows he cumulaive disribuion of he available bandwidh difference beween and for he 4 wrong cases (he dashed curve), and for all 86 cases (he solid curve). The resul shows ha if wo links have similar available bandwidh, Pahneck has a bias owards he firs link. This is because he probing packe rain has already been sreched by he firs choke link, so he second choke link can be hidden. As a comparison, we apply he heavy-race o boh and in experimen ou of he 77 valid probings correcly idenify as he boleneck; 2 probings correcly idenify as he boleneck; and 8 probings miss he real boleneck link and idenify as he only boleneck. Again, if muliple links have similar available bandwidh, we observe he same bias owards he early link.

7 Table 3: The number of imes of each hop being a candidae choke poin. Rouer }¹NºI»:¼ ž ŸJ ºI»:¼ ½ Experimen 5 Reverse Pah Queuing To sudy he effec of reverse pah ueuing, we se he capaciies of and o 5Mbps and 2Mbps, and apply exponenial-load in boh direcions on all links (excep he wo edge links). The average load on each link is se o 3 of he link capaciy. We had 8 valid probings. The second row in Table 3 liss he number of imes bcj)+ Œ?" ha each hop is deeced as a candidae choke poin (i.e., wih ). We observe ha each hop becomes a candidae choke poin in some probings, so reverse pah raffic does affec he deecion accuracy of RPTs. However, he use of probing ses reduces he impac of reverse pah raffic. We analyzed he 8 valid probings as 8 ses of consecuive probings each. The las row e Table 3 shows how ofen links are idenified as choke poins ( _`\ ¾" œ ) by a probing se. The real boleneck, hop 6, is mos freuenly idenified as he acual boleneck (las choke poin), alhough in some cases, he nex hop (i.e., hop 7) is also a choke poin and is hus seleced as he boleneck. This is a resul of reverse pah raffic. Normally, he rain lengh on hop 7 should be he same as on hop 6. However, if reverse pah raffic reduces he gap beween he hop 6 ICMP packes, or increases he gap beween he hop 7 ICMP packes, i will appear as if he rain lengh has increased and hop 7 will be labeled as a choke poin. We hope o une he deecion algorihm o reduce he impac of his facor as par of fuure work. 3.3 Validaion of Bandwidh Bounds A number of groups have shown ha packe rains can be used o esimae he available bandwidh of a nework pah 26, 8, 2]. However, he source has o carefully conrol he iner-packe gap, and since Pahneck sends he probing packes back-o-back, i canno, in general, measure he available bandwidh of a pah. Insead, as described in Secion 2.3, he packe rain rae a he boleneck link can provide a rough upper bound for he available bandwidh. In his secion, we compare he upper bound on available bandwidh on he boleneck link repored by Pahneck wih end-o-end available bandwidh measuremens obained using IGI/PTR 8] and Pahload 2]. Since boh IGI/PTR and Pahload need wo-end conrol, we used RON nodes for our experimens, as lised in he BW column in Table 4; his resuls in nework pahs for our experimen. On each RON pah, we obain Pahneck probings, 5 IGI/PTR measuremens, and Pahload measuremen 2. The esimaion for he upper bound in Pahneck was done as follows. If a boleneck can be deeced from he probings, we use he median packe rain ransmission rae on ha boleneck. Oherwise, we use he larges gap value in each probing o calculae he packe rain rae and use he median rain rae of he probings as he upper bound. Figure 7 compares he average of he available bandwidh esimaes provided by IGI, PTR, and Pahload (µ axis) wih he upper bound for he available bandwidh provided by Pahneck ( axis). The measuremens are roughly clusered in hree areas. For low bandwidh pahs (boom lef corner), Pahneck provides We force Pahload o sop afer flees of probing. If Pahload has no converged, we use he average of he las 3 probings as he available bandwidh esimae. Table 4: Probing sources from PlaneLab (PL) and RON. ID Probing AS Locaion Upsream Tes- B G S O M Source Number Provider(s) bed W E T V H aros 652 UT 7 RON 2 ashburn 7 DC 24 PL 3 bkly-cs 25 CA 25, 3356, PL 423, columbia 4 NY 635 PL 5 diku 835 Denmark 263 PL 6 emulab 755 UT 2 7 frankfur 3356 Germany 23, 78 PL 8 grouse 7 GA 23, 78 PL gs274 PA 55 bkly-inel 78 CA 23 PL inel 78 CA 23 RON 2 jfk 354 NY 23, 78 RON 3 jhu 5723 MD 78 PL 4 nbgisp 8473 OR 3356 PL 5 norel 85 Canada 477 RON 6 nyu 2 NY 657, 78 RON 7 princeon 88 NJ 78 PL 8 purdue 7 IN 782 PL 2 rpi NY 635 PL 2 uga 347 GA 663 PL 2 umass 24 MA 24 PL 22 unm 3388 NM 23 PL 23 uah 755 UT 2 PL 24 uw-cs 73 WA PL 25 vineyard 78 MA 2, 6347 RON 26 rugers 46 NJ 78 PL 27 harvard MA 663 PL 28 depaul 23 CH 6325, 663 PL 2 orono 23 Canada 663 PL 3 halifax 65 Canada 537 PL 3 unb 6 Canada 855 PL 32 umd 27 MD 86 PL 33 darmouh 755 NH 3674 PL 34 virginia 225 VA 23 PL 35 upenn 55 PA 663 PL 36 cornell 26 NY 635 PL 37 mazu 3356 MA 78 RON 38 kais 78 Korea 38 PL 3 cam-uk 786 UK 88 PL 4 ucsc 573 CA 252 PL 4 ku 246 KS 37 PL 42 snu-kr 488 Korea 4766 PL 43 bu MA 2 PL 44 norhwesern 3 CH 6325 PL 45 cmu PA 55 PL 46 mi-pl 3 MA PL 47 sanford 32 CA 663 PL 48 wusl 2552 MO 24 PL 4 msu 237 MI 356 PL 5 uky 437 KY 2 PL 5 ac-uk 786 UK 3356 PL 52 umich 237 MI 356 PL 53 cornell 26 NY 635 RON 54 lulea 283 Sweden 653 RON 55 ana 354 CA 23, 78 RON 56 ccicom 364 UT 3356, 2 RON 57 ucsd 7377 CA 252 RON 58 uah 755 UT 2 RON BW: measuremens for bandwidhesimaion; GE: measuremens for general properies; ST: measuremens for sabiliyanalysis; OV: measuremens for overlay analysis; MH: measuremens for mulihominganalysis. denoes he wo probing hoss obained privaely. Pahneck esimae of upperbound (Mbps) available bandwidh (Mbps) Figure 7: Comparison beween he bandwidh from Pahneck wih he available bandwidh measuremen from IGI/PTR and Pahload. a fairly igh upper bound for he available bandwidh on he boleneck link, as measured by IGI, PTR, and Pahload. In he upper lef region, here are low bandwidh pahs for which he upper bound provided by Pahneck is significanly higher han he available bandwidh measured by IGI, PTR, and Pahload. Analysis

8 CDF gap difference (us) Figure 8: Disribuion of sep size on he choke poin. fracion of pahs deeced d_rae value conf value Figure bcj)o : Sensiiviy of Pahneck o he values of and _s\. shows ha he boleneck link is he las link, which is no visible o Pahneck. Insead, Pahneck idenifies an earlier link, which has a higher bandwidh, as he boleneck. The hird cluser corresponds o high bandwidh pahs (upper righ corner). Since he curren available bandwidh ools have a relaive measuremen error around 3 8], we show he wo 3 error margins as doed lines in Figure 7. We consider he upper bound for he available bandwidh provided by Pahneck o be valid if i falls wihin hese error bounds. We find ha mos upper bounds are valid. Only 5 daa poins fall ouside of he region defined by he wo 3 lines. Furher analysis shows ha he daa poin above he region corresponds o a pah wih a boleneck on he las link, similar o he cases menioned above. The four daa poins below he region belong o pahs wih he same source node (lulea). We have no been able o deermine why he Pahneck bound is oo low. 3.4 Impac of Configuraion Parameers The Pahneck algorihms described in Secion 2.3 use hree configuraion parameers: he hreshold used o pick candidae choke poins (^_`\`R = g^ bšc)+ ), he confidence value ( =.), and he deecion rae ( _s\ =.5). We now invesigae he sensiiviy of Pahneck o he value of hese parameers. To show how he go^ hreshold for he sep size affecs he algorihm, we calculaed he cumulaive disribuion funcion for he sep sizes for he choke poins deeced in he GE se of Inerne measuremens (Table 4, o be described in Secion 4.). Figure 8 shows ha over of he choke poins have gap increases larger han go^, while fewer han of he choke poins have gap increases around g^. Clearly, changing he sep hreshold o a larger value (e.g., 5gO^ ) will no bšc)+ change e resuls significanly. To undersand he impac of and _s\, we reran he Pahneck bcj)+ deecion algorihm by varying from.5 o.3 and _s\ from.5 o. Figure plos he percenage of pahs wih..5 bšc)+ a leas one choke poin ha saisfies boh he bcj)+ and _`\ hresholds. The resul shows ha, as we increase and _s\, fewer pahs have idenifiable choke poins. bšc)+ This e exacly wha we would expec. Wih higher values for and _s\, i becomes more difficul for a link o be consisenly idenified as a choke link. The fac ha he resuls are much less sensiive o _s\ bšc)+ han shows ha mos of he choke poin locaions are fairly sable wihin a probing se (shor ime duraion). The available bandwidh of he links on a pah and he locaion of boh choke poins and he boleneck are dynamic properies. The Pahneck probing rains effecively sample hese properies, bu he resuls are subjec o noise. Figure shows he radeoffs involved in using hese samples bcj)+ o esimae he choke poin locaions. Using high values for and _s\ will resul in a small number of sable choke poins, while using lower values will also idenify more ransien choke poins. Clearly he righ choice will depend bcj)+ on e he daa is used. We see ha for our choice of and _`\ values,. and.5, Pahneck can clearly idenify one or more choke poins on almos 8 of he pahs we probed. The graph suggess ha our selecion of hresholds corresponds o a fairly liberal noion of choke poin. 4. INTERNET BOTTLENECK MEASURE- MENT I has been a common assumpion in many sudies ha bolenecks ofen occur a edge links and peering links. In his secion, we es his popular assumpion using Pahneck, which is sufficienly ligh-weigh o conduc large scale measuremens on he Inerne. Using he same se of daa, we also look a he sabiliy of Inerne bolenecks. 4. Daa Collecion We chose a se of geographically diverse nodes from Planelab 4] and RON 3] as probing sources. Table 4 liss all he nodes ha we used for collecing measuremen daa for he analysis in his paper. Among hem, GE is used in Secions 4.2, 4.3, and 5, ST is used in Secion 4.4, OV is used in Secion 6., and MH is used in Secion 6.2. These nodes reside in 46 disinc ASes and are conneced o 3 disinc upsream providers, providing good coverage for norh America and pars of Europe. We carefully chose a large se of desinaions o cover as many disinc iner-as links as possible. Our algorihm selecs desinaion IP addresses using he local BGP rouing able informaion of he probe source, using a similar mehod as described in 24]. In mos cases, we do no have access o he local BGP able for he sources, bu we almos always can obain he BGP able for heir upsream provider, for example from public BGP daa sources such as RoueViews 6]. The upsream provider informaion can be idenified by performing raceroue o a few randomly chosen locaions such as and from he probe sources. In he case of mulihomed source neworks, we may no be able o obain he complee se of upsream providers. Given a rouing able, we firs pick a. or.2 IP address for each prefix possible. The prefixes ha are compleely covered by heir subnes are no seleced. We hen reduce he se of IP addresses by eliminaing he ones whose AS pah saring from he probe source are par of oher AS pahs. Here we make he simplificaion ha here is only a single iner-as link beween each pair of adjacen ASes. As he core of he Inerne is repeaedly raversed for he over 3, desinaions we seleced for each source, we would expec ha each of hese iner-as links is raversed many imes by our probing packes. Noe ha he desinaion IP addresses

9 à v ^ _ \ à v à v ^ _ \ à l! p = C p = Fracion of pahs Pah source ID (a) Disribuion of number of choke links per source. CDF boleneck link choke link populariy (b) Populariy of choke links and boleneck links. Figure : Disribuion and populariy of choke links. obained from his procedure do no necessarily correspond o real end hoss. In our experimens, each source node probes each desinaion once using Pahneck. Pahneck is configured o use a probing se of probing rains and i hen uses he resuls of he probing se o calculae he locaion of he choke poins as well as a rough esimae for he available bšc)+ " bandwidh for he corresponding choke links. We again use he and _`\ Á"X œ hresholds o selec choke poins. Due o he small measuremen ime, we were able o finish probing o around 3,5 desinaions wihin 2 days. 4.2 Populariy As described in previous secions, Pahneck is able o deec muliple choke links on a nework pah. In our measuremens, Pahneck deeced up o 5 choke links per pah. Figure (a) shows he number of pahs ha have o 5 choke links. We found ha, for all probing sources, fewer han 2 of he pahs repor more han 3 choke links. We also noiced ha a good porion of he pahs have no choke link. This number varies from 3 o 6 across differen probing sources. The reason why Pahneck canno deec a choke link is generally bcj)+ Œ "X ha he raffic e hose pahs is oo bursy so no link mees he and _`\ "X œ crieria. In our measuremens, we observe ha some links are deeced as choke links in a large number of pahs. For a link ha is idenified C Ž@Jc as a choke link by a leas one Pahneck probe, le à \ denoe he oal number of probes ha raverse and le v C Žc i Ž@Jc \ denoe he oal number >@ of probes of ha deec Žc as a choke link. We compue he R7vT link as follows: Žc R}v} >@ _s < C Žc i Ž@Jc C Ž@Jc \ \ The populariy of a boleneck link is defined similarly. Figure (b) shows he cumulaive disribuion of he populariy of choke links (dashed curve) and boleneck links (solid curve) in our measuremens. We observe ha half of he choke links are deeced _ in 2 or less of he Pahneck probings ha raverse hem. Abou 5 of he choke links are deeced by all he probes. The same observaions hold for he populariy of boleneck links. 4.3 Locaion In general, a link is considered o be an inra-as link if boh ends of belong o he same AS; oherwise, is an iner-as link. In pracice, i is surprisingly difficul o idenify a link a he boundary beween wo ASes due o he naming convenion 24] ha is currenly used by some service providers. In our experimens, we firs use he mehod described in 24] o map an IP address o is AS. We hen classify a link ino one of he following hree caegories: (i) Inra-AS link. A link is inra-as if boh ends of and is adjacen links belong o he same AS. Noe ha we are very conservaive in reuiring ha inra-as links fully reside inside a nework. (ii) Iner-AS link. A link is iner-as if he ends of do no belong o he same AS. The link is likely o be an iner-as link, bu i is also possible ha is one hop away from he acual iner-as link. (iii) Iner-AS link. A link is iner-as if boh ends of belong o he same AS and i is adjacen o an iner-as link. In his case, appears o be one hop away from he link where AS numbers change, bu i migh be he acual iner-as link. Noe ha, using our definiions, he iner-as links and iner-as links should conain all he iner-as links and some inra-as links ha are one hop away from he iner-as links. Figure (a) shows he disribuion of choke links and boleneck links across hese hree caegories. We observe ha for some probing sources up o 4 of boh he boleneck links and choke links occur a inra-as links. Considering our very conservaive definiion of inra-as link, his is surprising, given he widely used assumpion ha bolenecks ofen occur a he boundary links beween neworks. For a choke link in a à Ž probing se, we compue is normalized locaion (denoed by `Žˆ ) on he corresponding nework pah in he following = way. Le! Jh! 5 denoe he AS-level pah, where is he lengh of he AS pah. (i) If is in he -h AS along à he pah, frm hen Žˆ * Ã. (ii) If is he link beween he -h and ( )-h ASes, hen ŽˆL ÁÄfr«"X œ. Noe ha he value à à of Žˆ is in he range of, ]. The smaller he value of Žˆ, he closer he Ž Ž choke hžå link is C o V he " probing source. Given a se of probing ses ( ) ha deec as a choke link, he normalized locaion of link is compued as Å E n à ŽTE Since he boleneck link is he primary choke link, he definiion of normalized locaion also applies o he boleneck link. Figure (b) shows he cumulaive disribuion of he normalized locaions of boh boleneck and choke links. The curves labeled (unweighed) show he disribuion when all links have an eual weigh, while for he curves labeled (weighed) we gave each link a weigh eual o he number of probing ses in which he link is deeced as a boleneck or choke link. This is ineresing because we observed in Figure (b) ha some links are much more likely o be a boleneck or a choke link han ohers. The resuls show ha abou 65 of he choke links appear in he firs half of an end-o-end pah (i.e., à ŽˆN# " œ ). By comparing weighed wih unweighed curves, we also observe ha high-freuency choke links end o be locaed closer o he source. Finally, by comparing he curves for choke links and boleneck links, we observe ha boleneck locaions are more evenly disribued along he end-oend pah. These observaions are in par influenced by he definiion of choke link and boleneck, and by Pahneck s bias owards earlier