Efficient Internet Routing with Independent Providers David Wetherall University of Washington
Credits Ratul Mahajan, Microsoft Research Tom Anderson, University of Washington Neil Spring, University of Maryland djw / nicta / dec 2005 2
Independent providers affect routing Telstra Sprint ATT iinet Direct paths in the Internet aren t always short paths! djw / nicta / dec 2005 3
Competing interests are fundamental A basic feature of the Internet: Providers carry traffic for their competitors Flows compete for available bandwidth Pervasive in other computing systems: Interference in wireless networks Freeloading in peer-to-peer systems Email, denial-of-service, djw / nicta / dec 2005 4
A long-standing issue The most important change in the Internet architecture over the next few years will probably be the development of a new generation of tools for management of resources in the context of multiple administrations. -- David Clark, 1988 (DARPA design paper) djw / nicta / dec 2005 5
and still an open issue Our position is that accommodating this tussle [in which different stakeholders vie to favor their particular interests] is crucial to the evolution of the network s technical architecture. -- David Clark et. al., 2005 (Tussle paper in TON) djw / nicta / dec 2005 6
djw / nicta / dec 2005 7
This talk 1. How does the existence of independent providers affect routing? An empirical study of BGP 2. Can we design routing protocols that work well with competing interests? WISER, a new protocol based on barter djw / nicta / dec 2005 8
BGP Internet routing game Goal is to choose interconnection for traffic. Each provider: Selects the egress they prefer, e.g., nearest exit Agrees to accept packets from all ingresses Has little visibility into the workings of others djw / nicta / dec 2005 9
Consequences of the BGP game 1. Asymmetric paths 2. Inefficient paths the price of anarchy 3. Unstable paths djw / nicta / dec 2005 10
An example of inefficiency Seattle ATT San Francisco Sprint overload Paths are longer because ATT lacks insight into Sprint s network djw / nicta / dec 2005 11
Estimating overall routing inefficiency 1. Get a detailed map of the Internet Provider topology, interconnection, traffic 2. Compute the routes selected by BGP Across multiple providers 3. Compute the most efficient routes 4. Find the difference between them Unfortunately, maps are hard to come by because of independent providers! djw / nicta / dec 2005 12
Rocketfuel: an Internet mapping engine Probed Internet paths from monitors to many locations Merged found paths and annotated high-level maps Result is city-level maps of 55 diverse providers For our purposes: approximate latency from geography, traffic from population/gravity models Sprint, USA Telstra, Australia djw / nicta / dec 2005 13
Price of anarchy for BGP 100 10 1 cumulative % of of flows flows 1 2 3 4 5 6 latency path length reduction inflation relative to social optimal % 50 10 5 1 path length inflation 1.0 1.4 2.0 5.9 Most paths are good but some are very poor djw / nicta / dec 2005 14
Why inefficiency matters Poor paths (the tail of the curve) lead to operator intervention, which is undesirable. Costly, error-prone and difficult ( tweak and pray ) Ad hoc cooperation ( ) used in practice Likely to become worse as QOS takes off Even a small average inflation represents wastage Aim to carry traffic just as well with a smaller network and save money djw / nicta / dec 2005 15
djw / nicta / dec 2005 16
Overall goal for WISER, a new protocol optimum price of anarchy Route Efficiency social BGP Provider Freedom Want high efficiency and freedom for providers to make their own decisions djw / nicta / dec 2005 17
Making WISER practical 1. Limited information disclosure: commercial interests 2. Independent optimization criteria: business strategies Use agnostic costs 3. Win-win outcomes: no one should lose or losers won t play Barter: ISPs lose on some flows and gain on others djw / nicta / dec 2005 18
Approaches that don t work Unilateral optimization (automated tweak and pray ) Computing the social optimal Traditional mechanism design Using real money as the basis for optimization djw / nicta / dec 2005 19
Example of barter with agnostic costs 8 1 7 hops 7 3 1 ms 12 11 3 ms 2 2 hops 11 2 1 11 ms 6 1 hop 4 3 Both providers win when each trades small losses on some traffic for bigger gains on other traffic djw / nicta / dec 2005 20
WISER Internet routing game 1 7 D [1] D S 2 D [3] 1 D [11] 3 11 Goal is to select the interconnection for traffic 1. Providers determine the agnostic costs of internal paths 2. Downstream (receiver) announces its cost over each link 3. Upstream (sender) selects based on local and remote costs djw / nicta / dec 2005 21
Incentinghonest cost disclosure honest disclosure inflated disclosure normalized costs 2 7 2 1 2 7 5 4.3 30 2 0.7 10 3 1 11 110 7.3 Normalize costs so both providers have an equal say also helps with inherently unequal cost ranges djw / nicta / dec 2005 22
Incentinghonest use of remote costs 2 7 2 7 0.7 2 2 1 7.3 Upstream makes a virtual payment to the downstream a rough balance of payment per byte should emerge when both providers are sensitive to received costs Providers contractually stipulate this balance djw / nicta / dec 2005 23
Extending WISER to many providers c3 = c1 l + path cost D [c3] c1 l D [c1] D [c4] S D D [c2] c2 l D [c5] Each pair of providers run WISER using exchange rates to convert costs between providers djw / nicta / dec 2005 24
djw / nicta / dec 2005 25
Evaluating WISER Efficiency Is the price of anarchy consistently low? estimate using the same maps Robustness to dishonesty Can a dishonest provider harm an honest one? study with different provider strategies Implementation Is the overhead low? Is it easy to deploy? prototype to flesh out details (XORP, SSFnet) djw / nicta / dec 2005 26
Price of anarchy for WISER 100 10 1 BGP anarchy WISER Wiser 1 2 3 4 5 6 latency path length reduction inflation % 50 10 5 1 length inflation BGP 1.0 1.4 2.0 5.9 WISER 1.0 1.1 1.2 1.5 cumulative % of of flows flow relative to social optimum WISER is close to the social optimum (y-axis) win-win strategy is efficient for real networks barter between each pair of providers works well djw / nicta / dec 2005 27
WISER needs less provisioning 100 80 60 40 20 % of ISPs 0 Wiser anarchy WISER BGP 0 5 10 15 20 overprovisioning (%) cumulative % of proviers overprovisioning (%) Overprovisioningmeasures how much extra capacity providers need to handle failures WISER allows smaller networks to carry a given traffic load just as well djw / nicta / dec 2005 28
Social optimum is not win-win 100 80 60 40 20 % of ISPs 0-100-50 0 50 100 ISP gain (%) Wiser optima average distance reduction (%) relative to BGP WISER Social cumulative % of providers Under WISER, some providers don t win as much, but they don t significantly either djw / nicta / dec 2005 29
Barter on small sets is not sufficient 100 10 1 anarchy flow-pair bar Wiser BGP BGP on flow pairs WISER cumulative % of flows % of flows 1.0 1.5 2.0 path length inflation (relative to optimum) multiplicative inflation Bartering over many routes (WISER) helps providers trade small loses for large gains djw / nicta / dec 2005 30
Ordinal preferences aren t efficient 100 10 anarchy ordinal preferen Wiser BGP WISER with BGP MEDs WISER cumulative % of flows % of flows 1 1.0 1.5 2.0 multiplicative path length inflation (relative to optimal) BGP MEDs(ordinal) aren t sufficient for efficient routing; agnostic costs in WISER (cardinal) are djw / nicta / dec 2005 31
djw / nicta / dec 2005 32
Robustness to dishonesty Honest behavior is not provably the best shortterm option unlike classic mechanism design But we expect cheating to be uncommon in the long-term Harm to reputations, contract termination Yet WISER should minimize the gains of dishonesty Study the impact of a dishonest provider djw / nicta / dec 2005 33
Effects of dishonesty = all honest providers (WISER) = one dishonest provider (WISER) = one dishonest (no constraints) 100 80 60 40 20 % of % of honest providers ISPs 0-100 -50 0 50 100 provider ISP gain gain (%) (%) 100 80 60 40 20 0-100-50 0 50 100 provider ISP gain gain (%) % of % of dishonest providers ISP Honest provider in different scenarios. Its loss is small. Dishonest provider in different scenarios. Its gain is small. djw / nicta / dec 2005 34
Implementation as a BGP extension c3 = c1 l + path cost D [c3] c1 l D [c1] D [c4] S D D [c2] c2 l D [c5] Costs are carried with BGP messages and used in the decision process. Border routers communicate to track exchange rates and log virtual payments. djw / nicta / dec 2005 35
Low deployment barrier Implementation mirrors the current routing protocol; little disruption Nature of existing provider peering contracts is preserved Bilateral contracts with simple charging model Incrementally deployable start with a pair of providers who benefit djw / nicta / dec 2005 36
Low overhead 100 80 60 40 20 WISER BGP Wiser BGP 0 0 10 20 30 40 50 messages per second cumulative % of % failures of experiments XORP prototype added <6% to BGP code and 15%-25% CPU The sample graph above shows that WISER sends a similar number of routing messages as BGP djw / nicta / dec 2005 37
djw / nicta / dec 2005 38
Conclusions optimum WISER Route Efficiency social BGP Provider Freedom WISER delivers high efficiency for independent providers in a practical protocol. It is an existence proof! Our future work is to move along the road to deployment We hope to do the same for other networking problems djw / nicta / dec 2005 39
For more information 1. www.cs.washington.edu/homes/ratul 2. Negotiation-based Routing Between Neighboring Domains, Mahajan, Wetherall and Anderson, NSDI 2004 3. Quantifying the Causes of Path Inflation, Spring, Mahajan, and Anderson, SIGCOMM 2003. 4. Measuring ISP Topologies with Rocketfuel, Spring, Mahajan, Wetheralland Anderson, TON 2004 [Bennett Prize] Questions? djw / nicta / dec 2005 40