BGP Path Exploration Damping (PED) Mattia Rossi mrossi@swin.edu.au Centre for Advanced Internet Architectures (CAIA) Swinburne University of Technology Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 2
Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 3 About this talk Selected for the Applied Networking Research Prize (ANRP) based on a peer-reviewed paper: Paper published in the IEEE Journal on Selected Areas in Communications (JSAC), October 2010 1 A Technique for Reducing BGP Update Announcements through Path Exploration Damping. Geoff Huston, Mattia Rossi, Grenville Armitage Project sponsored by the Cisco University Research Program Fund 1 http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5586440 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 4
hat is Path Exploration Damping? Path Exploration Damping PED Algorithm intended to replace MRAI and RFD Methods to reduce update churn and convergence time in BGP BGP is the de-facto standard for inter-domain routing BGP Border Gateway Protocol MRAI Minimum Route Advertisment Interval RFD Route Flap Damping IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 5 PED implementations CAIA Implemented in Quagga Patch available for Quagga 0.99.13 Download at http://caia.swin.edu.au/urp/bgp/tools.html Cisco (by Mohammed Mirza) Implemented in CISCO IOS-XE Experimental Version 15.1 Running on Cisco ASR1002 (2RU) Currently tested at APNIC Pty. Ltd., Australia IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 6
Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 7 Simplified BGP speaker design Router 1 Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process Router 3 Router 4 RIB: Routing Information Base Routing Table Adj-RIB-(in,out): Adjacency RIB (in,out) IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 8
Basic BGP dynamics IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 9 Basic BGP dynamics Path: 1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 9
Basic BGP dynamics Path: 2,1 Path: 1 Path: 2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 9 Basic BGP dynamics Path: *2,1 3,2,1 Path: 1 Path: 2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 9
Basic BGP dynamics Path: *2,1 3,2,1 4,3,2,1 Path: 1 Path: 2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 9 Path Exploration Path: *2,1 3,2,1 4,3,2,1 Path: 2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 10
Path Exploration Path: *3,2,1 4,3,2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 10 Path Exploration Path: *4,3,2,1 A A: 5,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 10
Path Exploration A A: 5,3,2,1 A: 5,4,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 10 Path Exploration A: 5,3,2,1 A: 5,4,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 10
hat is Path Exploration? An update sequence lengthening the AS-path gradually until stability is reached IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 11 MRAI and RFD Need to decrease BGP chattiness and Path Exploration Minimum Route Advertisment Interval MRAI (RFC 1771) Apply 30s (default) delay on announcements Router 1 Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process MRAI Timer MRAI Timer Router 3 Router 4 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 12
Minimum Route Advertisment Interval MRAI IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 13 Minimum Route Advertisment Interval MRAI aiting 30s Path: 1 30 sec. IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 13
Minimum Route Advertisment Interval MRAI Path: 2,1 30 sec. aiting 60s Path: 1 Path: 2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 13 Minimum Route Advertisment Interval MRAI Path: *2,1 3,2,1 Delay 0-90s Path: 1 30 sec. Path: 2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 13
Minimum Route Advertisment Interval MRAI Path: *2,1 3,2,1 4,3,2,1 Delay 0-90s Path: 1 30 sec. Path: 2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 13 Minimum Route Advertisment Interval MRAI Path: *2,1 3,2,1 4,3,2,1 Path: 2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14
Minimum Route Advertisment Interval MRAI Path: *3,2,1 4,3,2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14 Minimum Route Advertisment Interval MRAI Path: *4,3,2,1 Update delayed IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14
Minimum Route Advertisment Interval MRAI Update delayed IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14 Minimum Route Advertisment Interval MRAI IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14
Minimum Route Advertisment Interval MRAI Path: *4,3,2,1 Update delayed IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14 Minimum Route Advertisment Interval MRAI A A: 5,4,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14
Minimum Route Advertisment Interval MRAI A: 5,4,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 14 MRAI and RFD Need to decrease BGP chattiness and Path Exploration Minimum Route Advertisment Interval MRAI (RFC 1771) Apply 30s (default) delay on announcements MRAI on withdrawals (RATE) allowed per RFC 4271 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 15
MRAI on withdrawals RATE Path: *2,1 3,1 4,3,1 Path: *1 Path: *3,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 16 MRAI on withdrawals RATE Path: *2,1 3,1 4,3,1 30 sec. Path: *1 Path: *3,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 16
MRAI on withdrawals RATE Path: *2,1 3,1 4,3,1 Data loss! Black hole!! Path: *1 Path: *3,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 16 MRAI and RFD Need to decrease BGP chattiness and Path Exploration Minimum Route Advertisment Interval MRAI (RFC 1771) Apply 30s (default) delay on announcements MRAI on withdrawals (RATE) allowed per RFC 4271 Route Flap Damping RFD (RFC 2439) Flapping = sequence of announcements and withdrawals Suppress flapping prefixes for 1 hour (or more) IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 17
Route Flap Damping RFD Router 1 Flapping prefix! Router 2 Adj-RIB-in RIB Adj-RIB-out RFD RFD Decision process Router 3 Router 4 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 18 Route Flap Damping RFD Path: *3,2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 19
Route Flap Damping RFD A A: 5,3,2,1 Path: 3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 19 Route Flap Damping RFD Path: *4,3,2,1 A: 5,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 19
Route Flap Damping RFD A A: 5,3,2,1 A: 5,4,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 19 Route Flap Damping RFD A: 5,3,2,1 A: 5,4,3,2,1 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 19
Route Flap Damping RFD Path: *2,1 3,2,1 4,3,2,1 RFD suppresses 1.0.0.0/8 for 1 hour Path: 1 Path: 2,1 Path: 3,2,1 here is my data? IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 19 hat are the problems? MRAI exhibits unpredictable behavior RATE creates black holes RFD penalizes prefix owners even if the misbehavior happened further upstream e want something better! IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 20
Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 21 PED algorithm Delay BGP announcements if the announced AS path is longer than the previously known AS path IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 22
PED algorithm illustrated Router 1 Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process PEDI Timer PEDI Timer Router 3 Router 4 PEDI: Path Exploration Damping Interval IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 23 PED algorithm illustrated Router 1 Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process per-peer setting per-prefix application per-peer setting per-prefix application Router 3 Router 4 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 23
PED algorithm illustrated Router 1 A: 1.0.0.0/8 first announcement / previously withdrawn Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process announce new best path no delay Router 3 Router 4 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 23 PED algorithm illustrated Router 1 update Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process longer path! queue update, start PEDI timer Router 3 Router 4 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 23
PED algorithm illustrated Router 1 update Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process shorter path! send update, delete PEDI timer Router 3 Router 4 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 23 PED algorithm illustrated Router 1 update Router 2 Adj-RIB-in RIB Adj-RIB-out Decision process withdrawal! send update, delete PEDI timer Router 3 Router 4 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 23
Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 24 PED Data analysis Experiments using 24 hours of real BGP updates Two datasets: 1. APNIC Pty. Ltd., Australia (2 peers) 2. University of Oregon Routeviews, US (5 peers) AS 293 AS 11537 AS 3727 AS 4777 AS 4608 AS 2914 AS 48285 AS 6447 Routeviews APNIC AS 131702 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 25
PED Data analysis Replayed using the Quagga-Accelerator PED Collector BGP cloud Router 1 Router 2 Quagga Accelerator IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 26 Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 27
5 s ec PEDI 0 s ec MRAI 75 s ec PEDI 70 s ec PEDI 65 s ec PEDI 60 s ec PEDI 55 s ec PEDI 50 s ec PEDI 45 s ec PEDI 40 s ec PEDI 35 s ec PEDI 30 s ec PEDI 25 s ec PEDI 20 s ec PEDI 15 s ec PEDI 10 s ec PEDI 30 s ec MRAI Reduction of update load APNIC num ber of m es s ages 160k 140k 120k 100k 80k 60k 40k 20k 20% 29% BGP update m es s ages Pr efix announcem ents Pr efix withdr awals 0 70 s ec PEDI 65 s ec PEDI 60 s ec PEDI 55 s ec PEDI 50 s ec PEDI 45 s ec PEDI 40 s ec PEDI 35 s ec PEDI 30 s ec PEDI 0 s ec MRAI 30 s ec MRAI IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 28 Reduction of update load Routeviews num ber of m es s ages 150k 100k 50k 18% BGP update m es s ages Pr efix announcem ents Pr efix withdr awals 32% 0 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 29
Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 30 Convergence time approximation Control Plane convergence (Optimality) Data plane convergence Forwarding path (Reachability) Convergence measured from a single point of view (Optimality) Router 2 Convergence defined as a route being stable for 5 minutes 240s 30s 30s 30s 120s 5min Time to converge IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 31
Optimality approximation APNIC data 800 700 600 Upper and lower quartile Mean Median 500 s econds 400 200 150 100 50 0 70s PEDI 65s PEDI 60s PEDI 55s PEDI 50s PEDI 45s PEDI 40s PEDI 35s PEDI 30s PEDI 0s MRAI 30s MRAI IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 32 Optimality approximation Routeviews data seconds 800 700 600 500 400 250 200 Upper and lower quartile Mean Median 150 100 50 0 70 sec PDI 65 sec PDI 60 sec PDI 55 sec PDI 50 sec PDI 45 sec PDI 40 sec PDI 35 sec PDI 30 sec PDI 25 sec PDI 20 sec PDI 15 sec PDI 10 sec PDI 5 sec PDI 0 sec MRAI 30 sec MRAI IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 33
Detailed convergence time analysis Investigate Reachability vs. Optimality Analysis of a whole BGP system Impact on convergence of 4 events causing instability: Link failure along the path alternative path exists Link recovery Prefix withdrawal Prefix announcement IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 34 Detailed convergence time analysis Experimental analysis over 20 testruns Simple example topology ASes are single BGP speakers Example: 30s MRAI on all ASes and 35s PEDI on all ASes AS 6 0 1 2 3 Prefix Origin: 1.0.0.0/8 0 1 2 IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 35
Optimality Announcement of initial route Stable System: * 1 * 2,1 *3,2,1 *4,3,2,1 6,13,12,11,10,1 *13,12,11,10,1 5,4,3,2,1 AS 6 Data to * 1 *10,1 *11,10,1 22,21,20,10,1 0 1 2 3 Prefix Origin: 1.0.0.0/8 0 1 2 *12,11,10,1 *10,1 *20,10,1 *21,20,10,1 12,11,10,1 Announcement of initial route at AS 6 : PED: 0 seconds MRAI: 60-120 seconds IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 36 Reachability Link failure Link failure between 0 and 1 : * 1 * 2,1 *3,2,1 *4,3,2,1 6,13,12,11,10,1 *13,12,11,10,1 5,4,3,2,1 AS 6 Data to * 1 *12,22,21,20,10,1 *22,21,20,10,1 0 1 2 3 Prefix Origin: 1.0.0.0/8 0 1 2 *12,11,10,1 *10,1 *20,10,1 *21,20,10,1 Reachability achieved (1 ) PED: 0 seconds MRAI: 0-4 or 29-30 seconds IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 37
Optimality Link failure Link failure between 0 and 1 : * 1 * 2,1 *3,2,1 *4,3,2,1 *5,4,3,2,1 13,12,22,21,20,10,1 AS 6 Data to * 1 *12,22,21,20,10,1 *22,21,20,10,1 0 1 2 3 Prefix Origin: 1.0.0.0/8 0 1 2 *12,22,21,20,10,1 *10,1 *20,10,1 *21,20,10,1 Optimality achieved: PED: 66 seconds (+-jitter) MRAI: 2-58 seconds IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 38 Optimality Link recovery Link recovery between 0 and 1 : * 1 * 2,1 *3,2,1 *4,3,2,1 6,13,12,11,10,1 *13,12,11,10,1 5,4,3,2,1 AS 6 Data to * 1 *10,1 *11,10,1 22,21,20,10,1 0 1 2 3 Prefix Origin: 1.0.0.0/8 0 1 2 *12,11,10,1 *10,1 *20,10,1 *21,20,10,1 12,11,10,1 Optimality achieved: PED: 0 seconds MRAI: 31-33 and 55-60 seconds IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 39
Optimality/Reachability Prefix withdrawn Prefix withdrawal at AS 6 0 1 2 3 Prefix Origin: 1.0.0.0/8 0 1 2 Optimality achieved (route withdrawn on every AS): PED: 0 seconds MRAI: 0 seconds IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 40 Optimality/Reachability Prefix (re)-announced Prefix announcement at * 1 * 2,1 *3,2,1 *4,3,2,1 6,13,12,11,10,1 *13,12,11,10,1 5,4,3,2,1 AS 6 Data to * 1 *10,1 *11,10,1 22,21,20,10,1 0 1 2 3 Prefix Origin: 1.0.0.0/8 0 1 2 *12,11,10,1 *10,1 *20,10,1 *21,20,10,1 12,11,10,1 Optimality achieved (same as initial announcement): PED: 0 seconds MRAI: 32-34, 58-60, 76-90 seconds IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 41
Outline Introduction Motivation Path Exploration Damping - PED Experimental results Reduction of update load MRAI and PED convergence time compared Conclusions and future work IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 42 Conclusions In this talk: PED decreases update load PED converges to Reachability as fast or faster than MRAI PED converges to Optimality slower than MRAI in one case In the paper: PED interacts well with MRAI PED can be deployed incrementally A single PED speaker is beneficial to the BGP system 35s PEDI is a safe defaulte value in the MRAI dominated Internet In the future: Dynamic PEDI per prefix More heuristics IETF-81 http://www.caia.swin.edu.au mrossi@swin.edu.au 28 July, 2011 43