Finding a Needle in a Haystack: Pinpointing Significant BGP Routing Changes in an IP Network Jian Wu (University of Michigan) Z. Morley Mao (University of Michigan) Jennifer Rexford (Princeton University) Jia Wang (AT&T Labs Research) 1
Motivation AS2 AS4 destination AS3 Failure Disruption Congestion A B AS1 A backbone network is vulnerable to routing D changes that occur in other domains. source C Mitigation 2
Goal Identify important routing anomalies Lost reachability Persistent flapping Large traffic shifts Contributions: Build a tool to identify a small number of important routing disruptions from a large volume of raw BGP updates in real time. Use the tool to characterize routing disruptions in an operational network 3
Interdomain Routing: Border Gateway Protocol AS 1 BR 12.34.158.0/24 I can reach 12.34.158.0/24 ebgp data traffic AS 2 ibgp I can reach 12.34.158.0/24 via AS 1 ebgp data traffic AS 3 12.34.158.5 Prefix-based: one route per prefix Path-vector: list of ASes in the path Incremental: every update indicates a change Policy-based: local ranking of routes 4
Capturing Routing Changes A large operational network (8/16/2004 10/10-2004) ebgp Updates ibgp Best routes CPE BGP Monitor Best routes ibgp Updates ebgp ibgp ebgp 5
Challenges Large volume of BGP updates Millions daily, very bursty Too much for an operator to manage Different from root-cause analysis Identify changes and their effects Focus on actionable events rather than diagnosis Diagnose causes in/near the AS 6
System Architecture EBR EBR BGP Updates (106 ) BGP BGP Update Update Grouping Grouping Events (10 5 ) Event Event Classification Classification Typed Events Event Event Correlation Correlation Clusters (10 3 ) Traffic Traffic Impact Impact Prediction Prediction Large Disruptions (10 1 ) EBR Persistent Flapping Prefixes (10 1 ) Frequent Flapping Prefixes (10 1 ) EBR EBR Netflow Data EBR From millions of updates to a few dozen reports 7
Grouping BGP Update into Events Challenge: A single routing change leads to multiple update messages affects routing decisions at multiple routers EBR EBR EBR BGP Updates BGP BGP Update Update Grouping Grouping Persistent Flapping Prefixes Events Approach: Group together all updates for a prefix with inter-arrival < 70 seconds Flag prefixes with changes lasting > 10 minutes. 8
Grouping Thresholds Based on our understanding of BGP and data analysis Event timeout: 70 seconds 2 * MRAI timer + 10 seconds 98% inter-arrival time < 70 seconds Convergence timeout: 10 minutes BGP usually converges within a few minutes 99.9% events < 10 minutes 9
Persistent Flapping Prefixes A surprising finding: 15.2% of updates were caused by persistent-flapping prefixes even though flap damping is enabled. Types of persistent flapping Conservative damping parameters (78.6%) Protocol oscillations due to MED (18.3%) Unstable interfaces or BGP sessions (3.0%) 10
Example: Unstable ebgp Session ISP EA ED Peer EB EC p Customer Flap damping parameters is session-based Damping not implemented for ibgp sessions 11
Event Classification Challenge: Major concerns in network management Changes in reachability Heavy load of routing messages on the routers Change of flow of the traffic through the network Events Event Event Classification Classification Typed Events, e.g., Loss/Gain of Reachability Solution: classify events by severity of their impact 12
Event Category No Disruption p AS 1 AS 2 ED EE No Traffic Shift EA EB No Disruption : ISP no border routers have any traffic shift. (50.3%) EC 13
Event Category Internal Disruption p AS 1 AS 2 ED EE EA EB Internal Disruption : ISP all traffic shifts are internal. (15.6%) EC Internal Traffic Shift 14
Event Category Single External Disruption p AS 1 AS 2 ED EE external Traffic Shift EA EB Single External Disruption : ISP only one of the traffic shifts is external (20.7%) EC 15
Statistics on Event Classification No Disruption Internal Disruption Single External Disruption Multiple External Disruption Loss/Gain of Reachability Events 50.3% 15.6% 20.7% 7.4% 6.0% Updates 48.6% 3.4% 7.9% 18.2% 21.9% First 3 categories have significant day-to-day variations Updates per event depends on the type of events and the number of affected routers 16
Event Correlation Challenge: A single routing change affects multiple destination prefixes Typed Events Event Event Correlation Correlation Clusters Solution: group the same-type, close-occurring events 17
EBGP Session Reset Caused most of single external disruption events Check if the number of prefixes using that session as the best route changes dramatically Number of prefixes session recovery session failure time Validation with Syslog router report (95%) 18
Hot-Potato Changes Hot-Potato Changes P EA 11 9 ISP EB 10 Hot-potato routing = route to closest egress point EC Caused internal disruption events Validation with OSPF measurement (95%) [Teixeira et al SIGMETRICS 04] 19
Traffic Impact Prediction Challenge: Routing changes have different impacts on the network which depends on the popularity of the destinations Clusters Traffic Traffic Impact Impact Prediction Prediction Large Disruptions BR E Netflow Data BR E BR E Solution: weigh each cluster by traffic volume 20
Traffic Impact Prediction Traffic weight Per-prefix measurement from netflow 10% prefixes accounts for 90% of traffic Traffic weight of a cluster the sum of traffic weight of the prefixes A small number of large clusters have large traffic weight Mostly session resets and hot-potato changes 21
Performance Evaluation Memory Static memory: current routes, 600 MB Dynamic memory: clusters, 300 MB Speed 99% of intervals of 1 second of updates can be process within 1 second Occasional execution lag Every interval of 70 seconds of updates can be processed within 70 seconds Measurements were based on 900MHz CPU 22
Conclusion BGP troubleshooting system Fast, online fashion Operators concerns (reachability, flapping, traffic) Significant information reduction millions of update a few dozens of large disruptions Uncovered important network behavior Hot-Potato changes Session resets Persistent-flapping prefixes 23