Pythia: Detection, Localization, and Diagnosis of Performance Problems

Transcription

1 Pythia: Detection, Localization, and Diagnosis of Performance Problems using perfsonar Partha Kanuparthy Constantine Dovrolis (PI) Georgia Institute of Technology

2 Intro Pythia is a data-analysis tool data from perfsonar Focus: performance problems Funded by DoE: 3 yrs (since Sept 11) This talk: detection, localization, WiP

3 Pythia: one tool, three objectives Detection: noticeable loss rate between ORNL and SLAC on 7/11/11 at 9::2 EDT Localization it happened at DENV-SLAC link Diagnosis it was due to insufficient router buffers

4 Detection First step: Is there a problem? Look for deviations from baseline but not monitor-related events! 6 delay 25 context-switch delay s rise! 4 15 delay (ms) 3 delay (ms) 1 2 baseline Congestion: NY-CLEV seq. 5 Monitor event: ALBU-ATL seq.

5 Detection Implementation A single pass through OWAMP timeseries Discard monitor-related events 7 6 delay Discard level-shifts (e.g., NTP synchronization; TODO: detecting routing changes) delay (ms) NTP level-shifts seq. The rest are network performance problems! 6 delay 35 delay 5 3 Congestion: NY-CLEV delay (ms) 4 3 delay (ms) seq seq.

6 Detection: In Practice Detection outputs congestion events > 1s long Monitor events Congestion events Congestion events / path / day start, end timestamps ESnet data: ESnet 2.2 million days, 33 monitors Internet2 data: Internet2 11, days, 9 monitors

7 How long are congestion events? ESnet, I2: 9% events were 1-2s long CDF.5 this is sufficient to affect app-performance delay increases by 1s of milliseconds ESnet Duration (in seconds) some events are common across paths CDF Internet D ura tio n (in se c o n d s)

8 Are lossy events common? Answer: No ESnet: no lossy congestion events Internet2: 6 of 2268 lossy congestion events <.1% loss rate as sampled by OWAMP CDF Internet Percentage of packets lost

9 Localization Follow-up to detection: Which link is bad? Link/path performance levels discrete: e.g., high delay, medium delay, low delay Localization: minimum number of bad links that can explain bad paths use greedy heuristic to solve iteratively

10 Localizing Bad Links Number of paths ESnet: 9 congestion events washcr1-aofacr2.es.net bad link localized for each starcr1-chicr1.es.net up to 75 paths affected by an event: Bad link bnlmr2-bnlowamp.es.net clevcr1-ip-bostcr1.es.net clevcr1-ip-bostcr1.es.net One Way Delay (in ms) bostcr1-ip-aofacr2.es.net chiccr1-ip-clevcr1.es.net Time (in seconds) clevcr1-ip-chiccr1.es.net

11 Localization: Internet2 Internet2: 266 congestion events in 22 days 3 bad links: 1 case 2 bad links: 6 cases 1 bad link: rest few bad links dominate 9% events: 25 2 Timeline of bad links: peaks around 7th March 211 ge rtr.kans ge rtr.chic xe rtr.hous ge rtr.kans (58% events) ge rtr.chic (25%) xe rtr.hous (6%) Link Events per day Day since 23rd Feb 211

12 Case Study Internet2: event with two bad links One Way Delay (in ms) path: CHIC to LOSA 28th Feb 211, :1:51 GMT Time (in seconds) 1 Localized bad links: ge rtr.kans ge rtr.losa One Way Delay (in ms) path: ATLA to KANS Time (in seconds) 1 Predicted bad link performance (avg): 26ms and 57ms O ne Way D elay (in ms) path: HOUS to LOSA Time (in seconds)

13 Diagnosis Work in progress What was the problem? Match signatures to identify known problems delays, losses, reordering, etc. Unknown signatures: store in database for future diagnosis operators can label them for reference

14 Pythia: Real-time System Work in progress Centralized process talks to perfsonar MAs to collect data traceroute MA OWAMP MA 1 OWAMP MA 2 Pythia server OWAMP MA 3 BWCTL MA... MA Preprocessing Detection Localization Diagnosis

15 Thank you

16 Pre-processing traceroute: compensate for * * * s Clock skew: use a 1s window to normalize delays Clock offset between monitors: use a 2s window to identify congestion events Identify simultaneous events across paths