Network Measurement (II) Network Tomography

Transcription

1 Network Measurement (II) Network Tomography Tomography Tomography: imaging by sections or sectioning The word is derived from the Greek tomos (slice) and graphein (to write). Digital geometry processing is used to generate a 3-d image of the inside of an object from a large series of 2-d X-ray images taken around a single axis of rotation. Computed tomography (CT) A medical imaging method employing tomography Part

2 Network Tomography Network tomography: Inferring network characteristics from indirect measurements Coined by Y. Vardi in 1996; a.k.a. network inference Q: Why not direct measurement? A: Infeasible or too expensive to measure directly Two classes of tomography problems Traffic matrix estimation: infer e2e traffic demands from link load measurements Performance tomography: infer link performance from e2e performance measurements Solution technique Construct a system of linear equations Ax = b Solve the linear inverse problem Challenge: more unknowns than equations! Part Outline Tomo-gravity Infer e2e traffic demands from link loads Gravity model + tomography Passive loss inference Infer link loss from passive e2e measurement Statistical methods Part

3 References Fast, accurate computation of large-scale IP traffic matrices from link measurements, Y.Zhang, M.Roughan, N.Duffield and A.Greenberg, SIGMETRICS An information-theoretic approach to traffic matrix estimation, Y.Zhang, M.Roughan, C.Lund and D.Donoho, SIGCOMM Server-based inference of Internet performance. V.Padmanabhan, L.Qiu, and H.Wang. INFOCOM Part Tomo-gravity 3

4 Internet Service Provider Networks A large ISP network has 100s of nodes, 1000s of links, 10000s routes, and over 1 petabyte(10 15 bytes) per day C B A Part Network Engineering Under a failure, routes change Want to predict new link loads C A B Reliability analysis Predict link loads under unexpected/planned router/link failures Traffic engineering Optimize routes to minimize congestion Capacity planning Forecast future capacity requirements Part

5 Network Engineering: State of Art in 2003 Guess, tweak, and pray Guess based on experience & intuition Manually tweak things, and hope the best Disadvantages Manual process Time consuming, error prone Not very reliable Intuition may be wrong Unexpected side effects (often at the worst moment) Suboptimal performance Wastes resource, wastes time, angry customers Only works for current traffic pattern Need to repeat the exercise when traffic pattern changes Need accurate, automated, scientific tools! Part More Scientific Network Engineering? Feldmann et al Shaikh et al Tomo-gravity Fortz et al A: "Well, we don't know the topology, we don't know the traffic matrix, the routers don't automatically adapt the routes to the traffic, and we don't know how to optimize the routing configuration. But, other than that, we're all set!" [Rexford2000, Kurose2003] Part

6 Central Problem: No Traffic Matrix Traffic matrix (TM) Gives traffic volumes between origin and destination Very difficult to directly measure TMs Direct measurement [Feldmann et al. 2000] Collect flow-level data around the whole edge of the network Combine with routing data Semi-standard router feature: Netflow Cisco, Juniper, etc. Not always well supported Potential performance impact on routers Huge amount of data (500GB/day) Widely available SNMP data gives only link loads Even this data is not perfect (glitches, loss, ) Part Tomo-gravity Solution Tomo-gravity infers traffic matrices Input: SNMP link loads, topology, routing policies Advantages Today s data no special instrumentation Fast: a few seconds for AT&T Accurate: average ~11% error Scalable: hundreds of nodes Sound: has solid foundation in information theory Robust: copes easily with data loss & glitches Flexible: can incorporate more detailed measurements In daily use for AT&T IP network engineering Reliability analysis, traffic engineering, and capacity planning Part

7 Problem 1 link 1 3 route 2 link 3 b 1 x2 x3 route 3 router route 1 b b b Only measure at links link x 1 x 0 x Problem: Estimate traffic matrix (x s) from the link measurements (b s) Part Problem 1 link 1 route 2 3 link 3 b 1 x2 x3 route 3 router route 1 Only measure at links link 2 2 b = Ax Typically massively under-constrained! Part

8 Approaches Existing solutions Direct solution (Singular Value Decomposition) Gravity modeling Tomographic approach New solutions Generalized gravity modeling Tomo-gravity Part How to Validate? Simulate and compare Problems How to generate realistic traffic matrices? Danger of generating exactly what you put in Measure and compare Problems: Hard to get Netflow (detailed direct measurements) along whole edge of network If we had this, then we wouldn t need SNMP approach Actually pretty hard to match up data Is the problem in your data: SNMP, Netflow, routing, Our method Use partial, incomplete Netflow data (covering 70% edge) Part

9 Direct SVD Solution The problem is massively under-constrained Part Simple Gravity Model Motivated by Newton s Law of Gravitation Assume traffic between sites is proportional to traffic at each site x 1 b 2 b 3 x 2 b 1 b 3 x 3 b 1 b 2 Assume there is no systematic difference between traffic in different locations Only the total volume matters Could include a distance term, but locality of information is not so important in the Internet as in other networks Part

10 Simple Gravity Solution Better than direct SVD solution, but still not very accurate Part Generalized Gravity Model Internet routing is asymmetric Hot potato routing: use the closest exit point Generalized gravity model For outbound traffic, assumes proportionality on per-peer basis (as opposed to per-router) peering links access links Part

11 Generalized Gravity Solution Fairly accurate given that no link constraint is used Part Tomographic Approach Apply the link constraints 1 route 3 route 2 router 2 3 route 1 b = Ax Part

12 Tomographic Approach Under-constrained linear inverse problem Find additional constraints based on models Typical approach: use higher order statistics (e.g. EM, Bayesian) Disadvantages Complex algorithm doesn t scale Large IP networks have 100s of nodes, 10000s of routes Relying on higher order statistics is not robust given the problems in SNMP data Artifacts, missing data Violations of model assumptions (e.g. non-stationarity) Relatively low sampling frequency: 1 sample every 5 min Unevenly spaced sample points Not very accurate at least on simulated TM [Medina et al. 2002] Part Our Solution: Tomo-gravity Tomo-gravity = tomo-graphy + gravity modeling Reduce problem size Exploit topological equivalence Find a solution, which satisfies the constraints, and is closest to the generalized gravity model solution tomo-gravity solution (x) generalized gravity solution (g) constraint subspace (b=ax) (from link measurements) Part

13 Closest? Least-squares 2 minimize x g p Weighted least-squares minimize x[p] g[p] p Square-root weights x[p] g[p] 2 w[p] w[p] 2 2 g[p] work best tomo-gravity solution (x) generalized gravity solution (g) constraint subspace (b=ax) (from link measurements) Part Foundation in Information Theory Mutual Information I(S,D) Information gained about source (S) from destination (D) I(S,D) = -(relative entropy with respect to independent S and D) Also given by Kullback-Leibler divergence from independence Minimum mutual information Minimize I(S,D) (subject to link load constraints b=ax) Why this criterion? In the absence of information, let s assume no information Minimal assumption about the traffic Large aggregates tend to behave like overall network? Framework for tomo-gravity Gravity model = independence (between S and D) Generalized gravity model = conditional independence Weighted least-squares = first-order approx. to K-L divergence x[p] x[p] K(x g) x[p] log x[p] 1 p p g[p] g[p] p x[p] x[p] 1 g[p] p x[p] g[p] p x[p] g[p] g[p] 2 Part

14 Tomo-gravity Solution Accurate: ±20% bounds for large flows; average ~11% error Fast: less than 5 seconds for entire AT&T backbone Part Distribution of Element Sizes Estimated and actual distribution overlap Part

15 Estimates over Time Consistent performance over time Part Robustness measured link load = real load + real load N(0, ) Reasonably accurate even when noise 0.1 Part

16 Additional Information: Netflow Detailed direct measurements at a few high-traffic locations can significantly improve accuracy Part Additional Information: Local Matrices for reference previous case Local matrices also significantly improve accuracy Part

17 Network Engineering: A New Paradigm Measure, model, and control As opposed to guess, tweak, and pray what if? user input TMs routing optimizer answers, suggested changes tomo-gravity topology, routing SNMP measurement topology routing network simulator Part Operational Experience Network reliability analysis Consider link loads under planned/unexpected failures Answers what if type questions about link/span/ router failures Allows comprehensive analysis of network risks What is the link most under threat of overload under likely failure scenarios Used daily by AT&T for Planned Cable Intrusions Part

18 Operational Experience Traffic engineering Used with OSPF optimization [IMC 2003] Get within 6% of OSPF optimum using true TM Get within 12% of absolute best (e.g. using MPLS) Has been applied on a more limited basis, in connection with reliability analysis Successfully prevented service disruption during simultaneous failures of multiple large links Capacity planning Estimated traffic matrices have been used in backbone capacity planning since Oct Allows quantitative planning As opposed to being overly conservative and using unnecessarily large over-provisioning factors Part Summary: Tomo-gravity Works Takes the best of tomography and gravity modeling Has solid foundation in information theory Simple, and quick: A few seconds Accurate: average ~11% error Satisfies link constraints (gravity model solutions don t) Scalable: hundreds of nodes Uses widely available SNMP data Highly robust Can work within the limitations of SNMP data Only uses first order statistics Interpolation very effective Limited scope for improvement Can easily incorporate additional constraints Follow-up work: combine Netflow with Tomo-gravity In daily use for AT&T IP network engineering Operational experience very positive Part

19 Passive Loss Inference Motivation Ethernet Web Server Diagnosis engine Why is it so slow? C&W AT&T UUNet Sprint AOL Qwest Earthlink It s so slow! Part

20 Network Diagnosis Diagnosis engine Network topology Netmon/tcpdump traces Trouble spots location Diagnosis results: Qwest access link: > Peering between UUNET and AOL: > Part Network Diagnosis (Cont.) Goal: Determine internal network characteristics using passive end-to-end measurements Primary focus: identifying lossy links Applications Trouble shooting Server selection Server placement Overlay network path construction Part

21 Previous Work Active probing to infer link loss rate multicast probes striped unicast probes Pros & cons accurate since individual loss events identified expensive because of extra probe traffic S S A B A B Part Problem Formulation server l 2 l 1 l 3 (1-l 1 )*(1-l 2 )*(1-l 4 ) = (1-p 1 ) (1-l 1 )*(1-l 2 )*(1-l 5 ) = (1-p 2 ) (1-l 1 )*(1-l 3 )*(1-l 8 ) = (1-p 5 ) clients l 4 l 5 l 6 l 7 l 8 p 1 p 2 p 3 p 4 p 5 Challenges: Under-constrained system of equations Measurement errors Part

22 3 methods 1. Random sampling 2. Linear optimization 3. Bayesian Inference using Gibbs sampling (We ll focus on the latter one) Part Random Sampling Each sample: Randomly assign loss rate to links with constrains Iterate R times and average server l 1 l 2 l 3 l 4 l 5 l 6 l 7 l 8 clients p 1 p 2 p 3 p 4 p 5 Part

23 Linear Programming Linear Equations LP Constraints i T and P j L j i P, L j log(1/(1 p )) i log(1/(1 l )) i & i Optimization Goal Part Gibbs Sampling D observed packet transmission and loss at the clients ensemble of loss rates of links in the network Goal determine the posterior distribution P( D) Approach Use Markov Chain Monte Carlo with Gibbs sampling to obtain samples from P( D) Draw conclusions based on the samples Part

24 Gibbs Sampling (Cont.) Applying Gibbs sampling to network tomography 1) Initialize link loss rates arbitrarily 2) For j = 1 : warmup for each link i _ compute P(l i D, {l i }) _ where l i is loss rate of link i, and {l i } = U ki l k 3) For j = 1 : realsamples for each link i _ compute P(l i D, {l i }) Use all the samples obtained at step 3 to approximate P( D) Part Gibbs Sampling Details P( l i where D,{ l }) P( D l L p P( l i ) j i 1 ) 1 l i jclients P( D l P( D l it (1 p j j (1 l i L L ) ) dl ) s ) j PDF for posterior distribution p i f j j Uniform prior distribution Likelihood function (assumes independence) Relationship btw link loss and path loss (assumes independence) Part

25 Performance Evaluation Simulation experiments Trace-driven validation Part Simulation Experiments Advantage: no uncertainty about link loss rate! Methodology Topologies used: randomly-generated: nodes, max degree = 5-50 real topology obtained by tracing paths to microsoft.com clients randomly-generated packet loss events at each link A fraction f of the links are good, and the rest are bad LM1: good links: 0 1%, bad links: 5 10% LM2: good links: 0 1%, bad links: 1 100% Link loss processes: Bernoulli and Gilbert Goodness metrics: Coverage: # correctly inferred lossy links False positive: # incorrectly inferred lossy links Part

26 Comparative Results Random sampling generates many false positives LP has a low cover rate (30-60%) Gibbs performs very well (80% with a 5% false positive rate) Part Random topologies Gibbs sampling for a 1000-node random topology (d = 10, f = 0.5) # links "# correctly identified lossy links" "# true lossy links" "# false positive" Confidence estimate for gibbs sampling works well and can be used to rank order the inferred lossy links. Part

27 Trace-driven Validation Validation approach Divide client traces into two: tomography and validation Tomography data set loss inference Validation set check if clients downstream of the inferred lossy links experience high loss Experimental setup Real topologies and loss traces collected from traceroute and tcpdump at microsoft.com during 12/20/2000 and 01/11/2002 Results For the small subset of inferences that could be validated, all the inferences are correct Likely candidates for lossy links: links crossing an inter-as boundary links having a large delay (e.g. transcontinental links) links that terminate at clients Part Summary Passive network tomography is feasible Gibbs sampling has high coverage (over 80%), and low false positive rate (below 5-10%) Future work: make loss inference in real time Part

28 Discussion Q: why does Gibbs sampling outperform LP? Sampling errors cause measured loss rates to differ from true loss rates Linear relationship no longer holds exactly e.g. 5 probes give limited precision in loss estimation Temporal loss variation cannot assign a single number to loss rate of a link Gibbs sampling gains robustness by working with distributions Q: is loss independence a strong assumption? Independence across packets: not a big problem paper tried Gilbert loss and get similar results Independence across links: previous work suggests that it s reasonable for the Internet Part Concluding remarks Network tomography infers quantities of interest based on indirect measurements Often pose the problem as an under-constrained linear inverse problem Research directions Dealing with under-constrainedness by leveraging structure of the problem E.g., traffic matrix has a low-rank approximation Extending tomography to non-linear inference (e.g. in wireless environment) Design of experiments: make fewest measurements that yield maximum information Part