Network Troubleshooting with Mirror VNets

Transcription

1 An-Institut der Technischen Universität Berlin Network Troubleshooting with Mirror VNets Andreas Wundsam M. Amir Mehmood Olaf Maennel Anja Feldmann FutureNet III Workshop,

2 Troubleshooting and evolving large scale networks is still an open problem

3 What's a large scale network? Distributed system Runs for a long time State distributed in many hosts + routers Under autonomous control Lots of users affected Distributed software + configuration

4 Admin challenges Routing optimization Re-convergence can take time Upgrade of faulty service component E.g., upgrade the super-node software in a telephony overlay How to avoid new bugs / ensure smooth upgrades?

5 Validation How do you test an update to the configuration or software of your system? simulation testbed

6 Simulation scalable simulation cheap accuracy? (wireless anyone!?)

7 Testbed may be more accurate but user behavior? testbed expensive to run on large scale? longtime?

8 Problems are nasty They like to surface in large scale, long term deployments with real user behavior/traffic. cannot be caught in small testbeds

9 Our approach leverage Virtual Networks

10 What is virtualization? Abstraction concept Hides the details of what's underneath Provides layer of indirection Enables Resource Sharing More than multiplexing! -- provides Isolation Transparence

11 Virtual Networks (VNets) combination of networ king and comput ing under unified management

12 Virtual Networks (VNets) combination of networ king and comput ing slices (slithers?) of routers links servers end hosts

13 Network Virtualization Provides coherent view and management of Virtual Networks (VNet) Allows to run multiple VNets On shared substrate Under separate administrative control In isolation

14 Mirror VNets Only output of Prod sent to ext entities Ext Host Ext Host Input mirrored to Prod and Mirror Prod Mirror Prod Mirror Prod Mirror Substrate Production VNet Mirror VNet

15 Typical use case Spawn Mirror Switch Prod

16 Typical use case Switch Prod Spawn Mirror Move over Users

17 Mirroring options Not all traffic has to be mirrored Not all mirrored traffic has to be transmitted Control plane traffic: <1% of data volume >90% bugs

18 Mirroring options Reducing Mirroring All traffic Selection (OF Rules, Layer 1-4) Sampling Reducing Transmission Full Transmission Packet Headers Re-synthesized based on stats

19 Mirror VNet case study Operator considers introduction of QoS because of customer complaints of lacking quality at peak load Uses a Mirror VNet to test solution, then switch over

20 Experiment setup BG Traffic Sender Node 1 Node 2 Node 3 BG Traffic Receiver VNET A Vnode 1A Vnode 2A Vnode 3A prod VNET B Vnode 1B Vnode 2B Vnode 3B shadow VOIP sender Vnet entry node Vnet transfer node Moni VOIP receiver /dev/null Traffic flow Vnet exit node

21 Metrics VOIP packet drops MoS: Mean Opinion Score using ITU-T E-Model non linear Quality Of Experience Metric values from 1 (worst) to 5 (best)

22 Experiment phases Phase 1: Things are running smoothly

23 Experiment phases Phase 2: Traffic spikes cause overload in VNode 2, quality degradation

24 Experiment phases Phase 3: Operator introduces Mirror VNet

25 Experiment phases Phase 4: Operator fixes problem in Mirror (introduces QoS)

26 Experiment phases Phase 5: Operator switches VNets

27 Experiment phases Phase 6: Operator dismantles VNET A

28 Experiment results (1) packet drops % phase 1 phase 2 phase 3 phase 4 phase 5 phase 6 Receiver VNET A VNET B BG Traffic BG traffic throughput [MBit/s] Experiment time [s]

29 Experiment results (2) Phases 1 low traffic 2 high traffic 3 start Mirror VNET B 4 enable QOS 5 make VNET B prod. 6 dismantle A

30 Discussion Benefits Resilience against operator mistakes Real user traffic Rollback / undo for networks

31 Discussion Limitations Trade-off between overhead and prediction Elastic/closed loop traffic limits prediction quality

32 Future work Further case studies, larger networks Predict elasticity of traffic, adapt mirror predictions

33 Thank you.