I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks

Transcription

1 I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks Nikhil Handigol With Brandon Heller, Vimal Jeyakumar, David Mazières, Nick McKeown NSDI 2014, SeaOle, WA April 2, 2014

2 Bug Story: Incomplete Handover A X Match: AcQon Src A, Dst B: Output to Y WiFi AP Y WiFi AP Z B 2

3 Network Outages make news headlines HosQng.com's New Jersey data center was taken down on June 1, 2010, igniqng a cloud outage and connec7vity loss for nearly two hours HosQng.com said the connecqvity loss was due to a so>ware bug in a Cisco switch that caused the switch to fail. On April 26, 2010, NetSuite suffered a service outage that rendered its cloud- based applicaqons inaccessible to customers worldwide for 30 minutes NetSuite blamed a network issue for the downqme. The Planet was rocked by a pair of network outages that knocked it off line for about 90 minutes on May 2, The outages caused disrupqons for another 90 minutes the following morning... InvesQgaQon found that the outage was caused by a fault in a router in one of the company's data centers. 3

4 TroubleshooQng Networks is Hard Today Forwarding State Forwarding State ping Forwarding State Forwarding State Forwarding State traceroute Lots and lots of graphs tcpdump/span/sflow SNMP Tedious and ad hoc Requires skill and experience Not guaranteed to provide helpful answers (source: NANOG Survey in AutomaQc Test Packet GeneraQon, Hongyi Zeng, et. al.) 4

5 We want complete network visibility ping SNMP Visibility Spectrum We want to be here traceroute sflow Complete visibility: every event that ever happened to every packet 5

6 Talk Outline 1. How to achieve complete network visibility An abstracqon: Packet History A plagorm: 2. Why achieving complete visibility is feasible Data compression MapReduce- style scale- out design 6

7 Packet History Forwarding State Forwarding State Forwarding State Forwarding State Packet history = Path taken by a packet + Header modificaqons + state encountered 7

8 Our TroubleshooQng Workflow Forwarding State Forwarding State Forwarding State Forwarding State 1. Record and store all packet histories 2. Query and use packet histories of errant packets 8

9 A plagorm to capture and filter packet histories of interest 9

10 Control Plane Flow Table State Recorder Postcard Collector 10

11 Match Control ACT Plane Flow Match Table ACT State Recorder Postcard Collector 11

12 Control Plane Flow Table State Recorder Postcard Collector 12

13 Control Plane Flow Table State Recorder Packet Header ID Output port Step 1: Generate postcards Postcard Collector Version Version - > Flow Table State 13

14 ReconstrucQng Packet Histories Step 2: Group postcards by generaqng packet Packet Header Packet Header ID Packet Output Header port ID Packet Output Header port Version ID Output port Version ID Output port Version Version 14

15 ReconstrucQng Packet Histories Step 3: Sort postcards using topology Topo- sort 15

16 1. <Match, AcQon> 2. <Match, AcQon> 3. <Match, AcQon> 4. <Match, AcQon> 5. <Match, AcQon> Control Plane Flow Table State Recorder 1. <Match, AcQon> 2. <Match, AcQon> 3. <Match, AcQon> 4. <Match, AcQon> 5. <Match, AcQon> <Match, AcQon> 2. <Match, AcQon> 3. <Match, AcQon> 4. <Match, AcQon> 5. <Match, AcQon> <Match, AcQon> 2. <Match, AcQon> 3. <Match, AcQon> 4. <Match, AcQon> 5. <Match, AcQon> Postcard Collector 16

17 Postcards Control Plane Flow Table State Recorder TroubleshooQng Apps Reachability errors IsolaQon violaqon Black holes Waypoint rouqng violaqon Postcard Collector Packet History Assembly API Packet History Filter: A regular- expression- like language to specify packet histories of interest Filtered Packet Histories TroubleshooQng TroubleshooQng TroubleshooQng ApplicaQon TroubleshooQng ApplicaQon ApplicaQon App 17

18 Bug Story: Incomplete Handover Packet History Filter Packet History Packet History Filter Pkts from server not reaching the client X WiFi AP Y WiFi AP Z Packet History X: inport: p0, outports: [p1] mods: [...] state version: 3 Y: inport p1, outports: [p3] mods:... X Y 18

19 TroubleshooQng Apps ndb: InteracQve network debugger ndb netwatch netwatch: Live network invariant monitor nprof: Hierarchical network profiler nprof netshark netshark: Network- wide wireshark 19

20 But will it scale? 20

21 Why generaqng postcards for every packet at every hop is crazy! Network Overhead 64 byte- postcard/pkt/hop Stanford Network: 5 hops avg, 1031 byte avg pkt 31% extra traffic! Processing Overhead Packet history assembly and filtering Storage Overhead 21

22 Why generaqng postcards for every not packet at every hop is ^ crazy! Cost is OK for low- uqlizaqon networks E.g., test networks, bring- up phase networks Single server can handle enqre Stanford traffic 22

23 Why generaqng postcards for every not packet at every hop is ^ crazy! Huge redundancy in packet header fields Only a few fields change IP ID, TCP seq. no. Postcards can be compressed to bytes/pkt Diff- based compression 23

24 Why generaqng postcards for every not packet at every hop is ^ crazy! Postcard processing is embarrassingly parallel Each packet history can be processed independent of other packet histories Assembly Filtering 24

25 Scaling Performance Postcards Compressed Postcard Lists Compressed Packet Histories Shuffle 25

26 Scaling Performance Postcards Compressed Postcard Lists Compressed Packet Histories Shuffle 26

27 Scaling Performance Postcards Compressed Postcard Lists Compressed Packet Histories Shuffle 27

28 Scaling Performance Postcards Compressed Postcard Lists Compressed Packet Histories Shuffle 28

29 Variants 29

30 - Assist moves postcard compression to switches Move postcard compression to switches with simple hardware mechanisms Shuffle 30

31 - HostAssist exploits visibility from the hypervisor Packet Header HV (1) Store packet header at the hypervisor (2) Add unique pkt ID Shuffle Mini- postcards contain only unique pkt ID and switch state version 31

32 Overhead ReducQon in Basic (naïve) : 31% extra traffic in Stanford backbone network - Assist: 7% Host- Assist: 3% 32

33 Takeaways Complete network visibility is possible Packet History: a powerful troubleshooqng abstracqon that gives complete visibility : a plagorm to capture and filter packet histories of interest Complete network visibility is feasible It is possible to collect and filter packet histories at scale 33

34 Every I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazie res, Nick McKeown {nikhilh,brandonh}@cs.stanford.edu, {jvimal,nickm}@stanford.edu, http: // www. scs. stanford. edu/ ~ dm/ addr/ Stanford University, Stanford, CA USA These authors contributed equally to this work API Abstract The complexity of networks has outpaced our tools to debug them; today, administrators use manual tools to diagnose problems. In this paper, we show how packet histories the full stories of every packet s journey through the network can simplify network diagnosis. To demonstrate the usefulness of packet histories and the practical feasibility of constructing them, we built, an extensible platform that captures packet histories and enables applications to concisely and flexibly retrieve packet 1. Host A cannot talk to Host B. Show me where packets from A intended for B are going, along with any header modifications. 2. I don t want forwarding loops in my network, even transient ones. Show me every packet that passes the same switch twice. 3. Some hosts are failing to grab IP addresses. hop://yuba.stanford.edu/netsight Show me where DHCP traffic is going in the network. 4. One port is experiencing congestion. Show me 34 the traffic sources causing the congestion.