I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks.

Transcription

1 I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks. Paper by: Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazières, and Nick McKeown, Stanford University Presentation by Rayhaan Jaufeerally May 19, 2017

2 Introduction

3 Aims To reduce toil associated with network management and debugging by creating better network infrastructure that provides insightful interfaces. Toil is defined as manual, repetitive work with no enduring value

4 About the paper Implements a framework for examining the activities of packets travelling through the network. Develops several real-world applications on top of this framework ( 4), 1. ndb Interactive Network Debugger, 2. netwatch Live Invariant Monitor, 3. netshark Path aware packet logger, 4. nprof Hierarchical Network Profiler, Examine scalability concerns of keeping records of all packets through compression and architecture optimizations. 2

5 Packet history Packet History A route that a packet takes through the network plus the switch state and header modifications made at each hop. 3

6 Why packet histories? Being able to see where packets are going is a key component of being able to debug sophisticated networks. 4

7 NetSight

8 Architecture Overview Data Plane Core 1 Core 2 S1 S2 S3 S4 VLAN_HISTORY_DATA VLAN_CTRL_HIST Control Plane Controller 1. Controller n NetServ_1 NetServ_n NetServ Coordinator History Plane 5

9 Postcards Postcard Packet headers + Version + Switch + Out port. Represents what happened to a packet at a particular hop in the network. The authors group postcards together to piece together a packet history. 6

10 Postcard example Host 1 Host 2 Switch 1 Router 1 Switch 2 H1$ ping H2 64 bytes from H2: icmp_seq=1, ttl=63, time=1.03ms 64 bytes from H2: icmp_seq=2, ttl=63, time=1.12ms 64 bytes from H2: icmp_seq=3, ttl=63, time=1.05ms 7

11 Postcard example continued An example sequence of postcards produced would be: Postcard 1 ICMP headers Version Switch 1 Input port 1, output port 2 Postcard 2 ICMP headers Version Switch 1 Input port 1, output port 2 Postcard 3 ICMP headers Version Switch 1 Input port 1, output port 2 8

12 Postcard collection In order to scale linearly, NetSight spreads load generated by postcards across servers, This load balancing is done by sharding over flow key (5 tuple). Switches send postcards to servers, which then preprocess the packets and then shuffle them to a final server which will do the history reassembling. This is where the 5 tuple is used. Postcards are sent from switches to the history plane over a separate VLAN to avoid generating postcards for postcards. Configuration data is pulled from the control plane directly using a transparent OpenFlow proxy. 9

13 History Assembly Postcards arrive out of order, but histories must be in order, so NetSight uses topology information to figure out the order that packets have traversed the network in. The authors chose not to use timestamps, which makes sense given how hard it is to coordinate time. Path Tables are used to group postcards for one packet together, to do this immutable header fields are used (e.g. not src ip, because that changes in NAT). Result: Vast majority of packets were uniquely identifiable in a 1 second window. Path table is a HashTable indexed by this packet ID. 10

14 NetSight API Postcard Filter (PF) Matches a packet at a particular hop. Passed into the system and all matching postcards returned. Can match: BPF, DPID, InPort, OutPort, Version Packet History Filter (PHF) This is a regular expression built upon PF s, for example: Start at Switch1: ^{{Switch1}} Go through Firewall2: {{Firewall2}} Pass through Switch1 and NAT3: {{Switch1}}.*{{NAT3}} 11

15 PHF matching The PHF matching engine is based on the Linux BPF compiler for the packet matching components, and the RE1 regex engine for the complete outer rule matching. This library outputs a Nondeterministic Finite automation which can be executed on input strings. The authors modified the regex library to match on packets instead of characters. 12

16 Relaxed assumptions Dropped postcards are not accounted for, and the system handles them on a best effort basis, Non-unicast packets are treated as graphs instead of a linear history, Due to modifications of elements in flow keys, NAT boxes may cause incomplete packet histories. 13

17 Compression NetSight leverages the fact that many of the 5-tuple items across postcards look similar, and use a diff based scheme to save space. This technique uses a (header, value) pair to encode the difference when a header changes between successive packets in the same stream. This stream is them piped through a generalize Van Jacobson compression algorithm for storage. 14

18 Compression (2) 15

19 Storage 1. In the WAN trace sample the authors used, the storage cost for postcards was 6.84MB/s which fits in a 1TB hard drive for a day. 2. Depending on the use case traces may be stored for a day up to a week. 3. The round length determines how much compression can be done, on a lightly loaded 10G link, a 1M postcard round takes around a second of processing, which is an acceptable delay for human operators. 16

20 Packet History Assembly Typical path length of 2-8 hops takes less than 100ns == 10 million packet histories per core per second. 17

21 Provisioning With 5 hops and 1000 byte packets, the authors were able to use a $2000 server to handle 6.1Gb/s of traffic. They then used several such servers to handle load from an entire campus backbone of 20,000 users. 18

22 Scaling 19

23 SwitchAssist Attempts to offload some of the overheads that the NetSight servers would do, to the switches. For example the postcard compression between flows can be done in the switch. With a size of 15 bytes per compressed postcard, the bandwidth goes from 31% to 7%. 20

24 HostAssist Since most hosts in a DC environment are virtual machine hypervisors, they can be leveraged to insert a globally unique packet ID in each packet, This means that hosts only need to extract this field instead of generating postcards, The switch creates a mini-postcard including this ID and what it did to the packet, This reduces the NetSight overhead to 3% from a naive approach which has a 31% overhead. 21

25 Applications built on NetSight

26 ndb Interactive Network Debugger Based on the concept of gdb for programs, ndb aims to provide a similar functionality for networks. 22

27 ndb: Reachability error Host 1 Host 2 Switch 1 Router 1 Switch 2 Router 2 Switch 3 Host 3 Figure 1: A network topology that encountered a routing problem from H1 to H3. Host1 cannot reach Host3, Why? 23

28 ndb for diagnosing the error ndb can be used to query the network using PHF to determine where the problem is occurring in order to debug the issue. We want to ask: Where are packets from H1 going such that they do not reach H3? ^{{--bpf "ip src H1 and dst H3" --dpid Switch1 --inport S1pH1}}[^{{--dpid Switch3 --outport Sw3H3}}] 24

29 ndb Illustrated Host 1 Host 2 Switch 1 Router 1 Switch 2 Router 2 Switch 3 Host 3 Figure 2: Illustration of possible NetSight output showing misrouted packets destined for H3 from H1. 25

30 netwatch: Live Invariant Monitor Allows network operators to specify constraints that the network should perform within. The system will fire an alert if this constraint is violated. A constraint can be for example, all traffic between two hosts needs to pass through a particular firewall. 26

31 netwatch invariant example Host 1 Switch 1 Host 3 Host 2 Switch 2 Switch 3 Host 4 Figure 3: Illustration of topology that can be used to enforce invariants on. 27

32 Example network invariant All HTTP traffic between hosts should go through Switch 1. This can be implemented by the special function waypoint_routing(traffic_class, waypoint_id) This function creates a PHF of the form: {{--bpf "traffic_class" --dpid not "waypoint_id"}}{{--dpid not "waypoint_id"}}*$ 28

33 Questions?