Troubleshoo>ng AmLight: Handling Network Events in a Produc>on SDN Environment

Transcription

1 Internet2 Technology Exchange Miami, Sep 26 th 2016 Troubleshoo>ng AmLight: Handling Network Events in a Produc>on SDN Environment Jeronimo Bezerra Florida Interna1onal University <jab@amlight.net> Marcos Schwarz Rede Nacional de Ensino e Pesquisa <marcos.schwarz@rnp.br>

2 Introduc>on to AmLight RFC 7426: SDN Terminology Tests pre-produc>on SDN Topologies What should be monitored? Control Plane Monitoring Data Plane Monitoring Future Outline 2

3 AmLight is a Distributed Academic Exchange Point Produc>on SDN Infrastructure since Aug 2014 Partnership involving FIU, NSF, ANSP, RNP, RedClara and AURA Connects two academic exchange points: AMPATH/Miami and SouthernLight/Brazil Carries Academic and Non-Academic/Commercial traffic L2VPN, IPv4, IPv6, Mul>cast Supports Network Programmability/Slicing Flow Space Firewall for Network Programmability/Slicing OESS for L2VPNs OGF Network Service Interface (NSI) enabled ONOS/SDN-IP for Academic IPv4 Currently 5 slices for experimenta>on (including Global ONOS SDN-IP) Currently, opera>ng with more than 800 flows (produc>on and experimenta>on) Web site: 3

4 AmLight SDN Stack Northbound: Users APIs AmLight s NRENs IDCP OSCARS NSI OpenNSA Other NRENs OESS OESS SDN-IP ONOS FIBRE Univ. Twente NOX NOX ONOS Internet2 Other Testbeds Virtualization/Slices (FlowSpace Firewall) Andes1 Ampath1 Ampath2 SouthernLight Physical Layer Southbound API: Andes2 4

5 SDN: Layers and Architecture Terminology This presenta>on will use the SDN terminology standardized through IETF RFC 7426: Four planes: Applica>on, Control, Forwarding Plane & Management Planes Interfaces: Service, Control Plane Southbound and Management Plane Southbound interfaces Services and Applica>ons Application Application Plane Service Network Services Abstraction Layer (NSAL) Control Plane Forwarding Plane Service Interface Service App App Service Control Abstraction Layer (CAL) CP Southbound Interface App Management Plane Management Abstraction Layer (CAL) MP Southbound Interface Device and Resource Abstraction Layer (DAL) Operational Plane 5

6 Tests pre-produc>on Before applying any change to the SDN environment, all planes, apps and services need to be validated in a controlled environment Same sogware and devices used in produc>on need to be available for tests Many tools and approaches available, for example, OFTest, Ryu Switch Test, Cbench and some commercial possibili>es Some tests might cause instability to the SDN stack (don t try these tests in produc>on) Special aien>on is required for the Control and Data planes Many publica>ons with different methodologies and tests Unittest OFTest Ryu Switch Test, Cbench,... OFTest Ryu Switch Test,... Application Forwarding Plane Application Plane Service Interface Service Network Services Abstraction Layer (NSAL) Control Plane Management Plane Service App App Service Control Abstraction Layer (CAL) CP Southbound Interface App Management Abstraction Layer (CAL) MP Southbound Interface Device and Resource Abstraction Layer (DAL) Operational Plane... 6

7 Troubleshoo>ng a produc>on SDN network Troubleshoo>ng a produc>on environment has different requirements It needs to be agile and least disrup>ve as possible It might need historical informa>on and understanding of traffic going through the network Tools have to be handy Legacy troubleshoo>ng tools are par>ally useful or completely useless OAM (Opera>on, Administra>on and Maintenance) is not supported by OpenFlow (yet) Ping, traceroute, SNMP, wireshark/tcpdump are somehow compromised Deep knowledge of the hardware and sogware plakorm is required: Usage of the hidden commands becomes part of your rou>ne Sugges>on: get a premium support contract Going through the level 2 TAC team will increase your stress and the network recovery >me 7

8 SDN Topologies: Star>ng Simple Usually, with just one SDN App, troubleshoo>ng is less complex One SDN App is connected through an out-ofband network to mul>ple OF switches Usually, the SDN App has full control of ports and VLANs SDN App Application Layer OpenFlow 1.x A good network sniffer and a Syslog server are the key to success here Helps validate the OpenFlow messages sent and received Eases access to error messages User A User B 8

9 SDN Topologies: Adding Complexity Different control planes in parallel tends to be a consequence of slicing More applica>ons to understand and track Different levels of sogware stability and debug Higher chances of network outages OESS ONOS/SDN-IP Testbed Application Layer Slicing/Par>>oning adds complexity: OpenFlow communica>on between OpenFlow switch and SDN App is not end-to-end: OF Switch -> Slicer or Slicer -> OF App Complexity to track which switch is talking to which SDN App and vice-versa OF doesn t carry DPID on each OF message User A FlowSpace Firewall User B Tradi>onal sniffers are not enough to track indirect OpenFlow messages 9

10 Control Plane: what should be monitored? Everything concerned to the OpenFlow communica>on: # of flows installed Avoid gepng close to the limits documented (weird stuff might happen) Rate of flowmods, PacketOut/PacketIn & Stats requests / second: Switch s CPU is directly affected by these rates # of OFP_FLOW_ERROR messages: Some messages might indicate that a crash is about to happen (FULL_TABLE) Flows dura>on: Helps to understand traffic disrup>on due to flows being reinstalled Flow and Port Counters (bps and pps) If slicing is a reality, collect counters per slice Most of the SDN apps don t provide such data, some provide through REST interfaces 10

11 Data Plane: what should be monitored? Data Plane Monitoring: In some cases, everything looks ok, but traffic is not flowing Some possible data plane black holes: A specific line card or interface discarding all traffic Due to an interface memory issue, flows are installed but traffic is discarded Interface down in one side but up in the remote and the SDN App doesn t understand that For instance: 10G LAN-PHY, Ethernet circuits and 100G long haul circuits In this case, depending of the side, the SDN App installs the circuits poin>ng to the affected link, discarding all traffic A specific installed flow entry crashed Due to an interface memory issue, one specific flow is compromissed and traffic is discarded Depending of the number of OpenFlow switches and flow entries, finding the problem might be extremely >me-consuming In these cases, in-band tests are required: Just a very few SDN Apps test in-band per link No SDN Apps test in-band per flow 11

12 Disclaimer: What you are about to see and hear is the AmLight s experience. We are not saying these are the best or recommended methods probably are not. Don t try them on your network! 12

13 Monitoring the OpenFlow messages with passive packet capture: Non-intrusive Almost risk-free Few tools available: Wireshark/tshark/tcpdump OpenFlow Flight Recorder AmLight OpenFlow Sniffer AmLight OpenFlow Sniffer was created to be CLI-based with support to environments with slicers: Dissects 100% of Doesn t require GUI or Xwindow End-to-end communica>on visualiza>on Colors to highlight important fields Many filters available to op>mize tshoot! Source: github.com/jab1982/ofp_sniffer Control Plane Monitoring Monitor msgs: OpenFlow Sniffer, OFFR User A libpcap OESS ONOS/SDN-IP FlowSpace Firewall Testbed Application Layer User B 13

14 Control Plane Monitoring [2] Monitoring All Applica>ons and Counters in a centralized NMS: Scripts collect info from SDN Apps REST interfaces and export via JSON Zabbix imports JSON data and save into a MySQL Database Currently, collec>ng data from OESS, ONOS, FSFW and switches Examples: OESS Monitoring: Zabbix + customized scripts ONOS/SDN-IP FlowSpace Firewall SNMP, REST, JavaAPI, etc Testbed Application Layer User A User B 14

15 Most of the SDN Apps use LLDP or BDDP for topology discovery Once the topology is discovered, these protocols are not used to monitor the topology Also, interval between LLDP/BDDP packets is not appropriated for link monitoring Data Plane Monitoring OESS ONOS/SDN-IP Testbed Application Layer An in-band tes>ng approach is needed to validate the Data Plane OESS does through its Forwarding Verifica>on module Most of other SDN Apps don t have anything equivalent Even though OESS/FVD validates the data path, it doesn t valite users flows A full port issue is detected, but a single flow issue is not User A FlowSpace Firewall Monitoring Data plane: Trunk ports: OESS FWD User B 15

16 Data Plane Monitoring [2] Monitoring individual flows is important but extremely complex Being proac>ve with all flows is desired but the interval between tests and number of flows needed to be taken into considera>on Using a reac>ve approach is the best sugges>on Users won t be happy, but your switches won t crash Approaches to test users flows are yet considered experimental A SDNTrace protocol was proposed: hip://sdntrace-protocol.readthedocs.io/en/latest/ 16 User A Monitoring User Flows: SDNTrace OESS ONOS/SDN-IP FlowSpace Firewall Testbed Application Layer User B

17 Data Plane Monitoring [3] AmLight's developed its own SDNTrace to test users flows without changing them Works through GUI or REST Very lightweight Very cheap, only two-four flow entries needed Traces L2 and L3 flows Yet under evalua>on at AmLight Developed in collabora>on with the Academic Network of Sao Paulo/Brazil Tracing a circuit is done in seconds instead of many minutes and can work with both Zabbix and Nagios Github: github.com/amlight/sdntrace 17

18 Future New tools/scripts/protocols are s>ll needed S>ll a long and painful journey ahead OpenFlow-OAM? Improvements to OpenFlow agents are being constantly released But new bugs are coming with them L Some SDN monitoring-only applica>ons are being proposed and developed AmLight is developing its own SDN Looking Glass to consolidate all passive and ac>ve monitoring ac>vi>es associated to the SDN environment (to be released by January) But side applica>ons are not ideal: it is important that all SDN Applica>ons incorporate troubleshoo>ng capabili>es in their core! 18

19 Off-topic: Sugges>ons to Network Engineers What is/will be our posi>on descrip>on? Network Engineers? SDN Engineers? Research Network Engineers? Maybe Network Engineers 2.0? It doesn t maier the descrip>on, it maiers that we have to evolve! With SDN, troubleshoo>ng is very different: instead of using CLI and sniffers, we need to read code and applica>on s logs Most of us hate sogware development, but it is >me to change our mentality At AmLight, I don t remember last >me I created a VLAN using a CLI If SDN becomes the next de-facto standard, it will happen in a few years We all s>ll have >me to learn and get prepared for this new reality Recommenda>ons: Learn Python or Java (JavaScript is a plus) Ryu is a very interes>ng OpenFlow controller to start with Join Ryu or ONOS mailing lists Mininet is your friend! 19

20 Internet2 Technology Exchange Sep 26th Troubleshoo>ng AmLight: Handling Network Events in a Produc>on SDN Environment Ques8ons??? Jeronimo Bezerra Florida Interna>onal University <jab@amlight.net>