NetFPGA : An Open-Source Hardware Platform for Network Research and Teaching. Nick McKeown, John W. Lockwood, Jad Naous, Glen Gibb

Transcription

1 NetFPGA : An Open-Source Hardware Platform for Network Research and Teaching Nick McKeown, John W. Lockwood, Jad Naous, Glen Gibb S T A N F O R D U N I V E R S I T Y SIGMETRICS Tutorial #2 Tuesday, June 12, 1:45-4:45 pm San Diego, California SIGMETICS NetFPGA Tutorial 1 S T A N F O R D U N I V E R S I T Y Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 2 S T A N F O R D U N I V E R S I T Y

2 A What does a router do? R3 R1 R4 D B E C D R2 Destination D E F Next Hop R3 R3 R5 R5 F SIGMETICS NetFPGA Tutorial 3 S T A N F O R D U N I V E R S I T Y A B 20 bytes C TTL What does a router do? R3 R1 R Ver HLen T.Service Total Packet Length Fragment ID Flags Fragment Offset D Protocol Header Checksum R2 Source Address Destination Destination Address Next Hop R5 Options D (if any) R3 E Data R3 F R5 D F E SIGMETICS NetFPGA Tutorial 4 S T A N F O R D U N I V E R S I T Y

3 What does a router do? R3 A R1 R4 D B E C R2 R5 F SIGMETICS NetFPGA Tutorial 5 S T A N F O R D U N I V E R S I T Y Basic Components of an IP Router Management & CLI Routing Protocols Routing Table Software Control Plane Forwarding Table Switching Hardware Datapath per-packet processing SIGMETICS NetFPGA Tutorial 6 S T A N F O R D U N I V E R S I T Y

4 Per-packet processing in an IP Router 1. Accept packet arriving on an incoming link. 2. Lookup packet destination address in the forwarding table, to identify outgoing port(s). 3. Manipulate IP header: e.g., decrement TTL, update header checksum. 5. Buffer packet in the output queue. 6. Transmit packet onto outgoing link. SIGMETICS NetFPGA Tutorial 7 S T A N F O R D U N I V E R S I T Y Generic Datapath Architecture Header Processing Data Hdr Data Hdr Lookup Update Queue IP Address Header Packet IP Address Next Hop Address Table Buffer Memory SIGMETICS NetFPGA Tutorial 8 S T A N F O R D U N I V E R S I T Y

5 CIDR and Longest Prefix Matches The IP address space is broken into line segments. Each line segment is described by a prefix. A prefix is of the form x/y where x indicates the prefix of all addresses in the line segment, and y indicates the length of the segment. e.g. The prefix 128.9/16 represents the line segment containing addresses in the range: / / / SIGMETICS NetFPGA Tutorial 9 S T A N F O R D U N I V E R S I T Y Classless Interdomain Routing (CIDR) / / / / / Most specific route = longest matching prefix SIGMETICS NetFPGA Tutorial 10 S T A N F O R D U N I V E R S I T Y

6 Techniques for LPM in hardware Linear search Direct lookup Currently requires too much memory Updating a prefix leads to many changes Tries Deterministic lookup time Easily pipelined But requires multiple memories/references TCAM (Ternary CAM) Simple and widely used But low-density, high-power Gradually being replaced by new algorithms SIGMETICS NetFPGA Tutorial 11 S T A N F O R D U N I V E R S I T Y An IP Router on NetFPGA Management & CLI Exception Processing Routing Protocols Routing Table Software Linux user-level processes Forwarding Table Switching Hardware Verilog on NetFPGA PCI board SIGMETICS NetFPGA Tutorial 12 S T A N F O R D U N I V E R S I T Y

7 NetFPGA Router Function 4 Gigabit Ethernet ports Fully programmable FPGA hardware Low cost Open-source FPGA hardware Verilog base design Open-souce Software Drivers in C and C++ SIGMETICS NetFPGA Tutorial 13 S T A N F O R D U N I V E R S I T Y NetFPGA Platform Major Components Interfaces 4 Gigabit Ethernet Ports PCI Host Interface Memories 36Mbits Static RAM 512Mbits DDR2 Dynamic RAM FPGA Resources Block RAMs Configurable Logic Block (CLBs) Memory Mapped Registers SIGMETICS NetFPGA Tutorial 14 S T A N F O R D U N I V E R S I T Y

8 NetFPGA System CAD Tools Monitor Software User Space Linux Kernel Traffic Generator Packet Forwarding Table PCI PCI-e VI VI VI VI NetFPGA Router Hardware NIC (nf2c0.. 3) (eth1.. 2) SIGMETICS NetFPGA Tutorial 15 S T A N F O R D U N I V E R S I T Y NetFPGA System Implementation NetFPGA Blocks Virtex-2 Pro FPGA 4MB ZBT SRAM 64MB DDR2 DRAM PCI Host Interface 4 Gigabit Ethernet ports Intranet Test Ports Dual or Quad Gigabit Etherents on PCI-e Internet Gigabit Ethernet on Motherboard Processor Dual-Core CPU Operating System Linux CentOS 4.4 SIGMETICS NetFPGA Tutorial 16 S T A N F O R D U N I V E R S I T Y

9 NetFPGA Lab Setup Traffic Generator CPU x2 Monitor Software CAD Tools PCI-e PCI NIC Net-FPGA Internet Router Hardware (eth1.. 2) (nf2c0.. 3) Traffic Generator CPU x2 Monitor Software CAD Tools PCI-e PCI NIC Net-FPGA Internet Router Hardware Traffic Generator CPU x2 Monitor Software CAD Tools PCI-e PCI NIC Net-FPGA Internet Router Hardware Video Client SIGMETICS NetFPGA Tutorial 17 S T A N F O R D U N I V E R S I T Y Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 18 S T A N F O R D U N I V E R S I T Y

10 Demo 1 Topology of NetFPGA Routers Video Server HD Display SIGMETICS NetFPGA Tutorial 19 S T A N F O R D U N I V E R S I T Y Demo 1 Setup for the Reference Router Each NetFPGA card has four ports Port 2 connected to local host Ports 1 and 3 connect to Ethernet switch Topology: Ring host NetFPGA Ethernet Switch Video Client SIGMETICS NetFPGA Tutorial 20 S T A N F O R D U N I V E R S I T Y

11 NetFPGA System Setup Traffic Generator CPU x2 Monitor Software CAD Tools PCI-e PCI NIC Net-FPGA Internet Router Hardware Traffic Generator CPU x2 Monitor Software CAD Tools PCI-e PCI NIC Net-FPGA Internet Router Hardware Traffic Generator CPU x2 Monitor Software CAD Tools PCI-e PCI NIC Net-FPGA Internet Router Hardware SIGMETICS NetFPGA Tutorial 21 S T A N F O R D U N I V E R S I T Y Gigabit Switch / Patch Panel SIGMETICS NetFPGA Tutorial 22 S T A N F O R D U N I V E R S I T Y

12 Demo 1 Demo 1: Logical Topology Video Server SIGMETICS NetFPGA Tutorial 23 S T A N F O R D U N I V E R S I T Y Shortest Path Video Client Demo 1 Working IP Router Objectives Become familiar with Stanford Reference Router Observe PW-OSPF re-routing traffic around a failure SIGMETICS NetFPGA Tutorial 24 S T A N F O R D U N I V E R S I T Y

13 Demo 1 Streaming Video through the NetFPGA Video server Source files /var/www/html/video Network URL : Video client Windows Media Player Linux mplayer Video traffic MPEG2 HDTV (35 Mbps) MPEG2 TV (9 Mbps) DVI (3 Mbps) WMF (1.7 Mbps) SIGMETICS NetFPGA Tutorial 25 S T A N F O R D U N I V E R S I T Y Demo 1 Step 1 Observe the Routing Tables The router is already configured and running on your machines The routing table has converged to the routing decisions with minimum number of hops Next, break a link SIGMETICS NetFPGA Tutorial 26 S T A N F O R D U N I V E R S I T Y

14 Demo 1 Step 2 - Dynamic Re-routing Break the link between video server and video client Routers re-route traffic around the broken link and video continues playing Video Client SIGMETICS NetFPGA Tutorial 27 S T A N F O R D U N I V E R S I T Y Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 28 S T A N F O R D U N I V E R S I T Y

15 Integrated Circuit Technology Full-custom Design Complementary Metal Oxide Semiconductor (CMOS) Semi-custom ASIC Design Gate array Standard cell Programmable Logic Device Programmable Array Logic Field Programmable Gate Arrays Processors SIGMETICS NetFPGA Tutorial 29 S T A N F O R D U N I V E R S I T Y Look-Up Tables Combinatorial logic is stored in Look-Up Tables (LUTs) Also called Function Generators (FGs) Capacity is limited by only by number of inputs, not complexity Delay through the LUT is constant A B C D Combinatorial Logic Z A B C D Z Diagram From: Xilinx, Inc SIGMETICS NetFPGA Tutorial 30 S T A N F O R D U N I V E R S I T Y

16 G1 G2 G3 G4 H1 F1 F2 F3 F4 CLB 4 LUT G 4 LUT F 3 LUT H Din M M M M Clk S D Q R S D Q R YQ Y XQ X Xilinx CLB Structure Each slice has four outputs Two registered outputs, two non-registered outputs Two BUFTs associated with each CLB, accessible by all 16 CLB outputs Slice 0 PRE LUT Carry D Q CE CLR Carry logic run vertically Signals run upwards Two independent carry chains per CLB LUT Carry D PRE CE Q CLR Diagram From: Xilinx, Inc (Courtesy Jeff Weintraub) SIGMETICS NetFPGA Tutorial 31 S T A N F O R D U N I V E R S I T Y Field Programmable Gate Arrays CLB Primitive element of FPGA Routing Module Global routing Local interconnect GRM Local Routing CLB PIP Macro Blocks Block Memories Microprocessor I/O Block 3rd Generation LUT-based FPGA Macro Block (up, Mem) Pad Routing CLB Matrix I/O SIGMETICS NetFPGA Tutorial 32 S T A N F O R D U N I V E R S I T Y

17 NetFPGA Block Diagram Four Gigabit Ethernet Interfaces NetFPGA platform FPGA w/provided infrastructure 1 PHY 1 PHY 1 PHY 1 PHY Host computer 1 MAC 1 MAC 1 MAC 1 MAC FIFO packet buffers Virtex II-Pro 50 FPGA with user-defined network logic * Hardware specified with - Verilog source code - Pre-generated cores * Software written for - Embedded PowerPCs - Soft core processors (Microblaze, LEON..) Control, PCI Interface Linux OS - NetFPGA Kernel driver User-defined software networking applications 64MB 18Mb DDR2 18Mb SRAM SDRAM SRAM 3 Gb SATA Board-Board Interconnect SIGMETICS NetFPGA Tutorial 33 S T A N F O R D U N I V E R S I T Y Details of NetFPGA Fits into Standard PCI slot Standard Bus : 32 bits, 33 MHz Provides Interfaces for processing network packets 4 Gigabit Ethernet Ports Allows hardware-accelerated processing Implemented with Field Programmable Gate Array (FPGA) Logic SIGMETICS NetFPGA Tutorial 34 S T A N F O R D U N I V E R S I T Y

18 Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Quick introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 35 S T A N F O R D U N I V E R S I T Y Hardware Description Languages Concurrent By Default, Verilog statements evaluated concurrently Express fine grain parallelism Allows gate-level parallelism Provides Precise Description Eliminates ambiguity about operation Synthesizable Generates hardware from description SIGMETICS NetFPGA Tutorial 36 S T A N F O R D U N I V E R S I T Y

19 Verilog Data Types reg [7:0] A; // 8-bit register, MSB to LSB // (Preferred bit order for NetFPGA) reg [0:15] B; // 16-bit register, LSB to MSB B = {A[7:0],A[0:7]}; // Assignment of bits reg [31:0] Mem [0:1023]; // 1K Word Memory integer Count; // simple signed 32-bit integer integer K[1:64]; // an array of 64 integers time Start, Stop; // Two 64-bit time variables From: CSCI 320 Computer Architecture Handbook on Verilog HDL, by Dr. Daniel C. Hyde : SIGMETICS NetFPGA Tutorial 37 S T A N F O R D U N I V E R S I T Y Signal Multiplexers Two input multiplexer (using if / else) reg y; (a or b or select) if (select) y = a; else y = b; Two input multiplexer (using ternary operator?:) wire t = (select? a : b); From: SIGMETICS NetFPGA Tutorial 38 S T A N F O R D U N I V E R S I T Y

20 Larger Multiplexers Three input multiplexer reg s; (a or b or c or select2) begin case (select2) 2'b00: s = a; 2'b01: s = b; default: s = c; endcase end From: SIGMETICS NetFPGA Tutorial 39 S T A N F O R D U N I V E R S I T Y Synchronous Storage Elements Values change at times governed by clock Din Clock D Q Dout Clock Input to circuit Clock 1 0 t=0 t=1 t=2 Clock Transition time Clock Event Example: Rising edge Flip/Flop Transfers Value From D in to D out on Clock event Din t=0 Dout t=0 S0 A B C A B Clock Transition SIGMETICS NetFPGA Tutorial 40 S T A N F O R D U N I V E R S I T Y

21 Finite State Machines Inputs (X) Copyright 2001, John W. Lockwood, All Rights Reserved Outputs (Z) =λ (X,S(t)) [Mealy] -or- =λ (S(t)) [Moore] Combinational Logic S(t) Q D... Q D δ S(t+1)= (X,S(t)) Next State State Storage SIGMETICS NetFPGA Tutorial 41 S T A N F O R D U N I V E R S I T Y Synthesizable Verilog : Delay Flip/Flops D-type flip flop reg q; (posedge clk) q <= d; D type flip flop with data enable reg q; (posedge clk) if (enable) q <= d; From: SIGMETICS NetFPGA Tutorial 42 S T A N F O R D U N I V E R S I T Y

22 Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 43 S T A N F O R D U N I V E R S I T Y Exercise 1 Make your own router Objectives: Learn how to build hardware Run the software Explore router architecture Execution Start synthesis Rerun the GUI with the new hardware Test connectivity and statistics with pings Explore pipeline in the details page Explore detailed statistics in the details page SIGMETICS NetFPGA Tutorial 44 S T A N F O R D U N I V E R S I T Y

23 Exercise 1 Step 1 - Build the hardware Start terminal, cd to Desktop/netfpga2.1/proje cts/tutorial_router/synth Start synthesis with make SIGMETICS NetFPGA Tutorial 45 S T A N F O R D U N I V E R S I T Y Exercise 1 Step 2 - Run Homemade Router cd to Desktop/ Type: java -jar router.nfp -- use_bin netfpga2.1/projects/tutori al_router/synth/nf2_top_ par.bin to use the just built router hardware The same interface should start again SIGMETICS NetFPGA Tutorial 46 S T A N F O R D U N I V E R S I T Y

24 Exercise 1 Step 3 Configure Router Click Load from File Select config file from the desktop Click Open The port configuration should be loaded and PW-OSFP would soon set the routing tables SIGMETICS NetFPGA Tutorial 47 S T A N F O R D U N I V E R S I T Y Exercise 1 Step 4 - Connectivity and Statistics Ping any addresses x.y where x is from 1-20 and y is 1 or 2 Open the statistics tab in the Quickstart window to see some statistics Explore more statistics in modules under the details tab SIGMETICS NetFPGA Tutorial 48 S T A N F O R D U N I V E R S I T Y

25 Exercise 1 Step 5 - Explore Router Architecture Click the Details tab of the Quickstart window This is the reference router pipeline a canonical, simple to understand, modular router pipeline SIGMETICS NetFPGA Tutorial 49 S T A N F O R D U N I V E R S I T Y Exercise 1 Reference Router Pipeline Five stages Input Input Arbitration Routing Decision and packet modification Output Queueing Output Packet-based module interface Pluggable design MAC rx CPU rx MAC rx CPU rx MAC rx CPU rx MAC rx CPU rx queue 0 queue 0 queue 1 queue 1 queue 2 queue 2 queue 3 queue 3 user_data_path Input Arbite r Output Port Lookup Output Queues MAC tx CPU tx MAC tx CPU tx MAC tx CPU tx MAC tx CPU tx queue 0 queue 0 queue 1 queue 1 queue 2 queue 2 queue 3 queue 3 SIGMETICS NetFPGA Tutorial 50 S T A N F O R D U N I V E R S I T Y

26 Exercise 1 Step 6 - Explore Output Queues Click on the Output Queues module in the Details tab The page gives configuration details and statistics SIGMETICS NetFPGA Tutorial 51 S T A N F O R D U N I V E R S I T Y Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 52 S T A N F O R D U N I V E R S I T Y

27 NetFPGA in the Classroom Stanford CS344: Build an Internet Router Courseware will be available later in 2007 Students work in teams of three (2 software, 1 hardware) Design and implement hardware and software in 8 weeks Software: CLI, PW-OSPF Show interoperability with other groups Add new features in remaining two weeks Firewall, NAT, DRR, Packet capture, Data generator, SIGMETICS NetFPGA Tutorial 53 S T A N F O R D U N I V E R S I T Y Networked FPGAs in Research 1. RCP: Congestion control New module for parsing and overwriting new packet New software to calculate explicit rates 2. Packet Monitoring (ICSI) Network Shunt 3. Deep Packet Inspection (FPX) TCP/IP Flow Reconstruction Regular Expression Matching Bloom Filters 4. Ethane: Network security New switch ( managed flow-table ) deployed 5. Buffer Sizing Reduce buffer size and measure effect on network performance. Need a way to set buffer size, and measure buffer occupancy. SIGMETICS NetFPGA Tutorial 54 S T A N F O R D U N I V E R S I T Y

28 Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 55 S T A N F O R D U N I V E R S I T Y Buffer Requirements in a Router Buffer size matters: Small queues reduce delay Large buffers are expensive Theoretical tools predict requirements Queuing theory Large deviation theory Mean field theory Yet, there is no direct answer. Flows have a closed-loop nature Question arises on whether focus should be on equilibrium state or transient state.. SIGMETICS NetFPGA Tutorial 56 S T A N F O R D U N I V E R S I T Y

29 Rule-of-thumb Source Router C Destination 2T Universally applied rule-of-thumb: A router needs a buffer size: B = 2T C 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines. Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design SIGMETICS NetFPGA Tutorial 57 S T A N F O R D U N I V E R S I T Y The Story So Far # packets at 10Gb/s 1,000,000 2T C (1) 10, T C (2) O(logW ) n (1) Assume: Large number of desynchronized flows; 100% utilization (2) Assume: Large number of desynchronized flows; <100% utilization SIGMETICS NetFPGA Tutorial 58 S T A N F O R D U N I V E R S I T Y

30 Using NetFPGA to explore buffer size Need to reduce buffer size and measure occupancy Alas, not possible in commercial routers So, we will use NetFPGA instead Objective: Use NetFPGA to understand how large a buffer we need for a single TCP flow. SIGMETICS NetFPGA Tutorial 59 S T A N F O R D U N I V E R S I T Y Why 2TxC for a single flow? Only W packets may be outstanding Rule for adjusting W If an ACK is received: W W+1/W If a packet is lost: W W/2 SIGMETICS NetFPGA Tutorial 60 S T A N F O R D U N I V E R S I T Y

31 Time Evolution of a Single TCP Flow Time evolution of a single TCP flow through a router. Buffer is 2T*C Time evolution of a single TCP flow through a router. Buffer is < 2T*C SIGMETICS NetFPGA Tutorial 61 S T A N F O R D U N I V E R S I T Y Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 62 S T A N F O R D U N I V E R S I T Y

32 Demo 2 Topology for Second Demonstration Routers connected in a star topology Port 1 is connected to central Ethernet switch Port 2 connects to local host Ports 0 and 3 unused L2 Switch Netperf Server SIGMETICS NetFPGA Tutorial 63 S T A N F O R D U N I V E R S I T Y VLAN Configuration for Star Configuration SIGMETICS NetFPGA Tutorial 64 S T A N F O R D U N I V E R S I T Y

33 Demo 2 Enhanced Router Objectives Observe router with new modules New modules: rate limiting, delay, event capture Execution Run event capture router Look at routing tables Explore details pane Start tcp transfer, look at queue occupancy Change rate/delay, look at queue occupancy SIGMETICS NetFPGA Tutorial 65 S T A N F O R D U N I V E R S I T Y Demo 2 Step 1 - Run Pre-made Enhanced Router Start terminal and cd to Desktop/ Type java -jar adv_router.nfp A familiar GUI should start SIGMETICS NetFPGA Tutorial 66 S T A N F O R D U N I V E R S I T Y

34 Demo 2 Step 2 - Load Configuration Click Load from File Select config2 file from the desktop Click Open The port configuration should be loaded and PWOSFP would soon set the routing tables SIGMETICS NetFPGA Tutorial 67 S T A N F O R D U N I V E R S I T Y Demo 2 Step 3 - Explore Enhanced Router Click on the Details tab A similar Pipeline to the one seen previously shows with some additions SIGMETICS NetFPGA Tutorial 68 S T A N F O R D U N I V E R S I T Y

35 Demo 2 Enhanced Router Pipeline Three modules added 1. Event Capture to capture output queue events (writes, reads, drops) MAC rx CPU rx MAC rx CPU rx MAC rx CPU rx MAC rx CPU rx queue 0 queue 0 queue 1 queue 1 queue 2 queue 2 queue 3 queue 3 user_data_path Input Arbiter Output Port Lookup Event Capture 2. Rate Limiter to create a bottleneck 3. Delay to model large RTT latency MAC tx CPU tx queue 0 queue 0 Rate Limiter Delay Output Queues CPU tx MAC tx CPU tx MAC tx CPU tx queue 1 queue 2 queue 2 queue 3 queue 3 MAC tx queue 1 SIGMETICS NetFPGA Tutorial 69 S T A N F O R D U N I V E R S I T Y Demo 2 Step 4 - Start TCP Transfer We will use netperf to run a large TCP transfer and look at queue evolution Start a terminal and cd to Desktop/netperf Type./run_tcp_stream SIGMETICS NetFPGA Tutorial 70 S T A N F O R D U N I V E R S I T Y

36 Demo 2 Step 5 - Start Event Capture Click on the Event Capture module under the Details tab This should start the configuration page SIGMETICS NetFPGA Tutorial 71 S T A N F O R D U N I V E R S I T Y Demo 2 Step 6 - Configure Event Capture Check Send to local host to receive events on the local host Check Monitor Queue 2 to monitor output queue of MAC port1 Check Enable Capture to start Event capture SIGMETICS NetFPGA Tutorial 72 S T A N F O R D U N I V E R S I T Y

37 Demo 2 Step 7 - Watch Queue Occupancy Click on Output Queue 2 tab At full speed, router queue size is almost always 0 or 1 packets. SIGMETICS NetFPGA Tutorial 73 S T A N F O R D U N I V E R S I T Y Demo 2 Step 8 Decrease Queue Size From the Quickstart Window, Click on the Configuration tab. Change the output queues size in bytes slider to 40kB SIGMETICS NetFPGA Tutorial 74 S T A N F O R D U N I V E R S I T Y

38 Demo 2 Step 9 - Decrease the Link Rate To create bottleneck and show the TCP sawtooth, link-rate is decreased. In the Details tab click the Rate Limit module Check Enable Set link rate to 1.953Mbps SIGMETICS NetFPGA Tutorial 75 S T A N F O R D U N I V E R S I T Y Demo 2 Step 10 - Add Delay Click on the Delay module under the Details tab Check the Enabled box Increase delay to 48ms SIGMETICS NetFPGA Tutorial 76 S T A N F O R D U N I V E R S I T Y

39 Demo 2 Step 11 - Look at Event Capture Results Click on the Event Capture module under the details tab. The sawtooth pattern should now be visible. SIGMETICS NetFPGA Tutorial 77 S T A N F O R D U N I V E R S I T Y Queue Occupancy Charts SIGMETICS NetFPGA Tutorial 78 S T A N F O R D U N I V E R S I T Y

40 Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 79 S T A N F O R D U N I V E R S I T Y Exercise 2 Enhance Your Router Objectives Add new modules to datapath Synthesize and test router Execution Open user_datapath.v, uncomment delay/rate/event capture modules Synthesize After synthesis, test the new system. SIGMETICS NetFPGA Tutorial 80 S T A N F O R D U N I V E R S I T Y

41 An aside: xemacs Tips We will be modifying the Verilog source code Slides show xemacs, but vim also available. xemacs: To undo, use ctrl+shift+'-' To cancel a multi-keystroke command, just type ctrl+g To select lines, hold shift and press the arrow keys. To comment some selected lines, type ctrl+c+c To uncomment a commented block, move the cursor to one of the lines inside the commented block and type ctrl+c+u To save type ctrl+x+s To search, type ctrl+s search_pattern SIGMETICS NetFPGA Tutorial 81 S T A N F O R D U N I V E R S I T Y Exercise 2 Step 1 - Open the Source We will modify the Verilog source code to add event capture, rate limiter, and delay modules We will simply comment and uncomment existing code Open terminal Type xemacs Desktop/ netfpga2.1/projects/tutorial_ro uter/src/udp/user_data_path.v SIGMETICS NetFPGA Tutorial 82 S T A N F O R D U N I V E R S I T Y

42 Exercise 2 Step 2 - Add wires Now we need to add wires to connect the new modules Search for new wires (ctrl+s new wires) then press Enter Uncomment the wires (ctrl+c+u) SIGMETICS NetFPGA Tutorial 83 S T A N F O R D U N I V E R S I T Y Exercise 2 Step 3 - Connect Event Capture Search for opl_output (ctrl+s opl_output) then press Enter Comment the four lines above (up, shift + up + up + up + up, ctrl+c+c) Uncomment the block below to connect the outputs (ctrl+s opl_out, ctrl+c+u) SIGMETICS NetFPGA Tutorial 84 S T A N F O R D U N I V E R S I T Y

43 Exercise 2 Step 4 - Add the Event Capture Module Search for evt_capture_top (ctrl+s evt_capture_top) the press Enter Uncomment the block (ctrl+c+u) SIGMETICS NetFPGA Tutorial 85 S T A N F O R D U N I V E R S I T Y Exercise 2 Step 5 - Connect the Output Queue to the Rate Limiter Search for port_outputs (ctrl+s ports_outputs, Enter) Comment the 4 lines above (select the four lines by using shift+arrow keys, then type ctrl+c+c) Uncomment the commented block by scrolling down into the block and typing ctrl+c+u SIGMETICS NetFPGA Tutorial 86 S T A N F O R D U N I V E R S I T Y

44 Exercise 2 Step 6 - Add Rate Limiter and Delay Scroll down until you reach the next Excluded block Uncomment the block containing the rate limiter and delay instantiations. (scroll into the block and type ctrl+c+u) Save (ctrl+x+s) SIGMETICS NetFPGA Tutorial 87 S T A N F O R D U N I V E R S I T Y Exercise 2 Step 7 - Build the hardware Start terminal, cd to Desktop/netfpga2.1/proj ects/tutorial_router/synth Start synthesis with make SIGMETICS NetFPGA Tutorial 88 S T A N F O R D U N I V E R S I T Y

45 Exercise 2 Step 8 Perfect the Router If interested, go back to Demo 2: Step 1 after synthesis is done and redo the steps with your own router. You can also change the delay, bandwidth and queue size settings to see how that effects the queue occupancy evolution. SIGMETICS NetFPGA Tutorial 89 S T A N F O R D U N I V E R S I T Y Background Basics of an IP Router The NetFPGA Platform The Stanford Reference Router Tutorial Outline Demo 1: Stanford Reference Router running on NetFPGA Programming NetFPGA Inside the NetFPGA hardware Breakneck introduction to Verilog Exercise 1: Make your own Reference Router Using NetFPGA for teaching and research Example of classroom use Examples of research Example of Enhanced Router: Observing and controlling queue size Motivation: Understanding buffer size requirements in a router Demo 2: An enhanced Reference Router Exercise 2: Enhancing the Reference Router SIGMETICS NetFPGA Tutorial 90 S T A N F O R D U N I V E R S I T Y

46 We re done! Congratulations! SIGMETICS NetFPGA Tutorial 91 S T A N F O R D U N I V E R S I T Y Acknowledgements NetFPGA Team : January 2007 Jianying Luo, Glen Gibb, Nick McKeown, Greg Watson, Jim Weaver, Jad Naous, Ramanan Raghuraman, Paul Hartke, John Lockwood SIGMETICS NetFPGA Tutorial 92 S T A N F O R D U N I V E R S I T Y

47 Acknowledgements Support for the NetFPGA project has been provided by the following companies and institutions Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or of the other sponsors supporting this project. SIGMETICS NetFPGA Tutorial 93 S T A N F O R D U N I V E R S I T Y References on the Web NetFPGA homepage Hardware Request Page SIGMETICS NetFPGA Tutorial 94 S T A N F O R D U N I V E R S I T Y