Linux Kernel forwarding. Masakazu Ginzado Co., Ltd.

Transcription

1 Linux Kernel forwarding Masakazu Ginzado Co., Ltd.

2 Agenda Cisco PC-Router Hardware Cisco PC-Router Software Cisco forwarding Linux Kernel forwarding Linux Kernel forwarding Linux PC-Router

3 Hardware Cisco PC-Router Memory Memory Memory Forwarding ASIC CPU

4 Software Cisco PC-Router Protection Loose Strict Multi-tasking Co-operative Pre-emptive Multi-Processor UP SMP

5 Cisco Process Switching ip_input(process) ip_input(process) Processor Routing Table / / ARP Table I/O memory cbb Interface Processor Interface Processor Network media Network media

6 Cisco Fast Switching ip_input(process) ip_input(process) Processor Route Cache Routing Table / / I/O memory ARP Table Interface Processor Interface Processor cbb Network media Network media

7 Cisco Express Forwarding CEF Table 1 octet octet octet octet octet octet octet Adjacency Table cbb076 FE0/ FE1/0

8 Hardware ASIC Forwarding Switch Fabric Interface FIA FIA RAWQ PSA memory Rx Buffer Manager PSA FIFO burst memory RAWQ Physical Layer Interface Module Tx Buffer Manager CPU CEF table FIA Fabric Interface ASIC PSA Packet Switching ASIC RawQ SRAM CEF table DRAM

9 Software vs Hardware PPS 400Mpps 300Mpps 200Mpps 100Mpps 0Mpps Cisco 7200 Cisco 7600

10 Linux Kernel Forwarding ip_rcv ip_rcv_finish ip_forward ip_forward_finish ip_send ip_route_input fib_lookup ip_output rt_hash_code IP Processing ip_finish_output Poll_Queue CPU1 Device 1 Device 2 Device 3 NAPI netif_receive_skb e1000_alloc_rx_buffers e1000_clean_rx_irq Rx_Ring device 3 eth_header eth_type_trans alloc_skb Kernel memory DMA engines dev_queue_xmit e1000_clean_tx_irq Completion queue kfree Root Qdisc device 2 Tx_Ring device 2 qdisc_restart hard_start_xmit e1000_xmit_frame TX-API net_rx_action Interrupt handler net_tx_action H/W Interrupt

11 Receive live-lock & NAPI Old API New API H/W IRQ CPU CPU H/W IRQ H/W IRQ disable IRQ polling packets enable IRQ NIC NIC Packet Packet

12 struct rt_hash_table struct rt_hash_table[] struct rtable struct rtable Src IP Dst IP ifindex struct neighbour

13 struct fib_table struct fib_table struct hlist_head struct fn_hash prefixlen 0 prefixlen 23 Prefix struct fn_zone struct fib_node prefixlen 24 struct hlist_head prefixlen 32 struct fn_zone *fn_zone_list

14 DEFAULT RT/FIXED DST route add default to byte UDP packet abount 850kpps DST DST DST FULL RT/FIXED DST route add /16 to route add /16 to abount 850kpps 64byte UDP packet DST DST DST DEFAULT RT/RANDOM DST route add default to abount 270kpps 64byte UDP packet DST DST DST FULL RT/RANDOM DST route add /16 to route add /16 to abount 270kpps 64byte UDP packet DST DST DST

15 DEFAULT RT/FIXED DST DEFAULT RT/RANDOM DST FULL RT/FIXED DST FULL RT/RANDOM DST 700, , ,000 PPS 400, , , ,000 0 VC6beta E6405 CentOS5 E6405 FreeBSD8 E6405 OpenBSD46 E6405 CentOS5 L5520 E6405 Core2 Duo 2.13GHz L5520 Xeon 2.27GHz

16 tg3_poll ip_route_input.text.acpi_processor_idle tg3_start_xmit_dma_bug kmem_cache_free.text.fn_hash_lookup ip_output netif_receive_skb qdisc_run kmalloc.text ip_rcv dst_destroy kfree acpi_os_read_port dst_alloc kmem_cache_alloc dev_queue_xmit.text.rt_may_expire OTHERS 100% 80% 60% 40% kmem_cache_alloc kfree ip_rcv kmalloc qdisc_run netif_receive_skb ip_output tg3_start_xmit_dma_bug dev_queue_xmit rt_may_expire dst_alloc dst_destroy fn_hash_lookup kmem_cache_free acpi_processor_idle 20% ip_route_input 0% tg3_poll DEF RT/FIXED DST FULL RT/FIXED DST DEF RT/RAND DST FULL RT/RAND DST Kernel Oprofile result by symbols

17 tg3.ko route.c slab.c processor_idle.c dev.c fib_hash.c dst.c sch_generic.c ip_output.c skbuff.c ip_input.c osl.c neighbour.c eth.c list_debug.c rcupdate.c fib_rules.c softirq.c ip_forward.c OTHERS 100% 80% 60% ip_forward.c ip_input.c skbuff.c ip_output.c sch_generic.c dev.c fib_rules.c neighbour.c dst.c fib_hash.c processor_idle.c 40% slab.c 20% route.c 0% tg3.ko DEF RT/FIXED DST FULL RT/FIXED DST DEF RT/RAND DST FULL RT/RAND DST Kernel Oprofile result by apps

18 Ethernet flow control Ehternet Switch PAUSE Frame NIC Buffer is full... Rx Ring

19 irqbalance CPU #1 CPU #2 from NIC #1 to CPU #1 from NIC #2 to CPU #2 APIC irqbalance Interupt NIC #1 NIC#2

20 UMA vs NUMA Core MA Nehalem CPU #1 CPU #2 CPU #1 CPU #2 Core0 Core1 Core0 Core1 Core0 Core1 Core0 Core1 Core2 Core3 Core2 Core3 Core2 Core3 Core2 Core3 L2 Cache L2 Cache L2 Cache L2 Cache L3 Cache L3 Cache FSB 10.66GB/s 10.66GB/s x 3 QPI 25.6GB/s Chipset Memory 32GB/s Chipset Memory 32GB/s 10.66GB/s x 2 Memory 21GB/s

21 ASIC

22 ... Today s high-end PCs are equipped with peripheral component interconnect (PCI) shared buses that fit into the multi-gigabit-persecond routing segment, for a price much lower than that of commercial routers. However, commercially available PC Network Interface Cards (NICs) lack programmability, and require not only packets to cross the PCI bus twice, but also to process them in software by the operating system, reducing routing performance. In this paper we describe and assess the performance of an FPGA-based NIC developed to overcome the main limitations of NICs commercially available today. --- Andrea Bianco, Robert Birke, Jorge M. Finochietto, Giulio Galante, Marco Mellia, Fabio Neri, Michele Petracca, Boosting the performance of PC-based software routers with FPGA-enhanced line cards

23 ASIC vs FPGA ASIC Application Specific Integrated Circuit FPGA Field Programmable Gate Array SRAM

24 KIT STARTER CYCLONE IV GX ALTERA $395.00

25 Memory CPU HDL(Hardware Description Language) library IEEE; use IEEE.std_logic_1164.all; PCI Exp bus Slow Path Fast Path PCI Exp PCI Exp entity RS_FF is port ( R, S in std_logic; Q, Q_B out std_logif ); end RS_FF; architecture RS_FF of RS_FF is signal R_S std_logic_vector( 1 downto 0) begin R_S <= R & S; process ( R_S ) begin case R_S is when "01" => Q <= '1'; Q_B <= '0';... FPGA FPGA Eth Phy Eth Phy

26 HDL

27 Benchmark by Intel 3.0Mpps 2.5Mpps 2.0Mpps 1.5Mpps 1.0Mpps 0.5Mpps Mpps Gbps 20.0Gbps 15.0Gbps 10.0Gbps 5.0Gbps 0Mpps Packet Size 0Gbps performed by Vyatta Community Edition software single Intel Xeon processor 5540 and Intel 82599EB 10 Gigabit Ethernet Controller Source Integrating Services at the Edge

28 Inside Cisco IOS Software Architecture (CCIE Professional Development) - http//my.safaribooksonline.com/ Cisco Express Forwarding - http//my.safaribooksonline.com/ Linux Kernel Source Code - http// Linux Software Router Data Plane Optimization and Performance Evaluation - http//citeseerx.ist.psu.edu/viewdoc/download?doi= &rep=rep1&type=pdf Eliminating Receive Livelock in an Interrupt-driven Kernel - http// Boosting the performance of PC-based software routers with FPGA-enhanced line cards - http// Integrating Services at the Edge - http//edc.intel.com/download.aspx?id=2977&returnurl=/default.aspx Vyatta Replacement Guide - Cisco - http//