Afternoon Session 20

Transcription

1 Afternoon Session 20

2

3 DPDK Sample Apps L2fwd Crypto Ethtool L2fwd IVSHME M Exception Path KNI Perf Thread VMDq Timer RxTx Callbacks PTP Client L3fwd VF L2fwd CAT TEP Term Quota & W mark Multi Process L2fwd Keep Alive IP Frag Packet Ordering L3fwd Power IP Pipeline VM Power Manager QoS Sched Load Balancer L2fwd Jobstats Hello World Packet Distrib Link Status Interrupt IPsec Sec GW Skeleton Netmap L3fwd ACL IPv4 Multicast CLI QAT L2fwd IP Reass Bond Vhost Xen QoS Meter L3fwd VMDq DCB Vhost DPDK Network Platforms Group 22

4 How many Pipeline? Run to Completion model Pipeline model Rx PTHREAD Work On Packet Tx Rx Worker Thread 1 Worker Thread 2 Tx Worker Thread n DPDK Flexible & Scalable With # of Cores DPDK Data Plane Development Kit

5 Objective Code Walkthrough of 1) Life Of a Packet from Rx to Pkt Processing to Tx 2) How Developer uses DPDK Objects and configures their Properties? 3) How DPDK Poll mode Driver gets initiated into all the Lcores in the system? Where To Begin?

6 DPDK? Answer: Simple! Through The Application they develop So, A Good Place to Start is to see as 1) How DPDK application interacts with DPDK? 2) What Parameters DPDK application passes to DPDK? 3) What DPDK objects the application configures? That should give us deeper insight into 1) DPDK Objects, 2) their Properties and 3) their Configurations Let Us See a DPDK application as how it interacts with DPDK

7 DPDK Objects & Their Properties Objects: Lcores, Ports, Rings, Mbufs, MemPools;; Properties: Burst_Sizes, Behavior

8 Achieving Performance Silicon Features Attribute Vector instruction set Huge-pages Comments CPU core supports vector instruction set for integer and floating point (SSE: 128-bit integer, AVX1:128-bit integer, AVX2: 256-bit integer) Intel CPUs support 4K, 2MB and 1GB page sizes. Picking the right page size for the data structure minimizes TLB thrashing. DPDK uses hugetlbfs to manage physically mapped huge page area Hardware prefetch Cache and memory alignment Intel Data Direct I/O (DDIO) Cache QoS CPU Frequency scaling/turbo Intel CPUs support prefetching data into all levels of the cache hierarchy (L1, L2, LLC). DPDK aligns all its data structures to 64B multiples. This avoids elements straddling cache lines and DDR memory lines fulfilling requests with single read cycles Is a methodology on Xeon E5 and E7 Platforms where packet I/O data is placed directly in LLC on ingress and sourced from LLC on egress Allows way-allocation control of LLC between multiple applications, controlled by software Allows the core to temporarily boost CPU frequency higher for single threaded performance NUMA Non-Uniform Memory Architecture as much as possible, DPDK tries to allocate memory as close to the core where the code is executing. 27

9 Achieving Attribute Performance Comments Software Concepts Complete user space implementation Software prefetch Allows quick prototyping and development. Compiler can aggressively optimize to use complete instruction set DPDK also uses SW prefetch instructions to limit the effect of memory latency for software pipelining Core-thread affinity Use of vector instructions Function in-lining Algorithmic optimizations Threads are affinitized to a particular core, and quite often have cores dedicated to certain functions. Prevents reloading L1, L2 with instructions/data when threads hop from core to core The code implements algorithms using as much of the instruction set as possible we use vector (SSE, AVX) instructions to implement some components providing significant speed up DPDK implements a number of performance critical functions in header files for easier compiler inlining. To implement functions common in network processing e.g. n-tuple lookups, wildcards, ACLs etc. Hardware offload libraries To complement performant software implementation when capability is available. E.g. 5-tuple lookups can be done on most modern NICs, and act in conjunction with a software classifier implementation Bulk functions Most functions support a bulk mode processing n packets simultaneously. Allows for software pipelining to overcome memory latency 28

10 Question: What Objects You Saw In Previous Foil?

11 Don t Remember? No Worries! What Objects You See below in Run To Completion Model?

12 What Objects you see? Posix Thread What are Rings? Where are

13 Rings for at least 2 Purposes 1) CPU to NIC communication Any other Purpose?

14 To Answer That, Let us Look at Pipeline Model

15 See Below. Ring for Inter Core Communication Is a Ring Bidirectional? Or Unidirectional?

16 1) Init Rx Ring 2) Init Tx Ring

17 Summing up 1) CPU Cores 2) NIC Ports 3) Rings Hardware RX & TX Rings to interface with NIC 4) Rings Software RX & TX Rings for Inter Core Communication 5) MBUF RX & TX Mbuf Memory Buffers (they also have a/c keeping rings) 6) MemPool Memory Pool to allocate Space for Mbufs from

18 What if I can do packet processing without writing code to move them? Setting a bit here & there. Configuring a file or two? I still want to use s/w - NFV & cores

19 Your Value Add With ip-pipeline: The Next Hands-On You will be doing 38

20 Lego Blocks! Imagine The Possibilities!! 39

21 Packet Framework You don t need to write code to move packets. Table 0 Port Out 0 Flow # Actions Flow # Actions Flow # Actions Table 1 Port In 0 Flow # Actions Port Out 1 Flow # Actions Flow # Actions Port In 1 Port Out 2 Standard methodology for pipeline development. Ports and tables are connected together in tree-like topologies, with tables providing the actions to be executed on input packets. 40

22 Your Value Add With ip-pipeline: The Next Hands-On You will be doing If you are familiar more with hardware based packet processing just setting up bits in control registers moves packets No need to write code to move packets Similar look and feel as setting bits in control register - here you set objects in Config files and rules in scripts Lego block type application generator Object model highly scalable and flexible Fast prototyping of your applications Great tool to plan ahead - Application to Architecture mapping Experimentation - Run To Completion Vs Pipeline Quick Evaluation of Processor Budgeting and Performance trade-off 41

23 Sample Pipeline: Enterprise Firewall NIC 0 RX NIC 1 RX HWQ 0.0 (RX) HWQ 1.0 (RX) RX (PT) (P1) SWQ0 SWQ1 Firewall (P2) SWQ2 SINK0 Flow Classif (P3) SWQ3 SINK1 Metering (Flow Act) (P4) SWQ4 SINK2 Routing (P5) HWQ 0.0 (TX) HWQ 1.0 (TX) NIC 0 TX NIC 1 TX Packet Framework Flexible to create/modify pipelines models Various packet processing stages in typical Enterprise Firewall Stages map directly to packet framework s nested, command-driven, table-based architecture Pipeline modules are instantiated, configured and inter-connected through a simple config files loaded at run-time. The config file holds configuration for each pipeline instance including its function, set of input/output ports, tables and actions. Pipeline instances are mapped to CPU cores, with one-to-one or many-to-one relationship. [pipeline 2] type = FIREWALL core = 2 args = "n_rules=4k pkt_type=qinq_ipv4" pktq_in = SWQ0 SWQ1 pktq_out = SWQ2 SINK0 msgq_in = MSGQ2 msgq_out = MSGQ3 42

24 DPDK IP-Pipeline Block Diagram # File <DPDK>/examples/ip_pipeline/edge_router_upstream.cfg : [PIPELINE1] type = FIREWALL core = 1 pktq_in = RXQ0.0 RXQ1.0 pktq_out = SWQ0 SINK0 n_rules = 4096 pkt_type = qinq_ipv4 [PIPELINE2] type = FLOW_CLASSIFICATION core = 1h pktq_in = SWQ0 pktq_out = SWQ1 SINK1 n_flows = key_size = 8 key_offset = 268 hash_offset = 128 Key mask = 00000FFF00000FFF [PIPELINE3] type = FLOW_ACTIONS core = 1h pktq_in = SWQ1 pktq_out = SWQ2 n_flows = n_meters_per_flow = 1 flow_id_offset = 132 ip_hdr_offset = 278 color_offset = 136 [PIPELINE4] type = ROUTING core = 1 pktq_in = SWQ2 pktq_out = TXQ0.0 TXQ1.0 SINK2 encap = ethernet_mpls mpls_color_mark = yes ip_hdr_offset = 278 color_offset =

25 Example: IP pipeline Firewall (ACL) RX Flow Classification Worker Routing Traffic Mgr TX Worker Traffic Mgr # Pipeline Ports IN Tables Ports OUT Actions 1 RX 4x HWQ 4x Stub 4x IP RAS 2 Flow Classification 4x IP RAS, Nx SWQ 1x Hash 3x SWQ NAT, Meter, Stats 3 Firewall 1x SWQ 1x ACL 1x SWQ Add flow 5 Routing & ARP Nx SWQ 1x LPM, 1x Hash 4x IP FRAG 6 Traffic Mgr In 4x IP FRAG 4x Stub 4x TM 7 Traffic Mgr Out 4x TM 4x Stub 4x SWQ 8 TX 4x SWQ 4x Stub 4x HWQ Dec TTL 44

26 Firewall (ACL) Example: IP pipeline RX Flow Classification Worker Routing Traffic Mgr TX Worker Traffic Mgr # Pipeline Ports IN Tables Ports OUT Actions 1 RX 4x HWQ 4x Stub 4x IP RAS 2 Flow Classification 4x IP RAS, Nx SWQ 1x Hash 3x SWQ NAT, Meter, Stats 3 Firewall 1x SWQ 1x ACL 1x SWQ Add flow 5 Routing & ARP Nx SWQ 1x LPM, 1x Hash 4x IP FRAG 6 Traffic Mgr In 4x IP FRAG 4x Stub 4x TM 7 Traffic Mgr Out 4x TM 4x Stub 4x SWQ 8 TX 4x SWQ 4x Stub 4x HWQ Dec TTL 45

27 Implementation using DPDK (2) Focus on functional correctness and block-level performance optimizations Each pipeline on separate CPU core LINK0 Step 2: Optimal mapping to CPU cores Identify optimal mapping of pipelines to CPU cores for single app instance (golden mapping) Maximize performance per physical CPU core: max(p/n), where: P=app throughput, N=#physical CPU cores Flow Classification Firewall Core 5 HT0 LINK0 Routing Core 1 HT1 TX Passthrough Traffic Mgr Firewall Core 1 HT1 Flow Classification Firewall Core 3 HT0 LINK1 TX Passthrough Core 4 HT0 TX Passthrough Routing Core 2 HT0 Core 2 HT1 Flow Classification LINK1 Core 1 HT0 Meter Core 2 HT0 LINK0 LINK1 Routing Core 2 HT0 Core 1 HT0 Step 3: Performance scale up Replicate the golden mapping several times Run multiple app instances on different CPU cores LINK1 Core 1 HT0 Meter Core 2 HT1 Routing LINK1 Routing TX Passthrough Flow Classification Firewall Core 4 HT0 Core 7 HT0 Traffic Mgr Core 2 HT0 LINK0 Meter Core 6 HT0 Core 1 HT0 LINK0 Core 3 HT0 Core 2 HT0 Core 1 HT0 Step 1: Functional development Meter Routing Core 3 HT1 Core 3 HT0 Traffic Mgr Routing Core 4 HT1 Traffic Mgr Core 4 HT0 Routing LINK2 LINK3

28 Dispatch Loop - 47

29 The MAIN ( )

30 49

31 TCP - VLAN

32 the core

33 Time Budget Overview & Architecture 40 minutes Ex 6. Add 2 more pipeline stages Ex 5. 1 Core includes Control plane Architecture = 40 minutes to 1 hour 5 to 8 examples 8 x 15 minutes = 2 hours Total = 2 hours 40 min + Q& A Ex 7. Add 1 additiona l core Ex 1 Run To Completion Ex 4 1 Core Doing All Data Plane Ex 2 Block All Firewall Ex 3 Firewall, Flow Classify, Router 52

34 What You will do in HANDS-ON in the class? Trace function call stack So that you can chart LIFE OF PACKET IN IP_PIPELINE You will ADD, You will DELETE additional pipeline stages You will generate egress traffic and analyse You will ping the pipeline stages You will add rules to the tables Firewall table, Flow Classifier Table, Router Table to name a few You will debug a SEGABORT program to root cause and fix it You will run performance microbenchmarks You will take home additional homeworks 53

35 What will be my call to action after leaving the class Create a new type of pipeline for some custom task Add a new CLI command for one of the pipelines Link CLI command: print a new field from the DPDK internal structure that is currently not printed: if link physical signal/cable is ON or OFF (struct rte_eth_link, field link_status), CPU socket where NIC is physically located, MTU size, etc Routing pipeline: decrement TTL and update checksum (see l3fwd app on how to do this efficiently), check TTL and drop packets with TTL = 0 Routing pipeline: add stats counters per route, print them using a new CLI command Flow classification pipeline: add stats counters per flow, print them using new CLI command Firewall pipeline: add stats counters per rule, print them using new CLI command 54

36 Open one more PuTTY to your HostVM-<M> Open 2 PUTTY 55

37 Open 2 Windows A) FIRST, In the 1 st Window, enter these commands ssh HostVM-<M> Password is password sudo su - cd /home/user/dpdk-pktprocessing vi 1_COMMANDS_YOU_HIGHLIGHT_FROM.txt B) NEXT, in the 2 nd Window, enter these commands ssh HostVM-<M> Password is password sudo su cd /home/user/dpdk-pktprocessing 1 3 Right Click in this Window to Paste 1) Only Highlight from the vi in this Window This is your Highlight to Copy From Window Don t Click. i.e., Don t Select. Only Highlight 2./00_installTools.sh 56

38 PLEASE READ OUTPut message of each script and act on it./00_installtools.sh./01_installlib.sh./02_setuphugepages.sh./03_map_io_userspace.sh 57

39 Just only Execute the script files./00. To./04 58

40 Now That 1) Huge page is set, 2) DPDK library is built, 3] i/o is mapped Time for building a Simple application? Shall we start with a sample Application first? That is your Best Known Method 59

41 Let us go to dpdk.org code browse Line # Function / Macro 63 Rte_eal_init( ) Variable Names 69 rte_eal_remote_launch( ) Lcore_id 52 Lcore_hello( ) 75 Rte_eal_mp_wait_lcore( ) 60

42 61

43 Application export EXTRA_CFLAGS= -O0 g make C./examples/helloworld RTE_TARGET=build gdb args./examples/helloworld/build/app/helloworld c 0x7 n 4 Look at Helloworld. map Question: Why Hello from the cores are not in sequence? Why are they out of order? 62

44 Ip-pipeline main( )

45 App_init ( ) 64

46 OUT IN Back End? pipeline_common_be.c 65

47 PASSTHROUGH PIPELINE - Packet Meta Data -- Source, Destination, Mask, Hash assthrough_be_8c-example.html 66

48 Out IT Goes 67

49 Flow Chart is Great! Can we look at the code? 68

50 Which one is interesting to you? 69

51 app_thread( ) 70

52 Completion PASS THROUGH YYou will be creating PCAP.out PCAP in PASS-THROUGH PASS-THROUGH PCAP Out 71

53 72

54 Pcap file for exercise 1 73

55 ./04_Build_IP_Pipeline.sh cd /home/user/dpdk-pktprocessing/examples/ip_pipeline cat config/pcap.cfg./build/ip_pipeline f config/pcap.cfg <STOP FOR QUESTIONS & DISCUSSIONS> PRESS ENTER QUIT 74

56 ls al config/eth*.pcap mv config/eth1.pcap config/eth1_runtocompletion.pcap mv config/eth2.pcap config/eth2_runtocompletion.pcap mv config/pcap.cfg.out config/runtocompletion_pcap.cfg.out cat config/runtocompletion_pcap.cfgout 75

57 76

58 Exercise 1: Discussion Questions What new info is added to cfg.out files Discuss Your Observations 77

59 ExErcise 2: FIREWALL BLOCKS ALL 78

60 TCP - VLAN

61 the core

62 Config file 81

63 Config file 82

64 Rules 83

65 action, routing, egress RX0 RX1 RX2 TX0 TX1 TX2 RX3 TX3 84

66 $ sudo apt-get install graphviz $./diagram-generator.py -f <configuration file> 85

67 2 more stages added in the later 86

68 Flow_upstream.txt 87

69 Firewall_upstream.txt 88

70 Exercise 2 Firewall blocking all flows 89

71 Exercise 2: Discussion Points 90

72 91

73 FLOW 92