Rx Stack Accelerator for 10 GbE Integrated NIC

Transcription

1 IEEE Hot Interconnects 20, Santa Clara, CA, Aug , 2012 Rx Stack Accelerator for 10 GbE Integrated NIC F. Abel, C. Hagleitner IBM Research Zurich Switzerland F. Verplanken IBM Systems & Technology Group La Gaude France 2009 IBM Corporation

2 EI3 RX EI3 TX L2 L2 L2 CI HEA/ MC L2 Comp BIC BIC RegX MC Crypto XML Discrete vs Integrated Network Interface Controller Discrete NIC (dnic) eripheral ASIC device Marketed as: Ethernet Controller Converged Controller Integrated NIC (inic) Sun Niagara 2 Freescale QorIQ TM family IBM oweren TM 410 mm 2 (1.43 billion transistors) CI/Ethernet Network Interface Card (NIC) inic ~3% BIC BIC AT 2 AT 3 BUS AT 0 AT 1 BEC Converged Network Adapter (CNA) DDR3 DDR3 DDR3 DDR3 LAN On Motherboard (LOM) Higher performance, lower latency Lower power consumption Significant cost reduction Alter the general-purpose nature of the computer complex 2

3 Outline Hardware context of this work oweren TM / Host Ethernet Adapter Functional requirements Architecture of the Rx Stack Accelerator Data path acket parser acket handler Results Implementation erformance Summary and conclusions 3

4 Hardware Context 4

5 Context: oweren TM / Host Ethernet Adapter (HEA) AT 0 AT 1 AT 2 AT 3 Mem HY Mem HY LL LL LL LL LL LL L L L L LL LL Comp / Decomp Crypto XML attern Engine 2MB L2 2MB L2 2MB L2 2MB L2 MC MC BIC BIC Bus Bus External Controller Root/E Engine CI Exp Gen 2 BIC Root Engine CI Exp Gen 2 BIC Quad-port 10 GbE NIC a.k.a Host Ethernet Adapter (HEA) bus Access Macro ervasive 4x 10GE MAC or 4x 1GE MAC EI3 EI3 EI3 x8 HY x8 HY x8 HY x4 HY Misc I/O 5

6 Overview of the Host Ethernet Adapter (HEA) 4 10 GbE state-of-art Ethernet controller featuring: I/O virtualization support Through 128 queue pairs Internal layer-2 switch for partition-to-partition data traffic Flexible queue selection and scheduling assist Rx and Tx protocol acceleration Low-latency through direct processor bus attachment and cache injection Multi-core scaling, Interrupt and receive coalescing assist, 9 KB Jumbo frame support, Memory address protection... Equivalent functionality and performance levels as modern discrete NICs Small footprint (13 mm 2 ) and low power (2.6 W) SoC Bus Host Interface Tx Acc. Tx Tx Tx Acc. Acc. Acc. Rx Acc. Rx Rx Rx Acc Acc Acc XGMAC XGMAC XGMAC XGMAC XGMAC XGMAC XGXSCS XGCS XGCS XGCS XGXSCS XGCS XGCS XGCS 6

7 RXACC Functional Requirements 7

8 Target Domain Requirements oweren TM targets network-facing applications Must cope with a large spectrum of protocols (13+), traffic characteristics and policies E.g. routers, firewalls, intrusion-prevention systems and network analytics Must be able to adapt to changes Handle other existing or emerging protocols Virtualized I/Os Must sustain 16 Gb/s (internal layer-2 switch) Ethernet: A trucking service for application data rotocol layered architecture (i.e., encapsulation) permutations Iv4t ISL DIX VLAN QinQ SA SNA oe Iv4 TC ICMv6 L1.5 L2 L2 L2 L2.5 L3 L4 ayload L2 8 MLS MLS MLS MLS MLS MLS Iv6 Iv6t UD Ipv6 Ipv6 Ext. Ext. Hdr. Ipv6 Ext. Hdr. Ipv6 Ext. Hdr. Hdr.

9 Distribution of the rotocol Stacks U, 558 L5, 424 L4, L4, 2118 RH, 186 LF, 329 MF, 331 FF, 651 I6t, 186 Iv6, I6, M, 85 A, 25 DIX, SA, 1079 SNA, 1078 Q, VLAN, 1183 QQ, 362, 972 I4t, 197 I4, 940 M1, 1788 M6, 394 M5, 665 M4, 941 M3, 1231 M2, 1510 MLS2,

10 10 Offloaded Service Requirements WHY TC Rx+Tx processing ~3000 instructions (assuming zero-copy and checksum offload) 10 GbE Occurrence of 64 B packets 67.2 ns (14.8 Mfps) A generally accepted rule of thumb 1 GHz / 1 Gb/s WHAT (business as usual) 'per-byte' operations check-summing 'per-packet' operations header processing: VLAN, MAC, flow identification, QoS determination, discard, errors, traffic steering HOW (different) Flexible way rogrammable parsing and processing (rule-based) Meta-data descriptors Save instructions per frame E.g., rotocol stack signature (31b), rotocol statck offsets (64b) D I X S A S N A V L A N Q i n Q o E M L S 1 M L S 2 M L S 3 M L S 4 M L S 5 M L S 6 I v 4 I v 4 t I v 6 I v 6 t F F r g M F r g L F r g R H d r L 4 L 5 N u N u N u N u N u I n c o m M a l f d A b o r t N u L4 rot. L2.5 os. L3 osition L4 osition ayload osition

11 Rx Stack Accelerator 11

12 Rx Stack rocessing Can be decomposed in three major tasks: (t1) arsing Identify protocol stack + position of protocol fields (t2) Data extraction Locate and retrieve data to be processed (t3) rocessing Execute instructions based on identified rules Filtering (MAC, VLAN), VLAN extraction, Rx queue assignment, checksum verification, flow determination, discard, counters increments, Main characteristics exhibited by protocol processing applications [Jantsch1998] (c1) Intensive use of pattern matching especially on headers (c2) Complex and control dominated flow many nested if-then-else and case structures (c3) Intensive use of irregular memory accesses various sizes and patterns 12

13 RXACC Architecture A Transport Triggered Architecture (TTA) w/ 5 transport buses (1 byte/bus) Cut-through architecture low latency efficient irregular mem. access (c3) clk = 625 MHz octaword-out Data ath Transport Src FU 1 (180 bits) meta-data acket Handler Sockets Dst acket arser (programmable) FU N Ctrl Frame signature saves instructions (avg.) IL architecture instruction-level parallelism parallel, independent, and pipelined coprocessors (FUs) performance scalability hardware efficiency hardware modularity ultimate RISC architecture - only 1 instruction move data octaword-in (16 B) rog. FSM + Balanced Routing Table search algorithm (B-FSM) efficient pattern-matching (c1) efficient one-cycle multiway branching (c2) 13

14 Data ath (D) 14

15 Multi-field acket Inspection & Data Extraction QQ/VLAN/SA/SNA/Iv6/UD Off.5 Off.4 Off.3 Off.2 Off.1 F Off.5 Off.4 Off.3 Off.2 Off.1 F 15 Current fields of interest Next fields of interest

16 Data ath Architecture Cut-through architecture Relaxed buffering (2 16B) + Low latency Flexible data extraction (any 5 bytes within the 32B window) Entire packet is exposed to the parser Can inspect and match any data of the frame D00 D01 D02 D03 D04 D05 D06 D07 D08 D09 D10 D11 D12 D13 D14 D15 Status HOST I/F Legend: L2 (DIX) L3 (Iv4) Extracted fields: Frmtr = 16 Offset1 = 2 Offset2 = 3 Offset3 = 7 Offset4 = 8 Offset5 = 9 R H Tot. Len. R L MAC DA Id. MAC SA r Hd. ot. Csum Ether Type Status SA DA Status M 00 M 0 B C N T SrcOp I/F Tranport I/F B1-B5 Frmtr Off(1-5) D00 D01 D02 D03 D04 D05 D06 D07 D08 D09 D10 D11 D12 D13 D14 D15 Status XGMAC I/F 16

17 acket arser () 17

18 acket arser Design space: micro-coded limited branch capabilities multi-ghz operation power finite state machine (FSM) efficient but inflexible programmable finite state machine (pfsm) high performance and energy efficient Iv4_SA S22 Rule Current State Input symbols Input 1 Input 2 Input 3 Next State riority R44 S22 XXXX_XXXXb XXXX_XXXXb XXXX_XXXXb S24 0 R52 S24 XXXX_0101b 0001_0001b X00X_XXXXb S65 0 Iv4_DA_rot S24 R53 S24 XXXX_XXXXb 0001_0001b X00X_XXXXb S82 1 R61 S24 XXXX_XXXXb 0010_1001b X00X_XXXXb S36 2 R112 S24 XXXX_0101b 0000_0110b X00X_XXXXb S Iv4_UD S65 Iv4_TC S44 Iv4_UL4 S36 State transition diagram State transition rules ('X' symbol represents a don't care bit) HW Engine We use the B-FSM architecture [van Lunteren2006] 'B' stands for Balanced Routing Table search algorithm (BaRT) Originally designed for longest matching prefix searches (i.e. routing table lookups)

19 Output H I/F acket arser Architecture Input D I/F B1 N -B3 N Data ath Controller FInc, Src1-Src5 Range Comparator RCC Address Generator H next-state State Register output Rule Selector input Transition Rule Memory B-FSM engine bits 256 rules (1 rule = 160 bits) S C B1 C -B3 C STATE RCR B1 B2 B3 S N Match 1 F Transition Rule Selector Match 2 G H' Match 3 LUT Match 4 R1 R2 R3 R4 FWR Transition Rule Memory (F, F', G, G', H') Reg. File A,B Stat Reg. Input Rule Structure 19 Test part Result part H part S C Input symbols S N Output symbols H sym.

20 acket Handler (H) 20

21 acket Handler Architecture Functional Units (FU) Independent coprocessors (I-blocks) Concurrent operation IL erformance Configurable through MMIO Implement TTA socket registers: O = operand register(s) T = trigger register (note: result registers are not implemented) FU example: the HASHER 32 meta-data Hasher (XORs) Symmetry Bit Mask MMIO config. FU 1 acket Handler FU N O 1 O 2 O 3 O N T Sockets Transport Dst Ctrl Transport Decode 21 Dst

22 Rule Format 160 bits CS Compare Value Compare Mask NS Src 1 Src 2 Src 3 Src 4 Src 5 Dst 1 Dst 2 Dst 3 Dst 4 Dst 5 Test art (62 bits) Result art (variable-length instruction) (87 bits) Frame pointer increment (Long/Short Fixed/Variable) Source of the operands (i.e. offsets w.r.t the frame pointer) Destination of the operands (i.e. functional unit registers) H art (9 bits) Simplified Instruction Set Frame pointer instructions e.g. ffpinc(4); // fixed increment e.g. vfpinc(1); // variable increment Move instruction e.g.: move(d(4), CSI.O1); // unicast destination e.g.: move(d(12), HSH.O1 & CSI.O1 & CST.O9); // multicast destination (socket write sharing) 22

23 erformance and Implementation Results 23

24 Implementation Results Area (in 45 nm SOI technology) 1 RXACC = 0.7 mm 2 4 RXACC = 21.5 % of HEA (entire HEA = 13 mm 2 ) Clock Frequency 625 MHz (27% of core frequency) ower (estimate) RXACC 0.15 W (entire HEA = 2.6 W) Transition Rule Memory Utilization 256 rules 64 x (4 x 160) bits = 40 kbits 68 states out of 128 (53 %) 221 rules out of 256 (86 %) Transition rule memory + ECC logic = 15 % of RXACC 24

25 RXACC Bandwidth Bandwidth (Gb/s) DIX/Iv4/TC DIX/Iv6/UD DIX/QQ/VLAN/oE/MLS/MLS/MLS/MLS/Iv4/TC DIX/QQ/VLAN/Iv6-HBH-ROUT-FRAG/TC Theoretical limit of 10 GbE media (Inter-frame gap=12b - reample=8b) rotocol stacks do not fit (missing bars) Frame length (bytes) 25

26 Summary & Conclusions rocessor compute complex + integrated network I/O complex IFF high-computation performance (1) + high-chip density (2) + low-power (3) + flexibility (4) RXACC delivers (1) + (2) + (3) + (4) (1) erformance 15 Mfps, 20 Gb/s (at relaxed clock frequency 625 MHz) Saves hundreds of CU cycles per frame (2-3) Area and power efficiency 0.7 mm 2 (45 nm SOI), 0.15 W (4) Flexibility protocol permutations rogrammable parsing and processing Rule-based Can parse new emerging standards (e.g. SDN) Can inspect header and payload ragmatic approach w/ one code-set per application RXACC = TTA + pfsm = novel Application Specific rocessor Key enabler for integrated NICs The architecture has headroom to scale towards GbE 26

27 Rx Stack Accelerator for 10 GbE Integrated NIC Thank you IEEE Hot Interconnects 20, Santa Clara, CA, Aug , IBM Corporation