Reference Design: NVMe-oF JBOF

Transcription

1 Reference Design: NVMe-oF JBOF 1

2 Composable Infrastructure Two Target Architectures RNIC RNIC Driver NVMe Driver NVMe Fabric Driver NVMe SSDs Application NVMe Driver NVMe Fabric Driver RNIC Driver RNIC Ethernet Fabric Target Server x86 Infrastructure Host Server Fuji Bridge PCIe Switch NVMe SSDs Expensive, hot SW replaced by efficient, consolidated HW Target JBOF 2

3 NVMe-oF Bridge Architecture PCIe Root Complex (PRC) PCIe Abstraction Layer (AL) FW Approximately 150 FSMs running in parallel Doorbell TFC Main Memory (TMM) Data Pages SQs CQs HW Some simple, e.g. buffer management Rx DMA DMA Req Comp Capsule SQE CQE Int NVMe Engine (NE) µp Some quite complex: RoCE v1, v2 iwarp /TCP TCP NVMe NVMe-oF Rx Packet Buffer Rx Parser/Router Rx FIFO DMA Req Comp RDMA Engine (RE) Transport Engine (TE) MPU µp TSD Mem Tx Framer Tx FIFO Slow-path implemented in FW Ethernet MAC 4

4 NVMe-oF Bridge Architecture PCIe Root Complex (PRC) PCIe Abstraction Layer (AL) FW Approximately 150 FSMs running in parallel Some simple, e.g. buffer management Rx DMA Doorbell DMA Req Comp Capsule TFC Main Memory (TMM) Data Pages SQs CQs SQE CQE Int NVMe Engine (NE) µp HW 3.5% of die Some quite complex: RoCE v1, v2 iwarp /TCP TCP NVMe NVMe-oF Rx Packet Buffer Rx Parser/Router Rx FIFO DMA Req Comp RDMA Engine (RE) Transport Engine (TE) MPU µp TSD Mem Tx Framer Tx FIFO Slow-path implemented in FW Ethernet MAC 5

5 Purely SW Architecture PCIe Root Complex (PRC) PCIe Access Layer External DRAM DRAM I/O Control Rx FIFO Tx FIFO Ethernet MAC 6

6 NVMe-oF Options High-End x86 server-based Up to 3.3GHz 200W+ / 100Gb Mid-range Network Processors Up to 3.0GHz 20-35W / 100Gb Low-power, low-cost Dedicated hardware ~0.5GHz 7W / 100Gb PCIe Root Complex (PRC) PCIe Root Complex (PRC) PCIe Access Layer PCIe Abstraction Layer (AL) Doorbell TFC Main Memory (TMM) DRAM I/O Control Rx DMA Rx Packet Buffer DMA Req Comp Capsule DMA Req Comp SQs CQs Data Pages SQE CQE Int NVMe Engine (NE) RDMA Engine (RE) Transport Engine (TE) µ P µ P TSD Mem Rx Parser/Router Tx Framer Rx FIFO Tx FIFO Rx FIFO MPU Tx FIFO Ethernet MAC Ethernet MAC 7

7 A Spectrum of Options Myriad choices for NVMe-oF / Composable Infrastructure target deployments Storage Servers Network Processors Dedicated Bridge Versatility H M L Cost H M L Power H M L Footprint H M L 8

8 Fuji Kazan Networks 1 st -gen NVMe-oF Target ASIC Industry s fastest target bridge Ports: Ethernet: PCIe: Management: Single 100Gb, 2 x 25/50Gb Gen3 x16, Gen3 2x8 Via I2C, PCIe, or Ethernet RDMA supported: RoCEv1, RoCEv2, iwarp Lossy networks supported on all three Latency: Sub-500ns through ASIC Power: 7W nominal, 13W worst-case Package: 21mm BGA Pricing: Shared under NDA OPN: KN-103-A0 MOQ: 220 parts = Best-in-class feature 9

9 Onyx Fuji-based Adapter Lowest power NVMe-oF adapter available Plug-and-play solution for JBOFs Ports: Ethernet: PCIe: Availability Single 100Gb, 50Gb, 25Gb Dual 50Gb, 25Gb Single Gen3 x16 Dual Gen3 x8 Samples: Now sampling Production Q Half Height OPN: KN-ONYX1.0 MOQ: 1 board Half Length 10

10 Lowest-Cost NVMe-oF JBOF Ideal Solution for Composable Infrastructure Off-the-shelf SSDs Kazan Fuji bridges PCIe switch(es) PCIe midplane Enclosure Services inside PCIe Switch or in separate BMC 11

11 Performance 12

12 Fuji / Onyx Performance IOPS 2.8M per 100Gb link (~95% link saturation) 4kB random read I/Os Bandwidth 11.4GB/s per 100Gb link (~97% link saturation) 128kB sequential read I/Os Latency Single I/O: <5 usec adder to NVMe-oF round-trip (vs DAS) Multiple I/Os: Essentially no latency adder! 13

13 Loaded Latency Optane vs. Outstanding Commands Average I/O Latency 500 µsec J/1QD1 4J/4QD 16 8J/8QD 64 16J/16QD 256 Native Fuji 14

14 Incremental Latency Optane SSD Target Incremental Latency % - 4K Reads % Benchmark case Queue Depth Benchmark case Real-world No latency penalty!!! 15

15 Thank you!! For more information: Bryan Cowger Kazan Networks 1544 Eureka Rd, Suite 250 Roseville, CA