Challenges of High-IOPS Enterprise-level NVMeoF-based All Flash Array From the Viewpoint of Software Vendor

Transcription

1 Challenges of High-IOPS Enterprise-level NVMeoF-based All Flash Array From the Viewpoint of Software Vendor Dr. Weafon Tsao R&D VP, AccelStor, Inc. Santa Clara, CA 1

2 Enterprise-level Storage System (ESS) Data Protection (DP) High Availability (HA) Snapshot Replication (Sync/Async) Thin-Provisioning Thin-Clone Deduplication Compression Disaster Recovery (DR) Data Reduction Santa Clara, CA 2

3 NVMeoF-based ESS Plain Storage System Computing Node Computing Node Computing Node Computing Node ESS Computing Node Computing Node NVMeoF ESS NVMeoF JBOF NVMeoF JBOF Computing Node Computing Node Computing Node Computing Node ESS ESS ESS ESS NVMeoF JBOF NVMeoF ESS Santa Clara, CA 3

4 Santa Clara, CA 4

5 Why NVMe and NVMf? Santa Clara, CA Good for Building High-IOPS AFA 5 The photo is designed by fanjianhua / Freepik

6 Dad! Dad! ~ an Interrupt Santa Clara, CA PROFESSOR Robert Kelly s kids invaded a live BBC News interview. 6

7 Photo is designed by David Shankbone, wikipedia NVMe & NVMf: Active DMA to Move Data Santa Clara, CA 7

8 Legacy & Thick Code Stack Designed by Raymond Henry, freepik Santa Clara, CA 8

9 Enjoy NVMeoF for Free! How you can get a home-made NVMeoF AFA? Santa Clara, CA 9 The photo is designed by mrsiraphol / Freepik

10 Global News Network 'AVING by Dooyoung Shin ( Commodity Hardware 10

11 RoCE NIC from Mellanox 3 Keys of HW Components for NVMeoF AFA Santa Clara, CA 11

12 Free Software Resources Install Linux Centos 7.2 or above for target-side Install Linux Centos 7.2 or above and upgrade kernel to 4.8 for initial-side Download SPDK: lock-free software Multiple modules in SPDK: nvme driver, nvmeof target, iscsi target. User-layer application Replace the role of the Linux driver for nvme Santa Clara, CA The picture is designed by jannoon028 / Freepik 12

13 Target Modes: Passthough vs. Linux AIO Passthrough Memory Linux AIO NVMe commands to a physical nvme device NVMeoF Target NVMe commands to a virtual nvme device NVMeoF Target AIO LUN driver NVMe driver Linux aio Linux Bio NVMe ssd Nvme ssd LVM, MD.. SATA, SAS ssd Santa Clara, CA 13

14 Test Topology for a NVMeoF AFA Target-side Server Initial-side Node 1 PCIe 0 100GbE RoCE Initial-side Node 2 QPI target-side server - Intel(R) Xeon(R) CPU E GHz - Dc p3600 x 24 - Supermicro - Mellanox connectx-4 Initial-side node - Intel(R) Xeon(R) CPU E GHz - Supermicro - Mellanox connectx-4 PCIe 1 NVMe X24 Santa Clara, CA 14

15 Performance on Random WRITE over Different Ranges Tested NVMe SSD: Intel DC P3600, 800GB Santa Clara, CA 15

16 Measured Maximum Random WRITE IOPS ReactorMask 0x0101 AcceptorCore 0 MaxQueuesPerSession 4 MaxQueueDepth 129 [Subsystem1A] NQN nqn io.spdk:cnode1 Core 0 Mode Direct Listen RDMA :4420 NVMe 0000:06: [Subsystem1B] NQN nqn io.spdk:cnode1b Core 8 Mode Direct Listen RDMA :4420 NVMe 0000:83:00.0 Target-side Node X1: Intel(R) Xeon(R) CPU E GHz X2 24 nvme drives, 12 for CPU 0 and 12 for CPU 1 Santa Clara, CA 16 Initial-side Node X2: Intel(R) Xeon(R) CPU E GHz X2

17 Measured Maximum Random READ IOPS Target-side Node X1: Intel(R) Xeon(R) CPU E GHz X2 24 nvme drives, 12 for CPU 0 and 12 for CPU 1 Santa Clara, CA 17 Initial-side Node X2: Intel(R) Xeon(R) CPU E GHz X2

18 Measured Latency under Random WRITE over NVMeoF Target-side Node X1: Intel(R) Xeon(R) CPU E GHz X2 8 nvme drives connecting to CPU 0 Initial-side Node X2: Intel(R) Xeon(R) CPU E GHz X2 Fio run on the initial host with 8 threads through nvmeof interface. IO depth is configured for each fio thread. Santa Clara, CA 18

19 Challenges and Solutions High IOPS with Enterprise Features Santa Clara, CA The photo is designed by jigsawstocker /19Freepik

20 Missed Basic Features Initial-side NVMf driver for popular os dist. Host (Initial-side) L U N L U N In-cap read response packet RSP DATA NVMeoF Runtime attaching LUN/ns to target All Flash Array (Target-side) SPDK NVMf target LUN across multiple drives: using namespace Santa Clara, CA 20 ns ns ns ns ns SSD 0 SSD 1 SSD 2 SSD 3

21 Trade-off between Advanced Features and Performance SPDK NVMeoF Target NVMe-> Linux AIO-> BIO LVM/MD HA Scale-up Thin-clone Thin-provisioning Data protection NVMe ssd Santa Clara, CA 21

22 Performance on 4KB Random WRITE IOPS Under Different Target Modes Santa Clara, CA 22

23 IO Amplification (AMP) Data Protect Mechanism (DPM) on High IOPS SSD Write-in-place RAID -> 2R+2W at least Solution: RAID-aware device? multi-copy based? Write Buffer: Additional dual log devices Additional effort on hw Faster than nvme? Nvdimm RoW RAID Screenshot of Linux tool: iostat 4KB random WRITE: tps: 4KB transation per second User-perceived IOPS: ~ 108K Actual paid IOPS: ~36K*12, or 432K Santa Clara, CA 23

24 Performance under Different RAID5 over NVMeoF 4KB Random WRITE over ¼ range of capacity, No cache before RAID. consisting of 24 NVMe SSDs where 12 SSDs form a raid group and each SSD provides Read 400K iops, write 200K iops. Santa Clara, CA Fio run on the initial host through nvmeof interface 24

25 Option 1 Where to Deploy DPM? Initial-side L U N Target-side RAID ns ns ns SSD 0 SSD 1 SSD 2 Initial-side Target-side NVMeoF ns SSD 0 Option 2 L U N RAID ns ns SSD 1 SSD 2 Santa Clara, CA 25

26 Benefit of Initial-side RAID 4.8M WRITE ~4.4M ~2.1M ~1.4M ~216K 5.6M (2.8M WRITE + 2.8M READ) measured inference Santa Clara, CA 4KB Random WRITE over ¼ range of capacity, No cache before RAID. consisting of 24 NVMe SSDs where 12 SSDs form a raid group and each SSD provide Read 400K iops, write 200K iops. Fio run on the initial host through nvmeof interface 26

27 Issues of Small WRITE on Thin-Provisioning/Snapshot/ThinClone Zero on First WRITE Overhead of Power-loss Prevention or Limitation of Single-Thread Programming Copy on First WRITE 12 intel nvme DC P3600 form a RAID0 group 80GB pool for 300GB thin-lun and snapshot Fio random 4KB, write range 30GB, 3 threads LVM: linux lvm, CoW-style, stripe = 12, default FlexiRemap: AccelStor FlexiRemap, RoW-style TP: thin-provisioning, TC: thin-clone Santa Clara, CA 27

28 Performance on TP/Snapshot/TC over NVMeoF w/o penalty of ZoFW, CoFW AccelStor Next: A patent-pending solution SSD Vendors are welcome to discuss with us. Santa Clara, CA 28

29 Santa Clara, CA 29

30 Thank You AccelStor, Inc. Booth: 132 Santa Clara, CA 30