Intel Builder s Conference - NetApp

Transcription

1 Intel Builder s Conference - NetApp John Meneghini Data ONTAP NVMe-oF Target Architect Madhu Pai Data ONTAP NVMe-oF Transport Architect April 21, 2017 V NetApp, Inc. All rights reserved..

2 Introduction 1) Data ONTAP SAN Engineering FC & iscsi Transport engineering teams SCSI Target protocol engineering team QA and Host Interop teams 2) Active at T10, T11 and NVMexpress.org Many TPARs, TPs, and ECNs Many T10 and T11 Proposals 3) Adding support for NVMe-oF to Data ONTAP For more information see: 4) SPDK provides the following that NetApp wants leverage NVMe-oF virtual Target & Initiator protocol engine RDMA (and FC-NVMe) transports Libraries, unit tests, scripts and tools NetApp, Inc. All rights reserved.

3 What NetApp Wants To Use Current Future Storage Protocols iscsi Target vhost-scsi Target SCSI NVMe-oF* Target NVMe vhost-blk Target Object Integration Storage Services Block Device Abstraction (BDEV) NetApp bdev Virtual NVMe Linux Async IO Blob bdev Ceph RBD Libraries bdev json event copy conf nvmf trace log util RocksDB Ceph Drivers NVMe Devices NVMe-oF * Initiator NVMe * PCIe Driver Intel QuickData Technology Driver Core Application Framework 3

4 What Got My Attention Apl 2017: First SPDK Summit Jun 2016: NVMe-oF* Target Jan 2016: first external contributor Sept 2015: nvme driver on github 2013: spdk starts as INTEL internal project 4

5 NetApp Likes/Dislikes 1) SPDK Libraries, Modules and APIs (Likes) The BDAL API is what made SPDK work for NetApp NetApp likes the modularity and APIs in SPDK Improved modularity and APIs make it even better 2) DPDK and /usr/lib dependencies (Dislikes) The DPDK environment doesn t work for our application (Data ONTAP) Expand the SPDK EAL (env_dpdk) to abstract DPDK dependencies Abstract dependencies on Posix 3) Threading model (Dislikes) Would like a flexible, dynamic threading model supported by SPDK 4) Management plane (Dislikes) Would like to improve the NVMe-oF management APIs 5) Tired of chasing the tip of master, not knowing what will show up next NetApp, Inc. All rights reserved

6 NetApp s Vision for SPDK 1) Open source library that supports enterprise applications Well defined APIs that support a variety of use cases Support for multiple platforms and execution environments Don t even assume user space Enterprise class Reliability, Availability and Supportability 2) Community collaboration Better distribution lists (e.g. separate lists for code-reviews) Public bug reporting and code reviews Shared test environment and automation Feature branch development 3) Governance True open source project governed by community members Feature roadmaps and schedules NetApp, Inc. All rights reserved.

7 NetApp s Vision for SPDK - Simplified Functionality Development Effort Value-Add Proprietary Core Proprietary SPDK Shared Shared

8 Topics Agenda Platform Abstraction NVMe-oF Transport Improvements Support for NVM Protocol Features NVMe-oF Management APIs Enterprise Readiness with RAS NVMe-oF Target Threading Model NetApp, Inc. All rights reserved.

9 Platform Abstraction Improvements 1) Abstract dependencies on DPDK Improvements to env.h and env_dpdk/env.c Not all modules effected (e.g. vhost) 2) Abstract dependencies on Posix APIs and User libs Pthreads abstracted /usr/include, /usr/lib abstracted 3) Makefile improvements Better compile tool chain support Support for: armv8a, spaa2, thunderx, xgene1, power8 Support for different compilers Optional build targets Only build the libraries and applications I want C.f. dpdk/config NetApp, Inc. All rights reserved.

10 Agenda Platform Abstraction NVMe-oF Transport Improvements Support for NVM Protocol Features NVMe-oF Management APIs Enterprise Readiness with RAS NVMe-oF Target Threading Model NetApp, Inc. All rights reserved.

11 FC-NVMe and RDMA Transports 1) Interested in SPDK FC-NVMe and RDMA Transports 2) Changes to struct spdk_nvmf_transport expected FC-NVMe and RDMA differences Feature branch desired 3) Changes to SGL infrastructure expected Some applications require more than 2 SGL entries 4) RDMA Transport Improvements Verbs API in rdma.c to abstract user OFED libraries C.f. Linux NVMe-oF RDMA layering NetApp, Inc. All rights reserved.

12 Agenda Platform Abstraction NVMe-oF Transport Improvements Support for NVM Protocol features NVMe-oF Management APIs Enterprise Readiness with RAS NVMe-oF Target Threading Model NetApp, Inc. All rights reserved

13 NVM Protocol Improvements (Virtual Target) 1) Additional NVM Commands Abort and Identify command improvements Persistent Reservations Fuse Operations, Compare and Write 2) Additional Controller features Scaling Controllers, QPs and Namespaces Support for static Controllers Namespace sharing Namespace mapping Submission Queue flow WRR with Urgent Priority Arbitration 3) In-band Namespace Management NVMe Specification Improvements Name Space Management and Attach command support NetApp, Inc. All rights reserved

14 NVMe-oF Improvements Storage Storage Protocols NVM Protocol Virtual Direct NVMe-oF Target RDMA FC-NVME Objectives: Complete SPDK EAL Storage Services Block Device Abstraction Layer (BDAL) BDAL Extension Module Verbs API User Verbs FCT API FCT Add POSIX Abstractions Add FC-NVMe Transport Add FCT and Verbs API Develop NVM Protocol Hardware Drivers DPDK EAL + Posix Libraries SPDK EAL and POSIX abstractions Linux Kernel (RHEL 7, Debian 8) Hardware RNIC Driver HCA FC Driver UIO HBA Current OFED New NetApp, Inc. All rights reserved. NVMe-oF Initiator

15 Agenda Platform Abstraction NVMe-oF Transport Improvements Support for NVM Protocol Features NVMe-oF Management APIs Enterprise Readiness with RAS NVMe-oF Target Threading Model NetApp, Inc. All rights reserved

16 Management Plane improvements (OOB) 1) Support for Administratively configuring Subsystems and Controllers Add a Subsystem with no Subsystem Ports or Namespaces Start a Subsystem or stop a Subsystem (forces disconnect) Configure number of Controllers per Subsystem Add and remove Hosts from a Subsystem (dynamic discovery service) 2) Support for Administratively configuring Namespaces Out of band Namespace Management Mapping NVMe-oF Hosts to NVMe Namespaces (ACL) Support both private and shared Namespaces (ACL) Adding and removing Namespaces (AEN support) 3) Support for Administratively configuring Subsystem Ports Subsystem port online/offline (disconnect) Subsystem port add/remove Dynamic updates to Discovery Service NetApp, Inc. All rights reserved.

17 Agenda Platform Abstraction NVMe-oF Transport Improvements Support for NVM Protocol Features NVMe-oF Management APIs Enterprise Readiness with RAS NVMe-oF Target Threading Model NetApp, Inc. All rights reserved

18 SPDK and DPDK Reliability, Availability, Supportability 1) Improve Logging (log.c) Well defined APIs Implementation needs to be abstracted (e.g. with a constructor module) 2) Improve Tracing (trace.c) Well defined APIs Implementation needs to be abstracted 3) First Failure Detection and Capture (FFDC) Trigger and dump traces and logs on error or exception 4) Add performance counters and histograms Counters in the perf path Programmable, not always on 5) Better error handling Don t panic or abort() on error NetApp, Inc. All rights reserved

19 Agenda Environmental Abstraction NVMe-oF Transport Improvements Support for NVM Protocol Features NVMe-oF Management APIs Enterprise Readiness with RAS NVMe-oF Target Threading Model NetApp, Inc. All rights reserved

20 SPDK Threading Model 1) Hybrid Polling Implemented in the Linux 4.10 kernel 2) SPDK Libraries should be thread model independent Different applications will use different threading models 3) Changes to the NVMe-oF Target threading model Scaling threads in an NVMe-oF Target Subsystem More than 1 thread per Subsystem Dynamic thread association with Controllers Based upon dynamic controller creation Dynamic thread association with Queue Pairs As QPs scale, threads scale NetApp, Inc. All rights reserved

21 SPDK Threading Model (Limitations) 1) The number of threads are static. This results in wasted cores and cycles if we pre-provision the NVMe-oF threads. 2) Flexible thread model where there is a dynamic association of threads and their work. 3) The model of binding a subsystem to a thread does not scale in our environment. 4) The queuing architecture supported by the FC hardware does NOT tailor well for dynamic creation/deletion of queue pairs as done in RDMA NetApp, Inc. All rights reserved

22 SPDK Threading Model (Requirements) 1) Dynamic threading model that activates and quiesces threads. 2) Dynamically create an NVMe-oF Subsystem and associate it with a thread. 3) Break the subsystem IO traffic to thread affinity. 4) Support the same lockless semantics. 5) Enhancements should be compatible with future releases of SPDK NetApp, Inc. All rights reserved

23 SPDK Threading Model (Terminology/Extensions) 1) Hardware Queue Pair (HWQP) is the basic unit for polling The HWQP specifies a set of FC-NVMe queues that work together to provide the Send/Completion QP. In the new model, a thread would poll one or more HWQPs. An HWQP has affinity to a thread. 2) Subsystems would still have a thread affinity for the event handling. The thread owning the subsystem is the Master thread (for that subsystem) 3) IO and Admin queue pairs (IOQP/AQP) are spread across many HWQPs. 4) Threads polling the HWQPs are Poller threads. A Master thread could be a Poller thread (depending on the subsystem) NetApp, Inc. All rights reserved

24 HWQP Layout NetApp, Inc. All rights reserved

25 SPDK Threading Model (Operation) 1) All subsystem data owned by the Master thread. 2) Poller threads process IO by polling the HWQPs (and hence the IOQPs). 3) Master thread propagates a cache of needed data to the Poller threads that is needed during the lifecycle of an IO. 4) Poller threads route NVMe admin (and fabric) commands to the Master thread for processing. 5) Master thread co-ordinates out-of-band management commands and verifies that required caches in the Poller threads are set up correctly NetApp, Inc. All rights reserved

26 SPDK Threading Model (Operation) NetApp, Inc. All rights reserved

27 Thank You Questions? NetApp, Inc. All rights reserved.