Distributed Block Storage using NVMe-over- Fabric

Transcription

1 Distributed Block Storage using -over- Fabric Sujoy Sen, Senior Principal Engineer Mohan J Kumar, Fellow Intel 2018 Storage Developer Conference. Intel Corporation. All Rights Reserved. 1 Acknowledgements: Scott D Peterson, Reddy Chagam

2 Introduction +-of+distributed Scale-Out Storage (DSS) > Element of {, -of, DSS} Overview of -of and DSS Issues with DSS clients Solution Options Proposed Concept Summary 2018 Storage Developer Conference. Intel Corporation. All Rights Reserved. 2

3 -over-fabric Surface Drives to remote systems Host #1 Host #2 Host #3 Appears as drive/namespace to remote hosts -of virtual namespaces Low latency, efficient and high performance 1:1 mapping between physical and virtual namespace Low footprint initiator (user or kernel mode) Industry standard interface () and fabric protocol Widespread industry adoption and eco-system support #1 # Storage Developer Conference. Intel Corporation. All Rights Reserved. 3

4 Distributed Scale Out Storage Block Storage provided by a cluster of cooperating nodes Host #1 Host #2 Host #3 Block device distributed across nodes for HA and performance Block Block Block Block Capacity and Performance can be scaled by adding storage nodes Solution specific client (host) access module and interface Solution specific communication protocol Open Source and commercial offerings #1 #2 # Storage Developer Conference. Intel Corporation. All Rights Reserved. 4

5 Impact of solution-specific storage client Requires drivers for every operating system and hypervisor Requires solution-specific management Host is strongly coupled with the storage service Adds lifecycle management complexity Storage cluster attack surface extended to host May have significant resource footprint at the host Takes away valuable resources from applications Challenging to embed in Infrastructure Accelerators 2018 Storage Developer Conference. Intel Corporation. All Rights Reserved. 5

6 Gateways Host #1 Host #2 Host #3 Standards based client replaces custom client iscsi, -of etc. Smaller client footprint Decoupled from the storage service cluster Lower Performance Gateway Node Adds latency due to an extra hop/node Gateway can become a bottleneck #1 #2 #3 Management Complexity and increased cost 2018 Storage Developer Conference. Intel Corporation. All Rights Reserved. 6

7 Multiple Gateways Add multiple gateways Performance Alleviates gateway bottleneck Extra-hop Latency Various assignment policies Random Dynamic Load balanced More management complexity Higher cost Host #1 Host #2 Host #3 Gateway Node #1 Gateway Node #2 Gateway Node # Storage Developer Conference. Intel Corporation. All Rights Reserved. 7

8 Integrated Gateways Integrate gateways into the storage node Reduces cost, some reduction in management complexity Performance Extra-hop latency due to I/O forwarding Gateway bottleneck adds CPU pressure to storage node Host #1 Host #2 Host #3 Gateway Gateway Gateway #1 #2 # Storage Developer Conference. Intel Corporation. All Rights Reserved. 8

9 Distributed Integrated Gateways Natively distribute volume to the correct storage service node Host #1 Host #2 Host #3 Requires -of volume to be mapped to Target Tuples Requires -of volumes to support extensible placement mechanisms #1 #2 # Storage Developer Conference. Intel Corporation. All Rights Reserved. 9

10 Proposal: Distributed Self-learning -of Introduce the concepts of Redirector Host Application Hint Table Start LBA, Length, Target Hints Initiator redirector directs I/O to the correct -of target based on the hint table Redirector -of Initiator Hint Lookup Hint Update Hint Lookup Target Redirector forwards I/O to the correct storage node and propagates hints backward and forward Allows fully decoupled and legacy clients Initiator redirector updates local hint table based on received hints about a volume -of Target Redirector I/O1 -of Target Redirector Eventually learns about the volume s distribution across nodes Storage Service Storage Service Storage Developer Conference. Intel Corporation. All Rights Reserved. 10

11 Hints The hints allows flexible mapping of LBA to nodes Algorithmic or specific maps can be supported Simple Hint Start LBA Length Read/Write Target Extent List Simple Hints Striping Hints Hashing Hints Striping Hint Start/End LBA Stripe Size Number of Extents Target Extent List Allows -of to support storage services ranging from pair-wise HA to massive scale out Hashing Hint Object Size Object Name Fmt Hash Function Hash Table Page 2018 Storage Developer Conference. Intel Corporation. All Rights Reserved. 11

12 -of Changes Re-use existing -of elements Log Pages for Hints Namespace specific log pages Log Page must be consistent during read operation AER for Hint notification Redirector Capability Discovery Allow Initiators to discover Redirector capable Targets Use Supported Capabilities in GetFeatures command 2018 Storage Developer Conference. Intel Corporation. All Rights Reserved. 12

13 Summary Host #1 Host #2 Host #3 Distributed Scale-out Storage can benefit from standards-based host interfaces is the ideal host interface for scale-out storage -of can be naturally extended to support scale-out storage DSS DSS+ -of GW -of -of -of #1 DSS + Multiple -of GW DSS+ Distributed SL -of Distributed Storage Service Standard Host Interface #2 Gateway Availability #3 Gateway Performance Bottleneck Low Latency 2018 Storage Developer Conference. Intel Corporation. All Rights Reserved. 13