Multi-Host Sharing of NVMe Drives and GPUs Using PCIe Fabrics

Transcription

1 Multi-Host Sharing of NVMe Drives and s Using PCIe Fabrics Vincent Haché Principal Applications Engineer, Microsemi Corporation Santa Clara, CA 1

2 Introduction Increase in use of s and NVM in DC System designers need: Efficient resource deployment High-BW, low-latency interconnect Flexible, composable architectures There are restrictions in standard PCIe that present challenges for system design Santa Clara, CA 2

3 PCIe Hierarchy Restriction PCIe hierarchy is restrictive, making scale out challenging Host 2 Host 3 Santa Clara, CA 3

4 PCIe Hierarchy Restriction: Santa Clara, CA 4

5 PCIe Hierarchy Restriction: Host 2 Host 2 Santa Clara, CA 5

6 PCIe Hierarchy Restriction: Host 3 Host 3 DSP DSP DSP DSP DSP DSP DSP DSP DSP Santa Clara, CA 6

7 PCIe Hierarchy Restriction (continued) The multiple links required for transparent scale out complicates design and decreases efficiency Host 2 Host 3 Santa Clara, CA 7

8 PCIe Single Domain Restriction PCIe is single domain Unused s are stranded Complicated, non-standard NT drivers required for sharing PCIe Switch SSD PCIe Switch SSD PCIe Switch Host 2 Host 3 SSD Santa Clara, CA In use Unused 8

9 PCIe Fabrics for Scaling Fabric routing is proprietary, non-hierarchical Fabric links are shared among hosts FW-Virtualized Host Domains Host 2 Host 3 Translation Translation Translation SSD SSD SSD Fabric Domain Santa Clara, CA 9

10 PCIe Fabrics for Scaling (continued) FW running on embedded CPU virtualizes a simple switch compliant with the PCIe spec Host 2 Host 3 DSP SSD DSP SSD DSP SSD SSD SSD SSD Fabric Domain Santa Clara, CA 10

11 PCIe Fabrics for Scaling (continued) Embedded CPU handles the control plane, but data is routed directly by switch HW Control Plane Data Plane Host 2 Host 3 DSP SSD DSP SSD DSP SSD SSD SSD SSD Fabric Domain Santa Clara, CA 11

12 Device Sharing on a PCIe Fabric Unused devices can be dynamically assigned (no longer stranded) Low-latency, high-bw P2P within the rack Standard drivers to simplify system development Santa Clara, CA Fabric Switch SSD Fabric Switch SSD Fabric Switch SSD Host 2 Host 3 In use Unused 12

13 Multi-host Sharing of SSDs Fabric resources assigned by function SR-IOV: appears as multiple functions Host 2 Host 3 DSP SSD DSP SSD SSD DSP SSD SSD SSD SSD VF VF Virtual Configuration Space Santa Clara, CA Presents a standard NVM for in-box drivers 13

14 Demo: Multi-host Sharing of NVMe and s Dynamic partitioning of s and multi-host sharing of SR-IOV SSDs in real time Standard host drivers in Windows Server 2016 and Ubuntu Server LTS P2P transfers across the fabric CUDA P2P BW test and TensorFlow cifar10 image classification multi- training algorithm

15 Demo: Multi-host Sharing of NVMe and s (continued) Fabric Domain All s are assigned to to increase performance Multi-host sharing of SR- IOV NVM device SSD Host 2 VF VF Santa Clara, CA 15 Fabric managed through UART

16 Demo: Multi-host Sharing of NVMe and s (continued) Domain Virtualized as a Spec-compliant PCIe Switch NVM VF Appears as a Standard NVM Device Santa Clara, CA 16

17 Demo: Multi-host Sharing of NVMe and s (continued) CUDA P2P Bandwidth Santa Clara, CA Running Tensorflow Model 17

18 Demo: Multi-host Sharing of NVMe and s (continued) workload completes and s are released back into fabric pool Fabric Domain SSD Host 2 VF VF Santa Clara, CA 18

19 Demo: Multi-host Sharing of NVMe and s (continued) Spare s are assigned to Host 2 Fabric Domain SSD Host 2 VF VF Santa Clara, CA 19

20 Demo: Multi-host Sharing of NVMe and s (continued) Domain Virtualized as a Spec-compliant PCIe Switch NVM VF Appears as a Standard NVM Device Windows Host Still Running During Dynamic Reassignment Santa Clara, CA 20

21 Demo Multi-host Sharing of NVMe and s CUDA P2P Bandwidth PCIe spec-compliant host domain Simple management Standard drivers Dynamic reassignment appears as speccompliant surprise-plug Santa Clara, CA 21

22 Summary Microsemi s Switchtec Switches enable new architectures for next-gen solutions Benefits of PCIe fabrics with : Scalable, low-latency, cost-effective Simple Management (PCIe, UART, TWI, Ethernet) Multi-host sharing of SR-IOV NVMe devices Standard host drivers Santa Clara, CA Live Demo at Booth #213 22