NVMe Management Interface (NVMe-MI)

Transcription

1 Architected for Performance NVMe Management Interface (NVMe-MI) Peter Onufryk Microsemi Corp. NVMe-MI Workgroup Chair Austin Bolen Dell EMC NVMe-MI Workgroup Vice Chair

2 NVM Express, Inc Companies defining NVMe together Board of Directors 13 elected companies, stewards of the technology & driving processes Chair: Amber Huffman Technical Workgroup NVMe Base and NVMe Over Fabrics Chair: Amber Huffman Management Intf. Workgroup Out-of-band management over SMBus and PCIe VDM Chair: Peter Onufryk Vice Chair: Austin Bolen Marketing Workgroup NVMexpress.org, webcasts, tradeshows, social media, and press Co-Chairs: Janene Ellefson and Jonmichael Hands Interop (ICC) Workgroup Interop & Conformance Testing in collaboration with UNH-IOL Chair: Ryan Holmqvist 2

3 RASM (Reliability, Availability, Serviceability, Manageability) Reliability A measure of the reliability of systems and components. Better reliability increases availability and reduces replacement cost. Availability A measure of system uptime. Unreliable components, inability to manage, and inability to service reduce availability. RASM Manageability A measure of how easy it is to inventory systems/components, configure, perform updates, and monitor and report failures. Serviceability A measure of how easy it is to recover a system to full operation following a failure. Customers choose suppliers who provide the features that are important to them. Customers care about TCO (Total Cost of Ownership). Consequently, in the server space, MHz is not the only thing that's important: TCO is greatly affected by the RASM features of the servers. When server OEMs and users talk, their focus is RASM: Reliability, Availability, Serviceability, and Manageability. To a customer, RASM means dollars. Adding or improving on RASM reduces TCO. The cost of downtime is extremely high. According to IMEX Research*, the average cost of an unplanned outage runs into the hundreds of thousand of dollars. (Reference 2) Better RASM = Reduced TCO 3

4 Management Fundamentals Pillars of Systems Management Inventory Configuration Monitoring Change Management Management Operational Times Deployment (No OS) Pre-OS (e.g. UEFI/BIOS) Runtime Auxiliary Power Decommissioning 4

5 What is the NVMe Management Interface 1.0a? A programmable interface that allows out-of-band management of an NVMe Storage Device Field Replaceable Unit 5

6 NVM Express Roadmap Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 NVM Express (NVMe) NVMe 1.2 Nov 14 Namespace Management Controller Memory Buffer Host Memory Buffer Live Firmware Update NVMe May 16 NVMe NVMe May 17 Sanitize Streams Virtualization NVMe (next) IO Determinism Persistent Memory Region Persistent Event Log Multipathing NVMe Over Fabrics (NVMe-oF) NVMe-oF 1.0 May 16 Transport and protocol RDMA binding NVMe-oF (next) Enhanced Discovery In-band Authentication TCP Transport Binding NVMe Mgmt. Interface (NVMe-MI) NVMe-MI 1.0 Nov 15 Out-of-band management Device discovery Health & temp monitoring Firmware Update NVMe-MI 1.0a April 17 NVMe-MI 1.1 Enclosure Management In-band Mechanism Storage Device Extension Released NVMe specification Planned release * Subject to change 6

7 Benefits of NVMe-MI and Standardization Benefit OEM Drive Vendor Clear requirements and specification Industry standard compliance program Industry standard tools End User Ability to source NVMe-MI drives from multiple vendors Reduces need for drive vendors to develop proprietary management features Lower TCO over life of NVMe Storage Device Allows product differentiation 7

8 Sample End User Use Cases Use Case Benefit Inventorying Health monitoring Wear monitoring Temp. monitoring Power monitoring and configuration Perf. monitoring Configuring Change Mgmt. Asset management. Re-provisioning systems. Track quality of components. Identify bad drives for quick replacement. Replace drives nearing wear-out to avoid failure. Fan throttling reduces power, noise, and fan wear. Power throttling to save energy and cool system. Look for performance bottlenecks. Format drives for initial use. Crypto erase drives for re-provisioning or decommissioning. Update drive firmware for bug fixes and security patches. 8

9 Field Replaceable Unit (FRU) FRU Definition (Wikipedia) A circuit board, part or assembly that can be quickly and easily removed from a computer or other piece of electronic equipment, and replaced by the user or a technician without having to send the entire product or system to a repair facility. 9

10 NVMe Architecture (review) NVM Subsystem - one or more controllers, one or more namespaces, one or more PCI Express ports, a non-volatile memory storage medium, and an interface between the controller(s) and non-volatile memory storage medium PCIe Port PCIe Port PCIe Port NVM Subsystem with One Controller and One Port NVM Subsystem with Two Controllers and Two Ports 10

11 NVMe Storage Device NVM Storage Device One NVM Subsystem with one or more ports and an optional SMBus/I2C interface VPD Single Ported PCIe SSD Dual Ported PCIe SSD with SMBus/I2C 11

12 Vital Product Data (VPD) Utilizes IPMI Platform Management FRU Information Storage Definition with NVMe-MI extensions The VPD may be accessed using two methods NVMe-MI commands over MCTP SMBus/I2C interface using I2C operations as defined by IMPI Platform Management FRU Information Storage Definition Common Header VPD Elements Product Info Area (optional) NVMe MultiRecord Area NVMe PCIe Port MultiRecord Area Internal Use Area (optional) Chassis Info Area (optional) Board Info Area (optional) 12

13 Out-of-Band Management and NVMe-MI Out-of-Band Management Management that operates with hardware resources and components that are independent of the operation system control NVMe Out-of-Band Management Interfaces SMBus/I2C PCIe Vendor Defined Messages (VDM) IPMI FRU Data (VPD) accessed over SMBus/I2C PCIe Bus Host Processor Host Operating System PCIe Root Port PCIe Port Application NVMe Driver PCIe Root Port PCIe VDM PCIe Bus Management Controller (BMC) BMC Operating System Application BMC Operating System NVMe-MI Driver PCIe Port SMBus/I2C SMBus/I2C SMBus/I2C NVMe NVM Subsystem 13

14 NVMe-MI Protocol Layering Management Applications (e.g., Remote Console) Management Applications (e.g., Remote Console) MCTP over SMBus/I2C Binding Management Controller (BMC) NVMe Management Interface Management Component Transport Protocol (MCTP) MCTP over PCIe VDM Binding Application Layer Protocol Management Layer Applications (e.g., Remote Console) Transport Layer SMBus/I2C PCIe Physical Layer PCIe SSD 14

15 SMBus/I2C Topologies and Addressing Other SMBus Device Other SMBus Device Mux Control Shared SMBus/I2C Other SMBus Device Other SMBus Device Management Controller NVMe SSD Management Controller NVMe SSD SMBus/I2C NVMe SSD Requires Unique SMBus/I2C addresses Segmented SMBus/I2C SMBus/I2C SMBus/I2C Mux NVMe SSD NVMe SSD NVMe SSD SMBus/I2C During Auxiliary Power (if supported) I2C serial EEPROM read/write access at default SMBus/I2C address 0xA6, but may be modified using ARP During Main Power MCTP Endpoint at default SMBus/I2C address 0x3A, but may be modified using ARP I2C serial EEPROM read/write access If auxiliary power was provided, then SMBus/I2C address shall be maintained if modified using ARP; otherwise, the default address is 0xA6 SMBus/I2C address may be modified using ARP Repeated SMBus/I2C Addresses Supported 15

16 NVMe-MI Message NVMe-MI Message NVMe-MI MCTP Message Assembly 16

17 NVMe-MI Message Taxonomy NVMe-MI Message Request Message Response Message Command Message Control Primitive Success Response Error Response NVMe-MI Command NVMe Admin Command PCIe Command 17

18 NVMe Storage Device Single Ported PCIe SSD Dual Ported PCIe SSD with SMBus/I2C PCIe Port PCIe Port 0 PCIe Port 1 NVMe Controller PCI Function 0 Controller Management Interface Management Endpoint NVMe Controller PCI Function 0 Controller Management Interface Management Endpoint NVMe Controller PCI Function 1 Controller Management Interface NVMe Controller PCI Function 0 Controller Management Interface Management Endpoint Management Endpoint SMBus/I2C NVM Subsystem NVM Subsystem 18

19 Command Servicing State Diagram for Command Slots 19

20 Control Primitives Control Primitive Pause Resume Abort Get State Replay Description Suspend transmission Resume paused transmission Reinitialize command slot Retrieve state (e.g., errors) associated with a command slot Retransmit response message for last command message processed in a command slot 20

21 NVMe-MI 1.0a Command Set Overview Command Type Command Command Type Command NVMe Management Interface Specific Commands PCIe Command Read NVMe-MI Data Structure NVM Subsystem Health Status Poll Controller Health Status Poll Configuration Get Configuration Set VPD Read VPD Write Reset Vendor Specific PCIe Configuration Read PCIe Configuration write PCIe I/O Read PCIe I/O Write PCIe Memory Read NVMe Admin Commands Firmware Activate/Commit Firmware Image Download Format NVM Get Features Get Log Page Identify Namespace Management Namespace Attachment Security Send Security Receive Set Features Vendor Specific PCIe Memory Write Vendor Specific 21

22 NVMe Management Interface Specific Commands Command O/M * Description Read NVMe-MI Data Structure NVM Subsystem Health Status Poll Controller Health Status Poll Configuration Get M M M M Configuration Set M Set NVMe-MI configuration parameter VPD Read M Read Vital Product Data (VPD) VPD Write M Write Vital Product Data (VPD) Reset O Reset NVM Subsystem Retrieve information about the NVM Subsystem, Management Endpoint, or NVMe Controllers NVM Subsystem Information Port Information Controller Information Optional Commands Supported Used to efficiently determine changes in health status attributes associated with the NVM Subsystem (e.g., Unrecoverable error, reset required, PCIe status, Controller SMART / Health Information, composite temperature, composite, and controller status) Efficiently determines changes in health status attributes associated with one or more Controllers in the NVM Subsystem Get NVMe-MI configuration parameter (e.g., SMBus/I2C frequency and MCTP transmission unit size) * O = Optional, M=Mandatory 22

23 NVMe Admin Commands Command O/M * Description Firmware Activate/Commit Verifies that a valid firmware image has been downloaded and commits that O revision to a specific firmware slot Firmware Image Download Download all of a portion of a firmware image for a future update to the O controller Format NVM O Low level format of the NVM media associated with one or more Namespaces Get Features M Get NVMe configuration parameter Set Features O Set NVMe configuration parameter Get Log Page M Retrieve NVMe log page Identify M Retrieve information about the Controllers, Namespaces, or NVM Subsystem Namespace Management O Create or delete a Namespace Namespace Attachment O Attach or detach a Namespace from a Controller Security Send O Transfer command/data associated with security protocol Security Receive O Transfer command/data associated with security protocol * O = Optional, M=Mandatory 23

24 PCIe Commands Command O/M * Description PCIe Configuration Read O Read PCI Express configuration space PCIe Configuration Write O Write PCI Express configuration space PCIe I/O Read O Read PCI Express I/O space PCIe I/O Write O Write PCI Express I/O space PCIe Memory Read O Read PCI Express memory space (BAR memory & MMIO) PCIe Memory Write O Write PCI Express memory space (BAR memory & MMIO) * O = Optional, M=Mandatory 24

25 NVMe-MI Operational Times Power States Operations Supported During Power States 25

26 New Features Targeted for NVMe-MI 1.1 In-Band NVMe-MI Enclosure Management NVMe Storage Device Enhancement 26

27 Out-of-Band Management and NVMe-MI Out-of-Band Management Management that operates with hardware resources and components that are independent of the operation system control NVMe Out-of-Band Management Interfaces SMBus/I2C PCIe Vendor Defined Messages (VDM) IPMI FRU Data (VPD) accessed over SMBus/I2C PCIe Bus Host Processor Host Operating System PCIe Root Port PCIe Port Application NVMe Driver PCIe Root Port PCIe VDM PCIe Bus Management Controller (BMC) BMC Operating System Application BMC Operating System NVMe-MI Driver PCIe Port SMBus/I2C SMBus/I2C SMBus/I2C NVMe NVM Subsystem 27

28 In-Band Management and NVMe-MI Host Processor Host Operating System Application NVMe Driver Management Controller (BMC) BMC Operating System Application BMC Operating System NVMe-MI Driver In-band mechanism allows application to tunnel NVMe-MI commands through NVMe driver Two new NVMe Admin commands NVMe-MI Send NVMe-MI Receive PCIe Root Port PCIe Bus PCIe Port PCIe Root Port PCIe VDM PCIe Bus NVMe NVM Subsystem PCIe Port SMBus/I2C SMBus/I2C SMBus/I2C Benefits Provides management capabilities not available in-band via NVMe commands Efficient NVM Subsystem health status reporting Ability to manage NVMe at a FRU level Vital Product Data (VPD) access Enclosure management 28

29 Example Enclosure... NVMe Enclosure NVM Subsystem NVMe Controller Mgmt. Ep. Cntrl. Mgmt Intf. Enclosure Services Process Power Supplies Cooling Objects Temp. Sensors Other Objects Slot Slot Slot Slot NVMe Storage Device NVMe Storage Device NVMe Storage Device... NVMe Storage Device 29

30 Enclosure Management Native PCIe Enclosure Management (NPEM) Transport specific basic enclosure management Submitted to the PCI-SIG Protocol Workgroup (PWG) on behalf of the NVMe Management Interface Workgroup Approved by PCI-SIG on August 10, 2017 SES Based Enclosure Management Technical proposal being developed in NVMe-MI workgroup Comprehensive enclosure management 30

31 SES Based Enclosure Management SCSI Enclosure Services (SES) is a standard developed by T10 for management of enclosures using the SCSI architecture While the NVMe and SCSI architectures differ, the elements of an enclosure and the capabilities required to manage these elements are the same Example enclosure elements: power supplies, fans, display or indicators, locks, temperature sensors, current sensors, voltage sensors, and ports NVMe-MI leverages SES for enclosure management SES manages the elements of an enclosure using control and status diagnostic pages transferred using SCSI commands (SCSI SEND DIAGNOSTIC & SCSI RECEIVE DIAGNOSTIC RESULTS) NVMe-MI uses these same control and status diagnostic pages, but transfers them using the SES Send and SES Receive commands 31

32 Enclosure Management Protocol Layering Legacy SCSI Host Software NVMe Host Software Management Controller SCSI Translation PCIe / Fabric NVMe Controller with In-Band NVMe-MI Support NVMe-MI SES Send & SES Receive Commands SCSI Enclosure Services (SES) 32

33 NVMe-MI 1.0a NVMe Storage Device NVM Storage Device One NVM Subsystem with one or more ports and an optional SMBus/I2C interface VPD Single Ported PCIe SSD Dual Ported PCIe SSD with SMBus/I2C 33

34 NVMe Storage Device with Multiple NVM Subsystems M.2 Carrier Board from Amfeltec ANA Carrier Board from Facebook PCIe SSD PCIe Switch NVM Subsystem NVM Subsystem NVM Subsystem NVM Subsystem NVM Subsystem NVM Subsystem PCIe SSD 34

35 Multiple NVM Subsystems on an NVMe Storage Device and I2C/SMBus Topologies Host Host ARP Subsystem 1 E8h 3Ah Subsystem 1 VPD A0h optional ARP ARP ARP Subsystem 2 Subsystem 3 Subsystem 4 A0h VPD required Mux 3Ah 3Ah 3Ah Subsystem 2 Subsystem 3 Subsystem 4 Shared SMBus/I2C Segmented SMBus/I2C 35

36 NVMe-MI Ecosystem Commercial test equipment and conformance tests exist for NVMe-MI NVMe-MI 1.0a compliance testing program has been developed Compliance testing started in the May 2017 NVMe Plugfest conducted by the University of New Hampshire Interoperability Laboratory (UNH-IOL) 6 devices have passed compliance testing and are on the NVMe-MI Integrators List Servers are shipping that support NVMe-MI 36

37 Summary NVMe-MI 1.0a has been released Focused on managing NVMe Storage Devices (e.g., SSDs) SSDs and systems are shipping that support NVMe-MI 1.0a NVMe-MI 1.1 is nearing completion Technical work is scheduled for completion this year and a ratified specification is expected in Q1 18 Key new features in NVMe-MI 1.1 In-band NVMe-MI Enclosure Management NVMe Storage Device Enhancements 37

38 References 1. NVMe/NVMe-MI RASM RASM Manageability Reliability Serviceability Availability

39 Don t Miss the Next Webcast! Join us to learn about the evolution of the NVMe storage protocol and what s in store for its future, in 2018 and beyond in our next webcast titled: The Evolution and Future of NVMe Tuesday, December 19 th at 9:00am PT / 12:00pm ET. David Allen, NVMe Board Member and Seagate's Senior Director of Marketing Dr. J Metz, Board Member, and R&D Engineer for the Office of the CTO for Cisco 39

40 Architected for Performance Questions? Visit for more information on NVM Express technology Follow us: Twitter: LinkedIn: Facebook:

41 Architected for Performance