NVM Express TM Management Interface

Transcription

1 Architected for Performance NVM Express TM Management Interface August 11, 2015 John Carroll, Storage Architect, Intel Peter Onufryk, Storage Director, Product Development, PMC-Sierra Austin Bolen, Storage Technologist, Dell

2 Agenda NVMe Management Interface Overview Definition Comparison to NVM Express Specification interface Benefits over in-band management To standardize or not to standardize NVMe-MI Usage A real world example Automated Remote Health Monitoring NVMe-MI Architecture NVM Subsystem, Port, Management Endpoint, Command Slot Overview of Features/Functionality NVMe Management Commands NVMe Admin Commands PCIe Commands Control Primitives VPD Standardization Status 2

3 NVMe Management Interface What is the NVMe Management Interface? A programming interface that allows out-of-band management of an NVMe Field Replaceable Unit (FRU) or an embedded NVMe NVM Subsystem 3

4 Field Replaceable Unit (FRU) FRU definition (Wikipedia): A circuit board, part or assembly that can be quickly and easily removed from a computer or other piece of electronic equipment, and replaced by the user or a technician without having to send the entire product or system to a repair facility. 4

5 Management Fundamentals What is meant by management? Four pillars of systems management: Inventory Configuration Monitoring Change Management Management operational times: Deployment (No OS) Pre-OS (e.g. UEFI/BIOS) Runtime Auxiliary Power Decommissioning 5

6 In-Band vs Out-of-Band Management Host Processor Host Operating System PCIe Root Port NVMe Driver PCIe Root Port Management Controller (BMC) BMC Operating System PCIe Root Port NVMe-MI Driver SMBus/I2C NVMe driver communicates to NVMe controllers over PCIe per NVMe Spec MC runs on its own OS on it own processor independent from host OS and driver PCIe Bus PCIe Port PCIe SSD PCIe VDM PCIe Bus NVM Subsystem SMBus/I2C SMBus/I2C Two OOB paths: PCIe VDM and SMBus PCIe VDMs are completely separate from in-band PCIe traffic though they share the same physical connection 6

7 In-band vs Out-of-Band Management Cont. In-Band Management Application Many host OSes to support (Windows, Linux, VMWare, etc.) Several different flavors/distros of each New revisions of OS and NVMe driver released over time Developing and maintaining a management application for every OS variant is resource/cost prohibitive Management features vary per OS Out-of-Band Management Application Develop management application in one operating environment Works the same across any host OS the user installs Works across no OS cases (pre-boot, deployment) 7

8 Why Standardize NVMe Storage Device Management? Reduces Cost and Broadens Adoption Allows OEMs to source storage devices from multiple suppliers Eliminates need for NVMe storage device suppliers to develop custom OEM specific management features Consistent Feature Set All storage devices that implement management implement a common baseline feature set Optional features are implemented in a consistent manner Industry Ecosystem Compliance tests / program Development tools 8

9 A Real World Example Automated Remote Health Monitoring The Problem: Datacenter with hundreds of servers Each servers consists of dozens of Field Replaceable Units Some number of FRUs fail weekly (or even daily) Manually discovering and resolving issues due to failed FRUs is prohibitively time consuming and expensive The Solution: Each server has a BMC to manage all FRUs Each BMC is connected to a network accessible via a remote management console BMC detects NVMe FRU failures using NVMe-MI and reports failures to a remote administrator 9

10 10 Remote Health Monitoring Management Infrastructure Management Controller NVMe SSDs Host Processors Memory DIMMs NICs Power Supplies Server Management Controller NVMe SSDs Host Processors Memory DIMMs NICs Power Supplies Server Management Controller NVMe SSDs Host Processors Memory DIMMs NICs Power Supplies Server Management Controller NVMe SSDs Host Processors Memory DIMMs NICs Power Supplies Server Management Controller NVMe SSDs Host Processors Memory DIMMs NICs Power Supplies Server Remote Mangement Console Network

11 Remote Health Monitoring Set up Alerts Enable alerts for NVMe health events 11

12 Remote Health Monitoring Detect Error Using NVMe-MI Server Server Server Server Server NICs NICs Power Supplies NICs Power Supplies NICs Management Controller Management Controller Management Controller Management Controller Management Memory DIMMs Power Supplies NICs Memory DIMMs Power Supplies Host Processors Memory DIMMs Power Supplies Controller NVMe SSDs Host Processors Memory DIMMs Host Processors Memory DIMMs NVMe SSDs Host Processors NVMe SSDs Host Processors NVMe SSDs NVMe SSDs Network Remote Mangement Console Management Controller issues NVM Subsystem Health Status Poll command to NVMe drive NVMe drive responds indicating a Critical Warning bit is set Management Controller then issues a Controller Health Status Poll command to the drive NVMe drive responds indicating a Reliability Degraded error occurred Management Controller sends notification 12

13 Remote Health Monitoring Receive Alert alert sent if a drive health event occurs 13

14 Remote Health Monitoring Check Event Log Check event log on remote system 14

15 Remote Health Monitoring Drives Overview Review drive table to see status of all drives in the system 15

16 Remote Health Monitoring Drive Detail Expand a drive s view to see its details 16

17 Management Controller GUI Export Log Files Export logs from the drive 17

18 Remote Health Monitoring Check Log File Check the log for failure details 18

19 Remote Health Monitoring Blink Drive LED Admin blinks the indicator LED for the failed drive Local Datacenter Technician finds and replaces faulty drive 19

20 Agenda NVMe Management Interface Overview Definition Comparison to NVM Express Specification interface Benefits over in-band management To standardize or not to standardize NVMe-MI Usage A real world example Automated Remote Health Monitoring NVMe-MI Architecture NVM Subsystem, Port, Management Endpoint, Command Slot Overview of Features/Functionality NVMe Management Commands NVMe Admin Commands PCIe Commands Control Primitives VPD Standardization Status 20

21 NVMe Architecture (review) NVM Subsystem - one or more controllers, one or more namespaces, one or more PCI Express ports, a non-volatile memory storage medium, and an interface between the controller(s) and non-volatile memory storage medium PCIe Port PCIe Port x PCIe Port y NVM Subsystem One Controller/Port NVMe Controller PCI Function 0 NVM Subsystem Two Controllers/Ports PCI Function 0 NVMe Controller NSID 1 NSID 2 PCI Function 0 NVMe Controller NSID 3 NSID 2 NSID 1 NSID 2 NS A NS C NS A NS B NS B 21

22 NVMe Field Replaceable Units with NVMe-MI PCIe Port PCIe Port 0 PCIe Port 1 NVMe Controller PCI Function 0 Controller Management Interface Management Endpoint NVMe Controller PCI Function 0 Controller Management Interface Management Endpoint NVMe Controller PCI Function 1 Controller Management Interface NVMe Controller PCI Function 0 Controller Management Interface Management Endpoint Management Endpoint SMBus/I2C NVM Subsystem NVM Subsystem An NVMe FRU consists of one and only one NVM Subsystem with One or more PCIe ports An optional SMBus/I2C interface One or more Management Endpoints 22

23 VPD Vital Product Data Vital Product Data typically available in a serial EEPROM NVMe-MI defined standard VPD contents including: Device Form factor Initial and peak power usage by power rail RefClk/SRIS capability and more PCIe Port 0 PCIe SSD PCIe Port 1 NVM Subsystem Serial EEPROM SMBus/I2C NVMe-MI makes VPD contents accessible out-of-band 23

24 NVMe-MI Defines the Protocol for Managing NVMe Leverage existing PCIe and SMBus MCTP defines the transport layer Refer to for more info on MCTP NVMe-MI is the protocol for applications to information Management Applications (e.g., Remote Console) Management Applications (e.g., Remote Console) Management Component Transport Protocol (MCTP) MCTP over SMBus/I2C Binding SMBus/I2C Management Controller (BMC or Host Processor) NVMe Management Interface MCTP over PCIe Binding PCIe VDM Application Layer Protocol Management Layer Applications (e.g., Remote Console) Transport Layer Physical Layer 24

25 Types of MCTP Messages NVMe-MI Message Request Message Command Message NVMe-MI Command NVMe Admin Command PCIe Command Control Primitive Response Message Success Error Other MCTP Messages (e.g., MCTP control) 25

26 Command Slots Each Management Endpoint has two Command Slots to service Command Messages Each Command Slot follows this state machine Response Message Transmitted or Abort Idle Start of Message Abort or Error Response Required or Resume Transmit More Processing Required Process Abort Receive Command Message Received 26

27 Command Slots Each Management Endpoint has two Command Slots to service Command Messages Each Command Slot follows this state machine Response Message Transmitted or Abort Idle Start of Message Abort or Error Response Required or Resume Transmit More Processing Required Process Abort Receive Command Message Received 27

28 Management Interface Command Set Discover Device Capabilities Monitor Health Status Modify Configuration Command Configuration Set Configuration Get Controller Health Status Poll NVM Subsystem Health Status Poll Read NVMe-MI Data Structure Reset VPD Read VPD Write Vendor Specific O/M Mandatory Mandatory Mandatory Mandatory Mandatory Mandatory Mandatory Mandatory Optional 28

29 NVMe Admin Commands NVMe-MI defines mechanism to send existing NVMe Admin Commands out-of-band Admin Commands target a controller in the NVM subsystem Command Get Features Get Log Page Identify Firmware Activate/Commit Firmware Image Download Format NVM Namespace Management Security Send Security Receive Set Features Vendor Specific O/M Mandatory Mandatory Mandatory Optional Optional Optional Optional Optional Optional Optional Optional 29

30 PCIe Commands PCIe Commands provide optional functionality to read and modify PCIe memory Command PCIe Configuration Read PCIe Configuration Write PCIe Memory Read PCIe Memory Write PCIe I/O Read PCIe I/O Write O/M Optional Optional Optional Optional Optional Optional 30

31 Control Primitives Control Primitives enable a Management Controller to detect and recover from errors Control Primitives fit into a single packet and do not require message assembly Control Primitive Pause Resume Abort Get State Replay O/M Mandatory Mandatory Mandatory Mandatory Mandatory 31

32 Summary NVMe-MI standardizes out of band management to discover and configure NVMe devices NVMe-MI 1.0 specification under member review will be published on NVMe site after ratification Join NVMe to shape the future of NVMe-MI 32

33 Architected for Performance