Discovery Tool For Generation 3 Sas Controllers

Transcription

1 Discovery Tool For Generation 3 Sas Controllers Shriya Kulkarni Electronics Design Technology National Institute of Electronics and Information Technology, Kozhikode, Kerala, India Prashant Yendigeri System Applications Engineer, SCD LSI Corporation R&D, Bangalore, Karnataka, India Abstract- The SAS Controller is connected to end device (SAS/SATA) directly or through expanders. Discovery process is method of detecting all the end devices and Expanders connected to the host. It includes determining what changes have occurred when devices are added or removed from the topology, reporting any configuration errors in the topology to host and doing analysis on the topology to correctly setup port groups.gen 3 controllers implement Fast Path I/O Block which is used for sending I/O request messages to disks directly without firmware involvement unlike Gen1 and Gen2. Using Discovery Process, in a huge topology, where expanders are interconnected to form wide network, by chance if any one device fails (expander or hard disk drive) due to hardware failure or path breaks, then has to be informed to host as soon as possible otherwise it is possible that the host keeps on sending the I/O requests to that lost device or devices connected to expander in the topology if device lost is Expander. Normally, the firmware, using discovery process checks in circular fashion to find out which device is missing and then notifies to the host. This requires a good amount of time to complete in huge topologies, instead, using the new technique, tool could find out that particular device directly in real time using thread functions (running in host) without following the circular fashion. Keywords Gen 3 SAS (Serial Attached SCSI) Controllers, Expanders, Discovery process, SCSI (Small Computer System Interface, SATA (Serial ATA) Drive, SMP (Serial Management Protocol),STP (SATA tunneling Protocol),SSP (Serial SCSI Protocol) I.INTRODUCTION SCSI (Small Computer System Interface) is a set of ANSI standard electronic interfaces that allow computer to connect peripheral hardware such as disk drives, printers, and scanners faster and more flexibly than previous interfaces. So, it is possible to connect a new peripheral device to an existing computer with no hardware changes, the only thing is we need to write I/O driver. It also supports device independence. It means for host all magnetic disks look alike, all printers look alike. So the system does not need to modify when we replace a device with a different manufacture s device. Serial-attached SCSI (SAS) is a point to point serial protocol that moves data to and from the computer storage devices. The initiator (host, controllers) communicates with SAS device (peripheral devices, end devices) using SAS protocol. SAS is a point to point bus and attaches two devices together. The connection points are referred as ports and every device will have at least one, while expanders and HBAs will have several. A port can consist of just one phy or it may implement several. A Port that implements one phy is called Narrow Port and a port that implements several is called a Wide Port. A physical link ties two Phys to each other and consists of one differential pair of wires for a transmit path and another pair for receive path for a connection. The components known as Serial Attached SCSI Expanders (SAS Expanders) facilitate communication between SAS controller and large numbers of SAS devices. Expanders contain two or more external expander-ports. Expander devices contain expander ports and one SMP port. An expander port contains one or more Phys. An expander device shares its Phys with the SAS device(s) contained within the expander device set. A. SAS Generations- First Generation SAS supports the data rate of 3Gbps. Second-generation SAS (SAS-2) doubles the physical link rate to 6.0 Gbps. SAS-2 added self-configuring expanders. SAS-2 includes zoning capabilities to improve resource deployment flexibility, security, and data traffic management. SAS-2 maintains backward compatibility with SAS-1. Special Issue - IDEAS ISSN: X

2 SAS-2 devices (initiators, targets, or expanders) can support more than one communication speed. If any two linked devices support multiple speeds, the devices use the highest mutually supportable speed. The linked devices determine that speed during a speed negotiation process at start up. A sequential series of speed negotiation windows (SNW) characterizes this process. In SNW-1 and SNW-2, linked devices test established combinations of SAS speed(s), transmission amplitude, slew rate, de-emphasis, and spread spectrum clocking (SSC). In SNW-3, the linked devices negotiate link speed and SSC settings. Unlike SAS-1, SAS-2 allows for training of the transceiver mechanism (PHY) and for exchanging parameters. After SNW-3 has negotiated the speed and settings, a trainingspeed negotiation window (Train-SNW) tests the fastest mutually supported speed. Generation 3 SAS (Fig 1) supports data rate of 12Gbps.It also supports Fast Path Engine Block which is not present in the earlier generations. This Fast Path IO Engine handles all start and completion of successful IOs autonomously from firmware, thus allowing for improved IO throughput. In the event of an error or unhandled hardware condition, firmware support will be required to cleanup and restart or complete failed IOs. Additionally the fast path functionality requires some initial setup of the Fast Path IO Engine device structures during discovery to allow for fast path usage. It should be noted that the Fast Path IO Engine is a fully separate block of hardware from the SAS Core, so the register/interface to this block will differ from that of SAS Core. B. Sas Devices- There are three types of SAS devices: initiators, targets, and expanders (Figure 1 & 2). Initiator devices include host bus adaptors (HBAs) and controllers. The initiators attaches to one or more targets forming a SAS domain. Target devices include SAS hard disk drives (HDDs) or solid state drives (SSDs), SATA HDDs or SSDs, and SAS tape drives. Using expanders (low-cost, high-speed switches), you can increase the number of external ports and also the number of end devices connected to it. Figure 1.SAS Devices SAS initiators have multiple ports for connecting to internal and external targets. Each initiator port can have a single physical link (a narrow port) or two, four, or eight physical links (a wide port). You can connect SAS initiator ports to separate domains for fail-over redundancy. The targets attached to an initiator to create a larger SAS domain. However SATA drives have a single narrow port. We can have SAS and SATA devices in a single domain. The size of the expander s routing tables determines how many initiators and targets you can have in a domain. Expanders connect initiators, targets, and other expanders. They receive commands and data in one port and route them to another port based on the SAS address of the target. Expanders use three routing methods: direct, table, and subtractive. Direct routing forwards the commands and data to targets directly attached to the expander. Table routing forwards the commands and data to another expander. When an expander receives a SAS address that it does not recognize, it uses subtractive routing to forward the commands and data to another expander. Each routing method uses routing tables that are maintained in each expander. The expander creates the routing table during the discovery process.. SAS drives (both enterprise-class and midline) have two narrow ports. SAS drives use the same electrical and physical connection interface as SATA drives. SAS drives are highly reliable compared to SATA drives because of two ports. If one port fails then data can be transferred using other port unlike SATA drive. Hence SAS drives are mainly used in mission critical applications and SATA drive in non-mission critical applications. Special Issue - IDEAS ISSN: X

3 Figure 2.Topology The process by which the initiators in SAS network find out which devices are accessible in the system is called Discovery process. There can be as many as 16,384 addresses in SAS topology. Expanders play a big role in the process because they can be cascaded into sets which allow scaling of the topology. The expanders themselves may or may not take a active role in the discovery process, depending on whether they were designed to be selfconfiguring. Normally it falls to initiators to go through the steps, working with the expanders to which it has access, to check all possible paths for the addresses. Once that is done and the initiator has constructs its own address map, so that in future it can access this table for sending messages. II.SAS ARCHITECTURE It s a layered architecture facilitates a modular design (Fig 3). The layers communicate with each other by passing requests, indications, confirmations and other messages to each other. The Application Layer supports three protocols, SSP (Serial SCSI Protocol), SMP (Serial Management Protocol) and STP(SATA Tunneled Protocol).SMP is required for SAS initiator because that protocol is used to execute the Discovery Process. To access SAS drives, SSP support is needed and STP is used to access SATA drives. The Application Layer passes the commands to the Transport Layer, informing it about the packet protocol to be used and the type of command or response to send. The Application Layer at the top sends request to the SAS HBA Driver, which communicates the request to the HBA s register interface. Next, a port is selected based on the destination address and the request is forwarded to the Transport Layer of that port. The Transport Layer also supports these three protocols. The responsibilities include passing requests and confirmation between adjacent layers as well as handling some Link Layer errors. The Port Layer is limited only to SAS devices as SATA devices wont implement wide ports and expanders sorts out which Phy to use. The transport layer could have many threads in progress but frames get pooled together in port layer according to destination address. The Port Layer arranges for frame transmission on the link by either using existing connection or selecting a new Phy. The Link Layer is between Phy Layer and Port Layer. It is responsible for sending and receiving Address frames, sending and receiving identification sequence, Clock compensation, and management of primitives. The Phy Layer is responsible for 8b/10b encoding, generation of Phy reset sequence and also generation of OOB signaling.sas Physical Layer defines the bit level implementation between SAS transmitter and receiver. Special Issue - IDEAS ISSN: X

4 SMP STP SSP SAS HBA DRIVER Application Layer PO RT PO RT PORT TRANSPORT LAYER SAS HBA SMP SSP STP PORT LAYER phy LINK LAYER PHY LAYER PHYSICA L LAYER Figure 3 SAS Architecture III.GEN 3 SAS CONTROLLER The Gen 3 Controller lies between the Host and SAS/SATA topology including Expanders (Figure 4). The controller resides on the Host Bus Adapter (HBA) along with other peripherals. The controller firmware part consists of Controller Initialization Boot Loader MPT initiator/target IO Processor SAS Protocol Layer Controller Initialization and Boot Loader are mainly responsible for preparing the controller for executing the runtime firmware. The run-time firmware consists of the Fusion-MPT IT IOP and SAS protocol layer subsystems. These subsystems will complete the initialization of the controller hardware, enumerate and initialize the SAS topology, and will process MPI requests from the host system(s). NVDATA contains the default values for the MPI configuration pages. It is primarily used to customize firmware behavior for a particular system or host bus adapter implementation without requiring unique firmware. The Fusion-MPT Message Passing Interface Specification not only defines a hardware interface to the controller, it also specifies the communication protocols used by the host and controller along with the content of the MPI messages exchanged. The SAS protocol layer interface, also known as the PLI, is a firmware interface between the IO processor and SAS protocol layer subsystems. This design allows for third party controller firmware configurations that don t use the Fusion-MPT IT IOP subsystem. Instead, they can use the SAS protocol layer subsystem firmware along with their own IO processor and boot loader firmware. Special Issue - IDEAS ISSN: X

5 Therefore, in order to minimize incompatibilities between the SAS protocol layer subsystem and the various custom IO processor firmware implementations, the PLI is a carefully controlled and specified interface. PL or PL firmware is the SAS/SATA protocol layer firmware subsystem that provides an abstracted view of the SAS hardware. PL is responsible for configuring and controlling the hardware, the PL must also maintain detailed knowledge of all devices present in the SAS topology, as well as numerous internal settings related to external devices and the numerous features supported through the PL by the SAS controllers. Discovery is responsible for detecting the contents and properties of the SAS topology present in the current system configuration. This includes determining what changes have occurred when devices are added or removed from the topology, reporting any configuration errors in the topology to host, and doing analysis on the topology to correctly setup port groups, also includes detecting any protocol violations by the devices at any point of time, be it during discovery or during the course of data transfer. In the absence of self-configuring expanders, Discovery also performs expander route table configuration. The primary subsystems Discovery interacts with are the Device Manager, and SMP Engine. The Device Manager is called to handle any addition, deletion or change to device properties. In the first generation (GEN-1), this functionality was all contained within Discovery. Starting with GEN-2 and continuing in GEN-3, this functionality has been broken out into a separate subsystem. Additionally, this Device Manager interaction prevents Discovery from having any direct interaction with the SATA initialization process. The SMP Engine is called to transmit any of the SMP requests Discovery uses for topology analysis. Return calls from the SMP engine are received whenever a Discovery requested SMP completes. Application Layer HOST FIRMWARE 1. Controller Initialization 2. Boot Loader 3. IOP Subsystem 4. PL Subsystem SAS Protocol SAS/SATA Devices Figure 4 Gen 3 SAS Controller IV. PROPOSED ALGORITHM A. Current Solution- In huge topology, to find out all the attached devices, the discovery is done. The management application client performing the discovery process shall perform a level-order (i.e., breadth-first) traversal of the SAS domain to identify the end devices and expander devices in the SAS domain. The order of traversal shall be to discover: 1) The device to which the device containing the management application client is attached. 2) If the attached device is an expander device, every device attached to that expander device; and 3) For each expander device found, every device attached to that expander device. Special Issue - IDEAS ISSN: X

6 This order is repeated until all expander devices have been traversed. During discovery process, if an expander device is detected, the management application client shall use the SMP REPORT GENERAL and SMP DISCOVER and SMP Discover List commands to determine what is attached to each expander phy. If an expander device is detected and its CONFIGURABLE ROUTE TABLE bit is set to one in the SMP REPORT GENERAL function response, the management application client shall configure its expander route table. If an end device is detected, the management application client may attempt to open an SMP connection to determine if it contains an SMP target port, and use the SMP REPORT GENERAL function to determine additional information about the end device. End devices are not required to support SMP. The result of the discovery process is that the management application client has the necessary information to communicate with each SAS device and expander device in the SAS domain and each configurable expander device is configured with the expander route entries to allow routing of connection requests through the expanders. The discovery process may be aborted prior to completion if there is an indication that it may be based on incorrect information (e.g., arrival of a BROADCAST (CHANGE)).Thus if any device fails due to path break and the other device which is trying to connect this device receives OPEN_REJECT (NO DESTINATION).BROADCAST (CHANGE) will be indicated by expander and this message has to traverse back to controller which is initiator of the I/O messages. But before the controller will be notified, heavy I/O request messages have already been sent to that failed device. And also Gen 3 supports Fast Path I/O Block. Advantage of this is it handles all start and completion of successful I/O s autonomously from firmware thus allowing for improved I/O throughput. If error occurs, then firmware resends all the I/O s again unlike in Gen1 and Gen2 where firmware plays the role of transferring I/O messages which would add delay. So using discover process I will find out which all end devices are capable of supporting Fast Path I/O Engine capability, and then enabling that facility for the I/O message transfer between the host and that device. Also making table of all these Fast Path enabled devices so that other IO applications can use this information for routing purpose. Heavy I/O s Controll er E1 E2 E3 Level 1 BPP Level 2 E4 Figure 5 Current solution during Path Break B. Proposed Solution- In huge topology where heavy I/O requests are transmitted, the controller has to be notified without any delay about end device failure as it will keep on sending the I/O s to that particular device. In current solution, it has to traverse the path to reach the controller but in case of huge topology, the delay is significant. Hence the proposed solution will implement real time threads each expander which will be monitoring the device change and it will be updated to controller immediately when BROADCAST (CHANGE) message is received. And also implementing the API call to determine which all devices are capable of supporting fast path using the discovery process. In proposed solution, real time threads will be running in each expander device which keeps monitoring the topology status. When a device goes missing at any level and Broadcast (change) event reaches the expander, these threads updates the topology structure on priority. The Broadcast (change) event when reaches the controller, it is forwarded to host (Discovery tool) and firmware starts conventional discovery process. Once our tool gets the Broadcast (change) event, SMP Report General and SMP Discover list commands are sent to each of the expander threads. Each Special Issue - IDEAS ISSN: X

7 thread will send latest topology information to tool. So tool will have latest topology information on level by level basis but at a faster rate compared to conventional discovery process. Using this topology information tool will identify the missing device and take actions needed to such as flushing out IOs, updating its internal device structure, informing dependent applications about the topology change etc. Control -ler E1 E2 BPP E3 Thread E4 Figure 6.Proposed Solution V.CONCLUSION In the current scenario, if one device fails the BROADCAST (CHANGE) event is sent to all the Expanders and HBAs in the domain. The controller firmware starts looking for missing device on level by level basis. So if the missing device at the last expander level, then controller has to complete discovery on all above levels before it could reach to expander level to which missing device is attached. This process consumes considerable amount of time before controller updates its topology and informs the host of missing device. It is possible that the host has already sent heavy I/O s to the missing device before it is informed about topology update. It takes significant efforts at controller/firmware level and application level to flush out outstanding IOs and update status. To avoid this, real time thread functions will be running in each of the expanders monitoring topology structure which is stored in expander firmware at each level.,. Once Broadcast (change) event reaches, all threads update the topology. Tool once gets the event notification, it will send as many SMP Report General and SMP Discover list commands as the expander levels in topology and get the updated topology details. Tool will thus identify missing device and take appropriate actions. This technique is faster and prevents IOs being sent to missing device in a faster manner than through conventional discovery technique. REFERENCES [1] LSI Corporation, Storage Components Division Specification on SAS_Gen 3 Controller Firmware Architecture,Ver 1.1 [2] LSI Corporation, Storage Components Division Specification- Fusion-MPT Message Passing InterfaceV (Rev F) [3] ColtEDS2.4, Engineering Design Specification, Gen 3 SAS Core-Colt, LSI Document [4] LSI Expander_3 architecture spec, LSI Document. [5] Serial Attached SCSI Specification, SAS1.1. Special Issue - IDEAS ISSN: X