Average Latency observed by each Benchmark Process Request Size (MBytes)

Transcription

1 1 Emerging Serial Storage Interfaces: Serial Storage Architecture (SSA) and Fibre Channel - Arbitrated Loop (FC-AL) David H.C. Du, Taisheng Chang, Jenwei Hsieh, Sangyup Shim and Yuewei Wang Distributed Multimedia Research Center and Computer Science Department, University of Minnesota The authors would like to thank Dave Archer, Gary Delp, Larry Whitley and Walter Krapohl at IBM Rochester and Horst Truestedt, Edward Clausell, Richard Rolls, Michelle Tidwell and Howard Rankin at IBM Storage System Division for numerous discussions on the details of SSA. We also would like to thank Cort Fergusson, Mike Miller and Jim Coomes at Seagate Technology for providing us valuable information on FC-AL. January 6, 1997 DRAFT

2 2 Abstract Several serial storage interfaces have been proposed for constructing storage systems with high transfer bandwidth, large storage capacity, and fault tolerance feature. Among them, Serial Storage Architecture (SSA) and Fibre Channel - Arbitrated Loop (FC-AL) are considered as the next generation storage interfaces with broad industry support. Both technologies support simple cabling, long transmission distance, high data bandwidth, large capacity, fault tolerance, and fair sharing of link bandwidth. In this paper, a tutorial and a comparison of these two technologies are presented. The tutorial examines their interface specications, transport protocols, fairness algorithms, and capabilities of fault tolerance. The comparison focuses on their protocol overhead, ow control, fairness algorithms, and fault tolerance. The paper also summarizes the recently proposed Aaron Proposal which incorporates features from both SSA and FC-AL and targets at merging these two technologies. DRAFT January 6, 1997

3 I. Introduction Due to the rapid improvement of network infrastructure and computer processing power, many new applications such as virtual reality, digital library, video-on-demand, electronic commerce, etc are emerging at a very fast pace. These applications require a storage system which can support high data bandwidth, large storage capacity, and fault tolerance. When constructing a mass storage system, a designer requests a storage interface with the following capabilities: (1) a simple cabling scheme, (2) high data transfer rate, (3) capable of connecting a large number of storage devices, (4) fault tolerance and (5) cost eectiveness. The data transfer rate of storage devices is also increasing. For example, the bandwidth of magnetic disks has been improved by about 20% per year in the last few years due to the increasing recording density and faster rotational speed. Therefore, it is necessary to have high performance interfaces to take advantage of this improvement. Currently Small Computer System Interface (SCSI) channels are widely used for connecting storage devices to a host computer. However, SCSI channels are not designed with the aforementioned capabilities in mind. SCSI is a parallel bus interface which connects storage devices in a daisy-chained fashion. Although SCSI provides satisfactory performance for many traditional applications, it has some disadvantages including low data transfer bandwidth (the typical transfer rate is 20 MBytes/sec for fast/wide SCSI-2 channel), limited distance between devices, large connectors due to the parallel interface, and only small number of storage devices can be connected to a single channel due to the bandwidth limitation and overhead of SCSI bus. Moreover, its prioritized bus arbitration scheme leads to unfair bandwidth sharing among the devices attached to an SCSI channel. Even with the emerging Ultra SCSI channels running at 40 MBytes/sec, its performance is inherently limited by these deciencies. The limited bandwidth of SCSI interface forces a storage system designer to use many physical channels to connect a large number of disks in order to achieve high aggregate data transfer rate and to provide a large storage capacity. As an example to demonstrate this problem, we show a conguration of a mass storage system which was used in one of our previous studies. In this system fast/wide SCSI-2 channels were used to connect 1

4 disk arrays to a symmetric multiprocessing computer. This mass storage system consists of 24 RAID-3 disk arrays. Because one disk array (consisting of eight data and one parity disks) can deliver close to 16 MB/sec (this is close to the actual performance upper bound of a fast/wide SCSI-2 channel) throughput, only one disk array can be connected to one SCSI channel. Therefore, 24 SCSI channels are necessary to form such a large disk farm. Figure 1 shows two pictures of the connections between an SGI Onyx computer and 24 RAID-3 disk arrays. The left picture shows 24 disk arrays were connected to the backplane of an SGI Onyx computer via SCSI-2 host adaptors and cables. The other ends of the cables which connected to 24 disk arrays are shown in the right picture. This example demonstrates that it requires many SCSI channels and adaptors to set up a mass storage system with a large storage capacity and high aggregate data transfer bandwidth. This is usually only feasible for today's high-end servers and super-computers. Fig. 1. Use SCSI-2 channels to connect 24 RAID 3 disk arrays. To demonstrate the severeness of SCSI's unfair bus arbitration, we conducted a simple experiment on an SCSI-2 channel attached with ve disks on a multiprocessor computer. Five copies of a benchmark program were invoked with each process simultaneously accessing one dedicated disk repeatedly. Figure 2 shows the latencies observed by each 2

5 Average Latency (Sec) Average Latency observed by each Benchmark Process Disk 1 Disk 2 Disk 3 Disk 4 Disk Fig Request Size (MBytes) Average latency observed by each benchmark process with dierent data transfer sizes. benchmark process when the size of accessing data varied from 64 KB to 2 MB. As shown in the gure, there are signicant dierences among the latencies observed by dierent processes. In this test, Disk 1 always experiences the longest latencies while Disk 5 has the shortest latency (Disk 5 has the highest priority and Disk 1 has the lowest). The experiment result shows the impact of SCSI's unfair bus arbitration, especially when the SCSI is heavily loaded. Several serial storage interfaces with higher data transfer rates were developed as alternatives which may solve some of the SCSI's problems discussed before. These interfaces include Serial Storage Architecture (SSA) [1], [2], Fibre Channel - Arbitrated Loop (FC- AL) [4], [5], IEEE P1394 serial bus [9]. A serial interface uses compact connectors and serial links to simplify cabling schemes and reduce the cost. To reduce the cost of connection per device, it allows a large number (ranging from tens to more than one hundred) of storage devices attached to a single physical interface. Some of the serial interfaces also provide a mechanism for fault tolerance against single link failure. They also supports hot swappable devices and longer distance data transmission. Among the serial storage interface technologies, SSA and FC-AL are the two major technologies which are widely considered as the next generation storage interfaces. Both of them have broad industry support. SSA provides 20 MB/s of link bandwidth. The aggregate data bandwidth is 80 MB/sec 3

6 with two-in and two-out connections on a node. The full-duplex SSA interface supports fault tolerance against host, link, and adaptor failures. SSA devices (disks, tape drives, or hosts) can be congured as a string, a loop, or a switched topology. It also provides spatial reuse which allows multiple pairs of devices in a loop to communicate with each other simultaneously. Multiple hosts can be connected to a loop and access data concurrently for higher achievable throughput. It provides a fairness algorithm to enforce fair sharing of link bandwidth. FC-AL operates with 100 MB/sec link bandwidth. It provides high data bandwidth, fault tolerance, and an optional fairness algorithm. It supports any combination of hosts and storage devices up to a total of 127 in a single loop. As an enhancement to the Fibre Channel standard [7], devices in FC-AL can be connected by copper cables for low-cost storage attachment or optical cables for long distance transmission up to 10km with single-mode ber optics. The FC-AL standard includes a fairness algorithm which allows the storage devices connected on the same loop to fairly share the loop access. For performance consideration, some of the devices such as hosts may choose not to run the fairness algorithm. This allows a host to grab the loop access and send out commands earlier. A conguration with dual loops and multiple hosts oers fault tolerance against host, link, and adaptor failures. FC-AL supports a bypass circuit which may be used to keep a loop operating even when a device on the loop fails or is removed. In this paper a tutorial and a comparison of these two technologies are presented. The tutorial examines their interface specications, transport protocols, fairness algorithms, and capabilities of fault tolerance. Tutorials of SSA and FC-AL are presented in Section II and Section III respectively. The comparison in Section IV focus on their protocol overhead, ow control, fairness algorithms, and fault tolerance. The paper also summarize the recently proposed Aaron Proposal which incorporates features from both SSA and FC-AL and targets at merging these two technologies. The Aaron Proposal is briey discussed in Section V. Section VI concludes the paper. II. Serial Storage Architecture (SSA) The Task Group of Technical Committee X3T10.1 has been working on SSA standards 4

7 which maps the SCSI-2 protocol over SSA since The proposed SSA standard has completed and passed ANSI public review. It is currently in the process of being approved as an ANSI standard. Before we start introducing the technical details of SSA, we would like to give the readers an overview of SSA's features. We would like to provide a highlevel explanation about what is the SSA technology and how SSA can be used in many applications. The technical details about the protocol stack and transport protocol will be give in the following subsections. A. Overview Unlike SCSI which uses a shared bus to connect the devices, SSA uses a pair of pointto-point links to connect two devices together via a port on each device. A node (could be either a host adaptor or a storage device) with two ports can connect to two adjacent nodes by two pairs of links 1. Figure 3 shows one two-port node (the middle one in the gure) connected with two other two-port nodes. If all nodes are two-port nodes, then they can be connected as an SSA loop. An example of an SSA loop is shown in Figure 4. port port port port port port Router Router Router Fig. 3. Node interface Node interface Node interface One SSA two-port node (the middle one) connected with the other two two-port nodes. Fig. 4. An example of an SSA loop. 1 To simplify the description, we also use a duplex or bidirectional link in the paper. 5

8 The link bandwidth is 20MB/sec in the current specication 2. The host has the aggregate 80MB/sec bandwidth (two 20MB/sec links in and two 20MB/sec links out) in an SSA loop. The aggregate bandwidth can be doubled with future 40MB/sec links. Also, a host adaptor card with four ports can connect two SSA loops and achieve 160MB/sec aggregate bandwidth with 20MB/sec links. This provides a storage subsystem with large bandwidth. Compared to 20MB/sec SCSI, SCSI needs eight SCSI channels to provide the same bandwidth from the storage. For 40MB/sec Ultra SCSI, it still need four channels to provide the same bandwidth. One of the special features of SSA is the Spatial Reuse. This is because the links between two adjacent nodes are operating independently. Therefore, SSA is capable of supporting multiple simultaneous transmissions. Figure 5 shows an example of spatial reuse in an SSA loop with four hosts attached to it. One important advantage of this feature is, as shown in the Figure 5, each host can still have 80MB/sec aggregate bandwidth (which is the same as single host conguration) if each host only accesses its neighboring disks and there is no overlap among their access patterns. In fact, the aggregate bandwidth can be scaled up and each host on the loop can have 80 MB/sec aggregate bandwidth from an SSA loop if the aforementioned conditions are satised. In reality, each host may occasionally access far-away disks. Therefore, each host will have less than 80 MB/sec aggregate bandwidth. This also points out the importance of data allocation and load balance in an SSA loop. 40MB/sec 40MB/sec 40MB/sec 40MB/sec 40MB/sec 40MB/sec 40MB/sec 40MB/sec Fig. 5. Spatial reuse in an SSA loop with four hosts. Since there are two paths between each pair of nodes in an SSA loop, a single link failure 2 We assume 20MB/sec links bandwidth for the rest of this paper unless otherwise noted. 6

9 Link Failure Disk 3 Disk 2 Disk 1 Disk 4 20MB/s Traffic is rerouted to an alterante path 20MB/s Host (Initiator) Disk 5 Fig. 6. Disk 6 Disk 7 Fault Tolerance in a SSA Loop. can be tolerated. When a link in one path fails, an alternate path is available for data transmission. This redundant path eliminates a single point of failure. Figure 6 shows the scenario in which a single link failure is occurred. When the link from Disk 3 to Disk 4 fails, Disk 4 can send its data via the alternative route (through Disk 5 - Disk 6 - Disk 7) to the host. The detailed error handling procedures can be found in SSA standard document. The basic unit of information transmitted on SSA is a frame. A frame may consist of up to 128 bytes of data. The format of a frame will be discussed later. An SSA frame ows over each SSA node in a store and forward fashion. However, to reduce the long delay of traditional store and forward approach, SSA limits the maximum delay to ten character time (0.5 sec at 20 MB/sec) at each node if the intermediate node is allowed to forward this frame to the downstream node right away. Therefore, in the case of light load on the loop, the end-to-end delay can be close to the delay with cut-through routing approach. The destination node is responsible to drain a frame out from the loop. So each pair of source-destination nodes only use a portion of the loop. Each node can either forward frames from its upstream nodes or originate its own frames. SSA adopts a token based mechanism to ensure fairness of link sharing among the nodes. There is one token rotates in each direction (clock-wise and counter clock-wise). It ensures each node can originate certain amount of frames during each token rotation time. The token rotation time (or token cycle time) may vary from time to time depending on the trac load on the loop in that direction. 7

10 B. SSA Protocol Layers The proposed protocol stack is shown in Figure 7. The major function of each layer is outlined as follows. SSA-S2P (SSA SCSI-2 Protocol): It denes a mapping of the existing SCSI-2 protocol with the extension to SSA serial link. The goal of this protocol layer is to convert the existing SCSI-2 systems to be used in SSA and make such a migration easier. SSA-TL1 (SSA Transport Layer 1): This layer denes the transport layer functions for SSA. It includes ow control, acknowledgment, and fairness mechanism. SSA-PH1 (SSA Physical Layer 1): This protocol layer consists of the electrical characteristics of SSA interface and connectors. The Committee is dening another protocol stack which maps the SCSI-3 over SSA. This paper, however, will be focusing on the SCSI-2 over SSA, i.e., SSA-S2P - SSA-TL1 - SSA-PH1 protocol stack. Host Applications Disk SCSI-2 SSA-S2P SSA-TL1 SSA-PH1 SCSI-2 SSA-S2P SSA-TL1 SSA-PH1 Disk or Disk Array Fig SSA Loop The protocol stack mapping the SCSI-2 over SSA. C. SSA SCSI-2 Protocol: SSA-S2P SSA-S2P is a layer designed to minimize the changes required when convert the existing system and devices from SCSI-2 to SSA. This layer denes a data structure called SSA Message Structure (SMS) which is used to communicate control information between an initiator and a target 3. For examples, SCSI COMMAND SMS is used to transmit a Read or Write command from an initiator to a target. SCSI STATUS SMS can be used to 3 Note that throughout the paper an initiator is referred to a host and a target is referred to a disk. We shall use these terms interchangeably. 8

11 indicate the completion of a command. Table I and Table II show the corresponding SMSs transmitted between the initiator and the target for a read and write operation, respectively. TABLE I The typical activities for a read operation in SSA. Host SMSs or Data Frame on SSA Disk Command from upper layer SCSI CMD SMS! processing command fetching data prepared to receive data DATA READY SMS indicating data ready ready to receive data DATA REPLY SMS! allowed to transfer data First DATA frame sending data Last DATA frame sending data command complete informed SCSI STATUS SMS indicating command complete For a read command (Table I), a corresponding SCSI COMMAND SMS is transmitted from the initiator to the target. When the disk is ready to transfer data to the initiator, a DATA READY SMS is sent back to the initiator. After the initiator receives the SMS and is ready to receive data, it replies a DATA REPLY SMS to the disk. The disk then sends all the data corresponding to this command. At the end of data transfer, the disk sends a SCSI STATUS SMS to indicate the completion of this command. A write command follows the similar sequences as a read command is shown in Table II. TABLE II The typical activities for a write operation in SSA. Host SMSs or Data Frame on SSA Disk Command from upper layer SCSI CMD SMS! processing command DATA REQUEST SMS sending data First DATA frame!... sending data Last DATA frame! prepared to receive data ready to receive data command complete informed SCSI STATUS SMS indicating command complete 9

12 D. SSA Transport Layer: SSA-TL1 SSA uses 8B/10B encoding scheme. Each 8-bit data byte is encoded into a 10-bit character at the physical layer 4. Because of this 8B/10B encoding scheme, there are some bit patterns are not used to represent any data byte. SSA takes advantage of these special characters and uses them in its protocol layer. These special characters can be distinguished from other normal data bytes. Therefore, they can be inserted into any place of the transmission stream. We will introduce these special characters when we describe the data format and the SSA's transport protocol. We will show how they are used and helpful in improving the performance. The SSA frames are used to communicate between any two nodes. The frame format is shown in Figure 8. It consists of a one-byte long control eld, one to four bytes long address eld, one byte or two bytes long channel eld, up to 128 bytes data eld and four bytes CRC eld. The control eld identies the frame type. The address eld is use to route the frame. The rst bit in every address byte indicates if there is any other address byte following it. So the length of the address eld is actually dynamic and depends on the conguration. In an SSA loop, it only needs one byte address eld. The channel eld is used to identify which connection or application this frame is belong to when it arrives at the destination node. Up to 128 Bytes of data can be put in each SSA frame. Any two consecutive SSA frames must be separated by at least one FLAG. FLAG is one of the special characters we mentioned at the previous paragraph. The maximum eciency (the ratio of actual data length in each frame) of SSA is 128 (maximum data length)/136(1 Flag + 1 Control + 1 address + 1 channel data + 4 CRC) = 94%. In other words, theoretically SSA can utilize at most MB/sec out of 20MB/sec link bandwidth for actual data transmission. To avoid frame losses due to the receiver's buer overow, SSA uses a link-level creditbased ow control scheme to regulate transmissions between two adjacent nodes. A pair of RR (Receiver Ready) special characters is sent from a downstream node to a upstream node 4 Please note that when we said 20 MB/sec SSA link we meant the eective bandwidth. That is, the bandwidth for data transfer is 20 MB/sec although the physical hardware link speed is higher. We will use the same eective bandwidth whenever we mention link bandwidth throughout this paper. 10

13 1 char. 1-4 char. 1-2 char char. 4 char. FLAG Control Address Channel Data CRC FLAG Fig. 8. : SSA frame Data frame format for SSA. indicating enough buer space for the next frame. Note that downstream and upstream is from the data frame point of view. Each node can be a downstream node and also a upstream node for dierent directions. The upstream node requires to receive an RR character pair from the downstream node before it can start transmitting a new frame. When SSA starts operation, each node sends an RR pair to its upstream node of each link. This allows the upstream node to send an SSA frame. Every time the upstream node sends out an SSA frame, it has to wait for a new RR pair to be able to send out the next frame. How fast a new RR pair can be sent to the sender becomes an important factor on the performance. If a new RR pair arrives a long time after the sender sends out the previous frame, the sender has to wait for a long time before it can send the next frame. This results in a long idle time on the link and also a longer end-to-end delay. On the other hand, if a new RR pair can arrive before the sender nishes the transmission of the previous frame, the sender does not have to wait for the RR pair at all. To reduce the time needed for a new RR character to arrive at the sender, the receiver will send an RR pair as soon as it receives the control eld of an incoming frame if it has buer space for another frame (beside the one arriving). Since RR is a special character, the RR pair can be inserted into the transmission stream at any time. By these two approaches, if the receiver has buer space for the next frame, a new RR pair will arrive before the sender nishes the previous frame as long as the round-trip propagation delay and processing delay at the receiver is smaller than the frame transmission time. With 20MB/sec link, the transmission time of a 136 byte (the minimum frame size with 128 byte data eld and a FLAG) frame is 6.8 sec. The round-trip propagation time for 25m cable is about 175 nano seconds. To provide reliable transmissions, a link-level acknowledgment scheme is used. SSA 11

14 does not provide an end-to-end acknowledgment. It is up to the upper layer, for example, SCSI, to do that when there is a need. The transmission of each frame on an SSA link requires an acknowledgment from the downstream node. The acknowledgment is indicated by the arrival of an ACK character pair. The ACK is also one of the special characters. There is no sequence number information can be carried in this ACK character. Therefore, an acknowledgment only acknowledge one frame at a time. This results in at most only one outstanding frame (which is the frame that has been sent out completely but is not yet acknowledged) at any time. Similar as the ow control scheme, a late acknowledgment can also block a node from sending a new frame. To prevent this, a sender is allowed to start sending out the next frame even without receiving the acknowledgment for the previous frame. However, the tail FLAG of the second frame can not be sent before the acknowledgment comes back. Holding the tail FLAG of the second frame keeps the second frame in the state of transmitting which means the second frame is not yet complete and will not be considered as an outstanding frame. At the receiver side, the receiver should send an ACK pair to the sender right after it receives an whole frame with correct checksum. As in the RR ow control, the sender will not be blocked by this acknowledgment mechanism as long as the round-trip propagation time plus the checksum processing time is less than the transmission time. E. Fairness Algorithm As we mentioned at the overview, each SSA node can either forward or originate an SSA frame when it is allowed to send a frame to the downstream node. Normally, it will forward the frame from the upstream node rst. That is, it gives the trac from the upstream node higher priority. The reason of giving the higher priority to the frames from upstream nodes is to reduce the latency for the connections with longer paths. To prevent the starvation problem, the problem that a node never gets its chance to originate its frames, some mechanism is needed to ensure the fair sharing of the link among all the nodes. SSA adopts a token-based fairness algorithm to ensure fair sharing. There is one token rotating in an SSA loop in each direction (clockwise and counter clockwise). The token propagates on the loop and is used to govern the trac ow in the opposite direction 12

15 of the token rotation. When a node receives a token, it will switch the priority ordering between the frames from upstream node (we call them forwarded frames) and the frames to be originated from the node itself (we call them originated frames). It will store the frames coming from the upstream nodes in a buer. When this buer is full, the frame stream will be backed up. The upstream node will not be allowed to send more frames to this node due to the link-level ow control we described above. Two parameters, A quota and B quota, are used to regulate the amount of frames could be originated from each node. A quota denes the minimum number of frames a node is guaranteed to originate during each token rotation period (the time between when a node passed the token to the upstream node last time and the time when it passes the token again). When a node has originated at least A frames since it passed the token last time, the node is called Satised. The basic idea of the fairness algorithm is that it guarantees every node is Satised during each token rotation period. If a node is not Satised and has more frames to originate when it receives the token, it will hold the token and switch the priority such that it holds the frames from upstream nodes in a buer and originate its frames until it is Satised. After it is Satised, it should pass the token right away to the upstream node. To avoid the upstream node keep sending frames to the downstream nodes, B quota is used to regulate the maximum number of transmissions a node is allowed to originate during each token rotation period. If a node has originated B frames but the token has not arrived, it is not allowed to originate any more frames until the token arrives. When the token arrives eventually, the node should pass the token to the upstream node right away. This fairness algorithm is usually called SAT(ised) algorithm. The token is call SAT token. The SAT algorithm is extended to the case when there are more than one streams/connections is being originated from a node. The quotas will be dynamically scaled up by the number of connections at each node in this case. Although the SAT algorithm ensure the minimum and maximum numbers of frames a node could originate during each token rotation period, it does not necessarily result in even sharing. This is because the token rotation period that a node would experience varies from time to time and is dierent for dierent nodes. It highly depends on the 13

16 trac load on the loop, the load distribution among the nodes and the values of the A quota and B quota. The location of a node on a loop is also an important factor which would aect the number of frames it can originate. For example, the most upstream node has no upstream trac coming down to it. Therefore, this node does not need to wait for a token arrival to switch the priority order to originate its frames. On the other hand, the most downstream node will have a lot of frames coming from the other nodes from upstream and will most likely forward those frames most of time. Therefore, the most upstream node is more likely to be able to originate closer to B frames during each token rotation period while the most downstream node is more likely to be able to originate only A frames during each token rotation period in heavy load. Choice of A quota and B quota trades o the link utilization and the fair sharing. Choosing A = B will denitely ensure even sharing among the nodes. However, it comes with a cost of lower utilization when the trac load on the loop is light. On the other hand, setting B > A, as the scenario we gave above, will cause uneven sharing. There is no such a choice for A and B that results in optimal solution to all the scenarios. It really depends on the load and applications. III. Fibre Channel and FC-AL Fibre Channel is a high speed serial architecture that allows either optical or electrical connections at data rates from 25 MBytes/sec (Mbps) up to 100 MBytes/sec. Fibre Channel denes three topologies based on the capability and the existence of switches (also called fabric) between communicating ports (called N Ports). These include point-to-point topology, fabric topology and the Arbitrated Loop topology. In point-to-point topology, communication occurs between N Ports without using a switch fabric. In fabric topology, Fibre Channel uses the destination address in the frame header to route a data frame through a switch fabric to the destination N Port. The Arbitrated Loop topology allows more than two L Ports (ports which are capable of communicating in a loop topology) to communicate without a fabric. In Arbitrated Loop topology, only one pair of node can communicate at a time. Figure 9 shows these topologies. Fibre Channel provides three classes of service for dierent communication requirements. 14

17 Fabric Topology Arbitrated Loop Topology Fibre Channel Switch Connection Sub-Fabric Connectionless Sub-Fabric Fabric and Loop Topology Fig. 9. Topologies of Fibre Channel. Point-to-Point Topology These classes of service are distinguished by the method of connection setup and the level of delivery integrity. These classes of service are topology independent. Class 1 Service: It provides an acknowledged connection service with guaranteed bandwidth, end-to-end ow control, and in-order delivery. Class 2 Service: It is a frame-switched, acknowledged connectionless service that provides guaranteed delivery and buer-to-buer ow control. Class 3 Service: It is an unacknowledged connectionless service that lets data be sent rapidly to one device or multiple devices (with the help of a Fabric). Intermix: It is a concurrent Class 1, 2, and 3 services that enable parallel operations. It reserves full Fibre Channel bandwidth for dedicated Class 1 connections, but permit connectionless transmissions if bandwidth becomes available during idle Class 1 connections. FC-AL [4], [5] is an enhancement to the Fibre Channel standards. It denes additional signals and control mechanism to support Fibre Channel for operating with the loop. In a loop topology, communicating devices share a loop interface which only supports one pair of nodes to communicate at a time. A connection must be established between two L Ports before transferring Fibre Channel frames. FC-AL denes an arbitration scheme as an access protocol among L Ports. It is a prioritized protocol which grants accesses 15

18 of a loop to the L Ports with the highest priority. To prevent those L Ports with lower priorities from starvation, FC-AL denes a fairness algorithm which allows all L Ports have equal opportunity to access the loop. Since FC-AL is an enhancement to the Fibre Channel, we rst present a brief introduction to Fibre Channel in Section III-A. Then we describe FC-AL in Section III-B. A. Fibre Channel Fibre Channel has ve functional layers as shown in Figure 10. Among them, FC-0 to FC-2 are dened in Fibre Channel - Physical and Signaling Interface (FC-PH) [7]. FC- 3 (Common Services) is concerned with functions that span multiple N ports, including Striping (uses multiple N ports in parallel for their aggregate bandwidth), Hunt groups (allows more than one port to respond to the same alias address for higher eciency), and Multi-cast (delivers information to multiple destination ports). Upper Layer Protocols IP ATM HIPPI SCSI IPI FC-4 Protocol Mapping FC-LE FC-ATM FC-FP SCSI-FCP FC-I3 FC-3 Common Services FC-2 Framing Protocol and Flow Control FC-PH FC-1 Transmission Protocol FC-0 Fig. 10. Physical Layer Function layers of Fibre Channel. FC-4 (Protocol Mapping) provides a common and interoperable method for implementing upper-layer protocols over Fibre Channel. The protocols that Fibre Channel supports include SCSI, Intelligent Peripheral Interface (IPI), High-Performance Parallel Interface (HIPPI), Internet Protocol (IP). There is one FC-4 mapping protocol for each supported upper layer protocol. In Section III-A.2, we will describe one of the FC-4 protocols, Fibre Channel Protocol for SCSI[3]. A.1 Fibre Channel - Physical and Signaling Interface (FC-PH) 16

19 FC-PH denes three of Fibre Channel's functional layers, from FC-0 to FC-2. FC- 0 (Physical Layer) species a variety of physical media, drivers and receivers for a wide range of transmission speeds. The physical media can use single-mode or multi-mode ber, shielded twisted-pair, and coaxial-cable. Most of the commercially available products run at or Mbps at physical media. FC-1 (Transmission Protocol) denes the byte synchronization and the encode/decode scheme. An 8B/10B coding scheme is used for two types of transmission characters, data characters and special characters. Certain combinations of transmission characters are used to identify frame boundaries and transmit primitive function requests. Based on the 8B/10B coding, a physical link runs at Mbps (or Mbps) can support a data rate of 100 MB/sec (or 25 MB/sec). FC-2 (Framing Protocol and Flow Control) denes a set of building blocks to carry user data and ow control schemes to pace the transmission of frames. These building blocks include Frame, Sequence, Exchange and Protocol. Frames are based on a format as shown in Figure 11. The Data Field can carry up to 2112 bytes of data or up to 2048 bytes if there exist the Optional Headers. A Sequence is a set of related data frames transmitted unidirectionally with control frames, if applicable, transmitted in reverse direction. An Exchange consists of one or more Sequences. Fibre Channel also denes data transfer protocols and other protocols to manage the operating environment. We can use TCP/IP over Fibre Channel as an example to describe the hierarchy of building blocks. In this case, each TCP connection can be treated as an Exchange which is composed of one or more TCP/IP packets (as Sequences). A TCP/IP packet may be carried by a number of data frames. (4) (24) (0 to 2112) (4) (4) Idle Words SOF Frame Header Data Field CRC EOF Idle Words Fig. 11. Optional Header (64) Optional Header Payload (up to 2048) FC-2 general frame format. Unit = byte FC-2 also denes credit-base ow control schemes to pace the transmission of frames between nodes or between a node and a switch to prevent buer overow at the receiving side. The number of buers available at the receiving side is represented as Credits. The 17

20 credit information is sent from the receiver to the sender regularly. The sender uses a counter to manage the number of Credits which it received. The value of the counter is incremented with each credit it receives and decreased by one for each frame it transmits. During the transmission of frames, the sender restrains itself from transmitting more frames than the receiver can accommodate. A.2 Fibre Channel Protocol for SCSI Fibre Channel Protocol for SCSI (FCP) is one of Fibre Channel mapping protocols (FC- 4) which uses the service provided by FC-PH to transmit SCSI commands, data, status information between an SCSI initiator and an SCSI target. Each SCSI I/O operation is implemented as an individual Exchange consisting of a number of Sequences or Information Units. An Information Unit is a collection of data frames to be transmitted as a single Sequence by the Fibre Channel interface. A typical SCSI I/O operation consists of: (1) a command Sequence (FCP CMND) representing the desired operation, (2) zero or more transfer ready Sequence (FCP XFER RDY) and transfer data Sequence (FCP DATA), and (3) a response Sequence (FCP RSP) for the status information. TABLE III An example of the FCP read operation. Host Information Unit Disk Command request FCP CMND! Indicate command completion FCP XFER RDY FCP DATA... FCP RSP prepare data transfer data delivery request data in action prepare response message response For example, Table III shows the FCP mapping of an SCSI read operation to a series of FC-2 Sequences or Information Units. A command request (FCP CMND) is transferred by a host to a disk using a Sequence which only needs one frame for an SCSI read command. The disk follows the instruction contained in the read command and prepares the data. 18

21 When the desired data is ready to be transferred, the disk transmits one FCP XFER RDY (data delivery request) and one FCP DATA for each segment of data. This step is repeated until all data described by the SCSI command is transferred. FCP can take advantage of the multiplexing and shared bandwidth capabilities of FC Classes 2 or 3 Service. Multiple FCP I/O operations may be active at the same time. The maximum number of FCP I/O operations that may be active at one time depends on the queuing capabilities of a disk and the number of concurrent Exchanges supported by a Fibre Channel interface (the architectural limit is 65535). Class 1 and Intermixed classes of service may also be used to transfer the Information Units of FCP I/O operations. B. Fibre Channel - Arbitrated Loop (FC-AL) FC-AL allows Fibre Channel to operate in a loop topology. Its signaling interface supports a connection between two L Ports before they can exchange FC frames. FC-AL is logically located between FC-1 and FC-2 of the functional layers as shown in Figure 12. Figure 12 shows the Fibre Channel protocol layers in the context of a storage system with FC-AL. The upper-layer protocol used is SCSI. The Fibre Channel Protocol for SCSI (FCP) denes an FC-4 mapping layer for SCSI. In this example, FC-3 is not implemented. Applications SCSI-2/3 FCP FC-2 Host Disk Interface SCSI-2/3 FCP FC-2 Disk or Disk Array FC-AL FC-AL FC-1 FC-1 FC-0 FC-0 Out-bound Fibre Fig. 12. In-bound Fibre Out-bound Fibre In-bound Fibre Fibre Channel protocol layers with FC-AL. B.1 Arbitration An FC-AL loop is a shared medium for all of the attached L Ports. An L Port needs to arbitrate in order to access the loop. When more than one L Ports want to access the 19

22 loop, a priority scheme is used to decide which L Port wins the arbitration. The priority is based on the unique Arbitrated Loop Physical Address (AL PA) assigned to each L Port. L Ports with lower AL PAs have higher priorities. In most of the implementations, a host is assigned with a higher priority than disks. If a switch is also connected with the loop via an FL Port (a switch port which is capable of operating with an FC-AL loop), the highest priority (AL PA = 0x00) shall be assigned to the FL Port. Because the FL Port needs to handle the trac between L Ports in the loop and devices outside the loop. A loop is called a Public Loop if there exists an FL Port, otherwise it is called a Private Loop. TABLE IV Some of the FC-AL Primitive Signals. Primitive Signal Function ARBx Arbitrate Transmitted by an L Port (AL PA = x) to request access to the loop. OPNy Open Set up a circuit with a destination whose AL PA equals to y. CLS Close CLS is transmitted by an L Port to indicate that it is prepared to or has relinquished control of the loop. IDLE Idle Idle Table IV shows some of the Primitive Signals dened in FC-AL. These Primitive Signals are used to control accesses of a loop. There are other Primitive Signals for loop initiation and maintenance which are not listed here. Whenever an L Port wants to set up a circuit with another L Port. The L Port must arbitrate the control of the loop by sending out ARBx (arbitrate) with x as its AL PA. The ARBx Primitive Signals will travel the loop and reach all L Ports. When any arbitrating L Port (an L Port participating in arbitration) receives an ARBx, it compares its AL PA with the x value of the received ARBx. If its AL PA is smaller than the x value (which means it has higher priority), it sends out a new ARBx with x equals to its own AL PA. Otherwise, it shall forward the ARBx without any change. Eventually one of the arbitrating L Ports with the highest priority will receive its own ARBx and wins the arbitration. The L Port which won the arbitration can send out OPNy (open) to set up a circuit with another L Port whose AL PA is y. After a connection is established, they can exchange data frames and control frames according to FC-PH's specications. When either one of the two communicating L Ports nishes the transmission, it sends out a CLS (close) to 20

23 notify its partner. The other L Port will respond another CLS. After a pair of CLSs are exchanged between them, they relinquish control of the loop. The loop is then available again for other communications. To reduce the overhead of arbitration, an L Port which won the arbitration can open more than one circuits one by one without relinquishing control of the loop. In this case, the L Port sends out a CLS to close the current connection. Then sends out another OPNy to set up a circuit with another destination. This scheme allows an L Port opening more than one circuits without re-arbitrating the loop. For example, a host adaptor can arbitrate the loop once and send out multiple SCSI commands to dierent disks. B.2 Fairness Algorithm Like other prioritized protocols, FC-AL's arbitration scheme could lead to situations where an L Port with a low priority can not gain access of the loop. Thus, a fairness algorithm is dened to allow all L Ports to have an opportunity to arbitrate and win accesses of the loop. The basic idea of the fairness algorithm is that each L Port should only arbitrate and gain access of the loop once if there are other L Ports also arbitrating the loop. The fairness algorithm is enforced with one variable ACCESS maintained by each L Port and two special signals, ARB(F0) and IDLE. The default value of ACCESS is TRUE which allows an L Port to participate in arbitration. When an L Port wins the arbitration, it shall set its ACCESS to FALSE and restrain itself from arbitrating again until it receives an IDLE. When the wining L Port is opening a circuit, it sends out ARB(F0) to detect if other L Ports are also arbitrating. The ARB(F0) is a special ARBx whose x, equals to 0xF0, is lager than any possible AL PA. Any arbitrating L Port can change the ARB(F0) into its own ARBx. The ARBx (changed from ARB(F0) by some L Ports) or ARB(F0) circulates the loop and nally reaches the wining L Port. If an ARB(F0) is received by the wining L Port without change. This means no other L Ports are arbitrating. Otherwise, the wining L Port will receive an ARBx with x other than 0xF0. When an L Port intends to relinquish control of the loop, it sends out either an IDLE or an ARB(F0) depending on if there is any L Port was arbitrating access of the loop. If there is another L Port arbitrating (based on the received ARBx), it sends out an ARB(F0) that 21

24 stimulates the arbitrating process of other L Ports. If there are no L Ports arbitrating (the wining L Port received an ARB(F0)), the wining L Port sends out an IDLE. The IDLE will trigger all L Ports to set their ACCESS back to TRUE. This allows them to arbitrate the loop if they have data to transfer. The time between the rst L Port to win arbitration and an L Port to transmit an IDLE is called an access window. FC-AL's fairness algorithm sets up an access window in which all L Ports are given an opportunity to arbitrate and win accesses of the loop. An L Port can chose to use or not to use the fairness algorithm. In most implementations, the host adaptor does not use fairness algorithm, which allows it to promptly send out commands. Disks, on the other hand, usually use fairness algorithm to share the bandwidth with others t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 Time Idle Arbitrating Won Finished Fig. 13. An example of FC-AL fairness algorithm. An example of FC-AL fairness algorithm is illustrated in Figure 13. We use several gray levels to represent dierent states (idle, arbitrating, won arbitration, and nished) of a L Port and put its AL PA in the box. At time t1, L Port 3 won the arbitration and detected that there are other L Ports also arbitrating the loop. It restrains itself from arbitrating again until each arbitrating L Port has a chance to access the loop. The order of L Ports nishing their transmission is unpredictable since new L Ports may join the arbitration at any time. For example, L Port 1 joins the arbitration at time t2 and nishes before L Port 7. When the last L Port (L Port 7) won the arbitration, the ARB(F0) it sent out will return back to it without change. Therefore, it sends IDLE to conclude this access window and each L Port had an opportunity to access the loop. 22

25 B.3 Fault Tolerance The fault tolerance capability of FC-AL is accomplished by using two loops to connect all disks. All disks are attached to both FC-AL loops. One of the loops is used as the primary interface, the other loop is used for fault tolerance. In this case, if any loop fails, the disks can still be accessed through the other loop. FC-AL allows any combination of disks, tape drives, and hosts connect to the same loop. A networked le system consists of two hosts and a group of shared disks can provide better availability when any one of the hosts fails. FC-AL also denes a Bypass Circuit which can be used to keep a loop operating when an L Port is removed or fails. The Bypass Circuit provides the means to route the signal and bypass a failed L Port. In a disk array system implemented with FC-AL interface, the Bypass Circuits allow users to replace failed disks without shutting down the disk array. IV. Comparison between SSA and FC-AL In this section, comparisons of both technologies are discussed. Basic features are summarized in Table V. Features discussed include connectivity, topologies, spatial reuse, fairness algorithm, protocol overhead, and fault tolerance. TABLE V The comparison of SSA and FC-AL with single host Description SSA FC-AL (single loop) Fast/Wide SCSI Distance with copper 25m device to device 30m device to device 25m total length Distance with ber optic 2.5km device to device 10km device to device not supported Data bandwidth 80MB/s 100MB/s 20MB/s Number of attached devices Protocol Support SCSI SCSI/IPI/IP/HIPPI/ATM SCSI Error detection CRC detection CRC detection Parity Hot swappable devices yes yes yes with additional hardware Fault tolerance yes yes in dual loops no Fairness algorithm yes yes no Spatial reuse yes no no 23

26 The links between nodes in SSA and FC-AL are point-to-point connections. In SSA, each link communicates independently, and frames are transmitted using store and forward routing with the maximum of ten characters delay at each node if the intermediate nodes are allowed to forward the frame right away. In FC-AL, only one node can transmit at a time. A transmitting node is selected by an arbitration process. An SSA node is connected with two in and two out ports. In order to fully utilize the total link bandwidth, both inow and outow from a node need to be fully utilized to achieve the full data bandwidth. If a host generates only read commands, only half of the total bandwidth may be utilized because most of the trac for read commands are data frames from devices to a host. Because of independent links in SSA, spatial reuse is possible where more than one pair of nodes communicate simultaneously with each other. However, spatial reuse is not possible in FC-AL where only one node transmits data at a time. An SSA loop with multiple hosts may potentially increase the aggregate throughput due to spatial reuse. Both SSA and FC-AL oer exible topologies. SSA can be congured as a string, a loop, or a switched topology. FC-AL supports loop topology and loop with switch fabric topology. The Fibre Channel standard supports point-to-point and switched architecture. Hence multiple FC- AL subsystems can be connected by a bre channel switch. Both SSA and FC-AL allow multiple hosts in a single loop. Data sharing among multiple hosts can potentially increase data availability. SSA and FC-AL incur dierent protocol overhead. The protocol overhead includes two major parts, the framing eciency and the access overhead. The framing eciency is dened as the ratio of data portion (or payload) in a frame to the total frame size. The maximum framing eciency of SSA is about 94 percents. FC-AL's framing eciency is about 98 percents. These are derived by the maximum data size of 128 bytes out of the frame size of 136 bytes for SSA, and the maximum data size of 2112 out of the frame size of 2148 bytes for FC-AL. Access overhead for both technologies are also dierent. Each loop access in FC-AL requires an arbitration which results in higher overhead. On the other hand, SSA does not require any access arbitration. Therefore, although SSA has less framing eciency, it has less overall protocol overhead (in terms of percentages) for small size transactions. FC-AL has the higher framing eciency but higher access overhead in 24

27 small size transactions. However, FC-AL has less protocol overhead and may have less latency for large size requests (or commands) because of its higher bandwidth. Dierent ow control schemes are used to prevent buer overow at the receiving side. Flow control in SSA is enforced by credits between two adjacent nodes. FC-AL uses buer-to-buer ow control between the source and destination nodes. In SSA, multiple connections can be outstanding at the same time. SSA frames of dierent connections can be multiplexed and routed through a single link. FC-AL uses a connection oriented routing. A source node sets up a connection and transmits frames to the destination. While a source node in FC-AL transmits data frames, a destination node can send control information back to the source node. With FCP and SCSI protocol, multiple I/O operations can also be outstanding at the same time. The fairness algorithm of SSA is implemented by using a token scheme and two quotas, i.e. A quota and B quota. The values of the A quota and B quota are important to enforce the fair sharing of the link bandwidth. When A < B, upstream nodes can send more data than downstream ones because the upstream nodes have inherently higher priority over the downstream nodes. When the loop is heavily loaded, the nodes in downstream are allowed to transfer only up to A quota if upstream nodes are kept sending data. On the other hand, the upstream nodes use the data bandwidth up to B quota when available. The fairness algorithm in SSA is a part of the SSA protocol. On the other hand, a fairness algorithm in FC-AL is an optional feature. The fairness algorithm is enforced by keeping a state information at each node. Within a single access window, a node can win an arbitration only once. The fairness algorithm in FC-AL can be enforced partially or fully. Since the hosts in a loop need to send out commands promptly, they may not want to run the fairness algorithm. The fairness algorithm in FC-AL guarantees equal number of opportunities in accessing a loop. However, the equal number of opportunities do not translate into the equal amount of trac for all the nodes. SSA in a loop provides redundant paths to a device. It can tolerate a single link failure. A multiple host conguration in SSA oers fault tolerance against host, link, and adaptor failures. FC-AL with a single loop does not provide fault tolerance against a link failure. Hence FC-AL is often congured as dual loops. An FC-AL conguration with dual loops 25