A Discussion of Storage Area Network Architecture Relevant to Real-Time Embedded Applications

Transcription

1 A Discussion of Storage Area Network Architecture Relevant to Real-Time Embedded Applications System designers are often challenged to define architectures that can be characterized from end-to-end, offering low latency, high bandwidth data paths. Further, system scalability, redundancy, and component physical location are factors that require consideration, and often are parameters that become trade-offs when developing an embedded system solution. Existing fabric technologies applicable to storage systems that are available today are Ethernet Switched Fabric, and Fibre Channel Switched Fabric. Both of these technologies can be used to implement Storage Area Networks, which are networks of storage devices that can be connected together using Fibre Channel or Switched Ethernet, or both. SAN architectures extend the capabilities of storage topologies that were dependent on the bandwidth and physical limitations of SCSI. Further, architectures supporting multiple protocols are possible across the SAN. System designers can benefit from the deterministic low latency transport available through switched fabric, providing the flexibility to tackle many real-time deterministic applications, such as highbandwidth telemetry links, or mixed media streaming communication platforms. Lastly, the technology to implement SAN architectures is available today on various bus structures supporting multiple market segments. Future fabric technologies will extend the performance of SANs and provide a road map for system enhancement. Figure 1. Typical SAN Implementation with Fiber Channel Switches Image source: International Business Machines, Inc. Basic Description of a SAN SAN stands for Storage Area Network. The Storage Area Network is typically a dedicated network of storage devices connected together serving multiple computers as illustrated in Figure 1. The SAN separates the LAN, or local area network data from the network traffic generated between the storage device and the processing elements within a computer network. SANs allow an any-to-any relationship between computers and storage on the network. This allows parallel and overlapped accesses to take place. Further, any compute element can access data stored on any device within the network. This model is different from the traditional computer to disk model, where the drives are directly attached to the compute element. Additionally, in the direct attached case, the computer must be involved in all transactions to the storage, which is not the case with a SAN.

2 SANs are fundamentally scalable, so large multi-terabyte or multi-pedabyte arrays can be built. SANs are built around network interconnect technologies that allow the storage elements to be spread out over a relatively large distance, as opposed to the sub 25 meters or less for high bandwidth SCSI. Benefits of SAN Typical Benefits provided by SANs are: Any-to-Any storage relationship A true network of storage devices Block access to data Data-Centric storage model Easily implemented redundancy schemes High Bandwidth access and delivery Scalability of both bandwidth and capacity Not practically limited by physical distance Allows server free back-up to disk and to tape Disadvantages of SANs include: Network complexity Training, Support, Learning Curve Initial cost IT Model for SAN vs. the Embedded Model for SAN Typically we think of the SAN in a traditional IT role. In this role, the SAN provides a practical solution in providing data centric storage to multiple servers, allowing the servers to be connected to local area networks separated from the SAN. However, as practical as it may be to discuss SAN architecture employed in banks or in insurance companies, the use of SANs for real-time embedded solutions is coming of age. Until recently the DAS model has dominated real-time embedded solutions, including DAS RAID based solutions. One may ask, how can an embedded architecture benefit from the SAN? Within the embedded space, the possibility to aggregate bandwidth is of interest allowing very high bandwidth fat pipes to be fed to the disks. Further, redundancy is of interest, where RAID techniques must be employed. In large server or parallel processing applications it may also be desirable to have many CPUs share common storage, which is also possible. Network transport determinism is a fundamental issue in real-time systems. Fibre Channel provides a deterministic network transport. This may be one of the most compelling aspects of switched fabric SANs relative to system designers providing networks that must perform in real-time. Network Technology used in SANs Fibre Channel - a SAN Backbone At the backbone of most SANs is Fibre Channel. Fibre Channel is a network architecture based on standards that provide a scalable data transport for the movement of data. The OSI model applies to Fibre Channel with Physical, Transport, and Upper Level Protocols being defined and implemented. The characteristics of Fibre Channel are presented in summary, with the Fibre Channel frame format specified in Table 1. Figure 2 depicts the structural hierarchy.

3 Fibre Channel Overview Fibre Channel Characteristics Serial Transport Fiber Optic or Copper Physical Interface Gigabits per second full duplex extensible to > 4.25 Gigabits Gigabits per second links now shipping 8b/10b encoding Low Overhead with 1 x bit error rate Point-to-Point, Arbitrated Loop, or Fabric Topologies Supports transmission over long distances, up to 10 km Transport supports multiple protocols including IP, SCSI-3, IPI-3 Disk, IPI-3 Tape, HIPPI Classes of Service including: Class 1 through Class 6 SOF Header Data Field CRC EOF 1 word 6 words bytes 4 bytes 1 word Table 1. Fibre Channel Frame Format Figure 2. Fibre Channel Structural Hierarchy Typical Application Fibre Channel is typically used as the backbone of most SANs today. Fibre Channel Features from a Real-Time Embedded Perspective Fast data rate: 100 or 200 MBps full duplex bandwidth! Priority/Preemption Consistent and reliable delivery of data under maximum utilization No decrease in data throughput as a function of distance Bounded Latency allowing deterministic delivery calculations Efficient data delivery with much lower overhead than TCP/IP over Ethernet - on the order of 30X less processing power required of the CPU to move data Gigabit Ethernet for SAN Backbones, LAN and WAN Gigabit Ethernet is a data link transport that blends features from the Fiber Channel and conventional Ethernet (802.3) standards. Gigabit Ethernet is used in SAN and Networked Attached Storage, NAS, applications. The characteristics and features of Gigabit Ethernet are presented in summary, with references made in the suggested reading list for applicable standards and texts providing detailed information. Table 2 describes the frame format, while Figure 3 addresses the OSI model.

4 Gigabit Ethernet Overview Gigabit Ethernet Characteristics Serial Transport Copper or Fiber Optic Media 1.25 Gbps Data Rate 8b/10b encoding Transmission over moderate distance to 5 km Fabric Topology Typical Application Local Area Networks, Wide Area Networks, Networked Attached Storage, iscsi. Features of Gigabit Ethernet Standard Ethernet Framing Numerous protocols, including IP, UDP, TCP/IP Dedicated full duplex links (CSMA/CD defined for backwards compatibility) VLANS, QoS, link-layer flow Control flow control provides reliable transport of storage data via connectionless protocols like UDP/IP ad link aggregation allows scalability of IP SANs without performance penalties Ubiquitous standard for Networking, widely accepted Supports standard CAT5 cabling Preamble SFD Destination Address Length or Type Data Frame Check Sequence 56 bits 8 bits 48 bits 16 bits 46 to 1500 bytes 32 bits Table 2. Ethernet IEEE Frame Format Ethernet Structural Hierarchy Key: Ethernet Specific 7 6 Applicatio n Presentation Logical Link Control sublayer (LLC) Session Transport Network Data Link Physical Media Access Control sublayer (MAC) Phys ical Signaling sublayers Media Spe cifications Figure 3. OSI/IEEE Sub-layers Source: C.E. Spurgeon, Ethernet the Definitive Guide, 1 st ed., O Reilly, 2000

5 Protocols used in Storage Area Networks and Network Attached Storage Characteristics of SAN In a SAN, storage devices are allocated to a dedicated network where block I/O requests access devices directly. Characteristics of NAS NAS is an intelligent file oriented appliance, integrating processor and storage, where access is implemented using TCP/IP requiring file sharing protocols. Protocols used in Storage Area Networks - FCP SCSI-3 over Fibre Channel Protocol FCP is almost implied when discussing Fibre Channels and SANs. Here Fibre Channel acts as the transport for SCSI-3 command, response, status, and data blocks. - FCIP Fibre Channel over IP Protocol FCIP allows the encapsulation of Fibre Channel Frames within TCP/IP specifically for linking Fibre Channel islands over Wide Area Networks, this is also referred to as Fibre Channeling tunneling. This is a point-to-point protocol. - ifcp Internet Fibre Channel Protocol In this protocol, the Fibre Channel service is implemented by providing native Fibre Channel protocol over IP. It is used as a switch or gateway protocol. This protocol enables Fibre Channel switching over TCP/IP networks. The protocol uses TCP for congestion control, and error detection and recovery. This protocol is more Fibre Channel than IP centric. - iscsi Internet SCSI Protocol SCSI I/O encapsulated in IP Packets allows SCSI I/O to occur over Ethernet. iscsi can be used for Direct Attach and SAN connections to devices. Although iscsi resides on top of TCP/IP, it implements block I/O enabling use in the DAS or SAN interconnect models. In iscsi, the Fibre Channel protocol is eliminated in the SAN implementation. IBM has just introduced some leading edge storage products using iscsi. Refer to Lightweight Protocols for use in Real-Time Systems Protocols specific to real-time applications are available from several vendors, including SBS and Systran. These protocols are characterized as being lightweight in that they are efficient and make optimal use of the Fibre Channel transport. FXRI Raw Initiator a Protocol from Systran A Low Latency Protocol that bypasses the File System. This Protocol is used for high- Bandwidth transactions to storage where the ingest rate, or the delivery rate must be high, and predictable. Table 3 compares Fibre Channel Raw Initiator protocol with the File System protocol. Data rates to the storage system approach the Fibre Channel maximum bandwidth of 100 MBps in one direction.

6 Fe ature FXRI File Syste m Stores file and directory information No Yes Is the location of the data automatically No Yes stored on the disk? Are transactions guaranteed to complete Yes No immediately? Is performance predictable? Yes No Can data be accessed on other systems? Yes Depends on File System Can data be shared by a number of systems? Yes No Table 3. Raw Initiator vs. File System Protocol Point-to-Point Communications Protocol The FxLP Lightweight protocol can be used over Fibre Channel for point-to-point communications between nodes. This protocol uses SCSI-3 over Fibre to implement a low latency, high bandwidth peerto-peer communications. The protocol is efficient and bypasses the need for TCP/IP and the related overhead, while maintaining a socket like API for system implementation. High data rates with small packet sizes are possible. Building an Embedded SAN With the network technology and the protocols defined, the components necessary to implement a SAN can now be discussed. To implement an embedded SAN one needs to consider the following components: 1. Fibre Channel or Ethernet Interface to the fabric 2. Processing elements for connection to the SAN 3. Fibre Channel Switch (if a fabric topology is selected) 4. Ethernet Switch 5. Selections of a Storage Solution, including JBOD, or RAID With respect to implementation, a topology must be selected. In this case we will examine a fabric topology where several Fibre Channel pipes are connected to a single RAID. Ingest/Pump Application Digital Video Distribution Consider a typical SAN solution that requires multiple servers to be connected between a LAN and the SAN. In a high-resolution digital video distribution system, it is necessary to first store video content, and then play it. For instance, in distributing content to movie theaters, studios may opt to transmit the content digitally via a high speed down link to the theater. Once received, the content must be buffered to the storage system. Later, the content must be accessed for replay. In this case the content would be distributed to video projectors that may be fed from a LAN. Here Gigabit Ethernet may be used to distribute content to the individual projectors in a Multiplex. Since the data delivery to the projectors must be contiguous, and deterministic, a real-time O/S is selected to host the video pump applications. For this example, assume each CPU can pump two streams, each delivered to the projector via a Gigabit Ethernet port. The application requires several CPUs to be able to access data from the SAN and feed it into an Ethernet network. The number of CPUs is dependent on the number of projectors required. The system is scalable. To accomplish this, the LAN and SAN topologies must be selected. In order to

7 maintain a small physical footprint, the LAN implementation will use PICMG 2.16 CompactPCI Packet Switched Backplane topology to implement an embedded Ethernet backbone within the chassis. CompactPCI Packet Switched Backplane Supports Embedded Ethernet cpsb allows embedded Gigabit Ethernet connections to take place, two per slot within the chassis. Each of the connections is fed to a 24 port by 2 Gigabit Ethernet Switch. Further, the application requires several servers to be connected to the Embedded System Area Network, ESAN, in order to provide content to the local area network. The CPUs in this case can be 6U Pentium based CPUs, supporting dual Gigabit 2.16 compliant Ethernet ports routed to the backplane. Adequate processing power is provided with a 1.8 GHz Pentium IV, with One Gigabyte of on board memory. Appropriate I/O Bandwidth for the application is supported via an internal 64 bit wide, 66 MHz PCI bus to the I/O devices. Next we address the connections to the SAN. In this case Two Gigabit Fibre Channel PMCs are available from several vendors. These PMCs provide up to 100 MBps pipes, bi-directionally, between the fabric and the processor. 200 MBps PMCs are also available. Virtual Storage Architecture A Fast RAID Technology Lastly, we need to select the storage solution. Here we select a new technology that implements a high bandwidth storage array for use in the SAN environment. The RAID solution chosen uses a Virtual Storage Architecture that enables RAID delivered I/O bandwidth to exceed 400 Mbytes per second. Typical bandwidth supported by conventional RAID arrays is 240 Mbytes per second. With Virtual Storage Architecture, the RAID concept is to stripe data across multiple arrays of drives, using multiple XOR engines. In this way, sustained high bandwidth I/O can be supported, while maintaining data redundancy and rebuild capabilities. Virtual Storage Architecture is a trademarked product of Storage Computer ( Figure 4 shows the architecture described supporting three servers, each containing a Dual Gigabit Fibre Channel interface connected to a RAID array providing four One Gigabit Fibre Channel ports into the array. In this architecture a switch is used to allow the down link receiver access to the storage array, while the servers are connected to the array through the Fibre Channel switch.

8 Fibre Channel SAN based Video Distribution System Figure 4. Real-Time Video Distribution SAN Application Example CPUs supporting the Fabric Interfaces CPUs capable of supporting Fabric interfaces are available in Pentium, PowerPC, and SPARC technologies. Concurrent Technologies is producing a high performance Pentium CPU with dual Gigabit Ethernet. Many manufacturers, including Force Computers, SUN Microsystems, and Motorola are also supporting cpsb Ethernet on their CPUs making them ideal to implement LAN to SAN implementations for various applications. The ideal CPU supports PMC sites for the Fibre Channel interfaces, with Ethernet available from the front or rear. cpsb compatible Ethernet is available from the rear enabling Embedded System Area Networks to be created.

9 Operating System Considerations With respect to operating system implementation, drivers and stacks are available for real-time operating systems such as VxWorks. If the application does not require real-time processing, Linux or Solaris support Fibre Channel interfaces, as well as more complex software such as clustering packages. Management software is available under Unix for management of the SAN. Open Systems supported through Multiple Vendors SAN Networks can be created and are supported through products available from multiple vendors. Embedded SAN solutions can be created in a standards based open environment where a substantial track record exists for the technologies required. Summary SAN architectures employing fabric technology are viable for implementation and use within embedded systems. For military, medical and communication applications, switched Ethernet and Fibre Channel provide network technology that supports the requirements of real-time system implementation. Further, fabric specifications such as PICMG 2.16 and the future VITA 31 provide standards for system designers to build scalable systems that can benefit from high speed interconnect to embedded SANs. About the Author Ken Grob is Vice President of ACT/Technico, an Embedded Computing Systems Integrator. He holds a BSEE from Drexel University and has been in the Bus & Board industry since The company provides system architecture design, development, and implementation of complex solutions for communications, military/aerospace, medical and semiconductor equipment manufacturers. For more information about ACT/Technico products and services, contact: ACT/Technico One Ivybrook Blvd., Suite 180 Ivyland, PA (215) Standards Organizations and Trade Associations The following are some of the standard organizations supporting the technologies required to implement SANs. Fibre Channel Standards The National Committee for Information Technology Standards, Technical Committee T11, X Fibre Channel Physical and Signaling Standard: FC-PH, Initial core standard Storage Network Models The SNIA Shared Storage Model of the Storage Networking Industry Association, Information can be found on Fibre Channel, IP Storage Networks, and NAS.

10 Protocols Internet Engineering Task Force (IETF), : Protocols in development under the IPS Work Group; FCIP, Fibre Channel Over IP; ifcp, Internet Fibre Channel Protocol; iscsi, Internet SCSI Ethernet The latest Ethernet Standard, including Gigabit Ethernet definition: IEEE std 802.3, 2000 Edition, ISO/IEC : 2000 (E)), IEEE Standard for Information technology Telecommunications and information exchange between systems Local and metropolitan area networks Specific requirements Part 3: Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Specifications (includes 802.3ab, 802.3ac and 802.3ad) Available for download in pdf format at: CompactPCI Packet Switch Backplane Information on PICMG 2.16 IP Backplane for CompactPCI can be found at: specdirectory.stm#_picmg_2.16_ip: SCSI The National Committee for Information Technology Standards, Technical Committee T10, SCSI, SCSI-2, and SCSI-3 including SPI, Fast-20 (Ultra SCSI), Fast-40 (Ultra2 SCSI), Low Voltage Differential (LVD), SPI-3 (Ultra3 SCSI or Ultra160), SPI-4 (Ultra320), CAM, and much more. Suggested Reading Designing Storage Area Networks by Tom Clark, Addison-Wesley Fibre Channel for SANS by Alan F. Benner, McGraw-Hill Telecom Ethernet The Definitive Guide, by Charles E. Spurgeon, O Reilly Fibre Channel: Connection to the Future, The Fibre Channel Association IP Sans A Guide to iscsi, ifcp, and FCIP Protocols for Storage Area Networks, by Tom Clark, Addison-Wesley IBM Storage Networking, June 2000: Demystifying Storage Networking DAS, SAN, NAS, NAS Gateways, Fibre Channel, and iscsi, by David Sacks IBM Storage Consultant