COSC6376 Cloud Computing Lecture 17: Storage Systems Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston
Storage Area Network and Storage Virtualization
Single Disk Drive Storage Array SCSI device DAS NAS SAN iscsi Storage types Introduction To Storage: Overview of the storage subsystem
Direct attached storage (internal) Computer System CPU Memory Bus I/O - RAID Controller Disk Drives Introduction To Storage: Overview of the storage subsystem
Direct attached storage (internal) Data 12345 Computer System CPU Memory Bus I/O - RAID Controller Disk Drives Introduction To Storage: Overview of the storage subsystem
Direct attached storage (internal) 12345 Computer System CPU Memory Bus I/O - RAID Controller Disk Drives Introduction To Storage: Overview of the storage subsystem
DAS w/internal controller and external storage 12345 Computer System CPU Memory Bus I/O - RAID Controller Disk Enclosure Disk Drives Disk Drives Disk Drives Introduction To Storage: Overview of the storage subsystem
Attach large number of disks 4-Processor AMD Dual Core Opteron can handle 2.5 GB/s or more sequential read and write data traffic Each controller connects 8 point-to-point SATA Drives Average 400 MB/s/card PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB PCI-X SATA Controller PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 16 TB Raw Disk Space 12 TB RAID 5 PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 8 TB Mirrored PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB
Backblaze
Backblaze Backblaze Storage Pods: 67 terabyte 4U servers
Comparing internal and external storage Server Storage Disk Drives RAID Controller Internal Storage RAID controllers and disk drives are internal to the server SCSI, ATA, or SATA protocol between controller and disks RAID controller is internal SCSI or SATA protocol between controller and disks Disk drives are external Server RAID Controller Storage Disk Drives SCSI Bus w/ external storage Introduction To Storage: Overview of the storage subsystem
Direct attached storage Direct Attached Storage (DAS) Storage is captive behind the server Server CPU must handle user I/O requests, but also: User-database inquiries User file/print serving Data-integrity checking Communication with other devices Data access is file system and platform dependent Costly to scale; complex to manage Clients IP Network Servers Win Linux Win Linux Unix FC FC SCSI Direct-Attached Storage (DAS)
DAS w/external controller and external storage Storage System 12345 Computer System CPU Memory Bus Disk Enclosure Disk Drives Disk Drives Disk Drives HBA RAID Controller Introduction To Storage: Overview of the storage subsystem
DAS over Fibre channel Server HBA HBA is internal Fibre Channel protocol between HBAs and external RAID controller Disk drives and RAID controller are external Storage Disk Drives RAID Controller Introduction To Storage: Overview of the storage subsystem External SAN Array
I/O Transfer RAID Controller Contains the smarts Determines how the data will be written (striping, mirroring, RAID 10, RAID 5, etc.) Host Bus Adapter (HBA) Simply transfers the data to the RAID controller. Doesn t do any RAID or striping calculations. Dumb for speed. Required for external storage. Introduction To Storage: Overview of the storage subsystem
SCSI Small Computer System Interface
Terminology Initiator: A SCSI device that requests an operation to be performed by another SCSI device. Host Adapter: A Controller Board that translates host I/O requests into SCSI requests. Target: A SCSI device that performs an operation as requested by an initiator. Logical Unit: A Physical or Virtual device addressable through a target. LUN: Logical Unit Number. An encoded three-bit identifier for the logical unit.
SCSI configuration single initiator, single target LUN0 Initiator SCSI Bus Target LUN1 LUN2
SCSI configuration Single initiator, multiple targets Initiator 7 T Target 0 SCSI Bus Target 1 SCSI Bus Terminator T Target n
SCSI configuration Multiple initiators, multiple targets Initiator 7 T Target 0 SCSI Bus Target 1 Initiator 6 T Target n
The SCSI I/O Channel SCSI is the dominant protocol used to communicate between servers and storage devices in open system SCSI I/O channel is a half-duplex pipe for SCSI CDBs and data Parallel bus evolution Bus width: 8, 16 bits Bus speed: 5 80 Mhz Throughput: 5 320 MBps Devices/bus: 2 16 devices Cable length: 1.5m 25m A network approach can scale the I/O channel in many areas (length, devices, speed) TCP/IP Stack NIC Driver Ethernet NIC Applications Raw File System Block Device SCSI Generic Adapter Driver SCSI Adapter Half-Duplex SCSI I/O Channel SCSI Ethernet SCSI CDB: SCSI Command Descriptor Block Used to Relay SCSI Commands, Parameters, and Status between SCSI Initiators and SCSI Targets; Typically 6, 10, or 12 Byte Block IP Storage Networking FCIP/iSCSI. Steve Tegeler SCSI Initiator SCSI Target
Networked I/O Channel Networking the I/O Channel Same SCSI protocol (SCSI-3) carried over a network transport layer via serial implementation Transport must not jeopardize SCSI payload (security, integrity, latency) Two primary transports to choose from today: Fibre Channel and IP A networked I/O channel allows for multiple improvements: Distance limitations greatly increased High number of addressable devices Initiator Host System SCSI Channel Controller Network IP Storage Networking FCIP/iSCSI. Steve Tegeler Target and LUNs
Fibre channel networking Very common method for networking SCSI Fibre Channel provides high-speed transport for SCSI payload Fibre Channel SAN overcomes many shortcomings of DAS including: Addressing for up to 16-million nodes (24 bits) Loop (shared) and Fabric (switched) transport Speeds of 100 or 200 MBps (1 or 2 Gbps) Distance of up to 10km (without extenders) Support for multiple protocols Combines best attributes of a channel and a network Initiator Host System SCSI Fibre Channel HBA Fibre Channel Fabric Target IP Storage Networking FCIP/iSCSI. Steve Tegeler
IP: an alternate I/O transport Viable transport for I/O traffic Similar characteristics to Fibre Channel: Addressing for close to 4 billion nodes (IPv4) Primarily a switched transport (with routing) Ethernet speeds of 1/10 Gbps or various WAN speeds Support for multiple high-level protocols Cost and manageability advantages with IP IP knowledge base widespread in industry Initiator Host System SCSI IP Channel Adapter IP Network Target IP Storage Networking FCIP/iSCSI. Steve Tegeler
Fibre Channel
Fibre channel: what is it? Fibre Channel is a network protocol implemented specifically for dedicated storage networks Fibre Channel utilizes specialized Switches Host Bus Adapters RAID controllers Cables Introduction To Storage: Overview of the storage subsystem
Fibre channel components Server A Server B Server C HBA HBA HBA HBA HBA HBA switch switch Disk Drives RAID Controller RAID Controller FC Storage Array Servers Host Bus Adapters Cables Fiber optic or copper Fibre Channel Switches Two switches for redundancy Fibre Channel Storage Array Two RAID Controllers for redundancy 4 100+ disk drives per array A true storage network Multiple servers Multiple switches Multiple Storage Arrays Introduction To Storage: Overview of the storage subsystem
What is fibre channel? A high-speed transmission technology used as a peripheral channel or network backbone. FC1: Over 1G bits per second FC2: Over 2G bits per second It supports several common transport protocols like Internet Protocol (IP) and SCSI. It operates over copper and fiber optic cables at distances of up to 10 Kilometers. Up to 127 devices (SCSI: 15) Up to 10 km of cabling (3-15 ft. for SCSI) Physical interconnect can be copper or fiber optic It is supported by many suppliers.
Fibre channel (continued) How does it work? Serial interface Data is transferred across a single piece of medium at the fastest speed supported No complex signaling required Hot-pluggable Devices can be removed or added at will with no ill effects to data communications Provides a data link layer above the physical interconnect, analogous to Ethernet Sophisticated error detection at the frame level Data is checked and resent if necessary Introduction To Storage: Overview of the storage subsystem
Current standards: Performance from 266 megabits/second to over 4 gigabits/second High-bandwidth utilization with distance insensitivity. Support for multiple cost/performance levels, from small systems to super computers Ability to carry multiple existing interface command sets, including Internet Protocol (IP), SCSI, and audio and video
Fibre channel frame dissection Up to 2048 byte payload 4 byte checksum for each frame Introduction To Storage: Overview of the storage subsystem
Fibre channel What s with the funny name? Some background history required Originally developed to only support fiber optic cabling When copper cabling support was added, ISO decided not to rename the technology ISO changed to the French spelling to reduce association with fiber optics only medium Introduction To Storage: Overview of the storage subsystem
Fibre channel interface layers RAID Controller Disk Drive Fibre Channel Device Driver SCSI Protocol Fibre Channel Fiber Optic or Copper Cabling Introduction To Storage: Overview of the storage subsystem
SCSI vs. Fibre Channel Interface for internal storage to external disks Potential down time w/ SCSI Single bus RAID controller is SCSI hardware Standards: Ultra2 (80 MB/sec) Ultra 160 (160 MB/sec) Ultra 320 (320 MB/sec) Media specific (copper only) SCSI Limitations: Cables can t be any longer than 3 feet for single ended; 15 feet for LVD (low voltage differential) No more than 15 devices on a SCSI bus # of disk drives Used with SAN Lots of built-in redundancy with connections Redundant network HBA is fibre channel hardware Standards: FC1: 100 MB/sec FC2: 200 MB/sec Provides a data link layer above the physical interconnect Analogous to Ethernet FC is a network of devices It can be media independent- copper or fibre optic Fibre Channel limitations: Cable length: Up to 10 kilometers (more a limitation of cable than FC itself) Up to 127 devices # of disk drives Introduction To Storage: Overview of the storage subsystem
iscsi
iscsi: what is it? An alternate form of networked storage Utilizes a TCP/IP network Encapsulates native SCSI commands in TCP/IP packets Supported in Windows Server and Linux TCP/IP Offload Engines (TOEs) on NICs speed up packet encapsulation Introduction To Storage: Overview of the storage subsystem
iscsi storage NIC or iscsi HBA Server Public or Private Ethernet network SCSI commands are encapsulated in TCP/IP packets Server has a Network Interface Card or iscsi HBA iscsi HBAs use TCP/IP Offload Engine (TOE) NIC or iscsi HBA iscsi Storage Disk Drives RAID Controller Introduction To Storage: Overview of the storage subsystem
IP storage networking IP storage networking provides solution to carry storage traffic within IP Uses TCP: a reliable transport for delivery Applicable to local data center and long-haul applications Two primary protocols: iscsi Internet-SCSI used to transport SCSI CDBs and data within TCP/IP connections IP TCP iscsi SCSI Data FCIP Fibre-Channel-over-IP used to transport Fibre Channel frames within TCP/IP connections any FC frame not just SCSI IP TCP FCIP FC SCSI Data IP Storage Networking FCIP/iSCSI. Steve Tegeler
SCSI architectural model transports SCSI Applications (File Systems, Databases) SCSI Device-Type Commands SCSI Block Commands SCSI Stream Commands Other SCSI Commands SCSI Generic Commands SCSI Commands, Data, and Status SCSI Transport Protocols Parallel SCSI Transport FCP SCSI over FC iscsi SCSI over TCP/IP Layer 3 Network Transport TCP IP Layer 2 Network Parallel SCSI Interfaces Fibre Channel Ethernet, PPP, HDLC IP Storage Networking FCIP/iSCSI. Steve Tegeler
iscsi for storage consolidation IP access to open systems iscsi and Fibre Channel storage iscsi iscsi iscsi-enabled Hosts (Initiators) iscsi iscsi driver is loaded onto hosts on Ethernet network Able to consolidate servers via iscsi onto existing storage arrays iscsi Array (Target) iscsi iscsi Gateway IP Network Able to build Ethernet-based SANs using iscsi arrays IP Storage Networking FCIP/iSCSI. Steve Tegeler FC Fabric Storage Pool (Target) FC HBA- Attached Host (Initiator)
iscsi for remote block access Block access to remote storage over IP Application must tolerate latency for long distances Metro Ethernet services offer lower-latency transport alternative Remote backup over IP WAN Centralized management from centralized storage IP Storage Networking FCIP/iSCSI. Steve Tegeler Site B Remote Mirrors Storage Pool Site A iscsi-enabled Host FC Fabric iscsi IP WAN iscsi Device
iscsi architecture: software driver Host iscsi GW Device iscsi Host Driver iscsi GW Module SCSI Driver TCP/IP Driver FC HBA GigE NIC Fibre Channel IP Network iscsi TCP/IP Stack NIC Driver NIC Applications Block Device File System SCSI Generic Adapter Driver SCSI Adapter IP Storage Networking FCIP/iSCSI. Steve Tegeler iscsi Path Conventional SCSI Path
OS support OS vendors support native iscsi drivers Applications iscsi Software Driver File System Block Device SCSI Generic iscsi TCP/IP Stack NIC Driver NIC Adapter Adapter Driver SCSI Adapter IP Storage Networking FCIP/iSCSI. Steve Tegeler
iscsi HBAs and TCP offload engines (TOEs) Offloads TCP and, optionally, iscsi processing into hardware Relieves host CPU from: TCP processing 16-bit checksum per packet iscsi optional 32-bit header and data digests (CRC32C) TCP/IP Stack NIC Driver iscsi Driver TOE Driver TCP/IP Stack Applications Block Device SCSI Generic HBA Driver iscsi TCP/IP Stack File System Adapter Driver SCSI Adapter TCP Offload iscsi and TCP Offload IP Storage Networking FCIP/iSCSI. Steve Tegeler
Fibre channel vs. iscsi Fibre Channel The current market leader for shared storage technologies Provides the highest performance levels Designed for mission-critical applications Cost of components is relatively high, particularly per server HBA costs Relatively difficult to implement and manage iscsi Performance can approach Fibre Channel speeds A better fit for databases than NAS A good fit for Small to Medium Size Businesses Relatively inexpensive, compared to Fibre Channel Relatively easy to implement and manage Introduction To Storage: Overview of the storage subsystem
CPU % Example performance impact on CPU utility % FC vs. iscsi TOE vs. iscsi SW Driver iscsi SW Driver iscsi TOE FC HBA Throughput MB/s
SAN
Storage Area Network SAN: what is it? A network whose primary purpose is the transfer of data between storage systems and computer systems Fibre Channel is the primary technology utilized for SANs Can be implemented with dedicated iscsi networks Introduction To Storage: Overview of the storage subsystem
Benefits of SAN/consolidated storage Reduce cost of external storage Increase performance Centralized and improved tape backup LAN-less backup High-speed Consolidation with > 70TB of storage Introduction To Storage: Overview of the storage subsystem
NAS: what is it? Network Attached Storage Utilizes a TCP/IP network to share data Uses file sharing protocols like Unix NFS Storage Appliances utilize a stripped-down OS that optimizes file protocol performance Introduction To Storage: Overview of the storage subsystem
Networked attached storage NIC Server Public or Private Ethernet network All data converted to file protocol for transmission (may slow down database transactions) Server has a Network Interface Card No RAID Controller or HBA in the server NIC NAS Server Storage Disk Drives RAID Controller Introduction To Storage: Overview of the storage subsystem
NAS vs. SAN NAS SAN Access Methods File access Disk block access Access Medium Ethernet Fiber Channel Architecture Decentralized Centralized Transport Protocol Layer over TCP/IP SCSI/FC and SCSI/IP Efficiency Less More Sharing and Access Control Good Poor Typical Applications Web Database Typical Clients Workstations Database servers
Storage Virtualization
What and where can storage be virtualized? SNIA Storage Model Storage Networking Industry Association 2 * 6 3 Potential Areas of Virtualization File Level Virtualization Host Level Virtualization Network Virtualization Storage Virtualization aims to provide a layer of abstraction to manage storage and reduce complexity!!! 4 * * 5 Block Virtualization Device Virtualization 1 Storage Level Virtualization * Host aka Server Source: The Storage Networking Tutorials, SNIAVIRT- Page 20 http://www.snia.org/education/tutorials/ ** Device=aggregation of Host and Network (Meta Data)
SNIA storage virtualization taxonomy What is created Storage Virtualization Block Virtualization Disk Virtualization Tape, Tape Drive, Tape Library Virtualization File System, File/record Virtualization Other Device Virtualization Where it is done Host Based Virtualization Network Based Virtualization Storage Device/Storage Subsystem Virtualization How it is implemented In-band Virtualization Out-of-band Virtualization Storage Virtualization. EMC. 2009
Storage virtualization requires a multi-level approach Server Path management Volume management Replication Storage Network Path redirection Load balancing - ISL trucking Access control - Zoning Storage Volume management - LUNs Access control Replication RAID Storage Virtualization Storage Virtualization. EMC. 2009
Storage virtualization configuration Servers Servers Virtualization Appliance Virtualization Appliance Storage Network Storage Network Storage Arrays Storage Arrays Out-of-Band In-Band (a) (a) In out-of-band implementation, the virtualized environment configuration is stored external to the data path (b) The in-band implementation places the virtualization function in the data path (b) Storage Virtualization. EMC. 2009 Storage Virtualization
Block-level storage virtualization Ties together multiple independent storage arrays Presented to host as a single storage device Mapping used to redirect I/O on this device to underlying physical arrays Deployed in a SAN environment Non-disruptive data mobility and data migration Enable significant cost and resource optimization Servers Virtualization Applied at SAN Level Heterogeneous Storage Arrays Storage Virtualization Storage Virtualization. EMC. 2009
File-level virtualization Before File-Level Virtualization Clients IP Network Clients Clients After File-Level Virtualization IP Network Clients Virtualization Appliance File Server Storage Array File Server File Server Storage Array File Server NAS Devices/Platforms Every NAS device is an independent entity, physically and logically Underutilized storage resources Downtime caused by data migrations NAS Devices/Platforms Break dependencies between end-user access and data location Storage utilization is optimized Nondisruptive migrations Storage Virtualization. EMC. 2009 Storage Virtualization
How is storage virtualized at the enterprise level? Currently Networks are virtualized using Metadata or Storage Volume Controllers. There are two types of network virtualization Metadata or Storage Volume Controllers (SVC) are placed (inband) or in the path of data flow. Metadata or Storage Volume Controllers are placed (out of band) outside the path of data flow. Source: IBM Redbook Page 8 http://www.redbooks.ibm.com/redbooks/pdfs/sg246210.pdf