COSC6376 Cloud Computing Lecture 17: Storage Systems

Size: px

Start display at page:

Download "COSC6376 Cloud Computing Lecture 17: Storage Systems"

Hillary Dalton
6 years ago
Views:

1 COSC6376 Cloud Computing Lecture 17: Storage Systems Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston

2 Storage Area Network and Storage Virtualization

3 Single Disk Drive Storage Array SCSI device DAS NAS SAN iscsi Storage types Introduction To Storage: Overview of the storage subsystem

4 Direct attached storage (internal) Computer System CPU Memory Bus I/O - RAID Controller Disk Drives Introduction To Storage: Overview of the storage subsystem

5 Direct attached storage (internal) Data Computer System CPU Memory Bus I/O - RAID Controller Disk Drives Introduction To Storage: Overview of the storage subsystem

6 Direct attached storage (internal) Computer System CPU Memory Bus I/O - RAID Controller Disk Drives Introduction To Storage: Overview of the storage subsystem

7 DAS w/internal controller and external storage Computer System CPU Memory Bus I/O - RAID Controller Disk Enclosure Disk Drives Disk Drives Disk Drives Introduction To Storage: Overview of the storage subsystem

400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB PCI-X SATA Controller PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400

8 Attach large number of disks 4-Processor AMD Dual Core Opteron can handle 2.5 GB/s or more sequential read and write data traffic Each controller connects 8 point-to-point SATA Drives Average 400 MB/s/card PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB PCI-X SATA Controller PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 16 TB Raw Disk Space 12 TB RAID 5 PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 8 TB Mirrored PCI-X SATA Controller 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB 400 GB

9 Backblaze

10 Backblaze Backblaze Storage Pods: 67 terabyte 4U servers

11 Comparing internal and external storage Server Storage Disk Drives RAID Controller Internal Storage RAID controllers and disk drives are internal to the server SCSI, ATA, or SATA protocol between controller and disks RAID controller is internal SCSI or SATA protocol between controller and disks Disk drives are external Server RAID Controller Storage Disk Drives SCSI Bus w/ external storage Introduction To Storage: Overview of the storage subsystem

12 Direct attached storage Direct Attached Storage (DAS) Storage is captive behind the server Server CPU must handle user I/O requests, but also: User-database inquiries User file/print serving Data-integrity checking Communication with other devices Data access is file system and platform dependent Costly to scale; complex to manage Clients IP Network Servers Win Linux Win Linux Unix FC FC SCSI Direct-Attached Storage (DAS)

13 DAS w/external controller and external storage Storage System Computer System CPU Memory Bus Disk Enclosure Disk Drives Disk Drives Disk Drives HBA RAID Controller Introduction To Storage: Overview of the storage subsystem

14 DAS over Fibre channel Server HBA HBA is internal Fibre Channel protocol between HBAs and external RAID controller Disk drives and RAID controller are external Storage Disk Drives RAID Controller Introduction To Storage: Overview of the storage subsystem External SAN Array

15 I/O Transfer RAID Controller Contains the smarts Determines how the data will be written (striping, mirroring, RAID 10, RAID 5, etc.) Host Bus Adapter (HBA) Simply transfers the data to the RAID controller. Doesn t do any RAID or striping calculations. Dumb for speed. Required for external storage. Introduction To Storage: Overview of the storage subsystem

16 SCSI Small Computer System Interface

17 Terminology Initiator: A SCSI device that requests an operation to be performed by another SCSI device. Host Adapter: A Controller Board that translates host I/O requests into SCSI requests. Target: A SCSI device that performs an operation as requested by an initiator. Logical Unit: A Physical or Virtual device addressable through a target. LUN: Logical Unit Number. An encoded three-bit identifier for the logical unit.

18 SCSI configuration single initiator, single target LUN0 Initiator SCSI Bus Target LUN1 LUN2

19 SCSI configuration Single initiator, multiple targets Initiator 7 T Target 0 SCSI Bus Target 1 SCSI Bus Terminator T Target n

20 SCSI configuration Multiple initiators, multiple targets Initiator 7 T Target 0 SCSI Bus Target 1 Initiator 6 T Target n

The SCSI I/O Channel SCSI is the dominant protocol used to communicate between servers and storage devices in open system SCSI I/O channel is a half-duplex pipe for SCSI CDBs and data Parallel bus

5m 25m A network approach can scale the I/O channel in many areas (length, devices, speed) TCP/IP Stack NIC Driver Ethernet NIC Applications Raw File System Block Device SCSI Generic Adapter Driver

21 The SCSI I/O Channel SCSI is the dominant protocol used to communicate between servers and storage devices in open system SCSI I/O channel is a half-duplex pipe for SCSI CDBs and data Parallel bus evolution Bus width: 8, 16 bits Bus speed: 5 80 Mhz Throughput: MBps Devices/bus: 2 16 devices Cable length: 1.5m 25m A network approach can scale the I/O channel in many areas (length, devices, speed) TCP/IP Stack NIC Driver Ethernet NIC Applications Raw File System Block Device SCSI Generic Adapter Driver SCSI Adapter Half-Duplex SCSI I/O Channel SCSI Ethernet SCSI CDB: SCSI Command Descriptor Block Used to Relay SCSI Commands, Parameters, and Status between SCSI Initiators and SCSI Targets; Typically 6, 10, or 12 Byte Block IP Storage Networking FCIP/iSCSI. Steve Tegeler SCSI Initiator SCSI Target

22 Networked I/O Channel Networking the I/O Channel Same SCSI protocol (SCSI-3) carried over a network transport layer via serial implementation Transport must not jeopardize SCSI payload (security, integrity, latency) Two primary transports to choose from today: Fibre Channel and IP A networked I/O channel allows for multiple improvements: Distance limitations greatly increased High number of addressable devices Initiator Host System SCSI Channel Controller Network IP Storage Networking FCIP/iSCSI. Steve Tegeler Target and LUNs

23 Fibre channel networking Very common method for networking SCSI Fibre Channel provides high-speed transport for SCSI payload Fibre Channel SAN overcomes many shortcomings of DAS including: Addressing for up to 16-million nodes (24 bits) Loop (shared) and Fabric (switched) transport Speeds of 100 or 200 MBps (1 or 2 Gbps) Distance of up to 10km (without extenders) Support for multiple protocols Combines best attributes of a channel and a network Initiator Host System SCSI Fibre Channel HBA Fibre Channel Fabric Target IP Storage Networking FCIP/iSCSI. Steve Tegeler

24 IP: an alternate I/O transport Viable transport for I/O traffic Similar characteristics to Fibre Channel: Addressing for close to 4 billion nodes (IPv4) Primarily a switched transport (with routing) Ethernet speeds of 1/10 Gbps or various WAN speeds Support for multiple high-level protocols Cost and manageability advantages with IP IP knowledge base widespread in industry Initiator Host System SCSI IP Channel Adapter IP Network Target IP Storage Networking FCIP/iSCSI. Steve Tegeler

25 Fibre Channel

26 Fibre channel: what is it? Fibre Channel is a network protocol implemented specifically for dedicated storage networks Fibre Channel utilizes specialized Switches Host Bus Adapters RAID controllers Cables Introduction To Storage: Overview of the storage subsystem

27 Fibre channel components Server A Server B Server C HBA HBA HBA HBA HBA HBA switch switch Disk Drives RAID Controller RAID Controller FC Storage Array Servers Host Bus Adapters Cables Fiber optic or copper Fibre Channel Switches Two switches for redundancy Fibre Channel Storage Array Two RAID Controllers for redundancy disk drives per array A true storage network Multiple servers Multiple switches Multiple Storage Arrays Introduction To Storage: Overview of the storage subsystem

28 What is fibre channel? A high-speed transmission technology used as a peripheral channel or network backbone. FC1: Over 1G bits per second FC2: Over 2G bits per second It supports several common transport protocols like Internet Protocol (IP) and SCSI. It operates over copper and fiber optic cables at distances of up to 10 Kilometers. Up to 127 devices (SCSI: 15) Up to 10 km of cabling (3-15 ft. for SCSI) Physical interconnect can be copper or fiber optic It is supported by many suppliers.

29 Fibre channel (continued) How does it work? Serial interface Data is transferred across a single piece of medium at the fastest speed supported No complex signaling required Hot-pluggable Devices can be removed or added at will with no ill effects to data communications Provides a data link layer above the physical interconnect, analogous to Ethernet Sophisticated error detection at the frame level Data is checked and resent if necessary Introduction To Storage: Overview of the storage subsystem

30 Current standards: Performance from 266 megabits/second to over 4 gigabits/second High-bandwidth utilization with distance insensitivity. Support for multiple cost/performance levels, from small systems to super computers Ability to carry multiple existing interface command sets, including Internet Protocol (IP), SCSI, and audio and video

31 Fibre channel frame dissection Up to 2048 byte payload 4 byte checksum for each frame Introduction To Storage: Overview of the storage subsystem

32 Fibre channel What s with the funny name? Some background history required Originally developed to only support fiber optic cabling When copper cabling support was added, ISO decided not to rename the technology ISO changed to the French spelling to reduce association with fiber optics only medium Introduction To Storage: Overview of the storage subsystem

33 Fibre channel interface layers RAID Controller Disk Drive Fibre Channel Device Driver SCSI Protocol Fibre Channel Fiber Optic or Copper Cabling Introduction To Storage: Overview of the storage subsystem

34 SCSI vs. Fibre Channel Interface for internal storage to external disks Potential down time w/ SCSI Single bus RAID controller is SCSI hardware Standards: Ultra2 (80 MB/sec) Ultra 160 (160 MB/sec) Ultra 320 (320 MB/sec) Media specific (copper only) SCSI Limitations: Cables can t be any longer than 3 feet for single ended; 15 feet for LVD (low voltage differential) No more than 15 devices on a SCSI bus # of disk drives Used with SAN Lots of built-in redundancy with connections Redundant network HBA is fibre channel hardware Standards: FC1: 100 MB/sec FC2: 200 MB/sec Provides a data link layer above the physical interconnect Analogous to Ethernet FC is a network of devices It can be media independent- copper or fibre optic Fibre Channel limitations: Cable length: Up to 10 kilometers (more a limitation of cable than FC itself) Up to 127 devices # of disk drives Introduction To Storage: Overview of the storage subsystem

35 iscsi

36 iscsi: what is it? An alternate form of networked storage Utilizes a TCP/IP network Encapsulates native SCSI commands in TCP/IP packets Supported in Windows Server and Linux TCP/IP Offload Engines (TOEs) on NICs speed up packet encapsulation Introduction To Storage: Overview of the storage subsystem

37 iscsi storage NIC or iscsi HBA Server Public or Private Ethernet network SCSI commands are encapsulated in TCP/IP packets Server has a Network Interface Card or iscsi HBA iscsi HBAs use TCP/IP Offload Engine (TOE) NIC or iscsi HBA iscsi Storage Disk Drives RAID Controller Introduction To Storage: Overview of the storage subsystem

IP storage networking IP storage networking provides solution to carry storage traffic within IP Uses TCP: a reliable transport for delivery Applicable to local data center and long-haul applications

38 IP storage networking IP storage networking provides solution to carry storage traffic within IP Uses TCP: a reliable transport for delivery Applicable to local data center and long-haul applications Two primary protocols: iscsi Internet-SCSI used to transport SCSI CDBs and data within TCP/IP connections IP TCP iscsi SCSI Data FCIP Fibre-Channel-over-IP used to transport Fibre Channel frames within TCP/IP connections any FC frame not just SCSI IP TCP FCIP FC SCSI Data IP Storage Networking FCIP/iSCSI. Steve Tegeler

39 SCSI architectural model transports SCSI Applications (File Systems, Databases) SCSI Device-Type Commands SCSI Block Commands SCSI Stream Commands Other SCSI Commands SCSI Generic Commands SCSI Commands, Data, and Status SCSI Transport Protocols Parallel SCSI Transport FCP SCSI over FC iscsi SCSI over TCP/IP Layer 3 Network Transport TCP IP Layer 2 Network Parallel SCSI Interfaces Fibre Channel Ethernet, PPP, HDLC IP Storage Networking FCIP/iSCSI. Steve Tegeler

40 iscsi for storage consolidation IP access to open systems iscsi and Fibre Channel storage iscsi iscsi iscsi-enabled Hosts (Initiators) iscsi iscsi driver is loaded onto hosts on Ethernet network Able to consolidate servers via iscsi onto existing storage arrays iscsi Array (Target) iscsi iscsi Gateway IP Network Able to build Ethernet-based SANs using iscsi arrays IP Storage Networking FCIP/iSCSI. Steve Tegeler FC Fabric Storage Pool (Target) FC HBA- Attached Host (Initiator)

iscsi for remote block access Block access to remote storage over IP Application must tolerate latency for long distances Metro Ethernet services offer lower-latency transport alternative Remote

41 iscsi for remote block access Block access to remote storage over IP Application must tolerate latency for long distances Metro Ethernet services offer lower-latency transport alternative Remote backup over IP WAN Centralized management from centralized storage IP Storage Networking FCIP/iSCSI. Steve Tegeler Site B Remote Mirrors Storage Pool Site A iscsi-enabled Host FC Fabric iscsi IP WAN iscsi Device

42 iscsi architecture: software driver Host iscsi GW Device iscsi Host Driver iscsi GW Module SCSI Driver TCP/IP Driver FC HBA GigE NIC Fibre Channel IP Network iscsi TCP/IP Stack NIC Driver NIC Applications Block Device File System SCSI Generic Adapter Driver SCSI Adapter IP Storage Networking FCIP/iSCSI. Steve Tegeler iscsi Path Conventional SCSI Path

43 OS support OS vendors support native iscsi drivers Applications iscsi Software Driver File System Block Device SCSI Generic iscsi TCP/IP Stack NIC Driver NIC Adapter Adapter Driver SCSI Adapter IP Storage Networking FCIP/iSCSI. Steve Tegeler

iscsi HBAs and TCP offload engines (TOEs) Offloads TCP and, optionally, iscsi processing into hardware Relieves host CPU from: TCP processing 16-bit checksum per packet iscsi optional 32-bit header

44 iscsi HBAs and TCP offload engines (TOEs) Offloads TCP and, optionally, iscsi processing into hardware Relieves host CPU from: TCP processing 16-bit checksum per packet iscsi optional 32-bit header and data digests (CRC32C) TCP/IP Stack NIC Driver iscsi Driver TOE Driver TCP/IP Stack Applications Block Device SCSI Generic HBA Driver iscsi TCP/IP Stack File System Adapter Driver SCSI Adapter TCP Offload iscsi and TCP Offload IP Storage Networking FCIP/iSCSI. Steve Tegeler

45 Fibre channel vs. iscsi Fibre Channel The current market leader for shared storage technologies Provides the highest performance levels Designed for mission-critical applications Cost of components is relatively high, particularly per server HBA costs Relatively difficult to implement and manage iscsi Performance can approach Fibre Channel speeds A better fit for databases than NAS A good fit for Small to Medium Size Businesses Relatively inexpensive, compared to Fibre Channel Relatively easy to implement and manage Introduction To Storage: Overview of the storage subsystem

46 CPU % Example performance impact on CPU utility % FC vs. iscsi TOE vs. iscsi SW Driver iscsi SW Driver iscsi TOE FC HBA Throughput MB/s

47 SAN

48 Storage Area Network SAN: what is it? A network whose primary purpose is the transfer of data between storage systems and computer systems Fibre Channel is the primary technology utilized for SANs Can be implemented with dedicated iscsi networks Introduction To Storage: Overview of the storage subsystem

49 Benefits of SAN/consolidated storage Reduce cost of external storage Increase performance Centralized and improved tape backup LAN-less backup High-speed Consolidation with > 70TB of storage Introduction To Storage: Overview of the storage subsystem

50 NAS: what is it? Network Attached Storage Utilizes a TCP/IP network to share data Uses file sharing protocols like Unix NFS Storage Appliances utilize a stripped-down OS that optimizes file protocol performance Introduction To Storage: Overview of the storage subsystem

51 Networked attached storage NIC Server Public or Private Ethernet network All data converted to file protocol for transmission (may slow down database transactions) Server has a Network Interface Card No RAID Controller or HBA in the server NIC NAS Server Storage Disk Drives RAID Controller Introduction To Storage: Overview of the storage subsystem

52 NAS vs. SAN NAS SAN Access Methods File access Disk block access Access Medium Ethernet Fiber Channel Architecture Decentralized Centralized Transport Protocol Layer over TCP/IP SCSI/FC and SCSI/IP Efficiency Less More Sharing and Access Control Good Poor Typical Applications Web Database Typical Clients Workstations Database servers

53 Storage Virtualization

54 What and where can storage be virtualized? SNIA Storage Model Storage Networking Industry Association 2 * 6 3 Potential Areas of Virtualization File Level Virtualization Host Level Virtualization Network Virtualization Storage Virtualization aims to provide a layer of abstraction to manage storage and reduce complexity!!! 4 * * 5 Block Virtualization Device Virtualization 1 Storage Level Virtualization * Host aka Server Source: The Storage Networking Tutorials, SNIAVIRT- Page 20 ** Device=aggregation of Host and Network (Meta Data)

55 SNIA storage virtualization taxonomy What is created Storage Virtualization Block Virtualization Disk Virtualization Tape, Tape Drive, Tape Library Virtualization File System, File/record Virtualization Other Device Virtualization Where it is done Host Based Virtualization Network Based Virtualization Storage Device/Storage Subsystem Virtualization How it is implemented In-band Virtualization Out-of-band Virtualization Storage Virtualization. EMC. 2009

56 Storage virtualization requires a multi-level approach Server Path management Volume management Replication Storage Network Path redirection Load balancing - ISL trucking Access control - Zoning Storage Volume management - LUNs Access control Replication RAID Storage Virtualization Storage Virtualization. EMC. 2009

57 Storage virtualization configuration Servers Servers Virtualization Appliance Virtualization Appliance Storage Network Storage Network Storage Arrays Storage Arrays Out-of-Band In-Band (a) (a) In out-of-band implementation, the virtualized environment configuration is stored external to the data path (b) The in-band implementation places the virtualization function in the data path (b) Storage Virtualization. EMC Storage Virtualization

Block-level storage virtualization Ties together multiple independent

redirect I/O on this device to underlying physical arrays Deployed in a SAN

significant cost and resource optimization Servers Virtualization Applied at

58 Block-level storage virtualization Ties together multiple independent storage arrays Presented to host as a single storage device Mapping used to redirect I/O on this device to underlying physical arrays Deployed in a SAN environment Non-disruptive data mobility and data migration Enable significant cost and resource optimization Servers Virtualization Applied at SAN Level Heterogeneous Storage Arrays Storage Virtualization Storage Virtualization. EMC. 2009

59 File-level virtualization Before File-Level Virtualization Clients IP Network Clients Clients After File-Level Virtualization IP Network Clients Virtualization Appliance File Server Storage Array File Server File Server Storage Array File Server NAS Devices/Platforms Every NAS device is an independent entity, physically and logically Underutilized storage resources Downtime caused by data migrations NAS Devices/Platforms Break dependencies between end-user access and data location Storage utilization is optimized Nondisruptive migrations Storage Virtualization. EMC Storage Virtualization

60 How is storage virtualized at the enterprise level? Currently Networks are virtualized using Metadata or Storage Volume Controllers. There are two types of network virtualization Metadata or Storage Volume Controllers (SVC) are placed (inband) or in the path of data flow. Metadata or Storage Volume Controllers are placed (out of band) outside the path of data flow. Source: IBM Redbook Page 8

Storage Area Network (SAN)

Storage Area Network (SAN) 1 Outline Shared Storage Architecture Direct Access Storage (DAS) SCSI RAID Network Attached Storage (NAS) Storage Area Network (SAN) Fiber Channel and Fiber Channel Switch 2