Storage Area Networks SAN. Shane Healy

Transcription

1 Storage Area Networks SAN Shane Healy

2 Objective/Agenda Provide a basic overview of what Storage Area Networks (SAN) are, what the constituent components are, and how these components fit together to deliver high availability data storage. Core Components High Availability

3 What is a SAN? SAN: Storage Area Network A A network network whose whose primary primary purpose purpose is is the the transfer transfer of of data data between between computer computer systems systems and and storage storage elements elements and and among among storage storage elements. elements. The The combination combination of of aa high-speed high-speed transport transport with with Fibre/SCSI Fibre/SCSI protocol protocol results results in in an an efficient efficient means means to to deploy deploy servers, servers, disk disk arrays, arrays, and and tape tape subsystems subsystems and and frees frees all all components components from from the the constraints constraints of of parallel parallel SCSI SCSI architecture. architecture. SANs SANs are are extremely extremely high-speed, high-speed, back-end back-end networks networks dedicated dedicated to to storage. storage. SANs SANs provide provide raw raw storage storage in in the the sense sense that that the the servers servers "see" "see" the the storage storage as as physical physical disk disk drives. drives. Not Not aa replacement replacement for for the the SCSI SCSI protocol. protocol. Fibre Fibre Channel Channel Protocol Protocol encapsulates encapsulates SCSI SCSI with with extra extra packets. packets. Each Each SCSI SCSI packet packet is is within within aa FCP FCP frame. frame.

4 SAN Components A SAN consists of: aa communication communication infrastructure, infrastructure, which which provides provides physical physical connections connections aa management management layer, layer, which which organizes organizes these these connections connections storage storage elements elements computer computer systems SAN is made up of: A A protocol protocol Fibre Fibre Channel Channel Protocol Protocol A A switching switching method method FC FC switches switches Storage Storage medium medium with with intelligent intelligent controllers Servers Servers for for managing managing the the data data flow flow A A physical physical connection connection scheme Fabrics Fabrics

5 What does a SAN look like? A SAN typically looks physically much like this. Note that the servers are connected to the storage through fibre channel switches, not directly attached. Clients General Purpose Network LAN/WAN/FDDI Heterogeneous Servers Storage Area Network (SAN) DLT Library Array Controller with Disks

6 Non SAN Environment General Purpose Network LAN/MAN/WAN Heterogeneous Servers SCSI SCSI DLT Library Array Controllers with Disks Array Controller with Disks

7 What else can a SAN look like? Point-to-Point Arbitrated Loop Switched Fabric SAN Only 2 Devices Up to 126 Devices (Direct Connect) (Fibre Channel Hubs) Up to 16 Million Devices (Fibre Channel Switches)

8 Fibre Channel Topologies Point to Point Direct connection between server and storage via fibre cable(s) IDC Fibre Cable Server

9 Fibre Channel Loop (arbitrated loop) Data General Data General Fibre Channel HUB Resembles a ring or FDDI connections Usually uses a hub for arbitration EachDevicetakes turn tothe loopby an arbitrationschema

10 Fabric Switch provides addressing to specific LUN (Logical Unit Number or Disk drives) Data General Data General FIBRE CHANNEL SWITCH ROUTING IS DEPENDANT UPON A INTELLIGENT SWITCH VIA ADDRESSES ON THE FABRIC A MANY TOMANY RELATIONSHIP

11 What makes up a SAN? Servers Host Bus Adapters (HBA) Gigabit Interface Converters (GBIC) Cabling Switches Zoning Storage Controllers Disk Arrays Software

12 Servers They usually have some onboard disks for the operating system, but also have the ability to boot directly from the SAN. Typically at present in Intel Servers have a mirrored pair for the OS. On VMS they boot directly from the SAN. Multiple OS types are supported e.g. Windows, UNIX, Solaris, VMS etc. Blade technologies are being introduced in Intel where a server is essentially a motherboard. Currently Intel Primarily use HP servers, however multiple hardware platforms are supported e.g. IBM, DELL etc.

13 HBAs An HBA is a Host Bus Adapter It sits in the PCI (or PCIX) bus of a server and converts electrical signals into modulated laser signals. HBAs have replaceable GBICs (Gigabit Interface Converter) which contain the laser and receiver. The fiber cables plug into the GBIC, one to the transmit side, one to the receive side. Servers can (and typically do) have more than one HBA.

14 Fibre Channel Cables Cabling At At Intel, Intel, cabling cabling aa SAN SAN requires requires the the use use of of fiber fiber optic optic cables. cables. We We don't don't use use any any copper copper fibre fibre channel channel due to to a 30M 30M Limitation. Limitation. Fiber Fiber optic optic cable cable comes comes in in three three types types micron micron multi-mode multi-mode micron micron multi-mode multi-mode 99 micron micron single single mode mode We We use use different different cable cable types types to to accomplish accomplish different different objectives, objectives, mostly mostly to to address address distance distance requirements. requirements. Optical Core Size 50 micron 62.5 micron 9 micron Transport Mode Multi-mode Multi-mode single-mode Wavelength FC short wave FC short wave FC long wave Distance Limit 500 meters 175 meters 10,000 meters Color of Cover orange orange yellow

15 Multimode fiber Is optical fiber that is designed to carry multiple light rays or modes concurrently, each at a slightly different reflection angle within the optical fiber core. Multimode fiber transmission is used for relatively short distances because the modes tend to disperse over longer lengths (this is called modal dispersion). Single mode fiber Is optical fiber that is designed for the transmission of a single ray or mode of light as a carrier and is used for long-distance signal transmission. Single mode fiber has a much smaller core than multimode fiber.

16 Fibre Channel Cables Short Wave/Multi-mode 50 Micron Short Wave/Multi-mode Long Wave/Single-mode 62.5 Micron 9 Micron 125 micron cladding 50 micron core Orange Protective Cover Cladding is a reflective material that causes the light waves to be confined within the core material

17 Switches Fibre Fibre Channel Channel switches switches are are very very similar similar to to network network switches. switches. They They enable enable attached attached SAN SAN devices devices to to communicate communicate on on aa virtual virtual circuit circuit at at maximum maximum speed. Unlike Unlike aa hub hub or or aa fibre fibre channel channel loop, loop, switches switches do do not not share share bandwidth bandwidth between between all all connected connected devices. devices. We currently use Brocade switches at Intel. Brocade Silkworm series 2400, 2800, 3800, 3900, and Transport speeds of either 1 GB/sec or 2 GB/sec Future implementations will support 10 GB/sec. Each switch can either stand alone or become part of a "fabric," or a group of switches forming a single virtual switch.

18 Switched Fabrics Switches Switches can can be be joined joined together together to to form form aa "fabric," or or aa group group of of switches switches forming aa single single virtual switch. switch. Fabrics Fabrics can can be be configured configured to to support support many many security security features features such such as as zoning zoning and and fabric fabric logins. logins. Switches Switches are are combined by simply plugging plugging fiber fiber cables cables into into them, them, connecting connecting the the switches switches together. together. They then autosense autosense the the fabric fabric topology topology and and assume assume their their role in in it. it.

19 Switching (Physical) Physical Physical Topology Topology The physical The physical topology topology can can be be described described as as actual actual hardware hardware components components in in aa fabric fabric and and the the Fibre Fibre Channel Channel cabling cabling that that interconnects interconnects them. them. Design Design Considerations Considerations and and Topology Topology of of aa fabric fabric are: are: Number Number of of switches switches in in the the fabric fabric Number of hops between any Number of hops between any two two switches switches Number Number of of ports ports per per switch switch Number Number of of ISLs ISLs (Inter (Inter Switch Switch Links) Links) between between switches switches Physical Physical distance distance between between any any two two switches switches The number of tiers in the fabric is based The number of tiers in the fabric is based on on the the number number of of switches switches that that are are traversed traversed between between the the farthest farthest two two points points in in the the fabric. fabric. As As the the number number of of Tiers Tiers increase increase the the distance distance that that aa fabric fabric management management message message must must travel travel to to reach reach every switch in the fabric increases. It affects the time it takes to propagate and complete a fabric every switch in the fabric increases. It affects the time it takes to propagate and complete a fabric reconfiguration reconfiguration event event (e.g. (e.g. adding adding aa new new switch), switch), or or zone zone set set propagation propagation event. event.

20 Switching (Logical) Logical Logical Topology Topology In In contrast contrast to to aa fabric's fabric's physical physical topology, topology, aa logical logical topology topology is is concerned concerned with with where where the the Fibre Channel elements are attached around the fabric and the relationships (zoning) Fibre Channel elements are attached around the fabric and the relationships (zoning) that that define define how how these these elements elements will will be be used used together. together. When When describing describing aa fabric's fabric's logical logical topology, topology, itit is is in in terms terms of of its its number number of of logical logical tiers. tiers. The The number number of of logical logical tiers tiers in in the the topology topology is is based based on on the the number number of of switches switches traversed traversed between a server and the storage zoned to it. between a server and the storage zoned to it. Since Since servers servers and and storage storage can can be be located located anywhere anywhere in in the the fabric, fabric, there there may may be be several several different logical tier relationships in the same fabric. different logical tier relationships in the same fabric. Below Below demonstrates demonstrates an an example example of of multiple multiple logical logical tier tier relationships relationships in in the the same same fabric. fabric.

21 Zoning Zoning Zoning is is aa method method of of providing access access control in a Fibre Channel fabric fabric consisting consisting of of a single Fibre Channel switch switch or or cascaded cascaded switches. switches. Within Within the the fabric, fabric, the the SAN SAN administrator administrator can can group group server server HBA HBA ports ports (initiators) (initiators) with with storage storage ports ports (disk (disk or or tape tape drives) into zones. zones. The The individual individual switch switch ports ports or or device device World World Wide Wide Name(s) Name(s) are are referred referred to to as as zone zone members. members. A A zone zone effectively effectively partitions partitions a Fibre Channel Channel fabric fabric into into aa subset subset of of logical logical entities, entities, within within which which each each participant participant is able to connect and communicate communicate only only with with other other members members of of the the same same zone. zone.

22 Storage Arrays The The storage storage array array is is really really the the core core component component of of today s today s storage storage area area network. network. ItIt contains contains many, many, many many disks disks grouped grouped into into RAID RAID sets sets assigned assigned to to hosts hosts on the SAN They They are are typically typically the the size size of of aa refrigerator, refrigerator, but can can be be smaller smaller or or larger, larger, depending depending on on capacity. capacity. Storage Storage arrays arrays range range from from very very simple simple JBOD JBOD (Just (Just aa Bunch Bunch Of Of Disks) Disks) to to sophisticated sophisticated RAID RAID controlled controlled smart storage storage that that is is aware aware of of and and communicates communicates with with the the connected connected hosts. hosts.

23 RAID Short for Redundant Array of Independent (or Inexpensive) Disks, a category of disk drives that employ two or more drives in combination for fault tolerance and performance. RAID disks are then divided into LUN s (Logical Unit numbers) and presented to servers as partitions or raw space. There are number of different RAID levels: Disk Disk No No protection protection JBOD JBOD Some Some performance performance gain gain by by splitting splitting read read operations operations Protection against single disk failure Protection against single disk failure Minimum Minimum performance performance hit hit during during failure failure Performance Performance of of stripes stripes combined combined with with split split read read operations operations Protection Protection against against single single disk disk failure failure Minimum Minimum performance performance hit hit in in failure failure mode mode RAID-5: Striping with Parity RAID-1: RAID-1: Mirroring Mirroring RAID RAID 1/0: 1/0: Striped Striped Mirrors Mirrors RAID-3: RAID-3: Striped Striped Elements Elements Each Each data data element element striped striped across across disks disks -- parity parity kept kept on on the the last last disk disk in in the the RAID RAID group group Extremely Extremely fast fast read read access access from from the the disk disk Used for streaming media Used for streaming media Parity Parity protection protection against against single single disk disk failure failure Performance Performance penalty penalty during during failure failure Performance of striping Protection from single disk failure Parity distributed across member drives within the RAID Group Write performance penalty Performance impact if a disk fails in RAID Group Hot Spare Takes the place of failed disk within a RAID group Must have equal or greater capacity than the disk it replaces When failing disk is replaced, the hot spare restores the data to the replacement disk and returns to the hot spare pool

24 RAID

25 Software This This is is the the glue glue that that holds all all the pieces pieces together and makes the system work. work. There There is is software installed installed on: on: The The server server ItIt is is made made up up of of drivers drivers and and software software that that acts acts as as aa link link in in the the operating operating system system to to create create the the illusion illusion that that the the HBA HBA is is just just another another SCSI SCSI card card connected connected to to a dedicated dedicated disk. disk. The The switches switches The The switches switches all all have have software software that that enables enables them them to to route route requests requests between between hosts hosts and and storage. storage. The The storage storage frame frame The The storage storage frame frame can can have have sophisticated sophisticated software software that that is is used used to to create create RAID RAID groups, groups, Luns Luns (virtual (virtual disks), disks), and and assign assign storage storage to to hosts. hosts.

26 Server Software HBA HBA drivers drivers used used by by the the operating operating system system to to communicate communicate with the HBA installed in in the the PCI PCI slot. slot. ItIt makes makes the the HBA HBA look look to to the the system system like like aa SCSI SCSI card card and and enables enables the the out-of-the-box out-of-the-box operating operating system system software software to to issue issue SCSI SCSI commands commands over over the the fibre fibre channel channel network. network. Agent Agent software software an an optional optional component component that that not all SAN SAN vendors require. require. ItIt is is usually usually aa small small piece piece of server server code code installed installed on on the the server server to to communicate communicate through through the the fibre fibre channel channel network network or or through through the the Ethernet Ethernet network network with with the the storage storage frame. ItIt enables the storage frame frame to know about the server server so so that that itit can can assign assign storage and manage manage resources resources belonging belonging to to the the server. server. Multipath Multipath software software such such as as Secure Secure Path Path (HP) (HP) or or PowerPath PowerPath (EMC) (EMC) is is used used to to coordinate coordinate communication communication between between two two separate separate HBAs HBAs installed installed in in the the server. server. This This allows allows aa redundant redundant path path between between the the server server and and the the storage storage and and increases increases availability. availability.

27 Switch Software Software Software running on the switches is usually usually known known as as firmware. Responsible Responsible for for coordinating coordinating all all the the traffic traffic going going on on between between servers servers and and storage, storage, between between the the switches switches themselves, themselves, and and managing managing security. security. Our Our Brocade Brocade switches switches have have both both Telnet Telnet and and Web Web Server Server components components and and can can be be managed managed remotely remotely through through aa TCP/IP TCP/IP network network connection connection and and aa web web browser browser or or Telnet Telnet window. window. Switches Switches also also have have an an available available API API (application (application programming programming interface) interface) which which enables enables the the switches switches to to be be controlled controlled programmatically programmatically and and monitored. monitored.

28 Storage Software Storage Storage frames frames can can be be as as simple simple as as aa connected connected JBOD JBOD unit unit with with nothing nothing but but aa group group of of disks or or as complicated as as aa million million dollar dollar storage storage array array with with sophisticated SCSI SCSI or or Fibre Fibre Channel Channel controllers controllers and and disks disks Most Most of of what what is is deployed deployed at at intel intel has has aa high high level level of of intelligence intelligence and and sophistication. sophistication. For For example, example, Clariion Clariion storage storage arrays ship with a built in web server server component component that that can can be be used used to to manage manage the the entire entire frame frame or or even even an an entire entire enterprise enterprise of storage. Behind Behind the web server server is aa fully fully redundant redundant set set of of fibre fibre channel channel based based RAID RAID controllers controllers that that enable enable disks disks to to be be combined combined into into RAID RAID sets, carved into into virtual virtual disks, disks, and and assigned assigned to to hosts hosts (known (known to to the the array array through through agent agent software). software). On On HP HP storage storage typically typically a SAN SAN Appliance Appliance is is used used to to run run the management management software software and and provides provides the the same tasks. tasks.

29 Management Software The last piece of the puzzle. Means Means different different things things to to different different people, people, but but itit encompasses encompasses the the entire entire range range of of software software to to manage manage aa SAN SAN simple simple monitoring monitoring system health health provisioning provisioning (creating (creating and and assigning assigning storage) storage) tracking tracking and trending storage consumption Today Today there there are are few, few, ifif any, any, all-encompassing all-encompassing tools tools for for managing managing the the entire entire Intel Intel enterprise enterprise SAN, SAN, but these these tools tools are beginning to emerge. emerge.

30 High Availability Robust Robust SAN SAN designs designs include include at at least least two two data data paths paths between between any any server server and and disk disk storage storage subsystem, subsystem, such such as as RAID/Controllers RAID/Controllers etc. etc. By By using using redundant redundant data data paths paths in in conjunction conjunction with with FC FC director director switches, switches, aa true true redundant redundant environment environment can can be be achieved. achieved. Additionally, Additionally, the the any-to-any any-to-any connectivity connectivity of of SAN SAN devices devices gives gives the the flexibility flexibility and and portability portability for for data data requirements requirements in in dynamic dynamic environments. environments. ItIt can can provides provides redundancy redundancy in in SAN s SAN s and and still still achieves achieves data data availability availability even even in in the the event event of of multiple multiple component component failures. failures. Storage Storage frames frames and and servers servers typically typically include include dual dual redundant redundant devices, devices, e.g. e.g. Power Power supplies, supplies, HBA s, HBA s, Cables, Cables, Controllers Controllers etc. etc. Implementing Implementing High High Availability Availability supports supports non-disruptive non-disruptive upgrade upgrade (NDU) (NDU) of of storage storage system system software. software. While While aa server server is is running running normally. normally. IfIf aa failure failure occurs occurs e.g. e.g. aa controller controller failover, failover, software software such such as as Powerpath Powerpath or or SecurePath SecurePath attempts attempts to to fail fail over over (transfer) (transfer) the the I/Os I/Os to to aa different different controller controller or or Switch Switch port. port. Failover Failover Software Software (Powerpath/SecurePath) (Powerpath/SecurePath) does does not not protect protect against against single single HBA HBA failures. failures. To To protect protect against against such such failures failures dual dual HBA s HBA s are are required. required.

31 High Availability Setup 1 B A F E0 F E1 BE 0 BE1 F E0 Fabric B FE1 1 0 /1 0 ISL BE1 1 0 /1 0 Fabric A BE 0 ISL

32 High Availability HBA Failure 1 B A ISL s F E0 F E1 BE 0 BE1 F E0 FE1 1 0 /1 0 Fabric A BE1 1 0 /1 0 BE 0 Fabric B

33 High Availability Controller Failure ISL s B A BE 0 BE1 F E0 F E1 BE 0 BE1 F E0 FE1 1 0 /1 0 Fabric A /1 0 Embedded software performs internal failover on storage backplane Fabric B

34 High Availability Switch Failure 1 B A F E0 F E1 BE 0 Fabric A ISL s BE1 F E0 FE1 1 0 /1 0 BE1 1 0 /1 0 BE 0 Fabric B!

35 So What else can a SAN do? Clustering! Clustering! Snapshots Snapshots Clones Clones Faster Faster backups backups Data Data mining mining Bunkering Bunkering Fast Fast failover failover MS MS Cluster Cluster server server requires requires aa shared shared disk. disk. SANs SANs enable enable the the sharing sharing of of disks disks between between multiple multiple servers. servers. Take Take an an instant instant snapshot snapshot of of your your data data for for use use as as aa point-in-time point-in-time copy. copy. Create Create aa bit-for-bit bit-for-bit copy copy of of your your data data that that can can be be assigned assigned to to other other servers. servers. Connecting Connecting aa large large tape tape drive drive array array enables enables backups backups over over fibre fibre channel channel instead instead of of Ethernet Ethernet You You can can use use snapshots snapshots and and clones clones to to get get even even more more sophisticated. sophisticated. By By creating creating aa copy copy of of your your data, data, you you can can mine mine itit for for information information without without affecting affecting the the performance of the live data. performance of the live data. Create Create aa synchronous synchronous or or asynchronous asynchronous copy copy of of all all your your server s server s data data in in aa remote remote location location in in case of disaster. case of disaster. Having Having servers servers separated separated from from their their storage storage enables enables the the LUNs LUNs (disk (disk drives) drives) to to be be reassigned reassigned to to aa new new server server very very quickly. quickly.

36 SAN vs. Network Attached Storage (NAS) SAN SAN -- block block level level I/O I/O NAS NAS (Network (Network Attached Attached Storage) Storage) -- file file level level I/O I/O NAS NAS could could use use SAN SAN for for storage storage SAN SAN accommodates accommodates file file servers, servers, web web servers, servers, databases, databases, application application servers, servers, etc. etc. SAN SAN uses uses SCSI SCSI over Fibre Channel today, separating separating storage storage traffic traffic from from network network traffic. traffic. NAS NAS operates operates like a shared file server, storage is is accessed accessed across across Ethernet, Ethernet, usually sharing bandwidth bandwidth with applications. applications. SAN SAN integrates integrates backup backup within within its its high high speed speed storage storage network. network. NAS NAS backs backs up up data data across across Ethernet although they can use SCSI or or Fibre. Fibre. Operating Operating System System dependant dependant

37 File I/O Server vs. Block I/O Server Architecture Clients Or Servers File I/O Processing IP Network Server Disk Arrays (Custom O/S) NAS Server Clients Or Servers IP Network File I/O Processing Server SAN Storage (P100 O/S) SAN Server Block I/O (Fibre Channel)

38 Ethernet LAN Network Attached Storage OS directed file transfer Server Network Attached Storage Subsystem File transfer via the server operating system

39 Questions?