Solutions for Disaster recovery:

Transcription

1 Solutions for Disaster recovery: SAN Extension Design Rajinder Singh 1

2 Agenda Basic SAN Extension Principles Dual path, HA options Alternate Designs Hub-spoke, multi-hop SAN Extension Choices and Enhancements Optical: Extended B2B credits, B2B credit spoofing, Porttracking IP: Compression, Encryption, Application Acceleration MDS 9000 Product Family 2

3 Design Criteria Factors to consider for SAN Extension Applications using SAN Extension Synchronous/Asynchronous Replication, Data Backup/ Restore Application latency requirements Applications that use synchronous replication may be impacted Application throughput requirements Determines bandwidth requirements Transport options available What choices are available for SAN Extension 3

4 Typical SAN Design Site A Site B DC Interconnect Network Server Access Replication Fabrics Replication Fabrics Servers with two FibreChannel connections to storage arrays for high availability Use of multipath software is required in dual fabric host design SAN extension fabrics typically separate from host access fabrics Replication fabric requirements generally specified by array vendor 4

5 Basic HA SAN Extension Network Site A Yellow VSAN Extended Over Distance Site B DC Interconnect Network Blue VSAN Extended Over Distance High-Availability Replication Design: Conventional approach is dual fabrics (e.g., yellow VSAN and blue VSAN) over distance Client protection arrays provide protection against failures in either fabric May be augmented with additional network protection via portchannels and/or optical protection schemes 5

6 Fabric Consolidation with VSANs Application/Department -based SAN Islands Department #1 VSAN Cisco MDS 9000 Family SAN Island for Department #1 Common Storage Pool Shared Amongst VSANs SAN Island for Department #2 Department #2 VSAN Department #3 VSAN SAN Island for Department #3 Separate physical fabrics Over-provisioning ports on each island High number of switches to manage Collapsed Fabric with VSANs Common redundant physical infrastructure Less over-provisioning required lower $$ Fewer switches to manage Move unused ports non-disruptively 6

7 VSANs and Zones Virtual SANs and fabric zoning are very complimentary Hierarchical relationship provision VSANs then assign independent zones per VSAN VSANs divide up the physical infrastructure Zones provide added security and allow sharing of device ports VSANs provide traffic statistics VSANs only changed when ports needed per virtual fabric Zones can change frequently (backup) Ports are added/removed nondisruptively to VSANs Relationship of VSANs to Zones Physical Topology VSAN 2 ZoneA ZoneB VSAN 3 ZoneA Disk2 Host2 Disk4 ZoneD Host1 Disk1 Disk3 Disk1 Host4 Host3 Disk2 ZoneC 7

8 Inter-VSAN Routing (IVR) : Sharing Resources Across VSANs Allows sharing of centralized storage services such as tape libraries and disks across VSANs without merging separate fabrics (VSANs) Provides high fabric resiliency and VSAN-based manageability Distributed, scaleable, and highly resilient architecture Transparent to third-party switches Enables blade-per-vsan architecture for blade servers Engineering VSAN_1 IVR VSAN-specifc Disk Blade Server Marketing VSAN_2 IVR Blade Server VSAN_1 (access via IVR) HR VSAN_3 Marketing VSAN_2 IVR HR VSAN_3 Tape VSAN_4 (access via IVR) 8

9 SAN Extension Design: Adding Link HA Site A Site B Portchannels Portchannels Increase Resilience for High-Availability with or IP Links Appears as a single logical link (up to sixteen member links) Protecting the fabric from network failure Route portchannel member links over diverse geographic paths Load balancing on SRCID/ DESTID or SRCID/DESTID/ OXID basis (unidirectional per VSAN) SCSI exchange is smallest atomic unit, so frame order kept intact 9

10 SAN Extension Design: Adding Fabric HA Site A Distribute Green and Blue VSANs Between All Switches Portchannels All portchannels trunk both VSANs TE_Port (EISL) Each VSAN protected by: Portchannels over diverse paths Multiple switches/directors Same per VSAN load balancing: SRCID/DESTID, or SRCID/DESTID/OXID Site B 10

11 Other SAN Extension Implementations Hub and Spoke IP Network Central Site Remote Sites Multi-hop Primary DC DWDM Secondary DC IP Network Backup Site Synchronous Replication Asynchronous Replication/Backup 11

12 SAN Extension Technology Options Increasing Distance Data Center Campus Metro Regional National Dark Fiber Sync Limited by Optics (Power Budget) Optical CWDM DWDM SONET/SDH Sync (1,2Gbps) Sync (1,2,10Gbps per λ) Sync (1,2Gbps + subrate) Limited by Optics (Power Budget) Limited by BB_Credits Async IP MDS9000 IP Sync (Metro Eth) Async (1Gbps) 12

13 Dark Fiber Distance Based on Fiber Type, Optic Type, Link Speed Portchannel 2 x 1/2/4/10 Gbps Over Two Diverse Paths Diverse Paths Multiple Fiber Pairs Each Path Single 1/2/4/10 Gbps link per fiber pair SW (850nm) over 62.5/125µm multimode SW (850nm) over 50/125µm multimode LW (1310nm) over 9/125µm single mode Client protection only; Upper Layer Protocol (ULP), either SAN or application, responsible for failover protection 13

14 Coarse Wavelength Division Multiplexing (CWDM) 1470nm 1510nm 1550nm 1590nm OADM Mux/Demux 1490nm 1530nm 1570nm 1610nm 8-channel WDM at 20nm spacing (cf DWDM at <1nm spacing) 1470, 1490, 1510, 1530, 1550, 1570, 1590, 1610nm Colored CWDM SFPs (or GBICs) used in switches (no transponder required) Optical multiplexing done in CWDM OADM (optical add/drop multiplexer) Passive (unpowered) device; just mirrors and prisms Up to 30dB power budget (36dB typical) on SM fiber ~100km point-to-point or ~40km ring 1/2 Gigabit Fibre Channel and 1 Gigabit Ethernet currently 14

15 2-Site CWDM Storage Network Diverse Paths: One Fiber Pair Each Path Network MUX-4 MUX-4 Pass Pass MUX-4 MUX-4 Portchannel 4 x 2 Gbps Over Two Diverse Paths Network 2 Gbps CWDM SFPs HA resilience against fiber cut client protection 4-member portchannel 2 x 2 diverse paths Portchannel appears as single logical link E_Port or TE_Port for carriage of VSANs Load balance by src/dst (or src/dst/oxid) Fiber cut will halve capacity from 8 Gbps to 4 Gbps but not alter fabric topology no FSPF change MUX-8 would double capacity or leave spare wavelengths for GigE channels 15

16 CWDM optics without Multiplexor 4 fibre paths between each switch 2 Gbps CWDM SFPs different wavelengths Point to Point Portchannel 4 x 2 Gbps 2 Gbps CWDM SFPs same wavelength CWDM Optics do not require MUX If dark fiber available, can be used like typical SFPs Can use different wavelengths or the same wavelengths on all interfaces Use of optical attenuators may be required for shorter distance fiber runs Optical power meter used to measure signal strength 16

17 Dense Wavelength Division Multiplexing (DWDM) Higher density than CWDM 32 lambdas or channels in narrow band around 1550nm at 100GHz spacing (0.8nm) Erbium-Doped Fiber Amplifier (EDFA) amplifiable allows for longer distances than CWDM Carriage of 1 or 2 Gbps, FICON, GigE, 10GigE, 10G, ESCON, IBM GDPS Data center to data center Protection options: client, splitter, or linecard 17

18 DWDM Protection Alternatives for Storage Optical Splitter Protected Lambda Optical Splitter Protection Linecard or Y-cable Protection Y-cable Working Lambda Working Lambda Single transponder required Protects against fiber breaks Failover causes loss of light (and fabric change if only link) Dual transponders required More expensive than splitterbased protection Transmits over both circuits, but only one accepted Protected Lambda 18

19 DWDM HA Storage Network Topology Client protection recommended Diverse Paths One- Fiber Pair Each Path DWDM Ring Portchannel 2 x 2 Gbps Over Two Diverse Paths Fabric and application responsible for failover recovery Portchannel provides resilience Portchannel members follow diverse paths Single fiber cut will not affect fabric (no RSCNs, etc.) Use Src/Dst hash for load balancing (rather than Src/Dst/Oxid per exchange) for each extended VSAN 19

20 FibreChannel Over SONET/SDH Diverse Network Paths SONET/SDH Network Portchannel 2 x 2 Gbps over Two Diverse Paths over SONET/SDH (os) follows same distance rules as other optical technologies BB_Credits in Fibre Channel switch limits distance Outage in SONET/SDH network will not cause loss of light Recovers in <50ms May cause some loss BB_Credit loss from in flight traffic MDS9000 will recover lost BB_Credits 20

21 Extending Optical SAN Extension BB_Credits and Distance 1 Gbps ~2 km per Frame 2 Gbps ~1 km per Frame 4 Gbps ~½ km per Frame 16 Km BB_Credits are used to ensure enough frames in flight A full (2112 byte) frame is approx 2 km 1 Gbps, 1 km 2 Gbps and ½ km long at 4 Gbps As distance increases, the BB_Credits need to increase as well Insufficient BB_Credits will throttle performance no data will be transmitted until R_RDY is returned 21

22 Extending Optical SAN Extension FibreChannel Frame Buffering Traffic Flow BB_Credit Flow Control BB_Credit Flow Control BB_Credit Flow Control 2-8 BB_Credit BB_Credit 2-8 BB_Credit Receive Buffers Receive Buffers Buffer to buffer credits (BB_Credit) are negotiated between each device in a fabric; no concept of end to end buffering One buffer used per frame, irregardless of frame size; small frame uses same buffer as large frame frames buffered and queued in intermediate switches Hop-by-hop traffic flow paced by return of Receiver Ready (R_RDY) frames; can only transmit up to the number of BB_Credits before traffic is throttled 22

23 Extending Optical SAN Extension SAN Network Solutions for Increasing Distance MDS port FibreChannel Switching Module All ports have up to 255 BB_Credits Extends distances to 510 1G or 255 2G MDS /2-port Multiprotocol Services Module Extended BB_Credits available on ports 1-12 Up to 3500 BB_Credits can be configured on any one port Extends distances up to G or G MDS /24/48-port FibreChannel Switching Modules Extended BB_Credits available on any port Up to 4095 BB_Credits can be configured on any one port Extends distance up to G, G or 4G 23

24 Extending Optical SAN Extension Optical Solutions for Increasing Distance No Spoofing B2B Negotiation Frame R_Rdy SONET/SDH Network B2B Negotiation B2B Negotiation Spoofing Frame R_Rdy Frame Frame Ack Frame R_Rdy ONS SL-Series Card Negotiates up 255 BB_Credit with switch Spoofs R_RDYs to switch (Release 5.0) Has 1200 BB_Credits between SL cards Extends distances to G or G 24

25 Improving Optical Recovery PortTrack for Resilient SAN Extension Solutions 1) MDS Detects Link Failure 2) MDS Brings Down Array Port Optical/IP Network 3) Array Retries I/O on Alternate Path Optical/IP Network Arrays recover from a link failure via I/O timeouts. However, this can take several seconds or longer MDS PortTrack addresses this by monitoring the WAN/MAN link and if it detects a failure, it will bring down the corresponding link connected to the array The array after detecting a link failure will redirect the I/O to another link without waiting for the I/O to timeout 25

26 Improving Optical Recovery Port Tracking and ONS FLC or Squelching 4) MDS Brings Down Array Port 3) MDS Detects Link Failure 1) ONS Detects Link Failure Optical Network 2) ONS Brings Down Client transponder 5) Array Retries I/O on Alternate Path Optical Network The MDS port tracking feature can be used with the ONS Forward Laser Control (FLC) or ONS squelching feature to further track failures in the network, improving the ability to detect failed paths Forward Laser Control, squelching and port-tracking offer end to end path failure detection 26

27 Comparison of Data Recovery Port Tracking vs. No Port Tracking Port Tracking vs. No Port Tracking Bits/sec Fail ISL on Fabric A No PT, No FLC No PT, FLC Fabric A No PT, No FLC No PT, FLC Fabric B PT and FLC PT & FLC Fabric A PT and FLC PT & FLC Fabric B

28 MDS IP SAN Extension Design Diverse Network Paths Portchannel 2 x 1Gbps Over Two Diverse Paths Switch/Router Same port channeling and VSAN trunking rules apply as with links Portchannel individual IP links to alternate Ethernet switches/routers Each WAN link carries two IP tunnels Load balancing on SRCID/DESTID or SRCID/DESTID/OXID basis (unidirectionally per VSAN) Certain replication protocols require SRCID/DESTID load balancing FICON, IBM PPRC, HP CA- EVA 28

29 IP Frame Detail Ethernet Header IP Header TCP Header TCP Opts IP Header EISLopt Hdr Hdr SOF Frame Ethernet CRC IP Overhead for Ethernet Frames: 94 Byte Header + 4 Byte CRC = 98 Bytes EISL and Optional Headers If TE_Port, then 8 Bytes Added to Frame (After SOF) for VSAN Routing Max 2148 (E_Port) + EISL and Opt Headers Max FibreChannel frame is 2148 bytes plus optional extras IP will segment and reassemble frames if MTU too small (TCP payload on second or subsequent packets) Jumbo frames may increase performance IP MTU of 2300 avoids splitting of TCP frames 29

30 Storage Traffic and TCP Storage traffic: Quite bursty Latency sensitive (sync apps) Requires high, instantaneous throughput Traditional TCP: Tries to be network sociable Tries to avoid congestion (overrunning downstream routers) Backs off when congestion detected Slow to ramp up over long links (slow start and congestion avoidance) 30

31 MDS IP TCP Behavior Reduce probability of drops Bursts controlled through per flow shaping and congestion window control less likely to overrun routers Increased resilience to drops Uses SACK, fast retransmit and shaping Aggressive slow start Initial rate controlled by min-avail-bandwidth Max rate controlled by maximum-bandwidth Differences with Normal TCP: When congestion occurs with other conventional TCP traffic, IP is more aggressive during recovery ( bullying the other traffic) Aggression is proportional to the min-avail-bandwidth configuration 31

32 Frame Buffering: IP and Traffic Flow BB_Credit Flow Control TCP Windowing Flow Control BB_Credit Flow Control GigE Receive Buffers Slower WAN Link GigE IP Receive Buffers Backlog Here if Queue Can t Drain Due To: Slow WAN link and long RTT Packet loss and retransmissions Many sources (only one shown) Buffer too big IP presents a lower bandwidth pipe (if WAN link) Drain rate (send rate) depends upon bandwidth and congestion Slow ramp up of traditional TCP can cause frame expiry in some conditions Mixture of slow link (e.g. <DS3/E3; retransmissions, many sources, big buffers) 32

33 IP TCP Packet Shaping: MDS9000 Source Shaping Avoids Congestion at This Point Gigabit Ethernet 45Mbps Gigabit Ethernet Destination Traffic Flow Source Sends Packets at rate Consumable by Downstream Path Interpacket Gap to Accommodate Slow Downstream Link (45Mbps) Shaper sends at a rate consumable by the downstream path Immediately sends at minimum-bandwidth rate (avoids early stages of traditional slow start) Ramps up to maximum-bandwidth rate (using usual slow start and congestion avoidance methods) Requirements for shaper to engage: Min-available-bandwidth > 1/20 max-bandwidth SACK (Selective Ack) must be enabled 33

34 MDS9000 IP TCP Behavior Dedicated IP Link Minimum Bandwidth = Maximum Bandwidth Rate For example: a dedicated link Entire link is always available, so min bandwidth = max bandwidth IP will always send at 95% to 100% of max rate without ramp up Traffic is shaped at sending rate (max-bw) After retransmission (congestion), send er resumes at min (=max rate) Behavior mimics UDP blast but with benefits of retransmission capability and shaping 34

35 IP Data Compression Cisco uses R standard compression algorithms implemented in both hardware and software MDS9000 IP Storage Services Module Software-based compression for IP MDS9000 Multiprotocol Services Module Hardware and software-based compression and hardware based encryption for IP Three compression algorithms modes 1 3 plus auto mode Compressibility is data stream dependent All nulls or ones high compression (>30:1) Random data (e.g., encrypted) low compression (~1:1) Typical rate is around 2:1, but may vary considerably Application throughput is the most important factor 35

36 IP Data Compression and TCP Windowing Compression has the effect of a variable bandwidth path TCP window applies to data stream before compression If window size not increased, throughput will not increase Need to compensate with larger TCP max window size MDS9000 incorporates moving average feedback to dynamically adjust TCP window according to compression rate Feedback mechanism is not available when using IP networkbased compression solutions manual adjustment of TCP window size required 36

37 IP vs. SAN Network-Based Compression Network-Based Compression 7200 with VAM, VAM2, VAM2+ Advantages Compression for some or all IP traffic (as required) in shared network Disadvantages No feedback mechanism to TCP, need to manually adjust TCP window size Compression modules required for each network endpoint/path Compressed IP VAM2 Compressed IP SAN-Based Compression MDS9000 with IPS or MPS Module VAM2 Advantages Compression feedback for TCP window adjustments Compression on multiple IP tunnels through different network paths Disadvantages Compression of storage data only in a shared network 37

38 Protecting Data through Encryption Encryption in a SAN Extension Network - Secure Data for Business and Regulatory Requirements Data confidentiality sender can encrypt packets before transmitting them across a network Data integrity receiver can authenticate packets sent by the IPSec sender to ensure that the data has not been altered during transmission Data origin authentication receiver can authenticate the source of the IPSec packets sent; this service is dependent upon the data integrity service Anti-replay protection receiver can detect and reject replayed packets 38

39 Hardware-Based IPSec Encryption Remote Tape Backup Primary Site Remote Replication IP Network Tape Backup and Remote Replication Secured with IPsec Hardware-based GigE wire rate performance with latency ~ 10µs per packet Standards-based IPSec Encryption - implements R 2402 to 2410, & 2412 IKE for protocol/algorithm negotiation and key generation Encryption: AES (128 or 256 bit key), DES (56 bit), 3DES (168 bit) 39

40 IP vs. SAN Network-Based Encryption Network-Based Encryption 7200 with VAM, VAM2, VAM2+ Cisco Catalyst 6500 with VPNSM Advantages Encryption for some or all IP traffic (as required) in shared network Disadvantages Data not encrypted over part of IP network (SAN to switch or router) Encryption modules required for each network endpoint/path VAM2 Encrypted IP Encrypted IP VAM2 SAN-Based Encryption MDS9000 with MPS Module Encrypted IP Advantages Encryption on multiple IP tunnels through different network paths Disadvantages Encryption of storage data only in a shared network 40

41 Write Acceleration Enables extended distance capabilities for remote replication technologies Better performance using /IP-WA Up to 2X the performance over given distance Reduces effective I/O latency within SAN extension solutions Built into Services modules (IPS, MPS, SSM) - transparent to disk arrays Highly resilient solution no data stored in MDS 9000 switch IP Write Acceleration (WA) Write Acceleration Ratio (at various link speeds and write sizes) XFER_RDY WA or IP Network WRITE DATA STATUS WA Reduction in I/O Latency equal to one round trip time (RTT) XFER_RDY Ratio Gbps 622Mbps 155Mbps 45Mbps RTT (ms) 32kB 45M 32kB 155M 32kB 622M 32kB 1G 41

42 Write Acceleration Design Considerations SANOS 2.0 High Availability and Load Balancing: Can be used for native replication (SSM) or IP replication (IPS / MPS) Portchannel Switch/Router Portchannels may be used for HA Equal cost FSPF load balancing for IP Write Acceleration not supported Works with: EMC SRDF, Mirrorview, SANCOPY HDS TrueCopy HP CA-XP, CA-MVA IBM FlashCopy, FastT 42

43 SAN Extension Fabric Stability Connecting existing SAN fabrics or extending a SAN fabrics creates SAN design challenges Limit fabric control traffic such as RSCNs and Build/Reconfigure Fabric (BF/RCF) to local VSANs Connecting SAN fabrics with the same domain IDs Inter-VSAN Routing (IVR) can be used to address these challenges IVR only sends selective RSCNs to edge switches, preventing disruption of fabric services IVR with NAT allows two existing SAN fabrics with the same domain ID to be connected through a third transit VSAN 43

44 SAN Extension with IVR Site A Site B Local VSAN_5 Replication VSAN_10 Transit VSAN_20 (IVR) Replication VSAN_30 Inter-VSAN Connection between Completely Isolated Fabrics Any failure in transit VSAN_20 (network equipment, physical or logical failure) will not disrupt VSAN_10 or VSAN_30 fabric Works with any transport service (, SONET/SDH, DWDM/CWDM, IP) Host to Local Array Fabric is VSAN_5 Site A Replication Fabric is VSAN_10 Site B Replication Fabric is VSAN_30 SAN Extension Fabric is VSAN_20 44

45 MDS 9000 Fabric Switch Positioning Cisco positioned to extend reach all market segments Industry-Leading Investment Protection Across a Comprehensive Product Line Enterprise & Service Provider Small/Medium Business MDS 9000 Family Systems MDS 9124 (8/16/24 ports) MDS 9000 Modules Supervisor 1 and 2 MDS 9216A and 9216i (64 ports) MDS 9509 (336 ports) MDS 9506 (192 ports) 14-Port, 16-Port, 32-Port 12-Port, 24-Port, 48-Port (1 & 2 Gbps ) (1, 2 & 4 Gbps ) MDS 9513 (528 ports) SSM 4-Port 2-Port & 8-Port (10 Gbps ) (1 Gbps Ethernet) Mgmt. Cisco Fabric Manager OS Cisco MDS 9000 Family SAN-OS (Virtualization; Intelligent fabric Applications) 45

46 Agenda Basic SAN Extension Principles Dual path, HA options Alternate Designs Hub-spoke, multi-hop SAN Extension Choices and Enhancements Optical: Extended B2B credits, B2B credit spoofing, Porttracking IP: Compression, Encryption, Application Acceleration MDS 9000 Product Family 46

47 Q and A 47

48 48