Advanced Storage Area Network Design Blaise Pangalos Solutions Architect

Transcription

1

2

3 Advanced Storage Area Network Design Blaise Pangalos Solutions BRKSAN-2883

4 Agenda Introduction Design Principles Technology Overview Storage Fabric Design Considerations Data Center SAN Topologies Intelligent SAN Services Q&A

5 A Goal of this Session Target Smart Zoning FC Switch Domain ID FSPF FC/FCoE Switch Port-Channel Trunk FC Switch Initiator FCP NPV F-Port Port channel F-Port Trunk Fabric Interconnect

6 A Goal of this Session Smart Zoning Target /3 1/4 Domain ID FSPF 2/1 2/2 2/13 2/14 2/15 2/16 2/10 2/1 2/2 Port-Channel Trunk 1 2 1/1 1/2 1/3 1/4 D1 D2 D3 V12 DID 1 3 1/13 NPV F-Port Port channel F-Port Trunk Initiator FCP Fabric Interconnect

7 Introduction

8 An Era of Massive Data Growth Creating New Business Imperatives for IT 10X Increase in Data Produced (From 4.4T GB to 44T GB) By B IoT Devices (Will be Connected to Internet) 40% of Data Will Be Touched by Cloud 85% of Data for Which Enterprises Will Have Liability and Responsibility IDC April 2014: The Digital Universe of Opportunities: Rich Data and Increasing Value of Internet of Things

9 Evolution of Storage Networking. Enterprise Apps: OLTP, VDI, etc. Big Data, Scale-Out NAS Cloud Storage (Object) Compute Nodes Fabric REST API Fabric Block and/or File Arrays Multi-Protocol (FC, FICON, FCIP, FCoE, NAS, iscsi, HTTP) Performance (16G FC, 10GE, 40GE, 100GE) Scale (Tens of Thousands P/V Devices, Billions of Objects) Operational Simplicity (Automation, Self-Service Provisioning)

10 Enterprise Flash Drives = More IO Significantly More IO/s per Drive at Much Lower Response Time Response Time Msec SATA drives (8 drives) 15K rpm drives (8 drives) 100% Random Read Miss 8KB One Drive per DA Processor - 8 processors Enterprise Flash Drives (8 drives) IOPs Drive performance hasn t changed since 2003 (15K drives) Supports new application performance requirements Price/performance making SSD more affordable Solid state drives dramatically increase IOPS that a given array can support Increased IO directly translates to increased throughput

11 Design Principles

12 Fibre Channel Foundations Based on SCSI Host (Initiator) Disk (Target) Foundational protocol, forms the basis of an I/O transaction Host (Initiator) SCSI READ Operation SCSI I/O Channel SCSI WRITE Operation SCSI I/O Channel Disk (Target) Communications are based upon Point to Point Storage is accessed at a block-level via SCSI High-performance interconnect providing high I/O throughput The foundation for all block-based storage connectivity Mature - SCSI-1 developed in 1986

13 Fibre Channel - Communications Point-to-point oriented Facilitated through device login N_Port-to-N_Port connection Logical node connection point N_port Transmitter Receiver Host (Initiator) Receiver Transmitter Disk (Target) N_port Flow controlled Buffer-to-buffer credits and end-to-end basis Acknowledged N_port Transmitter Receiver Receiver Transmitter For certain classes of traffic, none for others Multiple connections allowed per device Host (Initiator) SAN (Switch)

14 Fibre Channel Addressing Dual Port HBA 10:00:00:00:c9:6e:a8:16 10:00:00:00:c9:6e:a8:17 50:0a:09:83:9d:53:43:54 Every Fibre Channel port and node has a hard-coded address called World Wide Name (WWN) Allocated to manufacturer by IEEE Coded into each device when manufactured 64 or 128 bits Host Switch Disk phx2-9513# show int fc 1/1 fc1/1 is up Hardware is Fibre Channel, SFP is short wave laser Port WWN is 20:01:00:05:9b:29:e8:80 Switch Name Server maps WWNs to FCID WWNN uniquely identify devices WWPN uniquely identify each port in a device 4 bits bits N-port or F_port Identifier 24 bits 24 bits IEEE Organizational Unique ID (OUI) Locally Assigned Identifier Format Identifier Port Identifier Assigned to each vendor Vendor-Unique Assignment

15 My port is up can I talk now? FLOGIs/PLOGIs Step 1: Fabric Login (FLOGI) Determines the presence or absence of a Fabric Exchanges Service Parameters with the Fabric Switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID) FC Fabric 3 Target Initializes the buffer-to-buffer credits Step 2: Port Login (PLOGI) E_Port Required between nodes that want to communicate Similar to FLOGI Transports a PLOGI frame to the designation node port In P2P topology (no fabric present), initializes bufferto-buffer credits 2 1 F_Port N_Port HBA Initiator

16 FC_ID Address Model FC_ID address models help speed up FC routing Switches assign FC_ID addresses to N_Ports Some addresses are reserved for fabric services Private loop devices only understand 8-bit address (0x0000xx) FL_Port can provide proxy service for public address translation Maximum switch domains = 239 (based on standard) 8 Bits Switch Switch Topology Model Domain Private Loop Device Address Model Public Loop Device Address Model Area Switch Domain 8 Bits 8 Bits Area Device Arbitrated Loop Physical Address (AL_PA) Arbitrated Loop Physical Address (AL_PA)

17 FSPF Fabric Shortest Path First Provides routing services within any FC fabric Supports multipath routing Bases path status on a link state protocol similar to OSPF Routes hop by hop, based only on the domain ID Runs on E ports or TE ports and provides a loop free topology Runs on a per VSAN basis. Connectivity in a given VSAN in a fabric is guaranteed only for the switches configured in that VSAN. Uses a topology database to keep track of the state of the links on all switches in the fabric and associates a cost with each link Fibre Channel standard ANSI T11 FC-SW2

18 FSPF phx # show fsp database vsan 12 FSPF Link State Database for VSAN 12 Domain 0x02(2) LSR Type = 1 Advertising domain ID = 0x02(2) LSR Age = 1400 Number of links = 4 NbrDomainId IfIndex NbrIfIndex Link Type Cost x01(1) 0x x x01(1) 0x x x03(3) 0x x x03(3) 0x x FSPF Link State Database for VSAN 12 Domain 0x03(3) LSR Type = 1 Advertising domain ID = 0x03(3) LSR Age = 1486 Number of links = 2 NbrDomainId IfIndex NbrIfIndex Link Type Cost x02(2) 0x x x02(2) 0x x phx # 16G D2 16G Port-Channel 2/13 2/14 2/15 2/ /1 1/2 1/3 1/4 8G D3

19 Fibre Channel FC-2 Hierarchy Multiple exchanges are initiated between initiators (hosts) and targets (disks) Each exchange consists of one or more bidirectional sequences Each sequence consists of one or more frames For the SCSI3 ULP, each exchange maps to a SCSI command OX_ID and RX_ID Exchange SEQ_ID Sequence Sequence Sequence SEQ_CNT Frame Fields Frame Frame Frame ULP Information Unit

20 What Is FCoE? It s Fibre Channel From a Fibre Channel standpoint it s FC connectivity over a new type of cable called Ethernet From an Ethernet standpoints it s Yet another ULP (Upper Layer Protocol) to be transported FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FC-1 Encoding FC-0 Physical Interface FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FCoE Logical End Point Ethernet Media Access Control Ethernet Physical Layer

21 Standards for FCoE FCoE is fully defined in FC-BB-5 standard FCoE works alongside additional technologies to make I/O Consolidation a reality T11 FCoE IEEE DCB FC on FC on Other other Network network Media media PFC ETS DCBX Lossless Ethernet Priority Grouping Configuration Verification FC-BB Qbb 802.1Qaz 802.1Qaz Standard Status Technically stable October, 2008 Completed in June 2009 Published in May, 2010 Sponsor Ballot July 2010 Published Fall 2011 Sponsor Ballot October 2010 Published Fall 2011 Sponsor Ballot October 2010 Published Fall

22 FCoE Is Really Two Different Protocols FCoE Itself Is the data plane protocol It is used to carry most of the FC frames and all the SCSI traffic The Two Protocols Have Two different Ethertypes Two different frame formats Both are defined in FC-BB-5 FIP (FCoE Initialization Protocol) It is the control plane protocol It is used to discover the FC entities connected to an Ethernet cloud It is also used to login to and logout from the FC fabric Uses unique BIA on CNA for MAC

23 Enode MAC Address Fibre Channel over Ethernet Addressing Scheme Enode MAC assigned for each FCID Enode MAC composed of a FC-MAP and FCID FC-MAP is the upper 24 bits of the Enode s MAC FCID is the lower 24 bits of the Enode s MAC FCoE forwarding decisions still made based on FSPF and the FCID within the Enode MAC FC Fabric Domain ID 10 Domain ID FCID Domain ID 11 FCID Fibre Channel FCID Addressing FC-MAC Address FC-MAP (0E-FC-xx) FC-MAP (0E-FC-xx) FC-ID FC-ID

24 What is an FCoE Switch? FCF (Fibre Channel Forwarder) accepts the Fibre Channel frame encapsulated in an Ethernet packet and forwards that packet over a VLAN across an Ethernet network to a remote FCoE end device FCoE Attached Storage FCF is a logical FC switch inside an FCoE switch Fibre Channel login happens at the FCF Contains an FCF-MAC address Consumes a Domain ID Nexus FCF FCoE encapsulation/decapsulation happens within the FCF NPV devices are not FCF s and do not have domains Nexus FCF FCoE MDS FC Want more? BRKDCT-1044 FCoE for the IP Network Engineer

25 FCoE is Operationally Identical Supports both FC and FCoE FCoE is treated exactly the same as FC After zoning device perform registration and then performs discovery Which are FCoE hosts? phx2-9513# show fcns database vsan 42 VSAN 42: FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE xac0600 N 50:0a:09:83:8d:53:43:54 (NetApp) scsi-fcp:target 0xac0700 N 50:0a:09:84:9d:53:43:54 (NetApp) scsi-fcp:target 0xac0c00 N 20:41:54:7f:ee:07:9c:00 (Cisco) npv 0xac1800 N 10:00:00:00:c9:6e:b7:f0 scsi-fcp:init fc-gs 0xef0000 N 20:01:a0:36:9f:0d:eb:25 scsi-fcp:init fc-gs

26 After Link Is Up, Accessing Storage FIP and FCoE Login Process Step 1: FIP Discovery Process Enables FCoE adapters to discover which VLAN to transmit & receive FCoE frames Enables FCoE adapters and FCoE switches to discover other FCoE capable devices Verifies Lossless Ethernet is capable of FCoE transmission Step 2: FIP Login Process FC or FCoE Fabric Similar to existing Fibre Channel Login process - sends FLOGI to upstream FCF Adds the negotiation of the MAC address to use Fabric Provided MAC Address (FPMA) FC-MAC FCF assigns the host a Enode MAC address to be used for FCoE forwarding ENode CNA VF_Port VN_Port FIP Discovery Target E_Ports or VE_Port

27 SCSI is the foundation for all O p erating System SCSI SCSI SCSI SCSI SCSI F CP F CP F CP i SCSI F CP F CP F CP F CIP T CP T CP F CoE IP IP L o s s l e s s E t h e r n e t Ethernet Ethernet Physical Wire

28 Additional Relevant Sessions Storage Networking Cisco Live San Diego BRKSAN FCoE for Small and Mid Size Enterprises BRKSAN Storage Area Network Extension Design and Operation BRKVIR Fiber Channel Networking for the IP Network Engineer and SAN Core Edge Design Best Practices BRKSAN Troubleshooting Cisco MDS 9000 Fibre Channel Fabrics BRKSAN Operational Models for FCoE Deployments - Best Practices and Examples BRKSAN SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric

29 Connectivity Types FC FCoE N F F N VN VF VF VN Target Switch Initiator Target Switch Initiator E TE E TE F NP VE VE VF VNP Switch Switch Blade Server Chassis Switch Switch Blade Server Chassis

30 Fibre Channel Port Types Summary Fibre Channel Switch Fabric Switch E_Port E_Port F_Port NP_Port NPV Switch Fabric Switch TE_Port TE_Port VF_Port VNP_Port NPV Switch Fabric Switch VE_Port VE_Port F_Port N_Port End Node G_Port VF_Port VN_Port End Node

31 The Story of Interface Speeds Protocol Clocking Gbps Encoding Data/Sent Data Rate Gbps MB/s 8G FC b/10b G FC b/66b ,275 10G FCoE b/66b ,250 16G FC b/66b ,700 32G FC b/66b ,400 Comparing speeds is more complex than just the apparent speed Data throughput is based on both the interface clocking (how fast the interface transmits) and how efficient the interface transmits (how much encoding overhead) 40G FCoE b/66b ,000

32 VSANs Introduced in 2002 A Virtual SAN (VSAN) Provides a Method to Allocate Ports within a Physical Fabric and Create Virtual Fabrics Analogous to VLANs in Ethernet Per Port Allocation Virtual fabrics created from larger cost-effective redundant physical fabric Reduces wasted ports of a SAN island approach Fabric events are isolated per VSAN which gives further isolation for High Availability FC Features can be configured on a per VSAN basis. ANSI T.11 committee and is now part of Fibre Channel standards as Virtual Fabrics

33 VSAN Assign ports to VSANs Logically separate fabrics Hardware enforced Prevents fabric disruptions RSCN sent within fabric only Each fabric service (zone server, name server, login server, etc.) operates independently in each VSAN Each VSAN is configured and managed independently phx2-9513# show fspf vsan 43 FSPF routing for VSAN 43 FSPF routing administration status is enabled FSPF routing operational status is UP It is an intra-domain router Autonomous region is 0 MinLsArrival = 1000 msec, MinLsInterval = 2000 msec Local Domain is 0xe6(230) Number of LSRs = 3, Total Checksum = 0x vsan database vsan 2 interface fc1/1 vsan 2 interface fc1/2 vsan 4 interface fc1/8 vsan 4 interface fc1/9 phx2-9513# show zoneset active vsan 43 zoneset name UCS-Fabric-B vsan 43 zone name UCS-B-VMware-Netapp vsan 43

34 Zoning & VSANs First assign physical ports to VSANs VSAN 2 Zone A Disk2 Host1 Disk3 Disk1 Zone C Then configure zones within each VSAN Assign zones to active zoneset Zone B Disk4 Host2 Zoneset 1 Each VSAN has its own zoneset VSAN 3 A zone consists of multiple zone members Zone B Zone A Host3 Disk6 Host4 Disk5 Members in a zone can access each other; members in different zones cannot access each other Zoneset 1 Devices can belong to more than one zone

35 Zoning examples Non-zoned devices are members of the default zone A physical fabric can have a maximum of 16,000 zones (9700-only network) Attributes can include pwwn, FC alias, FCID, FWWN, Switch Interface fc x/y, Symbolic node name, Device alias zone name AS01_NetApp vsan 42 member pwwn 20:03:00:25:b5:0a:00:06 member pwwn 50:0a:09:84:9d:53:43:54 device-alias name AS01 pwwn 20:03:00:25:b5:0a:00:06 device-alias name NTAP member pwwn 50:0a:09:84:9d:53:43:54 zone name AS01_NetApp vsan 42 member device-alias AS01 member device-alias NTAP

36 Number of ACL Entries The Trouble with sizable Zoning All Zone Members are Created Equal Standard zoning model just has members Any member can talk to any other member Recommendation: 1-1 zoning Each pair consumes an ACL entry in TCAM 10,000 8,000 6,000 4,000 2,000 Number of ACLs Result: n*(n-1) entries Admin pays price for internal inefficiency 0 Number of Members

37 Smart Zoning Operation Create zones(s) Add an initiator Add a target Today 1:1 Operation Zoning Zones Cmds ACLs Create zones(s) Add an initiator Add a target Feature added in NX-OS 5.2(6) Today Many Operation - Many Zones Cmds ACLs Create zones(s) Add an initiator Add a target Smart Zoning Zones Cmds ACLs Allows storage admins to create larger zones while still keeping premise of single initiator & single target Dramatic reduction SAN administrative time for zoning Utility to convert existing zone or zoneset to Smart Zoning 8 x I 4 x T

38 How to enable Smart Zoning New Zone Existing Zone

39 Inter-VSAN Routing VSAN 2 Zone A Zone B Zoneset 1 VSAN 3 Zone B Zoneset 1 Host3 Disk2 Host1 Disk4 Disk6 Disk3 Disk1 Host2 Host4 Disk5 Zone C Zone A Enables devices in different VSANs to communicate Allows selective routing between specific members of two or more VSANs Traffic flow between selective devices Resource sharing, i.e., tape libraries and disks IVR Zoneset A collection of IVR zones that must be activated to be operational

40 Trunking & Port Channels Trunking Trunk VSAN1 VSAN1 Single-link ISL or PortChannel ISL can be configured to become EISL (TE_Port) Traffic engineering with pruning VSANs on/off the trunk VSAN2 VSAN3 TE Port TE Port TE Port Port Channel Port Channel TE Port VSAN2 VSAN3 Efficient use of ISL bandwidth Up to 16 links can be combined into a PortChannel increasing the aggregate bandwidth by distributing traffic granularly among all functional links in the channel Load balances across multiple links and maintains optimum bandwidth utilization. Load balancing is based on the source ID, destination ID, and exchange ID E Port E Port If one link fails, traffic previously carried on this link is switched to the remaining links. To the upper protocol, the link is still there, although the bandwidth is diminished. The routing tables are not affected by link failure

41 N-Port Virtualization Scaling Fabrics with Stability N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a switch to act like a server/hba performing multiple fabric logins through a single physical link Physical servers connect to the NPV switch and login to the upstream NPIV core switch No local switching is done on an FC switch in NPV mode FC edge switch in NPV mode does not take up a domain ID Helps to alleviate domain ID exhaustion in large fabrics Blade Server F-Port N-Port FC1/1 FC1/2 FC1/3 NPV Switch Server1 N_Port_ID 1 Server2 N_Port_ID 2 Server3 N_Port_ID 3 NP-Port phx (config)# feature npiv F-Port FC NPIV Core Switch F_Port

42 NPV Uplink Selection NPV supports automatic selection of NP uplinks. When a server interface is brought up, the NP uplink interface with the minimum load is selected from the available NP uplinks in the same VSAN as the server interface. When a new NP uplink interface becomes operational, the existing load is not redistributed automatically to include the newly available uplink. Server interfaces that become operational after the NP uplink can select the new NP uplink. Manual method with NPV Traffic-Maps associates one or more NP uplink interfaces with a server interface. Note: Use of parallel NPV links will pin traffic to one NPV link. Use of SAN Portchannels with NPV actual traffic will be load balanced.

43 NPV Uplink Selection UCS Example NPV uplink selection can be automatic or manual With UCS autoselection, the vhbas will be uniformly assigned to the available uplinks depending on the number of logins on each uplink Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port

44 NPV Uplink Selection UCS Example NPV uplink selection can be automatic or manual With UCS autoselection, the vhbas will be uniformly assigned to the available uplinks depending on the number of logins on each uplink Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port

45 NPV Uplink Selection UCS Example NPV uplink selection can be automatic or manual With UCS autoselection, the vhbas will be uniformly assigned to the available uplinks depending on the number of logins on each uplink Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port

46 NPV Uplink Selection UCS Example NPV uplink selection can be automatic or manual With UCS autoselection, the vhbas will be uniformly assigned to the available uplinks depending on the number of logins on each uplink Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port

47 Uplink Port Failure Failure of an uplink moves pinned hosts from failed port to up port(s) Path selection is the same as when new hosts join NPV switch and pathing decision is made Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port Port is Down

48 Uplink Port Recovery No automatic redistribution of hosts to recovered NP port Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port Port is Up

49 New F-Port Attached Host New host entering fabric is automatically pinned to recovered NP_Port Previously pinned hosts are still not automatically redistributed Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port

50 New NP_Port & New F-Port Attached Host NPV continues to distribute new hosts joining fabric Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port F_Port New Port Added

51 Auto-Load-Balance npv_switch(config)# npv auto-load-balance disruptive This is Disruptive Disruptive load balance works independent of automatic selection of interfaces and a configured traffic map of external interfaces. This feature forces reinitialization of the server interfaces to achieve load balance when this feature is enabled and whenever a new external interface comes up. To avoid flapping the server interfaces too often, enable this feature once and then disable it whenever the needed load balance is achieved. If disruptive load balance is not enabled, you need to manually flap the server interface to move some of the load to a new external interface.

52 Blade System Blade System F-Port Port Channel and F-Port Trunking Enhanced Blade Switch Resiliency Blade N Blade 2 Blade 1 Blade N Blade 2 Blade 1 N-Ports N-Port F-Port Port Channel F-Port Port Channel F-Ports Core Director F-Port Trunking F-Port Trunking F-Port Core Director VSAN1 VSAN2 VSAN3 Storage F-Port Port Channel w/ NPV Bundle multiple ports in to 1 logical link Any port, any module High-Availability (HA) Blade Servers are transparent if a cable, port, or line cards fails Traffic Management Higher aggregate bandwidth Hardware-based load balancing F-Port Trunking w/ NPV Partition F-Port to carry traffic for multiple VSANs Extend VSAN benefits to Blade Servers Separate management domains Separate fault isolation domains Differentiated services: QoS, Security

53 A Goal of this Session Smart Zoning Target /3 1/4 Domain ID FSPF 2/1 2/2 2/13 2/14 2/15 2/16 2/10 2/1 2/2 Port-Channel Trunk 1 2 1/1 1/2 1/3 1/4 D1 D2 D3 V12 DID 1 3 1/13 NPV F-Port Port channel F-Port Trunk Initiator FCP Fabric Interconnect

54 A Goal of this Session Smart Zoning Target /3 1/4 Domain ID FSPF 2/1 2/2 2/13 2/14 2/15 2/16 2/10 2/1 2/2 Port-Channel Trunk 1 2 1/1 1/2 1/3 1/4 D1 D2 D3 V12 DID 1 3 1/13 NPV F-Port Port channel F-Port Trunk Initiator FCP Fabric Interconnect

55 Port Channeling & Trunking phx # show run interface fc 2/13-14 interface fc2/13 channel-group 1 force no shutdown interface fc2/14 channel-group 1 force no shutdown phx # show run interface san-port-channel 1 interface san-port-channel 1 switchport trunk allowed vsan 1 switchport trunk allowed vsan add D2 2/13 2/14 1 1/1 1/2 D3

56 Port Channeling & Trunking phx # show run interface fc1/1-2 interface fc1/1 channel-group 1 force no shutdown interface fc1/2 channel-group 1 force no shutdown phx # show run interface port-channel 1 interface port-channel1 switchport trunk allowed vsan 1 switchport trunk allowed vsan add D2 2/13 2/14 1 1/1 1/2 D3

57 Port Channeling & Trunking 4G phx # show run interface fc 2/15-16 interface fc2/15 switchport speed 4000 channel-group 2 force no shutdown interface fc2/16 switchport speed 4000 channel-group 2 force no shutdown phx # show run interface san-port-channel 2 interface san-port-channel 2 switchport trunk allowed vsan 1 switchport trunk allowed vsan add 12 switchport speed D2 2/13 2/14 2/15 2/ /1 1/2 1/3 1/4 D3

58 Port Channeling & Trunking 4G phx # show run interface fc 1/3-4 interface fc1/3 switchport speed 4000 channel-group 2 force no shutdown interface fc1/4 switchport speed 4000 channel-group 2 force no shutdown phx # show run interface port-channel 2 interface port-channel2 switchport trunk allowed vsan 1 switchport trunk allowed vsan add D2 2/13 2/14 2/15 2/ /1 1/2 1/3 1/4 D3

59 Port Channel Switch config phx # show run int fc 2/9-10 interface fc2/9 switchport mode F channel-group 3 force no shutdown interface fc2/10 switchport mode F channel-group 3 force no shutdown phx # show run int san-port-channel 3 interface san-port-channel 3 channel mode active switchport mode F switchport trunk allowed vsan 1 switchport trunk allowed vsan add D2 2/9 2/10 3 2/1 2/2 Fabric Interconnect

60 Port Channel FI Config 5548 D2 2/9 2/10 3 2/1 2/2 Fabric Interconnect

61 FLOGI Before Port Channel phx # show flogi database INTERFACE VSAN FCID PORT NAME NODE NAME fc2/9 12 0x :41:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/9 12 0x :02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 fc2/9 12 0x :02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 fc2/9 12 0x :02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 fc2/ x :42:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/ x :02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 fc2/ x :02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00 Total number of flogi = 7 phx # 5548 D2 2/9 2/10 2/1 2/2 Fabric Interconnect

62 FLOGI- After port channel phx # show flogi database INTERFACE VSAN FCID PORT NAME NODE NAME San-po3 12 0x :0c:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 San-po3 12 0x :02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 San-po3 12 0x :02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 San-po3 12 0x :02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 San-po3 12 0x :02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 San-po3 12 0x :02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00 Total number of flogi = 6 phx # 5548 D2 2/9 2/10 2/1 2/2 Fabric Interconnect

63 Cisco Prime Data Center Network Manager Feature Support and User Interface VMpath Analysis provides VM connectivity to network and storage across Unified Compute and Unified Fabric Visibility past physical access (switch) layer Standard & Custom Reports On Nexus and MDS platforms Dynamic Topology Views Rule-based event filtering and forwarding Threshold Alerting Integration via vcenter API

64 SAN Design Security Challenges SAN design security is often overlooked as an area of concern Application integrity and security is addressed, but not back-end storage network carrying actual data SAN extension solutions now push SANs outside datacenter boundaries Not all compromises are intentional Accidental breaches can still have the same consequences FC SAN design security is only one part of complete data center solution Host access security one-time passwords, auditing, VPNs Storage security data-at-rest encryption, LUN security External Dos or Other Intrusion Privilege Escalation/ Unintended Privilege Application Tampering (Trojans, etc.) SAN Unauthorized Connections (Internal) Theft Data Tampering LAN

65 SAN Security Secure management access Role-based access control CLI, SNMP, and web access Secure management protocols SSH, SFTP, and SNMPv3 Secure switch control protocols TrustSec FC-SP (DH-CHAP) RADIUS AAA and TACACS+ User, switch and iscsi host authentication Fabric Binding Prevent unauthorized switches from joining the fabric Device/SAN Management Security Via SSH, SFTP, SNMPv3, and User Roles SAN Protocol Security (FC-SP) VSANs Provide Secure Isolation Shared Physical Storage RADIUS or TACACS+ Server for Authentication iscsi- Attached Servers Hardware-Based Zoning Via Port and WWN

66 Slow Drain Slow Drain Device Detection and Congestion Avoidance This is a prerequisite for complex fabrics Devices can impart slowness in a fabric Feature of the fabric that ll expose that device for remediation

67 Storage Fabric Design Considerations

68 The Importance of Architecture SAN designs traditionally robust: dual fabrics, data loss is not tolerated Must manage ratios Fan in/out ISL oversubscription Virtualized storage IO streams (NPIV attached devices, server RDM, LPARs, etc.) Queue depth Latency Initiator to target Slow drain Performance under load: does my fabric perform the same Application independence Consistent fabric performance regardless of changes to SCSI profile Number of frames Frame size Speed or throughput

69 SAN Major Design Factors Port density How many now, how many later? Topology to accommodate port requirements Network performance What is acceptable? Unavoidable? Traffic management Preferential routing or resource allocation Fault isolation Consolidation while maintaining isolation Management Secure, simplified management High Performance Crossbar 2 QoS, Congestion Control, Reduce FSPF Routes Failure of One Device Has No Impact on Others Large Port Count Directors 1

70 Scalability Port Density Topology Requirements Considerations Number of ports for end devices How many ports are needed now? What is the expected life of the SAN? How many will be needed in the future? Large Port Count Directors Hierarchical SAN design Best Practice Design to cater for future requirements Doesn t imply build it all now, but means cater for it and avoids costly retrofits tomorrow

71 Traffic Management Do different apps/servers have different performance requirements? Should bandwidth be reserved for specific applications? Is preferential treatment/ QoS necessary? QoS, Congestion Control, Reduce FSPF Routes Given two alternate paths for traffic between data centers, should traffic use one path in preference to the other? Preferential routes

72 Network Performance Oversubscription Design Considerations All SAN Designs Have Some Degree of Oversubscription Without oversubscription, SANs would be too costly Oversubscription is introduced at multiple points Switches are rarely the bottleneck in SAN implementations Device capabilities (peak and sustained) must be considered along with network oversubscription Must consider oversubscription during a network failure event Remember, all traffic flows towards targets main bottlenecks Disk Oversubscription Disk do not sustain wire-rate I/O with realistic I/O mixtures Vendors may recommend a 6:1 to as high as 20:1 host to disk fan-out ratio Highly application dependent ISL Oversubscription Two-tier design ratio less than fan-out ratio Tape Oversubscription Need to sustain close to maximum data rate LTO-6 Native Transfer Rate ~ 160 MBps Port Channels Help Reduce Oversubscription While Maintaining HA Requirements Host Oversubscription Largest variance observed at this level. DB servers close to line rate, others highly oversubscribed 16Gb line cards non-oversubscribed

73 Fault Isolation Consolidation of Storage Single Fabric = Increased Storage Utilization + Reduced Administration Overhead Major Drawback Faults Are No Longer Isolated Technologies such as VSANs enable consolidation and scalability while maintaining security and stability Physical SAN Islands Are Virtualized onto Common SAN Infrastructure VSANs constrain fault impacts Faults in one virtual fabric (VSAN) are contained and do not impact other virtual fabrics Fabric #1 Fabric #2 Fabric #3

74 Data Center SAN Topologies

75 Denser Server Cabinets What are the implications? Uplinks change from 40 GE servers to 4x 10G servers Vertical Cabling Horizontal Cabling EoR X-Connect ToR Main X-Connect From 42U to ~58U DC Infrastructure Changes Denser: cabinets, cross-connects cable runs Horizontal Cabling: from 10G, through 40G to 100G longer distances Vertical Cable: match appropriate server connectivity choice Is SAN EoR economical now?

76 Structured Cabling Supporting new EoR & ToR designs Pricing advantage for manufactured cabling systems Removes guessing game of how many strands to pull per cabinet Growth at 6 or 12 LC ports per cassette Fiber-only cable plant designs possible

77 Core-Edge Highly Scalable Network Design End of Row Top of Rack Blade Server Traditional SAN design for growing SANs High density directors in core and fabric switches, directors or blade switches on edge Predictable performance Scalable growth up to core and ISL capacity Evolves to support EoR & ToR

78 Large Edge-Core-Edge/End-of-Row Design Large Edge/Core/Edge (3456 Usable Ports per Fabric) Traditional Edge-Core-Edge design Is ideal for very large centralized services and consistent host-disk performance regardless of location Full line rate ports, no fabric oversubscription 8Gb or 16Gb hosts and targets Services consolidated in the core Easy expansion Massive cabling if used for EoR designs Ports Deployed 6, Storage ports at 16Gb (optional 8Gb without changing bandwidth ratios) 240 ISLs to storage edge to 16Gb 48 ISLs from host edge to 16Gb A Fabric Shown, Repeat for B Fabric Used Ports Storage Ports 16Gb 8Gb 16Gb, or 8Gb Gb or 16Gb Host Ports 3360 Host ISL Oversubscription End to End Oversubscription 16Gb 16Gb storage 8Gb storage

79 SAN Top of Rack MDS 9148S SAN Top of Rack (5,376 Usable Ports) Ideal for centralized services while reducing cabling requirements Consistent host/target performance regardless of location in rack 8Gb hosts & 16Gb targets Easy edge expansion Massive cabling infrastructure avoided as compared to EoR designs Additional efficiencies with in rack IO convergence 352 Storage ports at 16Gb 4 ISLs from each edge to 16Gb A B Ports Deployed 5,376 Used Ports 5,344 4,224 16Gb Storage Ports 16Gb Host Ports 4,224 Host ISL Oversubscription End to End Oversubscription 16Gb 16G hosts Rack 48 Racks 44 Dual-attached servers per rack

80 Top-of-Rack Design - Blade Centers SAN Top of Rack Blade Centers (1,920 Usable Ports per Fabric) Ideal for centralized services Consistent host/target performance regardless of location in blade enclosure or rack 8Gb hosts & 16Gb targets Need to manage more SAN Edge switches/blade Switches NPV attachment reduces fabric complexity Assumes little east-west SAN traffic Add blade server ISLs to reduce fabric oversubscription Ports Deployed 1,920 8 ISLs from each edge to 8Gb 96 Storage ports at 16Gb A B Used Ports Storage Ports 16Gb 8Gb 16Gb, or 8Gb 960 8Gb Host Ports 2304 Host ISL Oversubscription End to End Oversubscription 8G 16Gb Storage 8Gb Storage Rack 12 Racks, 72 chassis 96 Dual-attached blade servers per rack

81 Medium Scale Dual Fabric Collapsed Core/Edge Design Medium Scale Dual Fabric (768 Usable Ports per Fabric) Ideal for centralized services Consistent host/target performance regardless of location 8Gb or 16Gb hosts & targets (if they exist) Relatively easy edge expansion to Core/Edge EoR design Supports blade centers connectivity 96 Storage ports at 16Gb A Fabric Shown, Repeat for B Fabric Ports Deployed 768 Used Ports 768@ 16Gb Storage Ports 16Gb Gb Host Ports Host ISL Oversubscription 16Gb N/A End to End Oversubscription 16Gb

82 Intelligent SAN Services

83 Enhancing SAN Design with Services Extend Fabrics FCIP Extended Buffer to Buffer credits Encrypt the pipe SAN Services extend the effective distance for remote applications SAN IO acceleration Write acceleration Tape acceleration SAN Extension with FCIP Enhance array replication requirements Reduces WAN-induced latency Improves application performance over distance Data Migration Data Migration with DMM Fabric is aware of all data frames from initiator to target IO Acceleration

84 SAN Extension with FCIP Fibre Channel over IP Encapsulation of Fibre Channel frames into IP packets and tunneling through an existing TCP/IP network infrastructure, in order to connect geographically distant islands Write Acceleration to improve throughput and latency Hardware-based compression Hardware-based IPSec encryption Array to Array Replication FCIP Tunnel TE Port

85 FC Redirect - How IOA Works Replication Starts Replication Starts Initiator to target Flow redirected to IOA Engine Flow accelerated and sent towards normal routing path IOA IOA MAN/WAN IOA IOA IOA= I/O Accelerator Initiator Target Initiator Target Virtual Initiator Virtual Target

86 Data Acceleration A fabric service to accelerate I/O between SAN devices Accelerate SCSI I/O Over both Fibre Channel (FC) and Fibre Channel over IP (FCIP) links For both Write Acceleration (WA) and Tape Acceleration (TA) I/O Acceleration Node platforms: MSM-18/4, SSN-16, MDS-9222i, MDS-9250i Uses FC Redirect IOA IOA MAN/WAN IOA IOA IOA= I/O Accelerator

87 IOA FCIP Tape Backup Large Health Insurance Firm MDS IOA Results 92% throughput FCIP increase Highly resilient Clustering of IOA engines allows for load balancing and failover Improved Scalability- Scale without increasing management overhead Significant reutilization of existing infrastructure- All chassis and common equipment re-utilized Flat VSAN topology- Simple capacity and availability planning

88 Extending Optical SAN Extension BB_Credits and Distance 2 Gbps FC ~1 km per Frame 4 Gbps FC 8 Gbps FC 16 Gbps FC ~0.5 km per Frame ~0.25 km per Frame ~0.125 km per Frame 16 Km phx2-9513(config)# feature fcrxbbcredit extended phx2-9513(config)# interface 1/1 phx2-9513(config-if)# switchport fcrxbbcredit extended 1000 phx2-9513# show interface 1/1 fc1/1 is up.. Transmit B2B Credit is 128 Receive B2B Credit is 1000 BB_Credits are used to ensure enough FC frames in flight A full (2112 byte) FC frame is approx 1 km 2 Gbps, ½ km 4 Gbps ¼ km long at 8 Gbps As distance increases, the number of available BB_Credits need to increase as well Insufficient BB_Credits will throttle performance - no data will be transmitted until R_RDY is returned

89 Data Mobility Application I/O Application Servers Data Mobility Manager Old Array Data Migration New Array Migrates data between storage arrays for Technology refreshes Workload balancing Storage consolidation DMM offers Online migration of heterogeneous arrays Simultaneous migration of multiple LUNs Unequal size LUN migration Rate adjusted migration Verification of migrated data Dual fabric support CLI and wizard-based management with Cisco Fabric Manager Not metered on no. of terabytes migrated or no. of arrays Requires no SAN reconfiguration or rewiring Uses FC Redirect

90 SAN Extension - CWDM Course Wavelength Division Multiplexing TX TX TX TX Optical transmitters Transmission 8 channels WDM using 20nm spacing Colored CWDM SFPs used in FC switch Optical multiplexing done in OADM Passive device Optical fiber pair OADM RX RX RX RX Optical receivers

91 SAN Extension - DWDM Dense Wavelength Division Multiplexing TX TX TX TX Optical transmitters Transmission Optical Splitter Protection Optical fiber pair DWDM devices RX RX RX RX Optical receivers DWDM systems use optical devices to combine the output of several optical transmitters Higher density technology compared with CWDM, <1nm spacing

92 Dense vs Coarse (DWDM vs CWDM) DWDM CWDM Application Long Haul Metro Amplifiers Typically EDFAs Almost Never # Channels Up to 80 Up to 8 Channel Spacing 0.4 nm 20nm Distance Up to 3000km Up to 80km Spectrum 1530nm to 1560nm 1270nm to 1610nm Filter Technology Intelligent Passive Site 1 Site 2 Site 1 Site 2 MDS ONS Array DWDM CWDM

93 Summary Drivers in DC are forcing change 10G convergence & server virtualization It's not just about FCP anymore. FCoE, NFS, iscsi are being adopted Proper SAN design is holistic in the approach Many design options Optimized for performance Some for management Others for cable plant optimization Performance, Scale, Management attributes all play critical roles Not all security issues are external Fault isolation goes beyond SAN A/B separation Consider performance under load Design for SAN services

94 Participate in the My Favorite Speaker Contest Promote Your Favorite Speaker and You Could Be a Winner Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress) Send a tweet and include Your favorite speaker s Twitter Two hashtags: #CLUS #MyFavoriteSpeaker You can submit an entry for more than one of your favorite speakers Don t forget to View the official rules at

95 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

96 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings Watch the related sessions listed earlier (!)

97 Thank you

98