Advanced Storage Area Network Design

Transcription

1

2 Advanced Storage Area Network Design Edward Mazurek Technical Lead Data Center Storage BRKSAN-2883

3 Agenda Introduction Technology Overview Design Principles Storage Fabric Design Considerations Data Center SAN Topologies Intelligent SAN Services Q&A 3

4 Introduction 6

5 An Era of Massive Data Growth Creating New Business Imperatives for IT 10X Increase in Data Produced (From 4.4T GB to 44T GB) By B IoT Devices (Will be Connected to Internet) 40% of Data Will Be Touched by Cloud 85% of Data for Which Enterprises Will Have Liability and Responsibility IDC April 2014: The Digital Universe of Opportunities: Rich Data and Increasing Value of Internet of Things 7

6 Evolution of Storage Networking. Enterprise Apps: OLTP, VDI, etc. Big Data, Scale-Out NAS Cloud Storage (Object) Compute Nodes Fabric REST API Fabric Block and/or File Arrays Multi-Protocol (FC, FICON, FCIP, FCoE, NAS, iscsi, HTTP) Performance (16G FC, 10GE, 40GE, 100GE) Scale (Tens of Thousands P/V Devices, Billions of Objects) Operational Simplicity (Automation, Self-Service Provisioning) 8

7 Enterprise Flash Drives = More IO Significantly More IO/s per Drive at Much Lower Response Time Response Time Msec SATA drives (8 drives) 15K rpm drives (8 drives) 100% Random Read Miss 8KB One Drive per DA Processor - 8 processors Enterprise Flash Drives (8 drives) IOPs Drive performance hasn t changed since 2003 (15K drives) Supports new application performance requirements Price/performance making SSD more affordable Solid state drives dramatically increase IOPS that a given array can support Increased IO directly translates to increased throughput 9

8 Technology Overview 15

9 Fibre Channel Foundations Based on SCSI Host (Initiator) Disk (Target) Foundational protocol, forms the basis of an I/O transaction Host (Initiator) SCSI READ Operation SCSI I/O Channel SCSI WRITE Operation SCSI I/O Channel Disk (Target) Communications are based upon Point to Point Storage is accessed at a block-level via SCSI High-performance interconnect providing high I/O throughput The foundation for all block-based storage connectivity Mature - SCSI-1 developed in

10 Fibre Channel - Communications Point-to-point oriented Facilitated through device login N_Port-to-N_Port connection Logical node connection point N_port Transmitter Receiver Host (Initiator) Receiver Transmitter Disk (Target) N_port Flow controlled Buffer-to-buffer credits and end-to-end basis Acknowledged N_port Transmitter Receiver Receiver Transmitter For certain classes of traffic, none for others Multiple connections allowed per device Host (Initiator) SAN (Switch) 17

11 Fibre Channel Addressing Dual Port HBA 10:00:00:00:c9:6e:a8:16 10:00:00:00:c9:6e:a8:17 50:0a:09:83:9d:53:43:54 Every Fibre Channel port and node has two 64 bit hard-coded addresses called World Wide Names (WWN) NWWN(node) uniquely identify devices PWWN(port) uniquely identify each port in a device Host Switch phx2-9513# show int fc 1/1 fc1/1 is up Hardware is Fibre Channel, SFP is short wave laser Port WWN is 20:01:00:05:9b:29:e8:80 4 bits 0002 Disk 12 bits N-port or F_port Identifier Allocated to manufacturer by IEEE Coded into each device when manufactured Switch Name Server maps PWWN to FCID 24 bits 24 bits IEEE Organizational Unique ID (OUI) Locally Assigned Identifier Format Identifier Port Identifier Assigned to each vendor Vendor-Unique Assignment 18

12 Port Initialization FLOGI and PLOGIGIs/PLOGIs Step 1: Fabric Login (FLOGI) Determines the presence or absence of a Fabric Exchanges Service Parameters with the Fabric Switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID) Initializes the buffer-to-buffer credits Step 2: Port Login (PLOGI) Required between nodes that want to communicate Similar to FLOGI Transports a PLOGI frame to the designation node port In P2P topology (no fabric present), initializes bufferto-buffer credits 2 FC Fabric 1 HBA E_Port F_Port N_Port 3 Target Initiator 19

13 FC_ID Address Model FC_ID address models help speed up FC routing Switches assign FC_ID addresses to N_Ports Some addresses are reserved for fabric services Private loop devices only understand 8-bit address (0x0000xx) FL_Port can provide proxy service for public address translation Maximum switch domains = 239 (based on standard) 8 Bits Switch Switch Topology Model Domain Private Loop Device Address Model Public Loop Device Address Model Area Switch Domain 8 Bits 8 Bits Area Device Arbitrated Loop Physical Address (AL_PA) Arbitrated Loop Physical Address (AL_PA) 20

14 FSPF Fabric Shortest Path First Provides routing services within any FC fabric Supports multipath routing Bases path status on a link state protocol similar to OSPF Routes hop by hop, based only on the domain ID Runs on E ports or TE ports and provides a loop free topology Runs on a per VSAN basis. Connectivity in a given VSAN in a fabric is guaranteed only for the switches configured in that VSAN. Uses a topology database to keep track of the state of the links on all switches in the fabric and associates a cost with each link Fibre Channel standard ANSI T11 FC-SW2 21

15 FSPF phx # show fsp database vsan 12 FSPF Link State Database for VSAN 12 Domain 0x02(2) LSR Type = 1 Advertising domain ID = 0x02(2) LSR Age = 1400 Number of links = 4 NbrDomainId IfIndex NbrIfIndex Link Type Cost x01(1) 0x x x01(1) 0x x x03(3) 0x x x03(3) 0x x FSPF Link State Database for VSAN 12 Domain 0x03(3) LSR Type = 1 Advertising domain ID = 0x03(3) LSR Age = 1486 Number of links = 2 NbrDomainId IfIndex NbrIfIndex Link Type Cost x02(2) 0x x x02(2) 0x x phx # 16G D2 16G Port-Channel 2/13 2/14 2/15 2/ /1 1/2 1/3 1/4 8G 22 D3

16 Fibre Channel FC-2 Hierarchy Multiple exchanges are initiated between initiators (hosts) and targets (disks) Each exchange consists of one or more bidirectional sequences Each sequence consists of one or more frames For the SCSI3 ULP, each exchange maps to a SCSI command OX_ID and RX_ID Exchange SEQ_ID Sequence Sequence Sequence SEQ_CNT Frame Fields Frame Frame Frame ULP Information Unit 23

17 What Is FCoE? It s Fibre Channel From a Fibre Channel standpoint it s FC connectivity over a new type of cable called Ethernet From an Ethernet standpoints it s Yet another ULP (Upper Layer Protocol) to be transported FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FC-1 Encoding FC-0 Physical Interface FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FCoE Logical End Point Ethernet Media Access Control Ethernet Physical Layer 24

18 Standards for FCoE FCoE is fully defined in FC-BB-5 standard FCoE works alongside additional technologies to make I/O Consolidation a reality T11 FCoE IEEE DCB FC on FC on Other other Network network Media media PFC ETS DCBX Lossless Ethernet Priority Grouping Configuration Verification FC-BB Qbb 802.1Qaz 802.1Qaz Standard Status Technically stable October, 2008 Completed in June 2009 Published in May, 2010 Sponsor Ballot July 2010 Published Fall 2011 Sponsor Ballot October 2010 Published Fall 2011 Sponsor Ballot October Published Fall

19 FCoE Flow Control IEEE 802.1Qbb Priority Flow Control Resume Ethernet Wire FCoE 3.3ms VLAN Tag enables 8 priorities for Ethernet traffic PFC enables Flow Control on a Per-Priority basis using PAUSE frames (IEEE 802.1p) Receiving device/switch sends Pause frame when receiving buffer passes threshold Two types of pause frames Quanta = = 3.3ms Quanta = 0 = Immediate resume Distance support is determined by how much buffer is available to absorb data in flight after Pause frame sent 26

20 ETS: Enhanced Transmission Selection IEEE 802.1Qaz Allows you to create priority groups Can guarantee bandwidth Can assign bandwidth percentages to groups Not all priorities need to be used or in groups 80%20% 80% FCoE 20% Ethernet Wire 27

21 FCoE Is Really Two Different Protocols FIP (FCoE Initialization Protocol) It is the control plane protocol It is used to discover the FC entities connected to an Ethernet cloud It is also used to login to and logout from the FC fabric Uses unique BIA on CNA for MAC Ethertype 0x8914 FCoE Itself Is the data plane protocol It is used to carry most of the FC frames and all the SCSI traffic Ethertype 0x8906 The Two Protocols Have Two different Ethertypes Two different frame formats Both are defined in FC-BB

22 FPMA - Fabric Provided MAC Address Fibre Channel over Ethernet Addressing Scheme FPMA assigned for each FCID FPMA composed of a FC-MAP and FCID FC-MAP Mapped Address Prefix the upper 24 bits of the FPMA FCID is the lower 24 bits of the FPMA FCoE forwarding decisions still made based on FSPF and the FCID within the FPMA FC Fabric Domain ID 10 Domain ID FCID Domain ID 11 FCID Fibre Channel FCID Addressing FPMA FC-MAP (0E-FC-xx) FC-MAP (0E-FC-xx) FC-ID FC-ID

23 What is an FCoE Switch? FCF (Fibre Channel Forwarder) accepts a Fibre Channel frame encapsulated in an Ethernet packet and forwards that packet over a VLAN across an Ethernet network to a remote FCoE end device FCF is a logical FC switch inside an FCoE switch FCoE Attached Storage Fibre Channel login happens at the FCF Contains an FCF-MAC address Consumes a Domain ID Nexus FCF FCoE encapsulation/decapsulation happens within the FCF NPV devices are not FCF s and do not have domains Nexus FCF FCoE MDS FC 30

24 FCoE is Operationally Identical Supports both FC and FCoE FCoE is treated exactly the same as FC After zoning device perform registration and then performs discovery Which are FCoE hosts? phx2-9513# show fcns database vsan 42 VSAN 42: FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE xac0600 N 50:0a:09:83:8d:53:43:54 (NetApp) scsi-fcp:target 0xac0700 N 50:0a:09:84:9d:53:43:54 (NetApp) scsi-fcp:target 0xac0c00 N 20:41:54:7f:ee:07:9c:00 (Cisco) npv 0xac1800 N 10:00:00:00:c9:6e:b7:f0 scsi-fcp:init fc-gs 0xef0000 N 20:01:a0:36:9f:0d:eb:25 scsi-fcp:init fc-gs 33

25 After Link Is Up, Accessing Storage FIP and FCoE Login Process Step 1: FIP Discovery Process Enables FCoE adapters to discover which VLAN to transmit & receive FCoE frames Enables FCoE adapters and FCoE switches to discover other FCoE capable devices Occurs over Lossless Ethernet Step 2: FIP Login Process Similar to existing Fibre Channel Login (FLOGI) process Sent to upstream FCF FCF assigns the host a FCID and FPMA to be used for FCoE forwarding Returns the FCID and the Fabric Provided MAC Address (FPMA) to the ENode FC-MAC FC or FCoE Fabric ENode CNA VF_Port VN_Port FIP Discovery Target E_Ports or VE_Port 34

26 SCSI is the foundation for all O p e r a t i n g S y s t e m S C S I S C S I S C S I S C S I S C S I F C P F C P F C P i S C S I F C P F C P F C P F C I P F C o E T C P T C P IP T C P IP L o s s l e s s E t h e r n e t E t h e r n e t E t h e r n e t Physical Wire 35

27 Connectivity Types FC FCoE N F F N VN VF VF VN Target Switch Initiator Target Switch Initiator E TE E TE F NP VE VE VF VNP Switch Switch Blade Server Chassis Switch Switch Blade Server Chassis 36

28 Fibre Channel Port Types Summary Fibre Channel Switch Fabric Switch E_Port E_Port F_Port NP_Port NPV Switch Fabric Switch TE_Port TE_Port VF_Port VNP_Port NPV Switch Fabric Switch VE_Port VE_Port F_Port N_Port End Node G_Port VF_Port VN_Port End Node 37

29 The Story of Interface Speeds Protocol Clocking Gbps Encoding Data/Sent Data Rate Gbps MB/s 8G FC b/10b G FC b/66b ,275 10G FCoE b/66b ,250 16G FC b/66b ,700 32G FC b/66b ,400 Comparing speeds is more complex than just the apparent speed Data throughput is based on both the interface clocking (how fast the interface transmits) and how efficient the interface transmits (how much encoding overhead) 40G FCoE b/66b ,

30 Design Principles 39

31 VSANs Introduced in 2002 A Virtual SAN (VSAN) Provides a Method to Allocate Ports within a Physical Fabric and Create Virtual Fabrics Analogous to VLANs in Ethernet Virtual fabrics created from larger cost-effective redundant physical fabric Per Port Allocation Reduces wasted ports of a SAN island approach Fabric events are isolated per VSAN which gives further isolation for High Availability FC Features can be configured on a per VSAN basis. ANSI T.11 committee and is now part of Fibre Channel standards as Virtual Fabrics 40

32 VSAN Assign ports to VSANs Logically separate fabrics Hardware enforced Prevents fabric disruptions RSCN sent within fabric only Each fabric service (zone server, name server, login server, etc.) operates independently in each VSAN Each VSAN is configured and managed independently phx2-9513# show fspf vsan 43 FSPF routing for VSAN 43 FSPF routing administration status is enabled FSPF routing operational status is UP It is an intra-domain router Autonomous region is 0 MinLsArrival = 1000 msec, MinLsInterval = 2000 msec Local Domain is 0xe6(230) Number of LSRs = 3, Total Checksum = 0x vsan database vsan 2 interface fc1/1 vsan 2 interface fc1/2 vsan 4 interface fc1/8 vsan 4 interface fc1/9 phx2-9513# show zoneset active vsan 43 zoneset name UCS-Fabric-B vsan 43 zone name UCS-B-VMware-Netapp vsan 43 41

33 Zoning & VSANs 1. Assign physical ports to VSANs VSAN 2 Zone A Zone B Disk2 Host1 Disk4 Disk3 Disk1 Host2 Zone C 2. Configure zones within each VSAN A zone consists of multiple zone members 3. Assign zones to zoneset Each VSAN has its own zoneset Zoneset 1 4. Activate zoneset in VSAN VSAN 3 Zone B Host3 Zone A Host4 Disk5 Members in a zone can access each other; members in different zones cannot access each other Zoneset 1 Disk6 Devices can belong to more than one zone 42

34 Zoning examples Non-zoned devices are members of the default zone A physical fabric can have a maximum of 16,000 zones (9700-only network) Attributes can include pwwn, FC alias, FCID, FWWN, Switch Interface fc x/y, Symbolic node name, Device alias zone name AS01_NetApp vsan 42 member pwwn 20:03:00:25:b5:0a:00:06 member pwwn 50:0a:09:84:9d:53:43:54 device-alias name AS01 pwwn 20:03:00:25:b5:0a:00:06 device-alias name NTAP member pwwn 50:0a:09:84:9d:53:43:54 zone name AS01_NetApp vsan 42 member device-alias AS01 member device-alias NTAP 43

35 Number of ACL Entries The Trouble with sizable Zoning All Zone Members are Created Equal Standard zoning model just has members Any member can talk to any other member Recommendation: 1-1 zoning Each pair consumes an ACL entry in TCAM 10,000 8,000 6,000 4,000 2,000 Number of ACLs Result: n*(n-1) entries 0 Number of Members 44

36 Smart Zoning Operation Create zones(s) Today 1:1 Operation Zoning Zones Cmds ACLs Create zones(s) Today Many Operation - Many Zones Cmds ACLs Create zones(s) Smart Zoning Zones Cmds ACLs x I 4 x T Add an initiator Add an initiator Add an initiator Add a target Add a target Add a target Feature added in NX-OS 5.2(6) Allows storage admins to create larger zones while still keeping premise of single initiator & single target Dramatic reduction SAN administrative time for zoning Utility to convert existing zone or zoneset to Smart Zoning 45

37 How to enable Smart Zoning New Zone Existing Zone 46

38 Zoning Best Practices zone mode enhanced Acquires lock on all switches while zoning changes are underway Enables full zoneset distribution zone confirm-commit Causes zoning changes to be displayed during zone commit zoneset overwrite-control New in NX-OS 6.2(13) Prevents a different zoneset than the currently activated zoneset from being inadvertently activated Note: Above setting are per-vsan 48

39 IVR - Inter-VSAN Routing VSAN 2 Zone A Zone B Zoneset 1 VSAN 3 Zone B Zoneset 1 Host3 Disk2 Host1 Disk4 Disk6 Disk3 Disk1 Host2 Host4 Disk5 Zone C Zone A Enables devices in different VSANs to communicate Allows selective routing between specific members of two or more VSANs Traffic flow between selective devices Resource sharing, i.e., tape libraries and disks IVR Zoneset A collection of IVR zones that must be activated to be operational 49

40 Forward Error Correction - FEC Allows for the correction of some errors in frames Almost zero latency penalty Can prevent SCSI timeouts and aborts Applies to MDS 9700 FC and MDS 9396S Applies to 16G fixed speed FC ISLs only switchport speed Configured via: switchport fec tts No reason not to use it! # show interface fc1/8 fc1/8 is trunking Port mode is TE Port vsan is 1 Speed is 16 Gbps Rate mode is dedicated Transmit B2B Credit is 500 Receive B2B Credit is 500 B2B State Change Number is 14 Receive data field Size is 2112 Beacon is turned off admin fec state is up oper fec state is up Trunk vsans (admin allowed and active) (1-2,20,237) 50

41 Trunking & Port Channels Trunking Trunk VSAN1 VSAN1 Single-link ISL or PortChannel ISL can be configured to become EISL (TE_Port) Traffic engineering with pruning VSANs on/off the trunk VSAN2 VSAN3 TE Port TE Port TE Port Port Channel Port Channel TE Port VSAN2 VSAN3 Efficient use of ISL bandwidth Up to 16 links can be combined into a PortChannel increasing the aggregate bandwidth by distributing traffic granularly among all functional links in the channel Load balances across multiple links and maintains optimum bandwidth utilization. Load balancing is based on the source ID, destination ID, and exchange ID E Port E Port If one link fails, traffic previously carried on this link is switched to the remaining links. To the upper protocol, the link is still there, although the bandwidth is diminished. The routing tables are not affected by link failure 51

42 N-Port Virtualization Scaling Fabrics with Stability N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a switch to act like a server/hba performing multiple fabric logins through a single physical link Physical servers connect to the NPV switch and login to the upstream NPIV core switch No local switching is done on an FC switch in NPV mode FC edge switch in NPV mode does not take up a domain ID Helps to alleviate domain ID exhaustion in large fabrics Blade Server NPV Switch phx (config)# feature npiv FC NPIV Core Switch F-Port FC1/1 Server1 N_Port_ID 1 F-Port FC1/2 Server2 N_Port_ID 2 NP-Port FC1/3 Server3 N_Port_ID 3 F_Port N-Port 52

43 Comparison Between NPIV and NPV NPIV (N-Port ID Virtualization) Used by HBA and FC switches Enables multiple logins on a single interface Allows SAN to control and monitor virtual machines (VMs) Used for VMWare, MS Virtual Server and Linux Xen applications NPV (N-Port Virtualizer) Used by FC (MDS 9124, 9148, 9148S, etc.), FCOE switches (Nexus 5K), blade switches and Cisco UCS Fabric InterConnects (UCS6100) Aggregate multiple physical/logical logins to the core switch Addresses the explosion of number of FC switches Used for server consolidation applications 53

44 NPV Uplink Selection NPV supports automatic selection of NP uplinks. When a server interface is brought up, the NP uplink interface with the minimum load is selected from the available NP uplinks in the same VSAN as the server interface. When a new NP uplink interface becomes operational, the existing load is not redistributed automatically to include the newly available uplink. Server interfaces that become operational after the NP uplink can select the new NP uplink. Manual method with NPV Traffic-Maps associates one or more NP uplink interfaces with a server interface. Note: Use of parallel NPV links will pin traffic to one NPV link. Use of SAN Portchannels with NPV actual traffic will be load balanced. 54

45 NPV Uplink Selection UCS Example NPV uplink selection can be automatic or manual With UCS autoselection, the vhbas will be uniformly assigned to the available uplinks depending on the number of logins on each uplink Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port 58

46 Uplink Port Failure Failure of an uplink moves pinned hosts from failed port to up port(s) Path selection is the same as when new hosts join NPV switch and pathing decision is made Blade Server F-Port Cisco UCS NPV Switch NP-Port 2 devices re-login FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port Port is Down 59

47 Uplink Port Recovery No automatic redistribution of hosts to recovered NP port Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port Port is Up 60

48 New F-Port Attached Host New host entering fabric is automatically pinned to recovered NP_Port Previously pinned hosts are still not automatically redistributed Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port 61

49 New NP_Port & New F-Port Attached Host NPV continues to distribute new hosts joining fabric Blade Server F-Port Cisco UCS NPV Switch NP-Port FC NPIV Core Switch FC1/1 FC1/2 FC1/3 FC1/4 FC1/5 FC1/6 F_Port F_Port F_Port New Port Added 62

50 Auto-Load-Balance npv_switch(config)# npv auto-load-balance disruptive This is Disruptive Disruptive load balance works independent of automatic selection of interfaces and a configured traffic map of external interfaces. This feature forces reinitialization of the server interfaces to achieve load balance when this feature is enabled and whenever a new external interface comes up. To avoid flapping the server interfaces too often, enable this feature once and then disable it whenever the needed load balance is achieved. If disruptive load balance is not enabled, you need to manually flap the server interface to move some of the load to a new external interface. aces/nx-os/cli_interfaces/npv.html#pgfid

51 Blade System Blade System F-Port Port Channel and F-Port Trunking Enhanced Blade Switch Resiliency Blade N Blade 2 Blade 1 Blade N Blade 2 Blade 1 N-Ports N-Port F-Port Port Channel F-Port Port Channel F-Ports Core Director F-Port Trunking F-Port Trunking F-Port Core Director VSAN1 VSAN2 VSAN3 Storage F-Port Port Channel w/ NPV Bundle multiple ports in to 1 logical link Any port, any module High-Availability (HA) Blade Servers are transparent if a cable, port, or line cards fails Traffic Management Higher aggregate bandwidth Hardware-based load balancing F-Port Trunking w/ NPV Partition F-Port to carry traffic for multiple VSANs Extend VSAN benefits to Blade Servers Separate management domains Separate fault isolation domains Differentiated services: QoS, Security 64

52 Port Channeling & Trunking - Configuration phx # show run interface san-port-channel 1 interface san-port-channel 1 switchport trunk allowed vsan 1 switchport trunk allowed vsan add 12 phx # show run interface fc 2/13-14 interface fc2/13 channel-group 1 force no shutdown interface fc2/14 channel-group 1 force no shutdown Nexus 5548 D2 fc2/13 fc2/14 1 fc1/1 fc1/2 MDS 9148 D3 67

53 Port Channeling & Trunking - Configuration phx # show run interface port-channel 1 interface port-channel1 switchport trunk allowed vsan 1 switchport trunk allowed vsan add 12 phx # show run interface fc1/1-2 interface fc1/1 channel-group 1 force no shutdown interface fc1/2 channel-group 1 force no shutdown Nexus 5548 D2 fc2/13 fc2/14 1 fc1/1 fc1/2 MDS 9148 D3 68

54 Port Channel Nexus switch config phx # show run int san-port-channel 3 interface san-port-channel 3 channel mode active switchport mode F switchport trunk allowed vsan 1 switchport trunk allowed vsan add 12 phx # show run int fc 2/9-10 fc2/9 Nexus 5548 D2 fc2/10 interface fc2/9 switchport mode F channel-group 3 force no shutdown fc2/1 3 fc2/2 interface fc2/10 switchport mode F channel-group 3 force no shutdown Fabric Interconnect 71

55 Port Channel FI Config 5548 D2 fc2/9 fc2/10 3 fc2/1 fc2/2 Fabric Interconnect 72

56 FLOGI Before Port Channel phx # show flogi database INTERFACE VSAN FCID PORT NAME NODE NAME fc2/9 12 0x :41:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/9 12 0x :02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 fc2/9 12 0x :02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 fc2/9 12 0x :02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 fc2/ x :42:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/ x :02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 fc2/ x :02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00 Total number of flogi = 7 phx # fc2/9 fc2/ D2 fc2/10 fc2/2 Fabric Interconnect 73

57 FLOGI- After port channel phx # show flogi database INTERFACE VSAN FCID PORT NAME NODE NAME San-po3 12 0x :0c:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 San-po3 12 0x :02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 San-po3 12 0x :02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 San-po3 12 0x :02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 San-po3 12 0x :02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 San-po3 12 0x :02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00 Total number of flogi = 6 phx # 5548 D2 2/9 2/10 2/1 2/2 Fabric Interconnect 74

58 Port-channel design considerations All types of switches Name port-channels the same on both sides Common port allocation in both fabrics ISL speeds should be >= edge device speeds Maximum 16 members per port-channel allowed Multiple port-channels to same adjacent switch should be equal cost Member of VSAN 1 + trunk other VSANs Check TCAM usage: show system internal acl tcam-usage 75

59 port-channel design considerations Director class Split port-channel members across multiple line cards When possible use same port on each LC: Ex. fc1/5, fc2/5, fc3/5, fc4/5, etc. If multiple members per linecard distribute across port-groups show port-resources module x 76

60 Port-channel design considerations Fabric switches Ensure enough credits for distance Can rob buffers from other ports in port-group that are out-of-service Split port-channel member across different forwarding engines to distribute ACLTCAM For F port-channels to NPV switches (like UCS FIs) Each device s zoning ACLTCAM programming will be repeated on each PC member For E port-channels using IVR Each host/target session that gets translated will take up ACLTCAM on each member Use following table: Ex. On a 9148S a six member port-channel could be allocated across the 3 fwd engines as follows: fc1/1, fc1/2, fc1/17, fc1/18, fc1/33 and fc1/34 Consider MDS 9396S for larger scale deployments 77

61 F port-channel design considerations Ports are allocated to fwd-engines according the following table: Switch/Module Fwd Engines Port Range(s) Fwd-Eng Number Zoning Region Entries MDS fc1/25-36 & fc1/ fc1/5-12 & fc1/ & MDS 9250i 4 fc1/5-12 & eth1/ fc1/1-4 & fc1/13-20 & fc1/ fc1/ ips1/ MDS 9148S Bottom Region Entries 78

62 F port-channel design considerations Switch/Module Fwd Engines Port Range(s) Fwd-Eng Number Zoning Region Entries MDS 9396S Bottom Region Entries 79

63 F port-channel design considerations Switch/Module Fwd Engines Port Range(s) Fwd-Eng Number Zoning Region Entries DS-X K DS-X K DS-X K DS-X K DS-X K Bottom Region Entries 80

64 F port-channel design considerations Switch/Module Fwd Engines Port Range(s) Fwd-Eng Number Zoning Region Entries DS-X K Bottom Region Entries 81

65 Internal CRC handling New feature to handle frames internally corrupted due to bad HW Frames that are received corrupted are dropped at the ingress port These frames are not included in this feature In rare cases frames can get corrupted internally due to bad hardware These are then dropped Sometimes difficult to detect New feature detects the condition and isolates hardware 5 possible stages where frames can get corrupted 82

66 Internal CRC handling Stages of Internal CRC Detection and Isolation The five possible stages at which internal CRC errors may occur in a switch: 1. Ingress buffer of a module 2. Ingress crossbar of a module 3. Crossbar of a fabric module 4. Egress crossbar of a module 5. Egress buffer of a module 83

67 Internal CRC handling The modules that support this functionality are: Cisco MDS Port 16-Gbps Fibre Channel Switching Module Cisco MDS Port 10-Gbps Fibre Channel over Ethernet Switching Module Cisco MDS 9700 Fabric Module 1 Cisco MDS 9700 Supervisors Enabled via the following configuration command: hardware fabric crc threshold When detected failing module is powered down New in NX-OS 6.2(13) 84

68 Device-alias device-alias(da) is a way of naming PWWNs DAs are distributed on a fabric basis via CFS device-alias database is independent of VSANs If a device is moved from one VSAN to another no DA changes are needed device-alias can run in two modes: Basic device-alias names can be used but PWWNs are substituted in config Enhanced device-alias names exist in configuration natively Allows rename without zoneset re-activations device-alias are used in zoning, IVR zoning and port-security copy running-config startup-config fabric after making changes! 85

69 Device-alias device-alias confirm-commit Displays the changes and prompts for confirmation MDS9710-2(config)# device-alias confirm-commit enable MDS9710-2(config)# device-alias database MDS9710-2(config-device-alias-db)# device-alias name edm pwwn MDS9710-2(config-device-alias-db)# device-alias commit The following device-alias changes are about to be committed + device-alias name edm pwwn 10:00:00:00:11:11:11:11 Do you want to continue? (y/n) [n] 86

70 Device-alias Note: To prevent problems the same device-alias is only allowed once per commit. Example: MDS9148s-1(config)# device-alias database MDS9148s-1(config-device-alias-db)# device-alias name test pwwn MDS9148s-1(config-device-alias-db)# device-alias rename test test1 Command rejected. Device-alias reused in current session :test Please use 'show device-alias session rejected' to display the rejected set of commands and for the device-alias best-practices recommendation. 87

71 Cisco Prime Data Center Network Manager Feature Support and User Interface VMpath Analysis provides VM connectivity to network and storage across Unified Compute and Unified Fabric Visibility past physical access (switch) layer Standard & Custom Reports On Nexus and MDS platforms Dynamic Topology Views Rule-based event filtering and forwarding Threshold Alerting Integration via vcenter API 88

72 SAN Design Security Challenges SAN design security is often overlooked as an area of concern Application integrity and security is addressed, but not back-end storage network carrying actual data SAN extension solutions now push SANs outside datacenter boundaries Not all compromises are intentional Accidental breaches can still have the same consequences FC SAN design security is only one part of complete data center solution Host access security one-time passwords, auditing, VPNs Storage security data-at-rest encryption, LUN security External DOS or Other Intrusion Privilege Escalation/ Unintended Privilege Application Tampering (Trojans, etc.) SAN Unauthorized Connections (Internal) Theft Data Tampering LAN 89

73 SAN Security Secure management access Role-based access control CLI, SNMP, and web access Secure management protocols SSH, SFTP, and SNMPv3 Secure switch control protocols TrustSec FC-SP (DH-CHAP) AAA - RADIUS, TACACS+ and LDAP User, switch and iscsi host authentication Fabric Binding Prevent unauthorized switches from joining the fabric Device/SAN Management Security Via SSH, SFTP, SNMPv3, and User Roles SAN Protocol Security (FC-SP) VSANs Provide Secure Isolation RADIUS or TACACS+ or LDAP Server for Authentication iscsi- Attached Servers Hardware-Based Zoning Via Port and WWN Shared Physical Storage 90

74 Slow Drain Slow Drain Device Detection and Congestion Avoidance Devices can impart slowness in a fabric Feature of the fabric that ll expose that device for remediation BRKSAN-3446 SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric White paper (2013) 91

75 Storage Fabric Topology Considerations 92

76 The Importance of Architecture SAN designs traditionally robust: dual fabrics, data loss is not tolerated Must manage ratios Fan in/out ISL oversubscription Virtualized storage IO streams (NPIV attached devices, server RDM, LPARs, etc.) Queue depth Latency Initiator to target Slow drain Performance under load: does my fabric perform the same Application independence Consistent fabric performance regardless of changes to SCSI profile Number of frames Frame size Speed or throughput 93

77 SAN Major Design Factors Port density How many now, how many later? Topology to accommodate port requirements Network performance What is acceptable? Unavoidable? Traffic management Preferential routing or resource allocation Fault isolation Consolidation while maintaining isolation High Performance Crossbar 2 QoS, Congestion Control, Reduce FSPF Routes Large Port Count Directors 1 Management Secure, simplified management 4 Failure of One Device Has No Impact on Others 94 94

78 Scalability Port Density Topology Requirements Considerations Number of ports for end devices How many ports are needed now? What is the expected life of the SAN? How many will be needed in the future? Hierarchical SAN design Large Port Count Directors Best Practice Design to cater for future requirements Doesn t imply build it all now, but means cater for it and avoids costly retrofits tomorrow 95

79 Scalability Port Density MDS Switch selection MDS 9148S 48 ports 16G FC MDS 9250i 40 ports 16G FC + 8 port 10G FCoE + 2 FCIP ports MDS 9396S 96 ports 16G FC MDS 9706 Up to 192 ports 16G FC and/or 10G FCoE and/or 40G FCoE MDS 9710 Up to 384ports 16G FC and/or 10G FCoE and/or 40G FCoE MDS 9718 Up to 768 ports 16G FC and/or 10G FCoE and/or 40G FCoE All MDS 97xx chassis are 32G ready! All 16G MDS platforms are full line rate 96

80 Scalability Port Density Nexus Switch selection Nexus 55xx Up to 96 ports 10G FCoE and/or 8G FC ports Nexus 5672UP Up to 48 10G FCoE and/or 16 8G FC ports Nexus 5672UP-16G Up to 48 10G FCoE and/or 24 16G FC ports Nexus5624Q 12 ports 40G or 48 ports 10G FCoE Nexus5648Q 24 ports 40G or 96 ports 10G FCoE Nexus5696Q Up to 32 ports 100G / 96 ports 40G / 384 ports 10G FCoE or 60 8G FC Nexus 56128P Up to 96 10G FCoE and/or 48 8G FC ports All Nexus platforms are full line rate 97

81 Traffic Management Do different apps/servers have different performance requirements? Should bandwidth be reserved for specific applications? Is preferential treatment/ QoS necessary? QoS, Congestion Control, Reduce FSPF Routes Given two alternate paths for traffic between data centers, should traffic use one path in preference to the other? Preferential routes

82 Network Performance Oversubscription Design Considerations All SAN Designs Have Some Degree of Oversubscription Without oversubscription, SANs would be too costly Oversubscription is introduced at multiple points Switches are rarely the bottleneck in SAN implementations Device capabilities (peak and sustained) must be considered along with network oversubscription Must consider oversubscription during a network failure event Disk Oversubscription Disk do not sustain wire-rate I/O with realistic I/O mixtures Vendors may recommend a 6:1 to as high as 20:1 host to disk fan-out ratio Highly application dependent ISL Oversubscription Two-tier design ratio less than fan-out ratio Tape Oversubscription Need to sustain close to maximum data rate LTO-6 Native Transfer Rate ~ 160 MBps Port Channels Help Reduce Oversubscription While Maintaining HA Requirements Host Oversubscription Largest variance observed at this level. DB servers close to line rate, others highly oversubscribed 16Gb line cards non-oversubscribed 99

83 Fault Isolation Consolidation of Storage Single Fabric = Increased Storage Utilization + Reduced Administration Overhead Major Drawback Faults Are No Longer Isolated Technologies such as VSANs enable consolidation and scalability while maintaining security and stability Physical SAN Islands Are Virtualized onto Common SAN Infrastructure VSANs constrain fault impacts Faults in one virtual fabric (VSAN) are contained and do not impact other virtual fabrics Fabric #1 Fabric #2 Fabric #3 100

84 Data Center SAN Topologies 101

85 Denser Server Cabinets What are the implications? Uplinks change from 40 GE servers to 4x 10G servers ToR Vertical Cabling Horizontal Cabling EoR X-Connect Main X-Connect From 42U to ~58U DC Infrastructure Changes Denser: cabinets, cross-connects cable runs Horizontal Cabling: from 10G, through 40G to 100G longer distances Vertical Cable: match appropriate server connectivity choice Is SAN EoR economical now? 102

86 Structured Cabling Supporting new EoR & ToR designs Pricing advantage for manufactured cabling systems Removes guessing game of how many strands to pull per cabinet Growth at 6 or 12 LC ports per cassette Fiber-only cable plant designs possible 103

87 Core-Edge Highly Scalable Network Design Traditional SAN design for growing SANs High density directors in core and fabric switches, directors or blade switches on edge Predictable performance Scalable growth up to core and ISL capacity Evolves to support EoR & ToR End of Row Top of Rack Blade Server MDS 9718 as core 105

88 Large Edge-Core-Edge/End-of-Row Design Large Edge/Core/Edge (2496 End Device Ports per Fabric) Traditional Edge-Core-Edge design Is ideal for very large centralized services and consistent host-disk performance regardless of location Full line rate ports, no fabric oversubscription 8Gb or 16Gb hosts and targets Services consolidated in the core Easy expansion Ports Deployed Used Ports Storage Ports 3456 per fabric 6,912 total 5,760 16Gb 6,240 8Gb Gb, or 960 8Gb 240 Storage ports at 16Gb (optional 8Gb without changing bandwidth ratios) 240 ISLs from storage edge to 16Gb 240 ISLs from host edge to 16Gb A Fabric Shown, Repeat for B Fabric MDS 9710 Host Ports ISL ports Host ISL Oversubscription End to End Oversubscription 3360 total 960 total 16Gb 16Gb storage 8Gb storage Gb or 16Gb 106

89 Very Large Edge-Core/End-of-Row Design Very Large Edge/Core/Edge (6144 End Device Ports per Fabric) Traditional Core-Edge design Is ideal for very large centralized services and consistent hostdisk performance regardless of location Full line rate ports, no fabric oversubscription 16Gb hosts and targets Services consolidated in the core Easy expansion Ports Deployed 12,288 Used Ports Storage Ports 16Gb 16Gb 576(288 per switch) Storage ports at 16Gb 768(48 per switch) ISLs from host edge to 16Gb 24 A Fabric Shown, Repeat for B Fabric MDS 9718 MDS 9710 Host Ports 8064 ISL ports 768 Host ISL Oversubscription 16Gb 4032 (252 per switch) 8Gb or 16Gb End to End Oversubscription 16Gb storage 107

90 SAN Top of Rack MDS 9148S SAN Top of Rack (5,376 Usable Ports) Ideal for centralized services while reducing cabling requirements 352 Storage ports at 16Gb MDS 9710 Consistent host/target performance regardless of location in rack 8Gb hosts & 16Gb targets Easy edge expansion Massive cabling infrastructure avoided as compared to EoR designs Additional efficiencies with in rack IO convergence Ports Deployed 5,376 Used Ports 5,344 4 ISLs from each edge to 16Gb 4,224 16Gb A B MDS 9148S Storage Ports 16Gb Host Ports 4,224 Host ISL Oversubscription End to End Oversubscription 16Gb 16G hosts 48 Racks 44 Dual-attached servers per rack Rack 108

91 Top-of-Rack Design - Blade Centers SAN Top of Rack Blade Centers (1,920 Usable Ports per Fabric) Ideal for centralized services 96 Storage ports at 16Gb MDS 9710 Consistent host/target performance regardless of location in blade enclosure or rack 8Gb hosts & 16Gb targets Need to manage more SAN Edge switches/blade Switches NPV attachment reduces fabric complexity A B Assumes little east-west SAN traffic Add blade server ISLs to reduce fabric oversubscription 8 ISLs from each edge to 8Gb Blade Center Ports Deployed 1,920 Used Ports Storage Ports 16Gb 8Gb 16Gb, or 8Gb 960 8Gb Host Ports 2304 Host ISL Oversubscription End to End Oversubscription 8G 16Gb Storage 8Gb Storage Rack 12 Racks, 72 chassis 96 Dual-attached blade servers per rack 109

92 Medium Scale Dual Fabric Collapsed Core Design Medium Scale Dual Fabric (768 Usable Ports per Fabric) Ideal for centralized services Consistent host/target performance regardless of location 8Gb or 16Gb hosts & targets (if they exist) Relatively easy edge expansion to Core/Edge EoR design Supports blade centers connectivity 96 Storage ports at 16Gb A Fabric Shown, Repeat for B Fabric MDS 9710 Ports Deployed 768 Used Ports 768@ 16Gb Storage Ports 16Gb Gb Host Ports Host ISL Oversubscription 16Gb N/A End to End Oversubscription 16Gb 110

93 POD SAN Design POD SAN Design Ideal for centralized services Consistent host/target performance regardless of location in blade enclosure or rack Storage ports at 16Gb 10/16Gb hosts & 16Gb targets Need to manage more SAN Edge switches/blade Switches MDS 9396S MDS 9396S NPV attachment reduces fabric complexity Add blade server ISLs to reduce fabric oversubscription 6 ISLs from each edge to 16Gb A B 8 ISLs from each edge to 8Gb MDS 9148S UCS FI 6248UP 6 Racks, 252 chassis 42 Dual-attached servers per rack 6 Racks, 288 blades 48 Dual-attached blade servers per rack Gb or Gb 111

94 FI UP, FI 6332 UCS SAN Design FI UP Use Case FI 6332 Use Case Nexus 7K/9K 40G 16G FC Nexus 7K/9K 40G 40G FCoE FI UP FI 6332 UCS B-Series B200 B260 B460 and 40G UCS C-Series C220 C240 C460 40G MDS 9700 Storage Array UCS B-Series B200 B260 B460 and 40G UCS C-Series C220 C240 C460 40G MDS 9700 Storage Array IOM 2304 IOM G 40G 16G FC 40G FCoE 112

95 Intelligent SAN Services 113

96 Enhancing SAN Design with Services Extend Fabrics FCIP Extended Buffer to Buffer credits Encrypt the pipe SAN Services extend the effective distance for remote applications SAN IO acceleration Write acceleration Tape acceleration SAN Extension with FCIP Enhance array replication requirements Reduces WAN-induced latency Improves application performance over distance Data Migration Data Migration with DMM IO Acceleration Fabric is aware of all data frames from initiator to target 114

97 SAN Extension with FCIP Fibre Channel over IP Encapsulation of Fibre Channel frames into IP packets and tunneling through an existing TCP/IP network infrastructure, in order to connect geographically distant islands Write Acceleration to improve throughput and latency Hardware-based compression Hardware-based IPSec encryption Array to Array Replication FCIP Tunnel TE Port 115

98 FC Redirect - How IOA Works Replication Starts Replication Starts Initiator to target Flow redirected to IOA Engine Flow accelerated and sent towards normal routing path IOA IOA MAN/WAN IOA IOA IOA= I/O Accelerator Initiator Target Initiator Target Virtual Initiator Virtual Target 116

99 Data Acceleration A fabric service to accelerate I/O between SAN devices Accelerate SCSI I/O Over both Fibre Channel (FC) and Fibre Channel over IP (FCIP) links For both Write Acceleration (WA) and Tape Acceleration (TA) I/O Acceleration Node platforms: MSM-18/4, SSN-16, MDS-9222i, MDS-9250i Uses FC Redirect IOA IOA MAN/WAN IOA IOA IOA= I/O Accelerator 117

100 IOA FCIP Tape Backup Large Health Insurance Firm MDS IOA Results 92% throughput FCIP increase Highly resilient Clustering of IOA engines allows for load balancing and failover Improved Scalability- Scale without increasing management overhead Significant reutilization of existing infrastructure- All chassis and common equipment re-utilized Flat VSAN topology- Simple capacity and availability planning 118

101 SAN Extension FC over long distance BB_Credits and Distance 2 Gbps FC ~1 km per Frame 4 Gbps FC 8 Gbps FC 16 Gbps FC ~0.5 km per Frame ~0.25 km per Frame ~0.125 km per Frame 16 Km phx2-9513(config)# feature fcrxbbcredit extended phx2-9513(config)# interface 1/1 phx2-9513(config-if)# switchport fcrxbbcredit extended 1000 phx2-9513# show interface 1/1 fc1/1 is up.. Transmit B2B Credit is 128 Receive B2B Credit is 1000 BB_Credits are used to ensure enough FC frames in flight A full (2112 byte) FC frame is approx 1 km 2 Gbps, ½ km 4 Gbps ¼ km long at 8 Gbps As distance increases, the number of available BB_Credits need to increase as well Insufficient BB_Credits will throttle performance - no data will be transmitted until R_RDY is returned 119

102 SAN Extension FCoE over long distance FCoE Flow Control For long distance FCoE, receiving switch Ingress Buffer must be large enough to absorb all packets in flight from the time the Pause frame is sent to the to time the Pause Frame is received A 10GE, 50 km link can hold ~300 frames That means 600+ frames could be either in flight or will be transmitted by the time the receiver detects buffer congestion and sends a Pause frame to the time the Pause frame is received and the sender stops transmitting Buffer Threshold Egress Buffer Frame Frame Latency Buffer Frame Frame Pause threshold Frame Frame Frame Frame Frame Ingress Buffer Frame Frame Frame Frame Frame Pause Latency Buffer turning is platform specific 120

103 Data Mobility Application I/O Application Servers Data Mobility Manager Old Array Data Migration New Array Migrates data between storage arrays for Technology refreshes Workload balancing Storage consolidation DMM offers Online migration of heterogeneous arrays Simultaneous migration of multiple LUNs Unequal size LUN migration Rate adjusted migration Verification of migrated data Dual fabric support CLI and wizard-based management with Cisco Fabric Manager Not metered on no. of terabytes migrated or no. of arrays Requires no SAN reconfiguration or rewiring Uses FC Redirect 121

104 SAN Extension - CWDM Course Wavelength Division Multiplexing TX TX TX TX Optical transmitters Transmission 8 channels WDM using 20nm spacing Colored CWDM SFPs used in FC switch Optical multiplexing done in OADM Passive device Optical fiber pair OADM RX RX RX RX Optical receivers 122

105 SAN Extension - DWDM Dense Wavelength Division Multiplexing TX TX TX TX Optical transmitters Transmission Optical Splitter Protection Optical fiber pair DWDM devices RX RX RX RX Optical receivers DWDM systems use optical devices to combine the output of several optical transmitters Higher density technology compared with CWDM, <1nm spacing 123

106 Dense vs Coarse (DWDM vs CWDM) DWDM CWDM Application Long Haul Metro Amplifiers Typically EDFAs Almost Never # Channels Up to 80 Up to 8 Channel Spacing 0.4 nm 20nm Distance Up to 3000km Up to 80km Spectrum 1530nm to 1560nm 1270nm to 1610nm Filter Technology Intelligent Passive Site 1 Site 2 Site 1 Site 2 MDS ONS Array DWDM CWDM 124

107 Summary Drivers in DC are forcing change 10G convergence & server virtualization It's not just about FCP anymore. FCoE, NFS, iscsi are being adopted Proper SAN design is holistic in the approach Many design options Optimized for performance Some for management Others for cable plant optimization Performance, Scale, Management attributes all play critical roles Not all security issues are external Fault isolation goes beyond SAN A/B separation Consider performance under load Design for SAN services 125

108 Additional Relevant Sessions Storage Networking Cisco Live Berlin BRKSAN SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric Friday 9AM 126

109 Call to Action Visit the World of Solutions for: Multiprotocol Storage Networking booth See the MDS 9718, Nexus 5672UP, 2348UPQ, and MDS 40G FCoE blade Data Center Switching Whisper Suite Strategy & Roadmap (Product portfolio includes: Cisco Nexus 2K, 5K, 6K, 7K, and MDS products). Technical Solution Clinics Meet the Engineer Available Tuesday and Thursday 128

110 Complete Your Online Session Evaluation Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt. All surveys can be completed via the Cisco Live Mobile App or the Communication Stations 129