Versatile architecture using Nexus 7000 with a mix of F and M modules to deliver FEX, FabricPath, MPLS, LISP and Multihop FCoE all at the same time

Transcription

1

2 Versatile architecture using Nexus 7000 with a mix of F and modules to deliver FEX,, PLS, LISP and ultihop FCoE all at the same time David Klebanov Technical Solutions Architect klebanov@cisco.com Umar Shafiq Technical Solutions Architect

3 After Lunch 3

4 Recommended Pre-requisites The following sessions are recommended to provide technical background BRKARC-3470 Cisco Nexus 7000/7700 Switch Architecture BRKDCT-2121 Virtual Device Context (VDC) Design and Implementation Considerations with Nexus 7000 BRKDCT-2048 Deploying Virtual Port Channel in NXOS (Not offered in SF) BRKDCT-2081 Cisco Technology and Design BRKSAN-2883 Advanced Storage Area Network Design BRKSAN-2378 Evolution of Connectivity options for FCoE environments BRKPL-2108 Designing PLS in Next Generation Data Center BRKRST-3045 LISP A Next Generation Networking Architecture Don t worry we will provide technology refreshers throughout the presentation 4

5 Icons and Color Coding Network () Cisco Nexus 7000/7700 Switch Network (Classical Ethernet) Routed Links FEX Fabric Links Cisco Nexus 5500/6000 Switch Cisco Nexus 2200 Fabric Extender (1Gb) FCoE Cisco Nexus 2200 Fabric Extender (1/10Gb) Fibre Channel Logical Links F Cisco Nexus 7000/7700 I/O odules (with color coding) ABC Command Line Interface Configuration Snippet 5

6 Consolidated Feature-rich Deployment Design Goals Cabling consolidation High speed SAN Unified Fabric FCoE Fabric N5K/N6K FC Fabric WAN connectivity Segmentation ultitenancy DS PLS obility aware routing Any workload anywhere Easy server provisioning ultitenancy F F LISP Hardware investment protection Design versatility 1Gb FEX 10Gb FEX 10Gb FEX Efficient cabling Simplified management Cost reduction CNA CNA 1Gb Server 1/10 Gb Server 10 Gb Server 6

7 Can t we just deploy it like this? N5K/N6K FCoE and FEX F2 F2E FCoE or FEX (+) DS FCoE Fabric FC Fabric PLS FP and PLS and LISP F2 F2E FP, no PLS/LISP (o) FP, no PLS/LISP (+) FEX, no FCoE (+) FEX, no FCoE (o) FP and FEX F2 F2E FP and FEX F F LISP LISP/PLS, no FP (o) How do we solve it today? FEX, no FP (o) 1Gb FEX 10Gb FEX 10Gb FEX + o Pending software support Not hardware capable CNA CNA 1Gb Server 1/10 Gb Server 10 Gb Server 7

8 Solution Overview Leveraging Nexus 7000 Virtual Device Contexts Design A Design B DS Storage VDC 1 FCoE Fabric N5K/N6K CE VDC 1 FC Fabric vpc CE VDC 2 FCoE Fabric Storage Storage VDC 2 VDC 1 FP vpc+ FP VDC 1 VDC 2 DS N5K/N6K FP VDC 1 FC Fabric Storage VDC 2 vpc+ FP VDC 2 CNA CNA CNA CNA 1Gb FEX 10Gb FEX 1Gb FEX 10Gb FEX 10 Gb Server 10 Gb Server 1Gb Server 1/10 Gb Server 1Gb Server 1/10 Gb Server 8

9 Discussion Agenda Fabric Extenders attachment to Nexus 7000 switches Fabric Extenders and Cisco on Nexus 7000 switches Cisco and PLS/LISP on Nexus 7000 switches ultihop FCoE on Nexus 7000 switches 9

10 FEX Attachment to Nexus 7000 Design A Design B N5K/N6K N5K/N6K DS Storage VDC 1 FCoE Fabric CE VDC 1 FC Fabric Storage VDC 2 FP VDC 1 vpc vpc+ CE VDC 2 FP VDC 2 DS Storage VDC 1 FCoE Fabric FP VDC 1 FC Fabric Storage VDC 2 vpc+ FP VDC 2 CNA CNA CNA CNA 1Gb FEX 10Gb FEX 1Gb FEX 10Gb FEX 10 Gb Server 10 Gb Server 1Gb Server 1/10 Gb Server 1Gb Server 1/10 Gb Server 10

11 Switching and Routing Component Cisco Nexus 7000 Nexus 7009 Nexus 7010 Nexus 7004 Nexus 7009 Nexus 7018 Nexus 7010 Nexus 7004 Nexus 7718 Nexus 7710 Nexus 7706 Nexus 7018 Design A Design B 1/2 modules are not supported in Nexus 7700 chassis CE VDC 1 1Gb FEX vpc 10Gb FEX CE VDC 2 FP VDC 1 1Gb FEX vpc+ 10Gb FEX FP VDC 2 Nexus 7700 F2E and only F2E and modules are different between N7000 and N7700 1Gb Server 1/10 Gb Server 1Gb Server 1/10 Gb Server 11

12 Port Extension Component Cisco Nexus 2000 Nexus 2248TP(E) Nexus 2224TP Nexus 2232PP Nexus 2232T(E) Nexus B22HP Nexus 2248PQ(E) Design A Design B CE VDC 1 vpc CE VDC 2 FP VDC 1 vpc+ FP VDC 2 1Gb FEX 10Gb FEX 1Gb FEX 10Gb FEX 1Gb Server 1/10 Gb Server 1Gb Server 1/10 Gb Server 12

13 Distributed Switch Cisco Nexus Distributed Switch Top of Rack Connecitivity, Centralized anagement Centralized anagement Nexus 7000/7700 Parent Switches Top of Rack Server Cabling Top of Rack Fabric Extenders Top of Rack Connectivity Redundancy Nexus 2000 Fabric Extenders In-rack servers 13

14 Cisco Nexus 7000 Parent Switch Requirements Fabric Extender Attachment -132XP-12L 10G -224XP-23L 10G -206FQ-23L 40G Parent I/O odule N77-F248XP-23E 10G -F248XP-25(E) 10G N77-48XP-23 10G N77-24FQ-25 40G -12FQ-25 40G Supervisor Sup 1* Sup 2/2E -SUP1-8GBUPG * End-of-Sale Design A Note: N7700 only Sup2E Software 2 6.1(1) F2 6.0(1) F2E 6.1(2) 6.2(6) N (2) N (6) NX-OS 14 Design B CLI Install and activate the feature - No special license is required (config)# install feature-set fex (config)# feature-set fex Note: Please refer to cisco.com for full configuration

15 Cisco Nexus 7000 Parent Switch Cabling Options Fabric Extender Attachment -224XP-23L -132XP-12L -F248XP-25(E) -206FQ-23L -12FQ-25 N77-48XP-23 N77-F248XP-23E N77-24FQ-25 Fiber / Twinax 10G Fiber / Twinax 40G 10G 10G Nexus 2232PP Nexus 2232T(E) Nexus 2232PP Nexus 2232T(E) Nexus 2248TP(E) Nexus 2224TP Nexus 2248TP(E) Nexus 2224TP 15

16 Cisco Nexus 7000 Parent Switch Cabling Options Fabric Extender Attachment -224XP-23L -132XP-12L -F248XP-25(E) -206FQ-23L -12FQ-25 N77-48XP-23 N77-F248XP-23E N77-24FQ-25 Fiber / Twinax 10G Fiber / Twinax 40G 40G 40G Nexus 2248PQ(E) Nexus 2248PQ(E) 16

17 FEX Integration Topology Connectivity Topology into Nexus 7000 Parent Switches FEX vpc vpc+ FEX Design A Design B Fabric Extenders can currently be attached in straight-through topology only* - Same topology for 1/2 and F2/F2E/ Each parent Nexus 7000 switch is fully redundant Different from the N5K/N6K N5K/N6K vpc N5K/N6K N5K/N6K vpc N5K/N6K FEX FEX FEX FEX FEX FEX * Check documentation for the latest information 17

18 FEX Integration Topology SoC Ports Symmetry on F2/F2E I/O odules F2/F2E SoC SoC Fabric Extender F2/F2E SoC SoC Fabric Extender F2/F2E SoC is a Switch on Chip and it defines a portgroup FEX uplinks must have symmetric connectivity into ports across port-groups Port-group can be dispatched on different F2/F2E odules ultiple FEX can share a same port-group Design B Port-Group {2,4} {6,8} Port-Group {2,4} {6,7} 18

19 FEX Attachment Scalability aximum FEX Attachment Count vpc vpc N7000 N7000 N7700 N7700 Cisco Nexus 7000 currently supports*: - Sup 1/2 supports 32 Fabric Extenders - Sup 2E supports 48 Fabric Extenders with 6.1(1) - Sup 2E supports 64 Fabric Extenders with 6.2(2) Cisco Nexus 7700 currently supports 64 Fabric Extenders* ix and match any supported FEX type on the same Nexus 7000/7700 parent switch * Check documentation for the latest information 19

20 Server Connectivity Topologies F vpc F Active/Active non-virtualized servers - Port channeling on server s - Leverage vpc/vpc+ on Nexus 7K - Supported on starting from NX-OS 5.2(1) FEX FEX Active/Active virtualized servers - Same as non-virtualized OR - No port channeling on server s o Virtual Switch Load Balancing X Can result in traffic blackholing during vpc Dual-Active condition Single Active/Standby Active/Active Non-Virt ed Hypervisor Active/Active Virtualized Single and Active/Standby X Can result in traffic blackholing during vpc Dual-Active condition 20

21 vpc Dual-Active Condition Explained vpc Primary vpc Domain vpc Keepalive vpc Peer-Link vpc Secondary vpc Dual-Active is triggered when a. vpc Peer-Link goes down b. vpc Keepalive stays up vpc Secondary shuts down all vpc member ports and vpc VLAN SVIs DS Storage VDC 1 Applies only to this design option FCoE Fabric N5K/N6K Upstream vpc Domain CE VDC 1 1Gb FEX 1Gb Server FC Fabric Storage VDC 2 FP VDC 1 vpc 10Gb FEX 1/10 Gb Server vpc+ CNA CE VDC 2 CNA 10 Gb Server FP VDC 2

22 Active/Standby Servers vpc Dual Active and Orphan Ports ( I/O odules) vpc Primary Po Standby vpc Peer-Link Po Dual Active Active vpc Secondary Active server is connected to vpc Secondary (important) - Network admin cannot predict which one will become active - This is an Orphan Port* During vpc Dual-Active condition vpc Secondary will shut down all vpc member port Uplink ports towards VDC are vpc member ports, they will be shutdown * Port in vpc Domain that belongs to non-vpc attached device 22

23 Active/Standby Servers Traffic Blackholing ( I/O odules) vpc Primary Po Standby vpc Peer-Link Po Dual Active E101/1/1 Active vpc Secondary By default, vpc Dual-Active does not shut down Orphan Ports, which is where Active/Standby servers are connected - Reason is Orphan Ports are not vpc member ports Active/Standby servers will keep on sending traffic towards vpc Secondary, which now has nowhere to send it to - vpc Peer-Link is down - Uplink was shutdown by vpc Dual-Active condition Server traffic is blackholed 23

24 Active/Standby Servers Solving Traffic Blackholing using Orphan Port Suspend Feature ( I/O odules) vpc Primary Po Po vpc Secondary Solution is to have vpc shut down Orphan Ports on vpc Secondary during Dual-Active condition - Orphan Port suspend feature (config)# interface ethernet 101/1/1 (config-if)# vpc orphan-ports suspend vpc Peer-Link Dual Active Active Standby E101/1/1 Active/Standby server will perform switchover and carry the traffic along the active path - Relies on healthy server operation Recommendation: Always prefer Active/Active port-channel server connectivity into vpc domain 24

25 Singly Connected Servers vpc Dual Active Traffic Blackholing ( I/O odules) vpc Primary Po vpc Peer-Link Dual Active Po Active vpc Secondary E101/1/1 vpc Primary Po vpc Peer-Link Dual Active Po Active vpc Secondary E101/1/1 orphan-port suspend feature does not make difference for singly connected servers - Can use OS scripts to take action based on status - With orphan-port suspend feature being OFF, server will only rely on transmission timeouts to determine loss of connectivity Singly connected servers need to be physically re-plugged to the vpc Primary or stay down until vpc domain fully recovers from Dual- Active condition vpc orphan-port suspend feature OFF vpc orphan-port suspend feature ON 25

26 vpc Primary Active/Active Virtualized Servers Solving Traffic Blackholing using Orphan Port Suspend Feature ( I/O odules) E102/1/1 Active Po vpc Peer-Link Dual Active Hypervisor Port Channeling Po vpc Secondary E101/1/1 Active vpc Primary Po E102/1/1 Active vpc Peer-Link Dual Active Hypervisor vpc orphan-port suspend feature Po vpc Secondary E101/1/1 Active 26 No Port Channeling on server - Traffic is pinned to an uplink by the virtual switch in the hypervisor - Ports are considered Orphan Ports During vpc Dual-Active V traffic pinned to server connected to vpc Secondary will be blackholed Solution is to use Orphan Port suspend feature or even better use Port Channeling to achieve Active/Active traffic forwarding - No need for Orphan Port suspend feature

27 vpc+ Dual-Active Condition and Design Leveraging F2/F2E/ I/O odules FP VDC vpc Primary Active E101/1/1 vpc+ Peer-Link Dual Active E101/1/1 Standby Standby vpc Secondary Active Traffic blackholing does not apply to design when utilizing F2/F2E/ I/O modules for FEX attachment into Cisco Nexus 7000 switches No vpc domain upstream, no vpc member ports to shut down in case vpc Dual-Active condition is triggered During vpc Dual-Active traffic is forwarded through network - No need for vpc Orphan Port Suspend feature 27

28 FEX Attachment to Nexus 7000 Switches Progress Checkpoint vpc All design conditions had been satisfied! FEX FEX Servers are connected to Fabric Extenders Fabric Extenders are attached to Cisco Nexus 7000 parent switches Single Active/Standby Active/Active Non-Virt ed Hypervisor Active/Active Virtualized Possible traffic blackholing scenarios had been addressed 28

29 Discussion Agenda Fabric Extenders attachment to Nexus 7000 switches Fabric Extenders and Cisco on Nexus 7000 switches Cisco and PLS/LISP on Nexus 7000 switches ultihop FCoE on Nexus 7000 switches 29

30 Cisco and FEXs on Cisco Nexus 7000 Design A Design B N5K/N6K N5K/N6K DS Storage VDC 1 FCoE Fabric CE VDC 1 FC Fabric Storage VDC 2 FP VDC 1 vpc vpc+ CE VDC 2 FP VDC 2 DS Storage VDC 1 FCoE Fabric FP VDC 1 FC Fabric Storage VDC 2 vpc+ FP VDC 2 CNA CNA CNA CNA 1Gb FEX 10Gb FEX 1Gb FEX 10Gb FEX 10 Gb Server 10 Gb Server 1Gb Server 1/10 Gb Server 1Gb Server 1/10 Gb Server 30

31 Cisco Selected Features and Characteristics STP From PODs to Fabric Optimum Low Latency Switching Conversational Learning Active VPC+ Classical Ethernet Active Layer 3 Layer 2 Anycast HSRP Unicast and ulticast ultipathing - High bandwidth and High Resilience *** 31 CE Device Classical Ethernet Integration (vpc+) All Active Default Gateway

32 Cisco Nexus and Edge Using F2, F2E and I/O odules Design B vpc+ FP VLANs Fabric Extenders are connected to F-series modules F2, F2E and I/O modules support Fabric Extender attachment F2, F2E and I/O modules support FEX FEX Hypervisor Server traffic is forwarded through Fabric Extenders, Nexus 7000 switches and on to the rest of the environment 32

33 Cisco Nexus and Edge Using F-series and -series I/O odules in the same VDC Design A FEX vpc+ FP VLANs FEX Hypervisor Fabric Extenders are connected to -series modules, we also need F-series for VDC VDC VDC F2 F2E No interop Server VLANs extended over the network must be set to mode mode VLANs cannot exist on -series I/O modules Lowest common denominator -series I/O modules do not support 33

34 Nexus 7000 Device Virtualization VDC Port Assignment FP VDC (1) FP VDC (2) Nexus 7000 is virtualized into two VDCs - Classical Ethernet VDC - VDC Ports on -series I/O modules are assigned to Classical Ethernet VDC - ust observe port ASIC boundaries CE VDC (1) CE VDC (2) Ports on F2, F2E or I/O modules are assigned to VDC - ust observe Switch-On-Chip boundaries 34

35 FP VDC and CE VDC Interconnect Virtual Port Channels FP VDC (1) CE VDC (1) vpc+ vpc FP VDC (2) CE VDC (2) VDCs are interconnected using external cables - Two VDCs cannot communicate internally - Can use TwinAx Classical Ethernet VDC participates in vpc domain VDC participates in vpc+ Domain Back-to-back vpc design 35

36 FEX Attachment and Connectivity FP VDC (1) CE VDC (1) 1Gb FEXs vpc+ vpc 10Gb FEXs FP VDC (2) CE VDC (2) Configuration Classical Ethernet and FEX Configuration F2, F2E or ports in FP VDC are used to connect to network Fabric Extenders are attached to - series I/O modules in a CE VDC All Classical Ethernet and FEX configurations are done in the Classical Ethernet VDC All configurations are done in VDC 36

37 Extending VLANs Between VDCs VLAN ode Setting and Trunking All VLANs are operating in FP VDC (1) CE VDC (1) FP VLANs vpc+ vpc All VLANs FP VDC (2) CE VDC (2) Classical Ethernet mode in CE VDC - Classical Ethernet mode is a default VLAN mode VLANs requiring transport are trunked between Classical Ethernet VDC and VDC over the vpc/vpc+ - Defined as mode VLANs in VDC n7k(config)# vlan <vlan> n7k(config-vlan)# mode fabricpath 1Gb FEXs 10Gb FEXs 37

38 Extending VLANs Between VDCs VDC and STP Root Priority FP VDC (1) CE VDC (1) 1Gb FEXs FP VLANs vpc+ vpc All VLANs 10Gb FEXs FP VDC (2) CE VDC (2) Useful Reminder ake sure that VDC has lower STP root priority - network must be the root of local STP domain - edge ports run RootGuard and will place VLANs in root-inconsistent state if better BPDUs are received from the local Classical Ethernet environment N7k(config)# spanning-tree vlan < vlan> priority 8128 STP Root Priority must match on both s in the FP VDC 38

39 Server Traffic in the Network

40 Global VLANs All VLANs are forwarded over Network Routed Leaf Layer 3 Layer 2 Leaf vpc+ FP VDC 1 FP VDC 2 R Spine Any VLAN anywhere - VLAN scope is entire FP network Routing functionality is provided by the Routed Leaf nodes - Please refer to the design sessions for more in-depth discussion about fabric routing functionality Routed Leaf nodes perform inter- VLAN routing sending traffic between source and destination server subnets FP VLANs VLAN Scope FP VLANs R Routing Decision Point 40

41 Local VLANs Some VLANs are forwarded over Network Routed Leaf Leaf R vpc+ FP VDC 1 FP VDC 2 Layer 3 Layer 2 Spine Locally significant VLANs - Some servers need to communicate only within their own Leaf Layer node block - No workload mobility across blocks Routing functionality is provided by the local Leaf nodes Leaf nodes Nexus 7000 switches perform inter-vlan routing in VDC sending traffic between source and destination servers FP VLANs VLAN Scope R Routing Decision Point 41

42 Interconnecting Global and Local VLANs Routing Between Leaf and Spine Routed Leaf Leaf vpc+ FP VDC 1 FP VDC 2 Transit VLAN Spine Layer 3 Layer 2 Layer 3 Layer 2 VLANs extended over the network (L3 at Routed Leaf nodes) need to communicate with VLANs that aren t (L3 at local Leaf nodes) Transit VLAN is used to route traffic between Routed Leaf nodes and local Leaf nodes Routed Leaf nodes and local Leaf nodes establish routing protocol adjacency across the Transit VLAN Local VLAN Scope Global VLAN Scope IGP Transit VLAN 42

43 Interconnecting Global and Local VLANs Routing Between Leaf and Spine Routed Leaf R Layer 3 Layer 2 Local Leaf nodes make routing decision sending the traffic towards the Routed Leaf nodes across the Transit VLAN Leaf R vpc+ FP VDC 1 FP VDC 2 Transit VLAN Spine Layer 3 Layer 2 S Routed Leaf nodes make routing decision sending the traffic towards the Layer 2 destination Leaf nodes Destination Leaf nodes forward the traffic to the destination servers - No routing decisions are made on destination Leaf nodes Local VLAN Scope Routing Decision Point R S Switching Decision Point Global VLAN Scope 43

44 Interconnecting Global and Local VLANs Routing Between Leaf and Spine Routed Leaf R Layer 3 Layer 2 In the opposite direction local Leaf nodes send the Global VLAN traffic towards the Routed Layer nodes Leaf R vpc+ Transit VLAN Spine Layer 3 Layer 2 S Routed Leaf nodes make routing decision sending the traffic towards the Layer 3 destination Leaf nodes across Transit VLAN Destination Leaf nodes route the traffic to the destination servers FP VDC 1 FP VDC 2 Local VLAN Scope Routing Decision Point R S Switching Decision Point Global VLAN Scope 44 44

45 Interconnecting Local VLANs Routing Between Leaf Nodes Routed Leaf Leaf R Transit VLAN Spine Layer 3 R Layer 3 Local to Local VLANs on a different pairs of Leaf Nodes communicate across the Transit VLAN Traffic does not cross Routed Leaf nodes vpc+ Layer 2 Layer 2 FP VDC 1 FP VDC 2 Local VLAN Scope Local VLAN Scope R Routing Decision Point 45

46 Pervasive Routing Dynamic Fabric Automation Spine RR RR RR RR Routing on all Leaf nodes - No need for Transit VLAN Host-routed network - BGP Dynamic SVI instantiation based on workload location Leaf RR Client vpc+ RR Client R RR Client RR Client Layer 3 Layer 2 For more information and details please refer to BRKDCT-2385 FP VDC 1 FP VDC 2 FP VLANs VLAN Scope FP VLANs 46 R Routing Decision Point

47 FEX and on Nexus 7000 switches Progress Checkpoint All design conditions had been satisfied! Servers connected to Fabric Extenders can communicate across network Global VLANs can be forwarded across network Global and Local VLANs can communicate with each other 47

48 Discussion Agenda Fabric Extenders attachment to Nexus 7000 switches Fabric Extenders and Cisco on Nexus 7000 switches Cisco and PLS/LISP on Nexus 7000 switches ultihop FCoE on Nexus 7000 switches 48

49 Cisco and PLS/LISP on Nexus 7000 Design A Design B N5K/N6K N5K/N6K DS Storage VDC 1 FCoE Fabric CE VDC 1 FC Fabric Storage VDC 2 FP VDC 1 vpc vpc+ CE VDC 2 FP VDC 2 DS Storage VDC 1 FCoE Fabric FP VDC 1 FC Fabric Storage VDC 2 vpc+ FP VDC 2 CNA CNA CNA CNA 1Gb FEX 10Gb FEX 1Gb FEX 10Gb FEX 10 Gb Server 10 Gb Server 1Gb Server 1/10 Gb Server 1Gb Server 1/10 Gb Server 49

50 Why PLS What s important in our case: Provides convenient and proven connectivity method beyond single Data Center fabric aintains end-to-end multi-tenancy properties and traffic segregation/segmentation Supported on the Cisco Nexus 7000 switches utilized in our design 50

51 Delivering and PLS PLS F2 F2E PLS * * stay tuned for software support In the meantime VDCs to the rescue! 51

52 and PLS Option1: Separate Tiers, Attached at the Routed Leaf PLS VDC (PLS Features) Tenant/ Org Tenant/ Org PLS Tenant/ Org Separate tiers of Cisco Nexus 7000 switches/vdcs Routed Leaf Point-to-point Layer 3 links VDC ( Features + Routing) Tenant/ Org vpc+ Tenant/ Org Tenant/ Org Server Default Gateway 52

53 and PLS ultitenancy Option1: Separate Tiers, Attached at the Routed Leaf VDC VRF Lite PLS VDC FP VL A VRF A 802.1Q trunk VRF A Interconnect VLAN1 FP VL B VRF B Interrconnect VLAN2 VRF B Interconnect VLAN3 PLS FP VL C VRF C VRF C Layer 2 SRV SVIs (DefGW) Interconn SVIs Layer 3 PLS SVIs* Label Switching * Can be sub-interfaces instead Server SVIs and Interconnect SVIs are mapped to the same VRF for any given tenant/org PLS SVIs are mapped to the PLS VRF for any given tenant/org VLANs are extended across PLS WAN 53

54 and PLS Option2: Separate Tiers, Attached at L2 Leaf PLS VDC (PLS Features + Routing) Leaf Tenant/ Org Tenant/ Org PLS vpc Tenant/ Org Separate tiers of Cisco Nexus 7000 switches/vdcs Server Default Gateway Layer 2 trunk links VDC ( Features) Tenant/ Org vpc+ Tenant/ Org Tenant/ Org 54

55 and PLS ultitenancy Option2: Separate Tiers, Attached at L2 Leaf VDC VLAN Trunking PLS VDC FP VL A 802.1Q trunk VRF A FP VL B Tenant/Org VLAN A Tenant/Org VLAN B Tenant/Org VLAN C VRF B PLS FP VL C VRF C Layer 2 SRV SVIs* (DefGW) Label Switching * Can be sub-interfaces instead FP VLANs are mapped to any given tenant/org Server SVIs are mapped to PLS VRF for any given tenant/org VLANs are extended across PLS WAN 55

56 Proxy-Routing Cisco Nexus 7000 with F2E and -series I/O modules in the same VDC F2E I/O modules are placed in the same VDC as the -series I/O modules - Cannot use F2 I/O modules SVI SVI F2E SVI Nexus 7000 VDC Internal Port-Channel Layer 3 (SVIs) is configured Internal Port-Channel is formed between the F2E and -series I/O modules Traffic received on the F2E I/O modules requiring routing function is forwarded to the -series I/O modules over the Internal Port-Channel links * F2E- proxy-routing requires NX-OS

57 and PLS Option3: Single Tier, Proxy Routing PLS features + Routing Tenant/ Org Tenant/ Org PLS Tenant/ Org Single tier of Cisco Nexus 7000 switch/vdc Server Default Gateway (-series Line Cards) Routed Leaf F2E vpc+ F2E Proxy-Routing features F2E F2E Tenant/ Org Tenant/ Org Tenant/ Org 57

58 and PLS ultitenancy Option3: Single Tier, Proxy Routing Internal Forwarding FP VL A Internal Port-Channel VRF A FP VL B VRF B PLS FP VL C VRF C Layer 2 SRV SVIs (DefGW) Label Switching FP VLANs are mapped to any given tenant/org Server SVIs are mapped to PLS VRF for any given tenant/org VLANs are extended across PLS WAN 58

59 Why LISP What s important in our case: Provide Virtual achine mobility aware routing beyond a single Data Center Supported on the Cisco Nexus 7000 switches utilized in our design 59

60 LISP Refresher Traffic Forwarding 1 DNS entry: D.abc.com A > > > LISP site /24 S 3 apping Entry ITR EID-prefix: /24 Locator-set: , priority: 1, weight: 50 (D1) , priority: 1, weight: 50 (D2) Network LISP apping System ITR Ingress Tunnel Router ETR Egress Tunnel Router > ETR West-DC D / /24 East-DC

61 LISP Host obility ulti-data Center Remote non-lisp Site Remote LISP Site Re-query Network xtr LISP PxTR LISP PLS Network PLS LISP apping System xtr xtr xtr xtr LISP LISP LISP LISP Data Center A xtr Ingress/Egress Tunnel Router Spine V igration 61 Spine V Data Center B

62 Delivering, PLS and LISP in a Single VDC LISP PLS LISP F2 F2E PLS LISP * * * stay tuned for software support In the meantime VDCs to the rescue! Again! 62

63 , PLS and LISP (multihop) Option1: Separate Tiers, Attached at Routed Leaf LISP ultihop NX-OS 6.2(8) LISP apping System Separate tiers of Cisco Nexus 7000 switches/vdcs PLS VDC (PLS Features + LISP Features) Routed Leaf PLS LISP xtr (Encap/Decap) LISP LISP -132XP-12L Line Card Point-to-point Layer 3 links VDC ( Features + Routing) vpc+ EID obility Detection 63

64 , PLS and LISP Option2: Separate Tiers, Attached at Layer 2 Leaf LISP apping System Separate tiers of Cisco Nexus 7000 switches/vdcs PLS VDC (PLS Features + Routing + LISP Features) VDC ( Features) Leaf PLS LISP LISP -132XP-12L Line Card vpc vpc LISP xtr (Encap/Decap) Layer 2 links 64

65 , PLS and LISP Option3: Single Tier, Proxy Routing LISP apping System Single tier of Cisco Nexus 7000 switch/vdc PLS features + Routing + LISP 1 PLS LISP LISP 1 LISP xtr (Encap/Decap) -132XP-12L Line Card Routed Leaf F2E vpc+ F2E Proxy-Routing features F2E F2E 65

66 , PLS and LISP Progress Checkpoint All design conditions had been satisfied! Connectivity is extended beyond a single Data Center ulti-tenancy segregation is preserved Routing is dynamically adjusted for Virtual achine mobility 66

67 Discussion Agenda Fabric Extenders attachment to Nexus 7000 switches Fabric Extenders and Cisco on Nexus 7000 switches Cisco and PLS/LISP on Nexus 7000 switches ultihop FCoE on Nexus 7000 switches 67

68 ultihop FCoE on Cisco Nexus 7000 Design A Design B DS Storage VDC 1 FCoE Fabric N5K/N6K CE VDC 1 FC Fabric CE VDC 2 FCoE Fabric Storage VDC 2 Storage VDC 1 FP vpc+ FP VDC 1 VDC 2 vpc DS N5K/N6K FP VDC 1 FC Fabric Storage VDC 2 vpc+ FP VDC 2 CNA CNA CNA CNA 1Gb FEX 10Gb FEX 1Gb FEX 10Gb FEX 10 Gb Server 10 Gb Server 1Gb Server 1/10 Gb Server Same for both designs 1Gb Server 1/10 Gb Server 68

69 D a t a C e n t e r B r i d g i n g Storage Transport Technologies Landscape FCoE and DCB Operating Systems SCSI Layer FCP FCP iscsi FCP FCP SRP Priority Flow Control 802.1Qbb FCIP ifcp FC FCoE TCP TCP IP IP Ethernet TCP IP IB Enhanced Transmission Selection 802.1Qaz Our solution is based on mapping of Fibre Channel frames over Ethernet transport, aka FCoE FCoE must run on a lossless Ethernet fabric, aka Data Center Bridging 69 Data Center Bridging Exchange

70 Fibre Channel Forwarder Important FCoE concept Fibre Channel Forwarder Ethernet Bridge FCF is a logical Fibre Channel switch inside an FCoE (Ethernet) switch Fibre Channel login happens at FCF Eth Port Eth Port Eth Port Eth Port Eth Port Eth Port FCoE encapsulation/decapsulation happens at FCF 70

71 Cisco Nexus 7000 Fibre Channel Forwarder Storage VDC Storage VDC Storage VDC vpc Data VDC Data VDC Nexus 7000 Nexus 7000 FCoE Configuration Data Ethernet Configuration FCF on Cisco Nexus 7000 is delivered in a form of Storage VDC - Requires Storage License Storage VDC acts as Director Class SAN switch - Single Storage VDC per All FCoE related configurations are carried out in Storage VDC Storage VDC is counted against total number of VDCs currently supported on Cisco Nexus 7000 switches 71

72 FCoE Server Connectivity Unified Wire and Shared Interfaces Data VDC Shared IF F2/ F2E Storage VDC Data VDC Storage VDC Nexus 7000 Nexus 7000 CNA CNA F2/ F2E Shared IF Servers utilize the concept of Unified Wire Unified Wire plugs directly into an F2/F2E* interfaces, which can accommodate both network data and FCoE traffic - Support for Fabric Extenders is upcoming Interfaces are shared between Data and Storage VDC - Shared Interface (config)# vdc fcoe type storage (config-vdc)# allocate shared interface e2/1 * line cards will allow FCoE with future software updates 72

73 Server Traffic Over Unified Wire Traffic Forwarding over Shared Interfaces Shared interfaces utilize 802.1Q trunking to carry both network Data Data VDC Etype Other Shared IF F2/ F2E Storage VDC FCoE Etype FCoE/FIP Storage VDC Nexus 7000 Nexus 7000 FCoE VLANs Data VLANs CNA FCoE VLANs Data VLANs CNA Data Data VDC Etype Other F2/ F2E FCoE Etype FCoE/FIP Shared IF data and FCoE VLANs Traffic with FCoE and FIP ethertypes received over shared interfaces appears in Storage VDC for processing (0x8906, 0x8914) Traffic with other ethertype values appears in Data VDC There is no communication between data VDC and Storage VDC! 73

74 Server Traffic Over Inter-Switch Links and Uplinks Dedicated Wire Traffic Forwarding DS F2/ F2E Data Data VDC FCoE Fabric Data VLANs N5K/N6K FCoE VLANs FCoE Storage VDC FC Fabric Dedicated Wire F2/ F2E Data VLANs F2/ F2E Data Data VDC N5K/N6K FCoE VLANs F2/ F2E FCoE Storage VDC Inter-Switch Links use Dedicated Wire where network data and FCoE traffic are carried over separate physical links Data VLANs are forwarded towards the network FCoE VLANs are forwarded towards the FCoE fabric 74

75 End-to-End Unified Fabric Inter-Switch Links and ultihop FCoE DS FCoE Fabric FCF VE N5K/N6K VE FCF FC Fabric Hop N Nexus 7000 operates in FC switching mode - Cannot be in NPV mode - Need to assign FC Domain ID and participate in fabric zoning Consider the following alternative F2/ F2E VE VE F2/ F2E Hop 2 Storage VDC FCF F2/ F2E FCF F2/ F2E Storage VDC Hop 1 Nexus 5K/6K NPV No Domain ID No Zoning CNA CNA Leverage FEX Side A Side B 75

76 Side A Side B Dynamic FCoE SAN Fabrics Topology Choice Target NPV NPV Dynamic FCF relationships Logical SAN separation vs physical VNP VF FCF VNP VF FCF Choice Lossless Side A Side B Side A Increased redundancy and resiliency (multipathing) Side B VF VF VE VE VE VE VNP VNP FCF FCF FCF FCF NPV NPV Initiator Initiator Target Note: Available now for Nexus 5K and 6K with NX-OS R7.0(1)N1(1) 76

77 ultihop FCoE on Nexus 7000 switches Progress Checkpoint Last one DS FCoE Fabric FCF VE N5K/N6K VE FCF FC Fabric All design conditions had been satisfied! FCoE Servers are connected to Cisco Nexus 7000 switches Storage VDC F2/ F2E FCF F2/ F2E VE VE F2/ F2E FCF F2/ F2E Storage VDC Cisco Nexus 7000 switches use device virtualization in a form of Storage VDC for all FCoE functionality Side A CNA CNA Side B ultihop FCoE carries storage traffic across the Unified Fabric 77

78 Key Takeaways Solid foundation for the Data Center fabrics Unrestrained server mobility domains DS FCoE Fabric N5K/N6K FC Fabric PLS LISP End-to-end multi-tenancy and V mobility aware forwarding ultipathing and all-active forwarding Versatility in scaled out server access Consolidated and unified network and storage fabrics CNA CNA 1Gb FEX 10Gb FEX 10 Gb Server 1Gb Server 1/10 Gb Server 78

79 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 79

80