Building Scalable Data Center Networks with NX-OS and Nexus 7000 / 7700 BRKDCT-3445

Transcription

1

2 Building Scalable Data Center Networks with NX-OS and Nexus 7000 / 7700

3 3

4 Building Scalable Data Center Networks with NX-OS and Nexus 7000 / 7700 Arkadiy Shapiro Technical Marketing Engineer, Technical Marketing Manager and Nexus 7000 / 7700 Scale Czar arshapir@cisco.com

5 Session Goals At the end of the session, the participants should understand: Key parameters of Nexus 7x00 and NX-OS scalability How to squeeze the most out of your Nexus 7x00 6

6 Session Non-Goals This session will not focus on: Technical details of mentioned features Nexus switch hardware architecture Scalability considerations for NX-OS running on switches other than Nexus 7x00 Scalability aspects of Dynamic Fabric Automation (DFA) or Application Centric Infrastructure (ACI) 7

7 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary 8

8 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary 9

9 Why is this important? Scale is top-of-mind for many IT decision makers common RFP question DC consolidations and business growth Higher scale requirements Scale is a key reason for Nexus 7000/7700 sales and Supervisor 2E upgrades 10

10 Scale means many things Hardware Number of ports? Number of linecards? Table sizes? Software Uni-dimensional how far can one feature go by itself? Multi-dimensional what if you put multiple features together? VLANs ARPs / sec VPCs FHRP groups Pseudowires Routes FP switch ID s Aggressive timers VRFs Labels FEX Bridge domains BFD sessions ACLs OTV MACs BGP peers 12

11 What Is Nexus 7000? Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7000 designed for general-purpose Data Center deployments, focused on 10G density plus 40G/100G Supervisor Engines I/O Modules Chassis Fabrics 13

12 What Is Nexus 7700? Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7700 designed for SP and MSDC Data Center deployments, focused on highdensity 40G/100G I/O Modules Supervisor Engine Chassis Fabrics 14

13 Nexus 7000 / 7700 I/O Module Scalability For Your Reference Parameter Line Rate 10G/40G/100G Ports (18 slot) M2 Series (no XL license) Nexus 7000 Nexus 7000 / 7700 M2 Series (XL license) F2/F2e Series 384/96/32 384/96/ F3 Series 768/192/96 (N7000) 768/384/192 (N7700) L2 Table 128K 128K 16K/SoC 64K/SoC L3 IPv4 128K 900K 32K 64K L3 IPv6 64K 350K 16K 32K Adjacency Table 1M 1M 16K 64K ECMP (32 verified) Netflow Full/Sampled Full/Sampled Sampled Sampled ACL 64K 128K 16K/SoC 16K/SoC SPAN/ERSPAN VRF 4K 4K 4K 8K 15

14 Nexus 7700 F3 24-Port 40G Module Architecture EOBC To Fabric Modules To Central Arbiters FSA CPU 1G switch x 6 x 12 Fabric ASIC Fabric ASIC x 12 Arbitration Aggregator LC Inband to ARB to FSA CPU 2 X 40G SoC 1 2 X 40G SoC 2 2 X 40G SoC 3 2 X 40G SoC 4 2 X 40G SoC 5 2 X 40G SoC 6 2 X 40G SoC 7 2 X 40G SoC 8 2 X 40G SoC 9 2 X 40G SoC 10 2 X 40G SoC 11 2 X 40G SoC Front Panel Ports (QSFP+) 17

15 Fabric Services Accelerator (FSA) Scale boost for all F3 I/O modules High-performance module CPU with onboard acceleration engines 6 Gbps inband connectivity from SOCs to FSA Multi-Mpps packet processing 2GB dedicated DRAM FSA Complex Dual-Core LC CPU EOBC Packet Acceleration Engines Planned scale boost for distributed fabric services: BFD and sampled NetFlow (NX-OS 7.1 Q3 CY14) Other potential applications: distributed ARP processing, STP offload, etc. 2GB DRAM I/O 2GB DRAM 6 x 1Gbps Module Inband 18

16 Supervisor Engine 2 / 2E Next generation supervisors providing control plane and management functions Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700) Base performance High performance One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM Connects to fabric via 1G inband interface Interfaces with I/O modules via 1G switched EOBC Second-generation dedicated central arbiter ASIC Controls access to fabric bandwidth via dedicated arbitration path to I/O modules N77-SUP2E N7K-SUP2/N7K-SUP2E ID and Status LEDs USB Expansion Flash ID and Status LEDs Console Port Management Ethernet USB Host Ports USB Log Flash USB Expansion Flash Console Port Management Ethernet 19

17 Nexus 7000 Supervisor Hardware Scalability Parameter Sup 1 Sup 2 Sup 2E CPU Dual-Core Intel Xeon Quad-Core Intel Xeon 2 x Quad-Core Intel Xeon Speed 1.66 Ghz 2.13 GHz 2.13 GHz Memory 8G 12 GB 32 GB External Flash Memory 2 External Compact Flash memory slots: Log 8GB Expansion 2GB 2 External USB slots Log 8GB Expansion 2GB 2 External USB slots Log 8GB Expansion 2GB NX-OS Release 4.0 or later 6.1 or later 6.1 or later Kernel 32 bit kernel space 32 bit user space 64 bit kernel space 32 bit user space 64 bit kernel space 32 bit user space 20

18 Nexus 7000 Supervisor Scale Guidance Sup 1 and Sup 2 Will support the same scale No more scale increases after NX-OS 6.2 Sup 2E is the lead platform for increased scale How do we achieve higher scale? NX-OS software infrastructure enhancements Scalability testing CoPP profiles adjustment Features and best practices 21

19 NX-OS 6.2 Software Release NX-OS 6.2 Hardware Software FCS of world class new DC switching platform Nexus 7700, NAM & additional FEX Many software features and enhancements VPLS, Anycast HSRP, FP multi topology, etc. Scale Scale and Convergence Improvements 22

20 Nexus 7x00 NX-OS Software Scalability Check Verified Scalability Guide for latest information These are single dimensional numbers listed for guidance only Numbers are system wide and can be within a single VDC or spread across VDCs 23

21 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary 24

22 Basics of STP Scale Key Parameters Number of RPVST+ logical ports Number of STP virtual ports (physical ports * VLANs) VPC example with 4000 VLANs: 16,000 STP virtual ports 8,000 RPVST+ logical ports VLAN VPC 1 Peer -link switch# show spanning-tree internal info global b "STP Port STP Port Count Summary Total stp_ports*instances: 2 Total ports*vlans : 8000 Total phy_ports*vlans :

23 STP Scale Best Practices Parameter Previous With NX-OS 6.2 RPVST+ logical ports 16,000 16,000 STP virtual ports 90, ,000 Use MST with less instances for highest VPC and VLAN scale Verify combined resource usage from all VDCs against verified scale 26

24 NX-OS 6.2 VPC Scale Profiles Feature Profile A Profile B Profile C Profile D VPC VDC VLANs per VDC VLAN/VPC SVI/VDC RPVST + logical ports 16k n/a MST used n/a MST used STP virtual ports 150k 150k 150k 150k HSRP Groups Supervisor 1 2E 2E 2E Use-case focus Access High VLAN scale for core / aggregation High VPC and VLAN scale for aggregation n/a MST used Highest VPC scale for aggregation 27

25 VPC Convergence at Scale Based on Profile B Peer Link Down < 2 Sec Convergence All VPC Leg Failure < 1 Sec Convergence Switch Power Down < 2 Sec Convergence Peer Link Up < 4 Sec Convergence All VPC Up < 6 Sec Convergence Switch Power Up < 13 Sec Convergence 28

26 Scaled VPC Convergence Best Practices Peer-link failure Scaled setup with large ARP table & large route table L3 convergence may take time Orphan port suspend on L3 uplinks Suspend L3 uplinks on peer-link failure Bring up when VPC legs are up L3 L2 SVI int eth 1/1 vpc orphan-port suspend SVI 29

27 Scaled VPC Convergence Best Practices Peer-link restoration Delay restore timer (on by default at 30 sec) Delay VPC Leg and orphan port bring up Delay restore interface-vlan timer Delay SVI Bring up Lower than delay restore timer ARP/ND synchronization Load-defer on access switch port-channels Currently Nexus 7x00 feature only L3 L2 delay restore 360 sec delay restore interface-vlan 120 sec ip arp synchronize ipv6 nd synchronize SVI SVI int po1 port-channel load-defer 120 sec 30

28 Nexus x00 Fabric Extender Scale Parameter Before With 6.2 Number of Fabric Extenders / server ports 48 / / 3072 Number of VLANS per Fabric Extender Number of VLANs per Fabric Extender server interface Number of subinterfaces per Fabric Extender server interface For highest VLAN per HIF scale, connect FEX uplinks to one F-series SoC instead of spreading across multiple SoCs

29 Nexus 7x00 Physical Port vpc + FEX Q3 CY14 NX-OS 7.1 (for FEX, VPC+, scale and F3 support) N7K-1 N7K-2 N7K-1 N7K-2 N7K vpc domain N7K vpc domain FEX101 e101/1/1 Po1 VPC1 VPC1 Po1 FEX102 e102/1/1 FEX101 e101/1/1 VPC1 VPC1 FEX102 e102/1/1 Port-channel vpc interface e101/1/1 switchport vpc 1 lacp mode active Physical port vpc vpc configuration on a physical Layer 2 port as opposed to a port-channel Front panel ports and FEX ports connected to F2/F2e/F3 only Improves scaling as separate PC interface not created for single-link VPC leg Key benefit: more than 744 host facing VPCs with FEX 35

30 Dual-homed FEX and Scalability Logically single-homed server Q3 CY14 NX-OS 7.1 N7K vpc domain N7K vpc domain N2K N2K vpc vpc Dual-homed or FEX AA server server Shipping option for Nexus 5000 / 6000, future option for Nexus 7x00 No VPC+ or Enhanced VPC in first release on Nexus 7x00 Consider scale implications of less FEX per system and less VPC 36

31 Nexus 7x00 FabricPath Scale Considerations Parameter Previous With 6.2 FP core ports / IS-IS adjacencies VPC Edge ports Switch IDs Topologies 1 8 For maximum VPC+ scale: Configure no portchannel limit Use less topologies Feature Profile A Profile B VPC VDC 2 1 VLANs per VDC VLANs per VPC HSRP groups

32 FabricPath MAC Scaling Problem Typical FabricPath network All these MAC addresses will need to be learned at the Border Leafs F2/F2e SoC will not be able to fit more than 16K MACs F3 is an option with 64K MAC At L2/L3 boundary, we are learning all MACs for all VLANs VLAN CE 6k MACs L3 L2 FP VLAN k MACs SID:100 WAN FP FP FP FP SID:10 FP FP FP ESID:1001 CE CE CE VLAN k MACs FP SID:200 SID:20 SID:300 SID:400 SID:500 FP VLAN k MACs attach module x show hardware internal forwarding l2 table utilization instance all FP CE BORDER LEAF VLAN k MACs SPINE LEAF 39

33 FabricPath MAC Scaling Option 1: Split VLANs across VDCs Split Border Leaf into multiple VDCs, terminate only a subset of SVIs in each Border Leaf learns MACs on those VLANs, for which it is a Default Gateway Additional cabling between SPINE and Border LEAF nodes 6k MACs L3 L2 CE VLAN k MACs + 7k MACs SID:100 = VLAN k MACs 13k MACs ESID:1001 FP FP FP FP SID:10 SID:200 WAN VLAN k MACs SID:600 SID:20 SID:300 SID:400 SID:500 VLAN k MACs 3k MACs CE CE CE CE FP ESID:1002 9k MACs SID:700 + VLAN k MACs = 12k MACs 40

34 FabricPath MAC scaling Option 2: Proxy Layer 2 Learning using M modules From NX-OS release 6.2.2, can place F2e module with M-module in the same VDC F2e module will work in Layer-2 mode only, additional functionality uses larger MAC address tables of M-modules Only applicable to Nexus 7000 with F2e + M modules You can achieve up to 128k MACs No L2 CE ports on border leaf CE VLAN k MACs L3 L2 CE VLAN k MACs M2 SID:100 F2e SID:10 WAN ESID:1001 CE VLAN k MACs M2 SID:200 F2e SID:20 SID:300 SID:400 SID:500 CE VLAN k MACs CE VLAN k MACs 42

35 FabricPath Overload Bit Improving convergence in scaled setups NX-OS Software Support (Leaf / Spine) Nexus 7x / Nexus 5500/ (0)N1(1) / 7.0(0)N1(2) fabricpath domain X default spf-interval lsp-gen-interval s1 s2 s3 s4 FabricPath Update with Overload Updates Bit cleared set fabricpath domain default set-overload-bit { always on-startup [seconds]} Additional Spine switch is brought up, too many IS-IS updates result in backoff Potential traffic black-holing; tuning IS-IS timers was prior workaround Spine can send updates to leafs with Overload Bit set to avoid being transit Spine clears Overload bit after time interval and starts attracting traffic 43

36 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary 44

37 First Hop Redundancy Protocol Scalability Parameter Previous With NX-OS Groups per module 500 No Limit HSRP groups per system with MGO (2000 master groups) VRRPv3 groups per system N/A 4000 with Pathways (2000 leader groups) CoPP adjustment to strict profile to remove per module limit If using dense profile, increase CIR value: class copp-system-p-class-important police cir 1400 kbps, bc 1500 ms Utilize HSRP MGO / VRRPv3 Pathways to achieve highest group scale HSRP v2 and VRRP v3 are the most recommended protocols 46

38 HSRP Multiple Group Optimization Set of slave groups tied to master group Slave groups may optionally send hellos at slower interval for MAC refresh Only master group failure can result in failure of all slave groups Required beyond a verified number of HSRP groups Strongly recommended when scaling VPC aggregation layer Group 1 Group 1 Group 2 Group 2 Group 3 Group 3 Group 4 Group 4 Note: Not compatible with Anycast HSRP today 47

39 FHRP Scaling Guidelines and Best Practices Number of groups can be any mix of IPv4 and IPv6. Example: 4K single-stacked VLANs or 2K dual stacked VLANs Same FHRP group ID can be used in different VLANs but not required Maximum number of master / leader groups is equal to maximum number of groups without MGO / Pathways Use as little master / leader groups as possible with HSRP MGO / VRRP Pathways interface Vlan5 ip address /24 hsrp version 2 hsrp 5 name hsrp-mgo-master preempt priority 110 ip interface Vlan101 ip address /24 hsrp version 2 hsrp 6 follow hsrp-mgo-master ip

40 Anycast HSRP 4-way Gateway Redundancy for FabricPath Networks Nexus 7x00 - NX-OS 6.2 Nexus 5500/6000 NX-OS 7.0(1)N1(1); 6.0(2)N2(1) for interop mode L3 L2 SVI HSRP Active Anycast SID 100 SVI HSRP Standby SVI HSRP Listen SVI HSRP Listen Virtual switch hosting the HSRP virtual MAC address FabricPath Use IS-IS for initial election based on bundle priority Each spine advertises a path to a virtual switch hosting the HSRP virtual MAC Use HSRP hellos for leafs to map vmac to Anycast Switch ID All devices in the network need to either support Anycast HSRP or interoperate with new TLV 49

41 Anycast HSRP Scale Guidelines Parameter With NX-OS Routers in group 4 Groups 2000 Bundles 64 on Nexus on Nexus 7700 Groups per bundle 200 (config)# interface vlan 100 (config-if)# hsrp 1 (config-if-hsrp)# ip (config)# hsrp anycast 100 ipv4 (config-hsrp-anycast)# vlan (config-hsrp-anycast)# switch-id 100 (config-hsrp-anycast)# priority 120 (config-hsrp-anycast)# no shutdown Bundle priority Overrides groups priority Optional to determine switch responding to broadcasts Configure more groups in one bundle instead of using more bundles 50

42 vpc+ and Anycast HSRP Do we still need it? VPC+ is not required but what about ARP synchronization? Can implement VPC+ to achieve better N-S traffic convergence on uplink failure vmac for HSRP Default Gateway is still tied to Anycast Switch ID (ASID), not Emulated Switch ID (ESID) of vpc+ Anycast HSRP Bundle L3 L2 ARP sync WAN ARP sync Active* Standby* Listen* Listen* SID:10 SID:20 SID:30 SID:40 ASID: 100 ASID: 100 ASID: 100 ASID: 100 SID:50 SID:60 SID:70 SID:80 51

43 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary Use right features, best practices and hardware to get the most out of Nexus 7x00 52

44 F3-M2 Interoperability with Selective VRF Download Scaling F3 FIB F3 TCAM 64K 70 bit entries per SoC Typically Layer 3 information is synchronized across I/O modules and SoCs in a VDC MPLS VDC Type F3 M2 + F3 F2/F2E + F3 Table Sizes F3 size F3 size F2E size Selective VRF download programs routes only for VRFs configured on F3 SoC interfaces Enabled by default in M2-F3 VDC Maximum IPv4 routes = 64K x 6 SoCs = 384K SoC 0 SoC 1 VRF A 64K routes SoC 2 SoC 3 SoC 4 VRF B 64K routes SoC 5 53

45 Equal Cost Multipathing on Nexus 7x00 16-way ECMP most common 32-way ECMP support for Layer 3 protocols (OSPF, IS-IS, EIGRP and BGP) Wider Layer 3 fabrics using F2, F2e and F3 modules With F2/F2e only, 16 < paths available < 32 results in less even traffic distribution as some paths may be more heavily loaded FIB: N7K-1 Spine/Agg Layer N7K-2 Up to 32 Devices/Paths N7K-32 L2 or L3 Fabric Network X FIB: Network X Next-Hop N7K-1 N7K-2.. N7K-32 Leaf/Access Layer ToR 54

46 IGP Scalability Guidelines Protocol Parameter Before With OSPF Neighbors / LSAs 300 / 15, / 100,000 Passive Interfaces VRFs EIGRP Neighbors / routes 300 / 15, / 100,000 Passive Interfaces VRFs Static Routes Passive interfaces scale - support for 4K SVI at Layer 2/3 boundary Reduced scale and no HA with aggressive timers use default! Use passive interface default Increase GR timeout value for highest scale with EIGRP: timers nsf signal 60 55

47 NX-OS Routing Protocol Instance Scale 16 instances per VDC / system for IGPs* 1 instance for BGP per VDC / 4 instances verified per system To achieve highest IGP peer scale, consider alternatives: VRFs network segmentation VDCs administrative separation Use-cases for multiple instances Peering with devices not supporting IETF GR Segmentation without VRFs OSPF 1 OSPF 16 * - From NX-OS 6.1 for OSPF From NX-OS 6.2 for IS-IS and EIGRP 56

48 BGP and MPLS Layer 3 VPN Scale Considerations Parameter Before With NX-OS 6.2 BGP peers 1,000 2,000 VRFs / MPLS Layer 3 VPNs 1,000 4,000 MPLS L3VPN routes (per-vrf labels) 500K 700K MPLS L3VPN routes (per-prefix labels) 300K 500K Unique BGP attributes 512K 920K Per-VRF labels for highest route scale Simplest ways to reach maximum L3 VPN scale Static routes + ebgp PE-CE Direct routes MPLS 57

49 L2 L3 L3 / MPLS L3 / MPLS High BGP Peer Scale Use-cases PE-CE for high scale MPLS Layer 3 VPN environments Layer 3 fabric architecture with ibgp (RR at spines) Layer 3 fabric architecture with ebgp Peering with BGP running on bare metal server or VM SVI Y PE L3 / MPLS Core PE AS 100 RR 1 RR 2 ToR 1 ToR 2 ToR 3 58

50 MPLS Inter-AS Option B Lite RFC 4364 WAN AS 100 Inter-AS Option A available, but drives up BGP session scale Phase 1 of InterAS Option B support VRF A VRF B DC1 AS 200 ibgp IP v4/v6 VRF A VRF B VRF A VRF B DC1 AS 300 ibgp IP v4/v6 VRF A VRF B Target common DC multitenancy use-cases Only IGP / static / direct routes can be advertised to MP-eBGP Can use per-vrf ibgp or static for back-up routing FabricPath DC VPC DC 59

51 VDC Layer 3 Resource Allocation Achieving Highest Route Scale Certain resources can be allocated and limited to a given VDC vrf u4route-mem u6route-mem m4route-mem m6route-mem Set vrf resource limits Set ipv4 route memory limits Set ipv6 route memory limits Set ipv4 route memory limits Set ipv6 route memory limits Recommended to modify maximum and / or minimum allocation for each resource 1MB minimum / 350MB maximum configurable value for u4route-mem 1MB minimum / 100MB maximum configurable value for u6route-mem u4route-mem and u6route-mem limits are only applied after a switchover or VDC / system reload More details: BRKDCT Virtual Device Context (VDC) Design and Implementation with Nexus

52 Resource Allocation Example Memory for Routing 1. Estimate RIB table size in MB based on number of routes: show routing [ipv4 ipv6] memory estimate routes < > next-hops <1-16> 2. Modify maximum and minimum to same value: limit-resource u4route-mem minimum 300 maximum Configure lowest values for admin VDC or default VDC not being used for production Free memory for other VDCs to use 300MB IPv4 40MB IPv4 9MB IPv4 1MB IPv4 Total of 350MB Core Agg Access Admin 61

53 BGP Multipathing in NX-OS Data Center requirement: ECMP! BGP was not designed for load-balancing Chooses best path Can ibgp full mesh without RRs scale? New features find applicability in DC Components of multipathing Advertise / receive multiple paths Install multiple paths in URIB Program multiple paths in FIB / hardware 62

54 ibgp AddPath in NX-OS draft-walton-bgp-add-paths-06 Nexus 3000 Nexus 7x00 Nexus 5500/6000 (from NX-OS 7.0(0)N1(1)) New BGP attribute adds path identifier to prefix (NLRI); signal multiple paths All BGP edge routers and RRs must support functionality (send and receive) NX-OS implementation for AddPath add-all-path flavor RR sends all paths prefix L2 prefix L2,L3,L4 RR1 RR2 prefix/24, ubest/mbest: 16/0 *via L2%default, [200/0], bgp-100, internal *via L3%default, [200/0], bgp-100, internal *via L4%default, [200/0], bgp-100, internal BGP Addpath Best-path L1 L2 L3 L4 63

55 ibgp Multipathing with AddPath and RR on a stick Where do multiple paths come from? 2-way multipathing with VPC / VPC+ RR MPLS / Layer 3 Core 4-way multipathing with Anycast HSRP RR MPLS / Layer 3 Core S1 SVI x S2 SVI x Anycast Bundle S1 SVI x S2 SVI x S3 SVI x S4 SVI x /24, ubest/mbest: 16/0 *via S1%default, [200/0], bgp-100, internal *via S2%default, [200/0], bgp-100, internal /24, ubest/mbest: 16/0 *via S1%default, [200/0], bgp-100, internal *via S2%default, [200/0], bgp-100, internal *via S3%default, [200/0], bgp-100, internal *via S4%default, [200/0], bgp-100, internal 64

56 ebgp Multipath Relax Installing multiple ebgp paths Nexus 3000 / 5500 / 6000 / 7000 Routes must be identical in terms of LOCAL_PREF, AS_PATH, MED, etc Knob allows for AS-PATH to be different as long as number of AS s is same ebgp at Data Center Edge Prefix x ISP A AS 1 WAN ISP B AS 2 router bgp 10 neighbor remote-as 100 address-family ipv4 unicast disable-peer-as-check ebgp inside Data Center with disable-peer-as-check AS 10 AS 20 AS 3 router bgp 100 bestpath as-path multipath-relax neighbor remote-as 10 address-family ipv4 unicast allowas-in AS

57 Nexus 7000 with Internet Route Feed Example: How many feeds can we accept? Scenario Multiple Internet feeds, each with ~440K IPv4 routes Each feed with ~70K BGP attributes Unique AS-PATH per feed Constraints BGP memory limit for attributes stored = ~920,000 attributes switch# show ip bgp ipv4 unicast summary BGP table version is , IPv4 Unicast config peers network entries and paths using bytes of memory BGP attribute entries [920000/ ] U4 / U6 RIB maximum configurable memory = 350MB / 100MB switch# show routing ipv4 memory estimate routes next-hops 7 Shared memory estimates: Current max 350 MB; routes with 32 nhs in-use 349 MB; routes with 7 nhs (average) Configured max 350 MB; routes with 32 nhs Estimate memory with fixed overhead: 349 MB; routes with 7 nhs 67

58 BFD Scale Considerations Parameter Previous With NX-OS 6.2 Sessions per module with sub-interface optimize N/A 1000 Sessions per module with 50 msec x 3 interval Sessions per module with 300 msec x 3 interval Not Tested 500 Sessions per system Sub-interface optimization for even higher scale ISIS PIM BGP OSPF M2, F2/F2e or F3 I/O modules recommended to achieve highest scale Stati c MPL S BF D SUP BF D LC More details: BRKDCT-2333 Data Center Network Failure Detection 69

59 BFD Off-load Addressing higher session scale SUP-BFD - BFD process running on Supervisor Engine Interfaces with LC-BFD processes Interfaces with BFD clients LC-BFD BFD process running on CPU of each I/O module Communicates with SUP-BFD process Generates BFD hellos (echo and async) Receives BFD hellos from peer (async) Support for stateful restart, SSO and ISSU Additional offload to FSA accelerators on F3* Supervisor Engine OSPF HSRP PIM BGP IS-IS Etc. SUP-BFD EOBC LC-BFD Module Inband Hardware I/O Module LC-BFD Hardware I/O Module * - Future in NX-OS 7.1 LC-BFD Hardware I/O Module 70

60 BFD Sub-interface Optimization Scaling up number of sessions per module Nexus 7x00 NX-OS 6.2 Single session selected among all v4/v6 sessions to run at fastest interval Rest of sessions run at slow interval (default: 2 sec) Failure of a fast session signaled to all slow sessions Use-case: high scale multi-tenancy with more routing protocol sessions than BFD sessions supported on I/O module VRF red sub-int 1 VRF blue sub-int 2 VRF green sub-int 3 VRF yellow sub-int 4 fast slow slow slow sub-int 1 sub-int 2 sub-int 3 sub-int 4 71

61 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary Use right features, best practices and hardware to get the most out of Nexus 7x00 72

62 OTV Key Scale Parameters Improvements in NX-OS * 32k 4,000 6* k 2,000 Sites * two ED per Site OTV extended VLANs MAC addresses across all the extended VLANs Multicast Data Groups 73

63 OTV Convergence Improvements in NX-OS 6.2 Large Scale (1500 VLANs) Small Scale (512 VLANs) <30 sec <10 sec <10 sec <3 sec Areas improved: Slow detection of site peer chassis failure BFD for OTV IS-IS Blackhole period during AED change Master-slave AED election/convergence Delay in learning and advertisement after AED election Optimizations 74

64 OTV Designs for Fast Convergence STP-based or vpc based sites Core STP Root OTV VDC OTV VDC OTV VDC OTV VDC Aggregation Aggregation West-A Access West-B East-B Access VPC East-A STP Single layer VPC Lead architecture 75

65 VPLS Scale Calculation Pseudowires and VFIs for Multi-point connectivity SITE 1 SITE 2 VPLS A VFI 1 Active VPLS B VFI 1 Standby MPLS CORE VPLS A VFI 1 Active VPLS B VFI 1 Standby Parameter With NX-OS 6.2 VPLS pseudowires 2000 Virtual Forwarding Instances 1000 Maximum number of pseudo-wires = X One VFI maps to one bridge domain or VLAN Can I achieve maximum extended VLAN scale here? Maybe Dual-homing and more sites requires more pseudo-wires! Example: With 2 dual homed site solution, each VFI requires 2 pseudo-wires Effective Number of VFIs = X / 2! 76

66 DCI Guidelines for Scale and Convergence OTV Use one overlay Best convergence under 512 VLAN scale For fast site peer failure detection Enable BFD between SVIs for site VLAN Enable Recursive Next Hop detection Ensure that no static routes are pointing to site-adjacent box VPLS More dual homed sites reduces number of VLANs supported for DCI 77

67 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary Use right features, best practices and hardware to get the most out of Nexus 7x00 78

68 Nexus 7x00 Classification Hardware Resources Key Limits Resource M I/O modules (with XL license) M I/O modules (without XL license) F2/F2e I/O modules* F3 I/O Modules* TCAM entries 128K (4 * 32K banks) 64K (4 * 16K banks) 16K (4 * 4K banks) 16K (4 * 4K banks) LOUs 104 (208 registers) 104 (208 registers) 104 (208 registers) 104 (208 registers) M2# show hardware access-list resource utilization module 4 INSTANCE 0x0 ACL Hardware Resource Utilization (Mod 4) Used Free Percent Utilization Tcam 0, Bank Tcam 0, Bank Tcam 1, Bank Tcam 1, Bank LOU F-series values are per forwarding engine (SoC instance) F3# show hardware access-list resource utilization module 2 INSTANCE 0x0 ACL Hardware Resource Utilization (Mod 2) Used Free Percent Utilization Tcam 0, Bank Tcam 0, Bank Tcam 1, Bank Tcam 1, Bank LOU INSTANCE 0x1 79

69 Forwarding Engine TCAM Bank Architecture Feature-to-bank mapping Fixed assignment, applicable to all modules TCAM Bank Mapping feature allows for better feature combination: hardware access-list resource feature bank-mapping Ingress PACL Netflow WCCP OTV FEX LISP Ingress QoS Netflow LISP egress VACL Netflow RBACL WCCP OTV LISP egress Netflow (SVI) RBACL OTV LISP TCAM 0 Bank 0 TCAM 0 Bank 1 TCAM 1 Bank 0 TCAM 1 Bank 1 ingress RACL VACL PBR DHCP ARP Netflow Netflow (SVI) Netflow sampler (SVI) WCCP BFD ERSPAN ingress QoS NetFlow NetFlow (SVI) NetFlow sampler WCCP egress RACL VACL Netflow (SVI) Netflow Sampler (SVI) CTS WCCP egress QoS Netflow Netflow sampler WCCP 80

70 TCAM Resource Pooling Applicable to M, F2, F2e and F3 I/O Modules Resource pooling chains all TCAM banks in a SoC to support configurations larger than the size of a single TCAM bank Global config command applies to all SoCs per module: hardware access-list resource pooling module <list> Resource pooling restricts feature configurations Only features that can coexist in same TCAM bank can be applied in parallel (first come, first served) TCAM 0 Bank 0 TCAM 0 Bank 1 TCAM 1 Bank 0 TCAM 1 Bank 1 81

71 ACL Layer 4 Operators Scalability Considerations Two ways for software to handle L4 operators (range, gt, lt, neq) Expand the ACE into multiple TCAM entries (i.e use eq instead) Allocate L4op pointer and program LOU register Command controls when option 1 vs option 2 occurs for ACEs: hardware access-list lou resource threshold <value> Default: if an ACE can be expanded into <=5 TCAM entries, no L4op allocated and no LOU register is used Tradeoffs Expansion results in more TCAM entry consumption L4op/LOU usage limited by 10 L4ops per policy and 208 LOU registers LOU usage eq No LOU gt 1/2 LOU lt 1/2 LOU neq 1/2 LOU range 1 LOU 82

72 ACE Expansion Example ip access lou-test permit tcp any any range deny ip any any interface vlan 99 ip access-group lou-test in With LOU threshold at default (5): switch# sh hardware access-list input entries detail mod 8 eg routed.*range [17672] permit-routed tcp / /0 range [0] Single CL TCAM entry switch# sh hardware access-list input l4ops mod 8 eg -i -a 3 lou head line 4 1 LOU (2 registers) Lou usage: Lou sw_id l4op_bit ref_count Operation allocated 0(AB) dest-port: RANGE(1024, 49151) switch# sh hardware access-list resource utilization module 8 eg "Tcam 1, Bank 0 ^LOU Tcam 1, Bank LOU

73 ACE Expansion Example With LOU threshold increased to 6: switch# sh hardware access-list input entries detail mod 8 eg routed.*range [12289] permit-routed tcp / /0 range [0] [12297] permit-routed tcp / /0 range [0] [12578] permit-routed tcp / /0 range [0] [12546] permit-routed tcp / /0 range [0] [18050] permit-routed tcp / /0 range [0] [17922] permit-routed tcp / /0 range [0] 6 CL TCAM entries switch# sh hardware access-list resource utilization module 8 eg "Tcam 1, Bank 0 ^LOU Tcam 1, Bank LOU Decide what works best based on your ACLs! 84

74 ACL Scaling Best Practices Use ingress ACLs instead of egress Due to ingress forwarding, egress ACLs will get programmed everywhere Ingress ACL on SVI all SoCs get programmed Application of large ACL takes time Edit ACL instead of making a copy and reapplying again Use config session mode for large ACLs optimizations done upfront, quicker application Disable atomic ACLs: no hardware access-list update atomic 85

75 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary Use right features, best practices and hardware to get the most out of Nexus 7x00 86

76 Scale Management Solutions in multiple areas: Monitor Report syslog, SNMP Prevent user options to limit used resources Feature-based, OnePK, central monitor script Do you need to address every area for every feature? 87

77 Feature based scale monitoring and reporting Examples Fixed warnings for FIB usage 2014 Jan 8 13:03:25 Nexus7706 %IPFIB-SLOT2-4- FLN_FIB_TCAM_RESOURCE_WARNING: FIB TCAM usage is at 95 percent for IPV4 unicast on instance 0 Custom warnings based on configured VRF route limit vrf context test address-family ipv4 unicast maximum routes warning-only address-family ipv6 unicast maximum routes warning-only 88

78 VDC Resource Allocation Prevent from going above supported scale Some resources can be allocated and limited to a given VDC: anycast_bundleid Set anycast bundle id resource limits m4route-mem Set ipv4 route memory limits m6route-mem Set ipv6 route memory limits module-type Controls which type of modules are allowed in this vdc monitor-session Monitor local/erspan-source session monitor-session-erspan-dst Monitor erspan destination session monitor-session-extended Extended Monitor local/erspan-source session monitor-session-inband-src Monitor inband source monitor-session-mx-exception-src Monitor Mx module exception source port-channel Set port-channel limits u4route-mem Set ipv4 route memory limits u6route-mem Set ipv6 route memory limits vlan Set VLAN limits vrf Set vrf resource limits *Resources as of NX-OS 6.2(6) 89

79 Monitor Script Nexus 7x00 NX-OS 7.1 Q3 CY14 New system EEM policy uses a script to: Monitor all features in the system Syslog / SNMP trap if close to limit and at / over limit Disabled by default Uni-dimensional scale only Benefit: simple way to monitor all features across all VDCs 90

80 Scale Monitoring with OnePK Q3 CY14 NX-OS 7.1 Requirement to notify when scale limits are violated Number of ARP entries > verified Number of MAC table entries > verified Number of VLANs > verified Create script to monitor any feature(s) Use onepk APIs (or VTY service set) to get current feature scale status on a switch onepk application (C/Java/Python) Compare this information with the pre-defined values switch# If used show scale logging > defined, create syslog or take any other action 1024 Jan 7 23:46:17 switch onep: %ONEP-2-SCALEAGENT: ( Vlan scale exceeded! Max vlan recommended:, 2000, vlan on the system:, 2001) 91

81 Nexus 7x00 Scale Monitoring with OnePK Python Application Example for VLAN scale scale_limits = {"max_vlans" : 2000, "port_channels" : 300} def scalenotification(): for switch in switches: ip = switch[0] appname = switch[1] username = switch[2] password = switch[3] ne = NetworkElement(ip, appname) ne.connect(username,password) vty = VtyService(ne) vty.open() vlan_summary = vty.write("sh vlan summary") vty.close() vlan_sum = re.search('(?<=vlansum-all-vlan\t)(.*)',vlan_summary) if int(vlan_sum.group(0)) > int(scale_limits["max_vlans"]): string_print = "Vlan scale exceeded!. Max vlan recommended:", scale_limits["max_vlans"],"vlan being used :", vlan_sum.group(0) ne.create_syslog_message (ne.onepsyslogseverity.onep_syslog_critical, str(string_print)); 1. Pre-define scale limits 2. Connect to the switch 3. Process and compare against the predefined scale limits 4. Create syslog incase of scale limit violation 93

82 Agenda Introduction Layer 2 scale considerations Layer 2 / 3 boundary scale considerations Layer 3 scale considerations DCI scale considerations Security scale considerations Scale Management Summary 94

83 Scalability Best Practices Should you ask for more? Maybe, but recommend to look into other ways to achieve higher scale Master-slave mechanisms for protocols: HSRP Multiple Group Optimization (MGO) VRRP v3 Pathways BFD sub-interface optimization Other solutions: STP scale - use MST IGPs use VRFs as opposed to multiple processes Physical VPC instead of more port-channels MP-BGP instead of more BGP sessions 95

84 Summary Scalability improvements with latest NX-OS software and Nexus 7x00 hardware Features and best practices to achieve higher scale New options to manage scale How far will you take your Nexus 7x00? 96

85 Related Cisco Live Milan 2014 Events Technical Breakout Sessions Session-ID BRKARC-3470 BRKARC-2081 BRKARC-3144 BRKDCT-2048 BRKDCT-2121 BRKDCT-2333 BRKDCT-2051 BRKDCT-2237 Session Name Cisco Nexus 7000/7700 Switch Architecture Cisco FabricPath Technology and Design Troubleshooting Cisco Nexus 7000 Series Switches Deploying Virtual Port Channel in NX-OS Virtual Device Context (VDC) Design and Implementation Data Center Network Failure Detection Overlay Transport Virtualization Versatile architecture using Nexus 7000 with a mix of F and M modules to deliver FEX, FabricPath, MPLS, LISP and Multihop FCoE all at the same time 99

86 Call to Action Visit the World of Solutions:- Cisco Campus Walk-in Labs Technical Solutions Clinics Meet the Engineer Lunch Time Table Topics, held in the main Catering Hall Recommended Reading: For reading material and further resources for this session, please visit 100

87 Complete Your Online Session Evaluation Complete your online session evaluation Complete four session evaluations and the overall conference evaluation to receive your Cisco Live T-shirt 101

88

89 Proxy Layer 2 Learning Configuration 1. Disable remote-mac learning on all Border Leaf nodes no mac address-table fabricpath remote-learning L3 L2 M2 SID:100 F2e WAN ESID:1001 M2 SID:200 F2e 2. Disable FabricPath core port MAC learning on all SOCs with core ports connected no hardware fabricpath mac-learning module <x> [port-group <x>] If you are using F2 leaf switches 3. Disable FabricPath core port MAC learning on all F2 SOCs with core ports connected no hardware fabricpath mac-learning module <x> [port-group <x>] FP FP FP FP SID:10 SID:20 FP FP FP FP FP FP SID:300 SID:400 SID:500 CE CE CE CE CE 4. Prune allowed VLAN lists on F2 CE edge ports switchport trunk allowed vlan <vlans> VLAN VLAN VLAN VLAN VLAN k MACs 5k MACs 7k MACs 3k MACs 4k MACs 103

90 DHCP Scale Considerations Snooping and Relay Parameter Previous With L3 links northbound v4 Snooping Bindings v4 Snooping VLANs , Access L2 links southbound v4 Relay Clients ,000 v6 Relay Clients N/A 10,000 Relay Agents Relay DHCP query Listen to DHCP reply (DHCP request) My MAC address is 11:22:33:44:55:66. What s my IP address? (DHCP request) Who is my default gateway? (ARP request) 4K dual stacked VLANs for DHCP relay! Use Option 82 to support highest snooping scale: ip dhcp snooping information option 104

91 ARP / ND Scale Considerations Parameter Previous With ARP packets / sec ARP glean packets / sec IPv6 ND packets / sec IPv6 ND glean packets / sec ARP table 128K 128K Key takeaway: faster convergence for aggregation / leaf upon bring-up of virtualized server row 105

92 Cisco TrustSec Scalability Guidelines NX-OS 6.2(6) has enhanced SGT functionality to support deployments with up to 200K IP SGT mappings Optimized method for processing IP-SGT mappings and downloading policy Optimized method to process IP-SGT mappings which is recommended with > 50K IP-SGT mappings: cts role-based batched programming 106

93