Towards Massively Scalable Ethernet: Technologies and Standards

Transcription

1

2 Towards Massively Scalable Ethernet: Technologies and Standards Ali Sajassi Distinguished Engineer, Cisco BRKSPG-2206

3 Agenda Introduction Addressing Aspects Optimal Forwarding VXLAN & EVPN-VXLAN TRILL & FabricPath SPB Multi-Pathing Interconnecting Ethernet Domains VPLS, H-VPLS & PBB-VPLS EVPN Summary

4 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 Scaling Layer-2 Networks to Millions of End-Points? Addressing Scalability, Mobility, Lookup Optimal Forwarding Routing vs. Bridging, Full Network Bandwidth use Efficient Interconnection of Ethernet Networks BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Addressing Aspects How to Avoid Keeping a Host-Route for Every Host on Every Network Element While Maintaining Mobility and Ease of Use?

7 Addressing Aspects Requirement Efficient Addressing Host Mobility Reduce/Avoid Unicast Flooding Reduce/Avoid Broadcasts Approaches Hierarchical addressing schemes Location-dependent addressing Control Plane Learning (where feasible) Location-independent addressing Control Plane Learning (where feasible) Broadcast offload using proxies/servers BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Addresses Location and Identity Identity addresses ( who ) MAC- addresses typically represent an Identity The manufacturer s MAC address issued to a physical Ethernet interface is a who address; it identifies a station regardless of what network, or where in that network, the station is attached. Virtual Machines (VM) typically re-use the server-assigned MAC addresses Location addresses ( where ) IP-addresses typically represent a Location (but also carry Identity) Some approaches (e.g. FabricPath, PBB-EVPN) assign to a device a MAC address (or IP-Address) that carries some geographical information. Typically locally administered MAC addresses * 46 bits can carry a network ID, a subnetwork ID, a switch ID, a port ID, and/or a host ID on that port. Switches can use mask-and-match lookups instead of 48-bit host-route lookups to forward L2 frames. * Locally administered MAC addresses: low-order two bits of the first byte are 10 ; globally administered manufacturers addresses: those bits are 00. BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 MAC addresses: Identity vs. Location Identity addresses Switches need to learn destination hosts MAC addresses If a host moves only the switches (rather than hosts) need to update their forwarding tables Location addresses Reduce the size of the (L2) forwarding table Hosts change addresses when they move: Requires notification of every host. Approaches to combine the two worlds (i.e. namespaces) Map n Encap : [VXLAN], [FabricPath], [TRILL], [EVPN], [OTV], [LISP], [8021Qbp] Translate: [PortLand], [MOOSE] BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 Identity and Location Addresses: Map n Encap Approaches Identity Addresses: Kept to the Edge Endpoint Identifier (EIR) [LISP] MAC-Address [OTV], [TRILL], [EVPN], [VXLAN] Location Addresses: Topologically aggregate-able; Can change while Identity stays fixed Routing Locator (RLOC) [LISP] Virtual Tunnel End Point Address [VxLAN] Overlay Interface Address [OTV] Rbridge-ID [TRILL] B-MAC [PBB-EVPN], [8021Qbp] Mapping Service Map Identity to Location Addresses (distributed e.g. routing protocol based, or centralized/server-based - think DNS, or through data-plane learning) Location Addresses Identity Addresses Location Address to Identity Address Mapping Service BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Combining location and identity Example: EVPN Network-based, L2-over-MPLS, Distributed (BGP based) Control Plane Learning MPLS labels PE performs EVPN MAC1 Payload MAC2 PE performs EVPN MAC1 Payload MAC2 PE1 B-MAC1 B-MAC2 PE2 MAC1 Payload MAC2 Mapping Service Mapping Service Host: MAC1 Host MAC1 Location IP-PE1 Host MAC1 Location IP-PE1 Host: MAC2 MAC2 IP-PE2 MAC2 IP-PE2 Learned table for attachment circuit MAC1 Int Eth 1 BGP MAC2 Int Eth 2 Learned table for attachment circuit BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Combining location and identity Example: OTV Network-based, L2-in-L3, Distributed (IS-IS based) Control Plane Learning MAC1 MAC2 Payload MAC1 Payload MAC2 Ingress Edge Device Egress Edge Device MAC1 Payload MAC2 Mapping Service Mapping Service Host: MAC1 Host MAC1 Location Int Eth 1 Host MAC1 Location Host: MAC2 MAC MAC2 Int Eth 2 ISIS 1. Layer 2 lookup on the destination MAC. MAC 2 is reachable through IP The Edge Device encapsulates the frame. 3. The transport delivers the packet to the Edge Device on the other site. 4. The Edge Device receives and de-capsulates the packet. 5. Layer 2 lookup on the original frame. MAC 2 is a local MAC. 6. The frame is delivered to the destination. BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Combining location and identity Example: VXLAN Network or host-based, L2-in-L3, C-MAC Data Plane Learning MAC1 MAC2 Payload MAC1 Payload MAC2 Ingress VTEP Egress VTEP MAC1 Payload MAC2 Mapping Service Mapping Service Host: MAC1 Host MAC1 Location Int Eth 1 Host MAC1 Location Host: MAC2 MAC MAC2 Int Eth 2 1. Layer 2 lookup on the destination MAC. MAC 2 is reachable through VTEP Ingress VTEP encapsulates the frame. 3. The transport delivers the packet to the egress VTEP on the other site. 4. Egress VTEP receives and decapsulates the packet. 5. Layer 2 lookup on the original frame. MAC 2 is a local MAC. 6. The frame is delivered to the destination. BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Combining location and identity Example: L2-LISP Network-based, L2-in-L3, Distributed Control Plane Learning (MR/MS/ALT) MAC1 MAC2 Payload MAC1 Payload MAC2 Ingress Tunnel Router (ITR) Map Cache Egress Tunnel Router (ETR) MAC1 Payload MAC End-System ID (EID: MAC1) Map Resolver (MR) Alternate Topology (ALT) Map Server (MS) End-System ID (EID): MAC2 Mapping Service EID-Prefix MAC1 MAC2 Locator(s) See: BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 A Note on MAC learning If you believe you need MAC learning, consider learning only if you have to All of the approaches (e.g. [FabricPath], [TRILL],[802.1Qbp],..) don t do C-MAC learning in the core Several deployments still perform C-MAC learning (from core-side) in data plane at the edge switch Conversational Learning Each forwarding engine distinguishes between two types of MAC entry: Local MAC MAC of host directly connected to forwarding engine Remote MAC MAC of host connected to another forwarding engine or switch Forwarding engine learns remote MAC only if bidirectional conversation occurring between local and remote MAC MAC learning not triggered by flood frames BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Conversational Learning Learning only the MAC addresses required xxx # of MACs xxx # of MACs MAC IF L2 Fabric MAC IF B 2/1 S11 S12 B STP Domain MAC IF A 2/1 C S12 MAC IF C 3/1 A S11 xxx # of MACs xxx # of MACs A C ALL MACs needs to be learned on EVERY Switch Large L2 domain and virtualization present challenges to MAC Table scalability Local MAC: Source-MAC Learning only happen to traffic received on CE Ports Remote MAC: Source-MAC for traffic received on Core-facing ports are only learned if Destination-MAC is already known as Local Example: FabricPath Implementation BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Controlling Broadcast Traffic Servers/Caches for Applications using Broadcast ARP suppression mechanism e.g., EVPN mechanism Flooding disablement of unknown unicast e.g., EVPN Example: ARP/ND Edge devices (e.g. with EVPN or OTV) maintain an ARP cache, which is populated by snooping ARP replies. Initial ARP requests are broadcasted to all sites, but subsequent ARP requests are suppressed at the Edge Device and answered locally. 4 1 Subsequent ARP requests (IP A) First ARP reques t (IP A) 5 ARP reply on behalf of remote server (IP A) 3 ARP Cache MAC 1 Cache Snoop & cache ARP reply Cache/ Server IP A 2 ARP reply MAC 2 IP B BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Optimal Forwarding How to Leverage the Entire Network Topology for Packet Forwarding and Approach Full Cross-Sectional Bandwidth?

19 Towards a new Layer-2 Control Protocol Why? Current Spanning Tree Non Optimal Forwarding Paths (see example) Parallel Paths cannot be leveraged Operational challenges in complex topologies Let s discuss 3 approaches Root IETF VXLAN / EVPN-VXLAN 2 IETF TRILL / Cisco FabricPath IEEE Shortest Path Bridging 802.1aq BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Optimal Forwarding IETF VXLAN and EVPN-VXLAN

21 VXLAN IETF Approach to Shortest Path Bridging over IP Virtual Extensible Local Area Network (VXLAN) RFC 7348 Main areas addressed by VXLAN: Shortest Path Forwarding with ECMP Overcome 4094 VLAN limitation MAC Table Size Scalability BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 VXLAN Basics VXLAN Tunnel End Point (VTEP) End Systems Performs VXLAN encapsulation/decapsulation Maps VLAN to VNID & back. Data-plane learning of MAC to remote VTEP IP association Local LAN Segment IP Interface VTEP Segment (VNID) mapped to IP multicast group in core. Multicast Group VTEPs join group using IGMP. Broadcast / Unknown Unicast / Multicast traffic sent to IP multicast group for VNID. VTEP IP Interface Local LAN Segment IP Network VTEP IP Interface Local LAN Segment Core network forwarding is regular IP End Systems End Systems BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 VXLAN Ethernet Data Encapsulation MAC-in-UDP Encapsulation to tunnel Ethernet over IP core network. 24-bit VXLAN Network Identifier (VNID) for scalable segmentation. Default behavior is to map the payload Ethernet frame VLAN to VNID on ingress, strip the VLAN and reconstruct it on egress. Optionally, VLAN may be left in the frame but is not used by VXLAN forwarding Source UDP Port used for entropy in ECMP Set to hash of payload Ethernet headers Src. VTEP Dst. VTEP Used for entropy (hash of payload) Set to 4789 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Packet Flow Multicast/Broadcast/Unknown Unicast 21 S-MAC: MAC-1 D-MAC: A.CC.02 Outer S-IP: IP-1 Outer D-IP: UDP VXLAN VNID: 10 ARP Request for IP B Src MAC: MAC-A DST MAC: FF.FF.FF.FF.FF.FF End System C MAC-C IP-C IP-1 MAC-1 VTEP-1 VTEP IP-3 MAC-3 Multicast Group IP Network 61 S-MAC: MAC-2 D-MAC: 43.1C.02.AB Outer S-IP: IP-2 Outer D-IP: IP-1 UDP VXLAN VNID: 10 ARP Response from IP-B Src MAC: MAC-B DST MAC: MAC-A IP-2 MAC-2 VTEP-2 MAC Address 61 VNID MAC-A 10 IP-1 Remote VTEP IP 31 1 ARP Request for IP B Src MAC: MAC-A DST MAC: FF.FF.FF.FF.FF.FF MAC Addres s VNID Remote VTEP IP 71 ARP Request for IP B Src MAC: MAC-A DST MAC: FF.FF.FF.FF.FF.FF ARP Response from IP-B Src MAC: MAC-B DST MAC: MAC-A End System A MAC-A IP-A MAC-B 10 IP-2 End System B MAC-B IP-B ARP Response from IP-B Src MAC: MAC-B DST MAC: MAC-A 51 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 VXLAN Scorecard Advantages Shortest path forwarding with ECMP Optimal multicast with PIM 16 Million segments (no more 4094 VLAN limitation) No MAC learning in the core (forward on VTEP IP addresses) Open Issues Unknown unicast flooding Fabric must support IP multicast (RFC 7348) Alternative is Manual Configuration of ingress replication flood lists (provisioning intensive). All Active Multi-chassis Redundancy BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 EVPN-VXLAN Let s add a (BGP) Control Plane End System MAC (& IP) address binding to VTEP IP advertised in EVPN MAC/IP Advertisement route. Eliminate need to flood unknown unicast. BGP scalability techniques apply: RR Hierarchy Constrained route distribution (RT per VNID) Support for End System Mobility (MAC Mobility BGP Attribute) 1 ARP Request for IP B Src MAC: MAC-A DST MAC: FF.FF.FF.FF.FF.FF VTEP + EVPN Edge Node End System C MAC-C IP-C EVPN MAC/IP Advertisement Route IP-1 NVE-1 End System A MAC-A IP-A IP-3 See: draft-ietf-bess-evpn-overlay 21 MAC Address NVE-3 L2 RIB Multicast Group VNID L2 RIB MAC Address IP Network Next- Hop MAC-A 10 Local VNID 31 Next- Hop MAC-A 10 IP-1 IP-2 NVE-2 End System B MAC-B IP-B L2 RIB MAC Address VNID BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public Next- Hop MAC-A 10 IP-1

27 EVPN-VXLAN Automatic VTEP Discovery Network Virtualization Edge Nodes (NVEs) exchange EVPN Inclusive Multicast Ethernet Tag routes. Enables automatic discovery of remote VTEPs Advertise multicast tunnel mode for broadcast/multicast traffic PIM-SSM Tree PIM-SM Tree BIDIR-PIM Tree Ingress Replication No longer required to provision remote VTEP IPs End System C MAC-C IP-C EVPN Inclusive Multicast Ethernet Tag Route IP-1 NVE-1 End System A MAC-A IP-A IP-3 VNID NVE-3 Remote VTEP 10 IP-2 10 IP-3 VNID IP Network Remote VTEP 10 IP-1 10 IP-2 EVPN Inclusive Multicast Ethernet Tag Route IP-2 NVE-2 End System B MAC-B IP-B EVPN Inclusive Multicast Ethernet Tag Route VNID Remote VTEP 10 IP-1 10 IP-3 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 EVPN-VXLAN Remote NVE Shared ESI All-Active Multi-Chassis Redundancy NVE-3 IP-3 IP-1 IP-2 None None NVEs discover multi-homed Ethernet Segments by exchanging EVPN Ethernet Segment routes. Multi-homed Segments can operate in Active/Standby or All-Active Mode. Remote NVE IP-2 IP-3 Shared ESI ESI1, ESI2 None IP-1 NVE-1 IP Network Ethernet Segment Routes C ESI1 NVE-2 IP-2 Remote NVE IP-1 IP-3 Shared ESI ESI1, ESI2 None A ESI2 D B BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 EVPN-VXLAN All-Active Multi-Chassis Redundancy Loop Prevention of Flooded Traffic Build broadcast/multicast flood list per source NVE NVE-3 IP-3 Flood based on Local Bias IP Network Source NVE Flood List for VNID 1 IP-1 Port D Source NVE IP-2 Flood List for VNID 1 Port A IP-1 NVE-1 NVE-2 IP-3 IP-2 Port B, Port C, Port D IP-3 Port A, Port B, Port C ESI1 C A ESI2 B D BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 EVPN-VXLAN All-Active Multi-Chassis Redundancy Prevent Duplicate Frames Designated Forwarder per ESI NVE-3 IP-3 IP Network Source NVE IP-1 Flood List for VNID 1 Port D Source NVE IP-2 IP-3 Flood List for VNID 1 Port A Port A, Port B, Port C IP-1 NVE-1 NVE-2 IP-2 IP-3 Port B, Port C, Port D NVE2 is DF for ESI2 NVE1 is DF for ESI1 C ESI1 A ESI2 D B BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 EVPN Integrated Routing and Bridging (IRB) EVPN enables intra-subnet bridging and inter-subnet routing using the same control plane. EVPN MAC/IP Advertisement Routes carry both MAC & IP address of end stations EVPN Default Gateway Extended Community allows advertisement/aliasing of default gateway MAC/IP between NVEs. IP Network IP Network NVE 1 Default Gateway MAC: M1 IP: IP1 Aliases M2 IP2 MAC/IP Advert Route w/ Default Gateway EC NVE2 Default Gateway MAC: M2 IP: IP2 Aliases M1 IP1 NVE 1 Default Gateway MAC: M1 IP: IP1 NVE2 Default Gateway MAC: M1 IP: IP1 Dedicated Default Gateway MAC/IP Per NVE Single Anycast Default Gateway MAC/IP for All NVEs See: draft-ietf-bess-evpn-inter-subnet-forwarding BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 EVPN IRB Symmetric Forwarding MAC VRF & IP VRF lookup performed on both ingress and egress NVEs S-MAC: M1 D-MAC: A.CC.02 Outer S-IP: IP1 Outer D-IP: IP2 UDP VXLAN VNID: 10 Dst MAC: M2 Src MAC:: M1 Dst IP: Src IP: IP Network Dst MAC: MAC2 Src MAC:: M2 Dst IP: Src IP: Dst IP: Src IP: NVE 1 Default Gateway IP VRF MAC: M1 IP: IP1 MAC VRF NVE2 IP VRF MAC VRF Default Gateway MAC: M2 IP: IP2 Dst MAC: M1 Src MAC:: MAC1 Dst IP: Src IP: /24 MAC /24 MAC2 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 EVPN IRB Asymmetric Forwarding MAC VRF & IP VRF lookup on ingress NVE & only MAC VRF lookup on egress NVEs S-MAC: M1 D-MAC: A.CC.02 Outer S-IP: IP1 Outer D-IP: IP2 UDP VXLAN VNID: 10 Dst MAC: MAC2 Src MAC:: M1 Dst IP: Src IP: IP Network Dst IP: Src IP: NVE 1 Default Gateway IP VRF MAC: M1 IP: IP1 MAC VRF NVE2 IP VRF MAC VRF Default Gateway MAC: M2 IP: IP2 Dst MAC: M1 Src MAC:: MAC1 Dst IP: Src IP: /24 MAC /24 MAC2 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Optimal Forwarding IETF TRILL and Cisco FabricPath

35 TRILL IETF Approach to Shortest Path Bridging TRILL (TRansparent Interconnect of Lots of Links) Main areas addressed by TRILL: Provide Shortest Path and Equal Cost Multi-Pathing for traffic Be Plug-n-Play BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 TRILL Basics IEEE Bridge RBridge TRILL Network is comprised of Routing Bridges (RBridges). Rbridge uniquely identified by a Nickname or RBridge-id (auto-created from ISIS system id) RBridges can be connected by LANs or RBridges can be connected by simple P2P links (incl. PPP see RFC 6361) Architecturally, RBridges run on top of an bridged network similar to Routers RBridges may be interconnected by classical 802.1Q bridges: Allows for gradual migration of existing networks RBridges do not participate in xstp, and drop BPDUs if they are received BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 TRILL Principles of Operation.1Q frame.1q frame.1q frame Frames are encapsulated with RBridge nicknames and further encapsulated with originating rbridge and next hop rbridge MAC address Header fields differ from 802.1ah A Outer C MAC Headers are swapped hop by hop (similar to routing) A E RBridge Header.1Q frame A E C D.1Q frame A E D E IEEE Bridge RBridge RBridges learn what MAC addresses are on their edge ports using general dataplane learning and MAY advertise them other RBridges Remote mac-address-to-rbridge binding learning: hardware or control plane Unknown unicast /multicast/broadcast frames flooded along pre-calculated distribution tree(s) BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 TRILL Forwarding RBridges use IS-IS for discovery and to synchronize Link State Databases TRILL uses these Link State Database to Compute pair wise bidirectional paths for unicast (per node and/or per VLAN) between all Rbridges For multicast, distribution trees are calculated rooted at (potentially) every rbridge; trees are given an rbridge-id/nickname as well TRILL adds to standard IS-IS Ships in the night with other protocols using ISIS TRILL Hellos Find out whether nodes are on a LAN or P2P link Designated Rbridge (DRB) Election Root-Bridge-IDs See also: RFC 6165 (Extensions to IS-IS for Layer-2 Systems) BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 TRILL Ethernet Data Encapsulation Outer Ethernet Header (link specific): Outer Destination MAC Address (RB2) Outer Destination MAC Address Outer Source MAC Address Outer Source MAC Address (RB1) Ethertype = IEEE 802.1Q Outer.VLAN Tag Information TRILL Header: Ethertype = TRILL V R M Op-Length Hop Count Egress (RB2) Nickname Ingress (RB1) Nickname Inner Ethernet Header: Outer-VLAN Tag Information: This is used only if two RBridges communicate across a standard 802.1Q network V: Version M: Multi-destination; indicates if the frame is to be delivered to a single or multiple end stations Opt-Length: >0 if an Option field is present Hop Count: Similar to TTL RBridge Nickname: Not the MAC address of the Rbridge, but the a TRILL ID for the RBridge (Egress Nickname used differently if M = 1) Inner Destination MAC Address Inner Destination MAC Address Inner Source MAC Address Inner Source MAC Address Multicast tree pruning: Requires inspection of customer Destination MAC Address and customer VLAN Ethertype = IEEE 0x8100 Ethertype = IEEE 0x893B Inner.VLAN Tag Information Inner.VLAN second part In case of Fine Grain Labeling: Second VLAN tag (see RFC 7172) See also: RFC 6325 and RFC 6327 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 Packet Flow Known Unicast M1 Perform MAC lookup on M2 to determine Egress RBridge RB3 Encapsulate in TRILL header & forward to next-hop RBridge RB1 MR1 RB8 MR8 RB 9 B1 B Q Cloud B2 Perform Egress RBridge nickname lookup on RB3 to determine next hop RBridge RB6 MR6 RB 7 TRILL Network RB5 MR5 B4 B Q Cloud B5 B7 Decapsulate TRILL header Perform MAC lookup on M2 to determine egress port RB 2 RB 4 RB3 MR3 M2 MR8 MR1 RB3 M2 M1 Outer MAC DA Outer MAC DA Outer MAC SA Outer MAC SA Etype = 802.1Q Outer VLAN Etype = TRILL V/M/R, TTL Egress RB-ID Ingress RB- Inner MAC DA ID Inner MAC DA Etype = 802.1Q Inner MAC SA Inner MAC SA Inner VLAN RB1 Changes Hop-to-Hop (MACs, VLAN, TTL) Unchanged From Ingress to Egress Outer MAC DA Outer MAC DA Outer MAC SA Outer MAC SA Etype = 802.1Q Outer VLAN Etype = TRILL V/M/R, TTL Egress RB-ID Ingress RB- Inner MAC DA ID Inner MAC DA Etype = 802.1Q Inner MAC SA Inner MAC SA Inner VLAN MR3 MR5 RB1 Payload. Payload. BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 Packet Flow Multicast/Broadcast/Unknown Unicast M1 Perform MAC lookup on G1 Encapsulate in TRILL header, set M bit and tree id (egress rbridge id) & forward to all- RB s mcast address RB9 MR9 RB1 MR1 RB8 MR Q Cloud Perform lookup on egress rbridge-id to determine distribution tree RB 6 RB 7 TRILL Network RB5 MR5 Note: All-RB=All-Rbridges = c Q Cloud Decapsulate TRILL header Perform MAC lookup on G1 to determine egress ports RB 2 RB 4 G1 G1 RB3 MR3 G1 All-RB- MCAST (or MR9) MR1 RB9 G1 M1 Outer MAC DA Outer MAC Outer MAC DA Outer MAC SA SA Etype = 802.1Q Outer VLAN Etype = TRILL V/M/R, TTL Egress RB-ID Ingress RB- Inner MAC DA ID Inner MAC DA Etype = 802.1Q Inner MAC SA Inner MAC SA Inner VLAN M=1 RB1 Changes Hop-to-Hop (MACs, VLAN, TTL) Unchanged From Ingress to Egress Outer MAC DA Outer MAC Outer MAC DA Outer MAC SA SA Etype = 802.1Q Outer VLAN Etype = TRILL V/M/R, TTL Egress RB-ID Ingress RB- Inner MAC DA ID Inner MAC DA Etype = 802.1Q Inner MAC SA Inner MAC SA Inner VLAN All-RB-MCAST MR5 M=1 RB1 Payload. Payload. BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 TRILL Benefits Shortest path delivery of unicast Layer 2 multi-pathing (ECMP) of unicast Optimal multicast delivery over shared trees Load-balancing over multiple trees. Per-VLAN/c-group pruning of trees via IGMP/PIM snooping. Fast convergence times, Minimal configuration Support for Shared Media and P2P links Loop Prevention and Mitigation (adds a TTL) Support for multi-homing (DRB election) Confines MAC Address learning to edge nodes, providing MAC address scalability similar to IEEE 802.1ah (MAC-in-MAC) BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Cisco FabricPath in a Nutshell Similarities with TRILL MAC-in-MAC -like encapsulation, includes TTL ISIS based control plane (unicast and multicast) No MAC learning in the Fabric, Forwarding based on Switch IDs ECMP for Multi-Path Load Balancing FarbicPath Additions Ethernet Conversational Learning at the Edges Interworking with STP-based Ethernet Access Domains All-Active Access Redundancy Multiple-Topologies Ethernet FabricPath Header FabricPath BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 FabricPath Conversational Learning S1 S2 S3 S4 L1 L5 L2 L3 L4 L6 L7 L8 L10 L11 L12 HIT LEARN MAC IF A 1/1 C S42 Decap L2 Fabric S11 S12 L9 S42 1/1 3/1 MISS Encap Decap Don t LEARN A B C MISS MAC IF C 3/1 FabricPath Port CE Port BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Multiple Topologies Topology: A group of links in the Fabric. By default, all the links are part of topology 0. Other topologies can be created by assigning a subset of the links to them. A link can belong to several topologies A VLAN is mapped to a unique topology L1 L5 L2 L3 L4 L6 L7 L8 L10 L11 L12 L2 Fabric L9 Topologies can be used for traffic engineering, security, etc. Topology 0 Topology 1 Topology 2 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Optimal Forwarding IEEE Shortest Path Bridging

47 802.1aq Shortest Path Bridging Motivation Traditional bridging based on RSTP/MSTP Non-optimal forwarding Manual configuration needed for disjoint trees and mapping of VLANs to these trees Approach: 802.1aq Shortest Path Bridging Optimal unicast and multicast forwarding Automatic Shortest Path Tree (SPT) management controlled by IS-IS Reuse existing 802.1ag/Y.1731 Ethernet OAM Same motivations as TRILL BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 802.1aq Shortest Path Tree per Bridge Original Concept Each bridge is the root of a separate shortest path tree instance Bridge G is the root of the green tree Bridge E is the root of the blue tree Both trees are active AND symmetric at all times Needed in Ethernet to have congruent multicast and unicast Root Root B G A Root C Root F Root D Root E Root Root B G A C F D E Blocked Ports Blocked Ports B G A C F D Root E BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 IEEE 802.1aq Variants Shortest Path Bridging MAC (SPBM) targets 802.1ah (PBB) networks where all addresses are managed Shortest Path Bridging VID (SPBV) is applicable in 802.1Q and 802.1ad (Q-in-Q) customer, enterprise or storage area networks SPB SPBV SPBM Enterprise Network Plug & Play Easy to operate Unknown addresses Access Network Reliability Bandwidth efficiency Unknown or managed addresses Metro Core Network Reliability Auto-discovery Load sharing Managed addresses MAC learning in data plane MAC learning in control plane BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 P802.1aq Shortest Path Bridging Shortest Path Bridging MAC Mode (SPBM) 802.1ah source B-MAC+B-VID for tree identification (assumes 802.1ah) Data-plane learning of C-MACs on BEB per 802.1ah IS-IS SPT (unicast & mcast) & CIST calculation/maintenance Control Plane Learning of B-MAC Service Discovery (I-SID registration) BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 Path Congruency and Symmetry Necessary if MAC learning is in the data plane Not necessary if MAC learning is in the control plane For both: SPBV and SPBB 6 unicast multicast Necessary for the proper operation of 802.1ag E-OAM and beneficial for clock distribution 6 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Loop Prevention and Mitigation Problem: Inconsistent view on network topology at different nodes may cause transient loops in case of a link-state control protocol Loop Prevention Agreement Protocol (AP) (see 802.1aq-2012, clause 13.17) Handshake mechanism between neighbors Extension to MSTP handshake Loop Mitigation Ingress Checking (Reverse Path Forwarding Check) Frames not arriving on the shortest path from the Source Bridge are discarded: No TTL needed Makes the tree directed: Good for loop prevention in most cases Transient loops may appear Severe problem for multicast/broadcast traffic A chance of network melt-down remains if one does not care Ingress filtering has to be modified BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Optimal Forwarding Multi-Path Forwarding

54 Multi-Path Load Balancing Approaches Classical ECMP [EVPN], [VXLAN], [OTV], [TRILL] Equal Cost Trees [802.1aq] Ethernet ECMP [802.1Qbp] BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 Classic Equal Cost Multi Path (ECMP) Pre-Requisite Link-Layer Routing Protocol which can compute two or more equal cost shortest paths between two nodes ECMP distributes the traffic per hop among the equal-cost paths Packet-based (in round-robin fashion): Can cause out-of-order packets Flow-based using hashing e.g. source and destination addresses (and potentially additional header fields): Effectiveness depends on the number and distribution of flows (according to the hash function) ECMP is leveraged by [EVPN], [VXLAN] [TRILL], [FabricPath] BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 Equal Cost Trees (ECT) Optimizations for 802.1aq Shortest Path Bridging 802.1aq SPB requires to compute a Shortest Path Tree (SPT) per Node Fast SPF to reduce the number of SPT Equal Cost Tree 802.1aq Equal Cost Tree (ECT) Algorithm allows to compute 16 different trees per node Deterministic masking operation by the source node (root of the SPF) to place traffic onto any of the 16 trees BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 802.1aq Equal Cost Trees (ECT) TRILL ECMP Example 802.1aq ECT Example Per-Hop (ECMP) vs. Global (ECT) Traffic Hashing Different from ECMP, ECT can only identify a maximum of 16 different paths between any source and destination pair Can lead to situations where certain links are not utilized at all (depending on the hash function used) BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 802.1Qbp ECMP Motivation and Requirements Support of per-hop ECMP Support of TTL for loop mitigation Support of Flow Hash To avoid deep packet inspection in the core To provide proactive service-level monitoring Flexible n-tuple hash algorithm for flow-identification Any edge node can choose any set of n-tuples and any hash algorithm to derive a flow id Support proactive service-level monitoring For a given flow-id, the path for that flow through the network be deterministic BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 DE UCA rsv1 Recap: Existing PBB/802.1ah Frame Format B-DA B-SA B-TAG B-VID I-TAG I-SID DA SA S-TAG S-VID C-TAG C-VID Payload PCP (3b) EtherType = I-Tag rsv2 (2b) I-SID (part 2) (16b) I-SID (part 1) (8b) PCP: Priority Code Point 3 bits DE: Discard Eligible 1 bit UCA: Use Customer Address 1 bit Rsv1: Reserved1 1 bit Rsv2: Reserved2 2 bits I-SID: Service ID 24 bits BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 DE IEEE 802.1Qbp-2014 / ECMP Frame Format B-DA B-SA F-TAG I-TAG I-SID DA SA S-TAG S-VID C-TAG C-VID PCP (3b) F-Tag Fields: EtherType = F-Tag (89-4b) Rsv (6b) Flow Hash (16b) TTL (6b) PCP: Priority Code Point 3 bits (copied from B-Tag) DE: Discard Eligible 1 bit (copied from B-Tag) Rsv1: Reserved 6 bits TTL: Hop Count 6 bits Flow Hash: 16 bits Payload BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 Interconnecting Ethernet Domains How to Connect Ethernet Domains Across a WAN in an Efficient Way?

62 Interconnecting Ethernet Domains The Legacy Approach : Virtual Private LAN Services (VPLS)

63 Interconnecting NG Ethernet Domains using Legacy VPN technology - VPLS DC Gateway DC-1 GW-1 GW-1 DC-2 Blocked port Spine GW-n WAN GW-m Leaf VM1 VM2 Short-comings: Can only support active/standby multi-homing ie, out of N GWs, only one can be active to a given VNI/VLANVPLS instance (very poor load-balancing and network resource utilization basically defeats the propose of NG Eth domain!!) Learning in data-plane over PWs requires flooding over WAN!! Requires full-mesh of PWs per VNI (that can result in scale issue because of O(n^2) PWs) Flooding using ingress replication can exhaust network resources when n is large VM3 VM4 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Hierarchical VPLS (H-VPLS) To alleviate scale issues related to PW and flooding, H-VPLS was proposed to break the WAN into hierarchical topology of spoke PEs (u-pes) and hub PEs (n- PEs) Spoke PWs were used between u-pes and n-pes; whereas, full-mesh of PWs were used among n-pes Reduce flooding and PW scale issue from O(n^2) to O(m)+O[(n/k)^2] At what price? At the expense of increase CapEx and OpEx Active PW Standby PW U-PE1 Access: H&S PW Overlay N-PE1 N-PE2 H-VPLS with MPLS to the Edge VPLS Core: Full Mesh of PW BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 Scaling H-VPLS Further Combining H-VPLS and PBB (802.1ah) To address MAC scale issue at n-pes and to further improve PW scale at n- PEs, H-VPLS with PBB encapsulation on u-pes were proposed (RFC 7080) Because of MAC-in-MAC encap, number of MACs and number of VPLS instance on n-pes can be significantly reduced At what price? At the expense of additional encap on u-pes and still having increases CapEx and OpEx of H- VPLS and only supporting active/standby multi-homing Active PW Standby PW U-PE1 Access: H&S PW Overlay N-PE1 N-PE2 H-VPLS with MPLS to the Edge VPLS Core: Full Mesh of PW BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 Interconnecting Ethernet Domains The New Approach: Ethernet VPN (EVPN)

67 Towards EVPN Requirements All-Active Redundancy with Load balancing on L2/L3/L4 flows Geo-redundant PE nodes Flexible Redundancy grouping Optimum unicast forwarding & Flow-based multi-pathing Multicast optimization w/ MP2MP See: RFC 7209 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Towards EVPN Solve Additional Challenges of Current VPLS for All-Active Redundancy Looping of Traffic Flooded from PE MAC1 MAC2 Duplicate Frames from Floods from the Core MAC1 MAC2 MAC Flip-Flopping over Pseudowire In case Port-Channel Load-Balancing does not produce a consistent hash-value for a frame with the same source MAC (e.g. non MAC based Hash-Schemes) MAC1 MAC2 MAC2 MAC2 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 BGP MPLS Based Ethernet VPNs Main Principles Leverage similarities with L3VPN as much as possible Determining Reachability to Unicast MAC Addresses Local learning: PE continues to learn MAC addresses over AC Remote learning: Distribution of MAC Addresses using BGP When multiple PEs announce the same MAC, hash to pick one PE Multicast Traffic Distribution Options: MP2MP LSPs, P2MP LSP or MP2P (w/ ingress replication) MP2P (like L3VPN) Trees for Unicast Distribution Full-Mesh of PWs no longer required Ethernet Segment (ES) with Ethernet Segment Identifier (ESI) PE PE Further segmentation using Ethernet-TAGs, to e.g. identify VLAN(s). Use of Ethernet-TAGs is optional, but allows for more efficient forwarding at MES (no C-MAC lookup necessary). BGP PE PE See: RFC 7432 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 Features PE & Multicast Tunnel Auto-discovery Auto-discovery of PEs in VPN instance similar to VPLS Access/Redundancy Auto-discovery Determine if access Ethernet Segment is Single-Homed, Multi-Homed Device (MC-LAG) or Multi- Homed Network (MST, G.8032 ) Host Mobility BGP Handshake to handle MAC/IP Movement from one PE to another BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 New BGP E-VPN NLRI Route Types Route MAC/IP Advertisement Route Usage Advertise MAC Address Reachability Advertise IP/MAC Bindings Ethernet A-D Route MAC Mass-Withdraw Aliasing Advertising Split-Horizon Labels Inclusive Multicast Ethernet Tag Route PE & Multicast Tunnel Auto-Discovery Ethernet Segment Route Access/Redundancy Auto-Discovery DF Election BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Operation Loop prevention for Multi-homed segments Designated Forwarder Election (per ESI and Ethernet TAG) All-Active: Only DF can forward BUM traffic to Ethernet Segment Active/Standby: Only DF can forward all traffic to/from Ethernet Segment Multi-homed PE implement (access) Split-Horizon procedures Multi-homed PE include ESI MPLS label, which identifies the source ESI A PE that receives a multicast/broadcast frame from the WAN filters out that frame over an AC whose ESI matches the one in the received frame PE2 is DF for ESI 1 and ESI2 ESI 1 ESI 2 Ethernet A-D Route ESI: ESI 1 ESI Label L2 PE1 PE2 Ethernet A-D Route ESI: ESI 1 ESI Label L1 PE3 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 EVPN Operations Example (1/4) M1 communicates with M2 (e.g. ARP) - Broadcast AGG1 BGP AGG4 M1 ESI=1 AGG2 PE1 PE3 AGG5 ESI=3 M2 C-MAC2 C-MAC1 ESI=2 AGG3 PE2 PE4 AGG6 EVPN MAC Advertisement Route next-hop: MES1 <C-MAC1, Label 100> Host M1 sends ARP with MAC SA = C-MAC1 and MAC DA=FF.FF.FF.FF.FF.FF PE1 learns C-MAC1 over its Agg2-PE1 AC and distributes it via BGP to other PEs All other PEs learn that C-MAC1 is reachable via PE1 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 EVPN Operations Example (2/4) M1 communicates with M2 (e.g. ARP) - Broadcast AGG1 AGG4 M1 ESI=1 AGG2 PE1 PE3 AGG5 ESI=3 M2 C-MAC2 C-MAC1 ESI=2 AGG3 PE2 PE4 AGG6 PE1 sends ARP over all local ACs for which PE1 is DF, & over P2MP LSP (of that EVI) Only a single AC per multi-homed ESI can be a designated forwarder (DF) to send (but not receive) multicast/broadcast messages to the customer site Any AC in the group (per ESI) can receive multicast/broadcast messages from customer site PE2 receives message but drops it at its AGG2-PE2 AC even though this AC is a DF for ESI=1 because ESI of the frame matches the ESI of the AC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 EVPN Operations Example (3/4) Reply from M2 to M1 (Unicast) AGG1 AGG4 M1 AGG2 PE1 PE3 AGG5 ESI=3 M2 ESI=1 ESI=2 AGG3 PE2 PE4 AGG6 ibgp L2-NLRI next-hop: MES4 <C-MAC2, Label 100> Host M2 sends response with MAC SA = M2 and MAC DA = M1 PE4 learns M2 over its Agg5-PE4 AC and distributes it via BGP to other PE devices All other PE devices learn that M2 sits behind PE4 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 EVPN Operations Example (4/4) Reply from M2 to M1 (Unicast) AGG1 AGG4 M1 AGG2 PE1 PE3 AGG5 ESI=3 M2 ESI=1 ESI=2 AGG3 PE2 PE4 AGG6 Since PE4 already knows that M1 sits behind PE1, it forwards the frame to PE1 If PE4 has two BGP ECMP for M1 (e.g., both PE1 & PE2 advertised M1), then it uses a hash based on L2/L3/L4 header to decide which of the two PEs to forward the frame to Upon receiving the frame, PE1 does a MAC lookup and forwards the frame to Agg2-PE1 AC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Interconnecting NG Ethernet Domains EVPN DC Gateway DC-1 GW-1 EVPN over MPLS GW-1 DC-2 Spine GW-n WAN GW-m Leaf VM1 VM2 Short-comings: Supports Active/Active (All-Active) multi-homing ie, every GW forward known unicast packet for every VNI/VLAN (very good load-balancing and network resource utilization complements LB capabilities of NG Eth domain!!) Learning is done in control-plane thus no need for flooding over WAN and thus no scale issue related to flooding!! No PWs and thus no scale issue related to PWs VM3 VM4 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 DC Gateway L2 Gateway L2 Stretch: E-W and S-N all-active or single-active L2 EVPN/VXLAN in the DC L2 DCI (E-W): EVPN/VXLAN with EVPN/VPLS interworking L2 to client (S-N): EVPN/VXLAN with VPLS/PW interworking VPN Client clien t Branch VPLS/PW/EV PN DC Gateway DC-1 WAN DC-2 Gateway Spine L2 EVPN (per-bd VNI) Leaf VM1 VM2 L2/L3 DC fabric L2/L3 DC fabric VM3 VM4 Leaf Leaf L2 EVPN/VXLAN Gateway L2 stretch: EVPN/VPLS (MPLS) Gateway L2 EVPN/VXLAN Leaf

79 DC Gateway IRB Anycast Gateway Integrated Routing and Bridging DC fabric is L2 only. All routing on DC gateway DC gateway is the L3 default gateway for VMs via EVPN IRB anycast gateway Support both L2 and L3 for the same VNI at the same time Internet Client VPN Client clien t DC Gateway DC-1 IRB Internet WAN Branch IRB IRB IRB DC-2 IP-VPN Internet VPLS/PW/EVP N Gateway (L3 anycast GW) Spine L2 EVPN (per-bd VNI) Leaf VM1 VM2 L2 only DC fabric L2 only DC fabric VM3 VM4 Leaf (L2 only) Leaf L2 EVPN/VXLAN Gateway Gateway L2 stretch: EVPN/VPLS (MPLS) L2 EVPN/VXLAN Leaf

80 DC Gateway L3 Gateway S-N Routing: all active EVPN/VXLAN to IP-VPN/MPLS interworking EVPN/VXLAN to global internet L3 EVPN between Leaf and GW (per-vrf VNI) Leaf is the L3 default gateway for VMs (with EVPN IRB anycast GW) and does inter-vxlan routing Internet Client Internet Branch VPN Client clien t IP-VPN Internet DC Gateway DC-1 WAN DC-2 Gateway Spine L3 EVPN (per-vrf VNI) Leaf VM1 VM2 L2/L3 DC fabric L2/L3 DC fabric VM3 VM4 Leaf (L3 anycast GW)

81 Inter-LATA Connectivity between EVPN & VPLS Networks PE3 M3 EVPN Network GW1 GW2 VPLS Network PE1 PE2 West Coast Mid West East Coast BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 EVPN Interop w/ different networks Remote Site 2 MPLS PE3 A/S EVPN w/ VPLS Seamless Integ A/S PE1 VPLS Remote Site 2 A/S PE2 EVPN- VXLAN DC 1 Single-Active ES PE4 Single-Home ES PE5 EVPN- VXLAN DC 2 Single-Active ES PE6 VPLS w/ Single-Active redundancy VPWS w/ Single-Active redundancy Each pair of VPWS PWs are modeled as a ves ES SH filtering may be applied to such Single-Active ES to prevent transient loop BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Summary

84 Technologies, Solutions, and Standards Are Converging Optimal Forwarding Multi-Pathing, Optimal Network Topology Use IS-IS Control Plane: Unicast & Multicast ECMP and enhancements Multiple Topologies Addressing Scalability, Mobility, Lookup Efficient Interconnection of Ethernet Networks *: 3x3x3 Cube World Record Holder: Feliks Zemdegs, Melbourne Winter Open 2011: 5.66s: * Map n Encap : Location and Identity addresses Avoid Flooding, control plane learning, broadcast proxies ISIS/BGP Control Plane Control Plane learning Native IP and MPLS transport Active-Active Multi-Homing Efficient Multicast BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 References [FabricPath]: FabricPath [LISP]: Locator/ID Separation Protocol [802.1Qbp] ECMP [EVPN]: BGP MPLS Based Ethernet VPN [TRILL]: Transparent Interconnection of Lots of Links [VL2]: VL2: A Scalable and Flexible Data Center Network [MOOSE]: Addressing the Scalability of Ethernet with MOOSE [PORTLAND]: PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric [SEATTLE]: Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises [MONSOON]: Towards a Next Generation Data Center Architecture: Scalability and Commoditization [VLB]: Valiant Load Balancing in Backbone Networks BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 85

86 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 87

88 Thank you

89 Please join us for the Service Provider Innovation Talk featuring: Yvette Kanouff Senior Vice President and General Manager, SP Business Joe Cozzolino Senior Vice President, Cisco Services Thursday, July 14 th, :30 am - 12:30 pm, In the Oceanside A room What to expect from this innovation talk Insights on market trends and forecasts Preview of key technologies and capabilities Innovative demonstrations of the latest and greatest products Better understanding of how Cisco can help you succeed Register to attend the session live now or watch the broadcast on cisco.com

90