Intelligent WAN 2.0 Traffic Independent Design and Intelligent Path Selection

Transcription

1

2 Intelligent WAN 2.0 Traffic Independent Design and Intelligent Path Selection Tech-WAN Jaromír Pilař Consulting Systems Engineer, CCIE #2910

3 Cisco Intelligent WAN (IWAN) AVC Private Cloud ISR-AX 3G/4G-LTE ASR1000-AX Virtual Private Cloud Branch WAAS kamai PfRv3 Internet Public Cloud Management & Orchestration Transport Independence Intelligent Path Control Application Optimization Secure Connectivity IPSec WAN Overlay Consistent Operational Model Optimal application routing Efficient use of bandwidth Performance monitoring Optimization and Caching NG Strong Encryption Threat Defense DMVPN Performance Routing AVC, HQoS, WAAS, Akamai Suite-B, CWS, ZBFW

4 Transport Independent Design

5 Cisco Intelligent WAN (IWAN) AVC Private Cloud ISR-AX 3G/4G-LTE ASR1000-AX Virtual Private Cloud Branch WAAS kamai PfRv3 Internet Public Cloud Management & Orchestration Transport Independence Intelligent Path Control Application Optimization Secure Connectivity IPSec WAN Overlay Consistent Operational Model Optimal application routing Efficient use of bandwidth Performance monitoring Optimization and Caching NG Strong Encryption Threat Defense DMVPN Performance Routing AVC, HQoS, WAAS, Akamai Suite-B, CWS, ZBFW

6 Hybrid WAN Designs Traditional and IWAN Active/Standby WAN Paths Primary With Backup TRADITIONAL HYID Intelligent WAN HYID Active/Active WAN Paths Two IPsec Technologies Data Center Data Center One IPsec Overlay GETVPN/ DMVPN/Internet Two WAN Routing Domains : ebgp or Static Internet: ibgp, EIGRP or OSPF Route Redistribution Route Filtering Loop Prevention ISP A DMVPN Internet ASR 1000 ASR 1000 SP V GETVPN ISP A DMVPN Internet ASR 1000 ASR 1000 SP V DMVPN DMVPN One WAN Routing Domain ibgp, EIGRP, or OSPF ISR Branch ISR Branch

7 IWAN Deployment DMVPN IWAN Prescriptive Design Transport Independent Design based on DMVPN Branch spoke sites establish an IPsec tunnel to and register with the hub site Data traffic flows over the DMVPN tunnels WAN interface IP address used for the tunnel source address (in a Front VRF) One tunnel per user VRF Over the Top Routing BGP or EIGRP are typically used for scalability IP routing exchanges prefix information for each site Per-tunnel QOS is applied to prevent hub site oversubscription to spoke sites IWAN POP1 MC1 R84 R85 R94 R95 R10 R11 R12 R / /24 DCI WAN Core INET IWAN POP / /24 MC2

8 What is Dynamic Multipoint VPN? DMVPN Is a Cisco IOS Software Solution for Building IPsec + GRE VPNs in an Easy, Dynamic, and Scalable Manner Relies on Two Proven Technologies Major Features Next-Hop Resolution Protocol (NHRP) Creates a distributed mapping database of VPN (tunnel interface) to real (public interface) addresses Multipoint GRE tunnel interface Single GRE interface to support multiple GRE/IPsec tunnels and endpoints Simplifies size and complexity of configuration Supports dynamic tunnel creation Configuration reduction and no-touch deployment supports: Passenger protocols: IP(v4/v6) unicast, multicast, and dynamic routing protocols Transport protocols (NBMA): IPv4 and IPv6 Remote peers with dynamically assigned transport addresses Spoke routers behind dynamic NAT; hub routers behind static NAT Dynamic spoke-spoke tunnels for partial/full mesh scaling Works with ; GRE tunnels and/or data packets in VRFs and switching over the tunnels Wide variety of network designs and options Compatible with Performance Routing

9 Over-the-Top WAN Design with Dynamic Multipoint VPN (DMVPN) Branch spoke sites establish an IPsec tunnel to and register with the hub site IP routing exchanges prefix information for each site BGP or EIGRP are typically used for scalability Only the WAN IP addresses need to be known by the WAN transport SECURE ON-DEMAND TUNNELS ASR 1000 IPsec VPN Hub Branch n ISR WAN interface IP address can be used for the tunnel source address Data traffic flows over the DMVPN tunnels When traffic flows between spoke sites, the hub assists the spokes to establish a site-to-site tunnel Per-tunnel QOS is applied to prevent hub site oversubscription to spoke sites ISR Branch 1 ISR Branch 2 Traditional Static Tunnels DMVPN On-Demand Tunnels Static Known IP Addresses Dynamic Unknown IP Addresses

10 Best Practice VRF-aware DMVPN Keeping the Default Routes in Separate VRFs Customer routing context (Global table) FVRF SP1 (SP1 routing context) FVRF SP2 (SP2 routing context) Different default routes possible within global table and towards SP infrastructures Configuration towards SP simplified, allows for simple infrastructure migration

11 DMVPN Configuration F-VRF IWAN POP vrf definition IWAN-TRANSPORT-1 address-family ipv4 exit-address-family Front-door VRF definition for Transport MC1 R84 R vrf definition IWAN-TRANSPORT-2 address-family ipv4 exit-address-family Front-door VRF definition for Internet Transport INTERNET R /24

12 DMVPN Configuration IPSec crypto ikev2 keyring DMVPN-KEYRING-1 peer ANY MC1 address pre-shared-key c1sco123 R84 R85 crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-1 match fvrf IWAN-TRANSPORT-1 match identity remote address authentication remote pre-share Maximize window size to authentication local pre-share keyring local DMVPN-KEYRING-1 eliminate future anti-replay issue INTERNET crypto ipsec security-association replay window-size 512 crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac mode transport crypto ipsec profile DMVPN-PROFILE set transform-set AES256/SHA/TRANSPORT set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-1 R10 crypto ikev2 dpd 40 5 on-demand Set DPD timers for Branch Configs ONLY IWAN POP /24

13 DMVPN Hub Configuration Interfaces & Routing TRANSPORT R84 interface GigabitEthernet0/0/3 description -TRANSPORT Put Transport Interface into MC1 vrf forwarding IWAN-TRANSPORT-1 Front-door VRF ip address interface Tunnel100 Instantiate DMVPN Tunnel R84 R85 bandwidth ip address no ip redirects ip mtu 1400 ip pim nbma-mode ip pim sparse-mode DMVPN Network ID: ip nhrp authentication cisco123 ip nhrp map multicast dynamic INTERNET ip nhrp network-id 100 ip nhrp holdtime 600 ip nhrp redirect Set DMVPN Ph3 ip tcp adjust-mss 1360 tunnel source GigabitEthernet0/0/3 Map to Physical Interface tunnel mode gre multipoint tunnel key 101 tunnel vrf IWAN-TRANSPORT-1 tunnel protection ipsec profile DMVPN-PROFILE-1 Tunnel endpoint is in Front-door VRF ip route vrf IWAN-TRANSPORT Default route for Tunnel endpoints IWAN POP

14 DMVPN Spoke Configuration Interfaces & Routing TRANSPORT R10 Interface GigabitEthernet0/1 vrf forwarding IWAN-TRANSPORT-1 ip address interface Tunnel100 ip address no ip redirects ip mtu 1400 ip pim dr-priority 0 ip pim sparse-mode ip nhrp authentication cisco123 ip nhrp network-id 100 ip nhrp holdtime 600 ip nhrp nhs nbma multicast ip nhrp nhs nbma multicast ip nhrp registration no-unique ip nhrp shortcut ip tcp adjust-mss 1360 if-state nhrp tunnel source GigabitEthernet0/1 tunnel mode gre multipoint tunnel key 101 tunnel vrf IWAN-TRANSPORT-1 tunnel protection ipsec profile DMVPN-PROFILE-1 ip route vrf IWAN-TRANSPORT Put Transport Interface into Front-door VRF Instantiate DMVPN Tunnel DMVPN Network ID: Multiple DMVPN Hub for Resiliency Set DMVPN Ph3 NHRP control i/f state Tunnel endpoint is in Front-door VRF Default route for Tunnel endpoints R /24

15 IWAN Routing Protocols Which protocol should I use? IWAN Profiles are based upon BGP and EIGRP for scalability and optimal Intelligent Path Control Scalability: BGP (Path Vector) and EIGRP (Advanced Distance Vector) provide best scale over large hub-and-spoke topologies like DMVPN OSPF (Link State) maintains a lot of network state which cannot be subdivided easily in large DMVPN networks Intelligent Path Control: PfR can be used with any routing protocols by relying on the routing table (RIB). Requires all valid WAN paths be ECMP so that each valid path is in the RIB. For BGP and EIGRP, PfR can look into protocol s topology information to determine both best paths and secondary paths thus, ECMP is not required.

16 EIGRP IWAN Design Overview Single EIGRP process for Branch, WAN and POP/hub sites Extend Hello/Hold timers for WAN Adjust tunnel interface delay to ensure WAN path preference primary, INET secondary Hubs Route tag filtering to prevent routing loops across DMVPNs Branch prefix summary route for spoke-to-spoke tunnels Spokes EIGRP Stub for scalability MC1 Set Interface Delay to influence best path Delay 1000 Hub MC /32 Delay 1000 R84 TAG DMVPN-1 DMVPN / /24 IWAN POP /16 IWAN POP /16 Delay 2000 R85 INET DMVPN-2 TAG DMVPN-2 EIGRP Stub R10 R11 R12 R / /24

17 IWAN Deployment BGP A single ibgp routing domain is used Extend Hello/Hold timers for WAN Hub: DMVPN hub routers function as BGP routereflectors for the spokes BGP dynamic peer feature configured on the route-reflectors Summary route to spokes Set Community for site local prefixes Spokes: peer to a redundant pair of route-reflectors in each DMVPN cloud Inbound route-map to set local-preference based on community Ensure that preferred path is MC / / /24 IWAN POP1 R84 R85 Community 10:84 Community 10:85 ibgp ibgp Local Pref 200 Hub MC /32 IWAN POP2 INET / /16 LP 50 R10 R11 R12 R / / /16 20

18 Deploying with user VRFs vrf definition TEST1 address-family ipv4 exit-address-family vrf definition TEST2 address-family ipv4 exit-address-family interface Tunnel 101 vrf forwarding TEST1 tunnel key 101 tunnel vrf IWAN-TRANSPORT-1 interface Tunnel 102 vrf forwarding TEST2 tunnel key 102 tunnel vrf IWAN-TRANSPORT-1 MC1 TRANSIT SITE R84 R85 INET DMVPN Tunnel per VRF Over the top routing per VRF SAF Peering per VRF R10 R11 R12 R / / / /24 Enterprise Branch Sites

19 Intelligent Path Control with PfR

20 Cisco Intelligent WAN (IWAN) AVC Private Cloud ISR-AX 3G/4G-LTE ASR1000-AX Virtual Private Cloud Branch WAAS kamai PfRv3 Internet Public Cloud Management & Orchestration Transport Independence Intelligent Path Control Application Optimization Secure Connectivity IPSec WAN Overlay Consistent Operational Model Optimal application routing Efficient use of bandwidth Performance monitoring Optimization and Caching NG Strong Encryption Threat Defense DMVPN Performance Routing AVC, WAAS, Akamai Suite-B, CWS, ZBFW

21 Intelligent Path Control with PfR Enterprise Use-Case Voice, video and critical applications take the best delay, jitter, and/or loss path Private Cloud Branch Other traffic is load balanced to maximize bandwidth Internet PfR monitors network performance and routes applications based on application performance policies PfR load balances traffic based upon link utilization levels to efficiently utilize all available WAN bandwidth Virtual Private Cloud Voice, video and critical applications will be rerouted if the current path degrades below policy thresholds

22 PfRv3 How it Works ISR ASR1K MC Traffic Classes Learning Active TCs MC Performance Measurements MC TC Path Define your Traffic Policy Learn the Traffic Measurement Path Enforcement Define path optimization policies on the Hub MC load balancing, path preference, application metrics DSCP Based Policies Application Based Policies Traffic flowing through the Border Routers (s) that match a policy are learned Traffic Classes Unified Performance Monitor Report the measured TC performance metrics to the Master Controller for policy compliance Unified Performance Monitor Master Controller directs path changes to keep traffic within policy Route Enforcement module in feature path

23 Intelligent Path Control Performance Routing Evolution IWAN 2.0 PfR/OER Internet Edge Basic WAN Provisioning per site per policy 1000s of lines of config PfRv2 Policy simplification App Path Selection Blackout ~6s Brownout ~9s Scale 500 sites 10s of lines of config PfRv3 Centralized provisioning AVC Infrastructure VRF Awareness Blackout ~ 2s Brownout ~ 2s Scale 2000 sites Hub config only

24 Configuration Improvement and Simplification PfRv2 PfRv3 MC Configuration on Hub site only pfr master policy-rules IWAN-pfr-map max-range-utilization percent 30 mc-peer domain 1 eigrp Loopback0 target-discovery logging border key-chain PFR-KEYCHAIN interface GigabitEthernet0/0.32 internal interface GigabitEthernet0/0.31 internal interface GigabitEthernet0/0.30 internal interface Tunnel2 external link-group internet interface Tunnel1 external link-group mpls learn traffic-class filter access-list DENY_GLOBAL_LEARN_LIST list seq 10 refname LEARN_VOICE_VIDEO traffic-class access-list VOICE_VIDEO filter ANCH_PREFIX count 2000 max throughput list seq 20 refname LEARN_CRITICAL traffic-class access-list CRITICAL filter ANCH_PREFIX count 2000 max throughput list seq 30 refname LEARN_BEST_EFFORT traffic-class prefix-list BEST_EFFORT_PREFIX count 2000 max throughput periodic 90 probe packets 20 pfr-map IWAN-pfr-map 10 match pfr learn list LEARN_VOICE_VIDEO set periodic 90 set delay threshold 200 set mode monitor fast set resolve delay priority 1 variance 5 set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set loss threshold set jitter threshold 30 set probe frequency 8 set link-group mpls fallback internet pfr-map IWAN-pfr-map 20 match pfr learn list LEARN_CRITICAL set periodic 90 set delay threshold 100 set mode monitor fast set resolve delay priority 1 variance 20 set resolve loss priority 5 variance 10 set probe frequency 8 set link-group mpls fallback internet pfr-map IWAN-pfr-map 30 match pfr learn list LEARN_BEST_EFFORT set periodic 90 set mode monitor passive router eigrp IWAN service-family ipv4 autonomous-system 1 sf-interface default shutdown exit-sf-interface sf-interface Loopback0 no shutdown hello-interval 200 hold-time 600 exit-sf-interface topology base exit-sf-topology neighbor Loopback0 remote 100 exit-service-family ip prefix-list BEST_EFFORT_PREFIX seq 10 permit /0 ip prefix-list BEST_EFFORT_PREFIX seq 5 deny /16 ip prefix-list BEST_EFFORT_PREFIX seq 10 permit /0 ip access-list extended CRITICAL permit ip any any dscp af31 ip access-list extended VOICE_VIDEO permit ip any any dscp ef permit ip any any dscp af41 permit ip any any dscp cs4 500 sites ~90 lines of configuration All MCs domain <MYNAME> vrf <name> master hub source-interface Loopback0 password IWAN load-balance class VOICE sequence 10 match dscp ef policy voice path-preference mpls fallback inet class VIDEO sequence 20 match dscp af41 policy real-time-video match dscp cs4 policy real-time-video path-preference mpls fallback inet class APPLICATION sequence 30 match dscp af31 policy low-latency-data 2000 sites <20 lines of configuration, all under domain On the hub only

25 PfRv3 Principles

26 IWAN Domain Concept A collection of sites sharing the same set of policies Each site runs Performance Routing components They exchanges services through the Enterprise Domain Peering framework Centralized configuration from a Domain Controller Overlay network per Transport for flexibility and simplification IWAN Domain Controller IWAN POP1 MC1 DC1 Transit DMVPN DCI WAN Core DCn Transit DMVPN INET IWAN POP2 MC2 R10 R11 R12 R13 Branch Branch Branch

27 PfRv3 Components MC/ ANCH Dual CPE MC Transit INET MC/ ANCH Single CPE The Decision Maker: Master Controller (MC) Apply policy, verification, reporting No packet forwarding/ inspection required Standalone of combined with a VRF Aware The Forwarding Path: Border Router () Gain network visibility in forwarding path (Learn, measure) Enforce MC s decision (path enforcement) VRF aware

28 PfRv3 Sites TRANSIT SITE Hub Site ID = ANCH SITE Site11 Site ID = MC/ ANCH Dual CPE MC TRANSIT INET MC/ ANCH Single CPE Controlled by a local Master Controller (MC) Site ID the IP address of the MC loopback One/Multiple s Each one/multiple links Site Type ANCH SITE Site10 Site ID = Transit Sites Enterprise POPs or Hubs Branch Sites - Stub

29 Transit Site Hub Master Controller MC/ ANCH Dual CPE DC/MC TRANSIT INET MC/ ANCH Single CPE domain IWAN vrf default master hub source-interface Loopback0 One of the MC is assigned the Domain Controller role Central point of provisioning for Domain policies DC + MC = Hub Master Controller

30 Branch Site MC/ ANCH Dual CPE DC/MC TRANSIT INET MC/ ANCH Single CPE Service Exchange Policies and Monitor configurations Site Prefixes domain IWAN vrf default master branch source-interface Loopback0 hub border source-interface Loopback0 master local

31 Policy/Monitor Distribution Policies Monitors DC/MC MC/ ANCH Dual CPE TRANSIT INET MC/ ANCH Single CPE Domain policies and monitor instances are configured on the Hub MC. Policies are defined per VRF Then distributed to branch sites using the peering infrastructure

32 Performance Policies - DSCP or App Based domain IWAN vrf default master hub load-balance class MEDIA sequence 10 match application telepresence-media policy real-time-video match application ms-lync policy real-time-video path-preference fallback INET class VOICE sequence 20 match dscp ef policy voice path-preference fallback INET class CRITICAL sequence 30 match dscp af31 policy low-latency-data Policies: DSCP or Application Based Policies (NBAR2) DSCP marking can be used with NBAR2 on the LAN interface (ingress on ) Default Class is load balanced

33 Built-in Policy Templates Voice Pre-defined Template Threshold Definition priority 1 one-way-delay threshold 150 (msec) priority 2 packet-loss-rate threshold 1 (%) priority 2 byte-loss-rate threshold 1 (%) priority 3 jitter 30 (msec) Real-time-video priority 1 packet-loss-rate threshold 1 (%) priority 1 byte-loss-rate threshold 1 (%) Low-latency-data priority 2 one-way-delay threshold 150 (msec) priority 3 jitter 20 (msec) priority 1 one-way-delay threshold 100 (msec) priority 2 byte-loss-rate threshold 5 (%) priority 2 packet-loss-rate threshold 5 (%) Pre-defined Template Bulk-data Best-effort scavenger Threshold Definition priority 1 one-way-delay threshold 300 (msec) priority 2 byte-loss-rate threshold 5 (%) priority 2 packet-loss-rate threshold 5 (%) priority 1 one-way-delay threshold 500 (msec) priority 2 byte-loss-rate threshold 10 (%) priority 2 packet-loss-rate threshold 10 (%) priority 1 one-way-delay threshold 500 (msec) priority 2 byte-loss-rate threshold 50 (%) priority 2 packet-loss-rate threshold 50 (%)

34 Discovery WAN Interface Define the Path Names on the Hub Border Routers Branch s discover Path Names and External Interfaces MC Hub Master MC MC/ ANCH Dual CPE TRANSIT HUB INET MC/ Using Smart Probes Hub generates Discovery packets over all available paths Discovery packets intercepted on spokes External interfaces and path names automatically discovered and added ANCH Single CPE

35 Performance Monitoring User Traffic Passive Monitoring MC/ SITE2 Dual CPE MC SITE1 INET MC/ SITE3 Single CPE Bandwidth on egress Per Traffic Class (dest-prefix, DSCP, AppName) Performance Monitor Collect Performance Metrics Per Channel - Per DSCP - Per Source and Destination Site - Per Interface

36 Performance Monitoring Smart Probing Smart Probing MC/ SITE2 Dual CPE MC SITE1 INET MC/ SITE3 Single CPE Integrated Smart Probes Traffic driven intelligent on/off Site to site and per DSCP Performance Monitor Collect Performance Metrics Per Channel - Per DSCP - Per Source and Destination Site - Per Interface

37 Performance Violation MC/ SITE2 Dual CPE MC SITE1 INET MC/ SITE3 Single CPE Threshold Crossing Alert (TCA) Sent to source site loss, delay, jitter, unreachable

38 Performance Violation NetFlow Export MC/ SITE2 Dual CPE MC SITE1 INET MC/ SITE3 Single CPE

39 Policy Decision MC/ SITE2 Dual CPE MC SITE1 INET MC/ SITE3 Single CPE Reroute Traffic to a Secondary Path

40 Key Points and Limitations Key Points DSCP or Application (NBAR2) based policies Control traffic in the global Routing Table and per VRF Passive Monitoring based on user traffic Leverage smart probing when needed Cooperation between pair of sites Exports of PfR information with NetFlow v9 to collectors Limitation: No IPv6 support yet (Roadmap) But infrastructure is IPv6 ready NBAR2 with Asymmetric Routing is not yet supported

41 PfRv3 A closer look

42 PfRv3 Monitoring Based on Performance Monitor PfRv3 defines multiple Performance Monitor Instances (PMI) Applied on all External interfaces MC Exports to MC Monitor1 Site Prefix Learning (egress direction) Monitor2 Aggregate Bandwidth per Traffic Class (egress direction) 1 1 Site Monitor3 Performance measurements (ingress direction)

43 Monitor2 Details Aggregate Bandwidth Per TC Bandwidth is measured egress direction. Using monitor #2 Collect bandwidth per Traffic Class MC PMI: [Egress-Aggregate] - #2 Trigger NBAR: no/yes Key Fields ipv4 destination prefix, ipv4 destination mask, pfr site destination prefix ipv4 pfr site destination prefix mask ipv4 [application id] ip dscp interface output Non-Key Fields counter bytes long counter packet long timestamp absolute monitoring-interval start SITE Exports to MC 2 Monitor-interval 30 sec default PMI: Performance Monitor Instance Traffic Flow

44 Source Site Egress Traffic IWAN POP MC1 Dst-Site-Pfx App DSCP Dst-Site-Id State BW Exit APP1 AF41? UK 24 1 Tu10 MC1 Hub MC /32 Source Destination DSCP App AF41 APP Collecting Traffic Class Traffic going outside a source site Initially based on Routing Information Captured by a Performance Monitor on the external interface on egress reports to its local MC Monitor interval is 30 sec / /24 INET R10 R11 R12 R / /24

45 What is a Traffic Class? Dst-Site-Pfx App DSCP Dst-Site-id State Exit APP1 AF41? UK?? N/A EF? UK?? MC1 Hub MC /32 IWAN POP N/A AF31? UK?? N/A 0? UK?? N/A EF? UK?? N/A AF31? UK?? N/A 0? UK?? N/A 0? UK?? 1 2 INET What is a Traffic Class (TC)? Destination Site Prefix DSCP value Application Name Each Master Controller owns a TC Database R10 R11 R12 R / / / /24 49

46 We need the Destination Site ID Why?? MC1? Dst-Site-Pfx App DSCP Dst-Site-id State Exit APP1 AF41? UK?? SITE INET Performance Measurement is done on the destination site Source site needs to get Performance Metrics from the destination site 50

47 Site Prefix Exchange Dynamic Based on User Traffic Source Prefix and Mask collected from Performance Monitor send to its local MC MC send information to all peers via Peering Every MC in the domain owns a Site Prefix database Site Prefix List Hub /16 R /24 R /24 R /24 R /24 Source Destination DSCP App AF41 XYZ R10 MC Site-Pfx Mask /24 Site /24 MC1 Site /24 1 Hub MC /32 1 IWAN POP Site / / /24 2 INET Site /24 R10 R11 R12 R / /24

48 Monitor1 Details Site Prefix collected with Performance Monitor Using monitor #1 MC PMI: [Egress-prefix-learn] - #1 Trigger NBAR: no Key Fields ipv4 source prefix, ipv4 source mask, Routing vrf input Non-Key Fields counter bytes long counter packet long timestamp absolute monitoring-interval start Site /24 Exports to MC 1 Monitor-interval 30 sec default Traffic Flow

49 Site Prefixes Static Configuration This allows configuring site-prefix manually instead of learning. This configuration should be used at the site if the site is used for transit. For example, Site A reaches Site B via Hub-Site, where Hub-Site is transit site. The configuration is used to prevent learning of Site A prefix as Hub-Site prefix when it is transiting from Hub. MC1 Hub MC /32 1 TRANSIT SITE 2 domain IWAN vrf default master hub source-interface Loopback0 site-prefixes prefix-list DC1_PREFIX ip prefix-list DC1_PREFIX seq 10 permit /16 Source Destination DSCP App AF41 AppXY

50 Source Site Egress Traffic IWAN POP Site Prefix List Hub /16 MC1 Hub MC /32 MC1 R /24 R /24 R /24 Source Destination DSCP App R / AF41 APP1 MC1 Dst-Site-Pfx App DSCP Dst- Site-id State BW Exit APP1 AF41 R10 CN 24 1 Tu100 INET Now the MC has the destination site information Traffic Class is now controlled R10 R11 R12 R / / / /24

51 Source Site Traffic Class Database Dst-Site-Pfx Dst-Site-id App DSCP State Exit R10 APP1 AF41 CN 1 Tu100 MC1 Hub MC /32 IWAN POP R10 N/A EF CN 1 Tu R10 N/A AF31 CN 1 Tu R10 N/A 0 CN 2 Tu R11 N/A EF CN 1 Tu R11 N/A AF31 CN 1 Tu R11 N/A 0 CN 2 Tu R12 N/A 0 CN 2 Tu INET Same process for all traffic to between all sites. Traffic Class database contains bandwidth, destination sites, and external path used R10 R11 R12 R / / / /24 55

52 Destination Site Ingress Traffic Channel Performance Traffic flow captured on the destination site Performance Monitor collects Performance Metrics Per Channel Default Monitor interval is 30 sec MC1 Hub MC /32 1 IWAN POP 2 INET Source Destination DSCP App AF41 APP1 3 R10 MC Channel Dst-Site-id Path DSCP BW Delay Jitter Loss 5 Hub Tu1 AF R10 R11 R12 R / / / /24

53 Monitor3 Details PMI [Ingress-per-DSCP] - #3 Performance is measured ingress direction: Actual Traffic Smart Probes if no traffic Using monitor #3 Monitors performance metrics per DSCP and per Site Export only when there is a performance issue (performance threshold crossed) Key Fields pfr site source id ipv4 pfr site destination id ipv4 ip dscp Interface input policy performance-monitor classification hierarchy Non-Key Fields transport packets lost rate transport bytes lost rate pfr one-way-delay network delay average transport rtp jitter inter arrival mean counter bytes long counter packets long timestamp absolute monitoring-interval start Monitor-interval 30 sec default SITE 1 3 R10 Traffic Flow 57

54 What is a Channel? Between Sites A Channel is a unique combination of: Interface Peer-Site id DSCP Created Based on real traffic observed on the s Added every time there is a new DSCP or a new interface or a new site added to the prefix database. Smart Probe is received On all exits (Present Channel, Backup Channel)

55 Channel Details IWAN POP MC1 Hub MC /32 Present Channel 3 Site 10 DSCP AF41 DMVPN1 1 2 Backup Channel 4 Site 10 DSCP AF41 DMVPN2 INET R10 R11 R12 R / / / /24

56 Channel Details MC1 Hub MC /32 1 IWAN POP 2 Between Any Pair of Sites that has traffic Present Channel 13 Site 11 DSCP EF DMVPN1 INET Backup Channel 14 Site 11 DSCP EF DMVPN2 R10 R11 R12 R / / / /24

57 Smart Probing MC INET MC 3 Site /24 3 Traffic Flow Help for Maesurement Over Channels Site /24 Without actual traffic sends 10 probes spaced 20ms apart in the first 500ms and another similar 10 probes in the next 500ms, thus achieving 20pps for channels without traffic. With actual traffic Lower frequency when real traffic is observed over the channel Probes sent every 1/3 of [Monitor Interval], ie every 10 sec by default Measured by Unified Performance Monitoring just like other data traffic 61

58 Smart Probing Zero SLA Support MC INET MC 3 Site /24 3 Traffic Flow Site /24 No SLA on the secondary path, but still probing all channels (DSCPs) 10 DSCP => 10 Smart Probes over the secondary path Waste of bandwidth, especially for metered interfaces (4G/LTE)

59 Path Enforcement Route Override

60 Performance Violation Performance notification exported ONLY when there is a violation on a specific channel MC1 Hub MC /32 Enterprise HQ Generated from ingress monitor attached on s to the source site MC 1 2 Based on Monitor interval (30 sec default, configurable) Via all available external interfaces. R10 TCA Delay DSCP AF41 Path INET R10 Channel Dst-Site-id DSCP Path BW Delay Jitter Loss 5 Hub AF41 Tu R10 R11 R12 R / / / /24

61 Performance Violation Detected on Dst Site Enterprise HQ Dst-Site-Pfx Dst-Site-id App DSCP State Exit R10 APP1 AF41 CN 1 Tu1 MC1 Hub MC / R10 N/A AF41 CN 1 Tu R10 N/A AF31 CN 1 Tu R10 N/A 0 CN 2 Tu R11 N/A EF CN 1 Tu R11 N/A AF31 CN 1 Tu R11 N/A 0 CN 2 Tu R12 N/A 0 CN 2 Tu2 1 2 INET R10 TCA Delay DSCP AF41 Path 3 R10 R11 R12 R / / / /24

62 Policy Decision Reroute TC MC computes a new path for each impacted TC MC tells the s to enforce the new path MC1 Hub MC /32 1 Enterprise HQ 2 INET 3 R10 R11 R12 R / / / /24

63 Reroute TC Path Enforcement Dataplane forwarding Activated on all but external interfaces Lookup per packet - output-if/next hop retrieved Packet Forwarded If no entry Uses FIB entry TC flows redirected to the new path over the auto mgre tunnels between the s No change in the routing table MC1 3 Hub MC / / /24 Enterprise HQ 2 INET R10 R11 R12 R / /24

64 Enterprise Deployment

65 PfRv3 and Parent Routes Make sure that all Border Routers have a route over each external path to the destination sites PfR will NOT be able to effectively control traffic otherwise. PfRv3 always checks for a parent route before being able to control a Traffic Class. Parent route check is done as follows: Check to see if there is an NHRP shortcut route If not Check in the order of BGP, EIGRP, Static and RIB If at any point, an NHRP short cut route appears, PfRv3 would pick that up and relinquish using the parent route from one of the routing protocols. PfR3 currently supports only one next-hop per multipoint interface. Routing has to be done such that only one next-hop per destination prefix is in the routing table per DMVPN tunnel interface.

66 Transit Site Hub MC TRANSIT SITE MC1 domain IWAN vrf default master hub source-interface Loopback0 enterprise-prefix prefix-list ENTERPRISE_PREFIX site-prefixes prefix-list DC1_PREFIX MC1 Policies Monitors R84 path R85 path INET 1 2 domain IWAN vrf default border master source-interface Loopback0 interface Tunnel100 description -- Primary Path -- domain IWAN path domain IWAN vrf default border master source-interface Loopback0 interface Tunnel200 description Secondary Path -- domain IWAN path INET INET IWAN POP is the central hub for the Enterprise Domain MC1 Hub MC 1 Hub, DMVPN Hub for 2 Hub, DMVPN Hub for INET MC/ MC/ MC/ / / / /24

67 Single CPE Branch Sites TRANSIT SITE MC1 Hub MC /32 R10 R11 domain IWAN vrf default master branch source-interface Loopback0 hub border master local source-interface Loopback0 R84 R85 INET Stub Site Combination of MC and on the same CPE R10 R11 R12 R / / / /24

68 Dual CPE Branch Sites TRANSIT SITE R12 domain IWAN vrf default master branch source-interface Loopback0 hub border master local source-interface Loopback0 MC1 Hub MC /32 R84 R85 R13 domain IWAN vrf default border master source-interface Loopback0 INET R10 R11 R12 R13 Stub Site One of the is also the MC / / / /24

69 Enterprise Domain Policy (DSCP or App Based) domain IWAN vrf default master hub load-balance class VOICE sequence 10 match dscp ef policy voice path-preference fallback INET class VIDEO sequence 20 match dscp af41 policy voice path-preference fallback INET class CRITICAL sequence 30 match dscp af31 policy low-latency-data When load balancing is enabled, PfRv3 adds a default class for match all DSCP (lowest priority compared to all the other classes) and we influence this traffic. When load balancing is disabled, PfRv3 deletes this default class and as a part of that frees up the TCs that was learnt as a part of LB they follow the routing table MC1

70 Policy Monitor Intervals domain IWAN vrf default master hub monitor-interval 2 dscp ef monitor-interval 2 dscp af41 monitor-interval 2 dscp cs4 monitor-interval 2 dscp af31 Monitoring Interval Fast Reaction Time 2 sec MC1 Advanced commands Carefully choose monitor interval for critical applications 78

71 Deploying with VRF Hub MC TRANSIT SITE interface Loopback1 vrf forwarding TEST1 interface Loopback2 vrf forwarding TEST2 MC1 GLOBAL: VRF TEST1: VRF TEST2: R84 R85 domain IWAN vrf TEST1 master hub source-interface Loopback1 vrf TEST2 master hub source-interface Loopback / /24 INET R10 R11 R12 R / /24 Enterprise Branch Sites

72 Deploying with VRF Hub MC Policies domain IWAN vrf TEST1 master hub load-balance class VOICE sequence 10 match dscp ef policy voice path-preference fallback INET class VIDEO sequence 20 match dscp af41 policy voice path-preference fallback INET class CRITICAL sequence 30 match dscp af31 policy low-latency-data [Cont d] vrf TEST2 master hub load-balance class VOICE sequence 10 match dscp ef policy voice path-preference fallback INET class CRITICAL sequence 30 match dscp af31 policy low-latency-data

73 Deploying with VRF Hub domain IWAN vrf TEST1 border master source-interface Loopback1 vrf TEST2 border master source-interface Loopback2 interface Tunnel101 description -- Primary Path vrf forwarding TEST1 domain IWAN path interface Tunnel102 description -- Primary Path vrf forwarding TEST2 domain IWAN path MC1 GLOBAL: VRF TEST1: VRF TEST2: Tu101 Tu102 R / /24 TRANSIT SITE R85 INET R10 R11 R12 R / /24 Enterprise Branch Sites

74 Deploying with VRF Branch MC/ TRANSIT SITE R10 domain IWAN vrf TEST1 master branch source-interface Loopback1 hub border master local source-interface Loopback1 vrf TEST2 master branch source-interface Loopback2 hub border master local source-interface Loopback2 MC1 Tu101 Tu102 GLOBAL: VRF TEST1: VRF TEST2: R / /24 R85 INET R10 R11 R12 R / /24 Enterprise Branch Sites

75 Redundant MC Anycast IP What happens when a MC fails? Traffic forwarded based on routing information ie no drop What happens when the Hub MC fails? Branch MCs keep their configuration and policies Continue to optimize traffic A backup MC can be defined on the hub. Using the same IP address as the primary Routing Protocol is used to make sure s and branch MC connect to the primary Stateless redundancy Backup MC will re-learn the traffic MC1 Hub MC /32 R / /24 MC2 TRANSIT SITE Backup Hub MC /30 R85 INET R10 R11 R12 R / /24

76 Dual POPs Different Prefixes Requirements: Separate datacenters/pops Separate prefix advertised from each datacenters to spokes POP2 Hub MC Configured as Branch IWAN POP1 MC1 EIGRP/BGP /16 DCI WAN Core / /16 R84 R85 R94 R95 EIGRP/BGP /16 EIGRP/BGP /16 IWAN POP2 MC2 EIGRP/BGP /16 INET R10 R11 R12 R / / / /24

77 Horizontal Scaling Architecture PfRv3 Multiple DMVPN Next Hop Limitation Limitations: PfRv3 manages traffic between Tunnel Interfaces, not multiple tunnels within a single Tunnel Interface Spokes have multiple next hops on the same DMVPN tunnel Interface Channel definition: local site id + remote site id + DSCP + color(sp) No differentiation for multiple channels within a color(sp) Solution: PfRv3 DMVPN Multiple Next Hop support Need to add sub-color to differentiate channels New channel definition MC1 Hub MC / Next Hop 1 Next Hop 2 TRANSIT SITE INET local site id + remote site id + DSCP + color(sp) + SP tag 1 with tag 1, 2 with tag 2 Targeted for Spring XE 3.15 / 15.5(2)T releases R10 R11 R12 R / / / /24

78 PfR Deployment Hub (XE 3.15 / 15.5(2)T ) MC1 domain IWAN vrf default master hub source-interface Loopback0 enterprise-prefix prefix-list ENTERPRISE_PREFIX site-prefixes prefix-list DC_PREFIX MC1 HUB SITE Site ID = Hub MC POP ID 0 MC2 1 2 domain IWAN vrf default border master source-interface Loopback0 interface Tunnel100 description -- Primary Path -- domain IWAN path path-id 1 domain IWAN vrf default border master source-interface Loopback0 interface Tunnel200 description Secondary Path -- domain IWAN path INET path-id 2 Path Id Path INET Id 2 DMVPN DMVPN INET IWAN POP is the central hub for the Enterprise Domain MC1 Hub MC 1 Hub, DMVPN Hub for 2 Hub, DMVPN Hub for INET R10 R11 R12 R / / / /24

79 Multiple POPs Common Prefixes DC1 Requirements: 2 (or more) Transit Sites advertise the very same set of prefixes Datacenter may not be collocated with the Transit Sites DCs/DMZs are reachable across the WAN Core for each Transit Site Branches can access any DC or DMZ across either POP(hub). And, DC/DMZs can reach any branch across multiple Transit Sites (hubs). Multiple s per DMVPN per site may be required for crypto and bandwidth horizontal scaling Targeted for Spring XE 3.15 / 15.5(2)T releases TRANSIT SITE1 MC / / DMVPN R10 R11 R12 R / /24 DCI WAN Core TRANSIT SITE / /16 DMVPN INET / /24 MC2

80 PfR Deployment Transit Site (XE 3.15 / 15.5(2)T ) MC2 domain IWAN vrf default master transit 1 source-interface Loopback0 enterprise-prefix prefix-list ENTERPRISE_PREFIX site-prefixes prefix-list DC_PREFIX MC1 TRANSIT SITE Site ID = MC2 Transit MC POP ID domain IWAN vrf default border master source-interface Loopback0 interface Tunnel100 description -- Primary Path -- domain IWAN path path-id 1 domain IWAN vrf default border master source-interface Loopback0 interface Tunnel200 description Secondary Path -- domain IWAN path INET path-id DMVPN Path Id 1 DMVPN INET Path INET Id 2 IWAN POP is the central hub for the Enterprise Domain MC1 Hub MC, MC2 Transit MC 1 Hub, DMVPN Hub for 2 Hub, DMVPN Hub for INET R10 R11 R12 R / / / /24

81 Monitoring Operations R83#sh domain one master traffic-classes summary APP - APPLICATION, TC-ID - TRAFFIC-CLASS-ID, APP-ID - APPLICATION-ID SP - SERVICE PROVIDER, PC = PRIMARY CHANNEL ID, BC - BACKUP CHANNEL ID, - BORDER, EXIT - WAN INTERFACE UC - UNCONTROLLED, PE - PICK-EXIT, CN - CONTROLLED, UK - UNKNOWN Dst-Site-Pfx Dst-Site-Id APP DSCP TC-ID APP-ID State SP PC/BC /EXIT / N/A ef 11 N/A CN 25/ /Tunnel / N/A ef 9 N/A CN 20/ /Tunnel / N/A af31 6 N/A CN 24/ /Tunnel / N/A af31 8 N/A CN 26/ /Tunnel / N/A af31 4 N/A CN 11/ /Tunnel / N/A default 5 N/A CN INET 8/NA /Tunnel200 [SNIP] Total Traffic Traffic Classes: Class 12 Site: Site Internet: Critical 0 TC Id Controlled Path Information - Channels R83#

82 Check Traffic Classes Details R83#sh domain one master traffic-classes Dst-Site-Prefix: /24 DSCP: ef [46] Traffic class id:25 TC Learned: 00:12:44 ago Present State: CONTROLLED Current Performance Status: in-policy Current Service Provider: INET since 00:06:01 Previous Service Provider: INET for 181 sec (A fallback provider. Primary provider will be re-evaluated 00:00:01 later) BW Used: 24 Kbps Present WAN interface: Tunnel200 in Border Present Channel (primary): 84 Backup Channel: 85 Destination Site ID: Class-Sequence in use: 10 Class Name: VOICE using policy User-defined priority 2 packet-loss-rate threshold 5.0 percent priority 1 one-way-delay threshold 150 msec priority 2 byte-loss-rate threshold 5.0 percent BW Updated: 00:00:14 ago Reason for Route Change: Delay Check Traffic Class Voice for site 10 Active Path used Check Channels used (Primary and Backup) Policies and reason for last change

83 Check Channel after TCA MC1#sh domain IWAN master channels beg 107 Channel Id: 107 Dst Site-Id: Link Name: DSCP: ef [46] TCs: 1 Channel Created: 00:15:03 ago Provisional State: Initiated and open Operational state: Available Interface Id: 11 Estimated Channel Egress Bandwidth: 0 Kbps Immitigable Events Summary: Total Performance Count: 0, Total BW Count: 0 ODE Stats Bucket Number: 1 Last Updated : 00:05:45 ago Packet Count : 2 Byte Count : 116 One Way Delay : 254 msec* Loss Rate Pkts: 0.66 % Loss Rate Byte: 0.0 % Jitter Mean Unreachable : usec : FALSE [SNIP] On Demand Export (ODE) Threshold Crossing Alert (TCA) TCA Statistics: Received:1 ; Processed:1 ; Unreach_rcvd:0 Latest TCA Bucket Last Updated : 00:05:45 ago One Way Delay : 266 msec Loss Rate Pkts: NA Loss Rate Byte: NA Jitter Mean : NA Unreachability: FALSE

84 PfRv3 Management

85 PfRv3 Exporter Configuration IWAN POP domain IWAN vrf default master hub collector port 2055 MC1 Hub MC / MC1 Enable exporter on the Hub MC Collector IP address Default UDP port 9995 INET Distributed through SAF to all MCs and s in the domain R10 R11 R12 R / / / /24

86 PfRv3 NetFlow Export List of Templates Exports from MC TCA record Route Change record Immitigable Event Summary Bandwidth Exports from s Egress Measurement Template Ingress Measurement Template All records available at:

87 Cisco IWAN Management On-Prem Management Specialized Management Cloud-Based Management Prime Infrastructure 2.2 End-to-End Assurance of Application Experience Application Aware Network Performance Management Automates Deployment and Lifecycle Management Single-pane view of IWAN IWAN deployment workflows Plug and Play DMVPN, QoS, AVC deployment and monitoring PfR v3 in Q License includes IWAN App and APIC- EM controller Integrates with Cisco AVC and PfR Monitor and analyze application traffic End-to-end flow visualization Flow & App-based Troubleshooting Fix and Verify in Realtime Eliminates manual building of WANs Automated SD-WAN orchestration Centralized hybrid WAN management Quick config updates and IOS upgrades Leverages onepk and REST APIs

88 LiveAction 4.1 PfR path change visualization Alert and report on PfR Out of Policy events Click on Alerts to get the details

89 LiveAction 4.1 PfRv3 TCA Alerts on DC1 Drill down to site pairs

90 LiveAction 4.1 PfRv3 Dashboard All Alerts Alerts / performance by Site Alerts / performance by Application Group

91 LiveAction 4.1 PfRv3 Dashboard Alerts / performance by Service Provider

92 Cisco Prime IWAN Automation and Orchestration Evolution Prime Capacity Planning, Troubleshooting, Change control Traditional Management Systems Cisco IWAN Apps IWAN Transport PKI Automation Mid CY2015 PnP Provisionin g Partners (future) Apps Security Intelligent Path Control Application Experience Evolution REST APIs APIC-EM Services (Partial) PKI Svc NetFlow Svc Network Svc Events Svc Inventory Svc PnP Svc APIC- EM OnePK/OpenFlow Device Abstraction Layer CLI

93 Key Takeaways

94 Performance Routing Phases Summary PfR/OER version 1 IOS 12.3(8)T, XE PfR version 2 IOS 15.2(3)T, IOS-XE 3.6 PfR version 3 IOS 15.4(3)M IOS-XE 3.13 PfR version 3 IOS 15.5(1)T IOS-XE 3.14 PfR version 3 IOS 15.5(2)T IOS-XE 3.15 Per Device provisioning Passive monitoring with Traditional NetFlow (TNF) Active monitoring with IP SLA Manual provisioning jitter probes 1000 s lines of configuration (pfr-map per site) Per Device provisioning Target Discovery (TD) Automatic provisioning of jitter probes Passive monitoring with Traditional NetFlow (TNF) Active monitoring with IP SLA 10 s lines of configuration PfR Domain One touch provisioning Auto Discovery of sites NBAR2 support Passive Monitoring (performance monitor) Smart Probing VRF Awareness IPv4/IPv6 (Future) <10 lines of configuration and centralized Zero SLA WCCP Support Transit Sites Multiple Next Hop per DMVPN Multiple POPs Syslog (TCA) Show last 5 TCA Blackout 6 seconds Brownout 9 seconds Limited scalability due to provisioning (~ tens of sites) Blackout 6 seconds Brownout 9 seconds Scale 500 sites Blackout ~ sub second Brownout ~ 2 sec Scale 2000 sites

95 Performance Routing Platform Support Cisco CSR-1000 Cisco ASR-1000 MC * Cisco ISR G2 family 3900-AX 2900-AX 1900-AX 890 MC Cisco ISR MC MC * support coming

96 Key Takeaways IWAN Intelligent Path Control pillar is based upon Performance Routing (PfR) Maximizes WAN bandwidth utilization Protects applications from performance degradation Enables the Internet as a viable WAN transport Provides multisite coordination to simplify network wide provisioning. Application-based policy driven framework and is tightly integrated with existing AVC components. Smart and Scalable multi-sites solution to enforce application SLAs while optimizing network resources utilization. PfRv3 is the 3 rd generation Multi-Site aware Bandwidth and Path Control/Optimization solution for WAN/Cloud based applications.available on ASR1k, ISR 4xxx, ISR-G2

97 Some useful links Cisco.com IWAN/PfR Page: PfRv3 Home Page LiveAction: The New LiveAction 4.1: LiveAction PfRv3 Demo: 105

98 Q & A 106

99