Using IPsec with Multiservices MICs on MX Series Routers

Using IPsec with Multiservices MICs on MX Series Routers Test Case April 2017 Version 1.0

Juniper Networks, Inc. 1133 Innovation Way Sunnyvale, California 94089 USA 408-745-2000 www.juniper.net Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice. The information in this document is current as of the date on the title page. Copyright 2017, Juniper Networks, Inc. All rights reserved. ii Copyright 2017, Juniper Networks, Inc.

Contents Overview... 1 Test Case Highlights... 2 Key Configuration Elements... 3 Configuring IPsec Tunnels... 3 Configuring the VRF Instance... 3 Results and Output from SFO01... 4 Results with IPsec Tunnels Using Physical IP Addresses... 7 Results with IPsec Tunnels Established Using Loopback Addresses... 8 Copyright 2017, Juniper Networks, Inc. iii

Overview This test case is for an IPsec encryption and multiservices MIC solution running on provider edge (PE) devices. The physical topology (shown in Figure 1) includes the following: MX240 devices running Junos OS Release 15.1F5-S6 Other switches used as customer edge (CE) devices, such as QFX5100 devices The Ixia product used to simulate BGP peers and generate traffic Figure 1 Physical Topology Figure 2 shows a local topology. MX Series routers are used as MPLS PE routers to carry traffic. In this example, a customer wants to secure critical traffic using IPsec VPN while retaining the current MPLS Layer 3 VPN environment already in place. There are two sites (SFO and NYC) and each one is multihomed for redundancy purposes. On each router, there are two virtual routing and forwarding (VRFs) instances; one for the legacy Layer 3 VPN environment where traffic is nonencrypted, and one where IPsec tunnels are used to transport the encrypted traffic. Each IPsec VRF is fully meshed with other IPsec VRFs in the network. For the test case shown in Figure 1, there are three sites (SFO, NYC, and ATL) with a total of four IPsec tunnels configured on each router. For example at location SFO, SFO01 has one IPsec tunnel connected to NYC01, and one IPsec tunnel connected to NYC02; likewise, SFO01 has one IPsec tunnel connected to ATL01, and one IPsec tunnel connected to ATL02. Copyright 2017, Juniper Networks, Inc. 1

Figure 2 shows a sample local topology using just the two sites SFO and NYC, where EBGP runs on top of the IPsec tunnels. Depending on the failure scenario, this ensures that once the EBGP peer is no longer reachable through the IPsec tunnel, traffic is diverted to the second IPsec tunnel. Figure 2 Local Topology Notes: The main goal for this test case is to have resiliency, and measure convergence time upon different failure scenarios. Scaling and throughput are not part of this test case. Test Case Highlights The following are the test case feature highlights: Deployment Two IPsec tunnel deployment methods were tested: one where the IPsec tunnels terminated on a physical interface, and the other used the loopback addresses. Configuration First, base configurations were used, then the IPsec environment was added on top of the configurations, and finally, EBGP was run on top of the IPsec tunnels. High Availability Load-balancing policy and BGP multipath were configured allowing traffic to be load-balanced equally between each IPsec tunnel from the local site to the remote sites. Failover Testing End-users impact was recorded for the following failover test cases: link failures, card failures, and ingress router failures. 2 Copyright 2017, Juniper Networks, Inc.

Key Configuration Elements Configuring IPsec Tunnels This section configures an IPsec tunnel between PE1 (SFO01) and PE2 (NYC01). You can adjust and reuse the configuration to create the other IPsec tunnels. 1. To configure the appropriate package, enter: set chassis fpc X pic Y adaptive-services service-package layer-3 2. To configure the ms-interface, enter: set interfaces ms-2/0/0 unit 1 family inet set interfaces ms-2/0/0 unit 1 service-domain inside set interfaces ms-2/0/0 unit 2 family inet set interfaces ms-2/0/0 unit 2 service-domain outside 3. To configure the service sets, enter: set services service-set sset1 next-hop-service inside-service-interface ms-2/0/0.1 set services service-set sset1 next-hop-service outside-service-interface ms-2/0/0.2 4. To configure the IPsec tunnels, enter: set services ipsec-vpn rule SFO01-NYC01 term 1 from source-address 0.0.0.0/0 set services ipsec-vpn rule SFO01-NYC01 term 1 from destination-address 0.0.0.0/0 set services ipsec-vpn rule SFO01-NYC01 term 1 then remote-gateway 10.6.6.6 set services ipsec-vpn rule SFO01-NYC01 term 1 then dynamic ike-policy ike_policy set services ipsec-vpn rule SFO01-NYC01 term 1 then dynamic ipsec-policy ipsec_policy set services ipsec-vpn rule SFO01-NYC01 term 1 then tunnel-mtu 9192 set services ipsec-vpn rule SFO01-NYC01 term 1 then anti-replay-window-size 4096 set services ipsec-vpn rule SFO01-NYC01 match-direction input set services ipsec-vpn ipsec proposal ipsec_proposal protocol esp set services ipsec-vpn ipsec proposal ipsec_proposal authentication-algorithm hmac-sha1-96 set services ipsec-vpn ipsec proposal ipsec_proposal encryption-algorithm aes-192-cbc set services ipsec-vpn ipsec policy ipsec_policy proposals ipsec_proposal set services ipsec-vpn ike proposal ike_proposal authentication-method pre-shared-keys set services ipsec-vpn ike proposal ike_proposal dh-group group14 set services ipsec-vpn ike proposal ike_proposal authentication-algorithm sha-256 set services ipsec-vpn ike proposal ike_proposal encryption-algorithm aes-256-cbc set services ipsec-vpn ike policy ike_policy mode main set services ipsec-vpn ike policy ike_policy proposals ike_proposal set services ipsec-vpn ike policy ike_policy pre-shared-key ascii-text "$ABC123" Configuring the VRF Instance This section configures the VRF on PE1 (SFO01) to support the PE1-PE2 (SFO01-NYC01) and PE1-PE3 (SFO01-NYC02) connections. You can adjust and reuse the configuration to create the other VRFs. To configure the vrf routing-instance, enter: set routing-instances vrf-ipsec instance-type vrf set routing-instances vrf-ipsec interface xe-1/1/4.1 set routing-instances vrf-ipsec interface ms-2/0/0.1 set routing-instances vrf-ipsec interface ms-2/0/0.3 set routing-instances vrf-ipsec interface lo0.1 set routing-instances vrf-ipsec route-distinguisher 64518:100 set routing-instances vrf-ipsec vrf-import import set routing-instances vrf-ipsec vrf-export block-static set routing-instances vrf-ipsec vrf-export no-100-exp set routing-instances vrf-ipsec vrf-table-label set routing-instances vrf-ipsec routing-options static route 10.66.66.66/32 next-hop ms- 2/0/0.1 set routing-instances vrf-ipsec routing-options static route 10.55.55.55/32 next-hop ms- 2/0/0.3 set routing-instances vrf-ipsec routing-options autonomous-system 64515 Copyright 2017, Juniper Networks, Inc. 3

set routing-instances vrf-ipsec protocols bgp group external-ce type external set routing-instances vrf-ipsec protocols bgp group external-ce local-address 192.168.100.1 set routing-instances vrf-ipsec protocols bgp group external-ce export static-to-bgp set routing-instances vrf-ipsec protocols bgp group external-ce peer-as 64513 set routing-instances vrf-ipsec protocols bgp group external-ce neighbor 192.168.100.2 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 type external set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 multihop set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 local-address 10.111.111.111 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 family inet unicast loops 2 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 peer-as 64514 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 local-as 64515 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 multipath multiple-as set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 neighbor 10.66.66.66 import test-import set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 neighbor 10.66.66.66 export no-advertise-200 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 neighbor 10.66.66.66 bfdliveness-detection minimum-interval 100 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc01 neighbor 10.66.66.66 bfdliveness-detection multiplier 3 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 type external set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 multihop set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 local-address 10.111.111.111 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 family inet unicast loops 2 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 peer-as 64516 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 local-as 64515 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 multipath multiple-as set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 neighbor 10.55.55.55 export no-advertise-200 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 neighbor 10.55.55.55 bfdliveness-detection minimum-interval 200 set routing-instances vrf-ipsec protocols bgp group ebgp-nyc02 neighbor 10.55.55.55 bfdliveness-detection multiplier 3 Results and Output from SFO01 The IPsec tunnels from SFO01 to both routers on the NYC site (the first tunnel uses loopback IP addresses, as configured above; the second tunnel uses physical interface IP addresses, not shown above): user@sfo01-re0# run show services ipsec-vpn ipsec security-associations Service set: SFO01-NYC01, IKE Routing-instance: default Rule: SFO01-NYC01, Term: 1, Tunnel index: 5 Local gateway: 10.1.1.1, Remote gateway: 10.6.6.6 <<< using loopback addresses to establish IPsec tunnels IPsec inside interface: ms-2/0/0.1, Tunnel MTU: 9192 Direction SPI AUX-SPI Mode Type Protocol inbound 1831896891 0 tunnel dynamic ESP outbound 3149752866 0 tunnel dynamic ESP Service set: SFO01-NYC02, IKE Routing-instance: default Rule: SFO01-NYC02, Term: 1, Tunnel index: 4 Local gateway: 10.0.11.1, Remote gateway: 10.0.11.2 <<< using physical interface IP IPsec inside interface: ms-2/0/0.3, Tunnel MTU: 9192 Direction SPI AUX-SPI Mode Type Protocol inbound 3102470741 0 tunnel dynamic ESP outbound 1430524634 0 tunnel dynamic ESP 4 Copyright 2017, Juniper Networks, Inc.

The route towards the remote EBGP peers reachable through the ms- interface: user@sfo01-re0# run show route 10.66.66.66 vrf-ipsec.inet.0: 17 destinations, 88 routes (17 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 10.66.66.66/32 *[Static/5] 1d 20:42:38 > via ms-2/0/0.1 user@sfo01-re0# run show route 10.55.55.55 vrf-ipsec.inet.0: 17 destinations, 88 routes (17 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 10.55.55.55/32 *[Static/5] 1d 20:53:29 > via ms-2/0/0.3 Both EBGP sessions are up: user@sfo01-re0# run show bgp neighbor 10.66.66.66 Peer: 10.66.66.66+62532 AS 64514 Local: 10.111.111.111+179 AS 64515 Group: ebgp-nyc01 Routing-Instance: vrf-ipsec Forwarding routing-instance: vrf-ipsec Type: External State: Established Flags: <Sync RSync> Last State: EstabSync Last Event: RecvKeepAlive Last Error: None Export: [ no-advertise-200 ] Options: <Multihop Preference LocalAddress AddressFamily PeerAS Multipath LocalAS Refresh> Options: <MultipathAs BfdEnabled PeerSpecficLoopsAllowed> Address families configured: inet-unicast Local Address: 10.111.111.111 Holdtime: 90 Preference: 170 Local AS: 64515 Local System AS: 64518 Number of flaps: 11 Last flap event: BfdDown Peer ID: 10.66.66.66 Local ID: 10.111.111.111 Active Holdtime: 90 Keepalive Interval: 30 Group index: 2 Peer index: 0 BFD: enabled, up NLRI for restart configured on peer: inet-unicast NLRI advertised by peer: inet-unicast NLRI for this session: inet-unicast Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: inet-unicast NLRI of received end-of-rib markers: inet-unicast NLRI of all end-of-rib markers sent: inet-unicast Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 64514) Peer does not support Addpath Table vrf-ipsec.inet.0 Bit: 20002 RIB State: BGP restart is complete RIB State: VPN restart is complete Send state: in sync Active prefixes: 1 Received prefixes: 14 Accepted prefixes: 14 Suppressed due to damping: 0 Advertised prefixes: 14 Copyright 2017, Juniper Networks, Inc. 5

Last traffic (seconds): Received 2 Sent 20 Checked 47 Input messages: Total 3737 Updates 37 Refreshes 0 Octets 72376 Output messages: Total 3724 Updates 30 Refreshes 0 Octets 71957 Output Queue[1]: 0 (vrf-ipsec.inet.0, inet-unicast) user@sfo01-re0# run show bgp summary Groups: 7 Peers: 11 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending bgp.l3vpn.0 29 29 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State #Active/Received/Accepted/Damped... < > 10.55.55.55 64516 3715 3689 0 7 1d 4:21:32 Establ vrf-ipsec.inet.0: 1/14/14/0 10.66.66.66 64514 3750 3738 0 11 1d 4:33:58 Establ vrf-ipsec.inet.0: 1/14/14/0 The 10.210.210/24 subnet behind the NYC site is learned by SFO01 through the two EBGP neighbors, and both nexthops are installed on SFO01 (load-balancing the traffic): user@sfo01-re0> show route 10.210.210/24 vrf-ipsec.inet.0: 17 destinations, 88 routes (17 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 10.210.210.0/24 *[BGP/170] 1d 07:33:38, localpref 100, from 10.66.66.66 AS path: 64514 64519 I, validation-state: unverified > via ms-2/0/0.3 via ms-2/0/0.1 [BGP/170] 1d 07:33:38, localpref 100, from 10.55.55.55 AS path: 64516 64519 I, validation-state: unverified > via ms-2/0/0.3 user@sfo01-re0> show route forwarding-table destination 10.210.210/24 extensive Routing table: default.inet [Index 0] Internet: Routing table: vrf-ipsec.inet [Index 4] Internet: Destination: 10.210.210.0/24 Route type: user Route reference: 0 Route interface-index: 0 Multicast RPF nh index: 0 Flags: sent to PFE Next-hop type: unilist Index: 1048586 Reference: 1 Next-hop type: indirect Index: 1048579 Reference: 2 Weight: 0x0 Next-hop type: unicast Index: 693 Reference: 4 Next-hop interface: ms-2/0/0.3 Weight: 0x0 Next-hop type: indirect Index: 1048580 Reference: 2 Weight: 0x0 Next-hop type: unicast Index: 692 Reference: 4 Next-hop interface: ms-2/0/0.1 Weight: 0x0 Test gear was used to measure the convergence times observed on the SFO01 device under test (DUT). 6 Copyright 2017, Juniper Networks, Inc.

Results with IPsec Tunnels Using Physical IP Addresses The following failure scenarios show the results of end-to-end re-convergence testing with IPsec tunnels established using physical IP addresses. The Packet Loss Duration (ms) column indicates the amount of traffic loss and the speed of convergence. On SFO01: Scenario: Cable pull xe-1/1/1 (SFO01-NYC02) Failover: On PE, using SFO01-NYC01 Scenario: Disabling both xe-1/1/0 (SFO01-NYC01) and xe-1/1/1 (SFO01-NYC02) through the CLI Failover: At CE, using SFO02-NYC0x Scenario: Removing the MS-MIC Failover: At CE, using SFO02-NYC0x Copyright 2017, Juniper Networks, Inc. 7

Scenario: Simulating power failure (pulling power cable) Failover: At CE, using SFO02-NYC0x Results with IPsec Tunnels Established Using Loopback Addresses The following failure scenarios show the results of end-to-end re-convergence testing with IPsec tunnels established using loopback addresses (which yielded slightly better results). The Packet Loss Duration (ms) column indicates the amount of traffic loss and the speed of convergence. On SFO01: Scenario: Disabling xe-1/1/0 (SFO01-NYC01) through the CLI Failover: On PE, using SFO01-NYC02 Scenario: Disabling both xe-1/1/0 (SFO01-NYC01) and xe-1/1/1 (SFO01-NYC02) through the CLI Failover: At CE, using SFO02-NYC0x 8 Copyright 2017, Juniper Networks, Inc.