K15344: Troubleshooting the IPsec tunnel between two BIG-IP AFM systems

Transcription

1 K15344: Troubleshooting the IPsec tunnel between two BIG-IP AFM systems Diagnostic Original Publication Date: Jun 25, 2014 Update Date: Jan 8, 2016 Issue You should consider using this procedure under any of the following conditions: Symptoms You are unable to bring up the IPsec tunnel between two BIG-IP AFM systems. Client traffic cannot traverse between two private networks through the IPsec tunnel. As a result of IPsec tunnel failing to properly establish between two BIG-IP AFM systems, you may encounter the following symptoms: The private network clients are unable to reach each other through the IPsec tunnel between the BIG- IP AFM systems. The private network clients sending Internet Control Message Protocol (ICMP) request packets receive ICMP unreachable status in the response packets. The system is still sending an ICMP unreachable status, even though the ISAKMP and ESP rules are configured on the BIG-IP AFM system. The IPsec tunnel fails to negotiate past phase one. The IPsec tunnel fails to complete phase two. The IPsec tunnel is up, but packets are not traversing the tunnel. Recommended Actions Configuring and establishing an IPsec tunnel between two BIG-IP AFM systems is similar to other BIG-IP systems. The additional step to configure a BIG-IP AFM system to support the IPsec tunnel is the deployment of firewall rules in the following contexts: Global Accept decisively incoming IPsec Encapsulating Security Payload (ESP) and Internet Security Association and Key Management Protocol (ISAKMP) traffic from each BIG-IP AFM system. Virtual server Optional granular control over the types of traffic allowed between the private networks that reside behind each BIG-IP AFM system.

2 The first step of troubleshooting a failed IPsec tunnel is to verify if the BIG-IP AFM system is configured with the necessary configuration, such as the following: Public Virtual Local Area Networks (VLANs) and IP addresses for both BIG-IP AFM systems to reach each other over WAN Private VLANs and IP addresses for the private networks behind each BIG-IP AFM system Default route or a specific route on the BIG-IP AFM system to reach the peer system, as well as the remote private network IPsec configuration, such as IKE Peers, IPsec Policies, and IPsec Traffic Selectors IP forwarding virtual servers for handling the unencapsulated traffic egressing from the IPsec tunnel between the two private networks Global firewall rules on both BIG-IP AFM systems to accept ESP and ISAKMP traffic The following illustration is an example IPsec setup based on this article: Note: The VLANs and the private IP addresses used in this example are for illustration purposes. Network Site-1 Site-2 Objects Client BIG-IP AFM Gateway W Gateway BIG-IP AFM Client VLAN A N IP Address / /24 /24 /28 /28 /28 /24 /24 Virtual Server --> ipsecencap ipsecdecap <-- --> ipsecdecap ipsec-encap <-- In this illustration, note the following: The gateways between Site-1 and Site-2 are routed over a WAN, for example the Internet. The gateway is directly connected to the BIG-IP AFM system on VLAN81 (public VLAN) for each site. The client is directly connected to the BIG-IP AFM system on VLAN1555 (private VLAN) for each site. The IPsec tunnel is configured to establish from the BIG-IP AFM system in Site-1 to the BIG-IP AFM system in Site-2, and the other way around, so that the traffic from clients on both sides traverses through this IPsec tunnel. If you have verified that the configuration on your BIG-IP AFM systems is valid for establishing an IPsec tunnel, refer to the Procedures section for more troubleshooting steps. Procedures When experiencing IPsec tunnel issues, you can determine the root cause by using the following troubleshooting steps: Verifying the status of the IPsec tunnel Reviewing the log entries for the IPsec tunnel Increasing the log level for the IKE daemon Re-establishing an IPsec SA

3 Verifying the firewall rules by performing a network trace capture Verifying the status of the IPsec tunnel You should first check if the ISAKMP security associations (SAs) exist on both BIG-IP AFM systems and if their cookies match. You should then check the IPsec (phase 2) status on both systems before verifying if the IPsec SAs exist on both BIG-IP AFM systems, and if their SPI values match. To do so, perform the following procedure on each BIG-IP AFM system: Impact of procedure: Performing the following procedure should not have a negative impact on your system Log in to the BIG-IP AFM command line. Display the ISAKMP SAs that are established on the system by typing the following command: 3. racoonctl -ll show-sa isakmp Display the IPsec SAs that are established on the system by typing the following command: racoonctl -ll show-sa ipsec BIG-IP and later Note: Prior to BIG-IP , the IPsec SAs may use localhost IP addresses in the /8 subnet. Beginning in BIG-IP , you should not see localhost IP addresses when viewing the IPsec SAs. However, the raccoon process may still log messages to the /var/log/racoon.log file using localhost addresses. Alternatively, in BIG-IP and later, you can display all established IPsec SAs by typing the following tmsh command: tmsh show net ipsec ipsec-sa all-properties Or if you want to filter the results based on a desired IPsec traffic selector, you can do so by using the following command syntax: 4. tmsh show net ipsec ipsec-sa traffic-selector <name of traffic selector> all-properties Repeat these steps on the remote BIG-IP AFM system. If the Cookies value from the ISAKMP SAs on both systems match, the ISAKMP has been successfully negotiated. If they do not match, you may want to review your IPsec configuration and/or the global firewall rules, as well as the upstream devices between the two BIG-IP systems. The following command output example shows a successful ISAKMP negotiation: Site-1 BIG-IP AFM system output Source Destination Cookies ST S V E Created Phase c0d38cf62807e78:3c87ca0118c8fe92 9 R 10 M : 52:40 1

4 Site-2 BIG-IP AFM system output Source Destination Cookies ST S V E Created Phase c0d38cf62807e78:3c87ca0118c8fe92 9 I 10 M : 54:29 1 Where: Column Possible Values Description 1 start phase 1 negotiation 2 msg 1 received 3 msg 1 sent ST (Tunnel Status) 4 msg 2 received 5 msg 2 sent 6 msg 3 received 7 msg 3 sent 8 msg 4 received 9 ISAKMP tunnel established 10 ISAKMP tunnel expired S I R Initiator Responder V <n> Version Number. 10 (0x10) means ISAKMP version 1.0 E (Exchange Type) B M U A I Base Identity Protection Authentication Only Aggressive Informational Phase 2 <n> Number of phase 2 tunnels negotiated If the SPI value from the IPsec SAs on both systems match, the IPsec tunnel has been successfully established. If they do not match, you may want to review your IPsec configuration and/or the global firewall rules, as well as the upstream devices between the two BIG-IP systems. For each IPsec tunnel, there are two entries to facilitate bi-directional communication. The SPI value for one BIG-IP AFM system matches the SPI value on the other BIG-IP AFM system for each direction. The following racoonctl -ll show-sa ipsec command output example from the Site-1 and Site-2 systems show matching values for each SPI endpoint, and a successful IPsec tunnel: Site-1 BIG-IP AFM system output [500] [500] esp-udp mode=tunnel spi= (0xe35dc3d9) reqid=13946(0x a) E: 20 cfe0bda3 9a cc319 6f08a063 3db5a5f1 seq=0x replay=64 flags=0x state=mature created: Jun 16 22:52: current: Jun 16 22:53:

5 diff: 23(s) hard: 1800(s) soft: 1440(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0 hard: 0 soft: 0 sadb_seq=1 pid=14996 refcnt= [500] [500] esp-udp mode=tunnel spi= (0x27cd3c4a) reqid=13945(0x ) E: 20 da9ecd b0 e8e8ae6f 32cbb90d effad45b seq=0x replay=64 flags=0x state=mature created: Jun 16 22:52: current: Jun 16 22:53: diff: 23(s) hard: 1800(s) soft: 1440(s) last: Jun 16 22:53: hard: 0(s) soft: 0(s) current: 3174(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 23 hard: 0 soft: 0 sadb_seq=0 pid=14996 refcnt=768 Site-2 BIG-IP AFM system output [500] [500] esp-udp mode=tunnel spi= (0x27cd3c4a) reqid=13950(0x e) E: 20 da9ecd b0 e8e8ae6f 32cbb90d effad45b seq=0x replay=64 flags=0x state=mature created: Jun 16 22:54: current: Jun 16 22:54: diff: 25(s) hard: 1800(s) soft: 1440(s) last: Jun 16 22:54: hard: 0(s) soft: 0(s) current: 3150(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 25 hard: 0 soft: 0 sadb_seq=1 pid=16382 refcnt= [500] [500] esp-udp mode=tunnel spi= (0xe35dc3d9) reqid=13949(0x d) E: 20 cfe0bda3 9a cc319 6f08a063 3db5a5f1 seq=0x replay=64 flags=0x state=mature created: Jun 16 22:54: current: Jun 16 22:54: diff: 25(s) hard: 1800(s) soft: 1440(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0 hard: 0 soft: 0 sadb_seq=0 pid=16382 refcnt=768 Similarly, in BIG-IP and later, if you use the tmsh show net ipsec ipsec-sa all-properties command to display the IPsec SAs, the Site-1 and Site-2 systems displays the following command output examples: Site-1 BIG-IP AFM system tmsh output IPsec::SecurityAssociations > tmm: 3

6 Direction: in; SPI: 0x27cd3c4a( ); Policy ID: 0x3679(13945) Protocol: esp; Mode: tunnel; State: mature Authenticated Encryption : aes-gcm128 Current Usage: 3450 bytes Hard lifetime: 1776 seconds; unlimited bytes Soft lifetime: 1416 seconds; unlimited bytes Replay window size: 64 Last use: 06/16/2014:22:53 Create: 06/16/2014:22: > tmm: 0 Direction: out; SPI: 0xe35dc3d9( ); Policy ID: 0x367a(13946) Protocol: esp; Mode: tunnel; State: mature Authenticated Encryption : aes-gcm128 Current Usage: 3150 bytes Hard lifetime: 1776 seconds; unlimited bytes Soft lifetime: 1416 seconds; unlimited bytes Replay window size: 64 Last use: 06/16/2014:22:53 Create: 06/16/2014:22:52 Total records returned: 2 Site-2 BIG-IP AFM system tmsh output IPsec::SecurityAssociations > tmm: 3 Direction: out; SPI: 0x27cd3c4a( ); Policy ID: 0x367e(13950) Protocol: esp; Mode: tunnel; State: mature Authenticated Encryption : aes-gcm128 Current Usage: 3402 bytes Hard lifetime: 1774 seconds; unlimited bytes Soft lifetime: 1414 seconds; unlimited bytes Replay window size: 64 Last use: 06/16/2014:22:54 Create: 06/16/2014:22: > tmm: 0 Direction: in; SPI: 0xe35dc3d9( ); Policy ID: 0x367d(13949) Protocol: esp; Mode: tunnel; State: mature Authenticated Encryption : aes-gcm128 Current Usage: 3726 bytes Hard lifetime: 1774 seconds; unlimited bytes Soft lifetime: 1414 seconds; unlimited bytes

7 Replay window size: 64 Last use: 06/16/2014:22:54 Create: 06/16/2014:22:54 Total records returned: 2 Procedure back out: None Reviewing the log entries for the IPsec tunnel The racoon daemon handles Internet Key Exchange (IKE) for IPsec and logs to the /var/log/racoon.log file. The following log entries from both systems describe how the IPsec tunnel is established when a client in Site-2 private network attempts to connect to a client in Site-1 private network: Impact of procedure: Performing the following procedure should not have a negative impact on your system. Site-1 BIG-IP system :28:19: INFO: respond new phase 1 negotiation: [500]<=> [500] :28:19: INFO: begin Identity Protection mode :28:19: INFO: received Vendor ID: DPD :28:19: INFO: Send INITIAL-CONTACT to : [500] :28:19: INFO: ISAKMP-SA established [500] [500] spi:d52aa488de332400: 6fbd9560b184dfd :28:21: INFO: respond new phase 2 negotiation: [500]<=> [500] :28:21: INFO: best sp match: /24[0] /24[0] proto=any dir=in :28:21: INFO: best sp match: /24[0] /24[0] proto=any dir=out :28:21: INFO: IPsec-SA established: ESP/Tunnel [0]-> [0] spi= (0x5f2301ea) :28:21: INFO: IPsec-SA established: ESP/Tunnel [0]-> [0] spi= (0x8d285288) Site-2 BIG-IP system :30:06: INFO: IPsec-SA request for queued due to no phase1 found :30:06: INFO: initiate new phase 1 negotiation: [500]<=> [500] :30:06: INFO: begin Identity Protection mode :30:07: INFO: received Vendor ID: DPD :30:07: INFO: Send INITIAL-CONTACT to : [500] :30:07: INFO: ISAKMP-SA established [500] [500] spi:d52aa488de332400: 6fbd9560b184dfd :30:08: INFO: initiate new phase 2 negotiation: [500]<=> [500] :30:09: INFO: IPsec-SA established: ESP/Tunnel [500]-> [500] spi= (0x8d285288) :30:09: INFO: IPsec-SA established: ESP/Tunnel [500]-> [500] spi= (0x5f2301ea) The following table provides an explanation for the process of how an IPsec tunnel is established: Site Log entry Description

8 2 IPsec-SA request for A packet matched an IPSec selector but there was no tunnel for it to queued due to no phase1 found. pass over. The packet has been queued while the tunnel is brought up. 2 initiate new phase 1 negotiation: [500]<=> [500] Starting ISAKMP with the IKE peer respond new phase 1 negotiation: [500]<=> [500] Site-1 BIG-IP system uses the localip when the first phase 1 (ISAKMP) packet arrived. Site-2 BIG-IP system is the responder while Site-1 BIG-IP system is the initiator. Both received Vendor ID: DPD Dead Peer Detection RFC3706. This is not mandatory, but both devices must support and offer the option in order for DPD to function. If one device does not support DPD, then DPD will not be used ISAKMP-SA established [500] [500] spi: d52aa488de332400: 6fbd9560b184dfd2 ISAKMP-SA established [500] [500] spi: d52aa488de332400: 6fbd9560b184dfd2 initiate new phase 2 negotiation: [500]<=> [500] best sp match: /24[0] /24[0] proto=any dir=in best sp match: /24[0] /24[0] proto=any dir=out IPsec-SA established: ESP /Tunnel [500]-> [500] spi= (0x8d285288) IPsec-SA established: ESP /Tunnel [500]-> [500] spi= (0x5f2301ea) Phase 1 is over. The ISAKMP exchange is complete and an ISAKMP Security Association (SA) now exists. The SPI matches on both units and Site-2 BIG-IP system is still referring to itself by the local IP. The BIG-IP systems now move to phase 2 (IPSec). The responder (Site-2 system) has found an IPSec policy that matches the security association criteria that the initiator (Site-1 system) has asked for. The tunnel is up. There are SPI values that also match across devices for the IPSec tunnel. Two SAs are created for two-way communication. Procedure back out: None Increasing the log level for the IKE daemon You may be able to determine potential issues of IPsec failing to establish by increasing the log level for the IKE daemon and reviewing the debug logs. To do so, perform the following procedure:

9 Impact of procedure: Increasing the log level may affect the performance of an overloaded BIG-IP system. You should only increase the log level when advised by F5 Support. F5 recommends reverting the log level back to the default (info) when the troubleshooting session is completed Log in to the Traffic Management Shell ( tmsh). Increase the log level for the IKE daemon by typing the following command: modify /net ipsec ike-daemon ikedaemon log-level debug Note: Changing the IKE daemon log level will reset the IPsec tunnels. Initiate IPsec traffic and review the log entries in the /var/log/racoon.log file. Ensure you proceed to the next step when you are done reviewing the log entries. When you are done reviewing the log entries, revert the log level back to the default level of info by typing the following command: 5. modify /net ipsec ike-daemon ikedaemon log-level info Save the change by typing the following command: save /sys config Procedure back out: tmsh modify /net ipsec ike-daemon ikedaemon log-level info Re-establishing an IPsec SA If traffic is not flowing between the two private networks, you can disable and re-enable an IKE peer on each BIG-IP system. By disabling an IKE peer, the ISAKMP SA as well as the IPSec SAs that are built on top of it will be torn down together. When the IKE peer is re-enabled on each system and the private network client starts sending traffic, both the ISAKMP SA and the IPSec SAs need to be re-negotiated so that the IPSec tunnel can be re-established between the two systems. In BIG-IP and later, you can now selectively delete specific IPSec SA only instead of disabling and re-enabling the IKE peer. To reestablished the IPSec tunnel between the two systems, the private network client will need to start sending traffic to the remote private network client. To re-establish an IPSec SA, perform one of the following procedures: Deleting the IPsec SA (BIG-IP and later) Disabling and re-enabling the IKE peer (BIG-IP and later) Impact of procedure: All traffic traversing through the IPsec tunnel may be momentarily disrupted until the IPsec tunnel is re-established. Deleting the IPsec SA (BIG-IP and later) Log in to the Traffic Management Shell ( tmsh). Delete the specific IPsec SA by using the following command syntax: delete /net ipsec ipsec-sa src-addr <source address> dst-addr <destination address>

10 For example, to delete IPsec SA between and , you would type the following command: delete /net ipsec ipsec-sa src-addr dst-addr Repeat the previous steps on the remote BIG-IP system, if necessary. Re-establish the IPsec tunnel by sending traffic from a client in the source private network to a client in the destination private network. Disabling and re-enabling the IKE peer (BIG-IP and later) Log in to the Traffic Management Shell ( tmsh). Disable the IKE peer by using the following command syntax: modify /net ipsec ike-peer <name of IKE peer> state <disabled enabled> For example, to disable site1-ikepeer, you would type the following command: 3. modify /net ipsec ike-peer site1-ikepeer state disabled Re-enable the IKE peer using the command syntax from the previous step, but replacing disabled with enabled. For example, to re-enable site1-ikepeer, you would type the following command: modify /net ipsec ike-peer site1-ikepeer state enabled Repeat the above steps on the remote BIG-IP system if necessary. Re-establish the IPsec tunnel by sending traffic from a client in the source private network to a client in the destination private network. Procedure back out: None Verifying the firewall rules by performing a network trace capture If the IPsec tunnel is verified to have established, but traffic between the two private networks continues to fail, you may need to configure the appropriate firewall rules on the virtual servers handling the unencapsulated traffic. You can verify this by performing a network trace capture for the affected traffic on both BIG-IP AFM systems while the private network client continues to attempt sending traffic. Impact of procedure: Depending on the traffic load, your system may experience slight performance degradation temporarily while performing network trace captures. For example, private network client sends ICMP echo packets to private network client while performing a network trace capture on both BIG-IP AFM systems using the following tcpdump command: tcpdump -ni 0.0 icmp and host and host Virtual servers without appropriate firewall rules

11 If the virtual servers are not configured with the appropriate firewall rule(s), the tcpdump capture may display traces similar to the following example: Site-1 BIG-IP AFM system output 20:39: IP > : ICMP net unreachable - admin prohibited, length 36 Site-2 BIG-IP AFM system output 20:40: IP > : ICMP net unreachable - admin prohibited, length 36 Virtual servers with properly configured firewall rules When the appropriate firewall rule(s) are configured on the virtual servers, the tcpdump capture may display traces similar to the following example: Site-1 BIG-IP AFM system output 04:12: IP > : ICMP echo request, id 20141, seq 1, length 64 04:12: IP > : ICMP echo reply, id 20141, seq 1, length 64 04:12: IP > : ICMP echo reply, id 20141, seq 1, length 64 The first entry in the example trace above (Site-1) shows the ICMP echo request ingresses the local BIG-IP AFM system through its private network interface, but does not egress the local BIG-IP AFM system through its public network interface. The ICMP echo request is encapsulated in an ESP packet before egressing the local BIG-IP AFM system through its public network interface. Similarly, on the remote BIG-IP AFM system, no ICMP echo request packet appears to ingress the remote BIG-IP AFM system through its public network interface. The ICMP echo request is encapsulated in an ESP packet when it ingresses the remote system. After the remote BIG-IP AFM system decapsulated the ESP packet, the ICMP echo request then ingresses the forwarding virtual server on the remote BIG-IP AFM system and egresses the remote BIG-IP AFM system through its private network interface. The first two entries in the example trace below (Site-2) exhibits this behavior. Site-2 BIG-IP AFM system output 04:12: IP > : ICMP echo request, id 20141, seq 1, length 64 04:12: IP > : ICMP echo request, id 20141, seq 1, length 64 04:12: IP > : ICMP echo reply, id 20141, seq 1, length 64 When the remote private network client returns the ICMP echo reply, the remote BIG-IP AFM system encapsulates it in an ESP packet before forwarding to the local BIG-IP AFM system. The local BIG-IP AFM system then proceeds to decapsulate the ESP packet and the ICMP echo reply ingresses the forwarding virtual server first before egressing the local BIG-IP AFM system via its private network interface. The last two entries in the example trace per Site-1 AFM output exhibits this behavior. If the traffic is not traversing as expected between the two private networks, you should review the firewall rules configured at the virtual server context and ensure the appropriate firewall rules are configured to allow the desired traffic between the two private networks.

12 Procedure back out: None Supplemental Information K15120: Configuring IPsec in Tunnel mode between two BIG-IP AFM systems using the tmsh utility K14137: The BIG-IP system may drop IPsec ESP response packets K14646: Multiple security associations negotiated for a single IPsec tunnel may cause intermittent traffic disruptions K15195: TMM may generate a core file during IPsec key exchange K15214: TMM CPU utilization may spike when using multiple IPsec tunnels