AppDirector Redundancy Mechanism

Transcription

1 AppDirector Redundancy Mechanism Technical Application Note North America Radware Inc. 575 Corporate Dr. Suite 205 Mahwah, NJ Tel International Radware Ltd. 22 Raoul Wallenberg St. Tel Aviv 69710, Israel Tel

2 Page Scope This document is a technical discussion of the redundancy mechanisms available for the AppDirector family of products. The document assumes previous knowledge of basic AppDirector product features, operation, and configuration. The redundancy mechanism and its supporting features discussed here apply specifically to the AppDirector products. The support of these features in other Radware products might vary. Overview AppDirector aligns server infrastructure operations with application front end requirements to eliminate traffic surges, server bottlenecks, connectivity disconnects and downtime to enable 24/7 application continuity and data center availability. To provide 24/7 availability all points of failure in the data center must be eliminated, including possibility of AppDirector failures. As such Radware recommends installing its devices in pairs, to provide fault tolerance in the case of a single device's failure. To achieve redundancy between pairs of devices, Radware recommends using Virtual Router Redundancy Protocol (VRRP). Working with VRRP enables to maintain dynamic redundancy using a logical entity called virtual router (VRRP was initially developed to provide high availability for routers, hence the name virtual router. However this protocol can be supported by a wide range of devices that are not routers as it is not a routing protocol - it does not advertise IP routes or affect the routing table in any way). With VRRP, IP Addresses are associated with the Virtual MAC Addresses that are owned by the main device, and are taken over by the backup device at fail-over time. AppDirector supports two redundancy modes: Active-Backup - A redundancy scenario involving 2 devices where one device is the designated Active device for all services (VIPs). In the event of a failure of the Active device, the Backup device temporarily assumes ownership of the VIPs. Active-Active - A redundancy scenario involving 2 devices where each device is both the Active device for some VIPs and the Backup device for other VIPs. In the event of a failure of one device, the other device temporarily assumes ownership of all VIPs. Radware also provides a proprietary redundancy method that uses Address Resolution Protocol to monitor the other device in a pair and to check its availability. The recommended redundancy mechanism for AppDirector is VRRP, however if for legacy reasons you are using the proprietary mechanism, please see Appendix 1 for details on this mechanism.

3 Page Stateful Failover Stateful failover allows a backup device to take over when a main device fails, without dropping existing sessions or breaking persistency. AppDirector supports stateful failover by mirroring the content of the tables that define session state: Client table, Session ID (persistency) table, Proximity table and DNS Persistency table. For better mirroring performance it is recommended to provide dedicated direct connection (cross-cable) between the AppDirector devices. This prevents delays and packet loss that can occur when mirroring data is sent via the network and is influenced by network load (see blue dotted line in Figure 1). Note: For increased reliability the direct connection between the devices can be implemented using a trunk (link aggregation). Physical IP Addresses versus Virtual IP Addresses Redundancy In redundancy configurations the main and the backup AppDirectors utilize both virtual and physical addresses. While the same virtual IP addresses are defined on both main and backup AppDirectors, different physical IP addresses are used for the main and backup devices. When a physical interface of the main AppDirector device is set as the default gateway of a server, when the backup AppDirector takes over, the server can work as usual because the backup AppDirector takes ownership on that IP address. The problem is that in this situation the server cannot ping its default gateway IP address, as the main device is down. To avoid this scenario, it is recommended to use virtual IP Interface addresses as default gateway of servers and other devices around AppDirector. To do that, create a virtual IP Interface address for each local subnet of AppDirector, and use this address in the relevant routing tables for hosts on that subnet. Make sure to set the same virtual IP Interface addresses as backup on the backup device.

4 Page Radware Redundancy Terms Interface Grouping Selective Interface Grouping This feature ensures that the backup device will take over even if only some of the active device physical ports are down. When this feature is enabled, a device will intentionally stop responding to ARPs on all of its ports if any of its physical ports is down, causing backup device to take over. Selective Interface Grouping allows defining which physical ports trigger interface grouping, and subsequently device failover, and which do not. When Interface grouping is triggered, it stops answering to ARP only on the ports that are included in Selective Interface Grouping. The most common example of a port that should not usually trigger device failover is management port. Note: To exclude a management port from triggering interface grouping you can either set it as Exclude in the Selective Interface Grouping table or configure it as Dedicated Management Port. Backup Interface Grouping Backup in VLAN Force Port Down Preempt Mode When this feature is enabled, it causes the device to take over only when all interfaces of the main device that are participating in redundancy are down. Respectively, it will release those interfaces only when all the main's interfaces are up. In Regular VLAN configuration, the device that is inactive can cause broadcast storms and lead to loops. Backup in VLAN, when enabled, prevent loop creation. This capability allows forcing down electrically, for a short period, physical ports belonging to a VLAN when the VLAN is disabled due to Interface Grouping activation. This allows the switches to which these ports are connected to clear their MAC tables and prevents them from continuing to send traffic to the wrong AppDirector device this problem is relevant mainly for Regular VLANs. This capability is supported only in VRRP configuration. This parameter defines the takeover procedure for a VR when the master device fails and then resumes functioning. When the master device for this VR resumes functioning, the

5 Page Preemption Mode decides whether it should retake control of the VR from the backup device (Preempt Mode set to True) or whether the backup device should maintain control of the VR (Preempt Mode set to False). Default is True. Note: The device that owns the IP address(es) associated with the VR always preempts independent of the setting of this flag. So to allow configurations where preemption is disabled, only virtual IP addresses should be associated to VR. VIP Virtual IP Interface (VIPI) The Virtual IP Address is an IP Address that is configured on a device, but not assigned to a specific physical interface. In this document we refer to VIP as virtual IP addresses that represent a service for which AppDirector provides load balancing, and do not include VIPI addresses (though both service VIPs and VIPIs are configured via the same L4 policy mechanism on the device). An IP address configured on AppDirector, which is not assigned to a physical port. This IP address is required in redundancy configurations to allow continuous access to AppDirector from any interface and to DNS resolving capability, no matter which AppDirector device is active. VRRP Introducing VRRP VRRP (Virtual Router Redundancy Protocol), defined in RFC 2338, is a standard protocol that enables dynamic device redundancy. If the main device fails, VRRP ensures that the backup device takes over and that traffic is forwarded to it. The basic concept in VRRP is that of a Virtual Router (VR). A VR has a Virtual Router Identifier (VRID) and one or more IP addresses associated with it. Each VR has a VRMAC, which is a MAC address associated with the VR. This saves the need for a MAC address update in case of a fail-over. The VRMAC address is determined by the VRID and does not need to be configured manually. Typically, the same VR is configured on multiple devices to achieve redundancy between them for the VR. Each device has a priority for a VR, and the main device for the VR is the device with the highest priority. Using VRRP, the main device constantly sends advertisements to other VRRP devices to indicate that it is online. When the advertisements stop, the main device is assumed to be inactive. A new main device is then selected for this VR; that is, the device with the next highest priority for that VR.

6 Page VRRP in Radware For a typical active-backup scenario, a VR is required for each interface of AppDirector. In a standard AppDirector setup (Figure 1), 2 VRs are required: VR-I: For the Internet side of the device. VR-S: For the server side of the device. You need to configure all VRs on each device, and associate the appropriate IP addresses with each VR. Typically, physical IP interfaces and virtual addresses are associated with VRs. However Radware recommends to associate virtual IP interfaces (VIPI) instead of physical IP interfaces. The following addresses should be associated to VRs: To the VR on the Internet side: a. VIPI for the Internet side subnet. b. VIPs. VIPs that belong to a different subnet than the Internet side physical interface do not have to be associated to the VR, unless the up-stream router has a leg on the VIPs subnet (even though AppDirector does not have such an IP interface). In case VIPs are not associated make sure the up-stream router has route configured for the VIPs using device VIPI as Next Hop Router. AppDirector allows to automatically associate VIPs to VR by enabling the VRRP Automated Configuration Updates parameter (only VIPs configured after the flag is enabled are automatically associated to VR). The VIP is associated to the VR whose Primary IP (device physical IP interface) is on the same subnet as the VIP. If no such VR is found (VR is not yet configured or VIP does not belong to the subnet on the Internet side) the VIP is not associated. c. Outbound NAT addresses when relevant. If Outbound NAT addresses belong to a different subnet than the Internet side physical interface, the same observation made for VIPs is valid. The VRRP standard limits the number of IP addresses that can be associated to a VR to 256. In cases where there are more than 255 addresses to be associated (VIPs and Outbound NAT together) there are 2 options: a. If the addresses belong to a different subnet than the Internet side physical interface, there is no need to associate them, as explained above. b. Configure multiple VRs on the same device interface and split the IP addresses between them. To the VR on the server side: a. VIPI for the server side subnet. b. Client NAT addresses when relevant. If Client NAT addresses belong to a different subnet than the server side physical interface, the same observation made for VIPs is valid.

7 Page Failover Duration For each VR user can configure the interval at which the VRRP advertisements are sent. The minimum defined by the VRRP RFC is 1 second. Failover will occur at 3 times the advertise interval plus up to 1 second for the skew time. For example, if the advertise interval is 1 second, the failover will occur within 4 seconds. (Refer to the VRRP RFC for more information on the skew time). It is possible to lower the failover period by using the VRRP Advertise Interval parameter that applies to all VRs on the device. This parameter can configure advertise interval in steps of 100ms with a minimum of 100ms this means that the minimum failover time can be ms. However please note that: a. Using this parameter does not conform to VRRP RFC b. When device is overloaded it might miss arriving VRRP advertisements Basic Configurations Routing Configuration Green Physical IP Interfaces Purple Virtual IP Interfaces Orange Virtual IP Internet AppDirector Servers Figure 1 Routing Configuration

8 Page Bridging Configuration Figure 2 - Bridging Configuration Direct Connection Figure 3 - Direct Connection

9 Page Configuration Recommendations Active-Backup To ensure proper AppDirector failover in Active-Backup configuration, Radware recommends performing the following steps on both devices: 1. Interface Grouping: a. Enable Interface Grouping on both devices b. Enable Backup Interface Grouping on both devices (default). c. Configure Selective Interface Grouping on both devices - the physical ports for all device interfaces that you want to participate in the failover process must be set to Include. All other ports should be set to Exclude. The following ports should be excluded: º Dedicated management ports (when using Dedicated Management Port parameter, that port is automatically excluded, no matter what the Selective Interface Grouping configuration is) º Ports that provide connectivity between two AppDirector devices for mirroring purposes. d. When VLANs are used, configure VLAN Up/Down criteria and VLAN Port Interface Grouping (for each VLAN port it is possible to decide whether it is included in the VLAN up/down criteria or not for example a port that is used for traffic connectivity to the other device, should be excluded). 2. Additional configuration in VLAN environment a. Regular VLAN º Enable Backup in VLAN parameter on both devices. º Enable Force Port Down functionality on both devices. b. Switch VLAN When using Switch VLAN loops can occur. Configurations that create loops in the network can be supported on AS4 and AS5 platforms by using the Rapid Spanning Tree protocol to prevent loops. Please see AppDirector Solution Guide for details on Spanning Tree configurations. If you are using other platforms please refrain from creating loop configuration. 3. Set Redundancy Admin Status to VRRP. 4. Configure Virtual Routers (VR) for all device interfaces that must participate in the failover (no VR is required for a dedicated management interface or mirroring only interface, for example). a. The same VRID must be used on both devices for the same interface.

10 Page b. The priority of all the VRs on the Active device must be higher than the priority of the same VRs on the Backup. It is recommended to use a maximum priority of 254 on the primary device (priority 255 has special meaning which overrides the preemption switch and renders the device unmanageable during failover events) and 100 on the backup. Note: All the interfaces for which VRIDs are defined, and only those interfaces, should be Included in Selective Interface Grouping. 5. Associate IP addresses to VRs as explained above. 6. Enable all VRIDs at once (All VRIDs Up). 7. Configure mirroring for stateful failover. If Preemption mode is enabled for all VRIDs: a. Active device º Select the tables that should be mirrored b. Backup device º Configure the mirroring address of the second device (in the configuration examples above, or , or ) º Enable mirroring If Preemption mode is disabled the following configuration is required on both devices: o Configure the mirroring address of the second device o Enable mirroring o Select the tables that should be mirrored Active-Active To ensure proper AppDirector failover in Active-Backup configuration, Radware recommends performing the following steps on both devices: 1. Configure Redundancy Status Primary for Layer 4 Policies on the device that should, in normal operation conditions, be the active device for this policy, and Backup on the other device. 2. Interface Grouping: a. Enable Interface Grouping on both devices b. Disable Backup Interface Grouping on both devices. c. Configure Selective Interface Grouping on both devices - the physical ports for all device interfaces that you want to participate in the failover process must be set to Include. All other ports should be set to Exclude. The following ports should be excluded:

11 Page º Dedicated management ports (when using Dedicated Management Port parameter, that port is automatically excluded, no matter what the Selective Interface Grouping configuration is) º Ports that provide connectivity between two AppDirector devices for mirroring purposes. d. When VLANs are used, configure VLAN Up/Down criteria and VLAN Port Interface Grouping (for each VLAN port it is possible to decide whether it is included in the VLAN up/down criteria or not for example a port that is used for traffic connectivity to the other device, should be excluded). 3. Additional configuration in VLAN environment a. Regular VLAN º Enable Backup in VLAN parameter on both devices. º Enable Force Port Down functionality on both devices. b. Switch VLAN When using Switch VLAN loops can occur. In the example in Figure 3 if the servers try to communicate between them they can create loops in the network. Configurations that create loops in the network can be supported on AS4 and AS5 platforms by using the Rapid Spanning Tree protocol to prevent loops. If you are using other platforms please refrain from creating loop configuration. 4. Set Redundancy Admin Status to VRRP. 5. Configure Virtual Routers (VR) for all device interfaces that must participate in the failover. a. 2 VRs must be configured on the Internet side device interface and 2 VRs must be configured on the server side device interface (if servers belonging to farms that are in main state on this device and servers belonging to farms that are in backup state on this device, are connected to the same device physical interface). Note: In Active-Active configuration the same server cannot be connected to two different farms that have different redundancy status on the same device. If for example we configure VRs with ID X and Y on the same physical interface, on devices A and B: º On device VRID X will have higher priority on device A than on device B while VRID Y will have lower priority on device A than on device B.

12 Page º 2 VIPIs are configured on the server subnet. One VIPI will be attached to VRID X on both devices and will be configured as default gateway on the servers for whom device A is master, and the other VIPI will be attached to VRID Y on both devices and will be configured as default gateway on the servers for whom device B is master. b. The same VRID must be used on both devices for the same VR. c. Each device will have a master VR (higher priority than on the other device) and a backup VR (lower priority than on the other device) on the Internet side and a master VR and a backup VR on the server side. It is recommended to use a maximum priority of 254 on the primary device (priority 255 has special meaning which overrides the preemption switch and renders the device unmanageable during failover events) and 100 on the backup. 8. Associate IP addresses to VRs as explained above: º To the master VR on the Internet side: VIPI, VIPs and Outbound NAT addresses for which this device should be master. º To the backup VR on the Internet side: VIPI, VIPs and Outbound NAT addresses for which this device should be backup. º To the master VR on the server side: VIPI and Client NAT addresses for which this device should be master. º To the backup VR on the server side: VIPI and Client NAT addresses for which this device should be backup. 6. Enable all VRIDs at once (All VRIDs Up). 7. Configure mirroring for stateful failover. In Active-Active redundancy configurations, mirroring can only be enabled in one direction (from device A to device B OR vice-versa, but NOT BOTH). a. Device whose active farms state should be mirrored º Select the tables that should be mirrored b. Device whose active farms state should not be mirrored, but provides backup for the first device farms º Configure the mirroring address of the other device º Enable mirroring Switch configuration recommendation It is strongly recommended to enable fast learning mode on the network switch ports to which Radware devices are connected, i.e., the Cisco Portfast command on Cisco Switches, and equivalent commands on the others. If fast learning mode is not enabled it causes failback (backup-to-active) to flip-flop, before failing back.

13 Page Appendix 1 - Proprietary Redundancy There are 2 basic functions inherent to Radware s proprietary redundancy mechanism: polling and teaching. These functions govern the redundancy operation between AppDirector devices. To illustrate these concepts we use the redundant configuration shown in Figure 1. Polling The two AppDirector devices poll each other on each of their shared interfaces using ARP. This way, one device can always know whether the other is up or down. Physical IP Interfaces are configured to poll other physical IP Interfaces. In the example above, AppDirector2 s and interface addresses would be configured to poll AppDirector1 s and addresses, respectively. The polling frequency and the number of retries before a device is considered down are configurable. Teaching Once an AppDirector interface considers the other AppDirector interface to be down, it must assume responsibility for the failed IP Address. For example, in Figure 1, if AppDirector1 fails and AppDirector2 decides to pick up for it, AppDirector2 must assume responsibility for IP Addresses and To do this, it does the following: 1. AppDirector2 teaches the x network that its port 1 MAC address is now the MAC address of AppDirector2 teaches the x network that its port 2 MAC address is now the MAC address of Once AppDirector2 has taught the network its own MAC addresses for and , it continues to ARP and As long as there is no answer, it will respond to ARPs destined to and , thereby continuing to assume responsibility for these addresses. AppDirector Layer 4 policies can be configured in one of two modes: regular or backup. If a layer 4 policy is configured as regular on AppDirector1, AppDirector1 assumes responsibility for its VIP and responds to ARP messages looking for the VIP with its port 1 MAC. AppDirector2 would not respond to ARPs to this VIP as long as it is inactive.

14 Page Additional Redundancy Terms The following terms are relevant to proprietary redundancy only: Fake ARP in Backup ARP in Interface Grouping In order to speed up the failover process from the backup device to the main device (when main device becomes operational after failure), the backup device sends ARPs advertising that the main device IP addresses now correspond to the MAC addresses of the main device. This is fake ARP, as one device (the backup) publishes the other device (the main). This parameter is enabled by default; however the fake ARP might confuse some Layer 3 switches, as they update their ARP tables by the source MAC of the packet, rather than by the MAC in the information part of the packet. In this case it can be disabled. Defines whether the device can send ARP requests (to servers for example) while interface grouping is active. Basic Configurations The basic configurations for which proprietary configuration can be supported are those shown in Figure 1and Figure 2. Direct connection configurations are not supported by proprietary redundancy mechanism. Configuration Recommendations Active-Backup To ensure proper AppDirector failover in Active-Backup configuration, Radware recommends performing the following steps: Active Device: 1. Configure Redundancy Status Primary for all Layer 4 Policies. 2. Interface Grouping: a. Enable Interface Grouping b. Disable Backup Interface Grouping c. Configure Selective Interface Grouping - the physical ports for all device interfaces that you want to participate in the failover process must be set to Include. All other ports should be set to Exclude (same as for VRRP). c. When VLANs are used, configure VLAN Up/Down criteria and VLAN Port Interface Grouping (same as for VRRP) 3. Leave Redundancy Admin Status as Disabled.

15 Page Backup Device 1. Configure Redundancy Status Backup for all Layer 4 Policies. 2. Interface Grouping: d. Disable Interface Grouping e. Enable Backup Interface Grouping 3. Enable Backup in VLAN 4. Set Redundancy Admin Status to Proprietary. 5. Configure in the IP Redundancy table all the device interfaces that must participate in the failover and their correspondent on the Main device (not for a dedicated management interface or mirroring only interface, for example). Active-Active To ensure proper AppDirector failover in Active-Active configuration, Radware recommends performing the following steps on both devices: 1. Configure Redundancy Status for all Layer 4 Policies - the Redundancy Status of each Layer 4 policy will be set to Primary on the device that should, in normal operation conditions, be the active device for this policy and to Backup on the other device. 2. Interface Grouping: a. Enable Interface Grouping on both devices b. Disable Backup Interface Grouping on both devices. c. Configure Selective Interface Grouping - the physical ports for all device interfaces that you want to participate in the failover process must be set to Include. All other ports should be set to Exclude (same as for VRRP). d. When VLANs are used, configure VLAN Up/Down criteria and VLAN Port Interface Grouping. 3. Enable Backup in VLAN parameter on both devices. 4. Set Redundancy Admin Status to Proprietary. 5. Configure in the IP Redundancy table all the device interfaces that must participate in the failover and their correspondent on the other device (not for a dedicated management interface or mirroring only interface, for example). Switch configuration recommendation It is strongly recommended to enable fast learning mode on the network switch ports to which Radware devices are connected, i.e., the Cisco Portfast command on Cisco Switches, and equivalent commands on the others. If fast learning mode is not enabled it causes failback (backup-to-active) to flip-flop, before failing back.