HPE Knowledge Article

Transcription

1 HPE Knowledge Article BIG-IP Local Traffic Manager (LTM) Series - load balancing not working as expected Article Number mmr_sf-en_us Environment BIG-IP-LTM-1500 series Issue Load balancing to show the correct "loading" values. load balancing load balancing not loading as expected Cause Configured two Virtual Servers: VS_Inline_1 VS_Inline_2 pointing to the same pool: ingress_proxy This pool is the only one whose load balancing method is Ratio (member). ingress_proxy pool has 12 pool members. 4 of the members have ratio=2 and the other 8 members have ratio=3. Theoretically, members with ratio 2 should receive 2 requests when members with ratio 3 have received 3 requests. However, members with ratio 2 are receiving more requests than members with ratio 3. It looks like a bad behavior of Ratio load balancing method : POOL MEMBER ingress_proxy/10.aaa.bb.cc:any active,up session enabled priority 0 ratio 2 (cur, max, limit, tot) = (47273, 52165, 0, 278.3M) (pkts,bits) in = (177.7G, 8.518T), out = (7.354G, 54.66T) requests (total) = > POOL MEMBER ingress_proxy/10.aaa.bb.cc:any active,up session enabled priority 0 ratio 3 (cur, max, limit, tot) = (39875, 48650, 0, 247.9M)

2 (pkts,bits) in = (5.336G, 7.586T), out = (6.536G, 48.42T) requests (total) = 45 Resolution VS_Inline_1 and VS_Inline_2 were using "persist OSN_ingress" : profile persist OSN_ingress { defaults from source_addr mode source addr timeout 360 Basically, any connections from clients to BIGIP will persist. So load balancing can be skewed due to the existence of this persistence entry. Factors affecting traffic distribution across pool members are : Load balancing methods The load balancing algorithm is the primary mechanism that determines how connections are distributed across pool members. You can define static or dynamic load balancing methods for a pool. Certain load balancing methods are designed to distribute requests evenly across pool members, and other load balancing methods are designed to favor higher performing servers, possibly resulting in uneven traffic distribution across pool members. Static Load balancing methods Certain static load balancing methods are designed to distribute traffic evenly across pool members. For example, the Round Robin load balancing method causes the BIG-IP system to send each incoming request to the next available member of the pool, thereby distributing requests evenly across the servers in the pool. However, when a static load balancing method such as Round Robin is used along with a BIG-IP configuration object that affects load distribution, such as a OneConnect profile or a persistence profile, traffic may not be evenly distributed across BIG-IP pool members as expected. Dynamic Load balancing methods Dynamic load balancing methods typically favor higher performing servers, and may result in uneven traffic distribution across pool members. Dynamic load balancing methods are designed to work with servers that differ in processing speed and memory. For example, when a dynamic load balancing method such as the Observed method is defined for a pool, higher performing servers will process more connections over time than lower performing servers. As a result, connection statistics for the higher performing servers will exceed those for lower performing severs. Note: For more information about changes in dynamic load balancing methods, refer to SOL6406: Overview of Least Connections, Fastest, Observed, and Predictive load balancing modes. Persistence A persistence profile allows a returning client to connect directly to the server to which it last connected. In some cases, assigning a persistence profile to a virtual server can create the appearance that the BIG-IP system is incorrectly distributing more requests to a particular server. However, when you enable a persistence profile for a virtual server, a returning client is allowed to bypass the load balancing method and connect directly to the pool member. As a result, the traffic load across pool members may be uneven, especially if the persistence profile is configured with a high timeout value. Source Address Affinity and NATs Source Address Affinity is a commonly used persistence profile that works by directing requests to the same pool member based on the source IP address of a datagram. If you configure a Source Address Affinity profile for a virtual server, and some of the virtual server connections originate from organizations that are connected to the internet through a NAT device, those connections may be persisted to the same pool member, creating the appearance that the BIG-IP system is incorrectly distributing more requests to a particular server. You can mitigate this behavior by configuring an alternate persistence profile for the virtual server, such as cookie persistence. Using Source Address Affinity as a fallback persistence method may also create the appearance that the BIG-IP system incorrectly

3 distributes more requests to a particular server. The BIG-IP system uses the fallback persistence method when it cannot use the specified primary persistence profile. For example, if cookie persistence is configured with source address affinity as a fallback, but a client does not send a persistence cookie, the fallback persistence method is used instead. Persistence through irules You can also configure persistence by writing an irule. Configuring session persistence through an irule allows you to enable a persistence type for particular requests. irule persistence can also create the appearance that the BIG-IP system is incorrectly distributing more requests to a particular server. OneConnect The OneConnect feature works by allowing the BIG-IP system to minimize the number of server-side TCP connections. OneConnect does this by distributing connections on a per-request basis and making existing connections available for reuse by other clients. When a OneConnect profile is enabled for a virtual server, the BIG-IP system is able to reuse open and idle server connections. When more open and idle server connections are available, the BIG-IP system creates fewer new connections. When the network contains a mixture of slow and fast servers, the BIG-IP system may not load balance traffic evenly. Monitors Health monitors verify connections for pool members and nodes on an ongoing basis, and mark pool members and nodes down when they fail to respond. When a device is marked down, the BIG-IP system redirects traffic to another pool member or node. This can result in uneven load balancing statistics across pool members. If you notice that traffic is not evenly distributed across BIG-IP pools members, as expected, you can verify whether one or more pool members / nodes are being marked down by a BIG-IP health monitor. To do so, review the /var/log/ltm file and look for error messages that appear similar to the following example: :3: Pool member :80 monitor status down :3: Pool member :443 monitor status down :3: Pool member :80 monitor status down :3: Pool member :443 monitor status up :3: Pool member :80 monitor status down :3: Pool member :8080 monitor status down. irules irule configurations often control how the BIG-IP system distributes connections, and some irules can result in uneven load balancing across pool members. For example, the following irule uses the node command, which causes the specified server node to be used directly, thus bypassing the load balancing method defined in the pool: when HTTP_REQUEST { if { [ ends_with ".jpg" { node UDP traffic When multiple UDP datagrams are sent from the same IP address and port to a UDP virtual server, the BIG-IP system, by default, sends all datagrams that arrive within the Idle Timeout period configured in the associated UDP profile to the same pool member to which the preceding datagram from the same source was sent. This can result in uneven load balancing statistics across pool members. Clustered Multi-Processing Clustered Multip-Processing (CMP) can be a significant factor in how the BIG-IP system distributes connections across active TMM instances, which in turn can affect distribution to pool members. More detail in respect to LOAD BALANCING and PERSIST : Enabling persistence for a virtual server allows returning clients to bypass load balancing When you configure a persistence profile for a virtual server, the BIG-IP LTM system tracks and stores session data, such as the pool member that serviced a client request. Persistence allows returning clients to bypass load balancing and connect directly to the server to which they last connected. Note: When a cookie persistence profile is configured for a virtual server, the BIG-IP system does not store the session data locally

4 in the persistence table; the BIG-IP system stores the session data in the cookie. Based on the following example configuration, when client connects to the my_virtual virtual server, the BIG-IP LTM system load balances the connection to one of the pool members in my_pool, and creates a persistence entry in the persistence table. When the client returns to the virtual server and the persistence entry has not expired, the BIG-IP LTM system bypasses load balancing and sends the connection directly to the server to which it last connected: virtual my_virtual { destination :http ip protocol tcp profile http tcp persist my_persist pool my_pool profile persist my_persist { defaults from source_addr mode source addr pool my_pool { lb_mode fastest min active members 1 member :80 member :80 member :80 Using persistence with priority group activation The priority group feature allows each member in the load balancing pool to be assigned a priority. While all the members of the highest priority group are available, traffic is load balanced to that priority group. If the number of available members in the highest priority group drops below a configured number, the BIG-IP LTM system activates the next highest priority group and sends traffic to those pool members. When a virtual server references a priority group pool, persistence may cause traffic to persist to a node in a lower priority group even after a higher priority group is reactivated. Based on the following example configuration, if both priority 3 members of the my_pool pool are marked down, the priority 2 member , begins taking connections. When a client connects to the my_virtual virtual server, the BIG-IP LTM system load balances the connection to the pool member, and creates a persistence entry in its persistence table. When the client returns to the virtual server and the persistence entry has not expired, the BIG-IP LTM system bypasses load balancing and sends the connection directly to the server to priority 2 member , regardless of whether a higher priority pool member has returned to service: virtual my_virtual { destination :http ip protocol tcp profile http tcp persist my_persist pool my_pool profile persist my_persist { defaults from source_addr mode source addr pool my_pool { lb_mode fastest

5 min active members 1 member :80 priority 3 member :80 priority 3 member :80 priority 2 If connections are persisted to a lower numbered priority pool member you can clear the persistence entries for the pool member using the following command syntax: bigpipe persist pool <pool_name> node <node_addr:port> delete For example, if connections are persisted to pool member :80, you can clear the persistence entries for the pool member by typing the following command: bigpipe persist pool my_pool node :80 delete However, there is one known issue that might also have an impact here : Method : Ratio (member) Ratio (node) Description : The BIG-IP system distributes connections among pool members or nodes in a static rotation according to ratio weights that you define. In this case, the number of connections that each system receives over time is proportionate to the ratio weight you defined for each pool member or node. You set a ratio weight when you create each pool member or node. When to use : These are static load balancing methods, basing distribution on user-specified ratio weights that are proportional to the capacity of the servers There is one known issue where fixed ratio load balancing may not correctly count persistent connections. This is the result of a known issue. The fixed ratio load balancing method may ignore the existing persistent connections when considering the pool member or node as a candidate for receiving a new load balanced (non-persistent) connection. This behavior can lead to specific nodes or pool members receiving a number of connections that exceed their actual ratio value when compared to other nodes or pool members in the pool. For example, for a given pool with one ratio 5 member and four ratio 1 members with a set of 72 connections, the expected distribution of connections with a fixed ratio would be: Member 1 (ratio 5): 40 connections Member 2 (ratio 1): 8 connections Member 3 (ratio 1): 8 connections Member 4 (ratio 1): 8 connections Member 4 (ratio 1): 8 connections However, due the condition previously described, the observed distribution may have a skewed ratio, such as: Member 1 (ratio 5): 60 connections Member 2 (ratio 1): 3 connections Member 3 (ratio 1): 3 connections Member 4 (ratio 1): 3 connections Member 4 (ratio 1): 3 connections Note: The incorrect observed ratio above is only an example intended to show how the distribution may appear, and is not intended to represent an actual alternate ratio that should be expected. The skewing of the ratio varies based on configuration and traffic profile. Due to the manner in which the fixed ratio load balancing method calculates availability, this higher connection count will tend toward a consistent subset of pool members with otherwise equal ratios. As the differential between the ratios of the respective members increases, as well as the disparity in size of the ratio groups (number of high ratio value members compared to lower ratio

6 value members) within the pool, this increased connection count for certain members may be more apparent. F5 Product Development tracked this issue as ID and it was fixed in BIG-IP version Copyright 2016 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice.the only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services.nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.