DHCP Failover: An Improved Approach to DHCP Redundancy

Transcription

1 Overview The DHCP Failover protocol specification and ISC s implementation of the protocol have problems that can cause issues in production environments, primarily in those environments where configurations change on a regular basis. Infoblox has fixed the bugs in the ISC implementation and enhanced the DHCP Failover protocol to increase its reliability and resiliency. This white paper provides an overview of DHCP Failover, specific details on what Infoblox has done to correct problems, and an appendix with a detailed description of the protocol. What is DHCP Failover? DHCP Failover is a way for two DHCP servers to share a common group of IP addresses. The addresses can be given out by either server and either server can take over for the other in the event of a failure. In order to maintain a common view of the lease states of all addresses and avoid conflicts, the servers stay synchronized with each other via the DHCP Failover protocol. Before the DHCP Failover protocol, the only way to get any level of redundancy with DHCP without introducing the chance of issuing duplicate IP addresses was to create split ranges. In a split range, you take a pool of addresses and split the addresses across multiple DHCP servers. For example, given the /24 network, DHCP server A might have the addresses from to , while server B might have to There are a number of problems with this solution, the most obvious of which is that it is wasteful of the useable IP address space (i.e., redundancy is provided by reserving IP addresses and not using them unless a failure occurs). This can lead to critical issues such as IP address exhaustion if failures persist for extended periods and the backup range is insufficient to supply addresses to all devices. In addition, devices that received an IP address from one range will get a different IP address if they try to renew from the backup server. Some devices, such as IP phones, may exhibit undesirable behavior, such as dropping calls, if their IP address changes. Using DHCP failover for the same scenario, both DHCP server A and DHCP server B know about all addresses from to , but each assigns only a subset of those addresses at any given time, keeping track of which addresses each assigned. If one of the peers (in DHCP failover, each server is called a peer) stops working for some reason, the peer that is still alive assumes all of the IP addresses, still remembering which addresses it really owns, and which it is serving on behalf of its unavailable peer. When the unresponsive peer comes back online, the two servers resync their lease information, and each server once again becomes responsible for its allotment of addresses. This provides a robust and dynamic system that continues to serve addresses in the event of a failure and does so without squandering valuable address space. Why doesn t everybody use DHCP Failover? There are a few reasons why people may not use DHCP Failover. First, not every DHCP server supports DHCP Failover. Additionally, there is no universally agreed standard for DHCP Failover. There is an RFC that describes the DHCP Failover protocol, but it was never ratified, so each vendor is free to implement its own version. This means that interoperability could be an issue if you tried to take two different vendors DHCP servers and run them together using DHCP Failover. Further, the draft RFC also did not cover some of the corner cases and, as a result, there are known conditions in which the protocol defined by the RFC will fail, sometimes in a manner from which recovery can be very difficult. Page 1 of 11

2 The Infoblox Advantage The Infoblox DHCP server is based on the ISC 3.0.x DHCP server. Infoblox has made some significant improvements to ISC DHCP server to address the implementation errors. In addition, Infoblox has made improvements to the DHCP Failover protocol itself that prevent the occurrence of deadlock and other unstable states that are possible based on the original RFC definition. The combination of the protocol enhancements and the implementation fixes make the Infoblox DHCP Failover implementation more reliable and robust than the ISC implementation or any product that is based on the standard ISC distribution. Below is a list of some of the issues with ISC s DHCP Failover implementation and the approaches that Infoblox has used to address them: Issue: The protocol RFC depends on the assumption that the failover configurations on both peers are exactly the same. Being out of sync can result in the issuance of duplicate IP addresses or leases not being provided. With the ISC DHCP server, there is no enforcement, synchronization, or checking to ensure that peers have identical configurations. Resolution: Infoblox DHCP failover provides a check to determine that both peers, when in the same Grid, have identical configurations. If the configuration check fails, the failover association will not connect until both peers have the same configuration. Issue: If the TCP connection between peers does not establish properly, the failover peers can become locked in the COMMUNICATIONS-INTERRUPTED state indefinitely. In this state, they will not issue leases to clients. Resolution: This failure condition was discovered and fixed, ensuring that the TCP connection between peers is always able to establish correctly (assuming network connectivity is available). Issue: Troubleshooting DHCP Failover can be time-consuming due to the lack of diagnostic information available to administrators. Resolution: Infoblox has greatly enhanced the information logged. For example: During recovery, a message is logged for every 1,000 leases sent in reply to the update request. During COMMUNICATIONS-INTERRUPTED, a message is logged every minute showing that a peer is trying to connect to the other peer. In addition, the Infoblox Grid Manager GUI has a screen which shows all failover associations and their real-time status. Issue: The ISC DHCP Failover implementation has not been optimized for performance. Resolution: The Infoblox DHCP Failover implementation has enhancements that decrease the time spent in the recovery states when both peers are in the same grid. For example: When a change is made to DHCP configuration that would cause a transition to a recover state, we may be able to reduce the RECOVER-WAIT time to zero (as opposed to waiting the full MCLT). For example, if a user changes the size of a range or assigns a range to a different failover association, the recalculation and recovery time is reduced, which translates to less downtime for those ranges. Peers will only exchange data for the specific ranges changed, and not all ranges, which may significantly reduce recovery time. Page 2 of 11

3 Issue: The ISC DHCP server does not permit simple manipulation of the leases database. Resolution: Since the Infoblox DHCP server uses its bloxsdb database for lease storage, leases can be cleared via the Infoblox Grid Manager on both peers at the same time, thereby avoiding any potential duplicate leases. DHCP administrators often use the Clear Lease function when troubleshooting networks (e.g., a client obtains a lease with a long lease time and the administrator wants to force it to renew, or after replacing a NIC card, the administrator wants to clear out the old NIC s lease). Page 3 of 11

4 APPENDIX: DETAILED DHCP FAILOVER PROTOCOL This Appendix provides a detailed description of the DHCP Failover protocol for those interested in a more indepth understanding of how it works. How Does DHCP Failover Synchronize Data? DHCP failover peers use lazy updates to keep each other up to date. Lazy updates use a best effort approach to keeping the data the same at all times, but do not demand that the data is always in complete synchronization. Each server follows a set of rules to prevent either server from doing something that may cause a problem later on. This means that each server can assign an address to a DHCP client before updating its peer, resulting in better performance. How Do IP Addresses Get Assigned While Using DHCP Failover? Lazy updates (as described above) work by creating set of rules for the two peers on how addresses should be handed out. If both DHCP servers follow the same set of rules, there is no chance the servers will assign different IP addresses to the same machine. There are three rules the servers follow: 1) The primary and secondary failover peers divide the list of free addresses, referred to as a pool, on each network defined for the failover association, into groups of free and backup addresses. Free addresses are available for assignment by the primary server to allocate to DHCP clients. Backup addresses are available for assignment by the secondary server. 2) Each DHCP server can allocate, or extend, leases for only a limited amount of time beyond the lease time known by its peer. This limited time is called the Maximum Client Lead Time, or MCLT, which is the maximum time the one server s idea of a lease expiration time can lead the other s. The MCLT is usually quite short (compared to the real lease time), usually no more than one hour. The server can keep extending the lease by MCLT indefinitely, but this requires much more frequent renewals. There is a way to increase this amount of time, but it requires the two servers to negotiate the acknowledged potential lease expiry time. When this time has been established, either peer can extend the client s lease for up to that amount of time, plus MCLT. Since the acknowledged potential lease expiry time is a fixed point in time and not a duration, like MCLT, the server must reestablish a new acknowledged potential lease expiry time every time it extends a lease. 3) In normal operations, a lease that has already been assigned to a client can not be assigned to another client unless both servers agree that the lease is no longer being used. Page 4 of 11

5 Communication Between Failover Peers Failover peers communicate using a TCP connection. When using DHCP Failover on a DNSone, the TCP port used is 519. The connection is asynchronous, so either server can establish this connection, and either server can send a message to the other at any time. This allows a connection to be made as soon as the second peer is started. As soon as a connection is established, the primary peer sends the secondary a CONNECT message. The CONNECT message contains identification and authentication information, as well as some information about how the primary is configured, particularly the MCLT value. If the secondary peer recognizes the primary peer and is able to communicate with it, it sends a CONNECTACK message. The CONNECTACK contains authentication information similar to that in the CONNECT message, as well as information on how the secondary is configured. After these two messages have been exchanged, the peers begin their failover communications. After the initial communications have been established, the peers exchange state information with each other and, if needed, synchronize their IP address databases. This process will be explained in more details later in this FAQ. When the peers first connect, and do any synchronizing that might be needed, the peers balance each address pool, making sure that each peer starts out with approximately the same number of IP addresses. During normal communications, when a DHCP server receives a DHCPREQUEST from a client, it responds with a DHCPACK back to the client, and then sends a binding update (BNDUPD) to its peer. When the peer receives the BNDUPD, it adds the update to a queue to be processed. After the update is processed, a binding acknowledgment (BNDACK) is sent back to the originating peer. These same BNDUPD and BNDACK messages are used throughout the synchronization process. As address are assigned from pools, pools can become unbalanced, leaving one peer with more addresses available than the other. When this occurs, the peer with fewer addresses performs a pool-rebalancing. Details on rebalancing will be described later. During quiet times (when no DHCP activity is taking place), the peers exchange CONTACT messages with each other to ensure there is no loss of connectivity. If no message is received by one of the peers for a certain amount of time, it is assumed that the peer connection is broken, and that peer begins working independently. A server can also send a DISCONNECT message in case it needs to leave the failover association to shut down for some reason. If this occurs, both servers close their connection. Lease Handling with DHCP While using DHCP Failover, it is no longer enough to know that a lease is either in use or available. There are many address binding states allowed during failover. These states are used to show whether an address is in use, which peer can allocate an available address, and an address s transition from being in use to being available for reallocation. The table below lists the binding states that an IP address can be in, excluding fixedaddresses and those assigned to BOOTP clients (as DHCP Failover does not support BOOTP). Page 5 of 11

6 State ABANDONED ACTIVE BACKUP EXPIRED FREE RELEASED RESET Description The address has been marked as abandoned due to a conflict with another machine already claiming to have that IP address. This may be a client machine with a hard-coded IP address that conflicts with a DHCP pool the administrator has defined. The address is in use by a client. The address is available for assignment by the secondary peer. The address is no longer used by the client, but is still bound to that client. Only after all FREE/BACKUP leases are assigned will requests be fulfilled by EXPIRED addresses. The address is available for assignment by the primary peer. The address has been released by the client, but is not yet available for reassignment. The address has been released by administrative action, but is not yet available for allocation. When a failover peer makes a change to any lease, a BNDUPD message is sent to the other peer. The update includes the new state of the lease, the time the change was made, the actual expiry time of the lease, along with other information that identifies the client and other information that the client sent. When the peer processes the update, it sends a BNDACK. As soon as the first peer receives the BNDACK, both servers have the same information about that lease. If the two peers are not operating normally, they may have different states for the same IP address. The rules controlling IP allocation protect both peers from making mistakes when their IP address databases are not in synch. From the table above, you can see that two states show an IP address is available for use: FREE addresses can be assigned by the primary server, and BACKUP addresses can be assigned by the secondary. Addresses are never available on both peers at the same time. As soon as an address is given to a client, whether it s assigned from the primary or the secondary, the lease is placed in the ACTIVE state. An address in the ACTIVE state can not be given to any other client until it is moved to the FREE or BACKUP state. This allows either the primary or secondary peer to extend a lease. As soon as a lease reaches its expiry time, it is changed from ACTIVE to EXPIRED. The server that changes the state updates its peer. When the peer receives the update, it moves the lease to the FREE state, and sends an acknowledgment to the first peer, which also moves the lease to the FREE state. At this point, the primary server is now free to allocate this lease. This two-way handshake is very important, as you would not want the lease to be placed in a FREE state if both servers did not know the lease was available. Failure to do this could cause one server to think the address was still in use and reassign it while the second server assigns the lease to a new client, causing a duplicate IP address on the network. RELEASED and RESET are handled using the same logic. Page 6 of 11

7 Lease Durations While Using DHCP Failover If failover is not in use, a DHCP client can request a lease time in the DHCPDISCOVER and DHCPREQUEST packets. If the client does not request a lease time, the DHCP server will assign it a lease time, which is configurable. If a requested lease time is acceptable, it will be used. If the lease time is smaller than a configured minimum or larger than a configured maximum, the lease is adjusted to the minimum or maximum time. When using failover, the lease time is called the desired lease time. It is called this because this is the lease time the server would like to use, depending on the state of the lease. There are three entities that remember the state of a lease: the DHCP client, the primary failover peer, and the secondary failover peer. The lease time must be set so that it does not expire before the primary or secondary peer thinks the lease will expire. Since the server assigning the lease always remembers this time, this problem really only exists on the other server. Since the lease time needs to be calculated before the peer knows about the lease, peer compares MCLT with the desired lease time and chooses the smallest value, which is what the client will use for its initial lease time. After the lease is accepted, the primary peer starts to calculate the potential lease expiry time (the time the client is expected to renew its address) plus the desired lease time. The primary peer assumes the client will renew after half of its current lease time, and tells the secondary peer the actual lease expiry time and also the potential lease expiry time. The secondary receives this time and records it in its version of the IP address state, and sends an acknowledgment. At this point, the primary receives the BNDACK and knows that both the primary and secondary have the same renewal time for the lease, called the acknowledged potential lease expiry time. The next time a client tries to renew, a new desired lease time is calculated, which will probably be the same as that calculated the last time, so the desired lease time is able to be used for the lease instead of MCLT. This same process occurs if the secondary receives a DHCPDISCOVER. Page 7 of 11

8 Failover Operational States So far, all explanations made have assumed that the failover association was functioning normally, but in the real world, this is not always the case. Normal Operation: Primary/Backup Configuration, and Load Balancing During NORMAL state, only one peer will respond to a DHCP message from a client. When using failover on a DNSone, a simple set of rules is used to balance load and determine which peer should respond to a client. This technique, described in RFC 3074, uses a secure hash algorithm to hash the client s identification information. When a peer receives a client s broadcast message, it performs a hash on the client s identification, which produces a number from 0 to 255. The server has a table with 256 entries, and uses the calculated hash value to lookup the corresponding entry in the table. If the entry is non-zero, the peer responds to the request, otherwise it drops the request. The primary server is configured with a load balancing value, and creates the table. The opposite version of the table is sent to the secondary server in the CONNECT message, assuring that the two peers will never have the same IP address to assign. After assigning an address, an update is sent to the peer so they both know the new state of the address. After each assignment, the peers recalculate the amount of free leases they have and, if necessary, rebalance the pool (described in more detail a little later.) Operations in the COMMUNICATIONS-INTERRUPTED State When communications between the two peers is interrupted, the peers enter the communicationsinterrupted state. In this state, neither peer knows the status of the other, so both peers provide DHCP services to any client that requests an address. Since address tables cannot be updated, a peer must stop issuing addresses as soon as it runs out of available addresses (FREE for the primary, BACKUP for the secondary). The peer may, however, renew any address that it already knows about, even if the lease was originally issued by the other peer. A disadvantage of communications-interrupted mode is that each peer can only allocate leases for MCLT seconds. This will result in a heavier load since lease will be renewed quicker than normal. Operations in the PARTNER-DOWN State For many reasons, including long network outages or planned upgrades, one peer of a failover association may become inoperable. In order to provide services during this outage, an administrator can put one peer in to a partner-down state. When a peer is placed in partner-down state, the remaining peer assumes control of all DHCP service. It safely assumes that its peer is not available, and can start assigning all IP addresses in the pool to clients needing addresses. As mentioned above, an unexpected failure could cause a peer to go in to the communicationsinterrupted state, which is not a desirable long-term state. If a peer goes in to a communicationsinterrupted state, an administrator can force the remaining server into the partner-down state. When a server changes state, it remembers the start time of state (STOS). When a server changes to partner-down, it can reclaim any FREE or BACKUP addresses that belong to its peer after MCLT plus STOS has passed. If an address is in the ACTIVE, EXPIRED, RELEASED, or RESET state, and the acknowledged potential expiry time is later than STOS, the server can free the IP address only after the Page 8 of 11

9 acknowledged potential expiry time plus MCLT, or after MCLT plus STOS, whichever comes last. This is due to the fact that the failed peer may have extended the lease to the acknowledged potential expiry time plus MCLT without updating its peer, but it may have also extended it to MCLT plus STOS, so the peer in the partner-down state must account for both possibilities. After a peer in the partner-down state has reclaimed all addresses that belonged to its peer, the MCLT has passed, and all the acknowledged potential expiry times have expired, the remaining peer operates as if it were no longer running in failover mode. It continues to do so until its peer reconnects to it. Due to this, it is imperative that the restarted peer not provide IP addresses until it has resynchronized with the peer in the partner-down state. This procedure is explained in the Operations in the RECOVER State section. If it were not to do this, it might hand out the wrong address to clients, or duplicate an already assigned address. Operations in the STARTUP state The STARTUP state is the state used when a peer first starts up. The server will not issue any IP addresses, and will wait until the peers can synchronize their address databases. There is one exception to this rule: If a server that has no memory of ever talking to its peer comes online, it goes straight to the RECOVER state. When starting two DHCP servers with a new failover association, the RECOVER state is, most likely, the state the two servers will first enter. Operations in the RECOVER State When a peer is in the RECOVER state, a complete update of the IP address database is done. This is the state that a peer will enter when it discovers that its peer was in the partner-down state, or when it realizes that it has never communicated with its peer before. A peer in the recover state will not answer any DHCP queries because it cannot ensure its data is accurate. The server will attempt to establish a connection to its peer if one has not been made already. After the connection is established, an update request (UPDREQ) message is sent to its peer. The peer reviews its entire address database and sends a BNDUPD for each and every IP address whose state has changed since the last time it communicated with the recovering peer in the normal state. If the recovering peer has no memory of communicating with its peer, it sends an update all request (UPDREQALL), and the peer sends information about every IP address it knows about. After the last update message is sent, an update done (UPDDONE) message is sent, and the recovering peer goes in to the RECOVER-WAIT state. The peer remains in this state for MCLT seconds. After MCLT seconds has passed the server moves to the RECOVER-DONE state. If its peer is in PARTNER-DOWN, or RECOVER-DONE, the peer moves to the NORMAL state. Upon seeing its peer move to NORMAL, it, too, moves to the NORMAL state, and begins issuing DHCP addresses. Operations in the POTENTIAL-CONFLICT State The RECOVER state is used to make a graceful transition back to the normal state. The problem is that an administrative error or incorrect state transition could make this impossible. There are a few situations that could cause this, and all involve one or both of the peers operating in partner-down state while the other is in the communications-interrupted state, or the partner-down state. This can happen, for example, because: A server fails and its peer is placed in partner-down state, but when the failed server comes back online it is unable to communicate with the peer. Page 9 of 11

10 A server administrator places one peer in partner-down state when the other is actually operating in the communications-interrupted state. Both of these scenarios could lead to a conflict, so both servers would immediately go in to the potentialconflict mode and stop serving DHCP addresses. Since this is an uncontrolled event, it is possible that both peers have issued the same IP address to different clients. The two peers should still remember that last time they communicated in the normal state, and will send updates to each other for any addresses whose states have changed. When a primary peer enters the potential-conflict state, it sends an UPDREQ message to the secondary peer. When the secondary enters the potential-conflict state, it waits for the primary to send the UPDREQ. When it receives the UPDREQ, it sends the primary BNDUPD message for all addresses whose states have changed, and for any addresses for which it has not received BNDACK messages. When the secondary has finished the updates, it sends an UPDDONE to the primary. The primary receives the UPDDONE and moves to the CONFLICT-DONE state, and starts answering clients DHCP requests. The secondary notices the peer changed state and sends an UPDREQ. The primary sends all of its updates to the secondary and sends an UPDDONE. The secondary then moves to the NORMAL state, and the primary seeing the state change also moves to the NORMAL state. After moving to the normal state, both peers can check to see if their pool of IP addresses is low and perform pool-rebalancing. Operations in the RESULTION-INTERRUPTED State It is possible that during the conflict-resolution process outlined in the Operations in the Potential- Conflict State, the connection between the two peers may be broken. Any peer still in the potentialconflict state when this occurs goes into the resolution-interrupted state. While in this state, the peer may extend existing leases, and may issue new leases from its pool of available addresses. Because the lease databases are not synchronized, the possibility of an IP address conflict exists during this state. Binding Update Conflicts The potential for an address conflict can exist in certain circumstances, and due to this fact, the BNDACK can include a REJECT-REASON. If a REJECT-REASON exists, the peer will reject the BNDUPD message. Pool Rebalancing While the two peers are operating in a NORMAL state, it is very common for the pool of free addresses and the pool of backup addresses to grow and shrink at different rates. This is particularly true because a lease that expires goes to the FREE state and never the BACKUP state. This means the BACKUP pool will get smaller more quickly than the FREE pool. When the secondary peer notices that the pool of addresses has become significantly out of balance, it sends a pool request (POOLREQ) message to the primary. The primary receives the POOLREQ, examines all of its pools to determine if any pools are out of balance. If so, it assigns addresses to the secondary to rebalance the pool by sending BNDUPD messages, marking the address BACKUP instead of FREE. When the primary determines the pool is out of balance, it sends a BNDUPD message to the secondary, making IP addresses FREE instead of BACKUP until the pool is balanced. If the secondary has issued one of the addresses to a client, it simply refuses the BNDUPD. Page 10 of 11

11 Conclusion The DHCP Failover protocol comprises a very complex set of rules that are created to maintain synchronization between two dynamic, autonomous systems and to avoid conflicts as much as possible. A network administrator, however, should ensure that a valid and stable communications path exists between the servers to avoid constant operational state changes. Server configuration changes must be done with care because configuration conflicts between two peers have been demonstrated to cause unpredictable results. Infoblox Grid Technology and the improved Infoblox implementation of the DHCP Failover protocol provide customers with a stable, reliable and complete solution for resilient DHCP services. Using Infoblox DHCP Failover in conjunction with Infoblox High Availability (HA) provides the ultimate in DHCP availability and offers the most resilient solution on the market. Page 11 of 11