Hierarchical Cache Design for Enhancing TCP Over Heterogeneous Networks With Wired and Wireless Links

Transcription

1 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 2, NO. 2, MARCH Hierarchical Cache Design for Enhancing TCP Over Heterogeneous Networks With Wired and Wireless Links Jian-Hao Hu, Gang Feng, Associate, IEEE, and Kwan Lawrence Yeung, Senior Member, IEEE Abstract TCP is a reliable transport protocol tuned to perform well in traditional networks made up of links with low biterror rates. Networks with higher bit-error rates, such as those with wireless links and mobile hosts, violate many of the assumptions made by transmission control protocol (TCP), causing degraded end-to-end performance. In this paper, we propose a twolayer hierarchical cache architecture for enhancing TCP performance over heterogeneous networks with both wired and wireless links. A new network-layer protocol, called new snoop (NS), is designed. The main idea is to cache the unacknowledged packets at both mobile switch center (MSC) and base station (BS), to form a two-layer cache hierarchy. If a packet is lost due to transmission errors in wireless link, the BS takes the responsibility to recover the loss. When a handoff occurs, the packets cached at MSC can help to minimize the latency of retransmissions due to temporal disconnection. NS can preserve the end-to-end TCP semantics and is compatible with existing TCP applications. Its implementation only requires code modification at the BS and MSC. Simulation results show that NS is significantly more robust in dealing with unreliable wireless links and handoffs as compared with the original Snoop scheme, as well as some other existing TCP enhancements. Index Terms Handoff, heterogeneous network, local loss recovery, new snoop (NS), transmission control protocol (TCP) enhancement. I. INTRODUCTION THE two growing technology trends, internet access and wireless access, would result in a strong combination that wireless data networks and services would dominate the marketplace. TCP is a transport layer protocol that has been designed, improved, and tuned to work efficiently on wired network where the packet loss is very small. Whenever a packet is lost, it is reasonable to assume that congestion has occurred on the connection path. Hence, TCP triggers congestion recovery algorithms when packet loss is detected. On the other hand, the bit-error rate of a wireless link is much higher and a wireless connection might be temporally broken due to a handoff or other temporal link impairment such as shadowing effect. As a result, the assumption that packet loss is (mainly) due to congestion is no Manuscript received February 14, 2000; revised August 28, 2001; accepted September 9, The editor coordinating the review of this paper and approving it for publication is V. O. K. Li. This work was supported in part by Competitive Earmarked Research under Grant HKU 1180/00E. J.-H. Hu and K. L. Yeung are with the Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong ( jhhu@eee.hku.hk; kyeung@eee.hku.hk). G. Feng is with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore ( egfeng@ntu.edu.sg) Digital Object Identifier /TWC longer valid and the original TCP cannot work well in a heterogeneous network with both wired and wireless links. In summary, the challenges for TCP over wireless networks are: high bit-error rates, handoff due to users mobility, asymmetric effects and latency variation, and low channel bandwidths. Since many network applications are built on top of TCP, and will continue to be in the foreseeable future, it is important to improve TCP performance with no (or minimal) modifications to the existing TCP implementations. This is the most promising way by which mobile devices can seamlessly integrate with the rest of the Internet. Recently, several reliable transport-layer protocols for networks with wireless links have been proposed to alleviate the poor end-to-end TCP performance in the heterogeneous network environments, including split connection approach [4], [12], fast-retransmit approach [9], link-level retransmissions [2], and selective acknowledgment [13]. However, these schemes have their own limitations with respect to one or more of the following aspects: TCP semantics transparency, application relinking requirement, software overhead or others. It is worthy noting that a good scheme called Snoop was proposed in [8], and [14]. The Snoop protocol introduces a module, called snoop agent, at the base station (BS). The agent monitors every packet that passes through the TCP connection in both directions and maintains a cache of TCP packets sent across the link that have not yet been acknowledged by the receiver. The snoop agent retransmits the lost packet if it has it cached and suppresses the duplicate acknowledgments (ACKs). Like other link-layer solutions, the snoop approach could also suffer from not being able to completely shield the sender from the wireless losses. In this paper, we focus on the case where a fixed host, e.g., an ISP server, using TCP to communicate with a mobile host via a path consisting of an error-immune wired link and an error-open wireless link, as shown in Fig. 1. A novel TCP enhancement scheme, called new snoop (NS), is proposed. The main idea is to cache the forwarded, but not-yet-unacknowledged packets at both mobile switch center (MSC) and BS, to form a two-layer cache hierarchy. If a packet is lost due to transmission errors in wireless link, the BS takes the responsibility to recover the loss. If the loss/interruption is due to a handoff, the MSC performs the necessary recovery. The local loss recovery scheme used at the BS is developed based on Snoop [8], [14], but with many original features and functions. We show that with the proposed hierarchical cache architecture, NS can effectively handle the packet losses caused by both handoffs and link impairments /03$ IEEE

2 206 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 2, NO. 2, MARCH 2003 Fig. 1. A TCP connection between a fixed host and a mobile host. This in turn gives significant improvement in end-to-end TCP throughput performance as compared with the original Snoop, as well as some other TCP enhancements. The rest of this paper is organized as follows. In Section II, we start with a brief review on the TCP architecture. Then, the most representative recent work on designing TCP enhancement schemes for heterogeneous networks is surveyed in Section III. In Section IV, the details of the two-layer hierarchical cache architecture and the NS protocol are presented. Simulation model and simulation results are then given in Section V. Finally, we conclude the paper in Section VI. II. A BRIEF REVIEW ON TCP The Internet is a best-effort network. Today s Internet infrastructure provides no guarantee on bandwidths or latencies and has no mechanisms to avoid packet losses. Packets between end-hosts are routed through a set of routers. Network congestion due to overload and router resource contention causes packet loss. This service model leads to a simple internal architecture that scales to handle a large number of communicating entities, but requires higher level protocols and applications to adapt to packet loss, varying available bandwidths and latencies. TCP is the standard for reliable unicast data transport in the Internet. It is used by virtually all unicast applications that require reliability. TCP achieves reliability by using cumulative ACKs that are sent by the receiver to the sender for all received data on a regular basis. A cumulative ACK acknowledges the successful receipt of all data received in-order so far. These ACKs are used by the sender to determine the successful delivery or loss of data. The TCP sender performs loss recovery by retransmission packets that it considers missing based on a lack of an ACK for it. Thus, TCP achieves reliability using an acknowledgment-repeat-request scheme. Currently, the TCP implementations contain a number of algorithms aimed at controlling network congestion. All of them are realized via defining a congestion window (CWND). This CWND is an estimate of the maximum number of packets that can be sent out without overloading the network. The TCP sender operates in one of two modes for congestion control [15]: slow start and congestion avoidance. They correspond to the nonsteady phase and steady-state phase, respectively. The goal of the slow start phase is to reach the steady state as quickly as possible. The variable SSTHRESH (steady-state threshold) is maintained at the sender to distinguish between the two phases. SSTHRESH is, thus, the estimate of the steady-state network capacity. As the window size CWND is smaller than SSTHRESH, the sender works in slow start mode. When the threshold is reached, the sender switches to congestion avoidance mode. During slow start, the sender starts with window size CWND packet and it grows by one for every ACK received. The window size is roughly doubled every round-trip time (RTT). In congestion avoidance mode, on the other hand, the window size growth is limited to approximately one packet every RTT. When a packet is lost, the subsequent packets arrived at the receiver are out of order. This triggers the receiver to generate an ACK with the same sequence number as the previous ACK. Such an ACK is referred as a duplicate ACK. The first and the second duplicate ACK are assumed due to a possible out-oforder routing by the sender (i.e., routed through different paths). But when the third duplicate ACK is received, the sender infers that a packet has been lost due to congestion. This then invokes the so-called fast-retransmit procedure by immediately retransmitting the lost packet. Then, the sender enters the fast recovery procedure until an ACK that confirms the correct reception of the retransmitted packet by the receiver is received. Another mechanism that triggers the packet retransmission by the sender is sender timeout. The sender maintains a retransmission timer for the last unacknowledged packet. The timeout value is reset each time a new ACK is received. When the timer expires, the sender enters the slow start mode by retransmitting the lost packet (i.e., the last unacknowledged packet), setting the threshold SSTHRESH to half of the current transmission window size and then resetting this window size to one. It should be noted that timeouts have a significant impact on throughput because they introduce a period of idle time and they close the transmission window size to one. Most TCP implementations use a 500-ms timer granularity for retransmission timeout. The TCP sender estimates [17] the RTT of the connection by measuring the time between the sending of a packet and the receipt of the ACK for that packet. The TCP protocol described above is called TCP Reno, one of the most popular TCP implementations in the Internet today. It is found that TCP Reno is not very efficient in handling multiple packet losses during a single congestion episode. As a result, an enhanced version called TCP New Reno [18] is proposed. In this paper, both TCP Reno and New Reno will be adopted in our performance evaluations. III. RELATED WORK A. The Split Connection Approach [4], [12] The Indirect-TCP (I-TCP) [4], [12] protocol was one of the first protocols to use this method. It involves splitting a TCP

3 HU et al.: HIERARCHICAL CACHE DESIGN FOR ENHANCING TCP OVER HETEROGENEOUS NETWORKS WITH WIRED AND WIRELESS LINKS 207 connection between a fixed and mobile host into two separate connections at the mobile base station: one TCP connection between the fixed host and the BS, and the other between the BS and the mobile host. Since the second connection is over a one-hop wireless link, there is no need to use TCP on this link. Rather, an optimized wireless link-specific protocol tuned for better performance can be used. The advantage of the split connection approach is that the flow and congestion controls of the wired and wireless links are separated. However, this approach suffers from breaking the end-to-end TCP semantics, requiring application relinking and higher software overhead. B. The Fast-Retransmit Approach [9] This approach addresses the TCP performance when communication resumes after a handoff. Unmodified TCP at the sender interprets the delay in a handoff process is caused by congestion (since TCP assumes that all delays are caused by congestion). When a timeout occurs, the system enters slow start and the window size is reduced to one. Usually, handoffs complete relatively fast (between a few tens to a couple of hundred milliseconds), and long waits are required by the mobile host before timeouts occur at the sender. This is because of the coarse retransmit timeout granularities (on the order of 500 ms) in most TCP implementations. The fast retransmit approach mitigates this problem by having the mobile host send a certain threshold number of duplicate acknowledgments to the sender. This causes TCP at the sender to immediately reduce its window size and retransmit packets starting from the first missing one (for which the duplicate acknowledgment was sent). The main drawback of this approach is that it only addresses handoffs and not the error characteristics of the wireless link. C. Link-Level Retransmissions [2] In this approach, the wireless link implements a retransmission protocol coupled with forward error correction at the data-link level. The advantage of this approach is that it improves the reliability of communications independent of the higher level protocol. However, TCP implements its own end-to-end retransmission protocol. Studies have shown that independent retransmission protocols can lead to degraded performance, especially when error rates become significant. A tight coupling of transport- and link-level retransmission timeouts and policies is necessary for good performance. In particular, information needs to be passed down to the data link layer about timeout values and policies for coexistence with the higher transport layer policy. D. Explicit Bad State Notification (EBSN) Scheme [16] Bakshi et al. [16] present a protocol that sends an EBSN to the sender when a wireless link experiences errors. The problem with EBSN is that, with bad network latency, presence or absence of EBSN at the sender will not reflect the actual channel state. The channel bad state can also be of a very short duration such that when EBSN reaches the sender, the channel is no longer in bad state. E. Selective Acknowledgments (SACKs) [13] Since standard TCP uses a cumulative ACK scheme, it often does not provide the sender with sufficient information to recover quickly from multiple packet losses within a single transmission window. Several studies have shown that TCP enhanced with SACK performs better than standard TCP in such situations. In SACK, each ACK contains information about up to three noncontiguous blocks of data that have been received successfully by the receiver. Each block of data is described by its starting and ending sequence numbers. Due to the limited number of blocks, the sender is informed about the most recent blocks received. F. Snoop Protocol [8], [14] The snoop protocol introduces a module, called the snoop agent, at the BS. The agent monitors every packet that passes through the TCP connection in both directions and maintains a cache of TCP packets sent across the link that have not yet been acknowledged by the receiver. A packet loss is detected by the arrival of a small number of duplicate ACKs from the receiver or by a local timeout. The snoop agent retransmits the lost packet if it has it cached and suppresses the duplicate ACKs. In the classification of the protocols in [8] and [14], the snoop protocol is considered as a link-layer protocol that takes advantage of the knowledge of the higher layer transport protocol (TCP). The main advantages of this approach are: 1) duplicate ACKs are suppressed and 2) local packet retransmission is performed. The per-connection state maintained by the snoop agent at the BS is soft and is not essential for correctness. Like other link-layer solutions, the snoop approach could also suffer from not being able to completely shield the sender from wireless losses. IV. TWO-LAYER HIERARCHICAL CACHE ARCHITECTURE AND NEW SNOOP TCP is a ubiquitous protocol and most existing network applications are using it, it is desirable to design enhancement scheme for TCP over wireless networks without changing existing TCP implementations. Using the proposed two-layer hierarchical cache scheme, no modifications to the two end hosts and the fixed network are required except at the MSC and BS. At these two points, the routers are modified to cache the unacknowledged packets going to mobile hosts using a new network-layer protocol called NS. Local retransmission over the wireless link is performed by the BS when a packet loss due to wireless transmission errors is detected. If a connection interruption due to a handoff occurs, the MSC performs the necessary recovery. We can show that the sender can be completely shielded from the instabilities of the wireless links and the user mobility. A. NS Protocol Layer Like Snoop protocol [8], [14], we add NS protocol above the IP layer as shown in Fig. 2. Its function is to monitor and cache the packets passing through the BS and the MSC, to abstract the code information from the TCP packets, and to utilize this information for local packet recovery and handoff. Fig. 2 shows the protocol layer structure for a single TCP connection between the

4 208 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 2, NO. 2, MARCH 2003 Fig. 2. host. Protocol layer for a direct connection between the sever and the mobile server and a mobile host. We can see that no TCP layer function is performed at both MSC and BS. Thus, the end-to-end TCP semantic is preserved. Before we delve into the detail design of NS, four major advantages over Snoop are summarized as follows. 1) Caching packets at both MSC and BS. NS at the BS focuses on the local retransmission of packets lost at the wireless link. NS at the MSC aims at handling unacknowledged packets during handoffs. 2) New measures for preventing BS congestion. The wireless link is usually the bottleneck in heterogeneous networks due to its narrow bandwidth and high processing delay. When congestion occurs at a BS, packets will be lost due to cache overflow. This has been ignored by original Snoop. In NS, new measures are designed to alleviate the congestion loss at BS. 3) Local fast retransmission of out-of-order packets at the BS. For both Snoop and NS, a cached packet will not be released from the BS buffer until an ACK that confirms its correct receipt by the mobile host is received. If an out-of-order packet arrives at a BS and is not forwarded to the mobile host in time, the local cache at a BS will overflow with a higher probability. In NS, priority is given to sending out-of-order packets, or fast retransmission of out-of-order packets. As a result, cached packets can be released faster to make room for new packets. 4) New ACK adjustment scheme for smoothing out the latency variation due to local loss recovery. When a packet is corrupted on the wireless link, BS retransmits the packet while holding the duplicate ACKs generated by the mobile host for this packet. This causes the latency gap, thus RTT estimation, between two successive ACKs as seen by the TCP sender to fluctuate significantly. This in turn, results in an inaccurate setting of retransmit timeout (RTO) timer at the sender, leading to unnecessary sender timeouts. To avoid this, an ACK adjustment scheme at the BS to smooth out the RTT delay variation is designed in NS. B. Loss Recovery for Packets Lost Due to Wireless Transmission Errors Like Snoop protocol [8], [14], NS caches forwarded, but not-yet-acknowledged packets at BS for possible local loss recovery over the wireless links. A cached packet is cleared from the buffer only if an ACK that confirms its correct receipt by the mobile host is received. However, the ways to handle the local buffer overflow using Snoop and our proposed NS are different. Using NS, a shared buffer for both packets waiting for transmission and packets waiting for ACKs (i.e., cached packets) is used. When a new packet arrives at the BS, it is discarded if the shared buffer is full. Otherwise, it waits in the shared buffer for its turn to be transmitted. Upon transmission, a copy of the transmitted packet remains in the buffer and becomes a cached packet. The cached packet is cleared from the buffer only if the corresponding ACK confirmed its correct receipt is received. Therefore, if a packet is corrupted in the wireless link, a copy of it can always be found in the BS buffer. Then, the BS will retransmit the corrupted packet and the original sender will be completely shielded from wireless link errors. Another advantage of shared buffer is the buffer utilization will be greatly improved because complete resource sharing policy is used. In NS when BS receives a sender retransmitted packet, as detected by its out-of-order packet sequence number, it will be forwarded to the mobile host with a higher priority. Once it has been forwarded, all subsequently received duplicate ACKs on the corresponding connection will be suppressed to temporarily suspend the sender s transmission rate. This can relax the potential congestion at the BS. It should be noticed that a retransmitted packet here should be considered as caused by congestion, not by wireless link error. This is because local loss recovery at the BS should have recovered all losses due to wireless link error. In [8] and [14], the authors did not provide a detailed description of how packets are buffered at the BS in Snoop. Suppose the buffer at the BS is divided into two portions, output buffer and cache buffer. A packet arrives and waits in the output buffer for transmission to the mobile host. Upon transmission, a copy of the transmitted packet (cached packet) is stored at the cache buffer until the ACK that confirms its correct receipt by the mobile host is received. Since the size of the cache buffer is fixed, a transmitted packet cannot be cached if the cache is full. Therefore, using Snoop does not guarantee a packet corrupted at the wireless link can be found at the BS. If this happens, the BS must forward the received duplicate ACKs to the original sender and this will trigger a fast-retransmit at the sender for recovering the loss later on. Unfortunately, in this case the loss caused by wireless transmission errors will be wrongly interpreted by the sender as traffic congestion. If the shared BS buffer approach is also adopted by Snoop, packets will be dropped when the shared buffer is full. When this happens, the subsequently arrived packets will be cached in the buffer even though they may have already been correctly received by the mobile host. Since ACK in TCP is cumulative, only duplicate ACKs corresponding to the earlier dropped packet will be generated by the mobile host, which cannot be used to release the cached packets from the BS buffer. Consequently, the BS buffer will be packed with correctly received, but not-yet-acknowledged packets. Later on when the sender retransmitted repair arrives, very likely it will be dropped again because of the fully occupied/packed BS buffer. This works like a positive feedback loop, the throughput of wireless link

5 HU et al.: HIERARCHICAL CACHE DESIGN FOR ENHANCING TCP OVER HETEROGENEOUS NETWORKS WITH WIRED AND WIRELESS LINKS 209 Fig. 3. The test model for congestion. Fig. 4. Throughput ratio between using snoop and new snoop protocols and using TCP Reno. will be significantly degraded. We call this problem congestion expansion. The impact of congestion expansion can be demonstrated using a simple example based on the heterogeneous network shown in Fig. 3. Each of the two TCP senders sends a 50-MB file to the sink through the BS. The bandwidth of the wireless link is 2 Mb/s and 76.8 kb/s, respectively. The BS buffer size is 64 packets and we assume the shared buffer scheme is used by both Snoop and NS. The TCP packet size is fixed and of size 512 B. The resulting throughput performance of both NS and Snoop against wireless packet loss rate (PLR) is shown in Fig. 4. Due to congestion expansion, we can see that the throughput of Snoop is much poorer than that of NS. In fact, due to congestion expansion, Snoop s throughput deteriorates with the decrease in PLR. On the other hand, since NS has taken the congestion into its design consideration, to be detailed later in this section and, therefore, it does not suffer from congestion expansion. Recall that the TCP protocol is a sliding window scheme implemented at the sender. Each TCP packet has an associated sequence number. A TCP packet is identified uniquely by its sequence number and size. For implementation, we propose to add a three-bit tag to each packet arrived/buffered at the BS. This tag is used only at the BS to indicate the status of the buffered packet, where the first bit of the tag denotes the associated packet is waiting for transmission or waiting for ACK ; the second bit denotes the packet is an out-of-order packet or an in-order packet ; and the third bit denotes the packet is a retransmitted packet or not. When a new packet arrives at the BS, it is assigned a tag with value 000 or 010, denoting the packet is waiting for its turn of transmission (to the mobile host). Tag 000 is used if it is an in-order packet, meaning that its sequence number is one plus that of the last received packet. Tag 010 is used if the new packet is an out-of-order packet. Upon transmission, the tag of the transmitted packet is converted to 100 or 110 to denote that this packet is a cached copy. It remains in the buffer until an ACK confirmed its correct reception by the mobile host is received. Finally, a packet is marked as a retransmitted packet with tag (where means do not care, it can be either zero or one) if it has been retransmitted by the sender or by the BS. Note that a retransmitted packet must be an out-of-order packet. Depending on the tag value, BS carries out the corresponding packet processing procedure, or ACK processing procedure as described as follows. 1) Data Packet Processing Procedure: Fig. 5 summarizes the flow chart for packet processing at BS. To be more specific, the following scenarios are considered. 1) When an in-sequence packet arrives and finds no free buffer at the BS, the packet is immediately discarded. Otherwise, attach tag 000 to the packet and place the packet to the transmission queue. When the packet is transmitted via the wireless link later, a timestamp is attached to its cached copy (for measuring the round-trip-time of the wireless loop as described later) and its tag becomes ) When an out-of-order packet that has not been cached before (i.e., no copy of this packet can be found at the BS) arrives and local buffer is available, attach tag 010 to the packet and place the packet to the transmission queue. Upon transmission, its tag becomes 110. It should be noted that although this packet is out-of-order, there is no need to give this packet a higher transmission priority over other packets waiting in the transmission queue. 3) When an out-of-order packet that has not been cached before arrives and local buffer is not available, if the sequence number of the new packet does not equal to the sequence number of the last received ACK, the packet is discarded. Otherwise, the mobile host is expecting this packet. That means the mobile host is waiting for this packet in order to generate an ACK to release some packets from the fully occupied BS buffer, and the wireless link is idle at this time. The new packet is immediately transmitted to the mobile via the idle wireless channel, and a copy of it is stored at a special one-packet buffer (which is not explicitly shown in Fig. 2). In this case, it is reasonable to assume that this packet has been lost previously in the fixed network, and right now it is a retransmitted copy of the previously lost packet. Therefore, a tag 111 is attached to this cached packet. We refer this as local fast retransmission of out-of-order packet. 4) When an out-of-order packet arrives and a copy of this packet is found in the BS buffer, this newly arrived duplicate packet is immediately dropped. This happens when some previous packet corrupted and caused a fast packet retransmission. The long delay for recovering the previously lost packet causes the current packet (i.e., the one just arrived) to timeout at the sender (since no ACK for it was issued by the mobile host). However, the previous copy of this packet has already been transmitted to the

6 210 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 2, NO. 2, MARCH 2003 Fig. 5. Flow chart for packet processing at BS. (Note: A new packet is a packet that has not been acknowledged before and a copy of it cannot be found in the cache at the time of its arrival.) Fig. 6. Flow chart for ACK processing at BS. (Note: A new ACK is an ACK with seq. number different from the last received ACK.) mobile host and the BS is waiting for the corresponding ACK. 2) ACK Processing Procedure: NS processes ACK with two major differences from Snoop. First, using Snoop all out-oforder packets are marked as retransmitted packets. If an out-oforder-packet is lost in the wireless link, the BS must wait for local RTO before triggering a local packet retransmission. If the wireless link transmission rate is high (e.g., 2 Mb/s or 384 kb/s) and the lower layer protocol processing time is long, the throughput degrades quickly while waiting for timeout. Using NS an out-of-order packet is judged as lost at the wireless link when the third duplicate ACK for it arrives at the BS. (Note that for an in-order packet, the first duplicate ACK signifies that this packet has been lost in the wireless link and a local retransmission can, thus, be immediately triggered.) The BS retransmits the packet immediately and the successive duplicate ACKs are held by the BS (i.e., without forwarding to the sender) until a new ACK is received. Using this approach, the sender will only receive two duplicate ACKs for the packet lost at the wireless link and that would not inappropriately (like in Snoop) trigger fast retransmission at the sender. The local loss recovery at the BS will cause extra delay (and delay variation) for ACKs to reach the sender. The length of this extra delay is a function of the packet loss probability on the wireless link. It is important to have a good estimation of the round trip delay time at the sender. This can avoid the inaccurate setting of the RTO value and, thus, preventing unnecessary retransmission by the sender. To achieve this, the BS should have a mechanism to adjust the forwarding time of each ACK toward the sender. The detail of the ACK adjustment is described in Section IV-D. This is the second important difference from Snoop. Fig. 6 shows the flow chart at the BS for processing ACKs. The following pseudo code describes the steps to be followed

7 HU et al.: HIERARCHICAL CACHE DESIGN FOR ENHANCING TCP OVER HETEROGENEOUS NETWORKS WITH WIRED AND WIRELESS LINKS 211 when an ACK is received by the BS. (The ACK adjustment scheme is not included.) 1) When a new ACK is received, the acknowledged packets in the buffer (including the one-packet special buffer) are flushed. The RTT estimate for the wireless link is updated. The same ACK is then forwarded to the sender. 2) When a spurious ACK is received, it is discarded without forwarding. A spurious ACK is an ACK with a sequence number less than that of the last received ACK. 3) When a duplicate ACK is received, if the requested packet is not found in the buffer, forward the duplicate ACK to the sender. if the requested packet is found in the buffer, if its tag is 100 (i.e., it is an in-order packet), retransmit the requested packet and reset the retransmit timer. Then, mark the packet as retransmitted with tag, where means either zero or one. Hold the duplicate ACK. if its tag is 110 (i.e., it is an out-of-order packet), if DUP counter, DUP counter DUP counter. if DUP counter, retransmit the requested packet and reset the RTO. Then, mark the packet as retransmitted with tag. Hold the duplicate ACK. Reset DUP counter. if its tag is (i.e., it is a retransmitted packet), hold the duplicate ACK. 4) When the RTO times out, check the sequence number of the associated timeout packet. If its sequence number is the same as that of the last received ACK, retransmit the timed out packet and reset the RTO. Mark the packet as retransmitted with tag. DUP_counter is an integer counter that counts how many duplicated ACKs have been received if the requested packet is an out-of-order packet. When DUP counter, a local retransmission of the requested packet takes place. As we have mentioned before, Snoop protocol does not have this feature and as a result, the local loss recovery of all out-of-order packets must be triggered by timeout. In many cases, this has been proved to be very inefficient. C. Loss Recovery for Packets Lost During Handoff So far we have considered the packets lost/corrupted at the wireless link due to transmission errors. In fact, many packets are also lost during handoffs when the transmission channel is Fig. 7. NS. Handoff processing in two-layer hierarchical cache architecture using temporally not available. The characteristic of this loss, however, is different from the loss due to congestion in wired networks, or loss at the wireless links. Handoff processing time, which is the time for completing handoff procedure, may last for several tens of milliseconds or higher. Packet loss due to handoffs can cause the TCP sender to remain idle for long periods of time (even after the handoff is completed), resulting in unacceptably low throughput. Besides, the potential wrong retransmission timeout estimation at the sender can lead to further throughput degradation, as discussed in Section IV-D. Snoop does not consider problems caused by handoffs. In NS, we propose to tackle this by storing unacknowledged packets at MSC for fast local loss recovery. For simplicity, we can assume that the buffer sizes at both BS and MSC are equal for each TCP connection. Fig. 7 shows a typical scenario of packet transmissions during a handoff in a network using the two-layer hierarchical cache scheme. When a mobile host finds a BS with a stronger signal power [19], a handoff request is sent to MSC via the old base station (BS1). The MSC then reroutes the TCP connection to the new base station (BS2). The MSC keeps track which packets have been sent to BS1 and during the handoff, it collects the following information from BS1: 1) which packets are waiting for ACKs from mobile host and 2) the associated local RTO values for packets waiting for ACKs. Then, the MSC sends all unacknowledged packets from its local cache plus their RTOs collected from BS1 to BS2. Then, BS2 takes over the role from BS1. The handoff is completed and the TCP connection is resumed. The buffer requirement at a MSC can be greatly reduced if a handoff can be predicted. Predicting a handoff is not difficult. We can, for example, start to cache the packets arrived at the MSC only when a handoff request from a mobile is received. If this duration is not enough, we can adjust the handoff initiation power threshold such that a handoff request can be initiated earlier. As a side remark, the third-generation mobile radio systems [1] are being standardized. These systems need to support a wide

8 212 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 2, NO. 2, MARCH 2003 Fig. 8. The GSM/UMTS architecture. Fig. 9. The Simulation Model. range of services including voice, data and multimedia services. In [1], a GSM/UMTS network subsystem (NSS) architecture as shown in Fig. 8 was proposed. This GSM/UMTS NSS is an evolution of the present GSM NSS. From Fig. 8, there are two modules in the core network at the MSC handling the traffic from PSTN/ISDN and Internet, respectively. Our proposed twolayer hierarchical cache architecture can nicely fit into it. D. ACK Adjustment for Handling Latency Variation Caused by Local Retransmission The TCP sender estimates the RTT of a connection by measuring the time between the sending of a packet and the receipt of the ACK for this packet. For each congestion window (CWND), the TCP sender can get a sample of the RTT. The new RTT and its deviation are calculated below RTT RTT (1) RTT (2) where is the mean deviation of RTT; is the sample of the RTT; is the control factor. The optimal value of (as suggested by RFC of IETF) equals to in order to have small oscillation between two RTTs. In TCP protocol, the RTO value of the sender is calculated as a function of the average and mean deviation of the RTT, where RTO=RTT (3) Most TCP implementations use a 500 ms timer granularity for the retransmission timeout. In heterogeneous networks, the local retransmission at the wireless link can cause the end-to-end RTT to increase significantly in a short time due to the long processing time and the low data transmission rate on the wireless link. As a result, big time gaps in the receiving ACK sequence are observed at the sender when local retransmission occurs. When there is no packet loss (thus, no local retransmission), the estimated delay variation will be small/normal and the RTO timer is set accordingly using (4). When a packet loss occurs subsequently (so local retransmission is triggered), the estimated RTO may not be large enough to prevent the sender from timeout. In this case, the timeout is unnecessary. In NS, we propose to use an ACK adjustment scheme to solve this problem. Each ACK that is to be forwarded to the sender will be held or delayed for a short period of time at the BS. This is to adjust the interval between successive ACKs. The sender will get a longer but more stable (i.e., with less delay variation) RTT measurement, and the probability of inappropriate timeout at the sender can then be reduced. The ACK delay interval,, is calculated as a function of the RTT and the packet loss probability of the wireless link, or RTT (4) where is the packet loss probability of the wireless link; RTT is the RTT of the wireless link. RTT is measured by the timestamp of each packet in the local buffer cache at the BS using the same formula in (1) above. When, BS will not hold any ACK since no local loss recovery is needed. When approaches 1, that means almost every packet needs to be locally recovered. In this case, the end-to-end delay should be increased by at least one RTT in order to get the packet recovered. In Section V-D, the impact of using ACK adjustment will be studied. V. PERFORMANCE EVALUATIONS In this section, we study the performance of NS under the hierarchical cache architecture using computer simulations. Fig. 9 shows our simulated network, which consists of an ISP server, a MSC, two BSs, and a mobile host. We assume that the mobile host is moving in one direction from left to right. A handoff occurs at the boundary between two cells that are covered by two adjacent base stations. Major simulation parameters are summarized in Table I. We assume that the links in the fixed network (i.e., Link_S_M and Link_M_B) are error-free, and the congestion in the fixed network only occurs at the BS. We also assume that some error-correction schemes have been implemented at physical and data link layers, therefore, the packet lost probability as seen by the TCP layer is a uniformly distributed random process. Since ACK packets are relatively small in size, the effect of ACK loss is ignored. For comparison, TCP Reno, New Reno, Reno with Snoop/NS, and New Reno with Snoop/NS are implemented. For all protocols except those with Snoop option, the BS is assumed to have a fixed (shared) buffer size of Size packets. If we also assume Snoop uses a shared buffer, Snoop

9 HU et al.: HIERARCHICAL CACHE DESIGN FOR ENHANCING TCP OVER HETEROGENEOUS NETWORKS WITH WIRED AND WIRELESS LINKS 213 TABLE I SIMULATION PARAMETERS will suffer from the serious congestion expansion problem, and its performance will be much poorer than that of using dedicated buffer approach. (Please refer to the example shown in Figs. 3 and 4.) In order not to be biased against Snoop, we assume a Snoop BS has dedicated output buffer and cache buffer, where the output buffer size is Size packets, and the extra cache buffer is infinite. A. Performance in Recovering Losses Caused by Wireless Transmission Error We first study the performance of NS in recovering losses caused by wireless transmission errors. (Note that the ACK adjustment scheme has been combined into NS.) The PLR of the wireless link under consideration is form 5 10 to We simulate the case that a mobile host retrieves a buck data file of 50 MB from the fixed server. Figs show the throughput of the wireless link with rates 384 kb/s, 2 Mb/s, and 144 kb/s, respectively. From them, we can see that using NS can significantly improve the throughput when PLR is higher than This indicates that the packet loss due to wireless link errors have been effectively recovered by NS. From PLR of 10 to 10, the throughput improvement is from times as compared with that of the pure TCP Reno and New Reno in Fig. 10, times in Fig. 11, and times in Fig. 12. We also find that the throughput of TCP New Reno is higher than that of TCP Reno, as New Reno can retransmit multiple packets in one fast recovery procedure 1 while Reno cannot. If there are multiple packets lost in one CWND, SSTHRESH will become very small after several fast recovery procedures using Reno. Comparing Figs , we can see that the throughput using 2 Mb/s is lower than that of 144 kb/s under high-packet loss conditions, while the vice versa is true under low-packet loss conditions. When the wireless link speed is high, more unacknowledged packets will be cached in the local BS buffer. If 1 For details, please refer to the literature on Reno and New Reno, e.g., [18]. the PLR is high, BS buffer will be overflowed easily. Then, the TCP sender will be forced into the fast retransmission and fast recovery procedures to deal with the lost packets. As a result, both SSTHRESH and CWND shrink and the throughput of the wireless link decreases. When the PLR is low enough, higher speed links can then achieve a higher throughput due to low buffer overflow probability. From Fig. 10, we can also see that the throughput of using NS is higher than that of Snoop. For PLR from 10 to 5 10, about and throughput improvements are obtained using New Reno and Reno, respectively. We also notice that the improvement using Reno is more than New Reno. This is because if multiple packets are lost due to buffer overflow, Reno has a higher probability to cause sender timeout. When PLR is low, the difference between Reno and New Reno is small. B. Performance in Recovering Losses Caused by Handoff Next, we focus on the performance of using NS to recover the packet losses during handoff. Assume that the mobile host moves at a constant speed from left to right as shown in Fig. 9. The handoff frequency, or the interhandoff arrival time, depends on the mobile host moving speed. Without explicitly modeling the mobile moving speed, we vary the interhandoff arrival time from 30 s to 180 s. For comparison, the handoff performance of Snoop is also obtained. The handoff processing time is assumed to be 20 ms for NS using the hierarchical cache architecture and 20 ms plus the packet transmission time (from the old BS to the new BS) for Snoop. Packet loss due to handoff is not considered for Snoop protocol. (If packet loss caused by handoff is considered, the performance of Snoop protocol will be even poorer.) Two wireless link PLRs are considered, 5 10 in Fig. 13 and 5 10 in Fig. 14. One corresponds to a high PLR case and the other corresponds to a low PLR case. Again results for three wireless link transfer rates of 144 kb/s, 384 kb/s, and 2 Mb/s are obtained. The TCP protocol used is TCP New Reno. From Fig. 13, we can see that the hierarchical cache scheme improves the wireless link throughput by 20%, 5%, and 2% as compared with that of using Snoop protocol at rates 384 kb/s, 2 Mb/s, and 144 kb/s, respectively. While keeping the wireless link PLR at 10, we compare Figs and find that handoff has a stronger performance impact to the case with 384 kb/s rate. The throughput drops about 20%, while the throughput drops for 2 Mb/s and 144 kb/s are about 1% and 0.5%, respectively. The reasons are as follows. Using 2 Mb/s link, it is likely that more unacknowledged packets are stored at the BSs local cache. Thus, more packets are discarded by the BS due to cache overflow. As a result the sender retransmits more packets during the fast recovery procedure because New Reno is used. Therefore, the throughput is mainly affected by the congestion in the fixed network (i.e., at the BS) not handoff. Using 144 kb/s link, BS can forward three packets during one wireless RTT (based on the parameters in Table I). Using 384 kb/s link, it can forward ten packets. Therefore, the handoff on throughput performance using 144 kb/s link is less serious. From Fig. 14 when the wireless link PLR is 5 10,wecan see that hierarchical cache scheme improves the throughput by

10 214 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 2, NO. 2, MARCH 2003 Fig. 10. Recovering losses caused by wireless link errors; 384 kb/s wireless link. Fig. 11. Recovering losses caused by wireless link errors; 2 Mb/s wireless link. Fig. 12. Recovering losses caused by wireless link errors; 144 kb/s wireless link. Fig. 13. Performance of using NS to recover losses caused by handoff; PLR =52 10., and as compared with that of using Snoop, at rates 384 kb/s, 2 Mb/s, and 144 kb/s, respectively. When the congestion is not the main cause for throughput degradation, the improvement of using hierarchical cache scheme becomes more significant, especially for 2 Mb/s link. Figs. 13 and 14 also show that the effect of the interhandoff arrival time is insignificant. This is because the time the sender needs to reach the steady state from slow start or fast recovery is far less than that of the interhandoff arrival time. Overall speaking, we can see that using hierarchical cache scheme, the handoff performance is much better than that of using Snoop.

11 HU et al.: HIERARCHICAL CACHE DESIGN FOR ENHANCING TCP OVER HETEROGENEOUS NETWORKS WITH WIRED AND WIRELESS LINKS 215 Fig. 14. Performance of using NS to recover losses caused by handoff; PLR = Fig. 15. Performance of using NS to recover losses caused by wireless link errors; 9.6 kb/s link. Fig. 16. Performance of using NS to recover losses caused by handoffs; 9.6 kb/s link. C. Performance of NS at a Low-Speed Wireless Link For existing cellular systems, the wireless link transfer rate is much lower and the typical transfer rate for data services using GSM is 9.6 kb/s. Therefore, we focus on the performance of the hierarchical cache scheme with low wireless link rate in this section. The three previous simulations are repeated here with a wireless link speed of 9.6 kb/s. The size of the download file is 500 kb for minimizing the effect of slow start on the throughput. The throughput against PLR is shown in Fig. 15. From it, we can see that using NS improves the throughput by 20 times when the PLR is higher than When the PLR is lower than 10, the performance of New Reno with NS is poorer than that without NS, and the performance of Reno with and without NS is comparable. Since the wireless link speed is rather low, the packet transmission time over wireless link is more than 600 ms (including 100 ms process delay). The probability of local cache overflow is, thus, higher when NS is used. Fortunately, GSM system is designed to provide voice services, its wireless PLR is relatively high (10 10 ). In this range, NS protocol can provide excellent performance. The throughput performance of using hierarchical cache scheme for handling handoffs is shown in Fig. 16. Similar to results in Section V-B, the improvement of using NS at low PLR is higher than that at high PLR. This is because when PLR is high, the main cause for throughput drop is due to congestion loss rather than handoffs. From the simulation results shown in Figs. 15 and 16, we can see that NS with hierarchical cache architecture can significantly improve the performance of the existing low bandwidth mobile systems. D. Performance of ACK Adjustment As described in Section IV-D, ACK Adjustment scheme is used in NS to reduce the timeout probability of the TCP sender. The NS protocol with and without ACK adjustment is studied in this section. We focus on the throughput difference between NS

12 216 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 2, NO. 2, MARCH 2003 Fig. 17. Performance of ACK adjustment in NS. with and without ACK adjustment. The wireless link transfer rates are 144 kb/s, 384 kb/s, and 2 Mb/s and the wireless link PLR is from 5 10 to The performance of ACK adjustment is summarized in Fig. 17. We can see that the throughput using NS with ACK adjustment is consistently better than that without ACK adjustment. The gain is about 1% when the wireless PLR is less than 10. When the PLR is high, throughput gain can be obtained. One would expect ACK adjustment scheme to give a better improvement when PLR is high. This is because the variation in ACK latency increases with the PLR because of the frequent local loss recovery. But on the other hand, TCP sender uses a coarse RTT estimation and the timer granularity is 500 ms, the enhancement of ACK adjustment is limited. VI. CONCLUSION In this paper, we have proposed a two-layer hierarchical cache architecture for enhancing the TCP performance in a heterogeneous network with both wired and wireless links. A new network layer protocol called NS was designed for implementing at both BS and MSC. The main idea is to cache packets at these two places for effectively recovering retransmissions caused by wireless link errors, as well as handoffs. We showed that the new scheme keeps the end-to-end TCP semantics and is compatible with existing TCP applications. The performance of the NS has been studied with different wireless link transfer rates. Simulation results showed that the proposed scheme is more robust in dealing with unreliable wireless links, as well as losses caused by handoffs than existing protocols. We also found that the proposed scheme can significantly improve wireless link throughput when wireless PLR is high ( 10 or 10 ), and the improvement for high speed wireless links is more significant than that for low speed wireless links. [5] H. Balakrishnan. (1996) An implementation of TCP selective acknowledgments. [Online]. Available: FTP: //daedalus.cs.berkeley.edu/ pub/tcpsack/ [6] H. Balakrishnan, V. N. Padmanabhan, and R. H. Katz, The effects of asymmetry on TCP performance, presented at the ACM MOBICOM 97, Budapest, Hungary, Sept [7] H. Balakrishnan, V. N. Padmanabhan, S. Seshan, M. Stemm, and R. H. Katz, TCP behavior of a busy web server: Analysis and improvements, Proc. IEEE INFOCOM, vol. 1, pp , Mar [8] H. Balakrishnan, S. Seshan, and R. H. Katz, Improving reliable transport and handoff performance in cellular wireless networks, ACM Wireless Networks, vol. 1, no. 4, pp , Dec [9] R. Caceres and L. Iftode, Improving the performance of reliable transport protocols in mobile computing environments, IEEE J. Select. Areas Commun., vol. 13, pp , June [10] R. Caceres and V. N. Padmanabhan, Fast and scalable handoffs in wireless internetworks, presented at the 1st ACM Conf. Mobile Computing and Networking, Berkeley, CA, Nov [11] A. DeSimone, M. C. Chuah, and O. C. Yue, Through-put performance of transport-layer protocols over wireless LANs, in Proc. GLOBECOM 93, Dec. 1993, pp [12] R. Yavatkar and N. Bhagwat, Improving end-to-end performance of TCP over mobile internetworks, in Proc. Mobile 94 Workshop on Mobile Computing Systems and Applications, Dec. 1994, pp [13] K. Fall and S. Floyd, Simulation-based comparisons of Tahoe, Reno, and SACK TCP, Comput. Commun. Rev., pp. 5 21, [14] H. Balakrishnan, S. Seshan, and R. H. Katz, Improving TCP/IP performance over wireless networks, presented at the 1st ACM Int. Conf. Mobile Computing and Networking (Mobicom), Berkeley, CA, Nov [15] W. Stevens, TCP slow start, congestion avoidance, fast retransmit and fast recovery, Internet Document RFC-2001, Jan [16] B. Bakshi et al., Improving Performance of TCP Over Wireless Networks, Texas A&M Univ., College Station, TX, Tech. Rep. TR , [17] P. Karn and C. Partidge, Improving round trip time estimates in reliable transport protocols, presented at the ACM SIGCOMM, Stoweflake Resort, Stowe, Aug [18] J. C. Hoe, Improving the start-up behavior of a congestion control for TCP, presented at the ACM SIGCOMM, Stanford, Ca, [19] T.-S. P. Yum and K. L. Yeung, Blocking and handoff performance analysis of directed retry in cellular mobile systems, IEEE Trans. Veh. Technol., vol. 44, pp , Aug REFERENCES [1] E. Berruto et al., Research activities on UMTS radio interface, network architectures, and planning, IEEE Commun. Mag., pp , Feb [2] E. Ayanoglu et al., A link-layer protocol for wireless networks, ACM/Baltzer Wireless Networks J., vol. 1, pp , Feb [3] A. Bakre and B. R. Badrinath, Handoff and system support for indirect TCP/IP, presented at the 2nd Usenix Symp. on Mobile and Location- Independent Computing, Ann Arbor, MI, Apr [4], I-TCP: Indirect TCP for mobile hosts, in Proc. 15th Int. Conf. Distributed Computing Systems (ICDCS), May 1995, pp Jian-Hao Hu was born in Yunnan, China, in He received the Ph.D. degree in communications from the University of Science and Technology of China (UESTC), Chengdu, China, in His Ph.D. thesis was on mobile satellite communication systems design. From 1999 to 2000, he was a Postdoctoral at City University of Hong Kong, and researched on mobile communication systems. Since 2000, he joined The University of Hong Kong, Hong Kong, where he is currently working on cdma 2000 system development in the university s 3G research center.

13 HU et al.: HIERARCHICAL CACHE DESIGN FOR ENHANCING TCP OVER HETEROGENEOUS NETWORKS WITH WIRED AND WIRELESS LINKS 217 Gang Feng (M 01 A 01) received the B.Eng. and M.Eng. degrees in electronic engineering from the University of Electronic Science and Technology of China, in 1986 and 1989, respectively, and the Ph.D. degree in information engineering from The Chinese University of Hong Kong, Hong Kong, in He worked for about one year in the Department of Electronic Engineering, City University of Hong Kong as a Senior Research Assistant. From 1989 to 1995, he was with the University of Electronic Science and Technology of China. Currently, he is an Assistant Professor in the Information Communication Institute of Singapore, Nanyang Technological University, Singapore. His research interests include routing and performance evaluation for high-speed networks, TCP enhancement over heterogeneous networks, and wireless networks. Recently, his work included reliable multicast, active network, flow and congestion control in the Internet. Kwan Lawrence Yeung (S 93 M 95 SM 99) received the B.Eng. and Ph.D. degrees in information engineering from The Chinese University of Hong Kong, Hong Kong, in 1992 and 1995, respectively. He joined the Department of Electrical and Electronic Engineering, The University of Hong Kong, in July 2000, where he is currently an Associate Professor. Before that, he has spent five years in the Department of Electronic Engineering, City University of Hong Kong as an Assistant Professor. During the summer of 1993, he visited the Performance Analysis Department, AT&T Bell Laboratories (now Bell Labs, Lucent Technologies), Holmdel, NJ. His research interests include personal and cellular mobile systems, optical networks, broadband packet switches, high-speed networking, and the next-generation Internet. Dr. Yeung received the Outstanding Young Researcher Awards in from The University of Hong Kong. He was ranked First Place in 1993 IEEE (HK) Student Paper Contest (postgraduate).