5 Amercan Control Conference June 8-, 5. Portland, OR, USA FrA3. Neural Network Control for TCP Network Congeston Hyun C. Cho, M. Sam Fadal, Hyunjeong Lee Electrcal Engneerng/6, Unversty of Nevada, Reno, NV 89557 Abstract - Actve Queue Management (AQM) has been wdely used for congeston avodance n Transmsson Control Protocol (TCP) networks. Although numerous AQM schemes have been proposed to regulate a queue sze close to a reference level, most of them are ncapable of adequately adaptng to TCP network dynamcs due to TCP s non-lnearty and tme-varyng stochastc propertes. To allevate these problems, we ntroduce an AQM technque based on a dynamc neural network usng the Back-Propagaton (BP) algorthm. The dynamc neural network s desgned to perform as a robust adaptve feedback controller for TCP dynamcs after an adequate tranng perod. We evaluate the performances of the proposed neural network AQM approach usng smulaton experments. The proposed approach yelds superor performance wth faster transent tme, larger throughput, and hgher lnk utlzaton compared to two exstng schemes: Random Early Detecton (RED) and Proportonal-Integral (PI)-based AQM. The neural AQM outperformed and RED, especally n transent state and TCP dynamcs varaton. I. INTRODUCTION The essence of congeston control strateges for TCP networks s to rapdly recover from network congeston, or to prevent an ncpent congeston. Ths can be acheved by dynamcally adjustng wndow sze at the source sde or controllng ncomng packets to a router at the lnk sde. Numerous TCP schemes that optmally adjust wndow sze for congeston avodance have been explored n the last decade. The frst wdely used scheme, TCP Tahoe, was later modfed to TCP Reno [], currently the most popular TCP. The congeston wndow n these protocols s based on the Addtve Increase Multplcatve Decrease (AIMD) algorthm: congeston wndow sze s ncreased by one packet per acknowledgement (ACK) but s halved f a source receves three duplcate ACK sgnals or does not receve any ACK wthn a gven round-trp tme (see []). Snce the development of TCP Reno, several researchers have suggested addtonal TCP functons to mprove network performance []. Whereas these algorthms operate at the source sde, AQM s mplemented at the lnk sde, especally for ncpent congeston avodance. In other words, AQM provdes congeston nformaton acqured from the lnk sde to the sources. The objectve of an AQM s prmarly to proactvely respond to network congeston as ts queue begns to ncrease. Rather than smply watng for a congested queue to overflow and then tal drop all subsequently arrvng packets, t mantans queue sze at a predefned level n the router. RED [3] s a popular example of an AQM scheme. In RED, the router calculates the drop probablty usng a current queue sze. The ncomng packets are passed, dropped or marked, based on ths probablty. By dscardng or markng a sngle packet, the router sends an mplct or explct warnng to the source. As a response to the warnng, the source s expected to adjust the congeston wndow sze to reduce ts transmsson rate. The drop probablty s often lnearly proportonal to queue length. Although RED s an effectve TCP congeston control [4], t can nduce network nstablty and major traffc dsrupton f not properly confgured. Hence, optmal parameter selecton for RED desgn under dfferent congeston scenaros has been a problem. Moreover, even f optmally selected, the parameter values must be adjusted n real-tme mplementatons because TCP dynamcs change wth the number of actve TCP flows. Many studes have addressed optmal parameter selecton for RED and ts varants. Floyd et al. proposed approprate parameter ranges n [5] and presented an adaptve RED n whch the best parameter settngs are based upon a traffc mx flowng through the router [6]. In [7], the authors proposed Random Exponental Markng (REM) as a modfcaton to RED. Ther am was to stablze the nput rate around the lnk capacty and mantan average queue sze around a small reference level. Feng et al. [8] presented a self-confgurng RED that adjusted the maxmum drop probablty accordng to the past hstory of the average queue sze. They also proposed BLUE [9], a new AQM mechansm. BLUE uses buffer overflow and lnk dle events, together wth average queue sze, to control congeston. Another approach, Stablzed RED (SRED) [], computes the drop probablty based on the estmated number of actve TCP flows and the nstantaneous average queue sze. In recent years, control-theoretc AQM approaches have been proposed, mostly usng lnear classcal control. In [], desgn gudelnes were proposed for choosng AQM parameters based on Proportonal (P) and Proportonal-Integral (PI) control. The authors lnearzed the nonlnear dfferental equaton TCP network model of -783-998-9/5/$5. 5 AACC 348
[] at an operatng pont to derve a transfer functon for P and ler desgn. Ther AQM scheme was compared to RED and found to be superor. Km and Low [3] formulated an AQM desgn problem for stablzng a gven TCP network descrbed by state-space models and proposed Proportonal-Dervatve (PD) and Proportonal-Integral-Dervatve (PID) AQM strateges. In [4], Dynamc-RED (DRED) was proposed to stablze queue dynamcs even f the number of actve TCP connectons s dynamcally vared. DRED ams to mantan queue sze close to a reference queue level by a dscrete ntegral control approach. Recently, more complex control methodologes have been proposed for AQM. Quet and Ozbay [5] appled H controller to AQM usng the lnearzed the TCP model of []. They derved the transfer functon of the controller and showed through computer smulaton that the proposed AQM was superor to PI control and RED. We note that the choce of control parameters s the key to satsfactory performance of a feedback control system. However, n practce, parameter choces for the nomnal model may be suboptmal due to system uncertanty or perturbaton. Thus, parameter values must be adjusted to adapt to operatonal changes. In addton, most control-theoretc AQM proposed to date are based on lnear models whle TCP networks are nonlnear tme-varyng stochastc systems. We present a more sophstcated adaptve control strategy for AQM n TCP networks usng a dynamc artfcal neural network AQM control. The control can promptly adapt ts operaton to the nonlnear tme-varyng and stochastc nature of TCP networks. Neural networks have been wdely appled n the last two decades n a varety of engneerng felds such lke sgnal processng, process control, communcaton systems, etc. [6]. They are teratvely traned by a proper learnng algorthm to mnmze a selected performance measure. As a result, unlke RED and classcal lnear control based approaches, neural networks are able to determne the optmal AQM system parameters values autonomously after adequate tranng. Followng tranng, the neural network operates as an adaptve and robust controller that can provde excellent performance even for envronmental condtons not ncluded n the tranng data set. The dynamc neural network controller presented n ths paper s traned to regulate the actual queue sze close to a reference value determned by network requrements. After tranng, the neural network operates as an adaptve controller under changes n TCP dynamcs. We choose a mult-layer recurrent (ncludng feedbac dynamc neural model because of ts well-known advantages. Ths model has been popular snce the md 99 s n many applcatons for dynamcal tme-varyng and nonlnear systems [7]. There are manly two methods for tranng recurrent neural networks: a back-propagaton-trough-tme algorthm [8] and a real-tme recurrent learnng algorthm [9]. For smplcty, we derve a learnng procedure by the general back-propagaton (BP) method [6]. To evaluate the proposed neural AQM, a TCP network topology ncludng a smple bottleneck, two routers on the lnk sde, and multple TCP sessons, s consdered. We use the TCP system model of [] for our neural AQM analyss and llustrate the advantages of our proposed methodology as compared to RED and PI-based AQM. The outlne of ths paper s as follows. In Secton II, we present our neural network AQM TCP congeston control. A learnng algorthm wth BP s derved for ths model n Secton III. Smulaton results and dscusson are gven n Secton IV. Our conclusons are gven n Secton V. II. NEURAL NETWORK AQM The block dagram of TCP congeston control wth the neural network AQM proposed n ths paper s shown n Fg.. q * - e p w Neural Network TCP Source Router Fg.. Neural network AQM of TCP network. In Fg., the congeston wndow sze, w of the TCP source s determned by the probablty, p calculated from the neural network. Queue dynamcs at the lnk sde s affected by w. The neural network control system mnmzes the error sgnal, e between the actual queue sze, q and the reference queue target value, q *. The loss probablty, p s the control nput to the TCP source. The neural network model used n ths paper s shown n Fg.. V e + Fg.. Recurrent neural network for AQM. We select a dynamc recurrent neural model ncludng one feedback connecton and a three-layer perceptron. The nput vector of ths neural network ncludes the error sgnal, e and the probablty, p as a feedback sgnal from ts output. Thus, the nput vector, u s gven by u e p T () The weght matrx n the frst layer s v v V () v m v m where m denotes the number of nodes n the second layer. In (), the frst column and the second column are related to the error sgnal and the feedback probablty, respectvely. + b Z - y q p 348
The weght vector n the second layer s (3 T m ) Thus, the dynamc behavor of the network s gven by T y y Vu k k b (4) where s the feedback gan, k denotes dscrete tme, and b s a bas connected wth unt nput. Fnally, the network output s obtaned from the actvaton functon p ( y), exp( ay) a (5) where a s a constant scalng factor. III. LEARNING PROCEDURE The neural network desgned n Secton II must be traned to optmze a TCP network performance measure. Durng network tranng, the weghts and the bas are teratvely updated untl they reach ther optmal values. In ths secton, we present a BP learnng algorthm for the proposed network and derve the rules for updatng the network weghts and bas. The tranng objectve s to mnmze the error sgnal J defned as * J q q (6) Adjustments of the weght matrx V, the weght vector, and the bas b, are governed by the delta rule as follows J vj (7) v j J (8) J b (9) b where s the learnng rate, =,,m, and j=,. The partal dfferental equatons n the rght sde of (7), (8), and (9) are expanded respectvely by usng the chan rule as J J q p y () v q p y j v j J J q p y () q p y J J q p y () b q p y b We calculate the three common terms n (), (), and () usng (5), (6), and the followng approxmatons [] J ( q * q) (3) q p a exp( ay) (4) y exp( ay) q q( q( k ) (5) p p( p( k ) These approxmatons descrbe the Jacoban of the TCP system. We use an approxmaton of the TCP system equaton assumng that t s not provded. The approxmate dervatve n (5) s determned by changng the nput p and the output q []. The rght hand sdes of (), (), and () nclude dervatves of y obtaned usng the followng equatons y( k ) u j (6) v ( j y( k ) ( j v u j j (7) y( k ) (8) b( By substtutng (3)-(8) n (7), (8), and (9), we fnally obtan the update rules v u j j (9) v u j j () j b () where * q( q( k ) a exp( ay) q q p( p( k ) exp( ay) () IV. SIMULATIONS We conducted a smulaton study to evaluate the performance of neural network AQM. We consdered the TCP network topology of Fg. 3, ncludng a smple bottleneck lnk between two routers and numerous TCP flows. n Sources Router Bottleneck Lnk Router Fg. 3. TCP Network model. n Destnatons The mathematcal model used for AQM desgn and smulaton s a flud-flow expresson []. The model descrbes the dynamcs of a queue and a congeston wndow wth the nonlnear dfferental equatons w( w( t R( ) w ( p( t R( ) R( R( t R( ) (3) N( x( C f N( x( C q ( otherwse (4) where C s a lnk capacty, N s the number of TCP connectons, x( s a transmsson rate defned as w( / R(, and the round-trp tme R( s calculated by R ( q( / C, where s a random propagaton delay tme. The specfcatons of the TCP network are from [], but some parameter values are modfed for our smulaton 348
scenaros. We select the packet sze as 5 bytes, C as 5 Mb/sec, and a maxmum q n a router as 8 packets. s unformly dstrbuted n [.6,.4] sec. We smulate RED and under same smulaton scenaros as our control for comparson purposes. We selected optmal parameter values for RED and from teratve numercal analyses usng (3) and (4) under the gven TCP specfcatons wth 4 TCP connectons. In RED, the mnmum and maxmum thresholds of an averagng queue sze are and 5 packets respectvely, the maxmum drop probablty s., and the weght n the movng average equatons for computng the averagng queue s.3. The ler used n ths smulaton s gven by p k p e( k e( dt (5) where k p s the proportonal gan and k s the ntegral gan. We selected k p =5-7 and k =5-5 for our smulatons, whch are of the same order as the values of [] and [5]. For the AQM neural network, we use three nodes n the hdden layer and a learnng rate of.. The ntal values of the weghts n the frst and second layers, and the bas, were unformly dstrbuted n [, ]. The neural network was traned to determne the optmal weghts and bas under the same smulaton envronment as RED and PI AQM. After teratve network tranng wth randomly chosen ntal weghts and bas, the optmal weghts that mnmze our control performance measure were.5536.447 V.4545.6853 (6).7.765.39.777.387 (7) and the bas b was.8444. We ran four smulaton scenaros to evaluate the three dfferent AQM approaches: RED, PI control, and neural network control. We also tested the robustness and adaptve capacty of the three schemes. To llustrate dynamc queue responses applyng these AQM methods, we solve the dfferental equatons n (3) and (4) numercally. Case I: We used 4 TCP flows (N=4) and a reference queue sze of packets for and neural AQM. Tme hstores of the queue sze for the three AQMs are shown n Fg. 4. We observe an ntal overshoot n all three AQMs after whch the responses drop to ther steady-state values. The overshoot saturates at 8 packets due to the maxmum queue sze n the router. In the steady state, large oscllatons contnue for RED whle the responses for PI control and neural AQM oscllate very closely to the reference level. PI and neural AQM control have consderably dfferent transent responses. For, the settlng tme s about 5 sec and s three tmes that of the neural AQM, and transent saturaton occurred as n RED. Ths behavor s a very serous problem that can potentally T result n network congeston especally traffc status changes rapdly. By contrast, neural AQM has a much faster response but does not result n saturaton. The comparson ndcates that neural AQM provdes stable control and a better transent response than PI-based AQM. 8 6 5 5 5 8 6 5 (a) RED 5 5 (b) PI and neural AQM Fg. 4. Queue dynamcs for fxed N. Case II-: In real-tme TCP mplementatons, the number of TCP flows vares randomly. Thus, we smulated the system wth tme-varyng TCP flows,.e., the number of TCP connectons was vared. We assume that N s progressvely ncreased by every 5 sec,.e. N =4 n [, 5] sec, N =34 n [5, ] sec, N =44 n [, 5] sec, and N =54 n [5, ] sec. The reference queue sze n and neural AQM s stll set to packets. Fg. 5 shows the queue dynamcs for ths smulaton scenaro. Fg.5 (a) shows that the response n RED s very smlar to Case I. In Fg. 5(b), both PI and neural AQM stll have the ntal overshoots at the startng tme as well as at the tmes when N s changed. The response for ntally saturates and has a larger settlng tme than neural AQM. For both, the overshoot ncreases whle the undershoot decreases as N ncreases. These results show that neural AQM outperforms PI AQM for dynamc reference queue level. 3483
8 8 6 6 5 5 5 5 8 6 5 (a) RED 5 5 (b) PI and neural AQM Fg. 5. Queue dynamcs for ncreasng N. 5 5 t (b) PI and neural AQM Fg. 6. Queue dynamcs for decreasng N. Case III: The smulaton n ths case s for comparsons of the two feedback control schemes: PI-based AQM and neural AQM. RED s not ncluded because t s not a feedback control AQM scheme. We vary the reference queue sze, but keep N fxed at 4. We set a reference queue level, q * as packets both n [, 5] sec and [,5] sec, and as packets both n [5,] and [5,] sec. Fg. 7 shows the smulaton results for the queue dynamcs. These results ndcate that neural AQM has better performance than for varyng reference queue sze. 8 Case II-: Ths example s the opposte scenaro of Case II-. In ths case, the number of TCP flows are dynamcally 6 decreased, that s, N=54 n [, 5], N=44 n [5, ], N=34 n [, 5], and N=4 n [5, ]. Tme 5 hstores of the queue sze for these three AQMs are plotted n Fg. 6. In ths case, undershoots occurred n both PI and neural AQM but we agan observe that the neural AQM has a superor transent response to PI-based AQM. The superor transent response of the proposed soluton n ths case drectly explans the hgher throughput compared to 5 PI-based AQM because the system can quckly adjust tself 5 to fully utlze the avalable bandwdth due to less traffc. Fg. 7. Queue sze for PI and neural AQM for varyng q *. 8 6 5 Table shows the mean values and varances of the throughput and the queue sze for each AQM. The queue sze of neural AQM has the smallest mean values and the largest throughput n all smulatons. Hgher throughput mples more effcent utlzaton of the network lnk. The mean queue sze of neural AQM remans closer to the reference queue value than, and the varance of ts throughput s the smallest for all cases. Hence, neural AQM provdes more stable queue management. 5 5 (a) RED 3484
Table. Smulaton results for RED,, and neural AQM Case I II- II- III Control Queue sze [Packets] Mean Throughput [Packets/sec] Queue sze [Packets] Varance Throughput [Packets/sec] RED 3. 5.9 4.44 4.3 PI 4.38 5.96.7 4 3.73 NN.6 6.6 3.7 3 3.8 RED 48.35.63 4.3 4 8.6 PI 6.99.78.56 4 5.3 NN 4.8.89 5.5 3 3.6 RED 48.3.6 4.4 4 8.53 PI 4.8.79 3.36 4.64 NN..97 8.59 3.37 PI 3.89 5.94.9 4 3.43 NN.5 6..34 4.66 V. CONCLUSION We presented a novel AQM methodology usng a dynamc neural network for TCP congeston control. The neural network acts as a feedback controller to mantan the actual queue sze close to a reference target. The neural network s traned by a BP algorthm. We appled the neural AQM to a sngle bottleneck network supportng multple TCP flows. Four scenaros were examned n the smulaton experments to compare neural AQM to RED and PI-based AQM. Whle PI AQM resulted n queue saturaton and larger overshoot, neural AQM reduced overshoot and elmnated saturaton. was more stable wth no packet loss due to congeston. Especally for the case of tme-varyng TCP dynamcs, the neural AQM was superor. We conclude that neural AQM s an effectve adaptve controller and provdes hgher Qualty of Servce (QoS) n TCP networks. Future work wll extend our results to more complex network scenaros, such as heterogeneous RTTs, short TCP connectons or nose dsturbance networks, and dfferent TCP data streams and wll nclude varous smulaton scenaros usng a network smulaton tool such as OPNET to verfy our results. REFERENCES [] V. Jacobson and M. Karels, Congeston avodance and control, Proceedngs of ACM SIGCOMM, pp. 34-39, 988. [] S. Floyd, A report on recent developments n TCP congeston control, IEEE Communcatons Magazne, vol. 39, no. 4, pp. 84 9,. [3] S. Floyd and V. Jacobson, Random early detecton gateways for congeston avodance, IEEE/ACM Transactons on Networkng, vol., no. 4, pp. 397-43, 993. [4] T. Bonald, M. May, and J.-C. Bolot, Analytc evaluaton of RED performance, Proceedng of IEEE INFOCOM, pp. 45-44,. [5] S. Floyd, Recommendatons on usng the gentle varant of RED, http://www.acr.org/ floyd/red/gentle.html,. [6] S. Floyd, R. Gummad, and S. Shenker, Adaptve RED: An algorthm for ncreasng the robustness of RED s actve queue management, http://www.cr.org/floyd/papers.html,. [7] S. Athuralya, V. H. L, S. H. Low, and Q. Yn, REM: actve queue management, IEEE Network, vol. 5, no. 3, pp. 48-53,. [8] W. Feng, D. D. Kandlur, D. Saha, and K. G. Shn, A self-confgurng RED gateway, Proceedngs of IEEE INFOCOM, pp. 3-38, 999. [9] W. Feng, D. D. Kandlur, D. Saha, and K. G. Shn, Stochastc far Blue: a queue management algorthm for enforcng farness, Proceedngs of IEEE INFOCOM, pp. 5-59,. [] Y. J. Ott, T. V. Lakshman, and L. H. Wong, SRED: stablzed RED, Proceedngs of IEEE INFOCOM, pp. 346-355, 999. [] C. V. Hollot, V. Msra, D. Towsley, and W. Gong, Analyss and desgn of controllers for AQM routers supportng TCP flows, IEEE Trans. on Automatc Control, vol. 47, no. 6, pp. 945-959,. [] V. Msra, W. B. Gong, and D. Towsley, Flud-based analyss of a network of AQM routers supportng TCP flows wth an applcaton to RED, Proceedngs of ACM/SIGCOM, pp. 5-6,. [3] K. B. Km and S. H. Low, Analyss and desgn of AQM based on state-space models for stablzng TCP, Proceedngs of Amercan Control Conference, pp. 6-65, 3. [4] J. Aweya, M. Ouellette, and D. Y. Montuno, A control theoretc approach to actve queue management, Computer Networks, vol. 36, pp. 3-35,. [5] P.-F. Quet and H. Ozbay, On the desgn of AQM supportng TCP flows usng robust control theory, IEEE Transton on Automatc Control, vol. 49, no. 6, 4. [6] S. Haykn, Neural networks: A comprehensve foundaton, Prentce Hall, 999. [7] L.R. Medsker and L.C. Jan, Recurrent neural networks: desgn and applcatons, CRC Press,. [8] P. J. Werbos, Back-propagaton through tme: what t does and how to do t, Proceedngs of the IEEE, vol. 78, no., pp. 55-56, 99. [9] R. J. Wllams and J. Peng, An effectve gradent-based algorthm for on-lne tranng of recurrent network trajectores, Neural Computaton, vol., pp. 49 5, 99. [] Guez, J. L. Elbert, and M. Kam, Neural network archtecture for control, IEEE Control Systems Magazne, vol. 8, no., pp. - 5, 988. 3485