1 Introduction

Transcription

1 Published in IET Computers & Digitl Techniques Received on 6th July 2006 Revised on 21st September 2007 ISSN Hrdwre rchitecture for high-speed rel-time dynmic progrmming pplictions B. Mtthews I. Elhnny Electricl nd Computer Engineering Deprtment, University of Tennessee, Knoxville, TN, USA E-mil: Abstrct: A novel hrdwre rchitecture for performing the core computtions required by dynmic progrmming (DP) techniques is introduced. The ltter pertin to vst rnge of pplictions tht necessitte n optiml sequence of decisions to be obtined. An underlying ssumption is tht complete model of the environment is provided, whereby the dynmics re governed by Mrkov decision process. Existing DP implementtions hve trditionlly focused on softwre-bsed mechnisms. Here, the uthors present method for exploiting the inherent prllelism ssocited with computing both the vlue function nd optiml policy. This llows for the optiml policy to be obtined severl orders of mgnitude fster thn trditionl softwre implementtions, estblishing the vibility of the pproch for demnding, rel-time pplictions. The well-known rentl cr mngement problem hs been studied s benchmrk for which field-progrmmble gte rry-bsed implementtion ws designed. The results highlight the dvntges of the proposed pproch with respect to the execution speed nd the sclbility properties. 1 Introduction A wide rnge of engineering pplictions, rnging from communictions [1 3], robotics [4, 5], to utomotive [6] systems, employ Mrkov decision processes (MDPs) s n underlying formlism to determine the optiml policy, or control scheme, in dynmic stochstic environment. An MDP is defined s qudruple ks, A, P A, Rl, where S denotes finite stte spce, A contins ll possible ctions tht cn be tken t prticulr stte, P A represents the probbility trnsition function jsj jsj A! [0, 1] nd R represents the mpping of stte-ction pirs to rewrds, R : jsj A! R. The MDP qudruple serves s perfect model for n ppliction spce, wheres the ctul plnning, which involves determining n optiml set of ctions tht must be tken to ccumulte mximum rewrd, is referred to s dynmic progrmming (DP) problem. Existing reserch efforts [7, 8] hve trditionlly focused on softwre-bsed relistions of DP techniques to determine the set of mpping of sttes to ctions, or policy, required to derive the optiml control utilising trditionl generl instruction processors. In contrst, rel-time systems often require custom solutions in order to meet stringent timing requirements often ssocited with these pplictions. To pply DP techniques in the frmework of rel-time pplictions, we propose n rchitecture for solving DP problems in custom digitl hrdwre, exploiting prllelism not possible in softwre-bsed relistions. This fcilittes the implementtion of lrge-scle rel-time decisionmking systems tht operte under uncertinty. Recent dvncements in VLSI technology hve led to n increse in logic density nd on-chip memory resources vilbility. An O(jSj 2 A) [9] memory requirement, principlly becuse of the probbility trnsition function, restricts DP lgorithms tht possess lrge stte set from being used in mny pplictions. This restriction is, in principle, becuse of the limited mount of on-chip storge vilble in current devices. Exploiting prllelism tht is n inherent ttribute of DP lgorithms, necessittes lrge, prllel memory in order to store the probbility trnsition function. Fortuntely, existing fieldprogrmmble gte rry (FPGA) devices [10, 11] now provide multiple megbits of distributed on-chip 164/ IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp & The Institution of Engineering nd Technology 2008

2 SRAM, s well s n bundnce of embedded fst multipliers tht cn be used to exploit the prllelism involved in DP computtions. Although the memory requirement is not eliminted by the rchitecture proposed in this pper, we do present techniques to reduce the memory bndwidth burden while, t the sme time, providing computtionl speedup tht is severl orders of mgnitude fster thn trditionl softwre-bsed schemes. The rest of the pper is orgnised s follows. In Section 2, we review the fundmentls of DP nd outline the policy itertion lgorithm nd its hrdwre relistion. In Section 3, vlue function optimistion method is presented, which is crucil to in reducing the memory requirements. A novel hrdwre rchitecture for the relistion of DP techniques, is presented in Section 4. FPGA-bsed implementtion results, ccentuting the performnce nd sclbility ttributes of the proposed pproch, re presented nd discussed in Section 5. Finlly, in Section 6, the conclusions re drwn. 2 Overview of DP DP is considered to be collection of lgorithms tht, given perfect model of the environment s represented by n MDP, llows for n optiml policy, or mpping of sttes to ctions, to be determined. In order to estblish n optiml policy, the probbility of trnsitioning to ech possible next stte, s 0, given ny stte nd ction pir, s nd, is given by P ss 0 ¼ Pr [s tþ1 ¼ s0 js t ¼ s, t ¼ ] (1) A common gol in DP ppliction is to derive n optiml policy tht mximises the vlue function V (s) ¼ mx p Vp (s) (4) Utilising (2) nd (4), the optiml vlue function, expressed in terms of the trnsition probbilities nd rewrd function, is given by V (s) ¼ mx E p ¼ mx s 0 " # 1 k¼0 g k r tþkþ1 js t ¼ s, t ¼ P ss 0[R ss 0 þ gv (s0)] (5) With the optiml vlue function defined, we next outline common methodology, known s generlised policy itertion (GPI), for rriving t the optiml policy. Policy itertion involves n evlution phse, where policy, p(s), is evluted to determine n optiml vlue function, nd n improvement phse, tht is employed to mke the current policy, p (s), greedy with respect to the vlue function. During the evlution phse, the vlue function improves ccording to the current policy. While in the improvement phse the policy is updted ccording to the current vlue function. As both the policy nd vlue function improve over successive itertions, they re gurnteed to converge totheoptimlvluefunctionndtheoptimlpolicy [13]. This pproch is illustrted in the lgorithm in Fig. 1 [14] To complement the bove, we define model such tht the rewrd obtined one step fter trnsitioning from stte s to stte s 0, by tking ction, tober tþ1, such tht its expecttion cn be represented in the form R ss 0 ¼ E[r tþ1 j t ¼, s t ¼ s, s tþ1 ¼ s0 ] (2) The rewrd function provides informtion tht cn be used to determine the vlue of the next stte, given tht ction is tken. The long-term vlue ssocited with given stte is provided by the stte-vlue function. The ltter is defined s the expected sum of discounted rewrds, where g is the discounting fctor, given tht the system begins in stte s nd follows policy p, such tht [12] " # V p 1 (s) ¼ E p g k r tþkþ1 js t ¼ s k¼0 (3) Figure 1 Policy itertion lgorithm & The Institution of Engineering nd Technology 2008 IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp / 165

3 Tble 1 Generic memory requirements Sttes Action Bits/vlue Memory required trnsition probbility Mbits rewrd function Mbits 3 Optimistion for hrdwre design Prior to discussing the hrdwre relistion of the policy itertion lgorithm, we first outline the ssocited limittions in terms of storge requirements. It is pprent tht both the trnsition probbilities, P ss 0, nd expected rewrd function, R ss 0, mndte n O(jSj 2 A) memory requirement. To illustrte the storge impct of this requirement, we first consider n ppliction comprising of 225 sttes, nd restrict the number of ctions tht cn be tken t ech stte to be 11. Furthermore, the trnsition probbilities nd rewrds re represented using 12-bit nd 9-bit vlues, respectively. The ggregte memory requirement, s shown in Tble 1, needed to store the trnsition probbilities nd rewrd function vlues is 11.7 Mbits, which exceeds the on-chip memory vilble in current FPGA devices. In order to efficiently mp the policy itertion lgorithm to hrdwre, it is impertive tht the storge requirements be reduced. To fcilitte such reduction, we first rewrite the policy improvement expressions s p(s) ¼ rg mx ¼ rg mx ¼ rg mx Pss 0[R ss þ 0 gv(s0 )] s 0 " # Pss 0R ss þ P 0 ss 0gV(s0 ) s 0 s 0 " # G(s, ) þ s 0 H ss 0V(s0 ) (6) where the vlue of the stte-ction pir, G(s, ), is given by G(s, ) ¼ s 0 P ss 0R ss 0 (7) for which the set of discounted trnsition probbilities, H ss 0, is given by H ss 0 ¼ gp ss 0 (8) We pply similr technique in n effort to reduce the storge requirements chrcterising the improvement phse. To tht end, the stte-vlue function my be expressed in the form V(s) ¼ s 0 P p(s) ss 0 ¼ G(s, p(s)) þ s 0 Rp(s) ss þ 0 gv(s0 ) H p(s) ss 0 V(s0 ) (9) The vlues for P ss 0 nd R ss 0 re provided by the MDP such tht the computtion of G(s, ) cn be performed offline in order to reduce the ssocited storge requirements from O(jSj 2 A)toO(jSjA) Unfortuntely, it is infesible to similrly reduce the number of unique trnsitionl probbilities from O(jSj 2 A). However, we cn decrese the number of bits required to store ech unique probbility vlue. We selected 7-bit representtion s sufficient resolution for these vlues. In representing the probbility s discrete set of potentil vlues, some error is introduced. This error is less thn onepercent per computtion nd is distributed evenly over wht is lredy n inherent pproximtion. Moreover, hving mde the storge requirements for the probbilistic vlues more moderte, we cn now offer greter dynmic rnge for ctul rewrd vlues. Exploiting the forementioned observtions, the totl memory requirement, s presented in Tble 2, is reduced from 11.7 to 3.89 MBits, which is suitble for FPGA-bsed implementtions. The finl observtion to be mde does not pertin to the storge properties, but rther to the reduction in computtion requirements resulting when utilising the offline processing method to condense the rewrd Tble 2 Reduced memory requirements Sttes Action Bits/vlue Totl memory trnsition probbility Mbits rewrd function kbits 166/ IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp & The Institution of Engineering nd Technology 2008

4 function. Recognising tht the product of the discount prmeter, g, nd trnsitionl probbilities, P ss0, cn be computed offline, we obtin smller number of on-chip multipliers required. This further decreses power, re nd ltency ssocited with the hrdwre relistion of the DP policy itertion lgorithm. 4 System rchitecture The hrdwre relistion of the policy itertion lgorithm, s shown in Fig. 2, offers modulr rchitecture with respect to the number of sttes tht cn be ddressed in prllel. Its gol is to produce n optiml ction selection in rel-time. The rchitecture is comprised of prllel vlue estimtion module, n policy evlution primitive nd control unit tht lterntes between policy evlution nd policy improvement modes to support the generlised policy itertion frmework. The sclbility property of this rchitecture is derived from the bility to dd vlueestimtion primitives, with fixed ltency cost, to the prllel vlue estimtion module. In estblishing the foundtions of DP, the generlised policy itertion lgorithm outlined n evlution phse tht optimises the vlue function, nd n improvement phse tht updtes the control policy. For both phses, itertion cross the stte spce is differentited primrily by the number of ctions required to perform n updte. For the improvement phse, the policy updte requires the vlue function to be computed for ll possible ctions, with the outcome resulting in storge of the ction tht produces the mximum vlue. The evlution phse only requires the vlue function to be computed for single ction, dictted by the policy, with the updte resulting in the storge of the computed vlue. For efficiency, we implement single, prllel vlue estimtion module tht receives the stte-ction pir to be evluted from the DP control module. During the policy improvement phse, the stte vlue supplied by the DP control module is fixed for every potentil policy ction vilble, wheres for the evlution phse the stte vlue chnges concurrently with ction vlue dictted by the selected control policy, p(s). From the perspective of the vlue estimtion module, there is no distinction between evlution nd improvement. Such distinction is only mde t the policy evlution primitive. Although the evlution nd improvement phses could be constructed using seperte modules, this would not be n efficient solution s the hrdwre resource requirement would double, given the ddition of second prllel vlue estimtion module. Figure 2 Hrdwre relistion of the policy itertion lgorithm Policy itertion rchitecture for FPGA implementtion b Prllel vlue computtion module & The Institution of Engineering nd Technology 2008 IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp / 167

5 The prllel vlue computtion, presented in Fig. 2, is constructed using S concurrent vlue-estimtion primitives, depicted in Fig. 3, to compute the product of the discounted trnsition probbility, H ss 0, nd the potentil next stte vlue, V(s 0 ), using 18-bit floting point multipliers. The result of the S prllel multiplictions is forwrded to tree of floting point dders, with depth of log (jsjþ1), for summtion. In trgeting brod rnge of pplictions, the rchitecture must be structured in mnner thn provides sclbility in terms of the number of sttes tht cn be supported. To do so, the selection of floting point multipliers nd dders, s opposed to fixed-point units, ws mde in n effort to mintin fixed dely nd constnt structure while retining high degree of ccurcy. The next stte vlues, V(s 0 ), re stored in S 18-bit registers tht re ll ccessed in prllel. In n effort to reduce memory requirements, the H ss 0 vlues re 7- bit encoded nd re distributed t equidistnt points in the rnge of [0,1]. These vlues re converted to 18-bit floting point vlues, by the H ss 0 decode unit, prior to being forwrded to the floting point multiplier. It should be noted tht encoded vlues must be stored in S RAM units for prllel ccess by the S vlue-estimtion primitives. Ech of the of S memory units corresponds to single next stte vlue, S 0, with its ddress represented by the current sttection pir. In connection with recent work tht hs exmined the effects of delyed informtion within the MDP frmework [15], it is pproprite to cknowledge tht our rchitecture is subject to similr concerns given the pipelined nture of the prllel vlue estimtion module, V(s). This pipelining results in n updte dely tht directly corresponds to the ssocited ltency of the vlue estimtion module nd the policy evlution Figure 3 Computtion unit structure primitive. In our rchitecture, vlue estimtes re not vilble instntneously, which cn result in convergence dely. For this reson, the system will select n ction bsed on previous vlues, rther thn the current ones. Given the significnt speedup provided by hrdwre implementtion, the reduced convergence rte, s will be demonstrted, is minor. The result of the prllel vlue computtion is pssed to n evlution module, which, depending on whether it is operting in the evlution or improvement phse, is responsible for updting either the vlue function or the control policy, respectively. In ddition to updting the vlue function nd control policy, it must lso determine whether they re close enough to the optiml solution. Addressing the evlution mode first, it is required tht t the beginning of ech itertion flg bit is set so s to indicte tht the optiml vlue function hs been found. As we iterte cross ech stte, the difference between the previously registered vlue nd the newly computed one is obtined. The result is then compred to smll vlue, 1, using floting-point comprtor module, to determine whether n optiml vlue hs been reched. If ny stte which produces vlue tht is not optiml the flg bit, indicting n optiml vlue function, is clered. Upon completion of the clcultions pertining to the finl stte, the DP controller tests the flg bit. If the ltter is set, the controller switches to policy improvement mode, otherwise the evlution module repets its itertive policy evlution process. The switch from policy evlution to policy improvement represents significnt chnge in functionlity of the evlution module. In policy improvement mode, the dder performs no ddition opertions nd merely serves s pipeline stge, with its output forwrded to mximum vlue module. The ltter comprises of register nd comprtor. The contents of the mximl vlue register re updted once newly computed result is found to be greter thn the existing one. Upon sweeping through the ction set for given stte, the ction tht yielded the mximl vlue is stored in the policy RAM. If, upon receiving the finl smple of prior evlution, it is determined tht n optiml vlue function hs been reched, the pipeline is flushed nd the system trnsitions to n improvement stte during which the process gin itertes over ll possible sttes. However, stte trnsitions do not occur until the results for ll possible ctions tht cn be tken from the current stte, hve been evluted. Once the finl stte-ction pir hs been identified, the system flushes its pipeline nd trnsitions to n evlution stte, the gol of which is to determine whether n optiml policy hs been found. If the evlution 168/ IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp & The Institution of Engineering nd Technology 2008

6 process determines the policy to be optiml, flg is sserted, the system flushes the pipeline nd wits for future instructions. 5 Implementtion results In this section, we employ the DP rchitecture presented in the prior section to solve the rentl cr mngement problem introduced in [16]. In this problem, mnger is responsible for distributing crs between two rentl cr loctions, ech contining mximum of 11 crs. For ech cr rented t either loction, the mnger is credited $10 by the ntionl compny. Vehicles rrive nd deprt the cr rentl loctions ccording to geometric distribution, Pr fn rrivlsg ¼ (1 2 l)l n21, where n is rndom vrible denoting the number of rriving, or deprting, vehicles. The rrivl probbilities, l rrivl, for loctions 1 nd 2, re 0.3 nd 0.5, respectively. Correspondingly, the deprture probbilities, l deprture, for loctions 1 nd 2, re 0.3 nd It should be noted tht, since ech loction is limited to 11 crs, ny rriving crs tht exceed the site limit re discrded. The mnger is permitted to move crs between the two loctions t cost of $2 per cr, with limit of 5 crs tht cn be moved in single night. DP formlism is pplied to determine the optiml policy in n effort to mximise the mnger s profit cross the two loctions. To solve the forementioned problem, the DP rchitecture hs been coded in Verilog HDL with the implementtion trgeting n Alter Strtix II EP2S180 FPGA device [10]. The design is comprised of 121 sttes with totl of 11 potentil ctions tht cn be tken from ech stte. The complete system implementtion chieved post-fit operting frequency of 103 MHz (9.7 ns). The design required dptive logic modules (ALMs), or 66% of the ALMs vilble on the trget FPGA device. Since the rchitecture requires single RAM such tht vlues for ech of the 121 sttes cn be ccessed in prllel, the design consumed 493 (64%) of the M4K RAMs vilble. Additionlly, single M512 RAM unit ws employed for storing G(s, ) vlues. The totl memory requirement ws Mbits, or 18% of tht vilble. In order to estblish the corse performnce gin tht cn be chieved when implementing the hrdwre design, softwre-bsed solution to the rentl cr mngement problem ws written. When simulted, the policy itertion lgorithm converged fter n verge of 33 evlution itertions nd 4 improvement itertions with n execution time of s. The optiml policy nd vlue function re shown in Figs. 4 nd 5, respectively. Figure 4 Optiml policy for rentl cr mngement problem Given tht the hrdwre design computes vlues in prllel, the equivlent hrdwre run time cn be determined by first estblishing the system ltency. The ltency contributed by the floting-point multipliction, l mult, is five clock cycles. The ltency due to the floting point dder, l dd, nd the flotingpoint comprtor units re four nd one clock cycles, respectively. The finl ltency component to be considered is result of miscellneous pipeline registers tht re needed to ensure mximum throughput, l pipe, nd is comprised of four clock cycles. The hrdwre runtime cn now be estblished in terms of the system ltency, policy improvement itertions (R i ), policy evlution itertions (R e ), the number of sttes (jsj), ctions (A) nd clock periods (t clk ). The number of clock cycles comprising single Figure 5 Optiml vlue function for rentl cr mngement problem & The Institution of Engineering nd Technology 2008 IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp / 169

7 policy evlution phse is c evl ¼jSjþl mult þ ( log 2 (jsjþ1) l dd ) þ l dd þ l compre þ l pipe (10) In the improvement phse, the clock cycle requirement must include the vlue computtion ltency for every ction, which is contrry to the evlution phse for which only one ction is selected. As such, the time to complete single improvement phse is given by c improve ¼ (jsja) þ l mult þ ( log 2 (jsjþ1) l dd ) þ l dd þ l compre þ l pipe (11) The optiml policy nd vlue function hrdwre computtions cn be expressed s function of the evlution phse nd improvement phse cycle requirements, c evl nd c improve, such tht t totl ¼ R i ((R e c evl ) þ c improve ) t clk (12) Further, we cn employ (10) nd (11), estblishing the hrdwre evlution cycles, c evl, to be 163 clock cycles nd the number of improvement itertion cycles, c improve, to be 1373 cycles. From these, the ggregte time cn be obtined by pplying (12), such tht the hrdwre computtion time to solve the rentl cr mngement problem is ms. Thus, the hrdwre-bsed solution yields four orders of mgnitude improvement in timing when compred to the softwre-bsed relistion. This considerble gin cn be ttributed to the considerble prllelistion ssocited in clculting the vlue function, which is in contrdiction to the sequentil processing inherent to the softwre-bsed pproch. Notbly, the policy-evlution processing requirement hs been reduced from n O(jSj 2 ) processing requirement, for the softwre implementtion, to n O(jSj log (jsj)) processing requirement in the corresponding hrdwre implementtion. Furthermore, the policy improvement phse results in similr processing reduction, where the softwre processing requirement is now O(jSj 2 A), compred to O(jSj (jsj) A) for the hrdwre. Additionl gins cn be ttributed to direct memory ccess, lck of instruction overhed (fetch, store, pipeline dely nd so on), nd exclusive ccess to the computtion units. In reducing the computtion time to the microseconds rnge,dpcnnowconsideredfornincresednumber of rel-time pplictions. 6 Conclusions In this pper, novel hrdwre-bsed rchitecture ws introduced for rel-time DP pplictions. It hs been demonstrted tht the proposed rchitecture cn yield speed gin of severl orders of mgnitude when compred to softwre-bsed systems. To tht end, the frmework is suitble for brod rnge of pplictions necessitting fst, rel-time sequentil decision mking. FPGA implementtion results were presented nd discussed to emphsise the vibility nd sclbility of the proposed rchitecture. 7 Acknowledgment This work hs been prtilly supported by the United Sttes Deprtment of Energy (DOE) reserch contrct DE-FG02-04ER25607 nd by Alter, Inc. reserch grnt. 8 References [1] JAVIDI T., TENEKETZIS D.: Sensitivity nlysis for n optiml routing policy in n d hoc wireless network, IEEE Trns. Autom. Control, 2004, 49, (8), pp [2] HAAS Z., HALPERN J.Y., LI L., ET AL.: A decision-theoretic pproch to resource lloction in wireless multimedi networks. Proc. 4th int. workshop Discrete lgorithms nd methods for mobile computing nd communictions DIALM 00, 2000, pp [3] USAHA W., BARRIA J.: Mrkov decision theory frmework for resource lloction in leo stellite considertions, IEE Proc. Commun., 2002, 149, (56), pp [4] LAROCHE P., CHARPILLET F., SCHOTT R.: Mobile robotics plnning using bstrct Mrkov decision processes. Proc. 11th IEEE Int. Conf. Tools with Artificil Intelligence ICTAI 99, Wshington DC, USA (IEEE Computer Society, 1999), p. 299 [5] FERGUSON D., STENTZ A.: Focussed processing of mdps for pth plnning. 16th IEEE Int. Conf. Tools with Artificil Intelligence, ICTAI 2004, pp [6] KANG J., KOLMANOVSKY I., GRIZZLE J.: Approximte dynmic progrmming solutions for len burn engine ftertretment. 38th IEEE Conf. Decision nd Control, 1999, vol. 2, pp [7] PADBERG F.: On the potentil of process simultion in softwre project schedule optimiztion. 29th Annul Int., Computer Softwre nd Applictions Conf., COMPSAC 2005, vol. 2, pp [8] KIM S., LEWIS M., WHITE C.C.: Optiml vehicle routing with rel-time trffic informtion, IEEE Trns. Intell. Trnsp. Syst., 2005, 6, (2), pp / IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp & The Institution of Engineering nd Technology 2008

8 [9] KOTSALIS G., DAHLEH M.: Model reduction of irreducible mrkov chins. Proc. 42nd IEEE Conf. Decision nd Control, 2003, vol. 6, pp [10] Alter Strtix II technicl documenttion, vilble t: [11] ilinx Virtex-4 technicl documenttion, vilble t: [12] SUTTON R.S.: On the significnce of mrkov decision processes. ICANN, 1997, pp [13] BERTSEKAS D.P., TSITSIKLIS J.N.: Neuro-dynmic progrmming (Athen Scientific, 1996) [14] BELLMAN R.E.: Dynmic progrmming (Princeton University Press, 1957) [15] KATSIKOPOULOS K., ENGELBRECHT S.: Mrkov decision processes with delys nd synchronous cost collection, IEEE Trns. Autom. Control, 2003, 48, (4), pp [16] SUTTON R.S., BARTO A.G.: Reinforecement lerning: n introduction (Cmrbridge, MA, MIT Press, 1998) & The Institution of Engineering nd Technology 2008 IET Comput. Digit. Tech., 2008, Vol. 2, No. 3, pp / 171