Applying Continuous Action Reinforcement Learning Automata(CARLA) to Global Training of Hidden Markov Models

Transcription

1 Applyng Contnuous Acton Renforcement Learnng Automata(CARLA to Global Tranng of Hdden Markov Models Jahanshah Kabudan, Mohammad Reza Meybod, and Mohammad Mehd Homayounpour Department of Computer Engneerng and Informaton Technology, AmrKabr Unversty of Technology (Tehran PolyTechnc, Tehran, Iran {kabudan, meybod, Abstract In ths research, we have employed global search and global optzaton technques based on Sulated Annealng (SA and Contnuous Acton Renforcement Learnng Automata (CARLA for global tranng of Hdden Markov Models. The man goal of ths paper s comparng CARLA method to other contnuous global optzaton methods lke SA. Experental results show that the CARLA outperforms SA. Ths s due to the fact that CARLA s a contnuous global optzaton method wth memory and SA s a memoryless one.. Introducton HMM s one of the most powerful methods for processng and modelng stochastc processes and sequences. The most popular method for tranng HMM s Baum-Welch (BW method that s a local search method and lke the other local methods suffers from local opta problem. To cope wth ths problem, global search and optzaton methods are used for tranng HMM whch can easly escape from local opta. The man goal of ths paper s a comparson of CARLA and SA as two global search methods for tranng HMMs. In secton two, we ntroduce HMM. In secton three, BW tranng method s brefly explaned. In secton four, SA-based and CARLA-based global search method s descrbed. Secton fve represents the results of the conducted experents, and fnally, secton sx concludes ths paper.. Hdden Markov Model HMM s a fnte-state system wth a number of states (. Ths system changes ts states probablstcally from a state to another state accordng to state transton probabltes. HMM has two types[]: Contnuous Densty HMM and Dscrete Densty HMM. Dscrete Densty HMM s rarely used for modelng and processng contnuous observatons. Assume that an observaton sequence wth te length T s to be generated by ths model. Observaton sequence s a sequence of observaton vectors as follows: O { o o... o t... o T } Each of the observaton vectors o t s a D-densonal vector and t s te ndex. Startng the te, for t, HMM goes nto a state (e.g. wth correspondng ntal state probablty( ( s are elements of ntal state probabltes vector. In state, vector o t s generated usng a multdensonal PDF b (o t. If the model s n state at te t-, then t enters nto state j at te t wth probablty a j (a j s are elements of state transton probabltes matrx A. An observaton sequence O s generated by HMM wth probablty P(O. Ths probablty s computed usng efcent methods lke Forward or Backward Procedure[]. In contnuous densty HMM, multdensonal PDF n state,.e. b (o, s usually a mxture of Gaussan (ormal multdensonal PDF lke: M b ( o c ( o, µ, C ( o, µ, C s a D-densonal Gaussan (ormal PDF wth Mean Vector and Covarance Matrx C of ths form: T ( o, µ, C.exp( ( o µ C ( o µ D (π.det( C Each of the observaton vectors o t s a D column vector. It s necessary that volume (hypervolume under ths mxture PDF be equal to, therefore: M c M s number of Gaussan PDFs n a mxture PDF of each state. Also, the followng constrants must be satsfed: π and a j j C - s nverse of Matrx C and det(c s the absolute value of the determnant of Matrx C. HMM model wth parameters, A and b( s shown as follows: λ ( π, A,b( b( n ths trplet s a set of parameters ncludng weghts (c, Mean vectors ( and Covarance matrces (C of Gaussan PDFs. 3. Baum-Welch Algorthm In real-world applcatons of HMM, we have usually one or more tranng observaton sequences generated by an unknown model. The goal s estatng Proceedngs of the Internatonal Conference on Informaton Technology: Codng and Computng (ITCC /04 $ IEEE

2 parameters of HMM model,.e., A and b(. The most popular method for tranng HMMs s BW algorthm. The goal of BW algorthm s fndng a model * such that the probablty of generatng observaton sequence O gven s maxzed[]: λ arg Max P( O λ λ As we know, the BW method s trapped n local maxa. We use global optzaton and search methods to escape from these maxa and fnd the global one of the HMM tranng problem. 4. Global Optzaton and Search Methods To solve hard optzaton and search problems n a near-optal way, global search and optzaton methods are used. Sulated Annealng (SA[], Genetc Algorthms (GA, Evolutonary Strateges (ES and Evolutonary Programmng (EP[3] are of ths category. A wde varety of global optzaton methods exsts[4]. Recently, a new global optzaton method called Contnuous Acton Renforcement Learnng Automaton (CARLA has been proposed for optzaton n contnuous domans[5,6]. Some of these methods (lke SA have a fner resoluton compared to some other methods (lke GA. From the global search methods, we chose SA for comparng to CARLA. 4.. Sulated Annealng Sulated Annealng s a global search (optzaton method whch s based on the laws of thermodynamcs. If the method follows the recommended Temperature Schedule for reducng temperature, then t guarantees fndng the global optum from a mathematcal pont of vew. Ths method has been appled n many applcatons for both contnuous and dscrete problems. SA follows the followng procedure for optzaton: A random ntal pont and a hgh ntal temperature s selected. In each teraton, t generates a new pont around the prevous pont usng a PDF g(x called Generatng Functon. It calculates cost functon E(x for ths pont, and probablstcally accepts ths pont wth probablty h(x whch s called Acceptance Functon. If the new pont s better than the prevous one, the probablty of acceptance wll be more than the probablty of rejecton. Of course, n the begnnng of the search, temperature s hgh and the search regon s wde, and the acceptance and rejecton probabltes are almost equal (0.5. When temperature decreases and the algorthm approaches optum ponts, search wdth s reduced, and accuracy and resoluton of the search are ncreased. In the followng secton, we explan the standard verson of SA,.e. Boltzmann Annealng (BA Boltzmann Annealng Boltzmann Annealng algorthm as follows:. A hgh ntal temperature(t and a random ntal pont (x s selected.. k, x * x 3. A random new pont x s generated around x * usng generatng functon g(x,x * D ( x x g( x, x π. T k.exp Tk 4. Cost functon for pont x,.e. E(x s calculated. 5. ew pont x s accepted wth probablty h(x. (If t s accepted then x * x, else x * doesn t change E E( x E( x h( x E + exp T k 6. kk+ 7. Temperature s reduced accordng to ths temperature schedule T T k. ln k 8. If the stop condton s met then algorthm stops, otherwse t goes to step 3. In the above algorthm, D s number of parameters of the cost functon E(x and x x s the Eucldean dstance between two D-densonal vectors x and x *. Acceptance functon h(x s usually a Barker functon (lke the form mentoned n step 5 of the above algorthm, but Metropols functon can also be used. In standard SA algorthm, generatng functon s of a Gaussan type and the correspondng SA s called Boltzmann Annealng (BA. If temperature schedule (step 7 of algorthm s not faster than nverselogarthmc schedule, then the method statstcally guarantees that t fnds the global optum n an nfnte number of teratons. The BA algorthm wth correspondng temperature schedule s very slow and needs a very hgh number of teratons for approachng global optum. If we use SA for global optzaton and global tranng of HMMs, the cost functon can be defned as follows: E Log P( Oλ 4.. CARLA One of the methods whch s used n optzaton and control, s Learnng Automaton (LA[7]. LA s a model whch nteracts wth an envronment. It apples an acton to the envronment and envronment evaluates ths acton. Accordng to the nputs that t receves from the envronment, LA corrects ts acton selecton mechansm wth a renforcement sgnal, and agan apples new acton to the envronment. Ths process s repeated nfntely or untl a predefned goal s obtaned. In standard models of LA, acton s a dscrete varable. Recently, a Contnuous Acton Renforcement Learnng Automaton (CARLA wth contnuous acton has been presented[5,6]. CARLA can be employed n the envronments wth contnuous parameters for global Proceedngs of the Internatonal Conference on Informaton Technology: Codng and Computng (ITCC /04 $ IEEE

3 optzaton tasks. It s a model for renforcement learnng n contnuous-parameter envronments. CARLA generates new value of acton usng an acton generaton functon called acton generaton PDF. If the number of system (envronment parameters s more than one, e.g. n mnzaton of a multdensonal cost functon, then a populaton of CARLA automata nteract wth the system n parallel and each automaton undertakes optzaton of one of the parameters of the system. These automata haven t any relatonshp or nteracton wth each other. They nteract only wth the system (envronment, and ther ndrect nteracton wth each other s through the system. Each automaton generates a new acton (new value of the parameter n the correspondng denson usng ts acton generaton PDF. System (Envronment, evaluates the set of actons of automata and sends renforcement sgnals to automata. Usng these renforcement sgnals, each CARLA automaton corrects ts acton generaton PDF. Ths process s repeated untl a predefned crteron s acheved. One of the advantages of ths method s that the CARLA has memory and saves many of the prevous ponts n ts memory wth respect to ther relatve portance. CARLA method operates n ths way:. At frst, acton generaton PDFs are ntalzed wth unform dstrbuton PDFs n the range of system parameters.. Usng acton generaton PDFs, each CARLA generates a new value of acton (parameter r. 3. Envronment response s calculated (cost functon n optzaton tasks. 4. A renforcement sgnal s calculated wth respect to the goodness of the envronment response or cost functon. 5. Acton generaton PDFs are corrected usng these renforcement sgnals. 6. If stop crteron s not met, t goes to step, otherwse the algorthm s stopped. Assume varable x s n the range [x mn, x max ]. At frst (n the frst teraton, acton generaton PDF s ntalzed wth ths unform PDF: f ( x, xmax In n-th teraton, acton r s generated usng acton generaton PDF f(x,n: r F( r, n f ( x, n dx z( n For generatng an acton, frstly a unform random number z(n between [0,] s generated. Then, the acton r s selected such that cumulatve dstrbuton functon F(r,n be equal to z(n. Cumulatve dstrbuton functon F(r,n s computed by numercal ntegraton n the nterval [x mn, r]. Samples of f(x,n s stored n memory and t s descretzed. Wherever the value of f(x,n s not avalable, ts value s calculated usng lnear nterpolaton. After generatng new value of acton,.e. r, ths acton s appled to the envronment and envronment response J(n(cost functon n optzaton tasks s obtaned. A renforcement sgnal s calculated for correctng acton generaton PDF: J med J( n β ( n mn max 0,, J med J mn Renforcement sgnal must be n the range [0,]. J med and J mn are Medan and Mnum of costs of R prevously vsted ponts (e.g. R500. If acton generaton PDF n n-th teraton be f(x,n, then the acton generaton PDF n (n+ -th teraton s corrected n ths way: α [ f ( x, n + β ( n H ( x, r] f x [, xmax ] f ( x, n + 0 else H(x,r s a gaussan functon wth mean r of ths form: g h ( x r H ( x, r.exp. ( x ( max g w ( xmax Parameters g h and g w determne the speed and the relatve resoluton (accuracy of the learnng or the search process. We set these parameters to 0.3 and 0.0 respectvely[5,6]. In ths formulaton, s a normalzaton coeffcent whch must be appled to the corrected PDF such that the hypervolume under new acton generaton PDF,.e. f(x,n+ be equal to one. 5. Experents In ths secton, we present the results of experents. Our goal s tranng an HMM wth Maxum Lkelhood (ML crteron by global methods. The HMM to be traned s a contnuous densty HMM wth three states and three Gaussan PDFs per state wth dagonal covarance matrces and two-densonal observaton vectors. Therefore the number of parameters wll be: P( ++M(D+ Wth 3, M3 and D, the number of HMM parameters s 57. Tranng s performed n multple observaton sequences case[], and K0 observaton sequences wth te length T0 are avalable. We want to maxze P(O. O s the set of K observaton sequences for tranng, and s the set of parameters of the HMM model. As we know, HMM tranng s a constraned optzaton problem wth these constrants: v c > 0, a j j, for all Where v s dagonal element of covarance matrx correspondng to -th denson. For convertng ths Proceedngs of the Internatonal Conference on Informaton Technology: Codng and Computng (ITCC /04 $ IEEE

4 constraned optzaton problem to an unconstraned one, we used ths heurstc mappngs: exp( a j aj exp( a c j M j exp( c exp( c v exp( v That s, a method lke SA, optzes the parameters π, a, c, v n an unconstraned manner, but the j converted and the mapped versons of these parameters,.e., a j, c and v, satsfy the constrants of the HMM tranng problem. ow, we explan the experents. 5.. SA In ths secton, SA method s used for global tranng of Hdden Markov Models. Sutable ntal temperatures n dfferent densons were set. Fgure shows a sample performance for BA method. Snce n the BA method, the rate of temperature reducton s very slow even n 0,000 teratons, we cannot reach fne resoluton. Therefore, search area remans wde and the change n the cost functon wll be sudden and consderable even n the end of the search process. Fgure. Sample performance of SA (Log Probablty 5.. CARLA In ths method, Max and Mn of HMM parameters were set wth respect to the tranng data. Each of CARLA automata tres to correct and adapt ts acton generaton PDF, and fnally ts deal acton generaton PDF s a Gaussan-lke PDF centered on the optum value of parameter n that denson. Fgure shows the shape of the acton generaton PDF after 000, 000, 3000 and 0,000 teratons. Fgure. The shape of the acton generaton PDF for the frst parameter of an HMM wth 57 parameters after 000, 000, 3000 and 0000 teratons CARLA has a good performance n optzaton tasks wth a few number of parameters (e.g. or 3, and wth a few number of local opta, but the CARLA performance s severely degraded when the number of parameters or the number of local opta grows, and the CARLA becomes slow. Fgure 3 shows sample performance of CARLA for tranng an HMM wth 57 parameters A Comparson between CARLA and SA Our goal n ths secton s comparng CARLA wth SA method for tranng HMMs. In all the experents, number of teratons s 0,000. For generalzaton and Proceedngs of the Internatonal Conference on Informaton Technology: Codng and Computng (ITCC /04 $ IEEE

5 consstency of the results, each method must be performed many tes and experental condtons must be the same for two methods. For ths purpose, we performed each experent 50 tes. That s, 50 dfferent and random ntal ponts and 50 random dfferent tranng set were generated. Of course, ntal condtons and tranng sets were exactly the same for two methods. Also, we used the same seed for startng random number generaton routnes for two methods. The average performance of each method was consdered as the Mean of Log-Probablty n dfferent executons of that method. Table shows average performances for two methods. [] Krkpatrck, S., Gelatt, C.D., Vecch, M.P., Optzaton by Sulated Annealng, Scence, Vol. 0, o. 4598, pp , May 983. [3] Bäck, T., Evolutonary Algorthms n Theory and Practce, Oxford Unversty Press, 996. [4] Stuckman, B.E., Easom, E.E., A Comparson of Bayesan/Samplng Global Optzaton Technques, IEEE Trans. on SMC, Vol., o. 5, Sep. 99. [5] Howell, M.., Gordon, T.J., Best, M.C., The Applcaton of Contnuous Acton Renforcement Learnng Automata to Adaptve PID Tunng, Proceedngs of IEE Control Semnar on Learnng Systems for Control, pp. -0, Brmngham, UK, May 000. [6] Howell, M.., Gordon, T.J., Contnuous Learnng Automata and Adaptve Dgtal Flter Desgn, UKACC Internatonal Conference on Control, Sep [7] arendra, K.S., Thathatchar, M.A.L., Learnng Automata: An Introducton, Prentce-Hall, 989. Fgure 3. Sample performance of CARLA method for tranng an HMM wth 57 parameters Table. Average Performance of CARLA and SA for 50 dfferent experents CARLA BA As the table shows, CARLA outperforms SA, and ths can be due to the fact that, SA hasn t any memory and has a very slow-decreasng temperature schedule, and the fnal resoluton of SA s much lower than the CARLA resoluton n a moderate number of teratons. 6. Conclusons In ths paper, we used two global optzaton and search technques for tranng hdden Markov models. Among the global optzaton methods, SA method and Contnuous Acton Renforcement Learnng Automaton (CARLA method were evaluated. Experental results show that the CARLA has a hgher performance compared to SA. Ths s due to the fact that CARLA s a contnuous global optzaton method wth memory and SA s a memoryless one. References [] Rabner, L.R., Juang, B.-H., Fundamentals of Speech Recognton, Prentce-Hall, 993. Proceedngs of the Internatonal Conference on Informaton Technology: Codng and Computng (ITCC /04 $ IEEE