Chinese Word Segmentation based on the Improved Particle Swarm Optimization Neural Networks

Transcription

1 Chnese Word Segmentaton based on the Improved Partcle Swarm Optmzaton Neural Networks Ja He Computatonal Intellgence Laboratory School of Computer Scence and Engneerng, UESTC Chengdu, Chna Department of Computer Chengdu Unversty of Informaton Technology Abstract The Chnese word segmentaton based on the mproved Partcle Swarm Optmzaton (PSO) neural networks s dscussed n ths paper. Frstly, a soluton s obtaned by searchng globally usng FPSO (Fuzzy cluster Partcle Swarm Optmzaton) algorthm, whch has strong parallel searchng ablty, encodng real number, and optmzng the tranng weghts, thresholds, and structure of neural networks. Then based on the optmzaton results obtaned from FPSO algorthm, the optmzaton soluton s contnuously searched by the followng BP algorthm, whch has strong local searchng ablty, untl t s dscovered fnally. Smulaton results show that the method proposed n ths paper greatly ncreases both the effcency and the accuracy of Chnese word segmentaton. Keywords Partcle Swarm Optmzaton(PSO), Fuzzy cluster Partcle Swarm Optmzaton(FPSO), Chnese word segmentaton, BP neural networks I. INTRODUCTION Chnese word segmentaton s one of the most dffcult research topcs n natural language processng. It s the base of Chnese nformaton processng. The smallest components of Chnese language are meanngful words nstead of characters[]. Unlke Englsh language, there s no blank space between two Chnese words. How to tell meanngful Chnese words from sentences, whch are serally stored n computer memory as Chnese character strngs, s called Chnese word segmentaton. In the process of Chnese word segmentaton by computer, denote an nput Chnese character strng as C C C 3 C n and the output as the sequental Chnese words W W W 3 W m, where W s a Chnese word wth ether only one or more Chnese characters[4]. Currently, Chnese word segmentaton s usually dealt wth neural networks especally BP neural networks that s wdely used[5][][3]. However, BP neural networks use gradent descent method and easly get nto local mnmum. That s, as the number of tranng samples ncreased, the nput and output relatonshp becomes complex. The convergence speed of networks becomes slower[]. In order to overcome ths shortage, researchers proposed many methods, such as methods to mprove the error functon and methods to mprove the exctaton functon etc., to mprove tranng speed of networks, but t stll can not avod local convergence. Convergence speed and accuracy of word segmentaton can be mproved by the neural networks traned by genetc algorthm at the same Ln Chen Department of Computer Chengdu Unversty of Informaton Technology Chengdu, Chna suelncl@6.com tme[]. But the tranng tme wll be ncreased because of the complex operaton of genetc algorthm. The accuracy wll be decreased by the redundant connectons of neural networks. In our research, Chnese word segmentaton based on the mproved FPSO neural networks s mplemented as follows: Frstly, a soluton s obtaned by searchng globally usng FPSO (Fuzzy cluster Partcle Swarm Optmzaton) algorthm, whch has strong parallel searchng ablty, encodng real number, and optmzng the tranng weghts, thresholds, and structure of neural networks. Then based on the optmzaton results obtaned from FPSO algorthm, the optmzaton soluton s contnuously searched by followng BP algorthm, whch has strong local searchng ablty, untl t s dscovered fnally[3]. Smulaton results show that the method proposed n ths paper greatly ncreases both of the effcency and accuracy of Chnese word segmentaton. II. INTRODUCTION OF FPSO ALGORITHM PSO optmzng BP neural networks focuses on how the partcle swarm can effectvely search the optmzaton soluton n the soluton space[]. However, along the ncreasng of teraton tmes and swarm types, the basc PSO algorthm s tme-consumng and requres a large number of teraton tmes when calculatng values wth hgh accuracy although t s able to converge to the global mnmum[6]. Further more, the nformaton between partcles may be too concentrated when evolvng to a certan number of generatons. Thus, most partcles gather around few ponts so that both of the prematurty and stagnaton may appear[7]. The basc PSO optmzng BP neural networks s mpossble f partcles are all premature and stagnaton[9]. The dfference between FPSO optmzng neural networks and PSO optmzng BP neural networks depends on the tme before optmzng the locaton and speed of PSO. Frstly, the partcle swarm s dvded nto subpopulatons by usng FCM algorthm and the optmzaton locaton of each subpopulaton s calculated[8]. Then the partcle n the partcle swarm updates the values of ts speed and locaton based on ts personal best (Pbest) and the optmzaton locaton of every subpopulaton. Therefore, the performance of FPSO optmzng BP neural networks s better than the PSO optmzng BP neural networks[3]. In studyng algorthms based on FPSO optmzaton, let x =(x, x,, x d ) be a group values of parameters, each /08 /$5.00@ 008 IEEE CIS 008

2 dmenson of a vector be the weghts or threshold, and d be the number of all weghts and thresholds n a neural networks. The ftness functon of every partcle s shown as follows[9]: I I ( Y y ) = (),, = popindex I n = n Here, n s the number of samples, Y, s the th deal output value of the th sample, Y, s the actual output value of the th sample, and popindex=,, popsze, popsze s the sze of the partcle types(e.: the number of partcles). III. DESIGNAND IMPLEMENTATION OF FPSO ALGORITHM OPTIMIZING BP NEURAL NETWORKS The steps of FPSO optmzng BP networks are shown as follows: ) Intalze the BP networks, and assgn the number of neurons n the nput layer, hdden layer, and output layer. ) Intalze the partcle swarm and the speed of every partcle: a) The locaton of partcle, the dmenson number of the speed vector (dmsze s the sze of the partcle types,.e. the number of partcles) dmsze = the number of connecton weghts from nput layer to hdden layer + the number of connecton weghts from hdden layer to output layer + the threshold number n hdden layer + the threshold number n output layer b) Whle ntalzng the partcle swarm and the speed of every partcle, the st and nd matrx x, all dmensons of the frst dmsze columns representng the partcle locatons, all dmensons of the last dmsze columns representng the partcle speeds, and the last dmenson representng the ftness of the partcle, are ntalzed frstly. c) Intalze the Pbest and Gbest of every partcle. Here, Pbest s the partcle's personal best, and Gbest s the partcle's global best. 3) Calculate the ftness of every partcle: a) A partcle s nput frst. For each sample, an output value of the networks can be calculated by usng the forwardng method of BP neural networks. Then the error s calculated by formula(). By followng the same process, errors of all samples are calculated. Next, the mean square dvatons (.e: the ftness of a partcle) of all samples are calculated by formula(). b) Return to step a), and nput all other partcles untl the ftness values of all partcles are calculated. I) In order to calculate convenently, the elements from column to column IN * HN n the ntalzed partcle matrx are assgned to the weght matrx from nput layer to hdden layer. Respectvely, () elements from column IN * HN+ to column IN* HN + HN * ON are assgned to the weght matrx from hdden layer to output layer, elements from column IN * HN + HN * ON + to column IN* HN + HN * ON + HN are assgned to thresholds of the hdden layer, and elements from column IN* HN + HN * ON + HN + to column IN * HN + HN * ON + HN + ON are assgned to thresholds of output layer. Here, IN s the number of neurons n the nput layer, HN s the number of neurons n the hdden layer, and ON s the number of neurons n the output layer[]. II) When calculatng usng the forwardng method of BP neural networks, the Sgmod functon s used n the hdden layer, whle the lnear Purelne functon s used n the output layer. The curve functon s adopted n the last layer of a network n BP network models, the output s lmted n a very small doman. If the lnear functon s adopted, the output can be a random value. Thus, the lnear functon s adopted n the last layer when BP networks are desgned n MATLAB,.e.: f(x)=x. 4) Compare the ftness values, and ensure the Pbest and Gbest of every partcle: If Present < Pbest and Pbest = Present, Pbest= x ; otherwse, Pbest s not changed; If Present < Gbest and Gbest=Present, Gbest= x ; otherwse, Gbest s not changed. Here, Present s the ftness of the current partcle, Pbest s the partcle s personal best, and Gbest s the partcle s global best. 5) Update the locaton and speed of every partcle: Let G={x, x,, x n } represent a data collecton composed of n partcals x ( =,,, n), whch s dvded nto K cluster C ( =,,, K), and sp ( =,,, K) be the optmzaton locaton searched by partcles n every cluster C untl current tme. The partcles update ther speeds and locatons by the formula shown as follows[3]: v ( t + ) = v ( t) + c r K = ( t)( pbest ( t) x ( t)) c ( sp ( t) x ( t) ) x ( t + ) = x ( t) + v ( t + ) (4) Here, the learnng factor c (=0,,, K) s a constant, and r (=0,,, K) s a random number n the doman of [0,]. The speed and locaton of partcles are updated by formula (3) and (4). Meanwhle, t s consdered whether both the updated speed and locaton are wthn the lmted domans. 6) Calculate the error caused by the algorthm usng the on-lne performance crteron or the off-lne performance crteron. The on-lne performance crteron are utlzed to evaluate the performance of networks, and the crteron are as follows: If fun(gbest)<eg, the algorthm s convergent; r (3)

3 otherwse, terate contnuously untl fun(gbest) reaches the assgned accuracy. Here, eg s the assgned accuracy for algorthm, and fun(gbest) s the ftness of the global best of the th teraton. 7) Compare f the number reaches the largest teraton number or satsfes the requrements of Step 6). If satsfyng the assgned accuracy, the algorthm s convergent. Thus, the weght and threshold of each dmenson n the global best Gbest obtaned from the last teraton s what we desre; otherwse, return to Step ), the algorthm s terated contnuously. The BP algorthm presented n ths paper, whch s based on FPSO optmzaton algorthm. From the dscusson above, we know that these two algorthms are smlar on two aspects: ) The weghts and thresholds of BP network are assgned as the value of each element n vectors of partcles. ) Whle the ftness of a partcle s calculated, the forward propagaton of BP algorthm s used. The defnton of the ftness functon for partcles s obtaned based on the mean square devaton of BP algorthm. IV. SIMULATION AND RESULTS ANALYSIS samples' expectng output and the segmentaton sentence are shown n TABLE I. In order to clarfy convenently, we name the tradtonal BP algorthm as Model, and the FPSO-based BP algorthm proposed n ths paper as Model. The output accuracy ratos of each sample n the 5 sentence samples after beng traned 3000 tmes, are shown n TABLE II ( due to the lmtaton of the paper, only the values of 9 output nodes are lsted n TABLE II). The analyss and comparson of the two algorthms are dscussed next. ) The comparson of the network convergence TABLE I. SAMPLE AND EXPECTING OUTPUT OF NEURAL NETWORKS Sample Expectng Output Ta-YZhen-Feng-Sh-D-Pao-Le ZheZh-Ge-Ta-PngDan-WuWe-Le 3 Ta-Cong-MaShang-XaLa Fgure. The error convergence curves of Model. 4 WuL-Xue-QLa-HenNan Ta-XueHu-Le-Je-FangCheng Whle FPSO-based BP algorthm s used n our research, the ntalzed parameter values are as follows: the partcle swarm sze s 0, the cluster number K s 3, and the studyng factors are set as c 0 = 0.5, c =0.5, c =0.5, c 3 =0.5. The studyng rato = 0.6, momentum factor =0.; the evolvng number s set as 3000 n order to evaluate the convergence performance of our algorthm; the sngle hdden layer network structure s adopted by consderng the effects that the number of hdden layers causes. At the same tme, the effects of node number n the hdden layer to the accuracy and convergence speed are also taken nto consderaton. The less the node number of the hdden layer s, the lower the network accuracy wll be and the worse the ftness wll be. On the contrast, the larger the node number of the hdden layer s, the longer the tranng tme wll be. Thus, the requrements of the on-lne performance evaluaton cannot be satsfed, even the pre-evaluaton accuracy of networks would be decreased. Fnally, the node number of the hdden layer s set as 60 after consderng both the numbers of the nput and output nodes and the number of samples, and recursvely conductng comparson experments. The tranng Fgure. The error convergence curve and partcle optmzaton of Model. The tranng goal curve of the pure BP algorthm s shown n Fg. wth err_goal = 0.00 and lr = 0.0. We have that ts response results TT = [ ] and the runnng tme elapsed_tme = 87.30s. When the error reaches 0.5 after Model has been traned 3000 tmes, the error curve has the ntendance of no change and the network error almost has no effect. Thus, the convergence error of 0.5 for the algorthm can far less meet the error requrement of 0.00 set before. The error curves of the tranng process for Model are shown n

4 TABLE II. OUTPUT ACCURACY RATIOS OF SAMPLES AFTER BEING TRAINED 3000 TIMES Sample Y Y Y3 Y4 Y5 Y6 Y7 Y8 Y9 Model Model Model Model Model Model Model Model Model Model TABLE III. OUTPUT RELATIVE ERROR OF DIFFERENT MODELS OF SAMPLE Model Model Model Average Error Output Relatve Error ( ) Y Y Y3 Y4 Y5 Y6 Y7 Y8 Y sub-graph of Fg.. FPSO-based BP algorthm converged to the assgned accuracy ε BP (ε BP =0.00) after runnng 8 tmes. The total runnng tme s elapsed_tme = 39.50s, and we have the recall result TT = [ ]. Obvously, the teraton number to tran networks when FPSO-based BP algorthm beng used s much less than that of beng used of the conventonal BP algorthm. Whle n sub-graph of Fg., 0 blue nodes represent 0 partcles, the red nodes wth * represent global best partcles, and the yellow traces show the process when partcles search the optmzaton. From all these, we can see that the FPSO algorthm costs less tme whle searchng the optmzaton soluton, and s not easly be premature and stagnant. ) Comparson of Generalzaton Error Dfferent output results for the testng data are obtaned when 30 samples are tested usng two algorthms. Part of the data s presented n TABLE III. It s shown from data n TABLE III that the generalzaton error reaches and the average error reaches when testng by the tradtonal BP algorthm. Whle the generalzaton error reaches 0. ~ and the average error reaches 0.79 when testng by the FPSO-based algorthm proposed n ths paper. Therefore, the generalzaton performance of FPSO-based algorthm s better than that of both the nhertance-algorthm-based BP algorthm and the basc BP algorthm. V. CONCLUSION The prncple and constructon on how to mprove the neural networks word segmentaton model based on the neural network algorthm of mprovng partcle swarm algorthms s dscussed n the paper. Large amount of smulaton conducted through MATLAB. From the experment results, we see that the mproved neural networks avod not only the problems that BP algorthm runs nto the local mnmum and converges slowly, but also the problems of long searchng tme, slow speed, easly appearng prematurty, stagnancy, etc., that are caused when the partcle swarm algorthms search for the optmzaton soluton by usng postve feedback prncple. Therefore, t s very meanngful to the automatc processng of Chnese nformaton that the practcablty and effcency of word segmentaton of our model can be ncreased further. REFERENCES [] Ynfeng, Desgn and analyss of Chnese word segmentaton automaton system based on neural network, Journal of the Chna Socety for Scentfc and Techncal Informaton (n Chnese), 998,,pp.4 49 [] Pan Hao, Hou Qnglan, "A BP Neural Networks Learnng Algorthm Research Based on Partcle Swarm Optmzer", Computer, Engneerng and Applcatons[J]. vol. 4.6, pp.4 43,006. [3] Ln Chen, Ja He, A Partcle Swarm Optmzaton Based on Fuzzy C- Means Clusterng, Journal of Southwest unversty for natonaltes [J]., Vol 33, No 4, pp ,007.4 [4] Ja He, Ln Chen, Hong Xu, Chnese Word Segmentaton Usng Backpropagaton Traned by Genetc Algorthm DCDIS A Supplement, Advances n Neural Networks, Vol.4(S), pp.46 40,007 [5] Bao-Y W and Shao-Mn Z, A Chnese text classfcaton model based on vector space and semantc meanng, {\t Machne Learnng and Cybernetcs}, Shangha, 004,vol.,pp [6] J.Kennedy, R.C.Eberhart. Partcle swarm optmzaton [A]. Proceedngs of the IEEE Internatonal Conference on Neural Networks[C]. 995,pp [7] P.J.Angelne.Evolutonary Optmzaton Versus Partcle Swarm Optmzaton:Phlosophy and Performance Dfference[R].Annual Conference Center on Evolutonary Programmng,San,998. [8] Xanghao L, Hongxng L etc, fuzzy cluster analyss and ts applcaton, Guzhou scence and technology press,994,pp.7, 0 0, [9] Yng Gao, Shengl Xe etc. Mult-subpopulaton optmzaton algorthm based on clusterng [J]. applcaton research of computers, 006,4,pp [0] R.C.Eberhart,Y.H.Sh.Evolvng Artfcal Neural Networks[R]. Proceedngs of Internatonal Conference on Neural Networks and Bran, Beng, 998.

5 [] Cchock A and Unbehauen R, Neural networks for opmzaton and sgnal processng. England: John Wlcy and Sons Ltd, pp. 4 50, 993. [] W. Schffmann, M. Joost and R. Werner, Optmzaton of the backpropagaton algorthm for tranng multlayer perceptrons techncal Report, Unversty of Koblenz, Insttute of Physcs, 993. [3] Hecht - Nelsen R, Theory of backpropagaton neural networks, Proc. IJCNN - 89, pp. 593, 989. Ths work was supported by Natural Scence and Technology Development Fund of CUIT (Chengdu Unversty of Informaton Technology) CSRF00705.