Application of Artificial Neural Networks for Rainfall Forecasting in Fuzhou City

Transcription

1 Applcaton of Artfcal Neural Networks for Ranfall Forecastng n Fuzhou Cty GONG Zhenbn MEE09203 Supervsor: Prof. A. W. Jayawardena ABSTRACT Accurate ranfall predcton s of great nterest for water management and flood control. In realty, physcal processes nfluencng the occurrence of ranfall are hghly complex, uncertan and nonlnear. Ranfall predcton s one of the most dffcult tasks n operatonal meteorology. In order to forecast ranfall n Fuzhou Cty, Artfcal Neural Networks (ANNs), a data drven technque based on the workng prncple of bologcal neurons are appled n ths study. Obectves are set to dfferentate what knds of atmospherc stuaton can trgger ranfall, and fnd out how much ranfall can be generated n such stuaton based on the avalable meteorologcal parameters records and real ranfall data. Two dfferent knds of ANNs,.e. Back Propagaton Neural Networks (BPNNs) and Support Vector Machnes (SVMs) are appled and compared wth each other. The results show that both BPNNs and SVMs can classfy rany days and non-rany days satsfactory, but the network ablty of calculatng the quanttatve precptaton s very poor because of ther poor generalzaton ablty. The results of senstvty analyss for BPNNs show that several free parameters, such as numbers of hdden neurons, learnng rates and momentum terms, have great nfluences on the network performance. By sutable data pre-processng, such as Prncpal Component Analyss (PCA), data normalzaton and data re-samplng, the network performance can be mproved a lot. Increasng the nput data nformaton, changng the real ranfall data nto sutable ran levels, applyng more complcated network topology and tryng dfferent actvaton functons are also recommended to mprove the quanttatve precptaton predcton for future works. Key words: ANNs, BPNNs, SVMs, quanttatve precptaton predcton, generalzaton ablty, Prncpal Component Analyss, data pre-processng, senstvty analyss INTRODUCTION Accurate ranfall predcton s of great nterest for water management and flood control. In realty, physcal processes nfluencng the occurrence of ranfall are hghly complex, uncertan and nonlnear. Ranfall predcton s one of the most dffcult tasks n operatonal meteorology. There are several ranfall forecastng methods, such as statstcal methods and Numercal Weather Predcton (NWP) models. Because of the nonlnear character of ranfall process, statstcal methods cannot generate good results. NWP models are based on very complcated physcal dynamcs. NWP have mproved the forecast accuracy of large-scale weather systems greatly, such as wnd feld and mass feld, but not good at meso-scale or small-scale weather systems. Ranfall s manly caused by meso-scale and small-scale weather systems. Local clmate stuatons also can affect the ranfall generaton process a lot. NWP models cannot solve ths local problem. Artfcal Neural Networks (ANNs) are a type of data drven technque based on the workng prncple of bologcal neurons. Snce the mechansms of ranfall are stll not understood well, ANNs are a good choce worth tryng to analyze the relatonshp between meteorologcal parameters and ranfall. The obectves are to dfferentate what knds of atmospherc stuaton can trgger ranfall, and fnd out how much ranfall can be generated n such stuatons. After fndng the optmal network parameters, Engneer, Fuzhou Meteorologcal Bureau, Fuan, Chna Research and Tranng Advsor, Internatonal Centre for Water Hazard and Rsk Management (ICHARM), Japan.

2 by nputtng the predcton results of varous meteorologcal parameters whch come from NWP models, the network can decde f t has ranfall and quantfy the amount of precptaton. Fuzhou ( 福州 ) s the captal cty of Fuan provnce. The cty s located n south-eastern Chna, the west sde of Tawan Strat. The ursdcton area s around 2,000 sq klometers, and the populaton s 6.39 mllon. The rany season s from Aprl to June. Durng the rany season, the contnuous heavy ranfall can cause basn wde flood n Mn Rver. Because of the flood control facltes along the rver, nowadays, ths knd of flood s not the maor threat to Fuzhou. The perod from July to September s the typhoon season. Durng ths perod, except the severe typhoon ranstorm, the strong convectve ranstorms, such as thunderstorm, occur very frequently. Most of ths knd of ranfall can cause severe waterloggng n the cty area, and also can trgger landsldes or debrs flows n western and northern hlly regon. DATA Daly ranfall data of Fuzhou Meteorologcal Observaton Staton from 948 to 2008 are selected to be the desred output data for tranng, valdatng and testng. The NCEP/NCAR (Natonal Center for Atmospherc Research) Re-analyss data downloaded from NOAA webste are chosen as nput data. The nput data contan varous meteorologcal parameters, ncludng ar temperature, specfc humdty, relatve humdty, geopotental heght, wnd and precptable water. Each parameter contans three dfferent layers varables rangng from 000mb to 850mb, except precptable water, whch s measured for the whole ar column. Besdes these measured data, the nput data also nclude the geopotental heght dfference between Fuzhou and three other surroundng grds data to express the nformaton of pressure gradent. Total number of nput varables s 3. THEORY AND METHODOLOGY Accordng to Haykn (999), a neural network contans large amounts of parallel dstrbuted processors made up of smple processng unts. These unts have a natural propensty for storng experental knowledge and makng t avalable for use. Neural networks acqure knowledge from ts envronment through a learnng process, and use ts nterneuron connecton strengths,.e. synaptc weghts, to store the acqured knowledge. By gvng a set of measured data, neural networks can learn the stmulus-response relatonshp wthn the data set through the tranng process (Mnns and Hall, 996). There are several knds of ANNs. BPNN and SVMs are selected n ths study. Multlayer Perceptrons wth back propagaton algorthm,.e. BPNN, s the most popular network archtecture n use today. BPNN ncludes two dstnct passes of computaton, the forward pass and the backward pass. In the forward pass the synaptc weghts reman unaltered throughout the network, and the functon sgnals of the network are computed on a neuron-by-neuron bass and propagate through the actvaton functon. Any non-lnear functon that s dfferentable can be used as an actvaton functon. From nput layer to hdden layer, the logstc sgmod functon s often used as the actvaton functon because of ts very smple dervatve that makes the subsequent mplementaton of the learnng algorthm much easer (Flood and Kartam, 994). The logstc sgmod functon s shown as below: Logsg( x), whch vares between the lmts [0,]. ax e The hyperbolc functon s also often used n many crcumstances, whch s shown as: Hyerbolc( x) a tanh( bx), whch vares between the lmts [-,]. Accordng to Lecun, sutable values for the constants a and b are: a=.759 and b=2/3.when sgnal s propagated from hdden layer to output layer, sometmes a lnear functon can be used as the actvaton functon, whch s more smple than the logstc functon or the hyperbolc functon. The backward pass s n opposton to the forward pass, whch starts at the output layer by passng the error sgnals back through the network layer by layer and recursvely computng the local gradent for each neuron, and then adustng the synaptc weghts of the connectons wth a hope that n the next 2

3 teraton the error wll be reduced (Haykn, 999). The error-correcton rule used n ths study s the Delta rule, whch s also named as Least Mean Squared (LSM) algorthm. Accordng to the delta rule, the correcton n appled to the synaptc weght connectng neuron to neuron s defned as: w w ( n) w ( n ) ( n) y ( n), where s the momentum term wth the range between 0 and, denotes learnng rate parameter, usually we can set 0.5, 0., (n) denotes the local gradent of nth tranng example, n s the synaptc weghts correcton of current tranng w example, w n s the synaptc weghts correcton of prevous tranng example. The local gradent (n) can be calculated by usng the dervatve of the actvaton functon. Then the synaptc weghts and bas can be updated step by step. Before applyng the BPNN, the nput data and desred output data need to be pre-processed. In ths study, there are 3 nput varables, so the dmensonalty should be reduced and the nput varables also should be decorrelated. The Prncpal Component Analyss (PCA) s a good choce to solve ths problem. After PCA, the nput data should be normalzed accordng to dfferent output range of dfferent actvaton functons. For example, for logstc functon, the nput data and the desred output data should be scaled nto range of 0 to. The fnal data pre-processng step s data balancng. As the number of tranng examples ncreases, n many applcatons, the neural networks would face wth mbalanced data sets, whch can cause the network to be based towards one class contanng more examples (Kotsants et al, 2006). Common solutons of mbalanced data set nclude: re-samplng the examples,.e. duplcatng tranng examples of the under-represented class (Lng, C. and L, C., 998) and downszng the examples,.e. removng tranng examples of the over represented class (Kubat, M. and Matwn, S., 997). In ths study, re-samplng the examples s selected n order to present as many data as possble to the network. Strctly speakng, Support Vector Machnes (SVM) s one knd of neural networks, and s another category of unversal feed forward networks poneered by Vapnk, whch can also be used for pattern classfcaton and nonlnear regresson (Haykn, 999). Because of ts remarkable characterstcs such as good generalzaton performance, the absence of local mnma, and sparse representaton of soluton, SVM has been recevng ncreasng attenton n areas rangng from pattern recognton to regresson estmaton (Cao and Tay, 2003). The SVM s based on Cover s theorem that a non-lnearly unseparable multdmensonal space may be transformed nto a new feature space where the patterns are lnearly separable wth hgh probablty (Haykn, 999). What SVM need to do s through an nner-product kernel functon to construct an optmal hyperplane whch can separate the dfferent features, n such a way that the margn of separaton between postve and negatve examples s maxmzed. Brefly speakng, the SVM s an approxmate mplementaton of the method of structural rsk mnmzaton whch seeks to mnmze an upper bound of the generalzaton error consstng of the sum of the tranng error and a confdence nterval. In SVM, the soluton to avod gettng stuck nto local mnma s dependent on a subset of tranng data ponts whch are referred to as support vectors (Cao and Tay, 2003). In order to solve pattern recognton problems, the frst step s to choose a sutable kernel functon for nonlnear mappng of an nput vector nto a hgh-dmensonal feature space that s hdden from both the nput and output. There are three common types of support vector machnes: polynomal learnng machne, radal-bass functon network, and two-layer perceptron. For radal-bass functon network, for example, the kernel functon s shown as: 2 K ( x, x ) exp( x x ). By usng ths kernel functon, a decson 2 2 surface s constructed. Ths decson surface s nonlnear n the nput space, but ts mage n the feature space s lnear. Gven the tranng sample ( x N, d ) ( x s the nput vector, d s the desred output), then the classfcaton problem can be transferred to fnd the Lagrange multplers ( N ) that maxmze the obectve functon: 3

4 N N N Q( ) dd K( x, x ), 2 N Ths functon subect to the constrants: () 0 and (2) 0 C, for =,2,,N, where C s d a penalty parameter, whch s a user-specfed postve parameter. By solvng functon of Q ( ) wth ts constrants, we can determne the Largrange multplers, and a hard classfer mplementng the optmal separatng hyperplane n the feature space, whch s gven by: N f ( x) sgn[ d K ( x, x ) b], Accordng to ths functon, the classfcaton purpose can be realzed. The regresson problem can be solved by smlar ways, but need to defne a Є-nsenstve loss functon frst. RESULTS AND DISCUSSION Fndngs from classfcaton problems: After performng PCA, the nput varables are reduced from 3 to 9. Feedng the nput data to the BPNN wth sequental mode to solve classfcaton problem s the frst task n ths study. In classfcaton problem, dfferent tranng examples are classfed nto two types, rany days and non-rany days, so the desred output data have only two values. Takng the logstc functon n BP neural networks as example, 0. denotes the day wth no ranfall (precptaton=0), and 0.9 denotes the day wth ranfall (precptaton>0). In order to solve ths problem, BP neural networks wth dfferent actvaton functons are attempted. Table : Results of Classfcaton Problem by applyng BPNN Logstc Functon n Hdden Layer Logstc Functon n output layer Lnear Functon n output layer Number of hdden neurons Tranng Valdaton Testng Tranng Valdaton Testng Hyperbolc Functon n Hdden Layer Number of hdden neurons Hyperbolc Functon n output layer Lnear Functon n output layer Tranng Valdaton Testng Tranng Valdaton Testng Table shows the results of computaton. The nput data set s dvded nto three parts, 80% for tranng, 0% for valdaton, and the rest 0% for testng. The network s run wth three dfferent numbers of hdden neurons. There are four knds of actvaton functon combnatons. The results show that the network performance wth dfferent actvaton functon combnaton and dfferent number of hdden neurons has no bg dfference. For tranng part, the accuracy range from 74% to 78%, and for testng part, the accuracy range from 77% to 8%. Ths accuracy s wthn the acceptable range. Table 2: Results of classfcaton by usng SVM For Classfcaton Kernel Functon Penalty parameter Kernel functon C Tranng Testng Lnear Functon % 77.47% Polynomal Functon % 77.73% Radal Bass Functon % 78.94% Sgmod Functon % 79.96% Table 2 shows the results calculated by usng SVM wth dfferent kernel functons. The penalty 5 parameter and kernel functon parameter are selected by tryng from 2 5 to 2 whch the nterval s based on powers of 2. The accuracy s the only performance measurement for ths classfcaton 4

5 problem. From the table, the accuracy s also wthn the acceptable range, and the radal bass functon has the best performance. The SVM method can be used to classfy ths complcated nonlnear nseparable problem well, but the results are not as good as BPNN. Fndngs from regresson problems: After decdng whether there wll be ranfall or not, the next task s to fgure out the quantty of precptaton that wll be. After data pre-processng, the data set s also dvded nto three parts for tranng, valdaton and testng. The network s run by usng dfferent combnatons of actvaton functons n hdden layer and output layer wth numbers of hdden neurons rangng from 5 to 4. Table 3 only shows the best performance of the four combnatons. Table 3. Statstcal performance of BPNN usng dfferent actvaton functons Actvaton Functons Hdden neurons Tranng Part Testng Part R CD EI RMSE R CD EI RMSE Logstc & Logstc Logstc & Lnear Hyperbolc & Lnear Hyperbolc & Hyperbolc From Table 3, all of the statstcal performances of tranng parts for dfferent actvaton functon combnaton are far better than testng parts. The range of correlaton coeffcent of tranng s 0.73 to 0.94, and the best performance comes from combnaton of two logstc functons, but the combnaton of hyperbolc functon and lnear functon shows relatvely poor performance. Results obtaned by comparng these performance ndces show that logstc functon has the best performance, and the lnear functon has the poorest performance. For testng part, all of statstcal performances are not good, that means the generalzaton ablty of the network ths study selected s very poor, and cannot nterpret the relatonshp between the nput varables and the precptaton. Ths may be because the complcaton of the problem tself or the nput varables ths study selected mght not be suffcent to model the mechansm of ranfall generaton. Table 4: Results of regresson problems by usng SVMs Kernel Functon Penalty parameter For Regresson Kernel functon Loss functon parameter parameter Tranng Testng C Є MSE R MSE R Lnear Polynomal Radal Bass Sgmod Table 4 shows the results of SVM regresson problem. Smlar to BP neural network, SVM cannot smulate the ranfall mechansm correctly. All correlaton coeffcents of observed ranfall and calculated output are not good enough. In ths case, polynomal kernel functon has the best performance, but the performance of sgmod functon s very poor. Fndngs from senstvty analyss: There are several free parameters whch can nfluence the network performance, such as the numbers of hdden neurons, the learnng rate, and the momentum term. By applyng dfferent numbers of hdden neurons, the results show that more hdden neurons always gve more accurate output. In ths study, when the number of hdden neurons ncreases from 3 to 5, the correlaton coeffcent (R) also mproves from 0.77 to 0.9, and the root mean square error (RMSE) decreases from 0.09 to 0.062, but when the number s over 5, the performance becomes worse. Ths ndcates that too many hdden neurons could cause over learnng problem. The testng results show that more hdden neurons would decrease the generalzaton ablty of the network. In order to test the nfluence of dfferent learnng rate and momentum term, a BP neural network wth 8 hdden neurons and logstc sgmod functon n hdden layer and lnear functon n output layer was tested. Dfferent learnng rates rangng from 0.03 to 0.3, n steps of 0.03 were tred. The results show that smaller learnng rates always generates better accuracy. The correlaton coeffcents between tranng output and desred output decreases from 0.88 to 0.72 as the learnng rate ncreases from

6 to 0.3, and so does the mnmum RMSE. Wth lower learnng rates, more tranng epochs to converge the network are needed, ndcatng that smaller learnng rate parameters cause smaller changes to the synaptc weghts n the network from one to the next teraton. After combned wth sutable momentum term, the slowness of convergence and network performance can be mproved. Ths can be proved by applyng dfferent momentum terms combned wth the learnng rate equal to 0.. The results show that when the momentum term s equal to 0.5 or 0.7, the network performance s the best, and the convergence tme s also shorter than others. CONCLUSIONS By the applcaton of multlayer perceptron neural networks wth back-propagaton algorthm and support vector machnes, the occurrence of a ranfall event can be predcted satsfactorly for dfferent atmospherc condtons. Both methods are good at classfcaton problem. For quanttatve precptaton analyss, nether method can generate good results n real applcaton. The generalzaton ablty of artfcal neural networks s poor. Before apply artfcal neural networks nto real practce, the network should be optmzed by tryng dfferent free parameters, such as dfferent numbers of hdden neurons, dfferent learnng rates and momentum terms. These free parameters can nfluence the network performance sgnfcantly. Also, sutable data pre-processng s very mportant and necessary before applyng ANNs, such as data normalzaton, data balancng, and prncpal component analyss. RECOMMENDATION Future study should be conducted to mprove the generalzaton ablty of the BPNNs. The possble methods are: ) modfy the nput data to nclude more nformaton from surroundng area and also the data from more upper layers, 2) classfy the quanttatve precptaton data nto dfferent levels, such as lght ran, moderate ran, and heavy ran, and replace the exact quanttatve precptaton wth sutable ran levels, 3) adust the topology of networks, ncrease hdden layers or ntroduce dfferent actvaton functons, and 4) fnd an effcent algorthm, such as Genetc Algorthm, to decde sutable network free parameters. ACKNOWLEDGEMENT I would lke to express my solemn grattude to my supervsor Prof. A. W. Jayawardena, Research and Tranng Advsor, ICHARM, PWRI, Tsukuba, Ibarak, Japan for hs overall supervson, nvaluable suggestons and ardent encouragement n every aspect of ths work. REFERENCES Cao L.J and Tay Francs E.H. (2003). Support vector machne wth adaptve parameters n fnancal tme seres forecastng. IEEE transactons on neural networks, Vol. 4, No.6, Flood I and Kartam N. (994). Neural networks n Cvl Engneerng: Prncples and understandng. Journal of Computng n Cvl Engneerng (ASCE), 8(2), Haykn, S. (999). Neural networks: A comprehensve foundaton. New York: Macmllan Publshng. Kotsants S.B, Kanellopoulos D and Pntelas P.E (2006). Data pre-processng for supervsed learnng. Internatonal Journal of Computer Scence, Vol No.2, -7. Kubat, M. and Matwn, S. (997). Addressng the curse of mbalanced tranng sets: one sded selecton. Proceedngs of the Fourteenth Internatonal Conference on Machne Learnng (ICML 997), Nashvlle, Tennessee, USA, July 8-2, 997, Lng, C. and L, C. (998). Data mnng for drect marketng: problems and solutons. Amercan Assocaton for Artfcal Intellgence, KDD-98, Mnns, A. W and Hall, M. J. (996). Artfcal neural networks as ranfall-runoff models. Hydrol. Sc. J.,4, 3,