Non-Linear Continuum Regression Using Genetic Programming

Transcription

1 Non-Lnear Contnuum Regresson Usng Genetc Programmng Ben McKay Mark Wlls Domnc Searson Dept. Chemcal & Process Engng. Unversty of Necastle NE 7RU, UK Gary Montague Abstract In ths contrbuton, genetc programmng s combned th contnuum regresson to produce to novel non-lnear contnuum regresson algorthms. he frst s a sequental algorthm hle the second, adopts a team-based strategy. Havng dscussed contnuum regresson, the modfcatons requred to extend the algorthm for non-lnear modellng are outlned. he results of applyng the derved non-lnear algorthms to the development of an nferental model of a food extruson process are then presented. he superor performance of the sequental algorthm, as compared to a smlar non-lnear partal least squares algorthm, s demonstrated. In addton, these results clearly demonstrate that the team-based strategy sgnfcantly outperforms the sequental approach. INRODUCION In ths paper Genetc Programmng (GP) s used to develop to non-lnear contnuum regresson algorthms for buldng nput-output models of chemcal process systems. A team-based strategy s nvestgated, and compared to a more tradtonal sequental approach. Why are chemcal engneers nterested n data-based modellng? Recent years have seen an ncrease n the emphass on product 'qualty', economc process performance, envronmental and safety ssues n the process ndustres, and these factors have placed sgnfcant demands on exstng operatonal procedures. Process montorng, optmsaton and advanced control have the potental to satsfy many of these demands. Hoever, n order to realse the benefts of these technques, t s generally necessary to have an accurate model of the process. Whle t may be possble to develop such a model from frstprncples (usng a detaled knoledge of the physcs and chemstry of a system), there are a number of drabacks to ths approach. As many ndustral process systems are extremely complex and relatvely poorly understood, the development of a realstc model can take a consderable amount of tme and effort. In addton, the modellng process nevtably nvolves a number of smplfyng assumptons that have to be made n order to provde a tractable soluton. herefore, a mechanstc model ll often be costly to develop and may be subject to naccuraces. Consequently, data-based modellng (usng plant data to buld an nput-output model that descrbes the response of process outputs to changes n nputs, thout attemptng to represent the underlyng process mechansms) presents a popular alternatve. Why use non-lnear models? Many chemcal process systems are non-lnear n nature. herefore, n order to acheve suffcent model accuracy, a non-lnear model structure s typcally requred. Why not use a standard GP algorthm? he performance of a standard GP algorthm hen used to develop non-lnear models of chemcal process systems (usng ether smulated or ndustral nput-output data) can be dsappontng hen compared to other non-lnear regresson technques such as feedforard neural netorks (e.g. see McKay, 997, Hden, 998). Generally, a GP algorthm s much sloer than compettve technques (especally for ncreasng numbers of nput varables) and the predcton errors on unseen (test) data are typcally hgher, ndcatng a less accurate model. Why the nterest n Contnuum Regresson? Our recent ork has shon that strateges that reduce the algorthm s search space can help the overall performance of a GP algorthm (hen used for model development). A number of multvarate statstcal modellng technques provde systematc procedures for the decomposton of an nput-output relatonshp. By extractng and structurng nformaton n an approprate manner, these technques can be used to reduce the effectve search space. In Hden et al. (998) t as demonstrated that the combnaton of GP th one such technque, knon as Partal Least Squares (PLS), could produce results of comparable accuracy to alternatve non-lnear regresson technques such as neural netorks. Hoever, PLS s merely one of a famly of lnear multvarate statstcal modellng

2 technques. It s Contnuum Regresson (CR) that provdes a unfed frameork encompassng all of these technques. Wse and Rcker (997) demonstrated the effectveness of CR, developng models of mproved accuracy hen compared to alternatve approaches (thout sacrfcng model robustness and generalsaton). Hoever, standard CR algorthms produce lnear models. In ths paper, e combne GP and CR to produce nonlnear CR algorthms. he objectve beng the development of an algorthm that can develop accurate and robust non-lnear models. Ho s ths paper organsed? he next secton provdes some background nformaton on lnear regresson. CR s then ntroduced and the range of technques t encompasses are explaned. Next, the modfcatons requred to extend the algorthm to a nonlnear strategy are outlned. o approaches are proposed, a sequental and a team-based algorthm. Havng detaled the algorthms, ther performance on a benchmark example are studed: he development of an nferental model of a food extruson process. After dscusson of the results, conclusons are dran and recommendatons for further ork are made. 2 LINEAR REGRESSION Gven a set of nput and output measurements for a process, a typcal modellng objectve s to obtan a relatonshp, that can be used to explan the varaton n an (n x ) output vector, y, n response to changes n an (n x m) nput matrx, X. hs can be expressed mathematcally as follos: y = f(x) + e () Where the functon f(.) s chosen so as to mnmse the vector of predcton errors, e. If f(.) s lnear n the parameters, then the nput-output model reduces to: y = Xr + e (2) here r s a (m x ) vector of regresson coeffcents. A famly of lnear multvarate statstcal modellng algorthms have been developed for the purpose of estmatng r. hree of the more common and effectve of these are Multple Lnear Regresson, Prncpal Component Regresson and Partal Least Squares. What s Multple Lnear Regresson? Multple Lnear Regresson (MLR) uses one of the many varants of the Least Squares technque (such as Batch Least Squares ) to parameterse Equaton 2. A dra-back of ths method s that t fals to produce accurate and robust models f the nput data s correlated (the columns of nput data are lnearly related to each other). hs s the norm hen data s collected from chemcal process systems. What s Prncpal Component Regresson? PCR avods the problems assocated th the modellng of correlated nput data (e. sngular solutons or mprecse parameter estmatons) by transformng the nputs nto a ne set of uncorrelated data (typcally of reduced dmensonalty), and then performng MLR beteen ths transformed data set and the output data. he procedure s based on a technque knon as Prncpal Component Analyss (PCA), one of the oldest and best documented multvarate statstcal technques (Pearson, 90). PCA s performed on the nput data, X, and generates a set of transformed nput varables (knon as prncpal components) that are uncorrelated and ranked n term of sgnfcance. hs allos the least sgnfcant prncpal components (ones that only descrbe process nose) to be dscarded pror to modellng. More rgorously, PCA rotates X to produce an (n x m) scores matrx, and an (m x m) loadngs matrx, V (descrbng the requred rotaton) such that, =XV. he loadng matrx V contans the egenvectors of the nput data correlaton matrx, X X. he scores matrx,, contans orthogonal projectons of the nput varables, sorted (from left to rght) n terms of decreasng contrbuton to the varance n the nput data. Retanng the frst p prncpal components leads to a reduced (n x p) scores matrx p and an (m x p) loadngs matrx V p. o perform PCR, batch least squares s used to regress p aganst y, gvng the (p x ) regresson vector b, such that y= p b + e. he nput-output mappng s then gven as, y = XV p b + e (3) Unfortunately, as PCA only consders the varaton n the nput data, there s no guarantee that the varaton represented by the major prncpal components ll correspond to the varaton that best descrbes the relatonshp th the output varable. What s Partal Least Squares? Partal Least Squares (PLS) as frst descrbed n Wold (966). As th PCR, the PLS algorthm overcomes problems assocated th measurement nose and correlaton beteen nput varables. Hoever, an addtonal advantage of PLS (as compared to PCR), s that by consderng both the nput varance and the nputoutput covarance, the algorthm provdes the opportunty for more accurate model development. A good ntroducton to the technque can be found n Gelad and Koalsk (986). PLS s commonly mplemented usng the NIPALS (Nonlnear Iteratve Partal Least Squares) algorthm. NIPALS sequentally extracts pars of latent vectors (analogous to the columns of the scores matrx n PCA) from the nput and output data. A unvarate (sngle nput, sngle output) regresson s performed at each stage, to model the relatonshp beteen these latent vector pars. he fnal model s then constructed by summng the contrbuton from each step. More rgorously, PLS (startng th =, X =X and, y =y) sequentally extracts latent vector pars, t and u, from X an y, n order of decreasng predctve poer. For the sngle output case, the vector u s equal to y,, hle the vector t corresponds to the projecton of X, n the

3 drecton most correlated to u (.e. t =X. here are the optmal nput projecton eghts).. he vectors t and u, are then regressed to obtan a unvarate lnear model th a regresson parameter b (such that u = b t + e ). As each latent vector s calculated the nformaton used s deducted from the nput-output data and these resduals (X + and y + ) are used to calculate the next set of latent vectors. hs procedure s repeated untl addtonal latent vector pars fal to mprove the model performance. he fnal model s then constructed by summng the contrbutons from each of the sgnfcant latent vector pars, y ˆ = b X (4) = here ŷ s the estmated output y. Substtutng the expresson for calculatng the nput resduals (X + = X - X ) nto Equaton 4 and recallng that X =X gves, yˆ = Xb + X ( I ) b + K + X ( I 2 K ( ) 2 ( ) Usng the fact that the are orthogonal (and thus j =0, for j), ths reduces to a form that s equvalent to Equaton 2, N ) b LV y = X b + e = Usng PLS, dffcultes assocated th modellng correlated nput varables are avoded by calculatng latent varables sequentally and only performng a unvarate regresson at each stage. Problems assocated th measurement nose are also overcome by projectng the nputs and outputs onto a loer dmensonal space. In addton, consderng both the nput varance and the nput-output covarance leads to mproved accuracy. Ho are the methods related? (5) (6) model For a lnear nput-output mappng, f all prncpal components or latent varables are retaned hen performng PCR or PLS, the to methods are equvalent to MLR. It s prmarly the dfference n emphass placed on varaton n the nput data compared to covarance beteen the nputs and the output that dstngushes the varous technques. A mathematcal frameork for unfyng these technques ll be ntroduced n the next secton. 3 CONINUUM REGRESSION Stone and Brooks (990) and Wse and Rcker (993) have demonstrated that MLR, PCR and PLS can be unfed under a sngle frameork knon as Contnuum Regresson. At one extreme of the contnuum s PCR, at the other s MLR, hle PLS s n the mddle. A qualtatve nterpretaton of ths concept can be obtaned by consderng ths contnuum n terms of the emphass that s placed on the varance n the nput data, hen compared to the covarance beteen the nputs and the outputs. Wth MLR, only the nput-output covarance s consdered hen formulatng the model (n fact the nputs are regarded as ndependent).wth PCR, a model s developed beteen the output and the sgnfcant prncpal components. hese components represent the most mportant varatons n the nput data, but ther covarance th the output s not necessarly maxmsed. Whle th PLS, a compromse beteen MLR and PCR s obtaned by consderaton of both the correlaton n the nput data and nput-output covarance. CR allos a smooth transton beteen these technques. What s the CR algorthm? he CR algorthm proposed by Wse and Rcker (993), frst creates a transformed data set by applyng a sophstcated scalng technque to the nput data. hs effectvely arps the nput space to place ether more or less emphass on exstng correlatons (dependng on a factor to be referred to as the contnuum coeffcent ). PLS (usng NIPLAS, as outlned n Secton 2) s then performed on the transformed data set. he frst step n transformng the nput data s to decompose X usng the sngular value decomposton (SVD) as, X = UΣV (7) here U s an (n x n) matrx of the egenvectors of XX and V s an (m x m) matrx of the egenvectors of the data correlaton matrx, X X. he dagonal elements of the (n x m) matrx Σ are the postve square roots of the egenvalues of X X and are called the sngular values (all other elements of Σ are zero). A modfed X matrx, (X µ ) s then formed as, X µ = UΣ µ V (8) here the matrx Σ µ s the sngular values rased to the poer µ (0 µ ). Where µ ll be referred to as the contnuum coeffcent. he standard NIPALS algorthm s then appled to X µ and y, gvng a fnal model of the follong form, y ˆ = N LV b = X In a smlar manner to Equaton 4, ths can be converted nto a form that s equvalent to Equaton 2 as follos, µ LV (µ ) y = XVΣ V b + e N = (9) (0) What s the qualtatve nterpretaton of the contnuum coeffcent? µ 0 CR tends toards MLR. µ = CR s equvalent to PLS. µ CR tends toards PCR.

4 4 NON-LINEAR CR USING GP o approaches for combnng GP and CR to create a non-lnear CR algorthm are proposed. he frst evolves nner models sequentally, hle the second evolves a team of nner models smultaneously. What s the sequental algorthm? he sequental non-lnear CR algorthm performs a GP run each tme a non-lnear nner model s requred (e. at each teraton of the latent varable loop). he mathematcal detals of the algorthm can be found n Appendx. At Step 8, a standard GP algorthm mnmses the cost functon J, by judcus choce of nonlnear functon, f (.). he optmal contnuum coefcent (µ) s obtaned by an outer loop that performs a undmensonal search usng Brent s method (Press et al., 992). What s the team-based algorthm? Rather than perform a GP run each tme a non-lnear nner model s requred, t s possble to evolve a team of nner models smultaneously usng a mult-populaton GP algorthm. Each team s composed of N p ndvduals (each an nner model), dran from N p separate populatons {P,, P Np }. eam members are assgned specfc tasks, based on the populaton from hch they are dran (eg. Members of populaton P are assgned the task of developng an nner model beteen the frst latent vector par, hle populaton P j contans canddates for the j th nner model). In ths manner, a team of non-lnear functons {f (.),,f Np (.)}.are evolved to mnmse the overall model predcton error (e n Equaton ). Ftness s calculated on a team bass. he ftness of a team s determned by mplementng the CR algorthm gven n Appendx (usng the nner models defned by the gven team). For the th latent varable par, at Step 8, the team Sequental eam-based Functon Set +, -, /, *, protected ^, protected, exp(), protected ln(), tanh() ermnal Set t,constants {t, t m,},constants Output: A sngle nner model for the A team of N P nner models. th latent varable par. Populaton Sze: Ftness functon Lnear rankng Crossover Probablty 0.7 Mutaton Probablty: 0.2 Drect Reproducton 0. Probablty Proporton of elte 0. ndvduals retaned: ermnaton crteron 40 Gens. 60 Gens. Parameter optmsaton Yes No otal # of runs 20 Average # ndvduals processed per run: 5.0x x0 5 able : Confguraton of the GP Algorthms member correspondng to the th nner model s evaluated. Each ndvdual n a team s then assgned the teams ftness. Each populaton s homogenous (e. only ndvduals from the same populatons can be crossed over). he crossover strategy employed does not mantan the ntegrty of teams (e. t s non-cohesve). he varous populatons are treated ndependently hen selectng ndvduals for crossover (rather than crossng over complete teams). Pror to ftness evaluaton, a teams contnuum coeffcent (µ) as determned n one of to ays. Ether, µ as optmsed usng Brent s method (th a probablty of 0.), or a random Gaussan perturbaton as made to the value of µ assocated th the fttest team of the prevous generaton. hs perturbed value of µ as then used to evaluate the teams ftness. 5 CONFIGURAION OF HE GP ALGORIHMS able shos the values of the most mportant GP algorthm control parameters, as ell as a summary of the termnal and functon sets used. It may be noted that the team-based algorthm used a sgnfcantly larger populaton than the sequental. Hoever, as the sequental algorthm as employng a parameter optmser (Levenberg-Marquardt non-lnear least squares), the sequental algorthm actually processed tce as many ndvduals per run. 6 EXPERIMENAL MEHOD For all of the results presented, the Root Mean Square () error beteen the actual and predcted output as calculated based on an unseen (or test) data set. hs test set s used to verfy the generalsaton of the model. Due to the stochastc nature of the GP algorthm, usng t on anythng but the smplest of systems produces dfferent results for every run. herefore, n order to nvestgate the performance of a GP-based algorthm for model buldng multple runs ere performed for each technque. 7 MODELLING A FOOD EXRUSION PROCESS As descrbed n Elsey et al (997), the parameters of ths smulated process have been ftted to plant data obtaned from a plot-scale APV Baker MPF 40 cookng extruder, processng a mxture of corn flour and ater. Gven the scre speed, feed rate, feed mosture content, feed temperature and barrel temperature profle, the model calculates the degree of gelatnsaton of the product. hs as selected as the process output of nterest n these studes. herefore, the termnal set for ths system conssts of four varables, scre speed, feed rate, feed mosture content and feed temperature. Four hundred steady-state readngs ere avalable for modellng. hs data as splt evenly th the frst 200 ponts beng used

5 to tran the models and the remanng 200 ponts reserved for testng. hs data set has become our benchmark example for comparng non-lnear modellng technques. A selecton of pertnent results are presented n able 2. hese results demonstrate that standard GP performs poorly hen compared to a feedforard neural netork (FANN). hs led to the development of to GP-based non-lnear PLS algorthms. he frst, GP_PLS (the so called quck and drty approach), s equvalent to the sequental algorthm presented n ths paper, hen the contnuum coeffcent s set to one. Whle the second, GP_PLS2, demonstrated that optmsaton of the nput projecton eghts ( ) can be benefcal n reducng the model predcton errors. hs s because, n lnear PLS the s calculated to acheve maxmum lnear covarance beteen t and u even though the nner models are n fact non-lnear. herefore, for a gven non-lnear functon, may not correspond to the drecton that produces the best approxmaton of u. Hden et al. (998) demonstrated that GP_PLS2 could produce results of comparable accuracy to a FANN. Hoever, t as concluded that optmsng as an unacceptable computatonal burden (especally th ncreasng numbers of nput varables), lmtng the applcaton of the algorthm. As such, n ths ork no attempt as made to optmse the nput projecton eghts. Fgure shos a comparson of the error dstrbutons obtaned usng the sequental and teambased algorthms (hle able 3 summarses the average Frequency 0 5 Sequental eam-based Error on est Set Fgure : Model error dstrbutons on the cookng extruder data. Method Mean est Mn. est Source FANN McKay (997) Standard GP Hden(998) PLS - Lnear GP_PLS GP_PLS Hden et al. (998) able 2: Summary of pror modellng ork on the cookng extruder data. Method Mean est Mn. est µ Best Model Best Model Sequental eam-based able 3: Results for the sequental and team-based algorthms on the cookng extruder data. and best errors). It may be noted that the sequental algorthm outperforms GP_PLS. hs s because the optmsaton of µ has a smlar effect to optmsng the nput eghts (n that t changes the projecton of the nput data n an attempt to maxmse the model ft). Hoever, as there s only one contnuum coeffcent (as compared to hch s an (m x ) vector, here m s the number of nput varables) the optmsaton problem becomes more tractable. It s also apparent that the team-based approach produces sgnfcantly loer errors on the test set, compared to the sequental algorthm. hs s despte the fact that the sequental algorthm processed more ndvduals (as shon n able ). he sequental algorthm fnds the best ft for each latent varable par n turn. hs procedure effectvely constrans the type of solutons the algorthm ll fnd. Whle resultng n good local solutons at each step, t s conjectured that t may be prone to beng trapped n local mnma. In comparson, the team-based algorthm optmses all of the nner models smultaneously. As Ftness s calculated on a team-bass, ths allos a tradeoff beteen team member contrbutons to the overall ftness measure. hs provdes the potental for escape from local mnma and the possblty of obtanng a more global optmum. he best set of nner models obtaned usng the team-based algorthm are gven by Equatons, 2, and 3, u = 2.6 exp(tanh(tanh(exp(4t )))) () u 2 = ( tanh(t 2 ) t2 + (2 t )/exp(t 2 ) (2) + u t 2 2 ) u 3 = 0.80 (t ) tanh(2t 3 ) (3) 8 CONCLUSIONS he food extruson benchmark example shos that nonlnear CR usng GP gves superor performance to a smlar mplementaton of non-lnear PLS. More specfcally, the sequental non-lnear CR algorthm outperforms GP_PLS of Hden et al.(998). It s conjectured that ths s because the optmsaton of µ has a smlar effect to optmsng the nput projecton eghts. Both change the projecton of the nput data n an attempt to maxmse the model ft. Hoever, optmsng the contnuum coeffcent s less computatonally ntensve than optmsng (hch s an (m x ) vector, here m s the number of nput varables). he case study also demonstrates the effectveness of the team-based approach to non-lnear CR, as compared to the sequental algorthm. By fndng the best ft for each latent varable par n turn, the sequental technque effectvely constrans the type of solutons the algorthm ll fnd. Whle concentratng on achevng good nner models at each step, t may be prone to beng trapped n local mnma. By optmsng all of the nner models smultaneously, the team-based algorthm allos tradeoffs beteen ndvduals contrbutons to the overall team

6 ftness. hs may allo the evoluton of a more globally optmal soluton. As explaned n Secton 4, the team-based algorthm employs a non-cohesve reproducton strategy. hs means that teams are broken up durng reproducton (e the ntegrty of the team s not mantaned). If only one type of team strategy s gong to ork (e. f there s only one good ay of performng each task), then dsruptng the ntegrty of the teams ll probably not have a detrmental effect. Whle e beleve that the non-lnear CR modellng task s an example of such a system (here the form of each nner model, and hence the task of each team member, s largely dctated by the data set), future ork ll examne the effect of mplementng a cohesve crossover strategy. As noted n Secton 7, the team-based non-lnear CR algorthm does not outperform GP_PLS2, the eght optmsed non-lnear PLS algorthm of Hden et al., (998). Hoever, t as consdered that optmsng the projecton eghts as currently an unacceptable computatonal burden. As such, fndng an mproved method of mplementng the nput eght optmsaton ll be a focus of future ork. 9 REFERENCES Elsey, J. Repenhausen, J. McKay, B., Barton, G.W. and Wlls, M.J., (997), Modellng and Control of a Food Extruson Process, Computers and Chemcal Engneerng, Vol. S2, pp S36-S366. Gelad, P. and Koalsk, B.R. (986), Partal least squares regresson: A tutoral, Analytca Chmca Acta, Vol.85, pp.-7. Hden, H.G. (998), Data-Based Modellng usng Genetc Programmng, PhD hess, Dept. Chemcal and Process Engneerng, Unversty of Necastle, UK. Hden, H.G., McKay, B., Wlls, M.J. and Montague, G.A. (998) Non-lnear partal least squares usng genetc programmng, Proc. 3 rd Annual Conf. On Genetc Programmng, Unversty of Wsconsn, Maddson, USA. July th., pp McKay, B., (997), Studes n Data-Based Modellng, PhD hess, Dept.Chemcal Engneerng, he Unversty of Sydney, Australa. McKay, B., Wlls, M.J. and Barton, G.W., (997), Steady-state modellng of chemcal process systems usng genetc programmng, Computers and Chemcal Engneerng, Vol.2, No.9, pp Pearson, K., (90), On lnes and planes of closest ft to systems of ponts n space, Phl. Mag., Ser.6, 2(), pp Press, W.H., Flannery, B.P., eukolsky, S.A and Vetterlng, W.., (992), Numercal Recpes n C: he Art of Scentfc Computng, 2nd Ed, Cambrdge Unversty Press, USA. Stone and Brooks (990), Contnuum regresson: Crossvaldated sequentally constructed predcton embracng ordnary least squares, partal least squares, and prncpal components regresson, J.R.Statst. Soc., B,52, pp Wse, B.M. and Rcker, N.L. (993) Identfcaton of fnte mpulse response models th contnuum regresson, Journal of Chemometrcs, Vol. 7, -4. Wold, S., (966), Non-lnear estmaton by teratve least squares procedures, Research Papers n Statstcs, Ed. Davd, F., Wley, Ne York, USA. APPENDIX : Non-Lnear Contnuum Regresson Algorthm Sequental Algorthm: he algorthm n able A represents a sngle functon evaluaton for a gven value of µ. hs s called repeatedly by Brent s method to optmse µ. Note that a full GP run s performed at Step 8 to optmse the ft of each nner model. eam-based Algorthm: he algorthm n able As called by a mult-populaton GP algorthm as a sngle ftness evaluaton, for a gven team of nner models, and a gven µ. () Mean centre and normalse the nput and output data (X and y) (2) Perform SVD on scaled nput data (X) UΣV = X (n x n)(n x m)(m x m) (3) Generate modfed nput data (X µ ) X µ = UΣ µ V (n x m) matrx (4) Set the output latent vector (u ) equal to the output. u = y (n x ) column vector (5) Calculate eghts ( ) that correspond to the drecton n = ( X ) u (m x ) column vector X th the greatest covarance th u. (6) Normalse to unt length = (m x ) column vector (Note. refers to the Eucldean Norm) (7) Calculate the latent vector t hch s the projecton of X onto t = X (n x ) column vector the drecton defned by. (8) Sequental: mnj eam-based: Use suppled f ( f (. ))( f (. )) (.) J = u t u t f(.) (9) Calculate resdual matrces X = X t (n x m) matrx + (n x ) column vector (0) If <, {Return to Step 5; = +} () Construct the fnal model y = y f ( t ) + yˆ = = f ( X ) able A: he non-lnear contnuum regresson algorthm. (m x ) column vector