MARS: Still an Alien Planet in Soft Computing?

Transcription

1 MARS: Stll an Alen Planet n Soft Computng? Ajth Abraham and Dan Stenberg School of Computng and Informaton Technology Monash Unversty (Gppsland Campus), Churchll 3842, Australa Emal: ajth.abraham@nfotech.monash.edu.au Salford Systems Inc 8880 Ro San Dego, CA 9208, USA Emal: dsten@salford-systems.com Abstract: The past few years have wtnessed a growng recognton of soft computng technologes that underle the concepton, desgn and utlzaton of ntellgent systems. Accordng to Zadeh [], soft computng conssts of artfcal neural networks, fuzzy nference system, approxmate reasonng and dervatve free optmzaton technques. In ths paper, we report a performance analyss among Multvarate Adaptve Regresson Splnes (MARS), neural networks and neuro-fuzzy systems. The MARS procedure bulds flexble regresson models by fttng separate splnes to dstnct ntervals of the predctor varables. For performance evaluaton purposes, we consder the famous Box and Jenkns gas furnace tme seres benchmark. Smulaton results show that MARS s a promsng regresson technque compared to other soft computng technques.. Introducton Soft Computng s an nnovatve approach to construct computatonally ntellgent systems that are supposed to possess humanlke expertse wthn a specfc doman, adapt themselves and learn to do better n changng envronments, and explan how they make decsons. Neurocomputng and neuro-fuzzy computng are wellestablshed soft computng technques for functon approxmaton problems. MARS s a fully automated method, based on a dvde and conquer strategy, parttons the tranng data nto separate regons, each wth ts own regresson lne or hyperplane [2]. MARS strengths are ts flexble framework capable of trackng the most complex relatonshps, combned wth speed and the summarzng capabltes of local regresson. Ths paper nvestgates the performance of neural networks, neuro-fuzzy systems and MARS for predctng the well-known Box and Jenkns tme seres, a benchmark problem used by several connectonst researchers. We begn wth some theoretcal background about MARS, artfcal neural networks and neuro-fuzzy systems. In Secton 6 we present expermentaton setup usng MARS and soft computng models followed by results and conclusons. 2. What Are Splnes? Splnes can be consdered as an nnovatve mathematcal process for complcated curve drawngs and functon approxmaton. Splnes fnd ever-ncreasng applcaton n the numercal methods, computer-aded desgn, and computer graphcs areas. V.N. Alexandrov et al. (Eds.): ICCS 200, LNCS 2074, pp , 200. Sprnger-Verlag Berln Hedelberg 200

2 236 A. Abraham and D. Stenberg Mathematcal formulae for crcles, parabolas, or sne waves are easy to construct, but how does one develop a formula to trace the shape of share value fluctuatons or any tme seres predcton problems? The answer s to break the complex shape nto smpler peces, and then use a stock formula for each pece [4]. To develop a splne the X-axs s broken nto a convenent number of regons. The boundary between regons s also known as a knot. Wth a suffcently large number of knots vrtually any shape can be well approxmated. Whle t s easy to draw a splne n 2-dmensons by keyng on knot locatons (approxmatng usng lnear, quadratc or cubc polynomal etc.), manpulatng the mathematcs n hgher dmensons s best accomplshed usng bass functons. 3. Multvarate Adaptve Regresson Splnes (MARS) The MARS model s a splne regresson model that uses a specfc class of bass functons as predctors n place of the orgnal data. The MARS bass functon transform makes t possble to selectvely blank out certan regons of a varable by makng them zero, allowng MARS to focus on specfc sub-regons of the data. MARS excels at fndng optmal varable transformatons and nteractons, as well as the complex data structure that often hdes n hgh-dmensonal data [3]. knots Fg.. MARS data estmaton usng spnes and knots (actual data on the rght) Gven the number of predctors n most data mnng applcatons, t s nfeasble to approxmate the functon y=f(x) n a generalzaton of splnes by summarzng y n each dstnct regon of x. Even f we could assume that each predctor x had only two dstnct regons, a database wth just 35 predctors would contan more than 34 bllon regons. Gven that nether the number of regons nor the knot locatons can be specfed a pror, a procedure s needed that accomplshes the followng: judcous selecton of whch regons to look at and ther boundares, and judcous determnaton of how many ntervals are needed for each varable A successful method of regon selecton wll need to be adaptve to the characterstcs of the data. Such a soluton wll probably reject qute a few varables (accomplshng

3 MARS: Stll an Alen Planet n Soft Computng? 237 varable selecton) and wll take nto account only a few varables at a tme (also reducng the number of regons). Even f the method selects 30 varables for the model, t wll not look at all 30 smultaneously. Such smplfcaton s accomplshed by a decson tree (e.g., at a sngle node, only ancestor splts are beng consdered; thus, at a depth of sx levels n the tree, only sx varables are beng used to defne the node). MARS Smoothng, Splnes, Knots Selecton, and Bass Functons A key concept underlyng the splne s the knot. A knot marks the end of one regon of data and the begnnng of another. Thus, the knot s where the behavor of the functon changes. Between knots, the model could be global (e.g., lnear regresson). In a classcal splne, the knots are predetermned and evenly spaced, whereas n MARS, the knots are determned by a search procedure. nly as many knots as needed are ncluded n a MARS model. If a straght lne s a good ft, there wll be no nteror knots. In MARS, however, there s always at least one "pseudo" knot that corresponds to the smallest observed value of the predctor. Fgure depcts a MARS splne wth three knots. Fg. 2. Varatons of bass functons for c = 0 to 80 Fndng the one best knot n a smple regresson s a straghtforward search problem: smply examne a large number of potental knots and choose the one wth the best R 2. However, fndng the best par of knots requres far more computaton, and fndng the best set of knots when the actual number needed s unknown s an even more challengng task. MARS fnds the locaton and number of needed knots n a forward/backward stepwse fashon. A model whch s clearly overft wth too many knots s generated frst, then, those knots that contrbute least to the overall ft are removed. Thus, the forward knot selecton wll nclude many ncorrect knot locatons, but these erroneous knots wll eventually, be deleted from the model n the backwards prunng step (although ths s not guaranteed). Thnkng n terms of knot selecton works very well to llustrate splnes n one dmenson; however, ths context s unweldy for workng wth a large number of

4 238 A. Abraham and D. Stenberg varables smultaneously. Both concse notaton and easy to manpulate programmng expressons are requred. It s also not clear how to construct or represent nteractons usng knot locatons. In MARS, Bass Functons (BFs) are the machnery used for generalzng the search for knots. BFs are a set of functons used to represent the nformaton contaned n one or more varables. Much lke prncpal components, BFs essentally re-express the relatonshp of the predctor varables wth the target varable. The hockey stck BF, the core buldng block of the MARS model s often appled to a sngle varable multple tmes. The hockey stck functon maps varable X to new varable X*: max (0, X -c), or max (0, c - X) where X* s set to 0 for all values of X up to some threshold value c and X* s equal to X for all values of X greater than c. (Actually X* s equal to the amount by whch X exceeds threshold c). The second form generates a mrror mage of the frst. Fgure 2 llustrates the varaton n BFs for changes of c values (n steps of 0) for predctor varable X, rangng from 0 to 00. MARS generates bass functons by searchng n a stepwse manner. It starts wth just a constant n the model and then begns the search for a varable-knot combnaton that mproves the model the most (or, alternatvely, worsens the model the least). The mprovement s measured n part by the change n Mean Squared Error (MSE). Addng a bass functon always reduces the MSE. MARS searches for a par of hockey stck bass functons, the prmary and mrror mage, even though only one mght be lnearly ndependent of the other terms. Ths search s then repeated, wth MARS searchng for the best varable to add gven the bass functons already n the model. The brute search process theoretcally contnues untl every possble bass functon has been added to the model. In practce, the user specfes an upper lmt for the number of knots to be generated n the forward stage. The lmt should be large enough to ensure that the true model can be captured. A good rule of thumb for determnng the mnmum number s three to four tmes the number of bass functons n the optmal model. Ths lmt may have to be set by tral and error. 4. Artfcal Neural Network (ANN) ANN s an nformaton-processng paradgm nspred by the way the densely nterconnected, parallel structure of the mammalan bran processes nformaton. Learnng n bologcal systems nvolves adjustments to the synaptc connectons that exst between the neurons [7]. Learnng typcally occurs by example through tranng, where the tranng algorthm teratvely adjusts the connecton weghts (synapses). These connecton weghts store the knowledge necessary to solve specfc problems. A typcal three-layer feedforward neural network s llustrated n Fgure 3. Backpropagaton (BP) s one of the most famous tranng algorthms for multlayer perceptrons. BP s a gradent descent technque to mnmze the error E for a

5 MARS: Stll an Alen Planet n Soft Computng? 239 partcular tranng pattern. For adjustng the weght ( w j ) from the -th nput unt to the j-th output, n the batched mode varant the descent s based on the gradent E ( ( ) for the total tranng set: Z j Zj (n) = * ( + * Z j (n ) Z j The gradent gves the drecton of error E. The parameters e and a are the learnng rate and momentum respectvely. Fg. 3. Typcal three-layer feedforward network archtecture 5. Neuro-fuzzy (NF) System We defne a NF [6] system as a combnaton of ANN and Fuzzy Inference System (FIS) [9] n such a way that neural network learnng algorthms are used to determne the parameters of FIS. As shown n Table, to a large extent, the drawbacks pertanng to these two approaches seem largely complementary. Table. Complementary features of ANN and FIS ANN FIS Black box Learnng from scratch Interpretable Makng use of lngustc knowledge In our smulaton, we used ANFIS: Adaptve Network Based Fuzzy Inference System [5] as shown n Fgure 5, whch mplements a Takag Sugeno Kang (TSK) fuzzy

6 240 A. Abraham and D. Stenberg nference system (Fgure 4) n whch the concluson of a fuzzy rule s consttuted by a weghted lnear combnaton of the crsp nputs rather than a fuzzy set. Fg. 4. TSK type fuzzy nference system premse parameters x consequent parameters A x A 2 A 3 f B 5 y B 2 B y 4 Fg. 5. Archtecture of the ANFIS Archtecture of ANFIS and the functonalty of each layer s as follows: Layer- Every node n ths layer has a node functon = µ A ( x ), for =, 2 or = µ B ( y ) 2

7 MARS: Stll an Alen Planet n Soft Computng? 24 s the membershp grade of a fuzzy set A ( = A, A 2, B or B 2 ), specfes the degree to whch the gven nput x (or y) satsfes the quantfer A. Usually the node functon can be any parameterzed functon.. A gaussan membershp functon s specfed by two parameters c (membershp functon center) and (membershp functon wdth) 2 x c JXDVVLDQ[F e 2 σ Parameters n ths layer are referred to premse parameters. Layer-2 Every node n ths layer multples the ncomng sgnals and sends the product out. Each node output represents the frng strength of a rule. 2 = w = µ A ( x ) µ B ( y ), =,2.... In general any T-norm operators that perform fuzzy AND can be used as the node functon n ths layer. Layer-3 Every -th node n ths layer calculates the rato of the -th rule s frng strength to the sum of all rules frng strength. w 3 = w =, =,2.... w + w 2 Layer-4 Every node n ths layer has a node functon 4 = w f = w ( p x + q y + r ), w s the output of layer3, and { } where p, q, r s the parameter set. Parameters n ths layer wll be referred to as consequent parameters. Layer-5 The sngle node n ths layer labeled computes the overall output as the summaton of all ncomng sgnals: = = = w f 5 verall output w f. w ANFIS makes use of a mxture of backpropagaton to learn the premse parameters and least mean square estmaton to determne the consequent parameters. A step n the learnng procedure has two parts: In the frst part the nput patterns are propagated, and the optmal concluson parameters are estmated by an teratve least mean square procedure, whle the antecedent parameters (membershp functons) are assumed to be fxed for the current cycle through the tranng set. In the second part the patterns are propagated agan, and n ths epoch, backpropagaton s used to modfy the antecedent parameters, whle the concluson parameters reman fxed. Ths procedure s then terated.

8 242 A. Abraham and D. Stenberg Fg. 6. 3D vew of Gas furnace tme seres tranng data (I/ relatonshp) 6. Expermental Setup Usng Soft Computng Models and MARS Gas Furnace Tme Seres Data: Ths tme seres was used to predct the C 2 (carbon doxde) concentraton y(t+) [0]. In a gas furnace system, ar and methane are combned to form a mxture of gases contanng C 2. Ar fed nto the gas furnace s kept constant, whle the methane feed rate u(t) can be vared n any desred manner. After that, the resultng C 2 concentraton y(t) s measured n the exhaust gases at the outlet of the furnace. Data s represented as [u(t), y(t), y(t+)]. The tme seres conssts of 292 pars of observaton and 50% was used for tranng and remanng for testng purposes. Fgure 6 shows the complexty of the nput / output relatonshp n two dfferent angles. ur experments were carred out on a PII, 450MHz Machne and the codes were executed usng MATLAB and C++. ANN tranng We used a feedforward neural network wth hdden layer consstng of 24 neurons (tanh-sgmodal node transfer functon). The tranng was termnated after 6000 epochs. Intal learnng rate was set at ANFIS tranng In the ANFIS network, we used 4 Gaussan membershp functons for each nput parameter varable. Sxteen rules were learned based on the tranng data. The tranng was termnated after 60 epochs. MARS We used 5 bass functons and selected as the settng of mnmum observaton between knots. To obtan the best possble predcton results (lowest RMSE), we sacrfced the speed (mnmum completon tme).

9 MARS: Stll an Alen Planet n Soft Computng? 243 Fg. 7. Gas furnace seres predcton usng soft computng models and MARS Performance and results acheved Fgure 7 llustrates the test results acheved for the gas furnace tme seres. Table 2 summarzes the comparatve performances of the dfferent soft computng models and MARS n terms of performance tme, tranng error and testng error obtaned. Table 2. Results showng performance comparson between MARS and soft computng models for gas furnace seres predcton Root Mean Squared Error Tranng tme Model BFlops Epochs Tranng Set Test Set (seconds) MARS ANN NF *Computatonal load n bllon flops (BFlops) 7. Concluson In ths paper we have nvestgated the performance of MARS and compared the performance wth artfcal neural networks and neuro-fuzzy systems (ANFIS), whch are well-establshed functon approxmators. ur experments to predct the

10 244 A. Abraham and D. Stenberg benchmark tme seres reveal the effcency of MARS. In terms of both RMSE (test set) and performance tme, MARS outperformed the soft computng models consdered. MARS can no longer be consdered an alen planet consderng the performance depcted n Table 2. It wll be nterestng to study the robustness of MARS compared to neural networks and neuro-fuzzy systems. Choosng sutable parameters for a MARS network s more or less a tral and error approach where optmal results wll depend on the selecton of parameters. Selecton of optmal parameters may be formulated as an evolutonary search [8] to make MARS fully adaptable and optmal accordng to the problem. References [] Zadeh LA, Roles of Soft Computng and Fuzzy Logc n the Concepton, Desgn and Deployment of Informaton/Intellgent Systems, Computatonal Intellgence: Soft Computng and Fuzzy-Neuro Integraton wth Applcatons, Kaynak, LA Zadeh, B Turksen, IJ Rudas (Eds.), pp-9, 998. [2] Fredman, J. H, Multvarate Adaptve Regresson Splnes, Annals of Statstcs, Vol 9, - 4, 99. [3] Stenberg, D, Colla, P. L., and Kerry Martn (999), MARS User Gude, San Dego, CA: Salford Systems, 999. [4] Shkn E V and Pls A I, Handbook on Splnes for the User, CRC Press, 995. [5] Jang J S R, Neuro-Fuzzy Modelng: Archtectures, Analyses and Applcatons, PhD Thess, Unversty of Calforna, Berkeley, July 992. [6] Abraham A and Nath B, Desgnng ptmal Neuro-Fuzzy Systems for Intellgent Control, The Sxth Internatonal Conference on Control, Automaton, Robotcs and Vson, (ICARCV 2000), December [7] Abraham A and Nath B, ptmal Desgn of Neural Nets Usng Hybrd Algorthms, In proceedngs of 6 th Pacfc Rm Internatonal Conference on Artfcal Intellgence (PRICAI 2000), pp , [8] Fogel D, Evolutonary Computaton: Towards a New Phlosophy of Machne Intellgence, 2 nd Edton, IEEE press, 999. [9] Cherkassky V, Fuzzy Inference Systems: A Crtcal Revew, Computatonal Intellgence: Soft Computng and Fuzzy-Neuro Integraton wth Applcatons, Kayak, Zadeh LA et al (Eds.), Sprnger, pp.77-97, 998. [0] Box G E P and Jenkns G M, Tme Seres Analyss, Forecastng and Control, San Francsco: Holden Day, 970.