Artificial Intelligence (AI) methods are concerned with. Artificial Intelligence Techniques for Steam Generator Modelling

Size: px
Start display at page:

Download "Artificial Intelligence (AI) methods are concerned with. Artificial Intelligence Techniques for Steam Generator Modelling"

Transcription

1 Artfcal Intellgence Technques for Steam Generator Modellng Sarah Wrght and Tshldz Marwala Abstract Ths paper nvestgates the use of dfferent Artfcal Intellgence methods to predct the values of several contnuous varables from a Steam Generator. The objectve was to determne how the dfferent artfcal ntellgence methods performed n makng predctons on the gven dataset. The artfcal ntellgence methods evaluated were Neural Networks, Support Vector Machnes, and Adaptve Neuro-Fuzzy Inference Systems. The types of neural networks nvestgated were Mult-Layer Perceptons, and Radal Bass Functon. Bayesan and commttee technques were appled to these neural networks. Each of the AI methods consdered was smulated n Matlab. The results of the smulatons showed that all the AI methods were capable of predctng the Steam Generator data reasonably accurately. However, the Adaptve Neuro-Fuzzy Inference system out performed the other methods n terms of accuracy and ease of mplementaton, whle stll achevng a fast executon tme as well as a reasonable tranng tme. Index Terms Artfcal Intellgence, Fuzzy Logc, Neuro- Fuzzy, Neural Networks, Support Vector Machnes I. INTRODUCTION Artfcal Intellgence (AI) methods are concerned wth machnes or computer systems that have the ablty to learn and solve problems, and as a result exhbt ntellgent behavour. Normally, ntellgent behavour s assocated wth characterstcs such as havng the ablty to adapt, learn new sklls, and form complex relatonshps []. There are several artfcal ntellgence methods that have been developed such as Neural Networks, Support Vector Machnes, and Neuro- Fuzzy Systems. These AI systems have been utlsed n dfferent applcatons for example: pattern recognton, predcton of process varables, and varous control applcatons. Each of these methods has dfferent approaches to adaptng and learnng n order to emulate ntellgent behavour. Such Artfcal Intellgence methods are partcularly useful n modellng complex relatonshps where the relatonshp cannot be computed drectly or easly nterpreted by a human. A well known Artfcal Intellgence method s Neural School of Electrcal and Informaton Engneerng P/Bag x3, Wts, 050 South Afrca Webste: Networks. Neural networks are nspred by the mechansms of the human bran and are capable of learnng complex relatonshps through the assocaton of examples of these relatonshps []. A neural network contnuously adapts or adjusts these complex relatonshps found n an example dataset untl t has learnt the relatonshps suffcently well. Neural networks model these complex relatonshps n terms of a set of free parameters (weghts and bases) that can be mathematcally represented by a functon [3]. There are numerous types of neural networks that can be mplemented such as Radal Bass Functon and Mult-layer Perceptons. Support Vector Machnes s a more recent Artfcal Intellgence method developed by Vapnk and hs colleges n 99. Support Vector Machnes are based on statstcal learnng where the goal s to determne an unknown dependency between a set of nputs and outputs, and ths dependency s estmated from a lmted set of example data [4]. In the case of classfcaton, the dea s to construct a hyper-plane as a decson surface n such a way that the margn of separaton between the dfferent classes s maxmzed [5]. In the class of functon approxmaton, the goal s to fnd a functon that has at most a certan devaton from the desred target for all the ponts n a dataset of examples used to model such dependences. Lke neural networks, t models complex relatonshps usng a mathematcal approach. Neuro-Fuzzy Systems are based on Fuzzy logc whch was formulated n the 960s by Zadeh [6]. These systems combne Fuzzy Logc and certan prncples of Neural Networks n order to model complex relatonshps. Fuzzy systems use a more lngustc approach rather than a mathematcal approach, where relatonshps are descrbed n natural language usng lngustc varables [6]. All of the AI methods mentoned requre a dataset n order to tran the AI systems to be able to model the complex relatonshps of the system beng modelled. Therefore, the AI system learns by example through a tranng process, and ths dataset s called a tranng dataset. Most AI methods can be used for functon approxmaton n whch predctons of contnuous varables can be generated. In ths paper, the nvestgaton nto certan AI methods for the applcaton of predctng certan varables from a Steam Generator wll be dscussed. The AI methods that were nvestgated were: Neural Networks (Radal Bass

2 Functon, Mult-Layer Percepton, Commttees, and Bayesan Technques), Support Vector Machnes, and Adaptve Neuro- Fuzzy Inference Systems. Each of theses AI methods were nvestgated and smulated n Matlab, n order to ascertan the performance of each method as well as ts strengths and weakness when appled to the stated applcaton. The man performance measures under consderaton are the accuracy obtaned, speed of tranng, and the speed of executon of the AI system on unseen data. The paper wll frst gve a basc foundaton of the theory of the AI methods used, and then the mplementatons and ther results wll be presented. Fnally, the key fndngs of the smulatons wll be dscussed. II. THE STEAM GENERATOR DATASET The problem requres modellng the nput-output relatonshp of data obtaned from a Steam Generator at Abbott Power Plant n the USA. The dataset used s avalable onlne, and contans 9600 samples. The model conssts of 4 nputs and 4 outputs. The nputs are the nput fuel, ar, reference level (nches), and dsturbance defned by the load level. The fuel and ar nputs have been scaled between 0 and. The outputs are the drum pressure (PSI), excess oxygen n exhaust gases (percentage), the level of water n the drum, and the steam flow (Kg/s). Both the nputs and outputs are n numerc form and have dfferent unts to express ther quanttes. The dea s to be able to predct the outputs for a specfc set of nputs for the steam generator. Therefore, the problem s a regresson problem as the goal s to predct the value of a number of contnuous varables. Ths data set wll be modelled usng the dfferent artfcal ntellgence methods mentoned above. III. ARTIFICIAL NEURAL NETWORKS A. General Theory of Artfcal Neural Networks Neural Networks were orgnally nspred by the mechansms used by the human bran to learn by experence and processes nformaton. The human bran conssts of many nterconnected neurons that form an nformaton processng network capable of learnng and adaptng from experence [, 7]. Inputs X X X N w k w k w kn Σ Bas b k k = Number of Neurons Actvaton Functon f(.) Fgure : Structure of a Generalsed Artfcal Neuron [5] Output y k Bascally, a neural network s a data modellng technque used to form an nput/output relatonshp for a specfc problem []. Therefore, an nput/output relatonshp must exst for the neural network to functon n predctng outputs or classfyng data. The basc component of a neural network s an artfcal neuron whch s a largely smplfed representaton of a bologcal neuron. Each neuron receves a number of nputs whch may be from the outputs of other neurons or the source data beng fed nto the network [, 7]. Each nput s multpled by a weght to determne ts nfluence or strength. These weghts are analogous to the adjustment of the synaptc connectons between neurons that occurs durng the learnng process n bologcal systems [, 7]. The weghted nputs and an external bas value are summed. The summed sgnal s passed through an actvaton functon to produce the neuron s output sgnal. The bas value has the effect of ncreasng or decreasng the sgnal nput passed to the actvaton functon and s smlar to the frng threshold of a bologcal neuron [, 7]. The actvaton functon lmts the ampltude range of the neuron s output sgnal [5]. An artfcal neuron represents the basc nformaton processng unt of any neural network. However, the general characterstcs of the artfcal neuron: actvaton functon used, bases, method of calculatng the weghts, and number of nputs; wll dffer dependng on the type of neural network and the problem beng modelled. Fgure shows the model of an artfcal neuron [5]. The general mathematcal model of a neuron can be descrbed by Equaton [5]. N yk = f w j= where: kj x j + b k y k = output of the kth neuron x j = the jth nput w kj = weghtng connectng the jth nput to neuron k. A neural network conssts of several layers of nterconnected artfcal neurons workng together to model a problem. The types of neural networks that wll be dscussed are feedforward neural networks, meanng that the sgnals can only travel n one drecton through the network structure: from the nputs towards the outputs. The nput layer conssts of several nputs (source data) to be modelled by the network, t does not have any neurons and no computaton s performed [5]. The network may have several hdden layers that ntroduce more adjustable weghtngs to the network, allowng a hgher order model of the data to be extracted [5, 3]. The fnal layer s the output layer of the network whch produces the overall network s outputs. Each layer of the network receves nputs from the prevous layer of the network, and passes ts outputs to the next layer. Normally, every node n a layer s connected to every other node n the followng layer (meshed) [5]. The basc structure of a feedforward network can be seen n Fgure [5]. ()

3 A neural network learns by example through tranng algorthms. Tranng results n an nput/output relatonshp beng determned for a specfc problem. Tranng can be supervsed or unsupervsed. The neural networks dscussed wll use supervsed tranng. Supervsed tranng nvolves havng a tranng dataset where numerous examples of nputs and ther correspondng outputs (targets) are fed to the network. The weghts and bases of the neural network are contnuously adjusted to mnmse the error between the network s outputs and the target outputs [, 5, 7]. Input Layer Neurons Hdden Layer Fgure : Basc Structure of a Feed-forward Artfcal Neural Network [5] Optmsaton technques can be used to determne the weghts of the network, snce t s a mnmsaton problem. Therefore, the knowledge or nformaton about the problem s contaned by the weghts and bases of the network. An mportant property of neural networks s ther ablty to generalse. Generalsaton refers to the ablty of the neural network to predct or produce reasonable outputs for nputs not seen durng the tranng or learnng process [5]. Thus, the nput/output relatonshp computed s vald for unseen data. Generalsaton s nfluenced by the sze of the tranng dataset and the archtecture of the neural network [5].The best generalsaton s normally acheved when the number of free parameters s farly small compared to the sze of the dataset [3]. However, a neural network can have poor generalsaton f t s under-traned (under-fttng) or over-traned (overfttng). Over-tranng occurs when the neural network fts the tranng data perfectly and results wth a functon approxmaton or boundary lne that s not smooth but erratc n nature [3]. The network effectvely memorses the data and therefore, has poor generalsaton on data not n the tranng set. Also, a neural network can be under-traned: there are not enough free parameters to suffcently form an nput/output relatonshp that captures the features of the problem [3]. B. Mult-Layer Percepton Neurons Output Layer Mult Layer Percepton (MLP) neural networks are a popular class of feed-forward networks (Fgure ). They were developed from the mathematcal model of the neuron (Fgure ), and consst of a network of neurons or perceptons []. An MLP network conssts of an nput layer (source data), several hdden layers of neurons, and an output layer of neurons. The hdden layers and the output layer can have dfferent actvaton functons. There are varous types of actvaton functons that can be employed. The actvaton functon of the hdden neurons must be nonlnear and are usually functons that are dfferentable [3]. Typcally, the hyperbolc tangent or logstc functons are used for the actvaton functon of the hdden neurons. However, the output actvaton functon may be lnear. Certan actvaton functons are more approprate for dfferent types of problems, therefore, the actvaton functon needs to be selected accordng to the problem. Normally, a lnear output actvaton functon s used for regresson problems as t does not lmt the range of the output sgnal []. A mult-layer percepton neural network represents a multvarate non-lnear functon mappng between a set of nput and output varables [3]. It has been shown that any contnuous functon can be modelled accurately wth one hdden layer, provded there s a suffcent number of hdden neurons [3, 5]. An MLP network wth one hdden layer can be mathematcally represented by Equaton [3]. Nhdden Nnput () () () () y k = f wkj f A wj x + wj0 + wk 0 () j= = where: k = number of outputs y k = the output at the k th node j = number of hdden neurons = number of nputs f A = actvaton functon of the hdden neurons f = actvaton functon of the output neurons x = the nput from the th nput node w j = weghts connectng the nput wth the hdden nodes w jk = weghts connectng the hdden wth the output nodes w 0j and w 0k = bases The complexty of the model s related to the number of hdden unts, as the number of free parameters (weghts and bases) avalable to adjust s drectly proportonal to the number of hdden unts. Tranng nvolves contnuously adjustng the values of the weghts and bases to mnmse the error between the network s output and the desred targets. Intally, the weghts and bases are set to random values, and then adjusted usng an optmsaton technque. However, such optmsaton technques are hghly susceptble to fndng local mnma, and there s no guarantee that a global mnmum has been found [3]. The best way to try and avod a soluton that s a local mnmum s to tran many networks takng the best network produced.

4 C. Radal Bass Functons Radal Bass Functons are two-layer feed-forward neural networks wth the actvaton functon of the hdden unts beng radal bass functons [5]. The response of the hdden layer unt s dependent on the dstance an nput s from the centre represented by the radal bass functon (Eucldean Dstance) []. Each radal functon has two parameters: a centre and a wdth. Therefore, the maxmum actvaton of a hdden unt s acheved when the nput concdes wth the centre vector. The wdth of the bass functon determnes the spread of the functon and how quckly the actvaton of the hdden node decreases wth the nput beng an ncreased dstance from the centre [3]. The most common radal bass functon used s the Gaussan bell-shaped dstrbuton. Normally, an RBF only has one hdden layer, and a lnear output layer. The nput layer smply passes the nput data to the hdden layer. An RBF network can be modelled mathematcally by Equaton 3 and the Gaussan actvaton functon s represented by Equaton 4. The bas parameters at the output layer compensate for the dfference between mean output values and mean target values [3]. k M ( ) = wkj j ( x) wk 0 y x φ + (3) where: φ ( x ) j (4) where: j= y k = the output at the k th node M = number of hdden nodes w kj = the weght factor from the j th hdden node to the k th output node w k0 = the bas parameter of the kth output node φ (x) = radal bass actvaton functon exp x u = σ j x = u j = σ = j nput vector centre vector of the jth hdden node wdth of bass functon An RBF s traned n two stages. The frst stage s an unsupervsed learnng process to determne the parameters of the radal bass functon for each hdden node [3]. Therefore, only the nput data s used durng ths process. These parameters are the centres and the wdths of the bass functons. There are a number of unsupervsed tranng algorthms to determne the parameters of the bass functons such as K-means clusterng. The second stage nvolves fndng the fnal layer weghts that mnmse the error between the network s output and the target values. Therefore, the second stage s done usng supervsed learnng. Snce the output layer s a lnear functon, the fnal layer weghts can be solved usng lnear algebra [3]. Both of these stages are relatvely fast, therefore, an RBF trans much faster than an equvalent MLP. The parameters of an RBF can be determned by supervsed tranng. However, the optmsaton process s no longer lnear, resultng n the process beng computatonally expensve compared to the two stage tranng process. The man dfference between MLPs and RBFs are that an MLP splts the nput space nto hyper-planes whle an RBF splts the nput space nto hyper-spheres []. D. Commttees Combnng the outputs of several neural networks nto a sngle soluton to gan mproved accuracy over an ndvdual network output s called a commttee or ensemble [8]. The smplest way of combng the outputs of dfferent networks together s to average the outputs obtaned [3]. The averagng ensemble can be expressed by Equaton 5 [3, 8], y K N = yk (5) N = where y k s the kth output, y k s the kth output of network, and N s the number of networks n the commttee. It can be shown that averagng the predcton of N networks reduces the sum-of-squares error by a factor of N [3]. However, ths does not take nto account that some networks n the commttee may generate better predctons than others[3]. In ths case, a weghted sum can be formulated n whch certan networks contrbute more to the fnal output of the commttee [3]. There are several other commttee methods to mprove the accuracy of the predcton obtaned, such as Baggng and Boostng. E. Bayesan Technques for Neural Networks The tranng of the neural networks usng the more standard approaches reles on the mnmsaton of a functon error (Maxmum Lkelhood Approach) [3]. Ths approach makes defnng the neural network model dffcult, and both tranng and valdaton datasets are necessary to determne the model that exhbts the best generalsaton. There wll always be a certan error between the predcted and the actual. If several networks wth dentcal archtectures are produced wth the same error, the weghts and bases wll not be the same each tme, as there s a level of uncertanty n the tranng process due to there beng many possbltes for parameters. In the Bayesan approach, a probablty dstrbuton functon s consdered to be represented over the weght space, to account for the uncertanty n determnng the weght vector [3]. Instead of attemptng to fnd a sngle set of weghts that mnmsed the error between the predcted and actual values. The probablty dstrbuton represents the degree of confdence assocated to the dfferent values for the weght vector [3]. Ths probablty dstrbuton s ntalsed to some pror dstrbuton, and then wth the ad of the tranng

5 dataset the posteror probablty dstrbuton can be determned and used to evaluate the predcted outputs for new nput data ponts [3]. The posteror probablty dstrbuton can be expressed usng Bayes Theorem and s shown n Equaton 6. P( D w) P( w) P ( w D) = P( D) (6) where D represents the target values of the tranng dataset, w s the vector representng the adaptve weghts and bases, P(w) s the probablty dstrbuton functon of the weght space n absence of any data ponts (Pror Probablty Dstrbuton), P(D) s a normalsaton factor, P(D w) s a lkelhood functon, and P(w D) s the posteror probablty dstrbuton. Usng Bayes Theorem allows any pror knowledge about the uncertan weght values to be updated based on the knowledge ganed from the tranng dataset to produce the posteror dstrbuton of the unknown weght values [3]. The posteror probablty dstrbuton gves an ndcaton of whch weght values for the weght vector are most probable [3]. The pror probablty dstrbuton should take nto account any nformaton known about the weghts [3]. From regularsaton technques, t s known that small weght values are favoured n order to produce smooth network mappngs. Therefore, the weght-decay regularsaton needs to be ncorporated n the pror probablty dstrbuton functon. For pror probablty dstrbuton that s a Gaussan functon, the form s shown n Equaton 6 [3], where W s the number of weghts and Z w s the normalsaton coeffcent. If the weght decay term s small then the p(w) s large. The quantty α s the coeffcent of weght-decay. P( w) = Z (6) where ZW W ( α ) π = α α exp( w w The Lkelhood probablty dstrbuton s gven by Equaton 7, and s an expresson of the dfference between the predcted output ( y(x,w) ) and the target output (t). The quantty β s the coeffcent of the data error [3]. P( D w) = Z (7) D ( β ) ) β N exp n= n n { y( x, w) t } where Z D π = β N The posteror probablty dstrbuton can be obtaned by applyng Bayes theorem and s gven below [3]. It can be seen that S(w) s dependent on the sum-of-squares error functon and a weght regularsaton term [3]. P( w D) = Z where S( W ) = βe Z S S exp ( S( w) ) ( 8) D N β = n= + αe n n { y( x, w) t } W α + w = ( α, β ) = exp( βe αe )dw The tranng process for the Bayesan approach nvolves determnng the approprate posteror probablty dstrbuton of the weght values [9]. In order, to make a predcton for a new nput vector, the output dstrbuton must be computed, and s gven by Equaton 9. Ths Equaton s effectvely takng an average predcton of all the models weghted by ther degree of probablty [3], and s dependent on the posteror probablty dstrbuton. Therefore, the traned network can make predctons on nput data t has not seen by usng the posteror probablty dstrbuton. P( y x W D n+ n, D) = P( y x, w) P( w D) dw (9) n+ n+ + The evaluaton of the probablty dstrbutons requres ntegraton over a multdmensonal weght space, and s not easly handled analytcally. One method to evaluate the ntegrals s to use a Gaussan Approxmaton whch allows the ntegral to be analytcally evaluated usng optmsaton technques [3]. Another common method used to solve these type of ntegrals s a random samplng method called Monte Carlo Technque [0]. Therefore, the Monte Carlo or the Hybrd Monte Carlo method s normally used to dentfy the posteror probablty dstrbuton of the weghts for a Bayesan neural network, by samplng from the posteror weght dstrbuton. F. Monte Carlo Methods In the Bayesan approach to neural networks, ntegraton plays a sgnfcant role as calculatons nvolve evaluatng an ntegral over the weght space. Monte Carlo s a method of approxmatng the ntegral by usng a sample of ponts from the functon of nterest [3]. The ntegrals that need to be evaluated are of the form [3], W

6 I = F( w) P( w D) dw (0) where F(w) s the ntegrand and P(w D) s the posteror dstrbuton of weghts. Ths ntegral can then be approxmated usng a fnte sum of the form, L I F( w ) L = () where w s the sample of weght vectors generated from the posteror probablty dstrbuton [3]. In order to generate samples of the weght vector space representatve of the P(w D), a random search through the weght space for areas were the dstrbuton s reasonably large s performed. Ths done usng a technque called Markov Chan Monte Carlo, where a sequence of weght vectors are generated, each new vector n the sequence dependng on the prevous weght vector plus a random component [3]. A random walk s the smplest method n whch each successve step s computed usng Equaton [3]. wn+ = wn () + ε ε s a random vector that allows more of the weght space to be explored. In order, to fnd samples of weght vectors that are representatve of the P(w D) dstrbuton, a procedure known as the Metropols Algorthm s used to select the sample weght vectors. The Metropols Algorthm rejects or accepts a certan sample of the weght space or state generated usng Equaton based on the followng condtons, f P( w f P( w n+ n+ D) > P( w D) < P( w P( wn wth probablty P( w n n D) accept state w D) accept state w + n D) D) n+ n+ areas of hgher posteror probabltes are favoured [3]. Ths gradent nformaton can be obtaned through the backpropagaton algorthm. The Hybrd Monte Carlo method s based on the prncples of Hamltonan mechancs that descrbe molecular dynamcs [0]. It s a form of the Markov Chan, however, the transton between states s acheved usng the stochastc dynamc model [9]. In statstcal mechancs, the state space of a system at a certan tme can be descrbed by the poston and momentum of all the molecules of the system at that tme [9]. The poston defnes the potental energy of the system and the momentum defnes the knetc energy of the system [9-0]. The total energy of the system s the sum of the potental and knetc energy, and can be represented by the Hamltonan equaton defned as, H = (3) ( w, p) E( w) + K( p) = U ( w) + p where w s the poston varable, p s the momentum varable, H(w,p) s the total energy of the system, E(w) s the potental energy, and K(p) s the knetc energy. The postons are analogous wth the weghts of a neural network, and potental energy wth the network error [0]. In ths equaton, the energes of the system are defned by energy functons representng the state of the physcal system (canoncal dstrbutons) [0]. In order to obtan the posteror dstrbuton of the network weghts, the followng dstrbuton s sampled gnorng the dstrbuton of the momentum vector [9]. P( w, p) = exp( H ( w, p)) (4) Z Hamltonan dynamcs are used to sample at a fxed energy n terms of a fcttous tme τ [9-0], and are shown n Equaton 5 and 6. Snce the dynamcs shown n Equatons 5 and 6 can not be smulated exactly, the equatons are dscretsed usng fnte tme steps gven by Equaton 7 and 9. [0]. In ths way the poston and momentum at tme τ + ε s expressed n terms of the poston and momentum at tme τ [0]. Ths method s known as the leap-frog method. These new states are accepted or rejected usng the Metropols crteron. Usng the above condtons, certan of the weght vector samples wll be rejected f they lead to a reducton n the posteror dstrbuton [3]. Ths procedure s repeated a number of tmes untl the necessary number of samples are produced for the evaluaton of the fnte sum for the ntegral. Due to hgh correlaton n the posteror dstrbuton as a result of the each successve step beng dependent on the prevous, a large number of the new weght vector states wll be rejected [3]. Therefore, a Hybrd Monte Carlo method can be used nstead. The Hybrd Monte Carlo methods uses nformaton about the gradent of P(w D) to ensure that samples through the dw H = dτ p dp H = dτ w = p E = w (5) (6)

7 Dscretsed Equatons ε pˆ τ + = pˆ (7) ε E w ( τ ) [ wˆ ( τ )] ε w ˆ ( + ) = ˆ ( ) + ˆ τ ε w τ εp τ + (8) ε ε E pˆ ( τ + ε ) = pˆ τ + [ wˆ ( τ + ε )] (9) w The basc steps n the mplementaton of the Hybrd Monte Carlo algorthm are [9, ]: () Randomly choose a trajectory drecton (λ) where λ s - for a backward trajectory and + for a forward trajectory. () Startng from a current state (w, p). Perform L leapfrog steps wth the step sze ε usng Equatons 6-9 to product a canddate state (w*, p*). Performng L leapfrog steps allows more of the state space to be explored faster. () Usng the Metropols crteron, accept or reject the (w*, p*) state. If the canddate state s rejected the old state (w, p) s kept as the new state. Otherwse, the canddate state s accepted and t becomes the new state. IV. SUPPORT VECTOR MACHINES Support Vector Machnes (SVM) were ntroduced by Vapnk and hs colleges n 99. They are based on statstcal learnng theory and are one type of kernel learnng algorthm n the feld of machne learnng [4]. SVMs can be used for both classfcaton and regresson problems. The goal of statstcal learnng s to determne an unknown dependency between a set of nputs and outputs, and ths dependency s estmated from a lmted set of example data. [4]. Therefore, the objectve of a SVM, lke neural networks, s to produce a model whch can predct the output values of a dataset prevously unseen. Thus, SVMs utlse supervsed learnng technques, and requre a tranng and testng dataset. In the case of classfcaton, the dea s to construct a hyper-plane as a decson surface n such a way that the margn of separaton between the dfferent classes s maxmzed [5]. These decson planes are defned to act as decson boundares separatng dfferent classes of objects. Fgure 3: Showng a Lnear SVM Regresson for a Dataset Illustratng the ε -Tube and Penalty Cost Functon [] In support vector regresson, the dea s to fnd a functon that has at most a devaton of ε from the desred targets for all the tranng data (ε -SV regresson) []. Thus, errors below the devaton are not of concern, and ponts outsde ths devaton are penalzed (Refer to Fgure 3). Therefore, a functon that approxmates all the nput-output pars wth the defned precson must actually exst and the optmsaton requred must be able to be feasbly solved []. In order, to account for data ponts that cannot be easly modelled, slack varables are normally ntroduced. In both classfcaton and regresson, the nputs are mapped nto a hgher dmensonal feature space by a functon φ (x) nduced by a kernel functon [4, ]. The SVM then fnds a lnear separatng hyper-plane wth the maxmal margn n ths hgher dmensonal space for the classfcaton case, and a set of lnear functons n ths hgher dmensonal space for the regresson case [4, 3]. There are dfferent types of kernel functons: lnear, polynomal, radal bass functon (RBF), and sgmod. Any functon that satsfes Mercer s Theorem can be used as a kernel functon [4]. The kernel functon s equal to the nner product of the two vectors (nput vector (x) and nput pattern of the th tranng sample (x )) nduced n the feature space and s gven by Equaton 0 [5]. T K x, x ) = φ ( x) φ( x ) (0) ( In the case of regresson, f gven a tranng dataset, {(x, t )} N, where the x s the nput vector and t s the target value, = an SVM approxmates the functon usng Equaton [3]. y = f ( x) = wφ ( x) + b () where φ (x) represents the hgher dmensonal feature space that the nputs are mapped to, w s the weght vector, and b s the bas. Snce n realty, not every pont wll be able to ft wthn the devaton defned, the Support Vector Machne mnmses the number of ponts outsde the devaton usng a penalty parameter []. Ths s acheved by mnmsng Equaton. If Equaton s transformed nto dual formulaton, t s expressed n terms of the kernel functon

8 and support vectors. Support vectors consst of the data ponts that st on the boundares of the acceptable regon defned byε and are extracted from the tranng dataset [5, 3]. Ths constraned optmsaton problem can be solved usng quadratc programmng wth the tranng data and, as a result, s guaranteed to fnd a global optmum [5, ]. Mnmse: w () Subject to : * ξ, ξ 0 N = * ( ) + C ξ +ξ t wφ( x ) b ε + ξ wφ( x ) + b t ε + ξ * where w - weght vector C - penalty parameter N - number of data ponts ε - devaton from functon ξ, ξ - Slack Varables * The constrants above deal wth a lnearε -nsenstve loss functon used to penalse the data ponts n the tranng dataset that are outsde the specfed devatonε. A loss functon s used to determne whch functon (f(x)) best descrbes the dependency observed n the tranng dataset [4]. The purpose of the loss functon s to determne the cost of the dfference between the actual and predcted outputs for a gven set of nputs. The ε -nsenstve loss functon s defned by Equaton 3 [4, 5]. As seen n Fgure 3, n regresson problems a ε -tube s formed around the functon, and any data ponts outsde ths ε -tube have an assocated cost gven by Equaton 3. Most data ponts should lay wthn the ε - tube, however, the slack varable allow some data ponts to le outsde the ε -tube [5]. There are two slack varables to account for the upper and lower bounds of the ε -tube. Both ε and C are user-defned parameters. The parameter C s a regularsaton parameter that controls the trade-off between the complexty of the machne and the number of data ponts that le outsde the ε -tube [5]. The devatonε, determnes the approxmaton accuracy enforced on the tranng data ponts [3]. For regresson, the parameters C and ε should be tuned smultaneously [5]. l ( f ( x), t) (3) f ( x) t ε f f ( x) t > ε = 0 otherwse There are other ε -nsenstve loss functons that can be used such as a quadratc ε -nsenstve loss functon. Also, a least squares cost functon can be used. Ths results n a Least Squares Support Vector Machne (LS-SVM) that has a few dfferent propertes to the orgnal Vapnk s SVM presented above. In a least squares SVM, the ε -nsenstve loss functon s replaced by a least squares cost functon whch corresponds to a form of rdge regresson [4]. The nequalty constrants that Equaton s subject to are replaced by equalty. As a consequence, the tranng process of a LS-SVM nvolves solvng a set of lnear equatons nstead of a quadratc programmng problem. The set of lnear equatons that result are of the dmenson N+, where N s the number of tranng samples [5]. In the case of a standard SVM, the quadratc programmng (QP) problem to be solved s roughly exponental to the sze of the tranng dataset [4]. Therefore, the number of tranng samples used to tran an SVM should be consdered carefully. However, the set of lnear equatons s stll not as tme and computatonally consumng to solve as the QP problem. In a LS-SVM, the weght vector that results from mnmzng the summed squared approxmaton error over all tranng samples s searched for, where the approxmaton error s the dfference between the SVM s output and the desred target output [5]. The equaton for a LS-SVM s shown below n Equaton 4 [5]. Mnmse : w + C Subject to : t N e = 0 = wφ( x ) + b + e where e = t - f(x ) (4) The man dfference between Neural networks and Support Vector Machnes s that support vector machnes mnmse an upper bound of the generalsaton errors nstead of mnmsng the error on the tranng dataset [3]. Support vector machnes utlse rsk mnmsaton, measured usng a loss functon. Normally, support vector machnes have a slower executon tme as there s lttle control over the number of support vectors defned [5]. An SVM has less parameters to tune than a neural network, and the optmsaton procedure can be performed effcently. In the case of the LS-SVM, the parameters that need to be tuned are the penalty or regularsaton constant and the devaton of the Gaussan functon, f an RBF kernel s used. Whle for a standard SVM the ε accuracy for the ε -nsenstve loss functon needs to determned addtonally. V. FUZZY LOGIC AND NEURO-FUZZY SYSTEMS Neuro-Fuzzy Systems are based on Fuzzy logc whch was formulated n the 960s by Zadeh. These systems combne Fuzzy Logc and certan prncples of Neural Networks n

9 order to model complex relatonshps. Fuzzy systems use a more lngustc approach rather than a mathematcal approach, where relatonshps are descrbed n natural language usng lngustc varables. Fuzzy Logc can deal wth ll-defned, mprecse systems [6], and therefore are a good tool for system modellng. Ths secton ntroduces the bascs of Fuzzy Logc and then explans Adaptve Neuro- Fuzzy Inference Systems that are based on the foundatons of Fuzzy Logc. A. Basc Fuzzy Logc Theory Fuzzy logc s a method of mappng an nput space to an output space by means of a lst of lngustc rules that consst of f-then statements [6]. Fuzzy logc conssts of 4 components: fuzzy sets, membershp functons, fuzzy logcal operators, and fuzzy rules [6, 7, 8]. In classcal set theory, an object s ether a member or s not a member of specfc set [7-8]. Therefore, t s possble to determne f an object belongs to a specfc set as a set has clear dstnct boundares, provded an object cannot acheve partal membershp. Another way of thnkng about ths s that the object s belongng to a set s ether true or false. A characterstc functon for a classcal set has a value of f the object belongs to the set and a value of zero f the object doesn t belong to the set [7]. For example, f a set X s defned to represent all possble heghts of people, one could defne a tall subset for any person who s above or equal to a specfc heght x, and anyone below x doesn t belong to the tall set but to a short subset. Ths s clearly nflexble as a person just below the boundary s labelled as beng short when they are clearly tall to some degree. Therefore, ntermedate values such as farly tall are not allowed. Also, these clear cut defned boundares can be very subjectve n terms of what a person may defne as belongng to a specfc set. The man am behnd fuzzy logc s to allow a more flexble representaton of sets of objects by usng a fuzzy set. A fuzzy set does not have as clear cut boundares as a classcal set, and the objects are characterzed by a degree of membershp to a specfc set [7-8]. Therefore, ntermedate values of objects can be represented whch s closer to the way the human bran thnks opposed to the clear cut-off boundares n classcal sets. A membershp functon defnes the degree that an object belongs to a certan set or class. The membershp functon s a curve that maps the nput space varable to a number between 0 and, representng the degree that a specfc nput varable belongs to a specfc set [7-8]. A membershp functon can be a curve of any shape. Usng the example above, there would be two subsets one for tall and one for short that would overlap. In ths way a person can have a partal partcpaton n each of these sets, therefore, determnng the degree to whch the person s both tall and short. Logcal operators are defned to generate new fuzzy sets from the exstng fuzzy sets. In classcal set theory there are 3 man operators used, allowng logcal expressons to be defned: ntersecton, unon, and the complement [7]. These operators are used n fuzzy logc, and have been adapted to deal wth partal membershps. The ntersecton (AND operator) of two fuzzy sets s gven by a mnmum operaton, and the unon (OR operator) of two fuzzy sets s gven by a maxmum operaton [7]. These logcal operators are used n the rules and determnaton of the fnal output fuzzy set. Fuzzy Rules formulate the condtonal statements whch are used to model the nput-output relatonshps of the system, and are expressed n natural language [6]. These lngustc rules are n the form of f-then statements whch use the logcal operators and membershp functons to produce an output. An mportant property of fuzzy logc s the use of lngustc varables. Lngustc varables are varables that take words or sentences as ther values nstead of numbers [7]. Each lngustc varable takes a lngustc value that corresponds to a fuzzy set [7], and the set of values that t can take s called the term set [8]. For example, a lngustc varable Heght could have the followng term set {very tall, tall, medum, short, very short}. A sngle fuzzy rule s of the form: f x s A then y s B (5) where A and B are fuzzy sets defned for the nput and output space respectvely. Both x and y are lngustc varables, whle A and B are the lngustc values or labels represented by the membershp functons [6]. Each rule conssts of two parts: the antecedent and the consequent [7]. The antecedent s the component of the rule fallng between the f-then, and maps the nput x to the fuzzy set A, usng a membershp functon. The consequent s the component of the rule after the then, and maps the output y to a membershp functon. The nput membershp values act lke weghtng factors to determne ther nfluence on the fuzzy output sets [7]. A fuzzy system conssts of a lst of these f-then rules whch are evaluated n parallel. The antecedent can have more than one lngustc varable, these nputs are combned usng the AND operator. Each of the rules s evaluated for an nput set, and correspondng output for the rule obtaned. If an nput corresponds to two lngustc varable values then the rules assocated wth both these values wll be evaluated. Also, the rest of the rules wll evaluated, however, wll not have an effect on the fnal result as the lngustc varable wll have a value of zero. Therefore, f the antecedent s true to some degree, the consequent wll have to be true to some degree [7]. The degree of each lngustc output value s then computed by performng a combned logcal sum for each membershp functon [7]. After whch all the combned sums for a specfc lngustc varable can be aggregated. These last stages nvolve the use of an nference method whch wll map the result onto an output membershp functon [9]. Fnally, a defuzzfcaton process s preformed n whch a sngle numerc output produced. One method of computng the degree of each lngustc output value s to take the maxmum of all rules descrbng ths lngustc output value [7, 9], and the output s taken as the centre of gravty

10 of the area under the effected part of the output membershp functon. There are other nference methods such as averagng and sum mean square [9]. Fgure 4 shows the steps nvolved n creatng an nput-output mappng usng fuzzy logc [0]. The use of a seres of fuzzy rules, and nference methods to produce a defuzzfed output consttute a Fuzzy Inference System (FIS) []. The fnal manner n whch the aggregaton process takes place and the method of defuzzfcaton can dffer dependng on the mplementaton of the FIS chosen. The approach dscussed above s that of the Mamdan based FIS. Input Varables Assgnment of Membershp Functons Applcaton Fuzzy Rules Aggregated Output Defuzzfcaton Numerc Output Fgure 4: Showng the Steps Involved n the Applcaton of Fuzzy Logc to a Problem [0] There are several types of fuzzy nference systems whch vary accordng to the fuzzy reasonng and the form of the f-then statements appled [6]. Another method of Fuzzy nference that s worth dscussng s the Takag-Sugeno-Kang method. It s smlar to the Mamdan approach descrbed above except that the consequent part s of a dfferent form and as a result the defuzzfcaton procedure s dfferent. The f-then statement of a Sugeno fuzzy system expresses the output of each rule as a functon of the nput varables, and has the form [], F u z z y L o g c Alternatvely, the output of a rule can be a constant. The fnal output of the Sugeno FIS s a weghted average of the outputs from each rule [6]. B. Adaptve Neuro-Fuzzy Inference Systems The man dffcultes wth Fuzzy Inference Systems are that t s dffcult to transform human knowledge nto the necessary rule base, as well as to adjust the Membershp functons to acheve a mnmzed output error for the FIS [6]. The purpose of an ANFIS (Adaptve Neuro-Fuzzy Inference System or Adaptve Network-Based Fuzzy Inference System) s to establsh a set of rules along wth a set of sutable membershp functons that s capable of representng the nput/output relatonshps of a gven system [6]. An adaptve network refers a mult-layer feed-forward type structure wth nterconnect nodes. However, some of the nodes are adaptve, meanng that such a node s output s dependent on several parameters belongng to t [6]. The lnks n an adaptve network only ndcates the flow of nformaton. An adaptve network utlzes a supervsed learnng algorthm n order to mnmze the error of the nput/output mappng requred, by adjustng the parameters of the adaptve nodes [6]. Therefore, a tranng dataset s necessary as the tranng process s smlar to that used by neural networks except the parameters of the adaptve nodes are beng adjusted nstead of the weghts of the lnks n the network. An ANFIS s a type of adaptve network wth the adaptve nodes representng membershp functons and the consequent equatons along wth ther correspondng parameters []. The goal of an ANFIS s to adjust the membershp functons and consequent equatons parameters to emulate the nput/output relatonshps of a gven dataset []. Therefore, an ANFIS s functonally equvalent to an FIS except that t has the ablty to learn and adapt through a tranng process usng nputoutput data pars to dscover the most approxmate parameters of the FIS to model the system accurately. The basc archtecture of a frst-order Sugeno (Takag-Sugeno- Kang) ANFIS wth nputs and rules s shown n Fgure 5 [, 6]. f x s A AND y s B then z = f(x,y) (6) If the output of each rule s a lnear combnaton of the nput varables plus a constant, then t s known as a frst-order Segeno fuzzy model, and has the form []: z = px + qy + c (7) Fgure 5: Showng the Archtecture of Frst-Order Sugeno ANFIS [, 6] In Fgure 5, Layer contans a seres of membershp functons whch determne the degree that the gven nput belongs to the specfc fuzzy set [6]. The membershp functon s parameters are changed, therefore, changng the

11 shape of the functon and the degree of membershp of the nput to a specfc fuzzy set. The nodes n ths layer are adaptve and the parameters are known as premse parameters []. In Layer, each node produces the product of the ncomng sgnals. Thus, determnng the fnal value or frng strength of each rule [6]. The nodes n Layer are fxed, normally performng a fuzzy AND operaton. Each node n Layer 3 calculates the normalzed frng strength by takng a rule s frng strengthens and dvdng t by the sum of all the rules frng strengths [6]. Layer 4 s an adaptve layer that has a node functon equal to the normalzed frng strength multpled by the frst-order Segeno fuzzy model functon. The fnal layer (layer 5) calculates the fnal output by summng all the ncomng sgnals [6]. Snce the normalzed frng strength and frstorder functon are the ncomng sgnals from the prevous layer, the output of layer 5 s effectvely a weghted average. All the respectve equatons can be found n []. An ANFIS s traned usng ether back-propagaton, or a hybrd tranng algorthm (a combnaton of least squares and back-propagaton). VI. IMPLEMENTATION AND RESULTS Each of the Artfcal Intellgence methods dscussed above were mplemented usng MATLAB. In ths secton, the mplementaton, results and observatons for each of these methods wll be dscussed. Also, the pre-processng performed on the Steam Generator dataset s dscussed A. Data Pre-processng Before a neural network, SVM, and Neuro-Fuzzy System should be mplemented, the dataset needs to be analysed and processed to nsure the best possble chance of acqurng the nput-output relatonshp of the dataset. Pre-processng the dataset that wll be fed nto the neural network or AI system s very mportant to the performance, generalzaton ablty, and the speed of tranng of the neural network [3]. On nspecton of the dataset, t was seen that there were no data ponts wth mssng values but there were a number of outlners. An outlner s an extreme pont that does not seem to belong to the dataset and may have an unjustfed nfluence on the model []. Snce two of the nputs were already scaled between 0 and, any samples that had a value greater than or less than zero for ether of these scaled nputs were consdered to be outlners. The outlners were smply removed from the dataset. There were 965 outlners n the dataset. Scalng of the data s mportant n neural networks and SVMs to equalze the mportance of each varable []. Snce dfferent varables can have values that dffer n orders of magntude, the varables wth the larger values wll appear more sgnfcant n determnng the outputs [3]. Thus, all nputs should be scaled to have the same range. Also, scalng s mportant as the actvaton functons n neural networks only have a lmted range before saturaton occurs. Both the nputs and the outputs were scaled between 0 and usng Mn-Max normalzaton to allow each varable to have equal mportance. Mn-Max normalzaton uses the maxmum and mnmum value of the varable to scale t to a range between 0 and, and s gven by Equaton 8 []. The outputs can be converted back to the orgnal scale wthout any loss of accuracy. x Scaled (8) x x = x x max mn mn Durng the collecton of the data, the samples can be stored n a specfc order. The dataset stored the samples n the sequental order n whch they were captured. Therefore, the samples were randomzed n order to break ths specfc order. Snce lttle else was known about the data, no other pre-process procedures were performed on the data ponts. The next step n the pre-processng was to partton the dataset nto 3 datasets: tranng, valdaton, and a testng dataset. Each dataset should contan a full representaton of the avalable values. The tranng dataset s used durng the supervsed tranng process to adjust the weghts and bases to mnmze the error between the network s outputs and the target outputs as well as for the tranng of the SVM and neuro-fuzzy system to adjust ther correspondng parameters. The valdaton data s used to perodcally check the generalzaton ablty of the network, SVM, or neuro-fuzzy system. The valdaton dataset s effectvely part of the tranng process as t s used to gude the selecton of the AI system. The test dataset s used as a fnal measure to see how the AI system performs on unseen data, and should only be used once. The resultng dataset has 8635 records of nputoutput sets whch were dvded nto the 3 datasets mentoned above. The same 3 datasets were used for the mplementaton of the neural networks, the SVMs, and the ANFIS. B. Performance Measures The man performance measure that was utlsed to evaluate the predcton ablty of the Artfcal Intellgence Methods was the Mean Squared Error (MSE). The Mean Squared Error s gven by Equaton 9. Ths equaton allows the contrbuton of each output to the total MSE to be calculated. R MSE = t( k) y( k) R k= R m = p R k = p= m R = p p= R k = (9) ( t ( k) y ( k) ) ( t ( k) y ( k) ) where R = sze dataset m = number of outputs p p

12 y = predcted value t = desred target value Other performance measures that were consdered are: the tme taken to tran the AI system, the tme taken to execute the AI system, and the complexty of the model produced by the AI method. C. Neural Networks Usng Standard Approaches The neural networks were mplemented usng the open source NETLAB Toolbox by Ian Nabney. Both the MLP and RBF neural networks were mplemented usng the standard approaches wth ths toolbox. The toolbox only constructs a layer feed-forward network for both the MLP and RBF. Therefore, there s only one hdden layer, and only the number of hdden nodes needed to be determned. The ntalsaton of the MLP and RBF networks nvolves determnng the actvaton functons used and the sze of the hdden layer. A lnear output actvaton functon s best for regresson problems, therefore, t was utlsed for both the MLP and RBF. In the case of an MLP, a lnear output actvaton functon does not saturate, and as a result can extrapolate a lttle beyond the tranng dataset []. However, the hdden nodes can saturate whch s one reason the nputs and outputs were scaled. In NETLAB, the hdden nodes of the MLP are the hyperbolc tangent. The hdden nodes of the RBF used the Gaussan functon as seen n Equaton 4. The generalsaton ablty of a network s determned manly by the correct model complexty and the number of tranng cycles. There are a number of methods to mprove the generalsaton ablty of a network such as determnng the model complexty, early stoppng, and regularsaton. Model complexty s represented by the number of hdden nodes n the network, as the hdden nodes are responsble for the number of adjustable parameters avalable n the network [3]. Therefore, a more complex model has a greater number of hdden nodes. However, f there are too many hdden nodes the system wll be unnecessarly complex and prone to modellng the system s data too well (over-ftted). Conversely f there are too few hdden nodes the network wll not be able to adequately model the system (under-ftted). One way to determne the optmal number of hdden nodes s to tran the neural network wth dfferent numbers of hdden nodes, and observe the tranng and valdaton errors obtaned. Note that a large number of hdden nodes wll slow the tranng process. The Early Stoppng technque uses a tranng as well as a valdaton dataset. The man dea behnd early stoppng s that the tranng error of the network wll gradually decrease as the number of tranng cycles ncreases. The degree the network s over-traned s measured usng the valdaton dataset as the valdaton error wll decrease at frst and then start to ncrease as the network s over-traned [3]. Therefore, tranng should be stopped at the pont before the valdaton error begns to ncrease. Regularsaton technques encourage weghts that produce smoother network mappngs. An over-ftted network models the tranng data almost exactly, resultng n the mappng produced by the network havng areas of large curvature [3]. Ths results n large weghts. The smplest regularsaton technque uses a weght-decay where mappngs wth large weghts are penalsed [3]. Regularsaton prevents over-tranng. The number of nputs and outputs are determned by the problem, and as stated there are 4 nputs and 4 outputs n the system beng modelled. Determnng the number of hdden nodes used s an teratve and expermental procedure, as t s dependent on the complexty of the relatonshps n the dataset. A rough estmate for the number of hdden nodes s to take half the sum of the total number of nputs and outputs []. Therefore, 4 hdden nodes were used as a startng pont and progressvely ncreased whle montorng the tranng process, to determne an approxmate number of hdden nodes. An alternatve approach would be to start wth a network wth a large number of hdden unts and prune t teratvely to fnd a network whch wll adequately and accurately model the data. The approach that was taken was to tran a network wth a fxed number of hdden nodes, perodcally stoppng the tranng process to determne the error on the valdaton dataset. Therefore, the tranng and valdaton errors durng the tranng process could be observed, and an ndcaton of the generalsaton ablty of the network determned. Ths was done for a varyng number of hdden nodes (4 0 for the MLP and 4-50 for RBF). For each number of hdden nodes, a number of networks were traned, as the optmsaton technques used wll result n a dfferent soluton each tme. Therefore, the dfferent solutons for the set number of hdden nodes could be compared and the best network selected. Also, a number of networks wth large numbers of hdden nodes were traned to see what the effect was on the resultng network (60, 70 and 00). Ths procedure was done, n order to determne an optmum number of hdden nodes and tranng cycles that could adequately and accurately model the system. The goal of ths procedure was to fnd a network that was powerful enough to adequately model the system and generalse well, whle beng easly traned. Usng the procedure dscussed above, t was found that for the MLP the most approprate number of hdden nodes and tranng cycles were 8 and 40 respectvely. Durng the experments carred out on the MLPs, a few observatons were made and are dscussed below. It was notced that ncreasng the number of hdden nodes beyond 8 ddn t seem to ncrease the accuracy by a sgnfcant amount to justfy utlsng a more complex network. The

13 accuracy obtaned for 8 and above hdden nodes was relatvely constant, and only slowly ncreased. Therefore, the least complex network (least number of hdden nodes) wth adequate accuracy was chosen as a more network complex wll take longer to tran and execute. An MLP wth 8 hdden nodes has 76 free parameters for the weghts and bases. Therefore, ncreasng the number of hdden nodes ncreases the number of free parameters that need to be adjusted durng the optmsaton process. Normally, the optmum number of tranng cycles occur at the pont were the valdaton error and the tranng error start to dverge (Early Stoppng). However, snce after a certan pont the valdaton and tranng error remaned relatvely constant (ran parallel to each other, only slowly decreasng), the pont where t started to reman constant was taken as the optmum number of tranng cycles. Ths pattern was observed for each MLP network mplemented and evaluated, and ndcates that the valdaton and tranng data must be smlar. Refer to Fgure 6, showng the pattern observed for 8 hdden nodes for one of neural networks traned. Ths fgure was obtaned by perodcally stoppng the tranng and notng the valdaton and tranng errors. comparatvely small. Determnng the number of tranng cycle necessary for RBF was not as easy as t was for the MLP, as the valdaton and tranng error was more jumpy than was observed wth the MLP. However, the valdaton error was relatvely steady after a certan pont and dd not ncrease: 50 for 30 hdden nodes and 00 for 50 hdden nodes. Fgure 7: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output for the Test Dataset appled to the MLP Fgure 6: Showng the MSE vs. the Tranng Cycles for one of the MLPs Traned. For RBF, 30 hdden nodes and 50 tranng cycles was determned to provde adequate accuracy, comparable to the MLP network mplemented. Intally, the number of hdden nodes nvestgated ranged from 4 to around 0; however, the error obtaned was much larger than that obtaned for the MLP wth the same number hdden of nodes. Therefore, n order to acheve a comparable accuracy to the MLP the number of hdden nodes had to be ncreased. The best accuracy was obtaned wth 50 hdden nodes and 00 tranng cycles. However, once agan ncreasng the number of hdden nodes above 50 resulted n the tranng error decreasng substantally but the valdaton error remaned relatvely constant. Therefore, t was decded that 50 hdden nodes were approprate as beyond ths there was not a great deal of mprovement on the valdaton error. Also, the accuracy dfference between that of 50 and 30 hdden nodes was Fgure 8: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output for the Test Dataset appled to the MLP

14 Fgure 9: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output 3 for the Test Dataset appled to the MLP found that the accuracy decreases wth respect to one or more of the other outputs. The MLP s effectvely tryng to model 4 separate functons at once, therefore, the hdden nodes may have been havng dffculty learnng n order to model all 4 functons at the same tme. Ths s referred to as cross-talk []. One way to attempt to solve ths problem would be to model each output as a separate network []. For both the MLP and RBF, t was dffcult to model Output 3. Ths can be seen from the Actual vs. Predcted plots of the frst 60 data ponts for the testng dataset n Fgures 9 and 3 for Output 3. An attempt was made to decrease the error contrbuton of Output 3 by adjustng the number of tranng cycles and hdden nodes. However, t made a small dfference to the error that Output 3 contrbuted to the total error, and caused the contrbuton to the total error of the other outputs to ncrease. The neural networks dd not seem to be able to model Output 3 as accurately as the other outputs of the system. A possble reason maybe that the dependency of Output 3 on the gven nputs s weak, therefore, more nput varables may need to be measured n order to model ths output more accurately. The followng performance measures were evaluated for each of the neural networks mplemented: () the tme taken to tran the network usng the tranng dataset, () the tme taken to execute or forward-propagate through the network for the testng dataset and () the MSE accuracy obtaned by the network on the testng dataset. The results are summarsed n Table, for the optmum networks obtaned for the RBF and MLP. The scaled conjugate gradent algorthm s used to optmse the MLP weghts and bases. The RBF network wth 30 hdden nodes s shown below as t has a comparable accuracy to the MLP obtaned. Fgure 0: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output 4 for the Test Dataset appled to the MLP A problem encountered wth the RBF network mplementaton was that the tranng functon for the RBF network n NETLAB (rbftran) encounters a dvde by zero error when the number of hdden nodes was substantally large and the code had to be modfed f smulatons were run wth large number of hdden nodes. Alternatvely, the tranng functon for the MLP (netopt) could have been used for the RBF tranng; however, t no longer uses the stage tranng process. A combnaton of early stoppng and regularsaton was used to determne theses optmum parameters. The values for alpha (weght decay coeffcent) and beta (Inverse Nose rato) were ntally set to the default values. However, they ddn t have to be changed sgnfcantly and were eventually set to 0.0 and respectvely. It was notced that by adjustng ether the hdden nodes or the number of tranng cycles, that dfferent outputs contrbuted more to the overall error of the system (Equaton 9). As a result, f an attempt was made to mprove the network accuracy wth respect to a certan output, t was Table : Performance Characterstcs for Indvdual MLP and RBF Networks MLP RBF Tme to Tran (s) Tme to Execute (s) MSE of Test Dataset No. of Hdden nodes 8 30 No. of Tranng Cycles From Table, t can be seen that the RBF wth comparable accuracy to the MLP took much longer to tran. The complexty of the MLP and RBF wth comparable accuracy s sgnfcantly dfferent. The MLP has 8 hdden nodes correspondng to 76 free parameters whle the RBF has 30 hdden nodes correspondng to 74 free parameters. Whle the RBF s supposed to be faster durng the tranng process [], the ncreased complexty of the network has ncreased the tranng tme sgnfcantly. Both the MLP and RBF gave smlar accuracy. The MLP was faster to execute than the RBF, whch was expected. The plots for the Actual vs. Predcted values for the frst 60 ponts for each output for the MLP are shown n Fgures 7-0, and for the RBF n Fgures -4.

15 Fgure : Showng the Predcted vs. Actual Values for the frst 60 ponts of Output for the Test Dataset appled to the RBF Fgure : Showng the Predcted vs. Actual Values for the frst 60 ponts of Output for the Test Dataset appled to the RBF Fgure 3: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output 3 for the Test Dataset appled to the RBF Fgure 4: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output4 for the Test Dataset appled to the RBF An RBF network wth the same number of hdden nodes as the MLP was mplemented to observe how long t would take to tran and execute n comparson to the MLP. If the networks have smlar complextes t was observed that the RBF traned faster than the MLP, and forward executed wth a smlar speed to the MLP. The RBF network wth 8 hdden nodes that was mplemented took 3 seconds to tran and 0.06 seconds to execute. However, the RBF model wth the 8 hdden nodes had a larger MSE than the MLP. D. Commttees A smple averagng commttee was mplemented for both the MLP and RBF usng the NETLAB Toolbox. The averagng commttee conssted of 0 MLP networks wth dentcal archtectures. Each network n the commttee was traned usng the same tranng dataset, however, each network was ntalsed dfferently and traned ndependently. The fnal output was taken as the average of the ndvdual outputs for each network. Also, a commttee consstng of 0 RBF networks was constructed the same way as descrbed for the MLP commttee. The archtectures of each network n the commttees used the optmum parameters found usng the standard approaches. It was found that the averagng commttees only margnally mproved the accuracy obtaned. The commttees took longer to tran and execute whch can only be expected as the commttee s effectvely 0 networks. Table, shows the results captured for the commttee mplemented. From Equaton 5, ncreasng the number of neural networks n the commttees would result n the error reducng as the number of neural networks ncreased. Table : Showng the results for the commttee networks consstng of neural networks wth dentcal archtectures. MLP Commttee RBF Commttee MSE of Test Dataset Tme to Tran (s) 80 9 Tme to Execute (s) Another mplementaton of a commttee that was tested was baggng. In baggng, each network s traned on a bootstrap

16 dataset. A bootstrap dataset s a dataset that s randomly created by selectng n ponts wth replacement, from the tranng dataset wth n patterns [3]. Ths means that some data ponts are chosen more than once and are duplcated n the bootstrap dataset, whle some data ponts wll not be selected at all. Then each bootstrap dataset created s used to tran a separate network, and the fnal output of the commttee s calculated by averagng the outputs of the networks created [3]. A commttee of 0 neural networks was created and each was traned wth a bootstrap dataset. The tranng tme was slghtly longer than that of the straght averagng commttee as the tranng tme ncluded the tme taken to create the bootstrap datasets. The commttee created usng baggng ncreased the accuracy slghtly from that of the smple averagng network but not sgnfcantly. The results obtaned for the commttee usng baggng are n Table 3. E. Bayesan Technques for Neural Networks The archtectures of the MLP and RBF used for the Bayesan technques were the optmum archtectures (number of hdden nodes, number of nputs and outputs, actvaton functons) found usng the standard approaches dscussed n the prevous sectons. Ths allows comparsons to be made between the results obtaned from both approaches. Table 3: Showng the results for the commttee networks usng baggng MLP Commttee RBF Commttee (Baggng) (Baggng) MSE of Test Dataset Tme to Tran (s) Tme to Execute (s) The Bayesan technques were mplemented usng NETLAB for the neural networks. NETLAB allows the mplementaton of the Bayesan technques to be done usng the Hybrd Monte Carlo algorthm. The Bayesan Network utlzng Hybrd Monte Carlo algorthm s mplemented usng NETLAB by the followng steps: the samplng s executed, each set of sampled weghts obtaned are placed nto the network n order to make a predcton, and then the average predcton s computed from the predcted values obtaned from each set of sampled weghts [3]. Snce Bayesan Technques don t requre cross-valdaton technques to determne parameters, a larger tranng dataset could be used. For the Hybrd Monte Carlo algorthm the followng parameters were adjusted to determne the best set of parameters to model the dataset: the step sze, the number of steps n each Hybrd Monte Carlo trajectory, the number of ntal states that were dscarded, and the number of samples retaned to form the posteror dstrbuton. At frst a step sze of was chosen, however, ths resulted n a large number of the samples beng rejected. Therefore, step szes less than were utlsed and the results from the experments noted. It was found that any step sze above 0.00 had hgh rejecton rate, and therefore, a low acceptance rate. As a result, step szes of 0.00 and were tested along wth the other parameters. A step sze of gave a 96% acceptance rate and the results are shown n Tables 4 and 5. If the step sze s extremely small, the Hybrd Monte Carlo algorthm wll take a long tme to converge to a statonary dstrbuton as the state space s beng explored n much smaller steps. If the step sze s large then too much exploraton may occur causng the Hybrd Monte Carlo algorthm to jump over the dstrbuton that s beng searched for, effectvely mssng t. It was notced that ncreasng the number of samples retaned dd not mprove the accuracy of the network. Therefore, 00 samples were eventually retaned whch s a relatvely small number of samples. The number of steps n a trajectory were modfy for dfferent runs, however, after a certan pont t ddn t seem to mprove the accuracy and the steps n a trajectory were set to 00. It was observed that too few steps n a trajectory ddn t allow enough of the sample space to be explored and the MSE was larger for a smaller number of steps. The number of samples omtted was chosen by observng the average number of samples rejected at frst, snce, once the other parameters had been chosen the acceptance rate was hgh, the number of samples omtted was set to a reasonably small number of 0. The coeffcent of data error (β) was vared and eventually set to 30. The coeffcent of weght-decay pror was set to the same used for the standard approach. Tables 4 and 5, show the results obtaned for some of the networks mplemented for the Bayesan MLP. In Table 5, Network 6 gves the best results, and the measures for ths network are shown n Table 6. Table 4: Showng some of the results obtaned n the mplementaton of the MLP usng Bayesan technques (Hybrd Monte Carlo) Network Network Network 3 MSE for Testng Dataset MSE for Tranng Dataset Step Sze No. of Samples Retaned No. Intal States Omtted No. of Steps n a Trajectory β α Table 5: Showng some of the results obtaned n the mplementaton of the MLP usng Bayesan technques (Hybrd Monte Carlo) Network 4 Network 5 Network 6 MSE for Testng dataset MSE for Tranng dataset Step Sze No. of Samples Retaned No. Intal States Omtted No. of Steps n a Trajectory β α Table 6: Showng the performance measures taken for Network 6 Bayesan MLP Tranng Tme (s).8 Executon tme (s).4

17 From, the results n Tables 4-6, t can be seen that the Bayesan MLP gave a better accuracy than the sngle MLP mplemented usng standard approaches. However, t took a substantal amount more tme to tran and execute compared to the sngle MLP. The Bayesan technques usng Hybrd Monte Carlo were attempted wth an RBF, however, dffcultes were experenced and no defnte results were obtaned. F. Support Vector Machnes The LS-SVMlab Toolbox for Matlab was used to smulate the SVM for the gven dataset. Ths toolbox mplements Least Squares Support Vector Machnes for both classfcaton and Regresson problems [4]. Another toolbox that mplemented the ε -nsenstvty loss functon SVM was found, however, to run a smulaton was extremely tme consumng even when the number of samples used to tran the SVM were substantally decreased. Snce an SVM determnes an unknown dependency between a set of nputs and an output, the toolbox handles the case of multple outputs by treatng each of the outputs separately. Therefore, there are effectvely 4 SVMs modellng the dataset. Ths s dfferent to the neural networks where one network was traned to model all 4 outputs. As a result, 4 SVMs were smultaneously mplemented and traned, one for each output of the dataset. The mplementaton of the LS-SVM requred the selecton of two free parameters snce a Radal Bass functon was used for the kernel functon. Therefore, the optmum values of the two free parameters needed to be determned: the wdth or bandwdth of bass functon (σ ), and the regularsaton or penalty parameter (C). An emprcal approach was taken n determnng the free parameters, and s smlar to the approach taken n [3]. Snce there are 4 outputs, the parameters for each correspondng SVM had to be determned. The procedure used s dscussed below wth respect to the determnaton of the parameters for modellng Output. The same procedure was used for the determnaton of the parameters for the other outputs n the dataset. Frst, the regularsaton constant was set at a value of 0 whle varyng the bandwdth of the bass functon for tranng data correspondng to Output. The bass functon s wdth was vared for values from 0.3 to 000. For a small σ, the tranng error was at ts mnmum; however, the valdaton error was very large. Ths gves an ndcaton that the LS- SVM s over-traned for small σ. At around σ =, the tranng and valdaton errors crossed and remaned constant for a whle. Then from about σ = 0, both the valdaton and tranng error started to ncrease whch ndcates that the SVM was not even able to model the tranng data well for large values of σ, and s under-traned. An approprate choce for the bandwdth of the bass functon was decded to be, from the above experments carred out. Secondly, the bandwdth of the bass functon was kept constant at whle the value of the regularsaton constant was vared. The regularsaton constant was vared between and 000 whle observng the tranng and valdaton errors. As the regularsaton constant (C) was ncreased both the valdaton and tranng error decreased together, untl a certan pont where the valdaton error started to ncrease whle the testng error contnued to decrease. Thus, for a small value of C t appears to under-ft the tranng data, and for large values of C the SVM appears to over-ft the tranng dataset. Therefore, the most approprate value for the regularsaton constant was 0, as beyond ths value the valdaton error starts to ncrease. The optmum parameter values chosen to model Output were C=0 and σ =. The optmum parameters for each of the SVM correspondng to the 4 outputs can be seen n Table 7. Fgures 4-7 show the Actual vs. the Predcted values of the frst 60 samples of the each output for test dataset appled to the LS-SVMs. Table 7: Showng the results obtaned for the mplementatons of the LS-SVM SVM Output Output Output3 Output4 MSE for Test Dataset Tranng Tme (s) Executon Tme (s).7 s.7.6 σ 0 0. C From Table 7, t can be seen that the LS-SVM took longer to tran and execute than the neural networks produced usng the standard approach. Even tough, the neural network was modellng 4 relatonshps t was much faster than the LS- SVM whch s only modellng one relatonshp at a tme. The results obtaned from LS-LVM were easly reproducble as opposed to neural networks where one can easly obtan dfferent and less accurate results when the smulaton s rerun, due to the optmsaton technques used. If the error of each of the SVMs are added together as f they are workng n a commttee to predcted each output of the Steam Generator separately, the effectve MSE would be approxmately Fgure 4: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output for the Test Dataset appled to the LS-SVM

18 Fgure 5: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output for the Test Dataset appled to the LS-SVM Fgure 6: Showng the Predcted vs Actual Values for the frst 60 ponts of Output 3 for the Test Dataset appled to the LS-SVM Fgure 7: Showng the Predcted vs. Actual Values for the frst 60 ponts of Output 4 for the Test Dataset appled to the LS-SVM G. Adaptve Neuro-Fuzzy Systems The Fuzzy Logc Toolbox for Matlab was used to smulate the Adaptve Neuro-Fuzzy System. The tranng process nvolves modfyng the membershp functon parameters of the FIS n order emulate the tranng dataset to wthn some error crtera []. The toolbox mplements a Sugeno-type system for the Adaptve Neuro-Fuzzy Inference System. It only supports a sngle output whch s obtaned usng a weghted average defuzzfcaton process. In the Fuzzy Logc Toolkt, the number of output membershp functons must be equal to the number of rules generated, and the output membershp functons must be lnear or a constant. Therefore, each output for the gven dataset was modelled separately, and a lnear output functon of the form n Equaton 7 was used. The toolkt allows for a tranng and valdaton dataset to be used, n ths case the Toolbox selects the model wth the mnmum valdaton data error []. The dea s that overtranng wll be avoded, as t s expected that the valdaton error wll decrease as tranng takes place untl a certan pont where the valdaton error begns to ncrease, ndcatng overtranng. The learnng process s smlar to neural networks except that dfferent parameters are beng adjusted. The dea s to talor the membershp functons to model the nput/output relatonshp of the dataset []. The ANFIS constructs a FIS n whch ts membershp functon parameters adjusted by a tranng algorthm. In ths way, the parameters of the membershp functons wll change through the process of learnng. The Toolkt uses ether back-propagaton or a hybrd method (least squares and back-propagaton) to tran the ANFIS. The Fuzzy Logc Toolbox has dfferent membershp functons avalable, of whch 8 can be used wth the Adaptve Neuro-Fuzzy System: Trangular functon, trapezodal, dfferent Gaussan functons, bell functon, Sgmodal Dfference functon (dfference of Sgmodal functons), Sgmodal product functon (product of Sgmodal functons), and polynomal P curves. The Sgmodal functons have the property of beng asymmetrcal opposed to the Gaussan whch s symmetrcal n nature []. In the Fuzzy Logc Toolkt the number of nput membershp functons and the type of membershp functon used could be modfed. The number of membershp functons was left at the default of per nput, gvng 8 nput membershp functons. Frstly, an FIS structure was ntalsed whch could then be adjusted to model the dataset provded. The generated FIS structure contaned 6 fuzzy rules, therefore, 6 output membershp functons. The ANFIS functon keeps track of the Root Mean Square Error (RMSE) of the tranng dataset at each epoch, as well as the valdaton error assocated wth a valdaton dataset. Dfferent nput membershp functons were tred for each output beng modelled. A curve of the tranng and valdaton errors vs. the tranng cycles was observed by plottng the values stored by the ANFIS functon n the toolbox. It was then possble to see how many epochs were necessary and the

Support Vector Machines

Support Vector Machines /9/207 MIST.6060 Busness Intellgence and Data Mnng What are Support Vector Machnes? Support Vector Machnes Support Vector Machnes (SVMs) are supervsed learnng technques that analyze data and recognze patterns.

More information

Machine Learning 9. week

Machine Learning 9. week Machne Learnng 9. week Mappng Concept Radal Bass Functons (RBF) RBF Networks 1 Mappng It s probably the best scenaro for the classfcaton of two dataset s to separate them lnearly. As you see n the below

More information

Classification / Regression Support Vector Machines

Classification / Regression Support Vector Machines Classfcaton / Regresson Support Vector Machnes Jeff Howbert Introducton to Machne Learnng Wnter 04 Topcs SVM classfers for lnearly separable classes SVM classfers for non-lnearly separable classes SVM

More information

Support Vector Machines

Support Vector Machines Support Vector Machnes Decson surface s a hyperplane (lne n 2D) n feature space (smlar to the Perceptron) Arguably, the most mportant recent dscovery n machne learnng In a nutshell: map the data to a predetermned

More information

Wishing you all a Total Quality New Year!

Wishing you all a Total Quality New Year! Total Qualty Management and Sx Sgma Post Graduate Program 214-15 Sesson 4 Vnay Kumar Kalakband Assstant Professor Operatons & Systems Area 1 Wshng you all a Total Qualty New Year! Hope you acheve Sx sgma

More information

12/2/2009. Announcements. Parametric / Non-parametric. Case-Based Reasoning. Nearest-Neighbor on Images. Nearest-Neighbor Classification

12/2/2009. Announcements. Parametric / Non-parametric. Case-Based Reasoning. Nearest-Neighbor on Images. Nearest-Neighbor Classification Introducton to Artfcal Intellgence V22.0472-001 Fall 2009 Lecture 24: Nearest-Neghbors & Support Vector Machnes Rob Fergus Dept of Computer Scence, Courant Insttute, NYU Sldes from Danel Yeung, John DeNero

More information

Feature Reduction and Selection

Feature Reduction and Selection Feature Reducton and Selecton Dr. Shuang LIANG School of Software Engneerng TongJ Unversty Fall, 2012 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components

More information

Sum of Linear and Fractional Multiobjective Programming Problem under Fuzzy Rules Constraints

Sum of Linear and Fractional Multiobjective Programming Problem under Fuzzy Rules Constraints Australan Journal of Basc and Appled Scences, 2(4): 1204-1208, 2008 ISSN 1991-8178 Sum of Lnear and Fractonal Multobjectve Programmng Problem under Fuzzy Rules Constrants 1 2 Sanjay Jan and Kalash Lachhwan

More information

Smoothing Spline ANOVA for variable screening

Smoothing Spline ANOVA for variable screening Smoothng Splne ANOVA for varable screenng a useful tool for metamodels tranng and mult-objectve optmzaton L. Rcco, E. Rgon, A. Turco Outlne RSM Introducton Possble couplng Test case MOO MOO wth Game Theory

More information

Subspace clustering. Clustering. Fundamental to all clustering techniques is the choice of distance measure between data points;

Subspace clustering. Clustering. Fundamental to all clustering techniques is the choice of distance measure between data points; Subspace clusterng Clusterng Fundamental to all clusterng technques s the choce of dstance measure between data ponts; D q ( ) ( ) 2 x x = x x, j k = 1 k jk Squared Eucldean dstance Assumpton: All features

More information

Lecture 5: Multilayer Perceptrons

Lecture 5: Multilayer Perceptrons Lecture 5: Multlayer Perceptrons Roger Grosse 1 Introducton So far, we ve only talked about lnear models: lnear regresson and lnear bnary classfers. We noted that there are functons that can t be represented

More information

S1 Note. Basis functions.

S1 Note. Basis functions. S1 Note. Bass functons. Contents Types of bass functons...1 The Fourer bass...2 B-splne bass...3 Power and type I error rates wth dfferent numbers of bass functons...4 Table S1. Smulaton results of type

More information

Classifying Acoustic Transient Signals Using Artificial Intelligence

Classifying Acoustic Transient Signals Using Artificial Intelligence Classfyng Acoustc Transent Sgnals Usng Artfcal Intellgence Steve Sutton, Unversty of North Carolna At Wlmngton (suttons@charter.net) Greg Huff, Unversty of North Carolna At Wlmngton (jgh7476@uncwl.edu)

More information

Machine Learning. Support Vector Machines. (contains material adapted from talks by Constantin F. Aliferis & Ioannis Tsamardinos, and Martin Law)

Machine Learning. Support Vector Machines. (contains material adapted from talks by Constantin F. Aliferis & Ioannis Tsamardinos, and Martin Law) Machne Learnng Support Vector Machnes (contans materal adapted from talks by Constantn F. Alfers & Ioanns Tsamardnos, and Martn Law) Bryan Pardo, Machne Learnng: EECS 349 Fall 2014 Support Vector Machnes

More information

Cluster Analysis of Electrical Behavior

Cluster Analysis of Electrical Behavior Journal of Computer and Communcatons, 205, 3, 88-93 Publshed Onlne May 205 n ScRes. http://www.scrp.org/ournal/cc http://dx.do.org/0.4236/cc.205.350 Cluster Analyss of Electrcal Behavor Ln Lu Ln Lu, School

More information

Edge Detection in Noisy Images Using the Support Vector Machines

Edge Detection in Noisy Images Using the Support Vector Machines Edge Detecton n Nosy Images Usng the Support Vector Machnes Hlaro Gómez-Moreno, Saturnno Maldonado-Bascón, Francsco López-Ferreras Sgnal Theory and Communcatons Department. Unversty of Alcalá Crta. Madrd-Barcelona

More information

CS 534: Computer Vision Model Fitting

CS 534: Computer Vision Model Fitting CS 534: Computer Vson Model Fttng Sprng 004 Ahmed Elgammal Dept of Computer Scence CS 534 Model Fttng - 1 Outlnes Model fttng s mportant Least-squares fttng Maxmum lkelhood estmaton MAP estmaton Robust

More information

Assignment # 2. Farrukh Jabeen Algorithms 510 Assignment #2 Due Date: June 15, 2009.

Assignment # 2. Farrukh Jabeen Algorithms 510 Assignment #2 Due Date: June 15, 2009. Farrukh Jabeen Algorthms 51 Assgnment #2 Due Date: June 15, 29. Assgnment # 2 Chapter 3 Dscrete Fourer Transforms Implement the FFT for the DFT. Descrbed n sectons 3.1 and 3.2. Delverables: 1. Concse descrpton

More information

y and the total sum of

y and the total sum of Lnear regresson Testng for non-lnearty In analytcal chemstry, lnear regresson s commonly used n the constructon of calbraton functons requred for analytcal technques such as gas chromatography, atomc absorpton

More information

SLAM Summer School 2006 Practical 2: SLAM using Monocular Vision

SLAM Summer School 2006 Practical 2: SLAM using Monocular Vision SLAM Summer School 2006 Practcal 2: SLAM usng Monocular Vson Javer Cvera, Unversty of Zaragoza Andrew J. Davson, Imperal College London J.M.M Montel, Unversty of Zaragoza. josemar@unzar.es, jcvera@unzar.es,

More information

NUMERICAL SOLVING OPTIMAL CONTROL PROBLEMS BY THE METHOD OF VARIATIONS

NUMERICAL SOLVING OPTIMAL CONTROL PROBLEMS BY THE METHOD OF VARIATIONS ARPN Journal of Engneerng and Appled Scences 006-017 Asan Research Publshng Network (ARPN). All rghts reserved. NUMERICAL SOLVING OPTIMAL CONTROL PROBLEMS BY THE METHOD OF VARIATIONS Igor Grgoryev, Svetlana

More information

An Iterative Solution Approach to Process Plant Layout using Mixed Integer Optimisation

An Iterative Solution Approach to Process Plant Layout using Mixed Integer Optimisation 17 th European Symposum on Computer Aded Process Engneerng ESCAPE17 V. Plesu and P.S. Agach (Edtors) 2007 Elsever B.V. All rghts reserved. 1 An Iteratve Soluton Approach to Process Plant Layout usng Mxed

More information

Learning the Kernel Parameters in Kernel Minimum Distance Classifier

Learning the Kernel Parameters in Kernel Minimum Distance Classifier Learnng the Kernel Parameters n Kernel Mnmum Dstance Classfer Daoqang Zhang 1,, Songcan Chen and Zh-Hua Zhou 1* 1 Natonal Laboratory for Novel Software Technology Nanjng Unversty, Nanjng 193, Chna Department

More information

The Research of Support Vector Machine in Agricultural Data Classification

The Research of Support Vector Machine in Agricultural Data Classification The Research of Support Vector Machne n Agrcultural Data Classfcaton Le Sh, Qguo Duan, Xnmng Ma, Me Weng College of Informaton and Management Scence, HeNan Agrcultural Unversty, Zhengzhou 45000 Chna Zhengzhou

More information

GSLM Operations Research II Fall 13/14

GSLM Operations Research II Fall 13/14 GSLM 58 Operatons Research II Fall /4 6. Separable Programmng Consder a general NLP mn f(x) s.t. g j (x) b j j =. m. Defnton 6.. The NLP s a separable program f ts objectve functon and all constrants are

More information

Classifier Selection Based on Data Complexity Measures *

Classifier Selection Based on Data Complexity Measures * Classfer Selecton Based on Data Complexty Measures * Edth Hernández-Reyes, J.A. Carrasco-Ochoa, and J.Fco. Martínez-Trndad Natonal Insttute for Astrophyscs, Optcs and Electroncs, Lus Enrque Erro No.1 Sta.

More information

Hermite Splines in Lie Groups as Products of Geodesics

Hermite Splines in Lie Groups as Products of Geodesics Hermte Splnes n Le Groups as Products of Geodescs Ethan Eade Updated May 28, 2017 1 Introducton 1.1 Goal Ths document defnes a curve n the Le group G parametrzed by tme and by structural parameters n the

More information

An Entropy-Based Approach to Integrated Information Needs Assessment

An Entropy-Based Approach to Integrated Information Needs Assessment Dstrbuton Statement A: Approved for publc release; dstrbuton s unlmted. An Entropy-Based Approach to ntegrated nformaton Needs Assessment June 8, 2004 Wllam J. Farrell Lockheed Martn Advanced Technology

More information

Outline. Self-Organizing Maps (SOM) US Hebbian Learning, Cntd. The learning rule is Hebbian like:

Outline. Self-Organizing Maps (SOM) US Hebbian Learning, Cntd. The learning rule is Hebbian like: Self-Organzng Maps (SOM) Turgay İBRİKÇİ, PhD. Outlne Introducton Structures of SOM SOM Archtecture Neghborhoods SOM Algorthm Examples Summary 1 2 Unsupervsed Hebban Learnng US Hebban Learnng, Cntd 3 A

More information

EXTENDED BIC CRITERION FOR MODEL SELECTION

EXTENDED BIC CRITERION FOR MODEL SELECTION IDIAP RESEARCH REPORT EXTEDED BIC CRITERIO FOR ODEL SELECTIO Itshak Lapdot Andrew orrs IDIAP-RR-0-4 Dalle olle Insttute for Perceptual Artfcal Intellgence P.O.Box 59 artgny Valas Swtzerland phone +4 7

More information

CHAPTER 3 SEQUENTIAL MINIMAL OPTIMIZATION TRAINED SUPPORT VECTOR CLASSIFIER FOR CANCER PREDICTION

CHAPTER 3 SEQUENTIAL MINIMAL OPTIMIZATION TRAINED SUPPORT VECTOR CLASSIFIER FOR CANCER PREDICTION 48 CHAPTER 3 SEQUENTIAL MINIMAL OPTIMIZATION TRAINED SUPPORT VECTOR CLASSIFIER FOR CANCER PREDICTION 3.1 INTRODUCTION The raw mcroarray data s bascally an mage wth dfferent colors ndcatng hybrdzaton (Xue

More information

Review of approximation techniques

Review of approximation techniques CHAPTER 2 Revew of appromaton technques 2. Introducton Optmzaton problems n engneerng desgn are characterzed by the followng assocated features: the objectve functon and constrants are mplct functons evaluated

More information

Outline. Discriminative classifiers for image recognition. Where in the World? A nearest neighbor recognition example 4/14/2011. CS 376 Lecture 22 1

Outline. Discriminative classifiers for image recognition. Where in the World? A nearest neighbor recognition example 4/14/2011. CS 376 Lecture 22 1 4/14/011 Outlne Dscrmnatve classfers for mage recognton Wednesday, Aprl 13 Krsten Grauman UT-Austn Last tme: wndow-based generc obect detecton basc ppelne face detecton wth boostng as case study Today:

More information

General Vector Machine. Hong Zhao Department of Physics, Xiamen University

General Vector Machine. Hong Zhao Department of Physics, Xiamen University General Vector Machne Hong Zhao (zhaoh@xmu.edu.cn) Department of Physcs, Xamen Unversty The support vector machne (SVM) s an mportant class of learnng machnes for functon approach, pattern recognton, and

More information

(1) The control processes are too complex to analyze by conventional quantitative techniques.

(1) The control processes are too complex to analyze by conventional quantitative techniques. Chapter 0 Fuzzy Control and Fuzzy Expert Systems The fuzzy logc controller (FLC) s ntroduced n ths chapter. After ntroducng the archtecture of the FLC, we study ts components step by step and suggest a

More information

Determining the Optimal Bandwidth Based on Multi-criterion Fusion

Determining the Optimal Bandwidth Based on Multi-criterion Fusion Proceedngs of 01 4th Internatonal Conference on Machne Learnng and Computng IPCSIT vol. 5 (01) (01) IACSIT Press, Sngapore Determnng the Optmal Bandwdth Based on Mult-crteron Fuson Ha-L Lang 1+, Xan-Mn

More information

Outline. Type of Machine Learning. Examples of Application. Unsupervised Learning

Outline. Type of Machine Learning. Examples of Application. Unsupervised Learning Outlne Artfcal Intellgence and ts applcatons Lecture 8 Unsupervsed Learnng Professor Danel Yeung danyeung@eee.org Dr. Patrck Chan patrckchan@eee.org South Chna Unversty of Technology, Chna Introducton

More information

Programming in Fortran 90 : 2017/2018

Programming in Fortran 90 : 2017/2018 Programmng n Fortran 90 : 2017/2018 Programmng n Fortran 90 : 2017/2018 Exercse 1 : Evaluaton of functon dependng on nput Wrte a program who evaluate the functon f (x,y) for any two user specfed values

More information

NAG Fortran Library Chapter Introduction. G10 Smoothing in Statistics

NAG Fortran Library Chapter Introduction. G10 Smoothing in Statistics Introducton G10 NAG Fortran Lbrary Chapter Introducton G10 Smoothng n Statstcs Contents 1 Scope of the Chapter... 2 2 Background to the Problems... 2 2.1 Smoothng Methods... 2 2.2 Smoothng Splnes and Regresson

More information

Mathematics 256 a course in differential equations for engineering students

Mathematics 256 a course in differential equations for engineering students Mathematcs 56 a course n dfferental equatons for engneerng students Chapter 5. More effcent methods of numercal soluton Euler s method s qute neffcent. Because the error s essentally proportonal to the

More information

SVM-based Learning for Multiple Model Estimation

SVM-based Learning for Multiple Model Estimation SVM-based Learnng for Multple Model Estmaton Vladmr Cherkassky and Yunqan Ma Department of Electrcal and Computer Engneerng Unversty of Mnnesota Mnneapols, MN 55455 {cherkass,myq}@ece.umn.edu Abstract:

More information

Meta-heuristics for Multidimensional Knapsack Problems

Meta-heuristics for Multidimensional Knapsack Problems 2012 4th Internatonal Conference on Computer Research and Development IPCSIT vol.39 (2012) (2012) IACSIT Press, Sngapore Meta-heurstcs for Multdmensonal Knapsack Problems Zhbao Man + Computer Scence Department,

More information

CS246: Mining Massive Datasets Jure Leskovec, Stanford University

CS246: Mining Massive Datasets Jure Leskovec, Stanford University CS46: Mnng Massve Datasets Jure Leskovec, Stanford Unversty http://cs46.stanford.edu /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, http://cs46.stanford.edu Perceptron: y = sgn( x Ho to fnd

More information

A Statistical Model Selection Strategy Applied to Neural Networks

A Statistical Model Selection Strategy Applied to Neural Networks A Statstcal Model Selecton Strategy Appled to Neural Networks Joaquín Pzarro Elsa Guerrero Pedro L. Galndo joaqun.pzarro@uca.es elsa.guerrero@uca.es pedro.galndo@uca.es Dpto Lenguajes y Sstemas Informátcos

More information

Performance Evaluation of an ANFIS Based Power System Stabilizer Applied in Multi-Machine Power Systems

Performance Evaluation of an ANFIS Based Power System Stabilizer Applied in Multi-Machine Power Systems Performance Evaluaton of an ANFIS Based Power System Stablzer Appled n Mult-Machne Power Systems A. A GHARAVEISI 1,2 A.DARABI 3 M. MONADI 4 A. KHAJEH-ZADEH 5 M. RASHIDI-NEJAD 1,2,5 1. Shahd Bahonar Unversty

More information

A Binarization Algorithm specialized on Document Images and Photos

A Binarization Algorithm specialized on Document Images and Photos A Bnarzaton Algorthm specalzed on Document mages and Photos Ergna Kavalleratou Dept. of nformaton and Communcaton Systems Engneerng Unversty of the Aegean kavalleratou@aegean.gr Abstract n ths paper, a

More information

BOOSTING CLASSIFICATION ACCURACY WITH SAMPLES CHOSEN FROM A VALIDATION SET

BOOSTING CLASSIFICATION ACCURACY WITH SAMPLES CHOSEN FROM A VALIDATION SET 1 BOOSTING CLASSIFICATION ACCURACY WITH SAMPLES CHOSEN FROM A VALIDATION SET TZU-CHENG CHUANG School of Electrcal and Computer Engneerng, Purdue Unversty, West Lafayette, Indana 47907 SAUL B. GELFAND School

More information

6.854 Advanced Algorithms Petar Maymounkov Problem Set 11 (November 23, 2005) With: Benjamin Rossman, Oren Weimann, and Pouya Kheradpour

6.854 Advanced Algorithms Petar Maymounkov Problem Set 11 (November 23, 2005) With: Benjamin Rossman, Oren Weimann, and Pouya Kheradpour 6.854 Advanced Algorthms Petar Maymounkov Problem Set 11 (November 23, 2005) Wth: Benjamn Rossman, Oren Wemann, and Pouya Kheradpour Problem 1. We reduce vertex cover to MAX-SAT wth weghts, such that the

More information

A mathematical programming approach to the analysis, design and scheduling of offshore oilfields

A mathematical programming approach to the analysis, design and scheduling of offshore oilfields 17 th European Symposum on Computer Aded Process Engneerng ESCAPE17 V. Plesu and P.S. Agach (Edtors) 2007 Elsever B.V. All rghts reserved. 1 A mathematcal programmng approach to the analyss, desgn and

More information

Parallelism for Nested Loops with Non-uniform and Flow Dependences

Parallelism for Nested Loops with Non-uniform and Flow Dependences Parallelsm for Nested Loops wth Non-unform and Flow Dependences Sam-Jn Jeong Dept. of Informaton & Communcaton Engneerng, Cheonan Unversty, 5, Anseo-dong, Cheonan, Chungnam, 330-80, Korea. seong@cheonan.ac.kr

More information

Incremental Learning with Support Vector Machines and Fuzzy Set Theory

Incremental Learning with Support Vector Machines and Fuzzy Set Theory The 25th Workshop on Combnatoral Mathematcs and Computaton Theory Incremental Learnng wth Support Vector Machnes and Fuzzy Set Theory Yu-Mng Chuang 1 and Cha-Hwa Ln 2* 1 Department of Computer Scence and

More information

Support Vector Machines. CS534 - Machine Learning

Support Vector Machines. CS534 - Machine Learning Support Vector Machnes CS534 - Machne Learnng Perceptron Revsted: Lnear Separators Bnar classfcaton can be veed as the task of separatng classes n feature space: b > 0 b 0 b < 0 f() sgn( b) Lnear Separators

More information

Biostatistics 615/815

Biostatistics 615/815 The E-M Algorthm Bostatstcs 615/815 Lecture 17 Last Lecture: The Smplex Method General method for optmzaton Makes few assumptons about functon Crawls towards mnmum Some recommendatons Multple startng ponts

More information

X- Chart Using ANOM Approach

X- Chart Using ANOM Approach ISSN 1684-8403 Journal of Statstcs Volume 17, 010, pp. 3-3 Abstract X- Chart Usng ANOM Approach Gullapall Chakravarth 1 and Chaluvad Venkateswara Rao Control lmts for ndvdual measurements (X) chart are

More information

Type-2 Fuzzy Non-uniform Rational B-spline Model with Type-2 Fuzzy Data

Type-2 Fuzzy Non-uniform Rational B-spline Model with Type-2 Fuzzy Data Malaysan Journal of Mathematcal Scences 11(S) Aprl : 35 46 (2017) Specal Issue: The 2nd Internatonal Conference and Workshop on Mathematcal Analyss (ICWOMA 2016) MALAYSIAN JOURNAL OF MATHEMATICAL SCIENCES

More information

GA-Based Learning Algorithms to Identify Fuzzy Rules for Fuzzy Neural Networks

GA-Based Learning Algorithms to Identify Fuzzy Rules for Fuzzy Neural Networks Seventh Internatonal Conference on Intellgent Systems Desgn and Applcatons GA-Based Learnng Algorthms to Identfy Fuzzy Rules for Fuzzy Neural Networks K Almejall, K Dahal, Member IEEE, and A Hossan, Member

More information

A MOVING MESH APPROACH FOR SIMULATION BUDGET ALLOCATION ON CONTINUOUS DOMAINS

A MOVING MESH APPROACH FOR SIMULATION BUDGET ALLOCATION ON CONTINUOUS DOMAINS Proceedngs of the Wnter Smulaton Conference M E Kuhl, N M Steger, F B Armstrong, and J A Jones, eds A MOVING MESH APPROACH FOR SIMULATION BUDGET ALLOCATION ON CONTINUOUS DOMAINS Mark W Brantley Chun-Hung

More information

Tuning of Fuzzy Inference Systems Through Unconstrained Optimization Techniques

Tuning of Fuzzy Inference Systems Through Unconstrained Optimization Techniques Tunng of Fuzzy Inference Systems Through Unconstraned Optmzaton Technques ROGERIO ANDRADE FLAUZINO, IVAN NUNES DA SILVA Department of Electrcal Engneerng State Unversty of São Paulo UNESP CP 473, CEP 733-36,

More information

Adaptive Virtual Support Vector Machine for the Reliability Analysis of High-Dimensional Problems

Adaptive Virtual Support Vector Machine for the Reliability Analysis of High-Dimensional Problems Proceedngs of the ASME 2 Internatonal Desgn Engneerng Techncal Conferences & Computers and Informaton n Engneerng Conference IDETC/CIE 2 August 29-3, 2, Washngton, D.C., USA DETC2-47538 Adaptve Vrtual

More information

Problem Definitions and Evaluation Criteria for Computational Expensive Optimization

Problem Definitions and Evaluation Criteria for Computational Expensive Optimization Problem efntons and Evaluaton Crtera for Computatonal Expensve Optmzaton B. Lu 1, Q. Chen and Q. Zhang 3, J. J. Lang 4, P. N. Suganthan, B. Y. Qu 6 1 epartment of Computng, Glyndwr Unversty, UK Faclty

More information

The Codesign Challenge

The Codesign Challenge ECE 4530 Codesgn Challenge Fall 2007 Hardware/Software Codesgn The Codesgn Challenge Objectves In the codesgn challenge, your task s to accelerate a gven software reference mplementaton as fast as possble.

More information

A New Approach For the Ranking of Fuzzy Sets With Different Heights

A New Approach For the Ranking of Fuzzy Sets With Different Heights New pproach For the ankng of Fuzzy Sets Wth Dfferent Heghts Pushpnder Sngh School of Mathematcs Computer pplcatons Thapar Unversty, Patala-7 00 Inda pushpndersnl@gmalcom STCT ankng of fuzzy sets plays

More information

User Authentication Based On Behavioral Mouse Dynamics Biometrics

User Authentication Based On Behavioral Mouse Dynamics Biometrics User Authentcaton Based On Behavoral Mouse Dynamcs Bometrcs Chee-Hyung Yoon Danel Donghyun Km Department of Computer Scence Department of Computer Scence Stanford Unversty Stanford Unversty Stanford, CA

More information

Performance Evaluation of Information Retrieval Systems

Performance Evaluation of Information Retrieval Systems Why System Evaluaton? Performance Evaluaton of Informaton Retreval Systems Many sldes n ths secton are adapted from Prof. Joydeep Ghosh (UT ECE) who n turn adapted them from Prof. Dk Lee (Unv. of Scence

More information

Improvement of Spatial Resolution Using BlockMatching Based Motion Estimation and Frame. Integration

Improvement of Spatial Resolution Using BlockMatching Based Motion Estimation and Frame. Integration Improvement of Spatal Resoluton Usng BlockMatchng Based Moton Estmaton and Frame Integraton Danya Suga and Takayuk Hamamoto Graduate School of Engneerng, Tokyo Unversty of Scence, 6-3-1, Nuku, Katsuska-ku,

More information

Active Contours/Snakes

Active Contours/Snakes Actve Contours/Snakes Erkut Erdem Acknowledgement: The sldes are adapted from the sldes prepared by K. Grauman of Unversty of Texas at Austn Fttng: Edges vs. boundares Edges useful sgnal to ndcate occludng

More information

Some Advanced SPC Tools 1. Cumulative Sum Control (Cusum) Chart For the data shown in Table 9-1, the x chart can be generated.

Some Advanced SPC Tools 1. Cumulative Sum Control (Cusum) Chart For the data shown in Table 9-1, the x chart can be generated. Some Advanced SP Tools 1. umulatve Sum ontrol (usum) hart For the data shown n Table 9-1, the x chart can be generated. However, the shft taken place at sample #21 s not apparent. 92 For ths set samples,

More information

Solving two-person zero-sum game by Matlab

Solving two-person zero-sum game by Matlab Appled Mechancs and Materals Onlne: 2011-02-02 ISSN: 1662-7482, Vols. 50-51, pp 262-265 do:10.4028/www.scentfc.net/amm.50-51.262 2011 Trans Tech Publcatons, Swtzerland Solvng two-person zero-sum game by

More information

Simulation: Solving Dynamic Models ABE 5646 Week 11 Chapter 2, Spring 2010

Simulation: Solving Dynamic Models ABE 5646 Week 11 Chapter 2, Spring 2010 Smulaton: Solvng Dynamc Models ABE 5646 Week Chapter 2, Sprng 200 Week Descrpton Readng Materal Mar 5- Mar 9 Evaluatng [Crop] Models Comparng a model wth data - Graphcal, errors - Measures of agreement

More information

Unsupervised Learning

Unsupervised Learning Pattern Recognton Lecture 8 Outlne Introducton Unsupervsed Learnng Parametrc VS Non-Parametrc Approach Mxture of Denstes Maxmum-Lkelhood Estmates Clusterng Prof. Danel Yeung School of Computer Scence and

More information

EECS 730 Introduction to Bioinformatics Sequence Alignment. Luke Huan Electrical Engineering and Computer Science

EECS 730 Introduction to Bioinformatics Sequence Alignment. Luke Huan Electrical Engineering and Computer Science EECS 730 Introducton to Bonformatcs Sequence Algnment Luke Huan Electrcal Engneerng and Computer Scence http://people.eecs.ku.edu/~huan/ HMM Π s a set of states Transton Probabltes a kl Pr( l 1 k Probablty

More information

R s s f. m y s. SPH3UW Unit 7.3 Spherical Concave Mirrors Page 1 of 12. Notes

R s s f. m y s. SPH3UW Unit 7.3 Spherical Concave Mirrors Page 1 of 12. Notes SPH3UW Unt 7.3 Sphercal Concave Mrrors Page 1 of 1 Notes Physcs Tool box Concave Mrror If the reflectng surface takes place on the nner surface of the sphercal shape so that the centre of the mrror bulges

More information

An Image Fusion Approach Based on Segmentation Region

An Image Fusion Approach Based on Segmentation Region Rong Wang, L-Qun Gao, Shu Yang, Yu-Hua Cha, and Yan-Chun Lu An Image Fuson Approach Based On Segmentaton Regon An Image Fuson Approach Based on Segmentaton Regon Rong Wang, L-Qun Gao, Shu Yang 3, Yu-Hua

More information

Helsinki University Of Technology, Systems Analysis Laboratory Mat Independent research projects in applied mathematics (3 cr)

Helsinki University Of Technology, Systems Analysis Laboratory Mat Independent research projects in applied mathematics (3 cr) Helsnk Unversty Of Technology, Systems Analyss Laboratory Mat-2.08 Independent research projects n appled mathematcs (3 cr) "! #$&% Antt Laukkanen 506 R ajlaukka@cc.hut.f 2 Introducton...3 2 Multattrbute

More information

Random Kernel Perceptron on ATTiny2313 Microcontroller

Random Kernel Perceptron on ATTiny2313 Microcontroller Random Kernel Perceptron on ATTny233 Mcrocontroller Nemanja Djurc Department of Computer and Informaton Scences, Temple Unversty Phladelpha, PA 922, USA nemanja.djurc@temple.edu Slobodan Vucetc Department

More information

Chapter 6 Programmng the fnte element method Inow turn to the man subject of ths book: The mplementaton of the fnte element algorthm n computer programs. In order to make my dscusson as straghtforward

More information

Novel Fuzzy logic Based Edge Detection Technique

Novel Fuzzy logic Based Edge Detection Technique Novel Fuzzy logc Based Edge Detecton Technque Aborsade, D.O Department of Electroncs Engneerng, adoke Akntola Unversty of Tech., Ogbomoso. Oyo-state. doaborsade@yahoo.com Abstract Ths paper s based on

More information

Computational Results of Hybrid Learning in Adaptive Neuro Fuzzy Inference System for Optimal Prediction

Computational Results of Hybrid Learning in Adaptive Neuro Fuzzy Inference System for Optimal Prediction Internatonal Journal of Appled Engneerng Research ISSN 0973-456 Volume 1, Number 16 (017) pp. 5810-5818 Research Inda Publcatons. http://.rpublcaton.com Computatonal Results of Hybrd Learnng n Adaptve

More information

TN348: Openlab Module - Colocalization

TN348: Openlab Module - Colocalization TN348: Openlab Module - Colocalzaton Topc The Colocalzaton module provdes the faclty to vsualze and quantfy colocalzaton between pars of mages. The Colocalzaton wndow contans a prevew of the two mages

More information

Air Transport Demand. Ta-Hui Yang Associate Professor Department of Logistics Management National Kaohsiung First Univ. of Sci. & Tech.

Air Transport Demand. Ta-Hui Yang Associate Professor Department of Logistics Management National Kaohsiung First Univ. of Sci. & Tech. Ar Transport Demand Ta-Hu Yang Assocate Professor Department of Logstcs Management Natonal Kaohsung Frst Unv. of Sc. & Tech. 1 Ar Transport Demand Demand for ar transport between two ctes or two regons

More information

A Unified Framework for Semantics and Feature Based Relevance Feedback in Image Retrieval Systems

A Unified Framework for Semantics and Feature Based Relevance Feedback in Image Retrieval Systems A Unfed Framework for Semantcs and Feature Based Relevance Feedback n Image Retreval Systems Ye Lu *, Chunhu Hu 2, Xngquan Zhu 3*, HongJang Zhang 2, Qang Yang * School of Computng Scence Smon Fraser Unversty

More information

Parallel matrix-vector multiplication

Parallel matrix-vector multiplication Appendx A Parallel matrx-vector multplcaton The reduced transton matrx of the three-dmensonal cage model for gel electrophoress, descrbed n secton 3.2, becomes excessvely large for polymer lengths more

More information

Quality Improvement Algorithm for Tetrahedral Mesh Based on Optimal Delaunay Triangulation

Quality Improvement Algorithm for Tetrahedral Mesh Based on Optimal Delaunay Triangulation Intellgent Informaton Management, 013, 5, 191-195 Publshed Onlne November 013 (http://www.scrp.org/journal/m) http://dx.do.org/10.36/m.013.5601 Qualty Improvement Algorthm for Tetrahedral Mesh Based on

More information

Using Neural Networks and Support Vector Machines in Data Mining

Using Neural Networks and Support Vector Machines in Data Mining Usng eural etworks and Support Vector Machnes n Data Mnng RICHARD A. WASIOWSKI Computer Scence Department Calforna State Unversty Domnguez Hlls Carson, CA 90747 USA Abstract: - Multvarate data analyss

More information

Lecture 4: Principal components

Lecture 4: Principal components /3/6 Lecture 4: Prncpal components 3..6 Multvarate lnear regresson MLR s optmal for the estmaton data...but poor for handlng collnear data Covarance matrx s not nvertble (large condton number) Robustness

More information

Optimizing Document Scoring for Query Retrieval

Optimizing Document Scoring for Query Retrieval Optmzng Document Scorng for Query Retreval Brent Ellwen baellwe@cs.stanford.edu Abstract The goal of ths project was to automate the process of tunng a document query engne. Specfcally, I used machne learnng

More information

Three supervised learning methods on pen digits character recognition dataset

Three supervised learning methods on pen digits character recognition dataset Three supervsed learnng methods on pen dgts character recognton dataset Chrs Flezach Department of Computer Scence and Engneerng Unversty of Calforna, San Dego San Dego, CA 92093 cflezac@cs.ucsd.edu Satoru

More information

SUMMARY... I TABLE OF CONTENTS...II INTRODUCTION...

SUMMARY... I TABLE OF CONTENTS...II INTRODUCTION... Summary A follow-the-leader robot system s mplemented usng Dscrete-Event Supervsory Control methods. The system conssts of three robots, a leader and two followers. The dea s to get the two followers to

More information

Compiler Design. Spring Register Allocation. Sample Exercises and Solutions. Prof. Pedro C. Diniz

Compiler Design. Spring Register Allocation. Sample Exercises and Solutions. Prof. Pedro C. Diniz Compler Desgn Sprng 2014 Regster Allocaton Sample Exercses and Solutons Prof. Pedro C. Dnz USC / Informaton Scences Insttute 4676 Admralty Way, Sute 1001 Marna del Rey, Calforna 90292 pedro@s.edu Regster

More information

A Robust LS-SVM Regression

A Robust LS-SVM Regression PROCEEDIGS OF WORLD ACADEMY OF SCIECE, EGIEERIG AD ECHOLOGY VOLUME 7 AUGUS 5 ISS 37- A Robust LS-SVM Regresson József Valyon, and Gábor Horváth Abstract In comparson to the orgnal SVM, whch nvolves a quadratc

More information

MARS: Still an Alien Planet in Soft Computing?

MARS: Still an Alien Planet in Soft Computing? MARS: Stll an Alen Planet n Soft Computng? Ajth Abraham and Dan Stenberg School of Computng and Informaton Technology Monash Unversty (Gppsland Campus), Churchll 3842, Australa Emal: ajth.abraham@nfotech.monash.edu.au

More information

An Application of the Dulmage-Mendelsohn Decomposition to Sparse Null Space Bases of Full Row Rank Matrices

An Application of the Dulmage-Mendelsohn Decomposition to Sparse Null Space Bases of Full Row Rank Matrices Internatonal Mathematcal Forum, Vol 7, 2012, no 52, 2549-2554 An Applcaton of the Dulmage-Mendelsohn Decomposton to Sparse Null Space Bases of Full Row Rank Matrces Mostafa Khorramzadeh Department of Mathematcal

More information

Fuzzy Filtering Algorithms for Image Processing: Performance Evaluation of Various Approaches

Fuzzy Filtering Algorithms for Image Processing: Performance Evaluation of Various Approaches Proceedngs of the Internatonal Conference on Cognton and Recognton Fuzzy Flterng Algorthms for Image Processng: Performance Evaluaton of Varous Approaches Rajoo Pandey and Umesh Ghanekar Department of

More information

Optimal Design of Nonlinear Fuzzy Model by Means of Independent Fuzzy Scatter Partition

Optimal Design of Nonlinear Fuzzy Model by Means of Independent Fuzzy Scatter Partition Optmal Desgn of onlnear Fuzzy Model by Means of Independent Fuzzy Scatter Partton Keon-Jun Park, Hyung-Kl Kang and Yong-Kab Km *, Department of Informaton and Communcaton Engneerng, Wonkwang Unversty,

More information

Parameter estimation for incomplete bivariate longitudinal data in clinical trials

Parameter estimation for incomplete bivariate longitudinal data in clinical trials Parameter estmaton for ncomplete bvarate longtudnal data n clncal trals Naum M. Khutoryansky Novo Nordsk Pharmaceutcals, Inc., Prnceton, NJ ABSTRACT Bvarate models are useful when analyzng longtudnal data

More information

Radial Basis Functions

Radial Basis Functions Radal Bass Functons Mesh Reconstructon Input: pont cloud Output: water-tght manfold mesh Explct Connectvty estmaton Implct Sgned dstance functon estmaton Image from: Reconstructon and Representaton of

More information

The Greedy Method. Outline and Reading. Change Money Problem. Greedy Algorithms. Applications of the Greedy Strategy. The Greedy Method Technique

The Greedy Method. Outline and Reading. Change Money Problem. Greedy Algorithms. Applications of the Greedy Strategy. The Greedy Method Technique //00 :0 AM Outlne and Readng The Greedy Method The Greedy Method Technque (secton.) Fractonal Knapsack Problem (secton..) Task Schedulng (secton..) Mnmum Spannng Trees (secton.) Change Money Problem Greedy

More information

Content Based Image Retrieval Using 2-D Discrete Wavelet with Texture Feature with Different Classifiers

Content Based Image Retrieval Using 2-D Discrete Wavelet with Texture Feature with Different Classifiers IOSR Journal of Electroncs and Communcaton Engneerng (IOSR-JECE) e-issn: 78-834,p- ISSN: 78-8735.Volume 9, Issue, Ver. IV (Mar - Apr. 04), PP 0-07 Content Based Image Retreval Usng -D Dscrete Wavelet wth

More information

CLASSIFICATION OF ULTRASONIC SIGNALS

CLASSIFICATION OF ULTRASONIC SIGNALS The 8 th Internatonal Conference of the Slovenan Socety for Non-Destructve Testng»Applcaton of Contemporary Non-Destructve Testng n Engneerng«September -3, 5, Portorož, Slovena, pp. 7-33 CLASSIFICATION

More information

Multicriteria Decision Making

Multicriteria Decision Making Multcrtera Decson Makng Andrés Ramos (Andres.Ramos@comllas.edu) Pedro Sánchez (Pedro.Sanchez@comllas.edu) Sonja Wogrn (Sonja.Wogrn@comllas.edu) Contents 1. Basc concepts 2. Contnuous methods 3. Dscrete

More information