Why Neural Networks? An Enduring Synthesis. Neural Networks. After s: Hebb, McCulloch and Pitts

Transcription

1 Neural Networks Donald H. Cooley An Endurng Synthess An endurng synthess for how the bran works wll enable us to explan how we rapdly and spontaneously adapt to nosy and complex envronments whose rules may change UNPREDICABILIY How do we cope wth the bloomng buzzng confuson of every day? (Wm. James) Central queston of bologcal ntellgence: How to acheve autonomous behavor n a changng and unpredctable world? What facultes are needed f an automaton s to acheve bologcal ntellgence? percepton see, hear, smell, touch cognton recognze, recall, plan, hypothesze, test acton exploratory and goal orented movements cogntve/emotonal recognze an object... so what? Why Neural Networks? Ever snce the 94 s t has been known that the: functonal unts that govern actons are dstrbuted patterns (electrcal & chemcal) across a network of cells or neurons. Hence: to smulate human behavor smulate neural networks 3 4 Why are bologcal NN s dffcult to study? Anmals are desgned to hde ther neural mechansms from behavoral ntrospecton. We want to analyze at the mcrolevel; whereas, we can only observe at the macro-level. After 94 94s: Hebb, McCulloch and Ptts Mechansm for learnng n bologcal neurons Neural-lke networks can compute any arthmetc functon 5 6

2 McCulloch-Ptts Neuron he McCulloch-Ptts Neuron, developed n 943 Bnary nputs / Bnary outputs / hreshold transfer functon Weghts were + (exctatory) or - (nhbtory) All exctatory connectons to a partcular neuron have the same weght; however, the values comng nto one unt Y do not have to be the same as those nto Y McCulloch-Ptts Neuron Each neuron has a fxed thresholdθ such that f the net value s > θ the neuron fres (outputs a ) θ s set so that nhbton s absolute,.e. any nonzero nhbtory nput wll prevent the neuron from frng It takes one tme step for a sgnal to pass over one connecton lnk 7 8 McCulloch-Ptts Neuron McCulloch/Ptts Neuron X.. X n X n+... X n+m w w -p -p nhbton s absolute hence > nw - p,.e. one nhbtor shuts t off Y θ Y wll fre (output a ) f t receves k or more exctatory nputs and no nhbtory nputs such that kw θ > (k-)w For the followng neurons, (the threshold) = X X Y X X Y What logc functons are mplemented by these networks? θ 9 McCulloch/Ptts Neuron For the followng neurons, (the threshold) = X X - - Z Z θ Y Although smple, a MP neuron can mplement any Boolean functon Logcal completeness AND OR NO What logc functon s mplemented by ths network?

3 After the MP Neuron Learnng/ranng/eachng he MP neuron seemed to have functonalty, but there was nothng about how to update or automatcally generate the weghts. 949 Donald Hebb Proposed a scheme for updatng a neuron s weghts Stated that nformaton (memores) could be stored n the connectons (synaptc weghts) Proposed a learnng scheme n whch synaptc weghts changed durng learnng 3 After the MP Neuron Learnng/ranng/eachng Hebb s Rule: When an axon of cell A s near enough to excte cell B and repeatedly takes place n frng t, some growth process or metabolc change takes place n one or both cells such that A s effcency as one of the cells frng B s ncreased. Is there bologcal support for Hebban learnng? Pavlov s dog 4 Perceptrons & Lnear Separablty 5 Applcatons Aerospace Hgh performance arcraft autoplots, flght path smulatons, arcraft control systems, autoplot enhancements, arcraft component smulatons, arcraft component fault detectors Automotve Automoble automatc gudance systems, warranty actvty analyzers Bankng Check and other document readers, credt applcaton evaluators 6 Applcatons Defense Weapon steerng, target trackng, object dscrmnaton, facal recognton, new knds of sensors, sonar, radar and mage sgnal processng ncludng data compresson, feature extracton and nose suppresson, sgnal/mage dentfcaton Electroncs Code sequence predcton, ntegrated crcut chp layout, process control, chp falure analyss, machne vson, voce synthess, nonlnear modelng 7 Applcatons Fnancal Real estate apprasal, loan advsor, mortgage screenng, corporate bond ratng, credt lne use analyss, portfolo tradng program, corporate fnancal analyss, currency prce predcton Manufacturng Manufacturng process control, product desgn and analyss, process and machne dagnoss, real-tme partcle dentfcaton, vsual qualty nspecton systems, beer testng, weldng qualty analyss, paper qualty predcton, computer chp qualty analyss, analyss of grndng operatons, chemcal product desgn analyss, machne mantenance analyss, project bddng, plannng and management, dynamc modelng of chemcal process systems 8

4 Applcatons Medcal Breast cancer cell analyss, EEG and ECG analyss, prosthess desgn, optmzaton of transplant tmes, hosptal expense reducton, hosptal qualty mprovement, emergency room test advsement Robotcs rajectory control, forklft robot, manpulator controllers, vson systems Speech Speech recognton, speech compresson, vowel classfcaton, text to speech synthess Applcatons Securtes Market analyss, automatc bond ratng, stock tradng advsory systems elecommuncatons Image and data compresson, automated nformaton servces, real-tme translaton of spoken language, customer payment processng systems ransportaton ruck brake dagnoss systems, vehcle schedulng, routng systems Vehcle recognton & countng 9 he Bran as a Neural Network he bran has ~ neurons Neurons respond slowly (mllseconds) here are many dfferent types of neurons Neurons communcate wth other neurons through synapses Each neuron may be connected to as many as, other neurons Axons connect neurons to one another axon etc. NEURON Synapses (synaptc juncton) nucleus axon dendrte note: In the bran, there may be as many as, connectons Response tme ~= mllseconds Neural Network Organzaton Neurons are organzed nto local crcuts. All of the neurons n a crcut/network may not be of the same type Neural networks perform specfc functons, e.g. vson 3 ermnology Neuron artfcal bologcal Neural Network Names NN ANN parallel dstrbuted processng model PDP connectvst/connectonsm model adaptve system self-organzng system neurocomputng neuromorphc system 4

5 An Artfcal Neuron NN Learnng x x n w w n f(x) Y = output = f(x) Y = n + exp( wx) = f(x) s sometmes called the neuron transfer functon. here are many such functons 5 Most mportant feature of NN s s ablty to learn Learnng mples that they are able to mprove ther performance he process of learnng nvolves adaptng or modfyng the network s free parameters to mprove performance 6 NN Learnng- Free Parameters What are a NN s free parameters? NN Learnng- Free Parameters What are a NN s free parameters? Weghts Bases 7 8 Basc Learnng Rules Error-Correcton Learnng here are 5 basc learnng rules Error correctng (optmzaton) Memory based (memorzaton) Hebban (bologcally nspred) Compettve (bologcally nspred) Boltzman (statstcal) 9 A partcular weght s adjusted accordng to the error of that neuron s output ξ( n) = ek ( n) ξ( n) = value of the error for tranng sample n e ( n) = d ( n) y ( n) => error sgnal k k k d ( n) = desred output, y ( n) = actual output k k 3

6 Error-Correcton Learnng he goal s to mnmze ξ, the cost or error functon he learnng rule s commonly referred to as the delta rule or the Wdrow-Huff learnng rule Error-Correcton Learnng wkj ( n) = ηek ( n) xj ( n) η = learnng rate he weght adjustment s proportonal to the product of the error sgnal and the nput sgnal of the synapse n queston η s a postve constant that corresponds to the rate of learnng 3 3 Delta Rule he adjustment made to a synaptc weght of a neuron s proportonal to the product of the error sgnal and the nput of the synapse n queston. Delta Rule Assumes that the error s drectly measureable X s a presynaptc value and v s a postsynaptc value Error-Correcton Learnng w ( n + ) = w + w ( n) kj k ( n) kj he precedng represents a closed-loop feedback system (CLFS) he stablty of a CLFS s determned by what s fed back he most mportant element of the feedback s the learnng rate η Learnng Rate Error correcton learnng occurs n a closed-loop feedback system Stablty s determned by the values of the parameters n the feedback loop(s) here s only one feedback loop and there s only one parameter (adjustable); namely η hus η s very mportant n how the network acts as t learns 35 36

7 Learnng asks Pattern Assocaton Assocatve Memory Pattern Recognton Classfcaton Functon Approxmaton Control Flterng Optmzaton In any problem (optmzaton), the basc dea s to fnd a (cost/error/etc.) functon to optmze and then teratvely change the parameters of the cost functon n such a way as to keep mprovng t untl t can no longer be mproved Optmzaton he four most mportant questons are What do we optmze? How (n what drecton) do we change the parameter(s) to optmze? How much do we change the parameters? How do we know when we are done? Optmzaton x k = ( x k + x k ) = α k p k x k p k - Search Drecton αk p k x k + α k - Learnng Rate 39 4 What to Optmze Assume a sngle lnear neuron, wth weght s w For each nput x() there s an assocated output y() here s also a desred output d(), and thus the error s e()=d()-y() - Overdamped

8 Underdamped η oo Large η oo Large η oo Large Can you gve an explanaton as to why the system would go unstable as shown n the precedng pcture? Can you gve an explanaton as to why the system would go unstable as shown n the precedng pcture? Note that movement s perpendcular to the gradent lne. he gradent lne(s) are lnes of constant slope If the dstance moved places the pont on a pont of greater slope then the next move wll do the same, etc Least Mean Square (LMS) Algorthm LMS algorthm s based on nstantaneous error values What t means s that we don t look at the error over a sequence of nputs, but only the current nput. By lookng at nstantaneous error values, we can only get an estmate of the error gradent 47 Least Mean Square (LMS) Algorthm What the theory behnd the LMS algorthm shows s that f we choose a small enough learnng rate, then over tme we wll fnd the w that gves the mnmum LMS error Lke lnear least-squares, we are usng a lnear neuron 48

9 Least Mean Square (LMS) Because the update s only an estmate of the gradent, unlke the steepest descent algorthm, the LMS algorthm follows an unpredctable path to the mnmum. Sometmes ths s called stochastc gradent descent Least Mean Square (LMS) he LMS algorthm works as follows: set w() = choose a value for η for each nput x(), and output d() compute e(n) = d(n)-w(n) x(n) w(n+)=w(n)+ η x(n)e(n) 49 5 LMS Example LMS Convergence Consderatons hat whch s changng the weght at each teraton s the learnng rate η, and the nput vector Stablty of the LMS algorthm, s a functon of the statstcal characterstcs of the nput and the sze of the learnng rate. η Stated another way we have to select accordng to the envronment n whch the X s are gven 5 5 Perceptron LMS bult on lnear neuron Perceptron bult on McCullch Pts neuron +/- for output If we consder a sngle neuron as a classfer, then we have the followng defntons m Perceptron v = w x + b = v > y = v 53 54

10 Perceptron Perceptron If we combne the bas wth the weght matrx w, and nclude an addtonal nput of for the x matrx x hen we have m vn ( ) = w( nx ) ( n) = w ( nxn ) ( ) = Perceptron Perceptron What s the relatonshp between the hyperplane (decson surface) and the weght vector? 57 What s the relatonshp between the hyperplane (decson surface) and the weght vector? Hyperplane defnes a lne (surface) for whch v= he hyperplane s perpendcular to the weght vector he bas defnes the dstance of the hyperplane from the orgn 58 Perceptron Perceptron Is a bas value mportant to a perceptron? Is a bas value mportant to a perceptron? Wthout the bas value, the hyperplane can only go through the orgn. 59 6

11 Example Let s say we have a group of lnearly separable values that we want a perceptron to recognze. Say the nput pars are (,) = class (,)=class (,) = class (,) = class 6 Example hus we want w and b such that F(w*x + b) = for class and for class We want a decson surface defned as [ w w b] x x = wx + wx + b = 6 Example here are an nfnte number of equaton coeffcents that wll satsfy ths relaton Draw a lne between the two regons, and then choose w s for coeffcents that are perpendcular to ths lne. Fnally, solve for b One soluton s [ 3] 63 Perceptron Learnng Rule x( n) = ( m + ) by nput vector = [, x ( n),..., x ( n)] w( n) = ( m + ) by weght vector = [ bn ( ), w( n),..., w ( n)] m b( n) = bas y( n) = output ( quantzed to ± ) = sgn( w x) d( n) = desred response η = learnng rate postve cons tant < m 64 Perceptron Learnng Rule. Intalze w()=. Input x(n) and compute y(n) 3. Update w as w(n+)=w(n) + η [d(n)-y(n)]x(n) Repeat steps and 3 untl no more weght changes 65 Heurstc Improvements Sequental vs Batch mode update Use sequental smpler, faster, requres less temporary storage Maxmze nformaton content Examples should contan maxmum nformaton Results n the largest tranng error Radcally dfferent from prevous examples 66

12 Heurstc Improvements Maxmze nformaton content (cont d) Always re-randomze tranng patterns For tranng patterns choose ones that are dffcult to recognze Potental problem exsts f dffcult pattern s actually an outler. Heurstc Improvements Actvaton Functon Generally learns faster f t s antsymmetrc ϕ( v) = ϕ( v) Not true of log-sgmod, but true of anh Heurstc Improvements Actvaton Functon (cont d) he next fgure s the log sgmod whch does not meet the crteron he fgure after that s the anh whch s antsymmetrc 69 7 Heurstc Improvements Actvaton Functon (cont d) For the anh functon, emprcal studes have shown the followng values for a and b to be approprate ϕ( v) = a tanh( bv) a = 759. b = 3 7 7

13 Heurstc Improvements Actvaton Functon (cont d) Note that ϕ() = and ϕ( ) = slope s max mum at v = ( t' s unty) Heurstc Improvements arget values Choose wthn range of output values Really should be some value from the maxmum For the tanh, wth a=.759, choose =.759, and then the targets can be +/ Heurstc Improvements Normalzng the nputs Preprocess nputs so that average over range s ~ or t s small compared to ts standard devaton Input values should be scaled so that ther covarances are approxmately equal ensures that weghts learn at about same rate If possble, nput values should be uncorrelated Covarance & Correlaton [ µ x µ y ] cov( XY, ) = E( x )( y ) cov( XY, ) corr( X, Y) = ρ = VAR( X ) VAR( Y) Correlaton & Covarance Note that for a postve correlaton t means as X ncreases, so does y, and vce versa Of the followng plots whch has the hghest covarance? 77 78

14 Heurstc Improvements Intalzaton t can be shown (approxmately, under certan condtons) that a good choce for weghts s to select them randomly from a dstrbuton wth µ = and σ = m = total number of weghts m Feature Detecton Hdden neurons play the role of feature detectors end to transform the nput vector space nto a hdden or feature space Each hdden neuron s output s then a measure of how well that feature s present n the current nput 79 8 Generalzaton Network generalzes well when for nontraned-on-data produces correct (or near correct) outputs. Can overft or overtran Generally want to select smoothest/smplest mappng of functon n absence of pror knowledge Generalzaton Influenced by four factors Sze of tranng set How representatve tranng set s of data Neural network archtecture Physcal complexty of problem at hand Often NN confguraton or tranng set fxed and so have only other to work wth 8 8 Generalzaton A commonly used value s N=O(W/ ) where O => s lke Bg-O, W= total number of weghts, =fracton of classfcaton errors permtted on test data. 83 Approxmatons of Functons NN acts as a non-lnear mappng from nput to output space Everywhere dfferentable f all transfer functons are dfferentable What s the mnmum number of hdden layers n a multlayer perceptron wth an I/O mappng that provdes an approxmate mappng of any contnuous mappng 84

15 Approxmatons of Functons Part of unversal approxmaton theorem hs theorem states (n essence) that a NN wth bounded, nonconstant, monotone ncreasng contnuous transfer functons and one hdden layer can approxmate any functon Says nothng about optmum n terms of learnng tme, ease of mplementaton, or generalzaton Approxmatons of Functons In general, for good generalzaton, the number of tranng samples N should be larger than the rato of the total number of free parameters (weghts) n the network to the mean-square value of the estmaton error Practcal Consderatons For hgh dmensonal spaces, t s often better to have -layer networks so that neurons n layers do not nteract so much. Cross Valdaton Randomly dvde data nto tranng and testng sets Further randomly dvde tranng set nto estmaton and valdaton subsets. Use valdaton set to test accuracy of model, and then test set for actual accuracy value Cross Valdaton leave-one-out ran on everythng but one and then test on t Repeat for all parttons RBF We have seen that a BPNN (backpropagaton neural network) can be used for functon approxmaton and classfcaton A RBFNN (radal bass functon neural network) s another network that can be used for both such problems 89 9

16 RBFNN vs BPNN RBFNN vs BPNN RBFNN has only two layers, whereas BPNN can have any number Normally all nodes of the BPNN have the same model, whereas the RBFNN has two dfferent models RBFNN has a nonlnear and a lnear layer Argument of RBFNN computes a Eucldean norm whereas a BPNN computes a dot product RBFNN trans faster than BPNN 9 RBFNN often leads to better decson boundares Hdden layer unts of RBFNN have a much more natural nterpretaton RBFNN learnng phase may be unsupervsed and thus could lose nformaton BPNN may gve a more compact representaton RBF here are general categores of RBFNN s Classfcaton separaton of hyperspace frst porton of chapter Functon approxmaton use a RBFNN to approxmate some non-lnear functon RBF-Classfcaton Cover s heorem separablty of patterns A complex pattern classfcaton problem re-cast n a hgh(er)- dmensonal space nonlnearly s more lkely to be lnearly separable than n a low-dmenson space 95 96

17 Cover s heorem Cover s heorem - Corollary Gven an nput vector X=[x, x k ] ( of dmenson k), f we recast t usng some set of nonlnear transfer functons on each of the nput parameters (of dmenson m>= k) then t s more lkely that ths new set wll be lnearly separable In some cases smply usng a nonlnear mappng and not changng (ncreasng) the dmensonalty s suffcent 97 he expected maxmum number of randomly assgned patterns (vectors) that are lnearly separable n a space of dmensonalty m s m Stated another way: m s a defnton of the separatng capacty of a famly of decson surfaces havng m degrees of freedom 98 XOR Problem In the XOR problem x x we want a output when the nputs are not equal, else output a. hs s not a lnearly separable problem Observe what happens f we use two non-lnear functons appled to x and x ϕ ( X ) = e XOR [, ] = [,] ϕ (,) = e = X (( ) + ( ) ϕ ( X ) = e ) = e X = XOR ϕ Input X (X) ϕ (X) (,).353. XOR If one plots these new ponts (next slde), they can see that they are now lnearly separable (,) (,) (,)

18 (,) XOR (,) (,) (,) What the prevous sldes show s that by ntroducng a nonlnearty, even wthout ncreasng the dmensonalty, the space becomes lnearly separable 3 4 Regularzaton An mportant pont to remember s that we always have nose n the data, and thus t s probably not a good dea to develop a functon that exactly fts the data. Another way to thnk of t s that fttng the data too closely wll lkely gve poor generalzaton he next slde s bad, and the one after that s good

19 9 Interpolaton here s sort of a converse to Cover s theorem about separablty. It s One can often use a nonlnear mappng to transform a dffcult flterng (regresson) problem nto one that can be solved lnearly. More Regresson/Interpolaton In a sense, the regresson problem bulds an equaton to gve us the nput/output relatonshp on a set of data. Gven one of the nputs we traned, we should be able to use ths equaton to get the output wthn some error he nterpolaton problem addresses the ssue of what value(s) do we get for nputs that we have not traned on RBF Regresson he RBF approach to regresson chooses F such that N F( X) = wϕ( X X ) = where ϕ( X X ) s a set of N arbtrary ( generally nonlnear) functons known as radal bass functons and denotes a norm usually Eucldean he X are the centers of the RBF' s RBF he radal Bass Functons most commonly used are N=# of data ponts F( x) = N = we σ x x 3 4

20 RBF hus, generally we accept a suboptmal soluton n whch the number of bass functons s <N, and thus the number of centers of bass functons s < N Classfcaton RBF s can also be used as classfers. Remember BPNN s tend to dvde the nput space (fgure a next slde) whereas RBF s tend to kernelze the space (fgure b next slde) 5 6 (a) BPNN (b) RBFNN Classfcaton One can nterpret the RBF s as posteror probabltes of the presence of a data pont n the nput space, and weghts can be vewed as posteror probabltes of class membershp We therefore have a -layer organzaton for a RBFNN used as a classfer

21 Learnng Strateges ypcally, the weghts of the two layers are determned separately,.e. fnd RBF weghts, and then fnd output layer weghts here are several methods for parameterzng the RBF s and selectng the output layer weghts Fxed Centers Randomly select from the tranng data some set of representatve x s and use these as the centers (the t s) m ϕ( x) = exp( x t ) =,,..., m d max m = number of centers, and d max s the maxmum ds tance between centers he effectve s tan dard devaton of these RBF' s s d max σ = m Fxed Centers From ths, what parameters are left to fnd? Fxed Centers From ths, what parameters are left to fnd? he w s of the output layer For ths, we need a matrx w such that w = ϕ X where X s the matrx of tranng data 3 4 Fxed Centers he problem wth the precedng s that ϕ s probably not square, and hence can t be nverted. o do ths we use the pseudo-nverse, and thus + w = ϕ X + ϕ = ( ϕ ϕ) ϕ ϕ = where { ϕ j } Fxed Centers m ϕ j = exp( xj t ) j =,,..., N =,,.., m d x s the jth nput vector of the tranng sample j max Example gven on page 84 of text 5 6

22 Pseudo-Inverse In order to nvert a matrx, t must meet certan crtera. One of those crtera s that t be square he Pseudo-nverse algorthm allows us to convert a non-square matrx nto an equvalent square matrx Pseudo-Inverse Remember that ultmately for a neural network we want W (a weght matrx) such that arget = WX hus W= X - If an nverse exsts, then the error can be mnmzed If no nverse exsts, then use the pseudo-nverse to get mnmum error 7 8 Pseudo-Inverse he pseudo-nverse s defned as W=X + Where X + = (X X) - X Example X =, t = [ ], X =, t = [ ] X X = = = X X X = ( ) = Self-Organzed Center Selecton Fxed centers may requre a relatvely large number of randomly selected centers to work well hs method (self-organzed) uses a two stage process (teratve) Self-organzed learnng stage estmate RBF centers Supervsed Learnng estmate lnear weghts Self-Organzed Learnng Stage For ths stage we need a clusterng algorthm that dvdes or parttons the data ponts nto subgroups Commonly use K-means clusterng Place RBF centers only n those regons of the data space where there are sgnfcant data of output layer 3 3

23 Self-Organzed Learnng Stage Self-Organzed Learnng Stage { } t ( n) denotes the centers of the RBF' s k m k = at teraton n 33 Intalzaton choose random values for the ntal set of centers t k (); subject to the restrcton that these ntal values must all be dfferent Randomly draw a sample vector from the nput space Smlarty matchng Let k(x) denote the best matchng center for nput x (closest) 34 Self-Organzed Learnng Stage Fnd k(x) at teraton n usng the mnmum dstance Eucldean crteron: k( x) = arg mn x( n) t ( n) k =,,..., m k where t k ( n) s the center of the kth radal bass functon at teraton n k Self-Organzed Learnng Stage Updatng Adjust the center of the closest RBF center as t ( n + ) = t ( n) + η[ x( n) t ( n)] < η < k k k Contnuaton Repeat untl no notceable changes n centers occur Generally, reduce the learnng rate over tme K-means Clusterng he prevous s dependent on the selecton of the ntal centers. Once the centers are found, must stll set and output layer weghts Adaptve K-means wth dynamc ntalzaton Randomly pck a set of centers c,..ck. Dsable all cluster center. Read an nput vector X 3. If the closest enabled cluster center c s wthn dstance r of X or f all cluster centers are already enabled, update c as η c = c + (X-c) 37 38

24 Adaptve K-means wth dynamc ntalzaton - Contnued 4. Otherwse, enable a new cluster ck and set t equal to X 5. Contnue untl a fxed number of teratons, or untl learnng rate has decayed to Next Step he next step s to set the wdths of the centers σ k How? 39 4 Next Step Can use a P nearest neghbor heurstc Gven a cluster center c k, select the P nearest neghborng clusters σ k Set to the root mean square of the dstances to these clusters Next Step σ k = c P k c p p 4 4 Fnally he last step s to set the weghts of the output layer Snce t s lnear, just use the delta tranng rule Note also, that there s no real requrement that the output layer be lnear, e.g. could be a BPNN 43 LVQ Learnng Vector Quantzaton Network Frst layer s compettve Output layer selects class (/) from output of neurons n compettve layer Output of a neuron s a f t s closest to the nput vector 44

25 LVQ For closeness, can we just use the dot product? LVQ For closeness, can we just use the dot product? Need some other dstance measure because dot product s nfluenced by vector magntude as much as angle (closeness) LVQ Input s P = arget or class s = LVQ For an nput P, want the output of the K competetve neurons to be all s except for one of them In the output layer there wll be one neuron wth -/ as weghts. hs neuron wll be hardlm neuron so ts output wll be / for the two classes SVM SVMs ntroduced n 99 by Boser, Guyon,Vapnk. Kernal machnes (KM) are a more general class of learnng machnes, SVM s are a subclass of KM s Kernal methods explot nformaton about the nner (dot, scalar) product between data tems 49 SVM Kernel generally s vewed as the set of features about a decson, e.g. defnes the vector space. Very complex decson problems can be defned n terms of dot products just have to recast (probably non-lnearly) nto a probably hgher dmenson space Remember Cover s heorem 5

26 SVM Bascally a lnear machne wth some nce propertes Means for non-lnearly separable applcatons need to apply cover s theorem apply non-lnear transformatons and possbly put n hgher dmensonal space SVM s more general than say MLBP learnng algorthm 5 5 Dot Product Remember, the functon WX + b = s SVM s he learnng algorthm for SVM s s dfferent from say BPNN s BPNN s seek to mnmze the error n the tranng data (emprcal rsk mnmzaton ERM) SVM s seek to mnmze what s called the structural rsk measure (SRM),.e. the error (rsk) n the testng data SRM places an upper bound on the generalzaton error.e. they generalze better SVM s - Classfcaton he tranng of a classfcaton SVM nvolves the fttng of a hyperplane such that the largest margn s formed between classes of vectors whle mnmzng the effects of classfcaton errors he vectors closest to the hyperplane are called support vectors 55 56

27 he optmal hyperplane s orthogonal to the shortest lne connectng the convex hulls of the two classes (dotted), and ntersects t half way. here s a weght vector w and a threshold b such that y ((w x ) + b) >. Rescalng w and b such that the pont(s) closest to the hyperplane satsfy (w x ) + b =, we obtan a form (w, b) of the hyperplane wth y ((w x ) + b) >=. Note that the margn, measured perpendcularly to the hyperplane, equals / w. o maxmze the margn, we thus have to mnmze w subject to y ((w x ) + b) = SVM he separaton (dstance) between the hyperplane and the closest data pont(s) s called the margn of separaton ρ he goal of the SVM s to fnd the hyperplane that maxmzes ρ When ρs maxmzed, ths decson surface s sad to be the optmal hyperplane. 59 Hyperplanes Remember that n an n-dmensonal space, a decson hyperplane s defned as [ bw,, w,... wn ] [,,... ] n. = W X + b = w w w OR = x x.. xn x x + b. xn 6 Hyperplane If d =+/-(d=desred output) then we have W X + b for d = + W X + b < for d = If w and b denote the optmum values of the weght and bas then we have W X + b = For X on the hyperplane 6 Hyperplane he dscrmnant functon then becomes g( X) = W X + b hs gves a measure of the dstance from X to the optmal hyperplane Another nterpretaton of X n terms of these parameters s X X r W = ρ + W 6

28 Hyperplanes g X W X b W X r W ( ) = + = ( p + ) + b W W X r W W = p + + b W because g( X ) = W X + b =, we have p p g X r W W r W ( ) = = = rw or r= W W g( X) W Hyperplanes he dstance from the orgn to the hyperplane s b / w. If b > the orgn s on the postve sde of the hyperplane he problem s to fnd the values for b and w from the tranng set {x, d } Hyperplanes By scalng b and w we can set the dscrmnants so that W X + b for d = + W X + b < for d = he data ponts {X, d } for whch the equaltes hold are called the support vectors, e.g. s s g( X ) = W X + b = Hyperplanes he support vectors le closest to the decson surface and thus they are the most dffcult to classfy For these support vectors, ther dstance to the hyperplane s s g( X ) r = W W r = W f d f d s s = + =

29 Matlab In Matlab Under the pull-down for help select Help->Matlab Help hs wll gve you a lst of topcs under whch you can read tutorals Clck on the Neural Network oolbox and expand t to gve the topcs ranng several dfferent functons tranb trans a network wth weght and bas learnng rules wth batch updates. he weghts and bases are updated at the end of an entre pass through the nput data. tranr trans a network wth weght and bas learnng rules wth ncremental updates after each presentaton of an nput. Inputs are presented n random order ranng 4 dfferent functons trans rans a network wth weght and bas learnng rules wth ncremental updates after each presentaton of an nput. Inputs are presented n sequental order. Perceptrons Remember lnearly separable Perceptron tranng rule If lnearly separable, and the learnng rate not too hgh, wll always fnd a soluton What does fnd a soluton mean? 7 7 Demop Demop Demop4 Demop5 Demop6 Perceptrons 73 Perceptrons Four functons are mportant for perceptron networks Newp creates a new perceptron network Int ntalzes a perceptron network Sm smulates (executes) a perceptrton network wth a set of values (nput & target) ran trans a network Updatedoes one tranng teraton 74

30 Perceptrons - newp Newp s for the creaton of a perceptron based neural network net=newp(pr,s) PR=> Rx matrx of max and mn values of nput elements s=> number of neurons Perceptrons - newp net=newp([ ],); one neuron wth a sngle nput. Range of values for the nput s to net=newp([,;-,],); two neurons each wth a two nputs. Range of values for the nputs are to, and - to respectvely Perceptrons - newp net=newp([,;-,],); net.b{}=[;]; sets bases to and net.b{} gves values of bases Perceptrons newp, ntalze net=newp([- ; 3,5],) weghts & bases ntalzed to net.iw{,}=[- ;,] net.b{}=[;] Perceptrons - sm Sm runs through the nput data and outputs the value acqured by the network for each nput he nput can be a sngle nput vector or multple n whch case t outputs for each Perceptrons - sm p={[;] [;] [-;3]}; sm(net,p) 79 8

31 Perceptrons - nt Intalzes or rentalzes network Int(net) sets parameters back to orgnal Perceptrons - nt Can change the way that a perceptron s ntalzed wth nt. net.bases{}.ntfcn = 'rands'; net = nt(net); wts = bases = Perceptrons - tran net = newp([- ;- +],); p =[; ]; t=[]; net.tranparam.epochs = 5; net = tran(net,p,t); RAINC, Epoch /5 RAINC, Epoch /5 RAINC, Performance goal met. MatLab GUI Matlab has a very powerful GUI for creatng and manpulatng networks nntool brngs t up It s explaned under matlab help->neural network toolbox->perceptrons->graphcal user nterface BP - GUI Create the nput and the target data Inputdata=[ 3; -] hs means there are 3 data values to tran on and they are (,),(,), and (3,-) arget=[ 3] hus for an nput of (,) we want an output of, etc. 85