Chapter 8: Adaptive Networks

Transcription

1 Chater : Adative Networks Introduction (.1) Architecture (.2) Backroagation for Feedforward Networks (.3) Jyh-Shing Roger Jang et al., Neuro-Fuzzy and Soft Comuting: A Comutational Aroach to Learning and Machine Intelligence, First Edition, Prentice Hall, 17 Introduction (.1) Almost all kinds of Neural Networks aradigms with suervised learning caabilities are unified through adative networks Nodes are rocess units. Causal relationshis between the connected nodes are eressed with links All or art of the nodes are adative, which means that the oututs of these nodes are driven by modifiable arameters ertaining to these nodes Ch. : Adative Networks 1

2 Introduction (.1) (cont.) A learning rule elains how these arameters (or weights) should be udated to minimize a redefined error measure The error measure comutes the discreancy between the network s actual outut and a desired outut The steeest descent method is used as a basic learning rule. It is also called backroagation Architecture (.2) Definition: An adative network is a network structure whose overall inut-outut behavior is determined by a collection of modifiable arameters The configuration of an adative network is comosed of a set of nodes connected by direct links Each node erforms a static node function on its incoming signals to generate a single node outut Each link secifies the direction of signal flow from one node to another Ch. : Adative Networks 2

3 A feedforward adative network in layered reresentation Eamle 1 : Parameter sharing in adotive network A single node Parameter sharing roblem Ch. : Adative Networks 3

4 Architecture (.2) cont.) There are 2 tyes of adotive networks Feedforward (acyclic directed grah such as in fig..1) Recurrent (contains at least one directed cycle) A recurrent adative network Ch. : Adative Networks 4

5 Architecture (.2) (cont.) Static Maing A feedforward adative network is a static maing between its inuts and outut saces This maing may be linear or highly nonlinear Our goal is to construct a network for achieving a desired nonlinear maing Architecture (.2) (cont.) Static Maing (cont.) This nonlinear maing is regulated by a data set consisting of desired inut-outut airs of a target system to be modeled: this data set is called training data set The rocedures that adjust the arameters to imrove the network s erformance are called the learning rules Ch. : Adative Networks 5

6 Architecture (.2) (cont.) Static Maing (cont.) Eamle 2: An adative network with a single linear node 3 f 3 ( 1, 2 ; a 1, a 2, a 3 ) a 1 1 a 2 2 a 3 where: 1, 2 are inuts; a 1, a 2 and a 3 are modifiable arameters A linear single-node adative network Ch. : Adative Networks 6

7 Architecture (.2) (cont.) Static Maing (cont.) Identification of arameters can be erformed through the linear least-squares estimation method of chater 5 Eamle 3: Percetion network 3 f 3 ( 1, 2 ; a 1, a 2, a 3 ) a 1 1 a 2 2 a 3 and 4 f 4 ( 1 ) 0 if if (f 4 is called a ste function) < 0 A nonlinear single-node adative network Ch. : Adative Networks 7

8 Architecture (.2) (cont.) The ste function is discontinuous at one oint (origine) and flat at all other oints, it is not suitable for derivative based learning rocedures use of sigmoidal function Sigmoidal function has values between 0 and 1 and is eressed as: 1 4 f4(3 ) 1 e 3 ( ) Sigmoidal function is the building-block of the multi-layer ercetron Architecture (.2) (cont.) Eamle 4: A multilayer ercetron 7 1 e [ ( w w w t )] 4,7 4 where 4, 5 and 6 are oututs from nodes 4, 5 and 6 resectively and the set of arameters of node 7 is {w 4,7, w 5,7, w 6,7, t 7 } 1 5,7 5 6,7 6 7 Ch. : Adative Networks

9 A neural network Backroagation for Feedforward Networks (.3) Basic learning rule for adative networks It is a steeest descent-based method discussed in chater 6 It is a recursive comutation of the gradient vector in which each element is the derivative of an error measure with resect to a arameter The rocedure of finding a gradient vector in a network structure is referred to as backroagation because the gradient vector is calculated in the direction oosite to the flow of the outut of each node Ch. : Adative Networks

10 Backroagation for Feedforward Networks (.3) (cont.) Once the gradient is comuted, regression techniques are used to udate arameters (weights, links) Notations: Assume we have L layers (l 0, 1,, L 1) N(l) reresents the number of nodes in layer l l,i reresents the outut of node i in layer l (i 1,, N(l)) f l,i reresents the function of node i Backroagation for Feedforward Networks (.3) (cont.) Princile Since the outut of a node deends on the incoming signals and the arameter set of the node, we can write: l,i fl,i (l 1,1,l 1,2,..., l 1,N(l 1), α, β, γ,...) where α, β, γ,, are the arameters of this node. Ch. : Adative Networks 10

11 A layered reresentation Backroagation for Feedforward Networks (.3) (cont.) Princile (cont.) Let assume that the training set has P atterns, therefore we define an error for the th attern as: E N(L) ( dk L,k ) k 1 Where: d k is the k-th comonent of the th desired outut vector and L,K is the k-th comonent of the redicted outut vector roduced by resenting the th inut vector to the network The task is to minimize an overall measure defined as: P E E 1 2 Ch. : Adative Networks 11

12 Backroagation for Feedforward Networks (.3) (cont.) Princile (cont.) To use the steeest descent to minimize E, we have to comute E (gradient vector) Causal Relationshis Change in arameter α Change in oututs of nodes containing α Change in network s outut Change in error measure Backroagation for Feedforward Networks (.3) (cont.) Princile (cont.) Therefore, the basic concet in calculating the gradient vector is to ass a form of derivative information starting from the outut layer and going backward layer by layer until the inut layer is attained Let s define as the error signal on ε l,i E the node i in layer l (ordered derivative) l,i Ch. : Adative Networks 12

13 Backroagation for Feedforward Networks (.3) (cont.) Princile (cont.) Eamle: Ordered derivatives and ordering artial derivatives Consider the following adative network Ordered derivatives & ordinary artial derivatives z g(, y) y f() Ch. : Adative Networks 13

14 Backroagation for Feedforward Networks (.3) (cont.) For the ordinary artial derivative, z g(, y) and y are assumed indeendent without aying attention that y f() For the ordered derivative, we take the indirect causal relationshi into account, z g(,f()) g(, y) g(, y) y f () y y The gradient vector is defined as the derivative of the error measure with resect to the arameter variables. If α is a arameter of the ith node at layer l, we can write: E α E l,i f * α l,i ε l,i f α l,i f () Where ε l,i is comuted through The indirect causal relationshi f * Backroagation for Feedforward Networks (.3) (cont.) The derivative of the overall error measure E with resect to α is: E α P 1 E α Using a steeest descent scheme, the udate for α is: E αnet αnow η α Ch. : Adative Networks 14

15 Backroagation for Feedforward Networks (cont.) Eamle.6 Adative network and its error roagation model In order to calculate the error signals at internal nodes, an error-roagation network is built Error roagation model Ch. : Adative Networks 15

16 Ch. : Adative Networks 16 Backroagation for Feedforward Networks (.3) (cont.) { { w,7 7 w, f f f * E f * E E ) 2(d E E ) 2(d E E ε ε ε ε ε Backroagation for Feedforward Networks (.3) (cont.) There are 2 tyes of learning algorithms: Batch learning (off-line learning): the udate for α takes lace after the whole training data set has been resented (one eoch) On-line learning (attern-by-attern learning): the arameters α are udated immediately after each inut outut air has been resented