Multilayer Feed-forward networks

Transcription

1 Multi Feed-forward networks 1. Computational models of McCulloch and Pitts proposed a binary threshold unit as a computational model for artificial neuron. This first type of neuron has been generalized in many ways. We present below the most common model: x 1 x 2 w 1 w 2 S y x n w n f(s) Figure 1 - The artificial model of neuron where x 1,..., x n represents the input signals (from the previous neuron or the network input); w 1,..., w n represents synapses (weights) associated to corresponded inputs; Ө - is the value of the neuron activation threshold (the offset); S - signal output neuron; f (S) - activation function; y the output (response) of neuron. This mathematical neuron computes a weighted sum of its input signals adds a value called the threshold and generates an output if this sum is above a certain threshold. S n i1 x i w i y f ( S) (1) In the previous equations x i represent the input signals, w i the weight associated with this entry. The term represent a threshold value (the offset) to move activation argument of function f. Different models are obtained by using various mathematical functions for f. Three common choices are shown in the following figure: step function, sign function and sigmoid function 1 of 5

2 t (a) Step function (b) Sign function (c) Sigmoid function 1, if step i ( x) 0, if x t x t 1, if x 0 1 sign ( x) sigmoid( x) x 1, if x 0 1 e 2. Error correction rules There are several rules of learning: supervised learning (learning with a teacher ) where to the network is provided a correct answer (output) for every input pattern; and unsupervised learning (or learning without a teacher) where the network does not require a correct answer associated with each input pattern in the training data. In the supervised learning paradigm, to the network is given a desired output for each input pattern. During the learning process, the actual output y generated by the network may be not equal the desired output d. The basic principle of error-correction learning rules is to use the error signal (d-y) to modify the connection weights to gradually reduce this error. Most general relationship for changes in synaptic coefficient w (weight) under this rule is (also called gradient trend in the opposite direction) E w (2) w where E is the global error (dependent on weights ) and is the learning rate (step size made in the gradient direction). This relationship is the basic principle of learning feed-forward multi networks. The basic ides to use the slope gradient to search in the space of possible hypotheses of weight vectors to find those weights that best approximate the training examples. This rule is important because it provides the basic principle of Back-propagation algorithm and can create and learn networks with many interconnected units. It is also important because the slope gradient can serve as a basis for learning algorithms that need to search in the hypothesis space, containing many different types of parameterized hypothesis. The slope gradient trying to determine the weight vector, which minimizes error function, starting with an arbitrary initial weight vector, which is modified in small steps repeatedly. At each step, weight vector is modified in the direction that produces a descending slope in surface of error function. This process continues until the global minimum error is reached. 3. Online / Offline Learning Feed-forward networks (with training based on the then error-correction principle) always will learn association between multiple input and output vectors. In this case the total error function is the sum of error functions corresponding to individual pairs input / output. This error can be minimized in two ways: 1 off-line is determined, for each pair of input / output, changes to the synaptic coefficients. These changes are applied only after all pairs from training data are presented. 2 on-line - change coefficients, computed for a pair of input/output, is applied immediately after presentation of the pair. The benefit of this method is that is generally faster and can leave some of the local minimum of error function. 2 of 5

3 4. Feed-forward Network with 3 s Input Hidden Output N1 N2 N3 Figure 2 The Network Architecture The Back-propagation algorithm learns the weights for a network on several levels, posing a fixed network with a lot of units and interconnections between them. Slope gradient is involved to try to minimize the squared error between network output value and target value for those outputs. Backpropagation learning problem is to search in the large space of hypotheses defined by all possible values of the weights for all units in the network. The algorithm described here for a feed-forward network containing two s of units with sigmoid activation function in the general case (Figure 2). Each unit of each is connected with all units on the previous. Input units are considered repeating unit having received valuable input to output. Are also presented formulas for the network so as to step forward and step backward. For backward step has been taken into account the error calculation formula shown in equation (5). The back-propagation rule presented here uses the following notations: W12[[i] the weights of connections between from hidden (h) and from input (i) Ө2[ - the threshold of from the hidden (h) W23[[ the weights of connections between from output (o) and from the hidden (h) Ө3[ - the threshold of from the (o) output Out1[i] the output value of from the input (i) the output value of from hidden (h) the output value of from the output (o) (the Scop [s] is the desired output value) F(.) Activation function for all N1, N2, N3 represent the number of nput, hidden and respectively output. The Forward Step N1 F( W12[[i]* Out1[i] 2[) (3) i=1 3 of 5

4 N2 F( W23[[* 3[) (4) N3 h=1 o E Out3o Goal (5) o=1 The Backward step 2 w * E w (6) where w can be Ө3[, W23[[, Ө2[, W12[[i] E 2*( Goal[ ) 3[ E 2*( Goal[ W 23[ [ E 2* 2[ N 3 o1 ( Goal[ W 23[ [ * ) (7) (8) (9) E 2* W12[ [ i] N 3 o1 ( Goal[ W 23[ [ * )* Out1[ i] If we consider the function F(.) as been sigmoid function, the value of derived function is easily determined according to the following equation: F ( x ) F ( x ) 1 F ( x ) 5. The Back-propagation algorithm: Each example from the training set is a pair of form value (the desired output of the network). x, y (10) (11), where x is an input vector and y is the target Will create a feed-forward network with: n nput units (depending on the size of the input vector), n hidden hidden units, and n out output units (depending on the size of the target vector). Number of units of hidden is chosen depending on the complexity of training data. 1. Initialize the weights to small random values; 2. Take an input sample; 3. Propagate the signal "Forward" through the network. Compute the output of all units from each (input, hidden and output ). The output of unit from the first (input ) is equal to their input. The output of from the hidden is computed according to equation (3); The output of from the output is computed according to equation (4); 4. Compute the error produced by the output of network using equation (5); 4 of 5

5 5. Compute the deltas for the preceding s by propagating the errors backwards in order to minimize the error produced by the current sample. Thus for each network the weights is computed by formula (6) the amount that should be applied so that the weight error is minimal; 6. Weights are updated accordingly equation (6) depending on how type of learning is used (online or offline). 7. Go to step 2 and repeat for the next pattern (sample) until the error in the output is bellow a pre-specified threshold (enough close to 0, recommended ) or a maximum number of iterations is reached. Problem: Create the architecture and implement the Back-propagation algorithm to learn the XOR logic function using both the online and offline learning rule. Because the sigmoid function reaches the value 0 to - and the value 1 to +, it is recommended, for speed convergence, to use the value 0.1 instead of 0 and the value 0.9 instead of 1. 5 of 5