Gdansk University of Technology Faculty of Electrical and Control Engineering Department of Control Systems Engineering

Transcription

1 Gdansk University of Technology Faculty of Electrical and Control Engineering Department of Control Systems Engineering Artificial Intelligence Methods Neuron, neural layer, neural netorks - surface of the neural netork response Laboratory exercises no T1 Auxiliary material for laboratory exercises Authors: Anna Kobylarz, mgr inż. Kazimierz Duzinkieicz, dr hab. inż. Michał Grochoski, dr inż. Gdańsk, 2015

2 The material contains selected parts of the lecture and additional information in order to create artificial neural netorks structures in the MATLAB environment using the Neural Netork Toolbox. In particular, it contains a description of the basic elements of neuron, activation functions, a description of the main instructions available in the MATLAB s toolbox and an example of creating and learning perceptron neural netork. Artificial neural netorks - a single neuron A neural netork is characterized by: 1. Functions according to hich a neuron responds to the inputs, called excitation functions and activation functions; 2. The structure of the connections beteen neurons, called netork architecture; 3. The method of determining the eights of those connections, called the learning algorithm. p 1 Weights 1,1 Excitation (propagation) function Activation function Response signal Input signals p 2. p j. 1,2. 1,j. g() b s Threshold n s a s f() Output Excitation signal p R 1,R Fig. 1. Artificial neuron model scheme. Where: R inputs number, S number of neurons in a layer, Fig. 2 shos ho a neuron is presented in the MATLAB documentation along ith the accepted indications of inputs, eights, threshold and output. Fig. 2. Scheme of a neuron ith a single input. Symbols and notation from MATLAB. 2

3 For this case, the dimensions of the matrixes describing the neuron are as follos: In cases here multiple signals on the input are given, the neuron and its scheme is as follos: p p; W ; b b ; a a Fig. 3. Scheme of neuron ith multiple (R) inputs. Where: R number of inputs, Where: R number of inputs, Fig. 4. Scheme of neuron ith multiple (R) inputs. Symbols and notation from MATLAB. The dimensions of matrixes describing this neuron are as follos: p p p 1 R 2 p ; R W ; b b ; a a 1, 2, Activation (transfer) functions of the neuron 3

4 In the Figs 5-7 examples of activation functions are shon, respectively step, linear and sigmoidal logistic (log-sigmoid) function. All instructions available in the toolbox are given in Table 1. Fig. 5. Hard limit activation function. Where: p input signal to neuron, n excitation signal of neuron, a output signal from neuron, eight value of neuron s input and b threshold value. Fig. 6. Linear activation function. Fig. 7. Log-sigmoid activation function. 4

5 Table 1. Activation (transfer) functions available in MATLAB s Neural Netork Toolbox. In Fig. 8 an example of a neuron ith a symmetrical hard limit activation function (perceptron) and to inputs is shon. Fig. 8. Neuron ith symmetrical hard limit activation function and to inputs perceptron. 5

6 Matrixes dimmensions for this neuron are as follos: p1 p ; 1,1 1,2 b p2 Artificial neural netorks neural layer W ; b ; a a Figure 9 and 10 sho a single neural netork layer ith a description of its parameters. Fig. 9. Neural netork layer scheme. Where: R inputs number, S number of neurons in layer, Fig. 10. Neuron ith hard limit activation function perceptron. Symbols and notation from MATLAB. 6

7 7 Matrixes dimmensions for neural layer are as follos: R 2 1 p p p p S,R S,2 S,1 2,R 2,2 2,1 1,R 1,2 1,1 W S 2 1 b b b b S 2 1 a a a a Artificial neural netorks multi-layer netork Figure 11 and 12 sho a multi-layer (three-layer) feedforard neural netork. As can be seen, the outputs of each layer are the inputs of another of the layers. Where: R number of inputs, s 1 - number of neurons in the first layer, s 2 - number of neurons in the second layer, s 3 - number of neurons in the third layer. Fig. 11. Scheme of multi-layer feedforard neural netork.

8 Where: R number of inputs, s 1 - number of neurons in the first layer, s 2 - number of neurons in the second layer, s 3 - number of neurons in the third layer. Fig. 12. Scheme of multi-layer feedforard neural netork. Symbols and notation from MATLAB. 8

9 Basic commands of MATLAB Neural Netork Toolbox ver. 8.2 Belo in Table 2. the important commands from the MATLAB s Neural Netork are shon. More commands can be obtained by typing help nnet, hile the details of a specific command, such as: the use of syntax, the algorithms, the application s example etc., can be obtained by typing help command ith any instruction, e.g. help nep. Table 2. MATLAB s Neural Netork Toolbox ver 8.2 commands. Command Short description Creating a netork netork Create a custom neural netork. nec Create a competitive layer. necf Create a cascade-forard backpropagation netork. neff Create a feed-forard backpropagation netork. nefftd Create a feed-forard input time-delay backpropagation netork. nelin Create a linear layer. nelind Design a linear layer. nep Create a perceptron. Net input (excitation) functions netprod Product net input function netsum Sum net input function Functions initializing netork parameters initlay Layer-by-layer netork initialization function Functions describing quality of netork s ork mae Mean absolute error performance function mse Mean squared normalized error performance function msereg Mean squared error ith regularization performance function. sse Sum squared error performance function Learning methods learncon Conscience bias learning function learngd Gradient descent eight and bias learning function learngdm Gradient descent ith momentum eight and bias learning function learnis Instar eight learning function learnlv1 LVQ1 eight learning function learnlv2 LVQ2 eight learning function learnos Outstar eight learning function learnp Perceptron eight and bias learning function learnpn Normalized perceptron eight and bias learning function learnh Widro-Hoff eight/bias learning function Processing of input and output data prestd Preprocesses the data so that the mean is 0 and the standard deviation is 1. poststd Postprocesses data hich has been preprocessed by PRESTD. trastd Preprocesses data using a precalculated mean and standard deviation. premnmx Preprocesses data so that minimum is -1 and maximum is 1. postmnmx Postprocesses data hich has been preprocessed by PREMNMX. tramnmx Transform data using a precalculated min and max. prepca Principal component analysis. trapca Principal component transformation. postreg Postprocesses the trained netork response ith a linear regression. 9

10 trainb trainbfg trainbr trainc traincgb traincgf traincgp traingd traingdm traingda traingdx trainlm trainoss trainr trainrp trainscg compet hardlim hardlims logsig poslin purelin radbas satlin satlins tansig errsurf maxlinlr initn initb cell2mat concur con2seq combvec mat2cell minmax nncopy normc normr seq2con sumsqr sim init adapt train Learning methods Batch training ith eight & bias learning rules. BFGS quasi-neton backpropagation. Bayesian Regulation backpropagation. Cyclical order eight/bias training. Conjugate gradient backpropagation ith Poell-Beale restarts. Conjugate gradient backpropagation ith Fletcher-Reeves updates. Conjugate gradient backpropagation ith Polak-Ribiere updates. Gradient descent backpropagation. Gradient descent ith momentum. Gradient descent ith adaptive learning rate backpropagation. Gradient descent /momentum & adaptive learning rate backpropagation. Levenberg-Marquardt backpropagation. One step secant backpropagation. Random order eight/bias training. RPROP backpropagation. Scaled conjugate gradient backpropagation. Activation functions Competitive transfer function Hard-limit transfer function Symmetric hard-limit transfer function Log-sigmoid transfer function Positive linear transfer function Linear transfer function Radial basis transfer function Saturating linear transfer function Symmetric saturating linear transfer function Hyperbolic tangent sigmoid transfer function Functions alloing easier analysis Error surface of single-input neuron Maximum learning rate for linear layer Functions initializing layer s parameters Nguyen-Widro layer initialization function By eight and bias layer initialization function Operations on vectors Convert cell array to numeric array Create concurrent bias vectors Convert concurrent vectors to sequential vectors Create all combinations of vectors Convert array to cell array ith potentially different sized cells Ranges of matrix ros Copy matrix or cell array Normalize columns of matrix Normalize ros of matrix Convert sequential vectors to concurrent vectors Sum of squared elements of matrix or matrices Operations on netorks Simulate dynamic system (neural netork) Initialize neural netork Adapt neural netork to data as it is simulated Train neural netork 10

11 hinton hintonb plotes plotpc plotpv plotep plotperf plotv plotvec nntool gensim Plotting Hinton graph of eight matrix. Hinton graph of eight matrix and bias vector. Plot error surface of single-input neuron Plot classification line on perceptron vector plot Plot perceptron input/target vectors Plot eight-bias position on error surface Plot netork performance. Plot vectors as lines from origin Plot vectors ith different colors Others Neural Netork Toolbox graphical user interface Generate a Simulink block to simulate a neural netork. 11

12 An example of creating and learning a perceptron neural netork. Belo are the basic commands used to create an example perceptron neural netork. The process of creating other netorks is similar, only the parameters and some instructions are different. It is possible to create a perceptron netork by using the nep command. Syntax: net = nep(pr,s,tf,lf) The parameters of this command are: PR RxQ matrix of Q1 representative input vectors. S - SxQ matrix of Q2 representative target vectors. TF Transfer (activation) function, default = 'hardlim'. LF Learning function, default = 'learnp'. It is not necessary to give the TF and LF parameters. The result of this command is creating a perceptron netork. Under the name net a structure is created. Within this structure all the information about this netork are stored. Netork initialization nep command also initializes (gives) the initial (zero) values of eights and thresholds. Weights connecting the inputs to the netork are stored in the structure net.iw. The thresholds are stored in the netork structure net.b. It is also possible to choose values of these elements independently, e.g.: net.iw{1,1}=[2 4] net.b{1}=[5] To restore the default eights and thresholds (initialization of the netork parameters) a command e.g. init should be used. net = init(net); Netork simulation To study the netork s response for an input vector a sim command should be used. The syntax of this command is available after typing help sim in the MATLAB s command indo. Mostly it is sufficient to specify hat netork is going to be used and to indicate input vector, in this case P: 12

13 Y = sim(net,p); It simulates response of netork net for input P. Defining input vector P as: P = [ ; ]; And target vector T: T = [ ]; The result is netork s response: Y = hich is not compliant ith expectations (target T). To achieve anted output values (accordingly to the target) a proper selection of eights and bias values is necessary, hich means that the netork has to learn ho to respond properly. Example Design perceptron netork consisting of a single neuron ith to inputs. The first input should be ithin the range [0; 1] and the second input ithin the range [-2; 2]. Save the proposed structure under the name net. In response to a predefined vector P, the netork should respond in accordance ith the vector T. net = nep([0 1; -2 2],1); Response for vector P: Y = sim(net,p) Y = Netork learning As e can see the netork s response is different from the vector T. In order to achieve the correct response a change of the eights and biases values is needed, either manually or in the learning process. Command train is used to start learning process. This statement evaluates the netork s default learning method (modification of eights and thresholds) in order to fit the response into the vector T. More options for train command can be obtained by typing help train. 13

14 net = train(net,p,t); Response of the netork after learning process is shon belo. Y = sim(net,p) Y = As it can be seen netork responded properly. It completes the example. It is possible to check ith hat values of eights and biases the task ends: net.iw{1,1} net.iw{1,1} =[1 1] net.b{1} =[-1] Changing the default parameters of netork After creating any netork it has various default parameters such as: learning method, the initialization method, the method of testing the quality of the netork, the netork learning process and many others. To check them typing the name of the created netork in MATLAB s command indo is needed. These parameters can also be changed ith usage of appropriate commands. Belo is some basic ones (for the perceptron netork created by nep command). Functions: adaptfcn: 'trains' initfcn: 'initlay' performfcn: 'mae' trainfcn: 'trainc' Parameters: adaptparam:.passes initparam: (none) performparam: (none) trainparam:.epochs,.goal,.sho,.time Changing default values of netork parameters (example values): net.trainparam.sho = 50; net.trainparam.lr = 0.05; net.trainparam.epochs = 300; net.trainparam.goal = 1e-5; 14