1. Approximation and Prediction Problems

Transcription

1 Neural and Evolutionary Computing Lab 2: Neural Networks for Approximation and Prediction 1. Approximation and Prediction Problems Aim: extract from data a model which describes either the depence between two variables (the variables can be one or multi-dimensional) or the variation in time of a variable (in order to predict new values in a time series). Examples: Experimental data analysis: o The input data are the measured values the analyzed variables (one variable is the predictor, the other is the predicted one); o The results are values of the predicted variables estimated for some given values of the predictor. Time series prediction: o The input data are previous values of a time-depent variable; o The result is an estimation of the next value in the time series. 2. Using the Matlab nftool for approximation problems nftool (Neural Network Fitting Tool) provides a graphical user interface for designing and training a feedforward neural network for solving approximation (fitting) problems. The networks created by nftool are characterized by: One hidden layer (the number of hidden units can be changed by the user; the default value is 20) The hidden units have a sigmoidal activation function (tansig or logsig) while the output units have a linear activation function The training algorithm is Backpropagation based on a Levenberg-Marquardt minimization method (the corresponding Matlab function is trainlm ) The learning process is controlled by a cross-validation technique based on a random division of the initial set of data in 3 subsets: for training (weights adjustment), for learning process control (validation) and for evaluation of the quality of approximation (testing). The quality of the approximation can be evaluated by: Mean Squared Error (MSE): it expresses the difference between the correct otputs and those provided by the network; the approximation is better if MSE is smaller (closer to 0) Pearson s Correlation Coefficient (R): it measures the correlation between the correct outputs and those provided by the network; as R is closer to 1 as the approximation is better. Exercise 1 a) Using nftool design and train a neural network to discover the depence between the data in the set Simple Fitting Problem.

2 b) Analize the network behavior by visualizing the dataset and the approximated function. Analyze the quality of the approximation by using the plot of R c) Compare the obtained results for different numbers of hidden units: 5, 10, 20. d) Save the network in an object named netfitting and simulate its functioning for different input data Hint: a) use nftool and select the corresponding dataset from Load Example Data Set b) visualize the graphs with PlotFit and PlotRegression c) change the architecture, retrain and compare the quality measures (MSE, R) for different cases d) after the network is saved, it can be simulated by using sim(netfitting,dateintrare) 3. Using Matlab functions to create and train feedforward neural networks The function used in Matlab to define a multi layer perceptron is newff. In the case of a network having m layers of functional units (i.e., m-1 hidden layers) the syntax of the function is: ffnet = newff(inputdata, desiredoutputs, {k 1,k 2,,k m-1 }, {f 1,f 2,,f m }, <training algorithm> ) The significance of the parameters involved in newff is: inputdata: a matrix containing on its columns the input data corresponding to the training set. Based on this, newff establishes the number of input units. desiredoutputs: a matrix containing on its columns the correct answers corresponding to the training set. Based on this, newff establishes the number of output units. {k 1,k 2,,k m-1 }: the number of units on each hidden layer {f 1,f 2,,f m }: the activation functions for all functional layers. Possible values are: logsig, tansig and purelin. The hidden layers should have nonlinear activation functions ( logsig or tansig ) while the output layer can have a linear activation function. <training algorithm> : is a parameter specifying the type of the learning algorithm. There are two main training variants: incremental (the training process is initiated by the function adapt) and batch (the training process is initiated by the function train). When the function adapt is used the possible values of this parameter are: learngd (it corresponds to the serial standard BackPropagation derived using the gradient descent method), learngdm (it corresponds to the momentum variant of BackPropagation), learngda (variant with adaptive learning rate). When the function train is used the possible values of this parameter are: traingd (it corresponds to the classical batch BackPropagation derived using the gradient descent method), traingdm (it corresponds to the momentum variant of BackPropagation), trainlm (variant based on the Levenberg Marquardt minimization method). Remark 1: the learning function can be different for different layers. For instance by ffnet.inputweights{1,1}.learnfcn='learngd' one can specify which is the learning algorithm used for the weights between the input layer and the first layer of functional units.

3 Exemple 1. Definition of a network used to represent the XOR function: ffnet=newff([ ; ],[ ],{5,1},{'logsig','logsig'},'traingd') Training. There are two training functions: adapt (incremental learning) and train (batch learning). Adapt. The syntax of this function is: [fftrained,y,err]=adapt(ffnet, inputdata, desiredoutputs) There are several parameters which should be set before starting the learning process: Learning rate - lr: the implicit value is 0.01 Number of epochs - passes (number of passes through the training set): the implicit value is 1 Momentum coefficient mc (when learngdm is used): the implicit value is 0.9 Example: ffnet.adaptparam.passes=10; ffnet.adaptparam.lr=0.1; Train. The syntax of this function is: [fftrained,y,err]=train(ffnet, inputdata, desiredoutputs) There are several parameters which should be set before starting the learning process: Learning rate - lr: the implicit value is 0.01 Number of epochs - epochs (number of passes through the training set): the implicit value is 1 Maximal value of the error: goal Momentum coefficient mc (when learngdm is used): the implicit value is 0.9 Example: ffnet.trainparam.epochs=1000; ffnet.trainparam.lr=0.1; ffnet.trainparam.goal=0.001; Simulating multi layer perceptrons. In order to compute the output of a network, one can use the function sim, having the syntax: sim(fftrained, testdata) RBF networks. These neural networks always contain only one layer of hidden units. These hidden units have activation functions with radial symmetry (this means that their graph is symmetric with respect to the Oy axis). The most frequently used activation function is the Gaussian one (specified in Matlab by radbas). The particularity of Matlab training for RBF function is the fact that the weights are established just during the network creation. There are to creation (including training) functions: newrbe (in,d,sigma) newrb(in,d,goal,sigma)

4 In both cases the parameters in and d represents elementals from the training set and sigma represents the width parameter associated to the Gaussian function. The function newrbe creates a network having as many hidden units as data are in the training set. The function newrb constructs the hidden layer by incrementally adding units until the training error becomes less than goal. Application 1. Linear and nonlinear regression Let us consider a set of two dimensional data represented as points in plane (the coordinates are specified by the user using the mouse coordinates of the point where a click is made can be captures by thematlab function ginput). Our aim is to train a neural network in order to capture the functional relationship between the coordinates of the points, i.e. the depence between the value of the ordinate (Oy axis) and abscissa (Ox axis). The network will be trained such that it minimizes the sum of squared distances between the input points and the graph of the estimated function). function [in,d]=regression(layers,hiddenunits,training,cycles) % if layers = 1 then a single layer linear neural network will be created % if layers = 2 then a network with one hidden layer will be created and % hiddenunits denote the number of units on the hidden layer % training = 'adapt' or 'train' % cycles = the number of passes (for adapt) and epochs (for train) % read the data clf axis([ ]) hold on in = []; d = []; n = 0; b = 1; while b == 1 [xi,yi,b] = ginput(1); plot(xi,yi,'r*'); n = n+1; in(1,n) = xi; d(1,n) = yi; inf=min(in); sup=max(in); % define the network if (layers==1) reta=newlind(in,d); % linear network designed starting from input data else ret=newff(in,d,hiddenunits,{'tansig','purelin'},'traingd'); if(training == 'adapt') % setting the parameters ret.adaptparam.passes=cycles; ret.adaptparam.lr=0.05;

5 % network training reta=adapt(ret,in,d); else ret.trainparam.epochs=cycles; ret.trainparam.lr=0.05; ret.trainparam.goal=0.001; % network training reta=train(ret,in,d); % graphical representation of the approximation x=inf:(sup-inf)/100.:sup; y=sim(reta,x); plot(in,d,'b*',x,y,'r-'); Exercises: 1. Test the ability of function regression to approximate different sets of data. Hint: for linear regression call the function as: regression(1,0, adapt, 1) (the last two parameters are ignored); for nonlinear regression the call should be: regression(2,5, adapt, 5000) 2. In the case of nonlinear regression analyze the influence of the number of hidden units. Values to test: 5 (as in the previous exercise), 10 and 20. In order to test the behavior of different architectures on the same set of data save the data specified in the first call and define a new function which does not read the data but receive them as parameters. Application 2. Prediction Let us consider a sequence of data (a time series): x 1,x 2,.,x n which can be interpreted as values recorded at successive moments of time. The goal is to predict the value corresponding to moment (n+1). The main idea is to suppose that a current value x i deps on N previous values: x i-1, x i-2,,x i-n. Based on this hypothesis we can design a neural network which is trained to extract the association between any subsequence of N values and the next value in the series. Therefore, the neural network will have N input units, a given number of hidden units and 1 output unit. The training set will have n-n pairs of (input data, correct output): {((x 1,x 2,,x N ),x N+1 ),((x 2,x 3,,x N+1 ),x N+2 ),,((x n-n,x n-n+1,,x n-1 ),x n )}. A solution in Matlab is: function [in,d]=prediction(data,inputunits,hiddenunits,training,epochs) % data : the series of data (one row matrix) % inputunits : number of previous data which influences the current one % hiddenunits : number of hidden units % traininb= 'adapt' or 'train' % epochs = number of epochs % preparing the training set L=size(data,2);

6 in=zeros(inputunits,l-inputunits); d=zeros(1,l-inputunits); for i=1:l-inputunits in(:,i)=data(1,i:i+inputunits-1); d(i)=data(1,i+inputunits); % creating the network ret=newff(in,d,hiddenunits,{'tansig','purelin'},'traingd'); if(training == 'adapt') ret.adaptparam.passes=epochs; ret.adaptparam.lr=0.05; reta=adapt(ret,in,d); else ret.trainparam.epochs=epochs; ret.trainparam.lr=0.05; ret.trainparam.goal=0.001; reta=train(ret,in,d); % graphical plot x=1:l-inputunits; y=sim(reta,in); intest=zeros(inputunits,1); intest=data(1,l-inputunits+1:l)'; reztest=sim(reta,intest); disp('predicted value:'); disp(reztest); plot(x,d,'b-',x,y,'r-',l+1,reztest,'k*'); Let us use the prediction function to estimate the next value in a series of Euro/Lei exchange rate. The data should be first read: data=csvread( dailyrate.csv ) ; The data in the file are daily rates from 2005 up to October 16, 2009 downloaded from Since the values are stored in reverse chronological order we construct a new list: data2(1:length(data))=data(length(data):-1:1) Then the prediction function can be called by specifying the data set, the number of input units, the number of hidden units, the type of the learning process and the maximal number of epochs:: prediction(data2, 5,10, adapt,2000) Exercise. 1. Analyze the influence of the number of input units on the ability of the network to make prediction (Hint: try the following values: 2, 5, 10, 15)

7 2. Analyze the influence of the number of hidden units on the ability of the network to make prediction (Hint: the number of hidden units should be at least as large as the number of input units) 3. Analyze the influence of training algorithm on the ability of the network to make prediction 4. Use a RBF network to solve the prediction problem (Hint: a variant is implemented in predictionrbf.m) Homework: Let us consider a set of data (file NrVisits.txt) containing the number of visits of a web server. Approximate the series of values using a neural network and estimate the next value in the series. You can try to use both a BackPropagation and a RBF network.