Aravind Baskar A ASSIGNMENT - 3

Transcription

1 Aravind Baskar A ASSIGNMENT - 3 EE5904R/ME5404 Neural Networks Homework #3 Q1. Function Approximation with RBFN (20 Marks) Solution a): RBFN-Exact Interpolation Method: Number of hidden units = number of data points Several trainings are made because noise adding to the function is random, which provide different solution based on randomly chosen values. The mean squared error for the five trials and the average error are Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Average MATLAB CODE: clear all; x_train = -1:0.05:1; x_test = -1:0.01:1; N_train=length(x_train); N_test=length(x_test); for trial=1:5 noise = randn(n_train,1); y_train = 1.2*sin(pi*x_train)-cos(2.4*pi*x_train)+0.3*noise'; y_test = 1.2*sin(pi*x_test)-cos(2.4*pi*x_test); sigma = 0.1; w_train= ones(n_train,n_train); for j=1:n_train,for i=1:n_train w_train(j,i)= exp(-((x_train(j)-x_train(i))^2)/(2*(sigma)^2));, w_train = inv(w_train)*y_train'; w_test = zeros(n_test,n_train); for j=1:n_test,for i=1:n_train w_test(j,i)= exp(-((x_test(j)-x_train(i))^2)/(2*(sigma)^2));, net_output = w_test*w_train; error = y_test' - net_output; disp(mse(error)); figure(trial); plot(x_train,y_train,'b^');hold on;

2 plot(x_test,y_test,'go');hold on; plot(x_test,net_output,'r');hold off; leg('training data','test data','rbfn Output','Location','NorthWest'); title(['rbfn - Exact interpolation method; trial-',int2str(trial)]); print(figure(trial),'-djpeg100',strcat('q1_a_',int2str(trial))) (a) (b) (c) (d) (e) Fig.1. RBFN-Exact Interpolation Method

3 This shows that RBFN Exact Interpolation method is over fitting because we included some noise to the exact solution, which provided inaccurate results with high error values. a) Follow the strategy of Fixed Centers Selected at Random (as described on page 37 in the slides of lecture five), randomly select 15 centers among the sampling points. Determine the weights of the RBFN. Evaluate the approximation performance of the resulting RBFN using the test set. Compare it to the result of part a). Solution (b): Fixed Centers Selected at Random In Fixed centers method, 15 center points are randomly assigned among the sampling points i.e., 15 hidden layers. (a) (b) (c) (d)

4 (e) Fig.2. Fixed Centers Selected at Random: No.of centers =15 The mean squared error for the five trials, when m=15 and the average error are Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Average Comparison: When comparing exact interpolation and fixed center method, the errors are small in fixed center method in some trials but in other few trials the error was extremely large. This is because the interpolation matrix is close to singular or badly scaled. The solution deps on the number of centers we choose (i.e. the number of hidden units). MATLAB CODE: % Fixed Centers selected at Random clear all; clc; x_train = -1:0.05:1; x_train = x_train(:); x_test = -1:0.01:1; x_test = x_test(:); N_train=length(x_train); N_test=length(x_test); M = 15; for trial=1:5 noise = randn(n_train,1); y_train = 1.2*sin(pi*x_train)-cos(2.4*pi*x_train)+0.3*noise; y_test = 1.2*sin(pi*x_test)-cos(2.4*pi*x_test); % select M centers randomly from the training points

5 centerind = randperm(n_train); centers = x_train(centerind(1:m)); dist = minmax(centers); dmax = (dist(2)-dist(1))^2; phimat = ones(n_train,m+1); for i=1:n_train for j=1:m phimat(i,j+1)=exp(-m*(x_train(i)-centers(j))^2/dmax); weights = inv(phimat'*phimat)*(phimat'*y_train); phimattest = ones(n_test,m+1); for i=1:n_test for j=1:m phimattest(i,j+1)=exp(-20*(x_test(i)-centers(j))^2/dmax); net_output = phimattest*weights; e = y_test - net_output; E = mse(e); disp(e); figure(trial); plot(x_train,y_train,'b^');hold on; plot(x_test,y_test,'go');hold on; plot(x_test,net_output,'r');hold off; leg('training data','test data','rbfn Output','Location','NorthWest'); title(['rbfn- Fixed Centers Selected at Random; trial-',int2str(trial)]); Solution (c): RBFN - Regularization Factor Method The regularization factors are varied between and 10 to see the effects on the exact interpolation method. The mean squared errors are Lamb 0.00 da 1 Error When comparing RBFN and Regularization method, the error are considerably small for lesser lambda values. But error shoot ups when lambda value is increased leads to under fit the curve. And also increase in lambda increases the smoothness of the curve.

6

7 Fig.4. Regularization Factor Method MATLAB CODE: clear all; x_train = -1:0.05:1; x_test = -1:0.01:1; N=length(x_train); N_test=length(x_test); noise = randn(n,1); y_train = 1.2*sin(pi*x_train)-cos(2.4*pi*x_train)+0.3*noise'; y_test = 1.2*sin(pi*x_test)-cos(2.4*pi*x_test); sigma = 0.1; W_train= ones(n,n); for j=1:n,for i=1:n W_train(j,i)= exp(-((x_train(j)-x_train(i))^2)/(2*(sigma)^2));, lambda = [ ]; for trial=1:length(lambda) w_train = inv(w_train'*w_train+lambda(trial)*eye(n,n))*w_train'*y_train'; W_test = zeros(n_test,n); for j=1:n_test,for i=1:n W_test(j,i)= exp(-((x_test(j)-x_train(i))^2)/(2*(sigma)^2));, net_output = W_test*w_train; error = y_test' - net_output; E(trial)=mse(error); disp(mse(error)); figure(trial); plot(x_train,y_train,'b^');hold on; plot(x_test,y_test,'go');hold on;

8 plot(x_test,net_output,'r');hold off; leg('training','test','rbfn Output','Location','NorthWest'); title(['regularization Factor method; lambda =',num2str(lambda(trial))]); loglam=log(lambda); plot(loglam,e,'-r*'); xlabel('log of Regularization Factor');ylabel('Mean Square Error'); Solution (d): RBFN Fixed center Vs Regularization Factor method Similar to previous case, error will increase with increase in lambda in regularization factor method. Lamb 0.00 da 1 Error The mean square error values are very less for lambda less than 0.5 when comparing to Fixed center RBFN method. For lambda greater than 0.5 i.e. increase in lambda value increases the curve smoothness but it also increase the mean square error which is greater than Fixed center error.

9 Fig.5. Regularization Factor method Vs Fixed Center RBFN

10 MATLAB CODE: clear all; clc; x_train = -1:0.05:1; x_test = -1:0.01:1; N_train=length(x_train); N_test=length(x_test); M=15; noise = randn(n_train,1); y_train = 1.2*sin(pi*x_train)-cos(2.4*pi*x_train)+0.3*noise'; y_test = 1.2*sin(pi*x_test)-cos(2.4*pi*x_test); index = randperm(n_train); mu = x_train(index(1:m)); minmaximum = minmax(mu); dmax = minmaximum(2)-minmaximum(1); dmax=dmax^2; W_train= ones(n_train,m); for j=1:n_train,for i=1:m W_train(j,i)= exp(-15*((x_train(j)-mu(i))^2)/dmax);, lambda = [ ]; for trial=1:length(lambda) w_train = inv(w_train'*w_train+lambda(trial)*eye(m,m))*w_train'*y_train'; W_test = zeros(n_test,m); for j=1:n_test,for i=1:m W_test(j,i)= exp(-15*((x_test(j)-mu(i))^2)/dmax);, net_output = W_test*w_train; error = y_test' - net_output; E(trial)=mse(error); disp(mse(error)); figure(trial); plot(x_train,y_train,'b^');hold on; plot(x_test,y_test,'go');hold on; plot(x_test,net_output,'r');hold off; leg('training','test','rbfn Output','Location','NorthWest'); title(['regularization Factor method; lambda =',num2str(lambda(trial))]); loglam=log(lambda);

11 plot(loglam,e,'-r*'); xlabel('log of Regularization Factor');ylabel('Mean Square Error'); Q2. Handwritten alphabet classification using Self-organizing Map (20 Marks) Solution: The hand written pictures are loaded as binary image from SOM_database.mat Learning rate = No.of iterations = 1000 Sigma calculated as per size of the image = 7.07 'U' 'U' 'L' 'O' 'O' 'O' 'S' 'S' 'E' 'E' 'U' 'U' 'U' 'L' 'L' 'O' 'S' 'L' 'L' 'L' 'U' 'U' 'U' 'U' 'L' 'L' 'L' 'L' 'L' 'L' 'U' 'U' 'N' 'N' 'N' 'N' 'L' 'L' 'L' 'L' 'M' 'N' 'N' 'A' 'A' 'A' 'A' 'L' 'L' 'L' 'N' 'N' 'N' 'N' 'A' 'A' 'A' 'A' 'R' 'R' 'M' 'M' 'M' 'A' 'A' 'A' 'A' 'A' 'R' 'R' 'M' 'M' 'M' 'A' 'A' 'A' 'A' 'A' 'R' 'R' Since SOM randomly initialized the values keeps on changes in every run. 1) For each neuron in the trained SOM, show the weights on a 10x10 map to visualize the result of the training process and make comments about them, if any. A simple way would be to reshape the weight of a neuron into a 20x16 matrix with double precision (since that is the dimension of the training image) and display it as an image. As seen, the weights look almost same as the label obtained in the previous part. Since learning rate = 0.1.

12 2) Use the generated SOM to classify the test images (found in test_data). The classification can be done in the following fashion. Apply a test sample to SOM, and determine the winner neuron. Then label the test sample with the one corresponding to the winner neuron (note that labels of all the neurons in the SOM have already been determined in the first step). Compare the label resulting from above classification scheme to the true label of the test sample, and calculate the recognition rate for the whole test set. The SOM is tested with the test images (found in test_data). The winner solution is determined by applying the test sample to SOM then test sample is labelled with corresponding winner neuron. Several trails are done because the initial weights are chosen random. At each trial gives different performance results because the values are assigned randomly. The average of five trials are taken and the performance in percentage given below Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Average 'U' 'M' 'A' 'L' 'U' 'N' 'M' 'M' 'N' 'A' 'E' 'E' 'E' 'E' 'E' 'E' 'L' 'E' 'E' 'E' 'U' 'U' 'U' 'U' 'U' 'U' 'U' 'R' 'U' 'U' 'L' 'R' 'N' 'N' 'R' 'A' 'R' 'R' 'R' 'A' 'A' 'A' 'A' 'A' 'A' 'A' 'A' 'N' 'A' 'A' 'L' 'L' 'L' 'U' 'U' 'L' 'L' 'U' 'U' 'L' 'M' 'N' 'M' 'M' 'M' 'M' 'M' 'A' 'M' 'M' 'O' 'O' 'O' 'O' 'L' 'L' 'O' 'O' 'O' 'O' 'S' 'S' 'S' 'S' 'S' 'S' 'S' 'S' 'L' 'S' When epochs = 1000; size=10x10; Learning rate=0.1; Recognition Rate = 72.22% Testing of SOM for other design parameters The parameters that can be varied and played around are a) Learning rate b) Size of the lattice c) Sigma d) Width e) Number of iterations The initial value of sigma deps on the size of the lattice as per the formula given. So we concentrate on the size and no.of iteration.

13 Size of lattice When we increase the size of lattice from 10x10 to 20x20, the accuracy rate also increased but the time consumption for learning also increased when comparing to lower size. Thus increasing in size also helps in improving the performance. But there is a tradeoff between time for computation and the performance. Iterations: Increase in no.of iteration increase the time conception. But the accuracy of prediction is considerably increased. Obtained label values for iteration. The weights here almost completely comply with the labels obtained. Also the accuracy obtained here is % which is higher compared to the previous part. Thus increasing the iterations definitely helps in improving the performance. But there is a trade-off between time for computation and the performance. 'N' 'A' 'L' 'R' 'R' 'E' 'E' 'E' 'S' 'S' 'N' 'A' 'R' 'R' 'E' 'E' 'E' 'S' 'S' 'S' 'N' 'N' 'A' 'R' 'E' 'E' 'E' 'O' 'O' 'S' 'M' 'M' 'M' 'M' 'R' 'R' 'R' 'O' 'O' 'O' 'M' 'M' 'N' 'N' 'M' 'R' 'O' 'O' 'O' 'O' 'N' 'N' 'M' 'M' 'U' 'L' 'L' 'L' 'O' 'O' 'N' 'M' 'M' 'M' 'U' 'L' 'L' 'L' 'U' 'U' 'M' 'M' 'M' 'M' 'U' 'L' 'L' 'L' 'U' 'U' The corresponding weights can be visualized as