Problem Set 2: From Perceptrons to Back-Propagation

Transcription

1 COMPUTER SCIENCE 397 (Spring Term 2005) Neural Networks & Graphical Models Prof. Levy Due Friday 29 April Problem Set 2: From Perceptrons to Back-Propagation 1 Reading Assignment: AIMA Ch Programming Assignment: More Matlab, More Perceptrons, and Back-Prop This assignment consists of two parts (1) enhancement of your Matlab percep learning function from the last assignment and (2) implementing back-propagation. Let s take these one at a time. 2.1 Enhancing the perceptron algorithm Computing RMS error Recall the error term D from the PERCEPTRON-LEARNING algorithm. This term expresses the Difference between the target output T and the actual output O, for each training pattern. Once this difference drops to zero (or close to zero), there is no need to continue training via the outer loop. Instead of looping some arbitrary number of times (like 1000), we can therefore stop looping when the error drops below a certain value, averaged over all the patterns. We don t really care whether the values of D are too big in the positive or negative direction, just that their magnitude is too big (for example, D j = 0.5 and D j = 0.5 are equally bad). Furthermore, we want to compute the average difference over all the patterns, to get an idea of how well our perceptron is working on the whole problem. That is why we took the average value of the absolute values D in the previous assignment. A more common way of expressing the average error is called the Root Mean Squared (RMS) Error, defined as p m E = ( Djk 2 )/(pm) j k where m is the number of outputs and p is the number of patterns, as before. 1 To compute the RMS error, you should set it to zero right before the pattern (j) loop, and add to it the sum over the square of D within the pattern loop: 1 Intuitively, D 2 j gets rid of the +/- sign on each D j, and the square-root function undoes the effect of squaring.

2 e(i) = e(i) + sum(d.* D); %.* means element-wise product Then right after the pattern loop you can divide this sum by the product p*m, and compute the square root of the resulting value using Matlab s built-in sqrt function. Once you have done this, you can change the outer for loop (for i = 1:1000) to an loop (for i=1:inf) and then halt this loop using a break statement right after you compute the RMS error E (if e(i) < 0.05, break, end). If you leave the final semicolon off the call to sqrt, you will be able to see the RMS error on each iteration of the learning algorithm. Note how few iterations it actually takes to learn the boolean OR and AND functions Scaling Up In the real world, we would probably use a built-in AND or OR function (or prefabricated circuit), instead of training a perceptron to learn such a simple task. Indeed, as its name suggests, the perceptron was designed to make decisions about percepts (images, sounds), just as human beings are able to do so effortlessly. For this part of the assignment, you will train your perceptron to classify bitmaps (images) of the digits zero through seven. First, download the Matlab file digits.mat from the class web page. This file contains the matrices dig0, dig1,..., dig7, containing bitmaps of the digits 0 through 7, as well as the vectors bin0, bin1,..., bin7, containing bit patterns for those numbers (0 = 000, 1 = 001,..., 7 = 111). To see the bitmaps more clearly, you can use Matlab s spy function; e.g., spy(dig3) will show you something that looks (roughly) like the digit 3. We need to put these patterns into one matrix containing all the digits and another containing all the binary targets. For the latter, we can just use Matlab s semicolon notation, as we did in the previous lab: >> bins = [bin0; bin1; bin2; bin3; bin4; bin5; bin6; bin7] For the bitmap images, we will have to turn each image into a vector, and then combine the vectors as above. We can do this using Matlab s reshape function, to turn each 5 4 matrix into a 1 20 vector: 2 >> digs = [reshape(dig0,1,20); reshape(dig1,1,20);...; reshape(dig7,1,20)] Now we can train our perceptron by simply typing w = percep(digs, bins), and test it by typing, e.g., ptest(w, reshape(dig0, 1, 20)). Finally, let s play a little with the graceful degradation feature of neural net models. Add a little noise to one of the digits by using the noise function, which you can download from the web page. This function adds a little noise, from 0 to 100 percent, to the bitmap: >> ptest(w, reshape(noise(dig3, 10), 1, 20)) How much noise is fatal to the perceptron s behavior? Test a few different numbers with different amounts of noise, and turn in a README with your results. 2 You might want to write a separate dig2mat function to replace the calls to reshape(dign, 1, 20.

3 2.2 Back-Propagation Implementing Back-Prop In this part of the lab we will use the formulas in the back-prop lecture to convert our percep program into a program for doing back-propagation. To get started, copy your enhanced percep.m to a new file bp.m. Then download the file bp.zip from the class web page, and unzip the file to get a bunch of small utility functions that I wrote to help you. Now you can start modifying bp.m to do back-prop. First, change the function header to reflect the new name, the two sets of weights returned (input hidden and hidden output), and the two additional input arguments (number of hidden units, learning rate) required by the back-prop algorithm: function [wih, who] = bp(i, T, nh, eta) Next, change the names of the variables encoding the number of input and output units to something more meaningful, and use these variables to create the initial random weight matrices (including bias weights): 3 ni = size(i, 2); no = size(t, 2); wih = randn(ni+1, nh); who = randn(nh+1, no); You should remove the line I(:,end+1) = 1;, because the attachment of biases is done already in the forward and update utility functions. The remainder of the program can be written using the comments on the next page as a guideline. Each comment corresponds to a call (or two) of one of the utility routines in bp.zip. Looking at the routines and the class notes on back-prop should help you figure out how to call the routines. You should also write a test program bptest.m, which will accept the weights (wih and who) computed by bp, as well as the input I, and will compute and show the resulting outputs (by leaving out a final semicolon from the line in which they are computed). Train and test your perceptron on the XOR function (b = [0; 1; 1; 0]), and experiment with a few different learning rates and numbers of hidden units. Turn in your bp.m and bptest.m, and some plots of your errors over training epochs, as you did in the previous section. You can use Matlab s title command to annotate each plot with these values; e.g., title( NH=2 Eta=0.3 ). 3 Be sure to change these variables everywhere they appear, by using global search-and-replace.

4 % for each pattern 1 <= j <= p do for j = 1:p end... % forward pass % error on output % delta on output % backprop to get error on hidden % delta on hidden % delta rule for weight update % report RMS error every 1000 iterations if mod(i,1000) == 0 fprintf( %d: %4.4f\n, i, E) end % update weights Speedup and Generalization Once you have successfully implemented backprop, try speeding it up by adding a momentum term, as described in the lecture notes. The momentum coefficient µ (mu) should be passed in as the fifth argument to your bp function. Run your modified back-prop algorithm a few times on the XOR problem, using µ = 0.9, to convince yourself that momentum can dramatically speed up convergence. Note down the approximate average difference in training time in your README file. Next, we ll experiment a bit with the tradeoff between learning speed and generalization. Download the file alpha.mat from the class web page, and load the data contained in it by typing >> load alpha in the Matlab interpreter. The data consist of bitmaps for the letters A and B, as well as training matrices containing these bitmaps, and binary codes for each bitmap (A = 01; B = 10). (You can see the bitmap patterns using spy(a) and spy(b).) Perform the following simple experiment: Train a hidden-layer net using your bp program on this data, with a small number of hidden units (e.g., two). Test this small network on the training data with 25 percent noise added, as you did for the digit-recognizing perceptron above. Treat outputs above 0.5 as 1, and outputs below 0.5

5 as 0 4., and record the number of times, out of ten tests, that the network correctly classifies both letters. Now train a much larger network (e.g., 40 hidden units) on the same task, and test it the same way. How do your two networks compare on this version of generalization? Put your results in your README file, along with a brief explanation of any difference you note. 4 You can have Matlab do this for you: bptest(wih, who, noise(25,alpha)) >.5