Homework 1. An Introduction to Neural Networks

Transcription

1 Hoework An Introduction to Neural Networks -785: Introduction to Deep Learning Spring 09 OUT: January 4, 09 DUE: February 6, 09, :59 PM Start Here Collaboration policy: You are expected to coply with the University Policy on Acadeic Integrity and Plagiaris. You are allowed to talk with / work with other students on hoework assignents You can share ideas but not code, you ust subit your own code. All subitted code will be copared against all code subitted this seester and in previous seesters using MOSS. Overview: Part : All of the probles in Part will be graded on Autolab. You can download the starter code hw.py fro Autolab as well. This assignent has 00 points total, including 90 that are autograded. Part : This section of the hoework is an open ended copetition hosted on Kaggle.co, a popular service for hosting predictive odeling and data analytics copetitions. All of the details for copleting Part can be found on the copetition page. Subission: Part : The copressed handout folder hosted on Autolab contains a single python file, hw.py. This contains a teplate for solving all of the probles and includes soe useful helper functions. Make sure that all class and function definitions originally specified as well as class attributes and ethods in hw.py are fully ipleented to the specification provided in this writeup and in the docstrings or coents in hw.py. Your subission ust be titled handin.tar gzip forat and it is inially required to contain a directory called hw, which contains the hw.py file ipleenting the classes, functions, and ethods specified in the writeup. After copleting Training Statistics hw will also contain iages. Other source files that are iported by hw.py are allowed. Please do not iport any other external libraries other than Nupy, as extra packages that do not exist in the autograder iage will cause a subission failure. Part : See the the copetition page for details. Local autograder: To ease your life, we provide in the handout for part a local autograder that you can use on your own code. It roughly has the sae behavior as the one on Autolab, except that it will copare your outputs and attributes with prestored results on prestored data while autolab directly copares you to a solution. In theory, passing one eans that you pass the other, but we recoend you also subit partial solutions on autolab fro tie to tie while copleting the hoework. To run the local autograder : ove your hw folder in local autograder/ and run runner.py fro the local autograder folder.

2 Neural Networks For part of the hoework you will write your own ipleentation of the backpropagation algorith for training your own neural network, as well as a few other features such as activations and batch-noralization. You are required to do this assignent in the Python Python version 3 prograing language. Do not use any autodiff toolboxes PyTorch, TensorFlow, Keras, etc - you are only peritted and recoended to vectorize your coputation using the Nupy library. In the file hw.py we provide you with classes whose ethods you have to fill. The autograder tests will copare the outputs of your ethods and attributes of your classes with a reference solution. Therefore we enforce for a large part the design of your code ; however, you still have a lot of freedo in your ipleentation. We recoend that you look through all of the probles before attepting the first proble. However we do recoend you coplete the probles in order as the difficulty increases, and questions often rely on the copletion of previous questions. Activations A nuber of activation classes including Sigoid, Tanh and ReLU ust be ipleented. Each of these coe in the for of a class which has both a forward and derivative ethods that ust be ipleented. The Identity activation has been ipleented for you as an exaple. All of these classes derive fro Activation which acts as an abstract base class / interface. You only need to ipleent the two ethods for each class. No helper functions or additional classes should be necessary. Keep your code as concise as possible and leverage Nupy as uch as possible. e indful that you are working with atrices which have a given shape as well as a set of values.. Softax Cross Entropy [5 points] For this hoework, we will be using the softax cross entropy loss detailed in the appendix of this writeup. Use the LogSuExp trick to ensure nuerical stability. Ipleent the ethods specified in the class definition SoftaxCrossEntropy. This class inherits the base Criterion class. Again no helper ethods should be needed and you should stride to keep your code as siple as possible. Use e-8 for eps... SoftaxCrossEntropy.forward Ipleent the softax cross entropy operation on a batch of output vectors. Hint: Add a class attribute to keep track of interediate values necessary for the backward coputation. Input shapes: x: batch size, 0 y: batch size, 0 Output Shape: out: batch size,

3 .. SoftaxCrossEntropy.backward Perfor the backward pass of softax cross entropy operation using interediate values saved in the forward pass. Output shapes: out: batch size, 0 3 Multi-Layer Perceptron MLP In this section of the hoework, you will be ipleenting a Multi-Layer Perceptron with an API siilar to popular Autoatic Differentiation Libraries like PyTorch, which you will be allowed and encouraged to use in the second part of the hoework. Provided in hw.py is a teplate the MLP class. Go through the functions of the given MLP class thoroughly and ake sure you understand what each function in the class does so that you can create a generic ipleentation that supports an arbitrary nuber of layers, types of activations and network sizes. This section is purely descriptive and for reference as you tackle the next probles. Nothing to ipleent... yet. The paraeters for the MLP class are: input size: The size of each individual data exaple. output size: The nuber of outputs. hiddens: A list with the nuber of units in each hidden layer. activations: A list of Activation objects for each layer. weight init fn: A function applied to each weight atrix before training. bias init fn: A function applied to each bias vector before training. criterion: A Criterion object to copute the loss and its derivative. lr: The learning rate. oentu: Moentu scale Should be 0.0 until copleting 3.6. nu bn layers: Nuber of atchnor layers start fro upstrea Should be 0 until copleting 3.5. The attributes of the MLP class The list of weight The list of weight atrix A list of bias A list of bias vector layers: A list of atchnor objects. Should be None until copleting 3.5. The ethods of the MLP class are: forward: Forward pass. Accepts a ini-batch of data and return a batch of output activations. backward: ackward pass. Accepts ground truth labels and coputes gradients for all paraeters. Hint: Use state stored in activations during forward pass to siplify your code. 3

4 zero grads: Set all gradient ters to 0. step: Apply gradients coputed in backward to the paraeters. train Already ipleented: Set the ode of the network to train. eval Already ipleented: Set the ode of the network to evaluation. Note: MLP ethods train and eval will be useful in 3.5. Note: Pay attention to the data structures being passed into the constructor and the class attributes specified initially. Saple constructor call: MLP784, 0, [64, 64, 3], [Sigoid, Sigoid, Sigoid, Identity], weight_init_fn, bias_init_fn, SoftaxCrossEntropy, 0.008, oentu=0.9, nu_bn_layers=0 3. Weight Initialization [0 points] Good initialization has been shown to ake a great difference in the training process. Ipleent two basic paraeter initialization functions rando noral weight init: Accepts two ints specifying the nuber of inputs and units. Returns an appropriately shaped ndarray with each entry initialized with an independent saple fro the standard noral distribution. zeros bias init: Accepts the nuber of bias units and returns an appropriately shaped ndarray with each entry initialized to zero. 3. Linear Classifier [9 points] Ipleent a linear classifier MLP with no hidden layers using the MLP class. We suggest you read through the entire assignent before you start ipleenting this part. For this proble, you will have to fill in the parts of the MLP class and pass the following paraeters: input size: The size of each individual data exaple. output size: The nuber of outputs. hiddens: An epty list because there are no hidden layers. activations: A single Identity activation. weight init fn: Function nae of a weight initializer function you will write for local testing and will be passed to you by the autograder. bias init fn: Function nae of a bias initializer function you will write for local testing and will be passed to you by the autograder. criterion: SoftaxCrossEntropy object. Consider that a linear transforation with an identity activation is siply a linear odel. You will have to ipleent the forward and backward function of the MLP class so that it can at least work as a linear classifier. All paraeters should be aintained as class attributes originally specified in the original handout code e.g. self.w, self.b There won t be any autograder tests that check for weight initializer ipleentations, but you will need to ipleent a rando noral initializer and a zeros initializer to test your ipleentation for

5 The step function also needs to ipleented, as it will be invoked after every backward pass to update the paraeters of the network the gradient descent algorith. After all parts are filled, train the odel for 00 epochs with a batch size of 00 and try visualizing the training curves using the utilities provided in test.py. 3.3 Non-Linearities [ points] Linear odels are cool, but we are here to build neural networks. Ipleent all of the non-linearities specified in hw.py. A saple ipleentation of the Identity activation is provided for reference. The output of the activation should be stored in the self.state variable of the class. The self.state variable should be further used for calculating the derivative during the backward pass. These are the following list of classes for which you would have to ipleent the forward and derivative function. The autograder will test these ipleentations individually.. Sigoid [4 points]. Hyperbolic Tangent Tanh [4 points] 3. Rectified Linear unit ReLU [4 points] 3.4 Hidden Layers [40 points] Update the MLP class that previously just supported a single fully connected layer a linear odel such that it can now support an arbitrary nuber hidden layers, each with an arbitrary nuber of units. Specifically, the hiddens arguent of the MLP class will no longer be assued to be an epty list and can be of arbitrary length arbitrary nuber of layers and contain arbitrary positive integers arbitrary nuber of units in each layer. The ipleentation should apply the Activation objects, passed as activations, to their respective layers. For exaple, activations[0] should be applied to the activity of the first hidden layer. While at risk of being pedantic, here is a clarification of the expected arguents to be passed to the MLP constructor once it can support arbitrary nubers of layers with an arbitrary assortent of activation functions. input size: The size of each individual data exaple. output size: The nuber of outputs. hiddens: A list of layer sizes nuber of units per layer. activations: A list of Activation objects to be applied after each linear transforation respectively. weight init fn: Function nae of a weight initializer function you will write for local testing and will be passed to you by the autograder. bias init fn: Function nae of a bias initializer function you will write for local testing and will be passed to you by the autograder. criterion: SoftaxCrossEntropy object. 5

6 3.5 atch Noralization atchnor [5 points] In this proble you will ipleent the atch Noralization technique fro Ioffe and Szegedy [05]. Ipleent the atchnor class and ake sure that all provided class attributes are set correctly. Apply the atchnor transforation of the first nu bn layers specified as an arguent to MLP. For the sake of the autograder tests, you can assue that batch nor will be applied to networks with sigoid non-linearities. 3.6 Moentu [0 points] Modify the step function present in the MLP class to include oentu in your gradient descent. The oentu value will be passed as a paraeter. Your function should perfor epoch nuber of epochs and return the resulting weights. Instead of calling the step function after copleting your backward pass, you would have to call the oentu function to update the paraeters. Also, reeber to invoke the zero grad function after each batch. Note: Please ensure that you shuffle the training set after each epoch by using np.rando.shuffle and generate a list of indices and perforing a gather operation on the data using these indices. 3.7 Training Statistics [0 points] In this proble you will be passed the MNIST data set provided as a 3-tuple of {train set, val set, test set}, each in the sae forat described in, and a series of network and training paraeters. training losses: A Nupy ndarray containing the average training loss for each epoch. training errors: A Nupy ndarray containing the classification error on the training set at each epoch Hint: You should not be evaluating perforance on the training set after each epoch, but copute an approxiation of the training classification error for each training batch. validation losses: A Nupy ndarray containing the average validation loss loss for each epoch. validation errors: A Nupy ndarray containing the classification error on the validation set after each epoch. Fill the function get training stats that, given a network, a dataset, a nuber of epochs and a batch size, trains the network and returns those quantities. Then execute the provided test file, that will run get training stats on MNIST for a given network and plot the previous arrays into files. Hand-in those iage files along with your code on autolab. Note: You will not be autograded on that part. Instead we will grade you anually by looking at your code and plots. We will check that your code and the statistics you obtain ake sense. Contrary to the other questions, we do not require an exact atch with the solution, only that you run your forward, backward and step functions in the right order, and use the partitions of the dataset correctly train and val at least, with shuffling of the training set between epochs. Note: We provide exaples of the plots that you should obtain. If you get siilar or better values at the end of the training loss around 0.3, error around 0., then you are very likely to get all the points. If you are a bit above but that the errors and losses still drop significantly during training, it s possible that you get all the points as well. References Sergey Ioffe and Christian Szegedy. atch noralization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/ , 05. URL 6

7 Appendix A Softax Cross Entropy Let s define softax, a useful function that offers a sooth and differentiable version of the ax function with a nice probabilistic interpretation. We denote the softax function for a logit x j of K logits outputs as σx j. e xj σx j = K Notice that the softax function properly noralizes the k logits, so we can interpret each σx j as a probability and the largest logit x j will have the greatest ass in the distribution. σx j = K j= e xj = For two distributions P and Q, we define cross-entropy over discrete events X as j= CE = P x log Qx = P x log Qx x X x X Cross-entropy coes fro inforation theory, where it is defined as the expected inforation quantified as log q of soe subjective distribution Q over an objective distribution P. It quantifies how uch inforation we our odel Q receive when we observe true outcoes fro Q it tells us how far our odel is fro the true distribution P. We iniize Cross-Entropy when P = Q. The value of cross-entropy in this case is known siply as the entropy. H = P x log P x = P x log P x x X x X This is the irreducible inforation we receive when we observe the outcoe of a rando process. Consider a coin toss. Even if we know the ernoulli paraeter p the probability of heads ahead of tie, we will never have absolute certainty about the outcoe until we actually observe the toss. The greater p is, the ore certain we are about the outcoe and the less inforation we expect to receive upon observation. We can noralize our network output using softax and then use Cross-Entropy as an objective function. Our softax outputs represent Q, our subjective distribution. We will denote each softax output as ŷ j and represent the true distribution P with output labels y j = when the label is for each output. We let y j = when the label is j and y j = 0 otherwise. The result is a degenerate distribution that will ai to estiate P when averaged over the training set. Let s foralize this objective function and take the derivative. 7

8 Lŷ, y = CEŷ, y = y i log ŷ i = = y i log σx i y i log ŷ, y = = = = e xi K y i log e xi K y i log e xi log e x k k= K y i log e xi + y i log k= K y i x i + y i log k= e x k e x k The next step is a little bit ore subtle, but recall there is only a single true label for each exaple and therefore only a single y i is equal to ; all others are 0. Therefore we can iagine expanding the su over i in the second ter and only one ter of this su will be and all the others will be zero. ŷ, y = K y i x i + log k= e x k Now we take the partial derivative and reeber that the derivative of a su is the su of the derivative of its ters and that any ter without x j can be discarded. ŷ, y = y j x j + = y j + = y j + = y j + K log k= K e xj e xj K e x k k= e x k That second ter looks very failiar, huh? 8

9 ŷ, y e xj = y j + x K j = y j + σx j = σx j y j = ŷ j y j After all that work we end up with a very siple and elegant expression for the derivative of softax with cross-entropy divergence with respect to its input. What this is telling us is that when y j =, the gradient is negative and thus the opposite direction of the gradient is positive: it is telling us to increase the probability ass of that specific output through the softax. atch Noralization atch Noralization coonly referred to as atchnor is a wildly successful and siple technique for accelerating training and learning better neural network representations. The general otivation of atchnor is the non-stationarity of unit activity during training that requires downstrea units to adapt to a non-stationary input distribution. This co-adaptation proble, which the paper authors refer to as internal covariate shift, significantly slows learning. Just as it is coon to whiten training data standardize and de-correlate the covariates, we can invoke the abstraction of a neural network as a hierarchical set of feature filters and consider the activity of each layer to be the covariates for the subsequent layer and consider whitening the activity over all training exaples after each update. Whitening layer activity across the training data for each paraeter update is coputationally infeasible, so instead we ake soe large assuptions that end up working well anyway. The ain assuption we ake is that the activity of a given unit is independent of the activity of all other units in a given layer. That is, for a layer l with units, individual unit activities consider each a rando variable x = {x k,..., x d } are independent of each other {x... x k... x d }. Under this independence assuption, the covariates are not correlated and therefore we only need to noralize the individual unit activities. Since it is not practical in neural network optiization to perfor updates with a full-batch gradient, we typically use an approxiation of the true full-batch gradient over a subset of the training data. We ake the sae approxiation in our noralization by approxiating the ean and variance of the unit activities over this sae subset of the training data. Given this setup, consider u to be the ean and σ the variance of a unit s activity over a subset of the training data, which we will hence refer to as a batch of data. For a training set X with n exaples, we partition it into n/ batches of size. For an arbitrary unit k, we copute the batch statistics u and σ and noralize as follows: u k σ k ˆx i x i u σ + ɛ x k i x k i u k 3 A significant issue posed by siply noralizing individual unit activity across batches is that it liits the set of possible network representations. A way around this is to introduce a set of learnable paraeters for 9

10 each unit that ensure the atchnor transforation can be learned to perfor an identity transforation. To do so, these per-unit learnable paraeters γ k and β k, rescale and reshift the noralized unit activity. Thus the output of the atchnor transforation for a data exaple, y i is given as follows, y i γˆx i + β 4 We can now derive the analytic partial derivatives of the atchnor transforation. Let L be the training loss over the batch and y the derivative of the loss with respect to the output of the atchnor i transforation for a single data exaple x i. = = γ 5 β = γ = σ Solve for ˆxi = = = β = γ = = σ σ 6 ˆx i 7 [ ] x i µ σ + ɛ x i µ σ + ɛ 3 0 = [ ] x i µ σ + ɛ = σ + ɛ + xi µ [ σ + ɛ = σ + ɛ + xi µ = σ + ɛ x i µ = σ + ɛ x i µ σ + ɛ 3 ] x i u + ɛ 3 x i u + ɛ x i µ x i µ 5 6 Now sub this expression for ˆxi into = = σ + ɛ x i µ σ + ɛ 3 x i µ 7 0

11 Notice that part of the expression in the second ter is just σ = = Now for the grand finale, let s solve for x i σ + ɛ + σ σ + ɛ σ x i µ x i µ x i = x i = [ ˆxi x i + σ σ + ˆx ] i x i x i 8 9 = + σ x i σ + 0 x i x i = = σ x i + + x i σ [ x i µ σ + ɛ x i = [ σ + ɛ ] + σ σ = [ ] σ + ɛ + σ = [ ] σ + ɛ + σ = [ ] σ + ɛ + σ = [ ] σ + ɛ + σ x i ] + σ + x i x i σ + 3 x i x i x j µ + 4 x i µ j= x i [ ] x i µ + 5 x i [ ] x i µ + x j 6 x i j= [ ] x i µ + [ ] 7 In suary, we have derived the following quantities required in the forward and backward coputation for a atchnor applied to a single unit: 8 Forward u = σ = ˆx i = x i u σ + ɛ x i 9 x i u 30 3

12 ackward = = γ 3 β = γ = σ = β = γ = σ 33 ˆx i 34 35