Implement NN using NumPy

Transcription

1 Implement NN using NumPy Hantao Zhang Deep Learning with Python Reading: Recommendation for Using Python Install anaconda on your PC. If you already have installed anaconda, remember to update: conda update all Use spyder in anaconda. See 2 1

2 Numerical Python (Numpy) NumPy is at the core of nearly every scientific Python application or module since it provides a fast N-d array datatype that can be manipulated in a vectorized form. Various views of multi-dimensional arrays It provides advanced array slicing methods (to select array elements) and convenient array reshaping methods Broadcast of operations to each element Making code compact and easy to read See 3 Numpy - ndarray NumPy's main object is the homogeneous multidimensional array called ndarray. This is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers. Typical examples of multidimensional arrays include vectors, matrices, images and spreadsheets. Dimensions usually called axes, number of axes is the rank [7, 5, -1] An array of rank 1, i.e., it has 1 axis of length 3 [ [ 1.5, 0.2, -3.7], An array of rank 2, i.e., it has 2 axes, the first [ 0.1, 1.7, 2.9] ] length 3, the second of length 3 (a matrix with 2 rows and 3 columns 2

3 Numpy array, broadcast In[1]: import numpy as np In[2]: a = np.array([1,2,3], float) In[3]: b = np.array([5,2,6], float) In[4]: a + b Out[4]: array([ 6., 4., 9.]) In[5]: a - b Out[5]: array([-4., 0., -3.]) In[6]: a * b Out[6]: array([ 5., 4., 18.]) In[7]: b**a Out[7]: array([ 5., 4., 216.]) In[8]: b**2 Out[8]: array([ 25., 4., 36.]) In[9]: a = np.array([[1, 2], [3, 4], [5, 6]], float) In[10]: b = np.array([-1, 3], float) In[1]1: a+b Out[11]: array([[ 0., 5.], [ 2., 7.], [ 4., 9.]]) In[12]: a*a Out[12]: array([[ 1., 4.], [ 9., 16.], [ 25., 36.]]) In[13]: a**2 Out[13]: array([[ 1., 4.], [ 9., 16.], [ 25., 36.]]) Numpy dot, shape, reshape, newaxis In[14]: v1 = np.array(range(0, 5)) In[15]: v2 = np.arange(5) In[16]: v1 Out[16]: array([0, 1, 2, 3, 4]) In[17]: v2 Out[17]: array([0, 1, 2, 3, 4]) In[18]: v1.dot(v2) Out[18]: 30 In[19]: np.dot(v1,v2) Out[19]: 30 In[20]: v1.shape Out[20]: (5,) In[21]: v3 = v1.reshape(1,5) In[22]: v3 Out[22]: array([[0, 1, 2, 3, 4]]) In[23]: v1[0] Out[23]: 0 In[24]: v3[0] Out[24]: array([0, 1, 2, 3, 4]) In[25]: v1[1] = 10 In[27]: v1 Out[27]: array([ 0, 10, 2, 3, 4]) In[28]: v3 Out[28]: array([[ 0, 10, 2, 3, 4]]) In[29]: v4 = v1[:, np.newaxis] In[30]: v4.shape Out[30]: (5, 1) In[31]: np.dot(v1,v4) Out[31]: array([30]) In[32]: np.dot(v3,v4) Out[32]: array([[30]]) In[33]: np.dot(v4,v3) Out[33]: array([[ 0, 0, 0, 0, 0], [ 0, 1, 2, 3, 4], [ 0, 2, 4, 6, 8], [ 0, 3, 6, 9, 12], [ 0, 4, 8, 12, 16]]) 3

4 Numpy Slicing ndarray In[1]: a = np.arange(9).reshape(3,3)+1 In[2]: a Out[2]: array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) In[3]: print(a[0]) Out[3]: [1 2 3] In[4]: print(a[1,2]) Out[4]: 6 In[5]: print(a[1,1:3]) Out[6]: [5 6] In[7]: print(a[:,1]) Out[7]: [2 5 8] In[8]: a[1,2] = 10 In[9]: a Out[9]: array([[ 1, 2, 3], [ 4, 5, 10], [ 7, 8, 9]]) In[10]: a[:,0] *= -1 In[11]: a Out[11]: array([[-1, 2, 3], [-4, 5, 10], [-7, 8, 9]]) In[12]: b = a.t In[13]: b Out[13]: array([[-1, -4, -7], [ 2, 5, 8], [ 3, 10, 9]]) In[14]: b[1,:] += 5 In[15]: b Out[15]: array([[-1, -4, -7], [ 7, 10, 13], [ 3, 10, 9]]) In[16]: a Out[80]: array([[-1, 7, 3], [-4, 10, 10], [-7, 13, 9]]) Numpy zeros, ones, rand In[30]: a = np.zeros(5) In[31]: b = np.ones(12) In[32]: a Out[32]: array([ 0., 0., 0., 0., 0.]) In[33]: b In[38]: m = np.random.rand(3,3) Out[33]: array([ 1., 1., 1.,..., In[39]: 1., m 1., 1.]) In[34]: c = b.reshape(3, 4) In[35]: c Out[35]: array([[ 1., 1., 1., 1.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]) In[36]: d = b.reshape(4, 3)+1 In[37]: d Out[37]: array([[ 2., 2., 2.], [ 2., 2., 2.], [ 2., 2., 2.], [ 2., 2., 2.]]) Out[39]: array([[ , , ], [ , , ], [ , , ]]) In[40]: n = np.dot(d, m) In[41]: n Out[42]: array([[ , , ], [ , , ], [ , , ], [ , , ]]) In[42]: n = np.dot(m, c) In[43]: n Out[44]: array([[ , , , ], [ , , , ], [ , , , ]]) 4

5 Numpy ndarray attributes ndarray.ndim the number of axes (dimensions) of the array i.e. the rank. ndarray.shape the dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m). The length of the shape tuple is therefore the rank, or number of dimensions, ndim. ndarray.size the total number of elements of the array, equal to the product of the elements of shape. ndarray.dtype an object describing the type of the elements in the array. One can create or specify dtype's using standard Python types. NumPy provides many, for example bool_, character, int_, int8, int16, int32, int64, float_, float8, float16, float32, float64, complex_, complex64, object_. ndarray.itemsize the size in bytes of each element of the array. E.g. for elements of type float64, itemsize is 8 (=64/8), while complex32 has itemsize 4 (=32/8) (equivalent to ndarray.dtype.itemsize). ndarray.data the buffer containing the actual elements of the array. Normally, we won't need to use this attribute because we will access the elements in an array using indexing facilities. Numpy sum, mean, max, min, transpose In[46]: arr = np.arange(10, 20). reshape(2,5) In[47]: arr Out[47]: array([[10, 11, 12, 13, 14], [15, 16, 17, 18, 19]]) In[48]: arr.sum() Out[48]: 145 In[49]: arr.mean() Out[49]: 14.5 In[50]: arr.max() Out[50]: 19 In[51]: np.sum(arr**2) Out[51]: 2185 In[52]: arr.t Out[52]: array([[10, 15], [11, 16], [12, 17], [13, 18], [14, 19]]) 5

6 Using Numpy arrays wisely Array operations are implemented in C or Fortran Optimized algorithms - i.e. fast! Python loops (i.e. for i in a: ) are much slower Prefer array operations over loops, especially when speed is important It also produces shorter code, often more readable mlnn.py The major features: 1. Provides a general, fully connected, multi layer neural network, 2. The cost function is quadratic error, though replaceable. 3. The activation function can be changed in the constructor 4. The mini patch size is a parameter 5. An epoch is roughly using every training example exactly once. 6

7 mlnn.py from sklearn.utils import shuffle import numpy from matplotlib.colors import ListedColormap import matplotlib.pyplot as plt def square_cost(actual_output, y): "" Return the sum of square cost, where y is the desired output. """ return np.sum((actual_output - y)**2)/2 def square_cost_derivative(actual_output, y): "" Return the partial derivatives of the cost function for the actual output. """ return (actual_output - y) mlnn.py def sigmoid(x): """The sigmoid function.""" return 1.0/(1.0+np.exp(-x)) def sigmoid_prime(x, y): """Derivative of the sigmoid function.""" return y*(1-y) def tanh(x): """ the hyperbolic tangent function """ return (1.0 - np.exp(-2*x))/(1.0 + np.exp(-2*x)) def tanh_prime(x, y): """ the derivative of hyperbolic tangent function """ return (1 + y)*(1 - y) 7

8 Computation in General NN There are L layers, l {1, 2,, L} plus input layer (l = 0); each layer is fully connected to the next. An example of 4-layer NN: 4 weight matrices: W 0 W 1 W 2 W 3 W i [j,k] = link weight from j th neuron in layer i to k th neuron in layer (i+1) 4 outputs: y 0 y 1 y 2 y 3 y 4 (plus y 0 x, the input) 4 sums: y 0 z 0 y 1 z 1 y 2 z 2 y 3 z 3 y 4 z i = y i W i + b i z i [j] = y[0]w[0, j] + y[1]w[1, j] + + y[r-1]w[r-1, j] + b i [j] where y = y i, W=W i, and r is the row number of W i y i+1 = a(z i ), where a is the activation function. Computation in General NN There are L layers, l {1, 2,, L} plus input layer (l = 0); each layer is fully connected to the next. An example of 4-layer NN: 4 weight matrices: W 0 W 1 W 2 W 3 W i [j,k] = link weight from j th neuron in layer i to k th neuron in layer (i+1) 4 outputs: y 0 y 1 y 2 y 3 y 4 (plus y 0 x, the input) 4 sums: y 0 z 0 y 1 z 1 y 2 z 2 y 3 z 3 y 4 z i = y i W i + b i z i [j] = y[0]w[0, j] + y[1]w[1, j] + + y[r-1]w[r-1, j] + b i [j] where y = y i, W=W i, and r is the row number of W i y i+1 = a(z i ), where a is the activation function. 8

9 class Network class Network: def init (self, net_arch): np.random.seed(60) # for reproducibility self.activation = sigmoid self.activation_derivative = sigmoid_prime # self.activation = tanh # self.activation_derivative = tanh_prime self.cost_derivative = square_cost_derivative self.num_layers = len(net_arch) - 1 self.sizes = net_arch self.weights = [np.random.randn(x, y) for x, y in zip(net_arch[:-1], net_arch[1:])] self.weights = np.asarray(self.weights) self.biases = np.asarray([np.random.randn(y) for y in net_arch[1:]]) Feed Forward Computation # It's simply feed forward computation. def feedforward(self, a): for w, b in zip(self.weights, self.biases): a = self.activation(np.dot(a, w) + b) return a # Training using backpropagation def SGD(self, train_x, train_y, epochs, batch_size, eta, test_data=none): # Plot the output of a binary function def plot_decision_regions(self, X, y, points=200): 9

10 XOR Example if name == ' main ': nn = Network(net_arch=[2,3,4,1]) print('net architecture:', nn.sizes) print('initial weights:\n', nn.weights) train_x = numpy.array([[0, 0], [0, 1], [1, 0], [1, 1]]) train_y = numpy.array([0, 1, 1, 0]) nn.sgd(train_x, train_y, 100, 1, 3) for a in train_x: print(a, nn.feedforward(a)) net.plot_decision_regions(train_x, train_y) # for binary input only Final prediction [0 0] [0 1] [1 0] [1 1] Computation in General NN There are L layers, l {1, 2,, L} plus input layer (l = 0); each layer is fully connected to the next. An example of 4-layer NN: 4 weight matrices: W 0 W 1 W 2 W 3 W i [j,k] = link weight from j th neuron in layer i to k th neuron in layer (i+1) 4 outputs: y 0 y 1 y 2 y 3 y 4 (plus y 0 x) 4 sums: y 0 z 0 y 1 z 1 y 2 z 2 y 3 z 3 y 4 z i = y i W i + b i a(z 0 ) = y 1 a(z 1 ) = y 2 a(z 2 ) = y 3 a(z 3 ) = y 4 y i+1 = a(z i ) 10

11 Backpropagation Multi-layer NN supervised learning Use Gradient Decent, and require differentiable activation and cost functions Error is propagated back through earlier layers Main Idea: Given a set of input/output examples { (x, y) }. Define the network as a function f(w, x) on weights w and x. Define the cost, say C = ½ f(w,x) y 2, and try to minimalize it. For each example (x, y), repeat the following: 1. compute f(w, x) 2. compute C/ w 3. update w by w = w ( C/ w) to decrease C. Computation in General NN There are L layers, l {1, 2,, L} plus input layer (l = 0); each layer is fully connected to the next. An example of 4-layer NN: Define C ½ (y L y) 2, i C/ y i, i C/ z i, then we have L = (y L y) i = a (z i ) i+1 i = W i i 4 weight matrices: W 0 W 1 W 2 W 3 W i [k,j] = link weight from k th neuron in layer i to j th neuron in layer (i+1) 4 outputs: y 0 y 1 y 2 y 3 y 4 (plus y 0 = x) 4 sums: y 0 z 0 y 1 z 1 y 2 z 2 y 3 z 3 y 4 z i = y i W i, y i+1 = a(z i ) 4 i & i : From z i [j] = y i [0]W i [0, j] + y i [1]W i [1, j] + + y i [r-1]w i [r-1, j] + b i [j] we have C/ b i = C/ z i = i, C/ W i [k,j] = y i [k]( C/ z i [j]) (or C/ W i = (y i 1) (1 i )) 11

12 Proof of i = W i i for i = L 1,, 2, 1 There are L layers, l {1, 2,, L} plus input layer (l = 0); each layer is fully connected to the next. W i [j,k] = link weight from j th neuron in layer i to k th neuron in layer (i+1) Assume z i = y i W i, i C/ y i, i C/ z i, L = (y L y), i = i+1 a (z i ). Assume also i is dropped from z i, y i, W i, i, z i for convenience: = W means i = W i i, equivalent to [j] = k W[j, k] [k] for all j. C/ y means i C/ y i, equivalent to [k] = C/ y[k] for all k. From z i = y i W i, we have z[k] = j y[j]w[j,k] and z[k]/ y[j] = W[j,k]. Hence [j] = C/ y[j] = k ( z[k]/ y[j])( C/ z[k]) = k W[j,k] [k], that is the same as [j] = k W[j, k] [k]. So = W. Note: From z i = y i W i, we get i = W i i. Here, W i serves dual roles: function h, where z i = h(y i ) and Jacobian matrix z/ y. That is the case for all linear function h. See z i = y i W i, y i+1 = a(z i ) C ½ (y L y) 2, i C/ y i, i C/ z i L = (y L y), i = a (z i ) i+1 and i = W i i C/ b i = i, C/ W i = (y i 1) (1 i ) def backprop(self, x, y): # feedforward y_act = x # y0 = x, the input y_acts = [x] # list to store all the activation vectors, y s, layer by layer z_wsums = [] # list to store all the weighted sum vectors, layer by layer for b, w in zip(self.biases, self.weights): z = np.dot(y_act, w) + b z_wsums.append(z) y_act = self.activation(z) y_acts.append(y_act) # backward propagation # nabla1: place holder for all layers of weights and biases nabla1_b = list(range(self.num_layers)) nabla1_w = list(range(self.num_layers)) 12

13 mnist.pkl.gz z i = y i W i, y i+1 = a(z i ) C ½ (y L y) 2, i C/ y i, i C/ z i L = (y L y), i = a (z i ) i+1, and i = W i i C/ b i = i, C/ W i = (y i 1) (1 i ) def backprop(self, x, y): # backward propagation theta = self.cost_derivative(y_act, y) delta = self.activation_derivative(z, y_act) * theta y_hat = np.expand_dims(y_acts[-2], axis=1) delta_hat = np.expand_dims(delta, axis=0) nabla1_w[-1] = np.dot(y_hat, delta_hat) nabla1_b[-1] = delta for l in range(2, self.num_layers+1): theta = np.dot(self.weights[-l+1], delta) delta = self.activation_derivative(z_wsums[-l], y_acts[-l]) * theta y_hat = np.expand_dims(y_acts[-l-1], axis=1) delta_hat = np.expand_dims(delta, axis=0) nabla1_w[-l] = np.dot(y_hat, delta_hat) nabla1_b[-l] = delta return (nabla1_b, nabla1_w) z i = y i W i, y i+1 = a(z i ) C ½ (y L y) 2, i C/ y i, i C/ z i L = (y L y), i = a (z i ) i+1, and i = W i i C/ b i = i, C/ W i = (y i 1) (1 i ) def update_batch(self, batch_x, batch_y, eta): nabla_w = np.zeros(self.weights.shape) nabla_b = np.zeros(self.biases.shape) for x, y in zip(batch_x, batch_y): nabla1_b, nabla1_w = self.backprop(x, y) nabla_w = nabla_w + nabla1_w nabla_b = nabla_b + nabla1_b self.weights -= eta*nabla_w self.biases -= eta*nabla_b # an alternative: # self.weights -= eta*nabla_w / batch_size # self.biases -= eta*nabla_b / batch_size 13

14 def SGD(self, train_x, train_y, epochs, batch_size, eta, test_data=none): n = len(train_x) if test_data: n_test = len(test_data[0]) for j in range(epochs): # stochastic requires to create random batches. train_x, train_y = shuffle(train_x, train_y) batches_x = [train_x[k : k+batch_size] for k in range(0, n, batch_size)] batches_y = [train_y[k : k+batch_size] for k in range(0, n, batch_size)] batches_x = np.asarray(batches_x) batches_y = np.asarray(batches_y) for batch_x, batch_y in zip(batches_x, batches_y): self.update_batch(batch_x, batch_y, eta) if test_data: print("epoch {} : {} / {}".format(j, self.evaluate(test_data), n_test)); 14