Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 15

Transcription

1 Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 15 Slides adapted from Jordan Boyd-Graber Machine Learning: Chenhao Tan Boulder 1 of 21

2 Logistics HW3 available on Github, due on October 19 First prelim next Monday Final project team formation due by October 12 Machine Learning: Chenhao Tan Boulder 2 of 21

3 Recap Outline Recap Machine Learning: Chenhao Tan Boulder 3 of 21

4 Recap Logistic Regression as a Single-layer Neural Network Input layer x 1 x 2... Single Layer o 1... What is the activation used in logistic regression? What is the objective function used in logistic regression? x d Machine Learning: Chenhao Tan Boulder 4 of 21

5 Recap Nonlinearity Options Sigmoid tanh ReLU (rectified linear unit) f (x) = f (x) = exp(x) exp(x) exp( x) exp(x) + exp( x) f (x) = max(0, x) softmax x = exp(x) x i exp(x i ) Machine Learning: Chenhao Tan Boulder 5 of 21

6 Recap Nonlinearity Options Machine Learning: Chenhao Tan Boulder 6 of 21

7 Recap Loss Function Options l 2 loss l 1 loss Cross entropy (logistic regression) (y i ŷ i ) 2 i y i ŷ i i i y i log ŷ i Hinge loss (more on this during SVM) max(0, 1 yŷ) Machine Learning: Chenhao Tan Boulder 7 of 21

8 Recap Forward propagation algorithm How do we make predictions based on a multi-layer neural network? Store the biases for layer l in b l, weight matrix in W l W 1, b 1 W 2, b 2 W 3, b 3 W 4, b 4 x 1 x 2 o x d Machine Learning: Chenhao Tan Boulder 8 of 21

9 Recap Forward propagation algorithm Suppose your network has L layers Make prediction for an instance x 1: Initialize a 0 = x 2: for l = 1 to L do 3: z l = W l a l 1 + b l 4: a l = g(z l ) // g represents the nonlinear activation 5: end for 6: The prediction ŷ is simply a L Machine Learning: Chenhao Tan Boulder 9 of 21

10 Recap Quiz 1 How many parameters are there in the following feed-forward neural networks (assuming no biases)? x 1 x 2... x 100 h 1... h 50 A. 100 * * * 5 h 1... h 20 B. 100 * * * o 1... o 5 Machine Learning: Chenhao Tan Boulder 10 of 21

11 Recap Neural networks in a nutshell Training data S train = {(x, y)} Network architecture (model) ŷ = f w (x) Loss function (objective function) L (y, ŷ) Learning (next week) Machine Learning: Chenhao Tan Boulder 11 of 21

12 Recap Quiz 2 Which of the following statements is true? (Suppose that training data is large.) A. In training, K-nearest neighbors takes shorter time than neural networks. B. In training, K-nearest neighbors takes longer time than neural networks. C. In testing, K-nearest neighbors takes shorter time than neural networks. D. In testing, K-nearest neighbors takes longer time than neural networks. Machine Learning: Chenhao Tan Boulder 12 of 21

13 Outline Recap Machine Learning: Chenhao Tan Boulder 13 of 21

14 Structured data Spatial information after_she_was_told_she_was_a_good_girl/ Machine Learning: Chenhao Tan Boulder 14 of 21

15 Convolutional Layers Sharing parameters across patches input image or input feature map output feature maps a i j = k k w ij x ij i=1 j=1 Number of filters Filter shape Stride size tikz-convolutional-layer/convolutional-layer.tex Machine Learning: Chenhao Tan Boulder 15 of 21

16 Quiz 3 A concrete example (assuming no biases, convolution with 4 filters, ReLU, max pooling, and finally a fully-connected softmax layer) input image (10*10) 4@6*6 4@3*3 5*1 4 filters 5*5, stride 1 Max pooling 2*2, stride 2 Machine Learning: Chenhao Tan Boulder 16 of 21

17 Quiz 3 How many parameters are there in the following convolutional neural networks? (assuming no biases, convolution with 4 filters, ReLU, max pooling, and finally a fully-connected softmax layer) input image (10*10) 4@6*6 4@3*3 5*1 4 filters 5*5, stride 1 Max pooling 2*2, stride 2 Machine Learning: Chenhao Tan Boulder 16 of 21

18 Quiz 3 How many ReLU operations are performed on the forward pass? (assuming no biases, convolution with 4 filters, ReLU, max pooling, and finally a fully-connected softmax layer) input image (10*10) 4@6*6 4@3*3 5*1 4 filters 5*5, stride 1 Max pooling 2*2, stride 2 Machine Learning: Chenhao Tan Boulder 16 of 21

19 Structured data Sequential information My words fly up, my thoughts remain below: Words without thoughts never to heaven go. Hamlet Machine Learning: Chenhao Tan Boulder 17 of 21

20 Structured data Sequential information My words fly up, my thoughts remain below: Words without thoughts never to heaven go. Hamlet language activity history Machine Learning: Chenhao Tan Boulder 17 of 21

21 Structured data Sequential information My words fly up, my thoughts remain below: Words without thoughts never to heaven go. Hamlet language activity history x = (x 1,..., x T ) Machine Learning: Chenhao Tan Boulder 17 of 21

22 Recurrent Layers Sharing parameters along a sequence h t = f (x t, h t 1 ) Machine Learning: Chenhao Tan Boulder 18 of 21

23 Recurrent Layers Sharing parameters along a sequence h t = f (x t, h t 1 ) Machine Learning: Chenhao Tan Boulder 18 of 21

24 Long short-term memory A commonly used recurrent neural network in natural language processing f t = σ(w f [h t 1, x t ] + b f ) i t = σ(w i [h t 1, x t ] + b i ) o t = σ(w o [h t 1, x t ] + b o ) C t = tanh(w C [h t 1, x t ] + b C ) C t = f t C t 1 + i t C t h t = o t tanh(c t ) Machine Learning: Chenhao Tan Boulder 19 of 21

25 Wrap up Neural networks Network architecture (a lot more then fully connected layers) Convulutional layer Recurrent layer Loss function Machine Learning: Chenhao Tan Boulder 20 of 21

26 What is missing? How to find good weights? How to make the model work (regularization, even more architecture, etc)? Machine Learning: Chenhao Tan Boulder 21 of 21