Neural Nets & Deep Learning

Transcription

1 Neural Nets & Deep Learning

2 The Inspiration Inputs Outputs Our brains are pretty amazing, what if we could do something similar with computers? Image Source: _med.jpeg

3 The Perceptron Idea dates from the 1950s Let i be our input features, try to predict a binary class Output = σ(i1w1 + i2w2 + i3w3 + i4w4) σ is some function that scales the sum to between 0 and 1. If output < 0.5 we say class poison, otherwise we predict class edible Color i1 Size of Cap i2 w1 w2 σ Height i3 Found in clusters? i4 Poisonous or Edible w3 w4 Image Source: -function-explained-plain-english-hsu/

4 Perceptron: Loss We have a model, how do we find weights that gives the best results? We need to define a measure of how bad our starting weights are (cost/loss) Must be smooth A small change in the weights = A small change in the loss This rules out accuracy on our training data, will change in discrete jumps Use the decimal output to represent a confidence of a class Can tweak for regression Bad Good Image Source: _happysad-800x480.jpg

5 Perceptron: Training with Gradient Descent Once we have our loss, we perform gradient descent. See what happens if we tweak our weights in different directions, If we get an improvement in our loss, keep going! Take slope at point and move in negative direction Once we stop improving, we re done! Image Source:

6 Image Source: d440fa32fc9e5300c894 Perceptron: The Problem So we have a model and a way to train it, how do we do? Not great : ( You might have noticed that what we re doing is essentially linear regression except squishing the output at the end Can only divide regions linearly Similar to how regression can only fit linear relations

7 Feed Forward Neural Net: The Solution What if we have multiple layers? h1 = i1w1,1 + i2w2,1 + i3w3,1 o = σ(h1wh1 + h2wh2 + h3wh3) Good idea, but still has the linear-ness problem Multiple linear combinations stacked is still a linear combination Inputs i1 h1 i2 h2 i3 h3 σ Output

8 Feed Forward Neural Net: The Solution Another idea: What if we squish the hidden layer as well as the output? h1 = σ(i1w1,1 + i2w2,1 + i3w3,1 ) Similar to what we did with log space! o = σ(h1wh1 + h2wh2 + h3wh3) No longer can only fit around linear boundaries! Note: Gradient descent is now a bit harder, but essentially the same idea New algorithm called backprop Inputs i1 σ1 i2 σ2 i3 σ3 σo Output

9 People came up with backprop & FFNNs in the 70s / 80s, but they were very, very slow (and hard to interpret) What changed?

10 GPUs

11 Feed Forward Neural Net: The Solution High-end 3D video games require lots of processing power But most of this processing is simple adds and multiplies CPUs are good at doing a lot of different operations pretty fast GPUs do simple operations SUPER fast (and in parallel) Multiply and add That sounds familiar. Neural Nets are now super fast!

12 Types of Neural Nets

13 Image Source: Applying NNs to Images Feed forward works for You have some features Predict an output What about images? Naive way: Take each pixel value and feed it in as a different feature (in a 100x100 image that s 10,000 features!) Is there a better way?

14 Convolutional Neural Nets (CNNs) Pass the same small neural net over different parts of the image Called a filter or kernel Learn the weights of this neural net These filters start picking up on higher level features Vertical line, horizontal line, etc. Stack enough of these and you get really good results! 99.9% on digit recognition! Image Source: es g002.png

15 Applying NNs to Series How do we work with series of things of non-fixed length? A sentence Stock prices A person s medical history One option is CNNs, same idea but in 1 dimension instead of 2 Fast but not the best, especially for language Doesn t capture long-term information, only the size of the filter

16 Image Source: deploy/html/images/technologies g002.png Recurrent Neural Nets (RNNs) Idea: Take a FFNN and have it output two things An answer you can use A state that it passed onto feed back into itself as a new input State contains info we want to keep around for the future Work really well for language, but is slow! French Word State English Word

17 Representing Words - Stock prices are easy, just numbers - What about words? - Could create a class for each word, apple is 55, food is 72 - These numbers are arbitrary, apple and pear are similar but could have totally different numerical ids - Idea: Create an embedding - Try to summarize a word with a point in space Similar words are near each other in the space - How do we create these embeddings?

18 Image Source:

19 Word2Vec Apple Words that are similar are used in similar contexts Make a neural net that given a word, predict the words around it Take the values that show up in the hidden layer and use that to represent the word I am going to eat a pear today I love eatings apples Both words are near eat and I [22.5, 45.2, ] List of 50 numbers Problems Hypothesis is kind of flawed, I am happy and I am sad Bias in language! Hidden Layer of 50 neurons 20% near eat 20% near I 60% other stuff

20 Reinforcement Learning How do we teach neural nets to play games? Create a neural network that given the current game board, predict a move Pretty much random at first If we make some moves and end up winning/scoring a point/surviving, reward it with our loss function! If we lose/get scored on/die don t reward those actions Slowly get better over time! Image Source: ver.original.width-440_q7hc1rg.jpg

21 Generative Adversarial Networks - Take the problem of colorizing black and white photos - Create two neural nets - One will generate color images from black and white ones One will try to discriminate between the generated color images and real color images - Slowly the generator will get better at creating forgeries - When you re done training, you have a program that can colorize images! Figure from pix2pix Isola et al

22 Other Tricks - A lot of deep learning research right now is trying random things and seeing what gives good results - Some tricks that people have come up with - Use a different activation for the middle instead of the standard σ (ReLU, ELU, etc.) Normalize input to layers (BatchNorm) Use fancy optimizers that do a lot of math (AdamOptimizer) To prevent overfitting, disable neurons randomly during training (Dropout) - Handicaps your model during training, but really helps! - Especially when you run through training data multiple times Image Source: HEJHD4z893g.png

23 Creating a Neural Net Model 1. Get your data, clean it, decide what your input/output is 2. Define your model What type/ tricks, how many layers This is hard! 3. Define your loss function What am I using to measure my performance? 4. Decide how you re going to train How long, with what optimizer, on one computer or many, etc.

24 Pros & Cons - Pros - Are really, REALLY powerful/accurate Very easy to make larger/smaller, lots of hyperparameters Work well in vastly different fields, same central ideas - Cons - - Are blackboxes, things go in, answers come out but often unclear why - Especially troubling when it gets things wrong - Application to medicine Still very simple, lack large-scale understanding Classic ML dangers: overfitting, bias, etc. People are working on fixing these things! Image Source: 0SYqipwn2DpAw.png

25 Yann LeCun - Giving a talk in List 4PM Father of Convolutional Neural Nets Director of research at Facebook Talk Title: How Could Machines Learn as Efficiently as Animals and Humans? - Check it out!

26 Demos: YOLO

27 Demos: Style Transfer

28 Demo: Image Translation with CycleGAN

29 Demo: Image Translation with CycleGAN

30 Demo: Celebrity Generator

31 Demo: Rock Paper Scissors - Play rock paper scissors with your camera