Deep (1) Matthieu Cord LIP6 / UPMC Paris 6

Transcription

1 Deep (1) Matthieu Cord LIP6 / UPMC Paris 6

2 Syllabus 1. Whole traditional (old) visual recognition pipeline 2. Introduction to Neural Nets 3. Deep Nets for image classification To do : Voir la leçon inaugurale de Yann LeCun au collège de France 2016 : Une question sur la leçon par mail matthieu.cord@lip6.fr Objet : Lecun cours polytech Indiquer l indice du temps dans la séquence

3 Representation 1. feature detection & representation 2. codewords dictionary image representation 3. Fei-Fei, Fergus, Torralba

4 Learning and Recognition Generative method: - graphical models 4. category models (and/or) classifiers Discriminative method: - SVM

5 Linear/kernel Classifier k(x,x1) α1 k(x,xi) αi f(x) αn xd k(x,xn)

6 Representation Learning and Recognition 1. feature detection & representation 2. codewords dictionary category models (and/or) classifiers category decision Fei-Fei, Fergus, Torralba

7 ~ Slide 7 of 47 Work on architecture: From shalow to deep From unsupervised to end-to-end supervised Image Feature extraction Local descriptors Feature coding Visual codes Pooling Image signature Classification Class label Dense SIFT K-Means Clustering Spatial Pyramids & Pooling SVM Classifier NN Kernels Fusion Local descriptors Visual dictionary Coding/pooling classifiers

8 Deep Visual Representations Image Label Shallow Architecture Difficult to model image data Models more complex functions Image Deep Architecture Multiple latent layers Goh

9 Deep Architectures: Bio & deep HMAX++ [Thériault et al., IEEE trans on IP, 2013]

10 Linear (binary) classifier: Perceptron [Rosenblatt, 1957,1962] a = w T x + w 0 g(a) = step(a) = step(w T x + w 0 ) +1 if w T x + w = otherwise Basic Neural Net

11 Linear discrimination -- as SVM Difference in training strategy

12 Learning rule for (linear) Perceptron Optimization (based on error loss) : Perceptron criterion: minimize distance to H for misclassified examples (x data, u desired output(-1,+1)): Algorithm: d(x, H ) = w T x i u i L(w) = x i missclass w T x i u i Repeat: Do if x i well classified with f(), continue else: w(t +1) = w(t) + η x i u i For all x i in training set (one epoch) Until convergence (or stop after a finite number of epochs) Rq: Hebb Rule: add to each connection a value proportional to input and output Discussion on having or not a learning rate

13 Neuron: non linearity function neuron k : " y k = g $ # j=0,d w jk x j % ' = g(a ) k & w jk : weight of the connection between cells j and k a k : activation of k g : activation function Sigmoid: 1 g(a) = 1 + e a g (a) = g(a)(1-g(a)) Rectified Linear Unit ReLU g(a) = max(0,a)

14 Multi Layer Perceptron (MLP) Feed forward multilayer net: several hidden layers

15 Expressive power of Multilayer Nets Biblio. book

16 Expressive power of Multilayer Nets

17 MLP training Min a cost function L(w) on training set {x,u} (empirical risk min) Gradient descent strategy for learning: Δw ij L w ij How to compute efficiently all weights updating for MLP: Algorithm: back-propagation of gradient

18 Application: image classification x y(x) M classes M output neurons 1 neuron / class

19 MLP example: brute force connection First Pb: Scalability Large images => extremely large number of trainable parameters

20 MLP example: brute force connection 2d Pb: Stability of the representation Expectation: Small deformation in the input space => similar representations Large (or unexpected) transfo in the input space => very dissimilar representations

21 MLP example: brute force connection Stability: Invariance/Robustness to (local) shifting, scaling, and other forms of (small) distortions?

22 MLP example: brute force connection Little or no invariance to shifting, scaling, and other forms of distortion Shift left

23 MLP example: brute force connection 154 input change from 2 shift left 77 : black to white 77 : white to

24 MLP example: brute force connection Scaling and other forms of distortions => same pb

25 Conclusion of MLP on raw data Brute force connection of images as input of MLP NOT a good idea No Invariance/Robustness of the representation because topology of the input data completely ignored Nb of weights grows largely with the size of the input image How keep spatial topology? How to limit the weight number?

26 How to limit the weight numbers? 1/ Locally connected neural networks Sparse connectivity: a hidden unit is only connected to a local patch (weights connected to the patch are called filter or kernel) Inspired by biological systems, where a cell is sensitive to a small sub-region of the input space, called a receptive field. Many cells are tiled to cover the entire visual field

27 How to limit the weight numbers? 2/ Shared Weights Hidden nodes at different locations share the same weights greatly reduces the number of parameters to learn Keep spatial information in a 2D feature map (hidden layer map) Computing responses at hidden nodes equivalent to convoluting input image with a linear filter (learned) A learned filter as a feature detector