ConvolutionalNN's... ConvNet's... deep learnig

Transcription

1 Deep Learning ConvolutionalNN's... ConvNet's... deep learnig Markus Thaler, TG208 Martin Weisenhorn, TB

2 Neural Networks Classification: up to now decision criteria derived from patters (features) classifier depends on how well the patterns correspond with the assumptions on the statistical models statistical properties are often not known (e.g. probability distribution, etc.) Neural Networks make no assumptions about statistical models a generalized model is trained with a set of pattern verctors

3 ... Neural Networks Perceptron for two pattern classes simplest form of neural network based on weighted sum of inputs x 1 x 2 x n 1 w 1 w 2... w n w n+1 d = n n ( x) wi xi + w + 1 i= activation function parameters w i retrived through training activation if if n i= 1 n i= 1 w x i w x i i i > w < w the decision surface (d(x) = 0) constitutes a n-dim hyperplane n+ 1 n

4 ... Neural Networks Discussion training - only two classes - iterative algorithm for linearly separable classes (see Gonzalez) - but in most cases pattern classes are not seperable Multi-layer neural networks - multiple classes - more than one layer of perceptrons - preceptron function non-linear (but not signum) - independent on seperability d(x) =

5 Multilayer Networks Example: 3 layer architecture fully connected MLP 1) x 1 x 2 x 3 x N weights w a input layer weights w b... hidden layer weights w c 1) Multi Layer Perceptron output layer activation function shape 1 shape 2 shape 3 shape

6 ... Multilayer Networks Example pattern vectors - 48 samples from normalized signatures reference patterns noisy reference patterns

7 ConvNets: CNN's 1) Convolutional Neural Networks (CNN') [1] biology inspired variants of Multi Layer Perceptrons cells of visual cortex - sensitive to sub-regions of the visual field (receptive fields) - exploit strong spatially local correlation in natural images - two types of cells simple cells respond to edge like pattern complex cells have larger receptive field, invariant to exact position of pattern CNN's emulate the behavior of the visual cortex 1) Convolutional Neural Networks

8 Overall View Basic Idea feature extractor: many different architectures are used datalab, "beyond image net", Thilo Stadelmann, Oliver Dürr

9 CNN's Feature Extraction: commonly used layers and functions Input layer Convolution or filtering layer - kernel with trained weights (filter, maps) - weights shared by all receptive fields - result: feature map or rectified feature map (see ReLU) Pooling layer - shrinks feature maps (reduces dimension of feature maps) Fully connected layer - classification by a fully connected neural neutwork ReLU (rectifier linear unit) - activation function: f(x) = max(0, x) speeds up training

10 Input Layer Input Layer original image may be grayscale, color - number of channels Hyperparamters 1) - image size - number of channels (gray, color) 1) Hyperparamters: tuning parameters selected by user soure: google research blog

11 Convolution Layer Convolution each filter - produces one map in next layer - combines generally all maps from previous layer each element of a feature map neuron (shared weights) input layer & feature maps in general zero padded 4 filters 6 filters many layers deep learning

12 ... Convolutional Layer Convolution filter sizes are typically 3x3 to 11x11 elements Hyperparamters - number of filters - size of filters - stride (often = 1, pooling for size reduction) Example - 96 example filters learned by Krizhevsky et al. (see also [3])

13 Pooling Layer Pooling subsampling of feature maps usually max-pooling Hyperparameters - pooling size - stride - (pooling type)

14 Fully Connected Layer Fully connected layer "tradional neural network" MLP Hyperparameters - number of neurons - number of layers

15 ReLU Rectifier Linear Units "activation function" speeds up training

16 Overall Architecture Basic architecture Input Feature Extractor Classifier Output convolutional layer, pooling layer, ReLU Hyperparamters(feature extractor) - number and type of layers - combination (sequence) of layers

17 Training Theory overview in general backpropagation - weights are updated according to cost function supervised classification for details see Literature (e.g. [4]) and WEB Training full training: train everything pre-training: use pre-trained system - need less samples - can be used in different application domains computational intensive - need powerfull computers (GPUs)... talk to somebody who has experience

18 Software Software and Libraries Examples: Caffe, Matlab, Tensorflow, Theano, etc. mostly open source and often by universities overview see

19 Literatur Literature/Information used [1] [2] [3] ( [4] Videos [5] How Convolutional Neural Networks work, Brandon Rohrer, Papers [6] Using Convolutional Neural Networks for Image Recognition, Samer Hijazi, et al. IP Group, Cadence [7] Beyond ImageNet - Deep Learning in Industrial Practice, Thilo Stadelmann, et al., Datalab ZHAW... an lot more on the WEB

20