3 October, 2013 MVA ENS Cachan. Lecture 2: Logistic regression & intro to MIL Iasonas Kokkinos

Size: px

Start display at page:

Download "3 October, 2013 MVA ENS Cachan. Lecture 2: Logistic regression & intro to MIL Iasonas Kokkinos"

Tiffany Dorsey
5 years ago
Views:

1 Machine Learning for Computer Vision 1 3 October, 2013 MVA ENS Cachan Lecture 2: Logistic regression & intro to MIL Iasonas Kokkinos Iasonas.kokkinos@ecp.fr Department of Applied Mathematics Ecole Centrale Paris Galen Group INRIA-Saclay

2 Administrative details 2 Make sure you have all received my from last week

3 Lecture outline 3 Recap & problems of linear regression Logistic Regression Introduction to Multiple Instance Learning

4 4 Learning problem Recover input-output mapping Output: y Input: x Method: f Parameters: w Aspects of the learning problem Identify methods that fit the problem setting Determine parameters that properly classify the training set Measure and control the complexity of these functions

5 Linear Classifiers Find linear expression (hyperplane) to separate positive and negative examples 5 Feature coordinate j x x i i positive : negative : x x i i w + b w + b 0 < 0 Each data point has a class label: +1 ( ) y t = -1 ( ) Feature coordinate i

6 Learning problem formulation 6 Given: Training set of feature-label pairs Wanted: simple that works well for simple: penalizing function complexity, to guarantee generalization works well: quantified by loss criterion

7 Linear regression 7 Linear Loss function: quantify appropriateness of for Introduce vector notation

8 Inappropriateness of quadratic loss We chose the quadratic cost function for convenience Single, global minimum & closed form expression 8 But does it indicate classification performance? Computed Decision Boundary Linear Fit Desired decision boundary

9 Inappropriateness of quadratic loss Consider transformation: 9 Quadratic loss is not robust to outliers and penalizes outputs that are `too good

10 Lecture outline 10 Recap & problems of linear regression Logistic Regression Training criterion formulation Optimization Interpretation Application to boundary detection Introduction to Multiple Instance Learning

11 Probabilistic interpretation of least squares 11 Outputs: continuous random variables linear function of inputs + zero mean Gaussian r.v. Optimal w: maximizes likelihood of observed outputs But the labels are discrete!

12 A probabilistic criterion for training a classifier 12 Training set: y: discrete observations: model as samples from Bernoulli distribution Find w that maximizes the likelihood of labels in the training set

13 Sigmoidal function & logistic regression 13 sigmoidal Training criterion from previous slide: How do we optimize it with respect to w? What does it mean?

14 Lecture outline 14 Recap & problems of linear regression Logistic Regression Training criterion formulation Optimization Interpretation Application to boundary detection Introduction to Multiple Instance Learning

15 Maximization by gradient ascent 15 Gradient ascent:

16 Maximization with Newton-Raphson 16 Newton method for finding roots of 1D function : Newton method for finding maxima of 1D function : condition Newton-Raphson method for finding maxima of N-D function :

17 Newton-Raphson for Logistic Regression 17 Jacobian: Hessian: In statistics: Iteratively Reweighted Least Squares (IRLS)

18 Lecture outline 18 Recap & problems of linear regression Logistic Regression Training criterion formulation Optimization Interpretation Application to boundary detection Introduction to Support Vector Machines

19 A compact expression for the loss 19 Using, :

20 Log loss: 20

21 Log loss vs. quadratic loss 21

22 Logistic vs Linear Regression 22 Logistic regression is more robust Linear Regression Logistic Regression

23 Lecture outline 23 Recap & problems of linear regression Logistic Regression Training criterion formulation Optimization Interpretation Application to boundary detection Introduction to Multiple Instance Learning

24 Boundary detection problem Object/Surface Boundaries 24

25 25 Signal-level challenges Poor contrast Shadows Texture

26 Machine Learning for Computer Vision Lecture 2 Fundamental challenges: can humans do it? 26

Learning-based approaches 27 Boundary or non-boundary?

combine the individual inputs. S. Konishi, A.Yuille, J.

Evaluating Edge Cues, IEEE PAMI, 2003 D. Martin, C. Fowlkes, J.

27 Learning-based approaches 27 Boundary or non-boundary? Use human-annotated segmentations Use any visual cue as input to the decision function. Use decision trees/logisitic regression/boosting/ and learn to combine the individual inputs. S. Konishi, A.Yuille, J. Coughlan, S.C. Zhu, Statistical Edge Detection: Learning and Evaluating Edge Cues, IEEE PAMI, 2003 D. Martin, C. Fowlkes, J. Malik. "Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture Cues", IEEE PAMI, 2004

28 Evaluation protocol 28 Threshold detector s output at certain level Match outputs to human ground-truth Count true/false positives (t/f)positives, misses (m) From precision, p and recall, r Quantify performance in terms of F-measure

29 29 Contours can be defined by any of a number of cues (P. Cavanagh) Slide credit: J. Malik, M. Maire

30 Cue-localization Gray level photographs 30 Objects from motion Objects from luminance Objects from disparity Line drawings Objects from texture Slide credit: J. Malik, M. Maire Grill-Spector et al., Neuron 1998

31 31 Local boundary cues Separate features per candidate orientation In specific: r (x,y) θ 1976 CIE L*a*b* colorspace Brightness Gradient BG(x,y,r,θ) Difference of L* distributions Color Gradient CG(x,y,r,θ) Difference of a*b* distributions Texture Gradient TG(x,y,r,θ) Difference of distributions of V1-like filter responses Slide credit: J. Malik, M. Maire

32 Boundary cues (horizontal) 32 Brightness Color (a, b channels) Texture

33 Boundary π/4 33 Brightness Color (a, b channels) Texture

34 Brightness π/2 34 Brightness Color (a, b channels) Texture

35 Brightness 3π/4 35 Brightness Color (a, b channels) Texture

36 Cue combinations with logistic regression 36 Slide credit: J. Malik, M. Maire

Exploiting global constraints: image Segmentation as Graph Partitioning Build a weighted graph G=(V,E) from image V: image pixels 37 E: connections between pairs of nearby

37 Exploiting global constraints: image Segmentation as Graph Partitioning Build a weighted graph G=(V,E) from image V: image pixels 37 E: connections between pairs of nearby pixels Partition graph so that similarity within group is large and similarity between groups is small -- Normalized Cuts [Shi & Malik 97] Slide credit: J. Malik, M. Maire

38 38 Wij small when intervening contour strong, strong when weak.. Slide credit: J. Malik, M. Maire

39 Normalized Cuts as a Spring-Mass system 39 Each pixel is a point mass; each connection is a spring: ( D W ) y = λdy Fundamental modes are generalized eigenvectors of (D - W) x = λdx Slide credit: J. Malik, M. Maire

40 Contour information from eigenvectors 40 Slide credit: J. Malik, M. Maire

41 Classifier 41 Logistic regression

42 Progress during the last 40 years 42 Humans Berkeley gpb, 08 Berkeley PB, 04 Canny+ Hysteresis ( 85) Prewitt, good feature= 5 years of work

43 Lecture outline 43 Recap & problems of linear regression Logistic Regression Training criterion formulation Optimization Interpretation Application to boundary detection Introduction to Multiple Instance Learning

44 Machine Learning for Computer Vision Lecture 2 Learning symmetry detection S. Tsogkas & I.K., Learning-based symmetry detection in natural images, ECCV

45 Multi-scale and multi-orientation features 45 S. Tsogkas & I.K., Learning-based symmetry detection in natural images, ECCV 2012

46 Learning boundary detection 46 Problem I: inconsistent orientation information Problem II: inconsistent location information

47 Sneaking into the fun room 47 mom s keychain grandma s keychain dad s keychain We know that dad cannot enter the fun room, either Which key should we try? Slide Credit: B. Babenko/T. Dietterich

48 Multiple Instance Learning 48 Typical Learning Multiple Instance Learning Positive bag: at least one instance should be positive Negative bag: no instance should be positive Slide Credit: K. Grauman

49 Noisy-or classifier combination N classifier responses, {p 1,...,p N } p i : probability of positive label (classifier i) 1 p i : probability of negative (classifier i) Negative bag: all instance classifiers are negative NY (1 p i ) p =1 i=1 Probability of bag being positive: NY (1 p i ) i=1 49

Algorithm pipeline 50 Input image P(x,y)

(P(x,y,θ,s)) 14x1 weight vector S. Tsogkas & I.

50 Algorithm pipeline 50 Input image P(x,y) Features Brightness Texture Color Spectra l P(x,y) Training Multiple Instance Learning (MIL) non-maximum suppression softmax θ,s (P(x,y,θ,s)) 14x1 weight vector S. Tsogkas & I.K., Learning-based symmetry detection in natural images, ECCV 2012

51 Feature extraction Color: CIE Lab color space. Texture: texton map.

Rectangle filters extract features at multiple scales and orientations.

51 51 Feature extraction Color: CIE Lab color space. Texture: texton map. Hard binning (32 bins for 3 color channels, 64 textons). Rectangle filters extract features at multiple scales and orientations. Differences of histograms ( gradients ) of color and texture content à symmetry indication. χ 2 distance à dissimilarity of feature content between adjacent rectangles. Integral images for fast extraction. S. Tsogkas & I.K., Learning-based symmetry detection in natural images, ECCV 2012

52 MIL Training 52 Input image θ = 22.5 θ = 45 Varying angle θ = 90 θ = 135 Varying scale θ = MIL training s = 4 s = 8 w w w = w s = 12 s = 24 s = 16 Probability map S. Tsogkas & I.K., Learning-based symmetry detection in natural images, ECCV 2012

53 Machine Learning for Computer Vision Lecture 2 Results Lindeberg Levinshtein This work State-of-the-art performance Large boost from color,

53 53 Machine Learning for Computer Vision Lecture 2 Results Lindeberg Levinshtein This work State-of-the-art performance Large boost from color, texture and spectral cues. Long contours (region proposals) S. Tsogkas & I.K., Learning-based symmetry detection in natural images, ECCV 2012 Ground-truth

54 54 Machine Learning for Computer Vision Lecture 2 Some more results Lindeberg Levinshtein This work Ground-truth

55 Lecture recap 55 Logistic Regression Training criterion Optimization Application to boundary detection Introduction to MIL

56 Logistic regression training cost 56 Log loss Quadratic loss

57 Lecture recap 57 Logistic Regression Training criterion Optimization Application to boundary detection Introduction to MIL

58 Maximization with Newton-Raphson 58 Newton-Raphson method for finding maxima of N-D function: Condition for maximum: Jacobian: Hessian:

59 Lecture recap 59 Logistic Regression Training criterion Optimization Application to boundary detection Introduction to MIL

60 Progress during the last 40 years 60 Humans Berkeley gpb, 08 Berkeley PB, 04 Canny+ Hysteresis ( 85) Prewitt, good feature= 5 years of work

61 Lecture outline 61 Recap & problems of linear regression Logistic Regression Training criterion formulation Optimization Interpretation Application to boundary detection Introduction to Multiple Instance Learning

62 Appendix 62 Newton Raphson = Iteratively Reweighted Least Squares Masking problem in multi-class linear regression & softmax

63 Newton Raphson = Iteratively Reweighted Least Squares 63 Newton Raphson: Hessian: Rewrite Update: Transform: Weighted Least Squares Fit:

64 Multiple classes & linear regression 64 K classes: one-of-k coding 4 classes, i-th sample is in 3 rd class: Matrix notation: Loss function: Least squares fit:

65 Multiple classes & linear regression 65 One linear discriminant per class: Problem: ambiguous regions Solution: assign to discriminant with largest score

66 Masking Problem in linear regression Class 1 Class 2 Class 3 66 Nothing ever gets assigned to class 2! 2D version:

67 Multiple classes & logistic regression 67 Soft maximum of competing classes: Discriminants (inputs) Softmax (outputs) Label probability: Similar steps for parameter estimation (Bishop s book)

68 Logistic vs Linear Regression, n>2 classes 68 Linear regression Logistic regression Logistic regression does not exhibit the masking problem

Part III: Affinity Functions for Image Segmentation

Part III: Affinity Functions for Image Segmentation Charless Fowlkes joint work with David Martin and Jitendra Malik at University of California at Berkeley 1 Q: What measurements should we use for constructing