Deformable Part Models

Size: px

Start display at page:

Download "Deformable Part Models"

Philippa Knight
6 years ago
Views:

1 Deformable Part Models References: Felzenszwalb, Girshick, McAllester and Ramanan, Object with Trained Part Based Models, PAMI 2010 Code available at hkp://

2 Recall: Dalal and Triggs, 2005 Detect upright pedestrians Histogram of oriented gradient feature vector Linear SVM classifier; sliding window detector 64X128 HoG descriptor

3 Today: Deformable Part Models Slide from Pedro Felzenswalb

4 Simple template- based object models lack ability to handle geometric due to and pose. Simple bag of words models have no trouble with but are limited in their ability to finely localize objects. Want to make more expressive models based on geometric of a canonical configura@on of object parts.

5 Problem: More expressive object models are difficult to train because they oaen use latent (unobserved/unlabeled) Example: learning a part- based model from images where only bounding boxes are labeled. What we have: bounding box What we want: torso, head, arms and legs

6 DPM Star- structured graph model to represent body parts and their geometric Mixtures of models to handle large in viewpoint / pose Latent- SVM formula@on Data mining of hard nega@ve examples PCA on HOG features for dimension reduc@on Performance on PASCAL challenge dataset

7 History: Pictorial Structures Slide from Pedro Felzenswalb

8 History: Pictorial Structures Goal: alignment of part model with features in an image. Slide from Ross Girshick

9 History: Pictorial Structures Felzenswalb showed that the best alignment of parts for a tree- structured model can be computed efficiently using dynamic programming. Slide from Ross Girshick

10 Pictorial Structures: Matching score maximizing Slide from Pedro Felzenswalb

11 From Pictorial Structures to DPM Specifies unary costs. detector. Pairwise costs. Slide from Pedro Felzenswalb

12 Slide from Pedro Felzenswalb

13 Dalal and Triggs, single template Slide from Pedro Felzenswalb

Root filter Part filters Score is sum of filter scores minus deforma@on costs Parts

14 Root filter Part filters Score is sum of filter scores minus costs Parts are represented at twice the of the root filter. Slide from Pedro Felzenswalb

15 Scoring with Linear Classifiers Slide from Pedro Felzenswalb

16 More Specifically... Like in Dalal&Triggs but for parts e.g. d i = [1,1,0,0] yields squared distance error Slide from Ross Girshick

17 Scoring with Linear Classifiers [w0,w1,...,wn, d1,d2,...,dn] [Φ(I,p0),..., Φ(I,pn), (dx 12,dy 12,dx 1,dy 1 ),..., (dx n2,dy n2,dx n,dy n )] Slide from Ross Girshick

18 Overview

19 Handling Large in Viewpoints Slide from Ross Girshick

20 Handling Large in Viewpoints Slide from Ross Girshick

Solu@on: Use Mixture Model Mixture component 1: Bicycles

21 Use Mixture Model Mixture component 1: Bicycles viewed from side Mixture component 2: Bicycles viewed from front/rear

Solu@on: Use a Mixture Model Person mixture: 2 components Car mixture: 3 components One model for side,

22 Use a Mixture Model Person mixture: 2 components Car mixture: 3 components One model for side, one for front and one for 45 degrees. One model for head+torso, one for full body Slide from Ross Girshick

23 Slide from Pedro Felzenswalb

24 Slide from Ross Girshick

25 Slide from Ross Girshick

26 Slide from Ross Girshick

27 Slide from Ross Girshick

28 Recall: SVM Training 1. Maximize margin 2/ w 2. Correctly classify all training data points: x x i i positive ( y negative( y i i = 1) : = 1) : x x i i w + b 1 w + b 1 Quadratic optimization problem: Minimize 1 2 w T w Subject to y i (w x i +b) 1 One constraint for each training point. rewrite

For constraints that are are sa@sfied (the appropriate inequality holds), the value of the corresponding Lagrange

29 Aside: SVM and Hinge Loss To maximize the margin, which is 2 / w hinge loss This is a consequence of encoding the constraints into the objec@ve func@on using Lagrange mul@pliers. For constraints that are are sa@sfied (the appropriate inequality holds), the value of the corresponding Lagrange mul@plier becomes 0. Intui@on: the inequality constraints fw(x + ) > 1 and fw(x - ) < - 1 contribute a linear penalty when they are not sa@sfied, but are not penalized when the inequali@es hold.

30 SVM vs LSVM Training SVM f w (x i ) = w. Φ(x i ) LSVM f w (x i ) = max w. Φ(x i, z) z in Z(x i ) Max over latent posi@ons for part x i This introduces some difficul@es during training

31 SVM vs LSVM Training SVM f w (x i ) = w. Φ(x i ) Convex problem, guaranteed op@mal solu@on! LSVM f w (x i ) = max w. Φ(x i, z) z in Z(x i ) Only semi- convex, no guaranteed op@mal solu@on

32 Slide from Ross Girshick

33 Slide from Ross Girshick

34 Slide from Ross Girshick

35 Slide from Ross Girshick

36 Slide from Ross Girshick

37 Slide from Ross Girshick

Since whole objec@ve func@on is no longer convex, there is no longer a

38 Since whole is no longer convex, there is no longer a single global op@mum, so no easy solu@on method. Slide from Ross Girshick

39 by upper bound Slide from Ross Girshick

40 Current of latent that yield upper bound for each example. Slide from Ross Girshick

Current es@mates of latent configura@ons that yield upper bound for each posi@ve

41 Current of latent that yield upper bound for each example. Yields upper bound for whole Slide from Ross Girshick

42 Current of latent that yield upper bound for each example. Yields upper bound for whole And it is upper bound Slide from Ross Girshick

43 Unfortunately, this isn t easy to op@mize either, not to men@on it is based on es@mates of upper bound latent configura@ons, which could be wrong. Slide from Ross Girshick

44 approach to solving this. NOT guaranteed to find unless you start with good Slide from Ross Girshick

45 approach to solving this. NOT guaranteed to find unless you start with good Slide from Ross Girshick

46 approach to solving this. NOT guaranteed to find unless you start with good Slide from Ross Girshick

47 approach to solving this. NOT guaranteed to find unless you start with good Slide from Ross Girshick

48 Easy to compute Slide from Ross Girshick

49 Easy to solve Slide from Ross Girshick

50 Can we learn that as well? Slide from Ross Girshick

Ini@alize a mixture model to handle different

51 a mixture model to handle different viewpoints Ini@aliza@on: Slide from Ross Girshick

52 Slide from Ross Girshick

Ini@alize parts for each component Refine by training.

53 parts for each component Refine by training. Note: parts are constrained to be symmetrically placed Slide from Ross Girshick

54 Summary: Model Learning Learn mixture model (component root filters) Learn part model for each mixture component LSVM used during both phases

Mining Hard Nega@ves Detec@ons problems are highly unbalanced (many more nega@ves than

55 Mining Hard problems are highly unbalanced (many more than We REALLY don t want to consider all nega@ve examples while training! Slide from Ross Girshick

56 Mining Hard Examples Set of training examples is HUGE (a single image can yield 10 5 examples for a scanning window classifier). Construct training data from the posi@ve instances and a selec@on of the hard nega@ve instances, via bootstrapping Train using subset of nega@ve examples Apply resul@ng classifier to all nega@ve examples, and take those incorrectly classified as a new set of hard nega@ves Retrain, and perhaps repeat

57 PCA Analysis of HOG Collected 36- dimensional HOG features at a variety of scales over large number of images PCA analysis to iden@fy top eigenvectors

58 Recall: HOG Features 64x128 Compute gradients Each cell contains a histogram of gradient orienta@ons, weighted by gradient magnitude

HoG Feature Extrac@on: Blocks 2x2 block of cells normalize [,,,,.

59 HoG Feature Blocks 2x2 block of cells normalize [,,,,..., ] Independent contrast normaliza@on over 2x2 blocks of cells.

60 HoG Feature Blocks 2x2 block of cells normalize [,,,,..., ] Independent contrast normaliza@on over 2x2 blocks of cells. Each histogram gets normalized 4 9 orienta@ons x 4 orienta@ons = 36 feature values for each cell

61 PCA Analysis of HOG Collected 36- dimensional HOG features at a variety of scales over large number of images PCA analysis to iden@fy top eigenvectors

62 PCA Analysis of HOG Collected 36- dimensional HOG features at a variety of scales over large number of images PCA analysis to iden@fy top eigenvectors Top 11 account for nearly all of the varia@on

PCA Analysis of HOG Projec@on onto PCA bases is expensive However, note that top eigenvectors are

63 PCA Analysis of HOG onto PCA bases is expensive However, note that top eigenvectors are constant over rows or cols to 13 dimensions: sum over each of 4 rows and 9 columns

64 Bounding Box Problem: and size of root filter (red rectangle) may not correlate perfectly with bounding boxes in human- labeled data Learn four linear regression to predict x1, y1, x2, y2 of bounding box from a vector containing loca@ons of each part (including root) and the width (scale) of the root. (2n+3)à x1 (or y1, x2, y2) (x1,y1) predict (x2,y2)

65 Non- Maximum Suppression Sliding window detectors tend to produce many overlapping on same object

Non- Maximum Suppression Greedy solu@on: run through list of detec@ons in descending

66 Non- Maximum Suppression Greedy run through list of in descending order of confidence, removing ones that overlap more than 50% with a higher- scoring detec@on.

67 Slide from Pedro Felzenswalb

68 Example from the Dataset Slide from Pedro Felzenswalb

69 Slide from Ross Girshick

70 Bounding box overlap of 50% needed, to detect Slide from Ross Girshick

71 Slide from Pedro Felzenswalb

72 Slide from Pedro Felzenswalb

73 Slide from Pedro Felzenswalb

74 Some False Car Person (Insufficient overlap) Horse

75 Some False Sofa BoKle Cat (Due to insufficient overlap)

76 Slide from Pedro Felzenswalb

77 Summary: DPM Star- structured graph model to represent body parts and their geometric Mixtures of models to handle large in viewpoint / pose Latent- SVM formula@on Data mining of hard nega@ve examples PCA on HOG features for dimension reduc@on Performance on PASCAL challenge dataset

Object Detection with Discriminatively Trained Part Based Models

Object Detection with Discriminatively Trained Part Based Models Pedro F. Felzenszwelb, Ross B. Girshick, David McAllester and Deva Ramanan Presented by Fabricio Santolin da Silva Kaustav Basu Some slides