Histogram of Oriented Gradients (HOG) for Object Detection

Size: px

Start display at page:

Download "Histogram of Oriented Gradients (HOG) for Object Detection"

Jason Boyd
5 years ago
Views:

1 Histogram of Oriented Gradients (HOG) for Object Detection Navneet DALAL Joint work with Bill TRIGGS and Cordelia SCHMID

Goal & Challenges Goal: Detect and localise people in images

appearance and clothing n Complex backgrounds n Unconstrained

sequences involves motion of the subject, the camera and the

2 Goal & Challenges Goal: Detect and localise people in images and videos n Wide variety of articulated poses n Variable appearance and clothing n Complex backgrounds n Unconstrained illumination n Occlusions, different scales n Videos sequences involves motion of the subject, the camera and the objects in the background Main assumption: upright fully visible people 2

3 Chronology n Haar Wavelets as features + AdaBoost for learning u Viola & Jones, ICCV 2001 u De-facto standard for detecting faces in images n Another approach: Haar wavelets + SVM: u Papageorgiou & Poggio, 2000; Mohan et al

Chronology n Edge templates from Gavrila et al n Based on

between edge fragments & detection task J J L L L Supports

Sensitive to edge detection & edge threshold Not resistant to

4 Chronology n Edge templates from Gavrila et al n Based on Information bottleneck principle of Tishby et al n Maximize MI between edge fragments & detection task J J L L L Supports irregular shapes & partial occlusions Window free framework Sensitive to edge detection & edge threshold Not resistant to local illumination changes Needs segmented positive images At par with then s-o-a 4

looking at two consecutive frames of a video n Leibe et

5 Chronology n Key point detectors repeat on backgrounds n Key point detectors do not repeat on people, even when looking at two consecutive frames of a video n Leibe et al, 2005; Mikolajczyk et al, 2004 Needed a different approach 5

6 Overview of Methodology Detection Phase Scan image(s) at all scales and locations Scale-space pyramid ` Extract features over windows Run linear SVM classifier on all locations Detection window Fuse multiple detections in 3-D position & scale space Focus on building robust feature sets (static & motion) Object detections with bounding boxes 6

7 HOG for Finding People in Images N. Dalal and B. Triggs. Histograms of Oriented Gradients for Human Detection. CVPR,

spatial cells Collect HOGs over detection window Linear SVM Overlap of Blocks N. Dalal and B.

8 Static Feature Extraction Input image Detection window Normalise gamma Compute gradients Weighted vote in spatial & orientation cells Cell Block Contrast normalise over overlapping spatial cells Collect HOGs over detection window Linear SVM Overlap of Blocks N. Dalal and B. Triggs. Histograms of Oriented Gradients for Human Detection. CVPR, 2005 Feature vector f = [...,...,...] 8

9 Overview of Learning Phase Learning phase Input: Annotations on training images Create fixed-resolution normalised training image data set Resample negative training images to create hard examples Encode images into feature spaces Encode images into feature spaces Learn binary classifier Learn binary classifier Object/Non-object decision Retraining reduces false positives by an order of magnitude! 9

10 HOG Descriptors Parameters n Gradient scale n Orientation bins n Percentage of block overlap Block Schemes n RGB or Lab, colour/gray-space n Block normalisation or L2-norm, L1-norm, v v / v v ε v /( v + ε ) 1 R-HOG/SIFT Cell C-HOG Center bin 10

11 11 Evaluation Data Sets MIT pedestrian database INRIA person database Train 507 positive windows Negative data unavailable Train 1208 positive windows 1218 negative images Test 200 positive windows Negative data unavailable Test 566 positive windows 453 negative images Overall 709 annotations+ reflections Overall 1774 annotations+ reflections

12 Overall Performance MIT pedestrian database INRIA person database n R/C-HOG give near perfect separation on MIT database n Have 1-2 order lower false positives than other descriptors 12

13 Performance on INRIA Database 13

14 14 Effect of Parameters Gradient smoothing, σ Orientation bins, β n Reducing gradient scale from 3 to 0 decreases false positives by 10 times n Increasing orientation bins from 4 to 9 decreases false positives by 10 times

15 15 Normalisation Method & Block Overlap Normalisation method Block overlap n Strong local normalisation is essential n Overlapping blocks improve performance, but descriptor size increases

Effect of Block and Cell Size 64 128 n Trade off between need for

16 Effect of Block and Cell Size n Trade off between need for local spatial invariance and need for finer spatial resolution 16

Descriptor Cues Input example Average gradients

inside a person are counted as negative n

17 Descriptor Cues Input example Average gradients Weighted pos wts Weighted neg wts Outside-in weights n Most important cues are head, shoulder, leg silhouettes n Vertical gradients inside a person are counted as negative n Overlapping blocks just outside the contour are most important 17

18 Multi-Scale Object Localisation Bias Multi-scale dense scan of detection window Clip Detection Score Final detections y Threshold s (in log) Η i x = [exp( s f ( x) = n i i ) σ,exp( s x w i exp ( x x ) / Η / 2 Apply robust mode detection, like mean shift i ) σ, σ ] y i s 1 2 i 18

equal to stride/cell size Relatively independent of scale

19 Effect of Spatial Smoothing n n Spatial smoothing aspect ratio as per window shape, smallest sigma approx. equal to stride/cell size Relatively independent of scale smoothing, sigma equal to 0.4 to 0.7 octaves gives good results 19

20 20 Effect of Other Parameters Different mappings Effect of scale-ratio n Hard clipping of SVM scores gives the best results than simple probabilistic mapping of these scores n Fine scale sampling helps improve recall

21 HOGs vs approaches till date HOG still among the best detector in terms of FPPI n See Dollar et al, CVPR 2009 Pedestrian Detection: A Benchmark miss rate (b) Typical aspect ratios VJ (0.85) HOG (0.44) FtrMine (0.55) Shapelet (0.80) MultiFtr (0.54) LatSvm (0.63) HikSvm (0.77) false positives per image 21

22 Results Using Static HOG No temporal smoothing of detections 22

23 Conclusions for Static Case n Fine grained features improve performance u Rectify fine gradients then pool spatially No gradient smoothing, [1 0-1] derivative mask Orientation voting into fine bins Spatial voting into coarser bins u Use gradient magnitude (no thresholding) u Strong local normalization u Use overlapping blocks u Robust non-maximum suppression Fine scale sampling, hard clipping & anisotropic kernel J Human detection rate of 90% at 10-4 false positives per window L Slower than integral images of Viola & Jones,

24 Applications to Other Classes M. Everingham et al. The 2005 PASCAL Visual Object Classes Challenge. Proceedings of the PASCAL Challenge Workshop,

25 Motion HOG for Finding People in Videos N. Dalal, B. Triggs and C. Schmid. Human Detection Using Oriented Histograms of Flow and Appearance. ECCV,

26 Finding People in Videos n Motivation u Human motion is very characteristic n Requirements u Must work for moving camera and background u Robust coding of relative motion of human parts n Previous works u Viola et al, 2003 u Gavrila et al, 2004 u Efros et al, 2003 Courtesy: R. Blake Vanderbilt Univ N. Dalal, B. Triggs and C. Schmid. Human Detection Using Oriented Histograms of Flow and Appearance. ECCV,

27 Handling Camera Motion n Camera motion characterisation u Pan and tilt is locally translational u Rest is depth induced motion parallax n Use local differential of flow u Cancels out effects of camera rotation u Highlights 3D depth boundaries u Highlights motion boundaries n Robust encoding into oriented histograms u Some focus on capturing motion boundaries u Other focus on capturing internal motion or relative dynamics of different limbs 27

28 28 Motion HOG Processing Chain Detection windows Input image Consecutive image Normalise gamma & colour Compute optical flow Flow field Magnitude of flow Compute differential flow Accumulate votes for differential flow orientation over spatial cells Differential flow X Differential flow Y Normalise contrast within overlapping blocks of cells Collect HOGs for all blocks over detection window Block Overlap of Blocks Cell

29 29 Overview of Feature Extraction Appearance Channel Input image Static HOG Encoding Consecutive image(s) Motion HOG Encoding Motion Channel Collect HOGs over detection window Linear SVM Object/Non-object decision Train Test 1 Test 2 Data Set 5 DVDs, 182 shots 5562 positive windows Same 5 DVDs, 50 shots 1704 positive windows 6 new DVDs, 128 shots 2700 positive windows

Coding Motion Boundaries n Treat x, y-flow components as independent images n Take their local gradients separately, and compute HOGs as in static images First

30 Coding Motion Boundaries n Treat x, y-flow components as independent images n Take their local gradients separately, and compute HOGs as in static images First frame Second frame Estd. flow Flow mag. Motion Boundary Histograms (MBH) encode depth and motion boundaries x-flow diff y-flow diff Avg. x-flow diff Avg. y-flow diff 30

31 Coding Internal Dynamics n Ideally compute relative displacements of different limbs u Requires reliable part detectors n Parts are relatively localised in our detection windows n Allows different coding schemes based on fixed spatial differences Internal Motion Histograms (IMH) encode relative dynamics of different regions 31

32 IMH Continued n Simple difference u Take x, y differentials of flow vector images [I x, I y ] u Variants may use larger spatial displacements while differencing, e.g. [ ] n Center cell difference n Wavelet-style cell differences

Flow Methods n Proesman s flow [ Proesmans et al.

Multi-scale pyramid based method, no regularization u Brightness

( A A + βi) A b on 5X5 window u 1 second per frame n MPEG-4

33 Flow Methods n Proesman s flow [ Proesmans et al. ECCV 1994] u 15 seconds per frame n Our flow method u Multi-scale pyramid based method, no regularization u Brightness constancy based damped least squares solution T T 1 T [ x, y] = ( A A + βi) A b on 5X5 window u 1 second per frame n MPEG-4 based block matching u Runs in real-time Input image Proesman s flow Our multi-scale flow 33

34 34 Performance Comparison Only motion information Appearance + motion n With motion only, MBH scheme on Proesmans flow works best n Combined with appearance, centre difference IMH performs best

35 Trained on Static & Flow Tested on flow only Tested on appearance + flow n Adding static images during test reduces performance margin n No deterioration in performance on static images 35

36 Trained on Static & Flow Tested on flow only Tested on appearance + flow n Adding static images during test reduces performance margin n No deterioration in performance on static images 36

37 37 Motion HOG Video No temporal smoothing, each pair of frames treated independently

38 r r r HOG, HOG, HOG, HOG, HOG, HOG, HOG, HOG, IMHwd and SVM IMHwd, Haar and SVM IMHwd and MPLBoost (K=3) IMHwd, Haar and MPLBoost HOG, IMHwd and SVM(K=4) IMHwd andand AdaBoost HOG MPLBoost (K=3) IMHwd, Haar and AdaBoost HOG, IMHwd and HIKSVM Haar and MPLBoost (K=4) HOG and HIKSVM IMHwd and Ess et HIKSVM al. (ICCV 07) - Full system 0.3 Recall-Precision for Motion HOG HOGfalse +1-precision IMHwd + HIK-SVM positives per image (c) Comparison to [11] (ETH-01) (e) Including motion features (ETH-02) recall recall recall HOG, IMHwd and SVM HOG, IMHwd, Haar and SVM HOG, IMHwd and MPLBoost (K=3) HOG, IMHwd, Haar andand MPLBoost (K=4) HOG, IMHwd, Haar MPLBoost (K=4) HOG, IMHwd HOG, Haarand andadaboost MPLBoost (K=4) HOG, IMHwd, Haar AdaBoost HOG, IMHwd andand HIKSVM HOG andand MPLBoost (K=3) HOG HIKSVM HOG, and HIKSVM EssIMHwd et al. (ICCV 07) - Full system false1-precision positives per image n n HOG, IMHwd and MPLBoost (K=3) HOG and MPLBoost (K=3) HOG, IMHwd and HIKSVM HOG and HIKSVM Ess et al. (ICCV 07) - Full system (h) Including motion features (ETH-03) (f) Comparison to [11] (ETH-02) false positives per image ETH (f) Comparison to [11] (ETH-02) ETH HOG, IMHwd, Haar and MPLBoost (K=4) HOG, Haar and MPLBoost (K=4) HOG, IMHwd and HIKSVM HOG and HIKSVM Ess et al. (ICCV 07) - Full system false positives per image (i) Comparison to [11] (ETH-03) Wojek et al, CVPR 09 Robust regularized flow + max in non-max suppression

39 Conclusions for Motion HOG n Summary u When combined with appearance, IMH outperforms MBH u Regularization in flow estimates reduces performance u MPEG4 block matching looks good but motion estimates not good for detection u Larger spatial difference masks help u Strong local normalization is very important u Relatively insensitive to number of orientation bins J Window classifier reduces false positives by 10 times L Slow compared to static HOG (probably not any more FlowLib from GPU4Vision) 39

40 Summary n Bottom-up approach to object detection n Robust feature encoding for person detection n Gives state-of-the-art results for person detection n Also works well for other object classes n Proposed differential motion features vectors for feature extraction from videos 40

41 Extensions n Real time feature computation (Wojek et al, DAGM 08; Wang et al, ICCV 09) n AdaBoost rejection cascade algorithms (Zhu et al, CVPR 06; Laptev, BMVC 06) n Part based detector for partial occlusions (Felzenszwalb et al, PAMI 09; Wang et al, ICCV 09) n Motion HOG extended (Wojek et al, CVPR 09; Laptev et al, CVPR 08) n Histogram intersection kernel (Maji et al, CVPR 2008, CVPR 2009, ICCV 2009) n Higher level image analysis (Hoiem IJCV 08) 41

42 Features for Object Detection n Local Binary Pattern u Wang et al, ICCV 2009 n Co-occurrence Matrices + HOG + PLS u Schwartz et al ICCV 2009 n Color HOG (Discriminative segmentation of fg/bg regions) u Ott & Everingham, ICCV

43 43 Founder & CEO Gesture Detection Using Webcams

44 Complete Lean Back Experience 44

45 Beta Launch in July 2011 n State of art work in research & engineering n Candidates for usability studies n Summer internships Contact: dalal@botsquare.com 45

46 46 Thank You Contact:

Histograms of Oriented Gradients for Human Detection p. 1/1

Histograms of Oriented Gradients for Human Detection p. 1/1 Histograms of Oriented Gradients for Human Detection Navneet Dalal and Bill Triggs INRIA Rhône-Alpes Grenoble, France Funding: acemedia, LAVA,