A HOG-based Real-time and Multi-scale Pedestrian Detector Demonstration System on FPGA

Size: px

Start display at page:

Download "A HOG-based Real-time and Multi-scale Pedestrian Detector Demonstration System on FPGA"

Theodore Boone
5 years ago
Views:

1 Institute of Microelectronic Systems A HOG-based Real-time and Multi-scale Pedestrian Detector Demonstration System on FPGA J. Dürre, D. Paradzik and H. Blume FPGA 2018

2 Outline Pedestrian detection with HOG/SVM HOG feature calculation Clustering and merging of overlapping detections Contributions of this work Trade-off between complexity and performance New merging method suitable for hardware implementation Results Comparison to other works Real-time demonstration system 1

HOG? Isn t it outdated?! Pedestrian detection based on feature description with histogram of oriented gradients (HOG) and SVM classification (Dalal / Triggs, CVPR, 2005) Detection performance: ~0.

3 HOG? Isn t it outdated?! Pedestrian detection based on feature description with histogram of oriented gradients (HOG) and SVM classification (Dalal / Triggs, CVPR, 2005) Detection performance: ~ FPPI State-of-the-art (CNN-based): ~ FPPI 1 Humans: ~ FPPI 1 HOG vs CNN - significant difference in computational complexity CNNs require 300x to 13,000x more energy to compute compared to HOG 2 For many applications HOG is still a suitable trade-off between complexity and performance! 1 Zhang, Towards Reaching Human Performance in Pedestrian Detection, TPAMI, Suleiman, Towards Closing the Energy Gap Between HOG and CNN Features for Embedded Vision, ISCAS,

HOG-features for pedestrian detection scaling image sliding window scaled image block Gaussian filter cell histogram window next scale next window feature extraction (HOG) feature vector

4 HOG-features for pedestrian detection scaling image sliding window scaled image block Gaussian filter cell histogram window next scale next window feature extraction (HOG) feature vector classification (SVM) buffer classification results of all windows of one scale window M φ gradient normalization merging classification results of all scales detections + + M G * w C * w φ,8 1 9 trilinear interpolation M G * w C * w φ,9 M G * w C 3

5 Merging of mutli-scale detections Dalal suggests the iterative Mean-Shift algorithm for clustering and merging of detections scales 4

6 Merging of mutli-scale detections Dalal suggests the iterative Mean-Shift algorithm for clustering and merging of detections Calculation of mass centers of the positive detection window centers Moving / shifting of all window centers towards mass centers Recalculate and shift until all centers remain still scales y x scales 4

7 Merging of mutli-scale detections Dalal suggests the iterative Mean-Shift algorithm for clustering and merging of detections Calculation of mass centers of the positive detection window centers Moving / shifting of all window centers towards mass centers Recalculate and shift until all centers remain still scales y x scales Computationally complex and irregular Complexity: O(n 2 ) yy mm = HH h yy mm n i=1 HH ii 1 2 tt PP ii ee 1 2 yy mm yy ii TT HH ii 1 yy mm yy ii n i=1 HH 1 HH 1 ii yy ii ii 2 tt PP ii ee 1 2 yy mm yy TT ii HH 1 ii yy mm yy ii 4

8 Contributions Publication Processing step 2009 Kadato 2011 Negi 2012 Mizuno et al Hahnle 2015 Yuan 2015 Rettkowski 2015 Ma 2018 this work Scaling HOG feat. extr. SVM classification Multi-scale processing Merging Many FPGA implementations to be found in literature Merging is rarely considered in hardware implementation A reason could be its complexity and irregularity Novel merging method, suitable for hardware implementation New approach for a trade-off between complexity and detection performance in HOG feature extraction 5

Architecture overview 1 pixel per clock, continuously image Parallel HOG/SVM calculation for different scales scaling sliding window scaled image window pixel stream next scale next window feature

9 Architecture overview 1 pixel per clock, continuously image Parallel HOG/SVM calculation for different scales scaling sliding window scaled image window pixel stream next scale next window feature extraction (HOG) feature vector scaling scaling classification (SVM) HOG feature extraction HOG feature extraction... HOG feature extraction buffer classification results of all windows of one scale classification results of all scales SVM classification SVM classification SVM classification merging detections... merging Algorithm detections Hardware structure 6

10 Trade-off between complexity and detection performance in HOG feature calculation Block Gaussian filter M G calculation of w C w C H block normalization N hierarchy level Cell x, y M histogram accumulation w φ y x Pixel P, x, y gradients φ calculation of w φ 7

11 Trade-off between complexity and detection performance in HOG feature calculation Block Gaussian filter M G calculation of w C w C H block normalization N hierarchy level Cell x, y M histogram accumulation w φ y x Pixel P, x, y gradients φ calculation of w φ Some complexity of the algorithm comes with jumping back and forth between hierarchy levels! 7

12 Trade-off between complexity and detection performance in HOG feature calculation block normalization N Block H x, y histogram accumulation hierarchy level Cell M w φ y x Pixel P, x, y gradients φ calculation of w φ Skipping the Gaussian filter and the inter-cell interpolation: Far less complex bottom-up approach Minor loss in detection performance of about ~3% 7

13 Comparison of HOG/SVM-impl. to other work Kadota Negi Mizuno Hahnle Yuan Rettkowski Ma this work this work Year FPGA LUTs Reg. DSPs BRAM [kbit] 2009 Intel Stratix II 2011 Xilinx Virtex Intel Cyclone IV 2013 Xilinx Virtex Xilinx Spartan Xilinx Zynq 2015 Xilinx Virxtex Intel Cyclone IV 2018 Intel Stratix V 3,794 [LUT6] 17,383 [LUT6] 34,403 [LUT4] 5,188 [LUT6] 9,955 [LUT6] 21,297 [LUT6] 98,642 [LUT6] 4,937 [LUT4] 3,529 [LUT6] 6, [18x18] 2, [25x18] 23, [18x18] 5, [25x18] 13, [18x18] 5,942 4 [25x18] 8, [25x18] 2, [9x9] 2, [27x27] Clock rate [MHz] Resolution Data rate FPS LUT eff. η Detection rate / FPPW Comment? x / - no Gaussian filter, interpolation + classification? x % / 0.2 (2x10-1 ) x % / (10-4 ) 1, x % / (10-3 ) x ? /? x % / 0.04 (4x10-2 ) 4, x (est.) % / (10-4 ) x % / (10-4 ) x % / (10-4 ) no Gaussian filter + interpolation, AdaBoost no Gaussian filter no interpolation no Gaussian filter + interp., AdaBoost, ext. DDR no Gaussian filter, linear bin-interpolation no Gaussian filter, linear bin-interpolation LUT Efficiency: ηη = pppppppppppp/ss αα #LLLLLLLL ff cccccc αα = 1.0 for 4 input LUTs αα = 1.5 for 6 input LUTs 8

14 Comparison of HOG/SVM-impl. to other work Kadota Negi Mizuno Hahnle Yuan Rettkowski Ma this work this work Year FPGA LUTs Reg. DSPs BRAM [kbit] 2009 Intel Stratix II 2011 Xilinx Virtex Intel Cyclone IV 2013 Xilinx Virtex Xilinx Spartan Xilinx Zynq 2015 Xilinx Virxtex Intel Cyclone IV 2018 Intel Stratix V 3,794 [LUT6] 17,383 [LUT6] 34,403 [LUT4] 5,188 [LUT6] 9,955 [LUT6] 21,297 [LUT6] 98,642 [LUT6] 4,937 [LUT4] 3,529 [LUT6] 6, [18x18] 2, [25x18] 23, [18x18] 5, [25x18] 13, [18x18] 5,942 4 [25x18] 8, [25x18] 2, [9x9] 2, [27x27] Clock rate [MHz] Resolution Data rate FPS LUT eff. η Detection rate / FPPW Comment? x / - no Gaussian filter, interpolation + classification? x % / 0.2 (2x10-1 ) x % / (10-4 ) 1, x % / (10-3 ) x ? /? x % / 0.04 (4x10-2 ) 4, x (est.) % / (10-4 ) x % / (10-4 ) x % / (10-4 ) no Gaussian filter + interpolation, AdaBoost no Gaussian filter no interpolation no Gaussian filter + interp., AdaBoost, ext. DDR no Gaussian filter, linear bin-interpolation no Gaussian filter, linear bin-interpolation LUT Efficiency: ηη = pppppppppppp/ss αα #LLLLLLLL ff cccccc αα = 1.0 for 4 input LUTs αα = 1.5 for 6 input LUTs 8

15 Architecture overview 1 pixel per clock, continuously image Parallel HOG/SVM calculation for different scales scaling sliding window scaled image window pixel stream next scale next window feature extraction (HOG) feature vector scaling scaling classification (SVM) HOG feature extraction HOG feature extraction... HOG feature extraction buffer classification results of all windows of one scale classification results of all scales SVM classification SVM classification SVM classification merging detections... merging Algorithm detections Hardware structure 9

New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1,42 1 2 3 0,85 Highly overlapping detections are most likely

16 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1, ,85 Highly overlapping detections are most likely multiple detections of the same object 4 Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,31 SVM-score 1,27 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,31 10

17 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1, ,85 Highly overlapping detections are most likely multiple detections of the same object 4 Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,31 SVM-score 1,27 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

18 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1, ,85 Highly overlapping detections are most likely multiple detections of the same object 4 Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,31 SVM-score 1,27 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

19 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1, ,85 Highly overlapping detections are most likely multiple detections of the same object 4 Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,31 SVM-score 1,27 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

20 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1, ,85 Highly overlapping detections are most likely multiple detections of the same object 4 Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,31 SVM-score 1,27 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

21 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1, ,85 Highly overlapping detections are most likely multiple detections of the same object 4 Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,31 SVM-score 1,27 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

22 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection 1, ,85 Highly overlapping detections are most likely multiple detections of the same object 4 Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,31 SVM-score 1,27 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

23 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection Highly overlapping detections are most likely multiple detections of the same object Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,42 1,31 SVM-score 4 1 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

24 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection Highly overlapping detections are most likely multiple detections of the same object Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false 1,42 1,31 SVM-score 4 1 New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found Position Index Score 1 1 1, , , ,85 10

25 New Merging Method Instead of complex mean-shift: plausibility check High SVM-score implies high likelihood of a true positive detection Highly overlapping detections are most likely multiple detections of the same object Objects always cause multiple overlapping detections, single non-overlapping detections are most likely false New algorithm suitable for hardware implementation 1. Sort detections by SVM-score 2. For each entry: scan list for 50% overlapping detections 3. Removal of overlapped detections 4. Entry is true positive, if at least 2 overlapped detections have been found 1 (e.g. Heapsort: O(n log n)) Position Index Score 1 1 1, , , ,85 Small arithmetic complexity, run-time complexity dominated by sorting 10

26 Demonstration System COTS Altera / Intel DE2-115 dev. board with Cyclone IV FPGA 9 scales (factor 1.1), 800x600, 20 FPS Resolution and framerate limited by SDRAM Camera IF RGB->grey bi. scaling bi. scaling HOG / SVM HOG / SVM... HOG / SVM FIFO FIFO FIFO SDRAM IF... VGA IF bounding box generator merging 11

27 Synthesis results Intel Cyclone IV EP4CE x600, MHz, 9 scales LUTs Reg. DSPs BRAM [kbit] Pedestrian Detector 47,175 25, ,222 - HOG/SVM (9 instances) 41,906 23, ,137 - Bilinear scaling (8 instances) 1, Merging 1, Infrastructure (Framework) 3,343 2, Camera Interface VGA Interface SDRAM Interface Total (% of available) 50,518 (44%) 27,295 (24%) 491 (92%) 1,765 (45%) Small hardware costs for merging module Number of scales limited by DSPs and BRAM routing Camera IF SDRAM IF RGB->grey HOG / SVM FIFO bi. scaling HOG / SVM FIFO bi. scaling HOG / SVM FIFO VGA IF bounding box generator merging 12

28 Summary For many embedded pedestrian detection applications HOG/SVM is still a suitable trade-off between complexity and performance Merging of overlapping detections was rarely considered in hardware implementations, supposedly because of its complexity and irregularity This work shows a new approach for the trade-off between algorithm simplifications and performance in HOG/SVM presents a highly efficient streaming architecture introduces a new method for merging, particularly suitable for hardware implementations shows a real-time demonstration system on a small COTS FPGA board 13

FPGA-based Real-Time Pedestrian Detection on High-Resolution Images

2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops FPGA-based Real-Time Pedestrian Detection on High-Resolution Images Michael Hahnle 1, Frerk Saxen 2, Matthias Hisung 1, Ulrich