Semi-supervised Learning on Real-time. Pedestrian Detection System

Transcription

1 23 rd ITS World Congress, Melbourne, Australia, October 2016 Paper number ITS-0236 Semi-supervised Learning on Real-time Pedestrian Detection System Kuo-Ching Chang 1*, Zhen-Wei Zhu 1, Han-Wen Huang 1, Chuan-Ren Lee 1 1. Automotive Research and Testing Center, Taiwan * No.6, Lugong S. 7th Rd., Lukang, Changhua County, #2323, kcchang@artc.org.tw Abstract This research proposed a pedestrian detection system in an outdoor scene by a pin-hole camera with field-programmable gate array. We used two different feature descriptors included histograms of oriented gradients and local weighted pattern which were descripted the shape and textural information for the image feature of pedestrians. Semi-supervised learning with linear support vector machine would be used as the classifier to detect pedestrians. A car to vulnerable road users both adult and child scenarios were selected, and the maximum test speed of demonstrated vehicle was 30 km/h. The average detection rates of our proposed system for both scenarios were 100% and 96.67%, respectively. Keywords: Pedestrian Detection, Local Weighted Pattern, Field-Programmable Gate Array Introduction The issue of advanced driver assistance systems (ADASs) [1] include lane departure warning (LDW) system [2], blind spot warning (BSW) system [3], parking assistance (PA) system [4], and autonomous emergency braking (AEB) system [5], pedestrian detection (PD) system [6] and so on, have become an important assistant systems to significantly avoid serious traffic accidents. According to the official road crash statistics in world health organization website, it shows about 1.25 million people die each year due to the road traffic crashes. Half of those dying are vulnerable road users such as pedestrians, motorcyclists and cyclists. In 2016, European New Car Assessment Programme (Euro NCAP) has published three crossing scenarios for evaluating star rating of AEB pedestrian system: (1) an adult runs from the driver s side of the vehicle, (2) an adult walks from the passenger s side, and (3) a child runs from between parked cars on the passenger s side. Three tests show the certainly critical

2 circumstance in our life. Therefore, the AEB pedestrian system will be the worth developing system to decrease the number of accidents and avoid the human injury. In the past decades, researchers have employed the pattern recognition technologies to figure out the target object in the key image. A straightforward approach to detect pedestrian was based on using a pin-hole camera by extracting features of the current image and employed histogram of oriented gradients (HOG) descriptor with support vector machine (SVM) classifier [7]. Huang et al [8] and Chang et al [9] combined the HOG and local weighted pattern (LWP) descriptors to detect pedestrian. Combining these two powerful descriptors including both shape and textural features could show significant performance. To solve the false-classified problems caused by previous training parameters, the concept of semi-supervised learning is applied widely in the field of machine learning. Chen et al [10] proposed regression-based techniques implemented in a unified active and semi-supervised regression framework with ability to perform transfer learning for people counting. Tang et al [11] treated tracking as a foreground/background classification problem and proposed an online semi-supervised learning framework with SVM. The proposed pedestrian detection system using the linear kernel function of SVM classifier with a semi-supervised framework to re-estimate support vectors from the false-classified samples and then replaced the parameters of the previous trained detector. Therefore, those false samples could be correctly classified after using semi-supervised learning. A significant number of research studies that deal with the hardware-accelerated for pedestrian detection have been proposed [12-13]. Lee et al [12] used HOG feature extraction accelerator that was based on two-dimensional histogram interpolation. The hardware implementation of HOG feature extraction for high resolution image (1080P) is proposed by [13]. In this paper, we combine the hardware accelerator into system on chip (SoC) system and built on Xilinx Zynq-7000 SoC. Using the hardware accelerator built in SoC system that our proposed system could significantly detect pedestrians in real-time. System Architecture The pedestrian detection system architecture with semi-supervised learning is proposed by three parts: (1) Image acquisition unit: a camera. (2) Main processing unit: embedded field programmable gate array (FPGA). (3) Parameter estimation unit: a server. The proposed system architecture is shown in Figure 1. In this research, embedded FPGA with semi-supervised learning were employed for detecting pedestrian and a single pin-hole camera was adopted to build this system. A series of images 2

3 would be grabbed by a pin-hole camera which was mounted on the windshield of the vehicle. A server was used to estimate the classifier parameter with linear support vector machine [14] and then update the information to FPGA. The region of interest (ROI) was set on the trajectory of the vehicle in this proposed system. When pedestrians passed through the area of ROI, the pattern recognition technologies were employed to recognize pedestrians and beep sound. Figure 2 is the flow chart of proposed system. Figure 1. Architecture of pedestrian detection system System Start Import Support Vectors Video Import Update Feature Extraction (HOG & LWP) Semi-supervised Learning Recognize No SVM Classifier Pedestrians Warning Figure 2. System process diagram for detecting pedestrians 3

4 Image Features Exaction Histogram of oriented gradient (HOG) [7] and local weighted pattern (LWP) [8-9] were feature descriptors for detecting pedestrians in our proposed system. Features were extracted from the 8-bit grayscale image. The pixel stream was converted by the image decoder which was implemented by FPGA. A detecting window was rectangular region of 64 pixels width and 128 pixels height. The step of detecting window size was 8 pixels that across an image. In order to calculate HOG features, for each detecting window was divided into 7 15 overlapping blocks. Each block was 16 pixels width and height which consisted 4 cells with 8 pixels width and height. The oriented gradients divided into 9 bins with orientation range from 0 to 180 in each cell. The resolution of each bin was fixed to 20. The block normalization used Lowe-style clipped L2 norm (L2-Hys[7]) which was included 36 features. Figure 3 showed the HOG descriptor with a detecting window, blocks, cells and corresponding oriented histogram. Figure 3. HOG descriptor structure The LWP descriptor was modified by local oriented pattern (LOP) descriptor whereas the difference of neighbor pixels for voting. The local binary pattern descriptor was the original form for LOP. The detecting window size and block size were the same as HOG descriptor. Each detecting window could be divided 32 non-overlapping blocks. In each block, we compared the intensity value of every 3 pixels width and 3 pixels height sub-window. The central pixel could compare with its eight neighbor pixels. Once the central intensity value was large than the one of neighbor value, it was displayed as 1, and otherwise displayed 0. After comparing intensity for each sub-window, there were eight values could be led to an 8-bit pattern. To reduce pattern form, we combine two transitions to be the same condition, where 0 to 1 or 1 to 0 was defined a transition [8]. Therefore, for each block included 59 features. For each feature, the logarithmic voting value was used to calculate by Equation (1). 4

5 8 v(xx) = log II(xx) II(yy ii ) + εε (1) ii=1, where II(xx) was the grayscale intensity of central pixel, II(yy ii ) was the grayscale intensity around the central pixel and εε was a tiny numerical value that avoid the undefined logarithmic voting value. The detail of the LWP descriptor was described in [8]. Figure 4 showed the LWP descriptor with a detecting window, blocks, sub-windows, corresponding transition pattern, and voting values. In summary, for each sample could be described two part features involving 3780 HOG features and 1888 LWP features. There were 5668 features and were extracted from each training and testing sample. We used the LIBLINEAR [14] which was a popular open source machine learning library to estimate support vectors by collective training samples. Semi-supervised Learning Figure 4. LWP descriptor structure A labeled data set was used to train the linear SVM classifier with off-line training algorithm. The set of training data was composed of positive sample set and negative sample set from real world scenarios. However, the amount of collecting negative samples for training is limited. It is hard to achieve diversity of negative samples perfectly. Consequently, we used the concept of semi-supervised learning to collect different type of negative samples. The candidate target samples were the detecting result from the initial off-line classifier and will further be classified by a prior strong classifier. The updated false-classified database will be collected by above process. The optimization process of off-line classifier is adapting to replace previous classifier s support vectors. By repeating those processes with a server, the 5

6 classifier will be improved efficiency and robustness. Figure 5 is the diagram that represents the processing of semi-supervised learning. Semi-supervised Learning Non-Pedestrians Video Pedestrians Video Support Vectors Update Feature Extraction (HOG & LWP) SVM Classifier No Re-estimate Classifier False Samples Yes Yes Candidated Target Samples New Negative Samples Not Pedestrians Strong Classifier Pedestrians Semi-supervised Learning End Figure 5. The flowchart of semi-supervised learning Field-programmable Gate Array We implemented our proposed system on Xilinx Zynq-7000 SoC system. It is composed of dual advanced RISC machine (ARM) Cortex-A9 CPUs and Xilinx Kintex-7 FPGA. ARM and FPGA share globally addressable 4GByte virtual memory and are communicated via interconnect in SoC system. It is based on ARM advanced microcontroller bus architecture design that can manage multiple transactions and is architected for low-latency paths communication between the ARM and FPGA. The FPGA has 64 bit memory channel which is control by direct memory access controller. It is running at 200 MHz under these architectures and could be running on maximum performance. The FPGA implement the HOG descriptor, LWP descriptor and linear SVM classifier. The pedestrian architecture has been implemented and has higher throughput than GPU and CPU. 6

7 We designed the detection system with three parts: (1) Feature extraction module. (2) Classifier module. (3) Video processer. The video signal is decoded from S-video input with national television system committee protocol by video processer. It was designed with pipeline and data-flow architecture using first in first out (FIFO) buffer and saved in the double data rate synchronous dynamic random-access memory. The feature extraction module includes HOG descriptor and LWP descriptor. We employ the pipeline architecture to extract the gradient and texture information from video sequences. To improve the calculated performance of cells histogram and blocks histogram, we use the FIFO buffer to store data between calculation architectures with parallel architecture. Finally, we designed parallel multi-classifier to calculate the result of pedestrian system. Implementation In order to demonstrate the pedestrian detection system, the front view of the camera with 46 horizontal degrees and FPGA embedded system was chosen for this approach test. It was mounted on the windshield of the vehicle with 720 pixels width and 480 pixels height. The INRIA pedestrian dataset was chosen to extract HOG and LWP features for estimating support vectors with linear support vector machine. The initial classifier was built in above dataset, and then used 250 non-pedestrian images and 250 pedestrian images with semi-supervised learning method for evaluating the robustness of support vectors. The test protocol a car to vulnerable road users was selected that include both adult and child scenarios. On account of the limitation of the equipment we had, tests were similar to Euro NCAPEuro NCAP s scenario tests and could be experimented the proving ground of automotive research and testing center (ARTC) campus. The demonstrated vehicle and pedestrian were a sport utility vehicle, and movable both adult and child pedestrian target. The pedestrian walking speeds ranging was round 2kilometres per hour (KPH) and struck at 50% frontal structure of the vehicles width. The scenarios, which are shown in Figure 6., were that demonstrated vehicle would go straight to the pedestrian in speed 10, 20 and 30 KPH and experiment would be done for 10 times for each test speed. The initial distance between the demonstrate vehicle was over 50 meters. When pedestrian from 3 to 25 meters, the algorithm would be employed to detect and recognize the pedestrian. The average detection rates for both adult and child scenarios were 100% and 96.67%, respectively.the average detection rates of our proposed system for both scenarios were 100% and 96.67%, respectively.the average detection rates of our proposed system for both scenarios were 100% and 96.67%, respectively. The average detection rates of our proposed system for both scenarios were 100% and 96.67%, respectively. Results are shown in Table 1 and Table 2. 7

8 (a) Adult pedestrian (b) Child pedestrian Figure 6. The testing scenario Table 1. Results of detection distance and detection rate with adult pedestrian scenario Times Detection distance (m) Average Average detection detection rate (%) System 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th distance (m) average KPH % detection rate (%) % % % Table 2. Results of detection distance and detection rate with child pedestrian scenario Times Detection distance (m) Average Average detection detection rate (%) System 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th distance (m) average KPH % detection rate (%) % % 96.67% Conclusion and Future Work In this proposed how a semi-supervised learning method for estimating support vectors to real time detect pedestrian with a pin-hole camera and FPGA embedded system. Results showed this system could successfully detect the pedestrian in front of the vehicle at speed of 10, 20 8

9 and 30 KPH. The next approach, the deep learning method will be adopted and convolution neural network will be used to detect not only pedestrians and also vehicles, cyclists and motorcyclists. The goal of our future proposal could be applied to active autonomous emergency braking and reduce traffic accidents. Acknowledgements This paper is supported by Department Industrial Technology, Ministry of Economic Affairs, Taiwan, under the research projects 105-EC-17-A References 1. Geronimo, D., A.M. Lopez, A.D. Sappa and T. Graf (2010). Survey on Pedestrian Detection for Advanced Driver Assistance Systems, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 7, pp Kozak, K., J. Pohl, W. Birk, J. Greenberg, B. Artz, M. Blommer, L. Cathey and R. Curry (2006). Evaluation of lane departure warnings for drowsy drivers, Proc. Hum. Factors Ergonom. Soc. 50th Annu. Meeting, pp Wu, B.-F., C.-C. Kao, Y.-F. Li and M.-Y. Tsai (2012). A Real Time Embedded Blind Spot Safety Assistance System, International Journal of Vehicular Technology, Vol. 2012, pp Wada M., K. S. Yoon and H. Hashimoto (2003). Development of advanced parking assistance system, IEEE Trans. Ind. Electron., vol. 50, no. 1, pp Kaempchen, N., B. Schiele and K. Dietmayer (2009). Situation assessment of an autonomous emergency brake for arbitrary vehicle-to-vehicle collision scenarios, IEEE Trans. Intell. Transp. Syst., vol. 10, no. 4, pp Dollar P., C. Wojek, B. Schiele, and P. Perona (2012). Pedestrian detection: An evaluation of the state of the art. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol 34, no. 4, pp Dalal, N., B. Triggs (2005). Histograms of Oriented Gradients for Human Detection, Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp Huang, P.-T., Y.-M. Chan, L.-C. Fu, S.-S. Huang, P.-Y. Hsiao, W.-Y. Wu, C.-C. Lin, K.-C. Chang, and P.-M. Hsu (2014). Pedestrian detection system in low illumination conditions through Fusion of image and range data, IEEE Conference on Intelligent Transportation Systems, pp Chang, K.-C., C.-K. Chen, and P.-K. Tseng (2015). Multi-sensor fusion adopted 2-D laser rangefinder and camera for pedestrian detection. In Proceedings 22 nd World Congress on ITS, Bordeaux. ERTICO (ITS Europe). 9

10 10. Chen, C., S. Gong, and T. Xiang (2013). From Semi-supervised to Transfer Counting of Crowds, The IEEE International Conference on Computer Vision, pp Tang, F., S. Brennan, Q. Zhao, and H. Tao (2007). Co-Tracking Using Semi-Supervised Support Vector Machines, IEEE 11th International Conference on Computer Vision, pp Lee, S., H. Son, J.-C. Choi, and K. Min (2012). HOG feature extractor circuit for real-time human and vehicle detection, Proc. IEEE Region 10 Conf. TENCON, pp Mizuno, K., Y. Terachi, K. Takagi, S. Izumi, H. Kawaguchi, and M. Yoshimoto (2012). Architectural study of hog feature extraction processor real-time object detection, in Proceedings of IEEE Workshop on Signal Processing Systems, pp Fan, R.-E., K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin (2008). LIBLINEAR: A library for large linear classification, Journal of Machine Learning Research, vol. 9, pp