Vision Based Person Detection for Safe Navigation of Commercial Vehicle

Size: px

Start display at page:

Download "Vision Based Person Detection for Safe Navigation of Commercial Vehicle"

Randall Gallagher
5 years ago
Views:

1 Vision Based Person Detection for Safe Navigation of Commercial Vehicle Songlin Piao and Karsten Berns University of Kaiserslautern, 67663, Germany, Abstract. Vision based solution of safe navigation for commercial vehicles with fish-eye camera is presented in this paper. This work aims to develop a system which is able to detect persons or objects around commercial vehicles for preventing any accidents. This is achieved by integrating classifier based window searching algorithm and ego-motion based algorithm into the system. The classifier is trained using cascaded support vector machine and ego-motion is estimated only based on captured images from camera. Test results show that designed system provides high detection rate when it is applied in commercial vehicle. 1 Introduction Nowadays, safety issue is becoming more and more important in autonomous mobile robots. The word Autonomous receives increasing popularities in the industry of mobile vehicles. One of the main challenges ahead of industry is to promote the safety rate during the autonomous operation. Vehicle manufactures try to reduce accident rate by integrating a diverse range of safety techniques into their product. However, it seems not easy for commercial vehicles to fulfill this requirement. There were 4216 fatalities caused by large trucks and buses in U.S. in 2012 according to Motor Carrier Safety Progress Report [1]. The reason of such kind of accidents can be categorized into two main types. First, there are many blind areas which driver cannot observe directly around these machines because of limited vision scope, the tremendous size and diverse shapes. Second, though various sensors are mounted around the vehicle to detect blind spots, the sensory systems posses only the functionality of measuring data and are not able to process them meanwhile. These realistic problems motivated us to develop and design safety system described in those scenarios. Similar as other object detection tasks, human detection is a challenging endeavor because of Sensing noise, Environmental variations, Similarity to background signal, Appearance variability and unpredictability, Similarity to other people and Active deception according to the statement [2]. Environmental variations may be the most significant factor due to unexpected or abrupt changes in outdoor environment affecting directly the quality of data sensing. For example, sudden variations of lighting, foggy, snowing or raining weather conditions could

2 2 introduce tremendous noises to the sensory system especially to vision systems. These kind of factors would directly have a strong impact on higher false positive rate and lower detection rate. False positive rate decides the usability whereas the detection rate leads to the performance of the safety system. Various types of sensors have been used to detect humans so far. Radar sensors are widely utilized in the automotive industry nowadays since it was first successfully implemented in the military industry decades ago. Typical long range radar sensor from Bosch is now already in 4th generation since its first introduction into the market in 1990 s [3]. Adaptive Cruise Control which is available in most cars nowadays employs this sensor. The radar sensors are also applied in human detection in [4], [5]. There are also system implementations which used laser scanners and thermal infrared systems as alternative ones. Braun et al. detected human face with RGB camera and tracked human legs using laser scanner in [6], and Fuerstenberg developed pedestrian protection system with only laser scanner in [7]. Although all of these sensors could achieve relatively high detection rate, they lose most of visible information so that it is not suitable for short range object detection. Instead, vision sensors capture all visible data and cost relatively cheaper. Based on real time requirements, one of the main problems is the reduction of computational effect of the detection algorithm. We managed to solve this problem by applying linear support vector machine (SVM) and integral images. Discussing about detection algorithms, a classifier needs to be trained offline for the specific object. Two kinds of greedy searching methods are mainly used. One is to fix the size of searching window and to change the size of the whole image in each pyramid layer. The other one is to fix the size of the whole image and to adapt the size of searching window for each scanning level. Dalal and Triggs used histogram of oriented gradients (HOG) to detect human [8]. They trained human detection classifier using linear SVM and used first type of searching method to scan whole image. Instead, Viola and Jones applied Haar-like features and the second searching strategy mentioned above to scan face area in the image [9]. The classifier was trained using Adaboost and the program could run in real time. The trick they used is integral image which could calculate sum of the pixels inside specific region within constant time complexity. The performance of the detection algorithm depends on features used to extract descriptor, length of descriptor, learning algorithm and window scanning strategy. The remainder of this paper is organized as follows. Section 2 introduces fisheye camera and its mathematical model used in the system. Section 3 discusses local binary pattern (LBP) feature applied into the system and window scanning algorithm. Section 4 introduces training procedure of the linear and kernel intersection SVM and its potential problems more in detail. Section 5 will show the process of estimating ego-motion of a vehicle relative to the ground plane based on vision sensor and the principles of detecting foreground object based on this ego-motion. The experimental result will be described in section 6 and section 7 gives conclusion and future work.

Comparing with stereo cameras which need too much time on calculating depth image in preprocessing step, only one calibration step is enough in fish-eye camera.

3 3 (a) Mobotix Camera (b) Captured Image (c) Re-projection Error (d) Calibration Fig. 1. Vision System Overview and Calibration Results 2 Fish-Eye Vision Sensor One of the big advantages of fish-eye camera is the view range of 360 degree. Comparing with stereo cameras which need too much time on calculating depth image in preprocessing step, only one calibration step is enough in fish-eye camera. Once it is calibrated and the height of camera is known, then the distance between detected object and the point which corresponds to the center of camera could be estimated. There are only a few literatures available on fish-eye camera. Scaramuzza et al. proposed omni-directional visual odometry in [10], and Gandhi and Trivedi used omni-camera to detect interesting events such as independently moving persons and vehicles in [11] by estimating parametric planar motion model. Both of them are not focused on solving safety issues using such sensor.

4 The vision sensor used in this work is Mobotix Hemispheric Camera 1 as shown in Fig. 1(a). It can provide 180 degree horizontal field of view and 160 degree vertical field of view.

4 4 The vision sensor used in this work is Mobotix Hemispheric Camera 1 as shown in Fig. 1(a). It can provide 180 degree horizontal field of view and 160 degree vertical field of view. One of the troublesome problems is that it is difficult to demonstrate human in the posture of upright in the fish-eye image scene, but most of state-of-art algorithms assume that humans are upright in the image scene. Therefore, it is necessary to do a pre-processing step to get a normal view. We exploit the algorithm and toolbox from [12] to calibrate the camera to get mapping relationship between world coordinate and camera coordinate. Fig. 1(b) is one of the image used for calibration with chessboard; Fig. 1(c) shows the distribution of re-projection error of each point for all the chessboard, different colors correspond to the different images of the chessboard; Fig. 1(d) displays the position of every chessboard with respect to the reference frame of the fish-eye camera. Then the critical area predefined inside omni-view can be mapped to the normal view with these parameters. In this procedure the reverse mapping technique is chosen to overcome aliasing effect. Final image processing will be done on the image size of 720 by 240 as shown in the right-handed side of Fig. 2. Width of the right image corresponds to angle inside left one and height of the right image corresponds to radius inside left one. It is not necessary to map all omni-view image but interesting area into the perspective view because the distortion is very large on the pixels which are near to the center of omni view. In the case of large distortion, sensors like ultrasonic or other vision based algorithms can be applied to detect objects in such area. Each region of interest could be represented by four parameters like angle min, angle max, radius min and radius max. The unit of angle and radius are respectively radian and pixel. These values are set to radian, radian, 40.0pixel, 270.0pixel in the designed system. Fig. 2. Projecting the image in fish-eye camera to a landscape view 3 Classifier based Detection A lot of work about vision based human detection have been done by other researchers. In the past, template matching based methods were used to detect 1

5 (a) (b) Fig. 3. Local Binary Pattern Descriptor humans inside the image, but recently, descriptor based methods are becoming preferring choice for human detection.

Then greedy searching strategy is applied to scan whole image with pre-trained classifier to judge the existence of human in specific position.

5 5 (a) (b) Fig. 3. Local Binary Pattern Descriptor humans inside the image, but recently, descriptor based methods are becoming preferring choice for human detection. A human detection classifier is trained by learning extracted positive and negative descriptors from the samples of training dataset. Then greedy searching strategy is applied to scan whole image with pre-trained classifier to judge the existence of human in specific position. Since HOG [8] was first proposed in 2005, many descriptors were proposed to get optimized balance between speed and accuracy. Some of the researchers tried to use GPU to get higher speed, but these kinds of implementations were computationally expensive so that it is not suitable for on-board robotic system. Wu et al. proposed a real-time human detection algorithm using contour cues with CENTRIST descriptor [13]. Inspired by [13], contour information is further used in the designed system to extract human descriptor, but Local Binary pattern (LBP) replaces the original CENTRIST feature. As seen in Fig. 3, each patch is divided into several grids, and then green box scans the whole patch 2 and makes final query descriptor. In case of Fig. 3, the patch is divided to 3 by 4 small grids. For each green box, the length of LBP descriptor is 256 and the total length of the final descriptor is = 1536 as there are 2 3 = 6 possible positions of green box. For each pixel inside green box the index of LBP descriptor can be calculated as shown in Fig. 3(b). The index can be represented as (C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 ) 2 where C i is set to 1 if the corresponding neighbor pixel value is higher than the current pixel value otherwise it is set to 0. Cascaded detector which combines linear SVM and histogram intersection kernel SVM(HIK SVM) is trained at the end as shown in Fig. 4. Human detector which has patch size of 90 by 40 and 108 by 36 have been trained, respectively. 2 Here patch is Sobel edge image.

6 6 Fig. 4. Cascade in Detection 4 Training Procedure SVM is proven as a good option for binary classification problem. As discussed in the previous section, there are two types of SVM, which are applied in the system; One is linear SVM and the other is HIK SVM. Given a training set which consists of positive set P and negative set N, linear SVM is trained by iterative approach. Each sample in the training set is represented by the extracted feature vector described in the previous section. In the first iteration, the classifier is trained by P and randomly selected subset N 1 from N. We note this classifier as L 1. Similarly, L 2 is trained by P and N 2, where N 2 is the hard example on the negative set N produced by classifier L 1. In general, classifier L n is trained by P and N n, where N n is the collection of the hard example which is classified as positive by all the classifiers L 1,, L n 1. Then the final linear SVM could be trained by the positive sample set P and combination of negative sample set from each iteration procedure, which can be denoted as N final = N 1,, N n. However, there is some problem in this linear SVM. If this final trained classifier L final would be applied to other negative test dataset N test, it is of high possibility to have hard examples inside the indistinct area as shown in Fig. 5(b). This indistinct area could be depicted as two positive and negative support vectors as shown in Fig. 5(a). HIK SVM is designed to solve this problem, it is trained by positive set P and hard examples produced on the new negative test set N test by final linear SVM classifier L final. By cascading these two types of SVM, false positive rate could be reduced to much lower level and precision of the classifier also remains as high as possible, but this cascaded structure cannot affect recall rate instead. According to our research, recall rate mainly depends on the selection of positive training set P. More details about recall rate would be discussed in the experiment section.

7 7 (a) (b) Fig. 5. SVM Training 5 Ego-Motion based Detection Ego-motion is very important for the vehicle localization, especially in absence of GPS information or with slowly updating frequency. This algorithm is used to detect objects with independent motion or with obviously different height to the ground plane. Gandhi and Trivedi proposed similar method in [11], but they considered parametric planer motion model, which means they first did transformation from omni-view to perspective view and then estimated the homography. On the contrary, homography matrix is estimated directly in the proposed system, which is faster than the previous one. If X 1 and X 2 are coordinates of the same scene point in camera frame 1 and 2, then their relationship can be represented as X 2 = RX 1 + T (1) where R and T are rotation and translation matrix. If assume X 1 lies on the plane represented by normal vector n R 3, then the relationship can be written as n T X 1 = h, (2) substituting Eq. (2) into Eq. (1), we get X 2 = (R + T nt h )X 1. (3) Eq. (3) is the definition of homogeneous relationship between X 1 and X 2 and this can be simply written as X 2 = HX 1. (4) It is important to mention that in Eq. (4) scaling factor is already considered. First, good corner points are calculated by Shi-Tomasi corner detection

8 (a) (b) Fig. 6. Ego Motion Example algorithm [14], and then these points are tracked by optical flow based KLT feature tracker. The result of this first part is shown in Fig. 6(a).

After estimation of final homography matrix H, back projection needs to be implemented from current frame to the previous frame using X1 = H 1 X2 to calculate corresponding point in the previous

8 8 (a) (b) Fig. 6. Ego Motion Example algorithm [14], and then these points are tracked by optical flow based KLT feature tracker. The result of this first part is shown in Fig. 6(a). Random sample consensus (RANSAC) algorithm is applied to estimate accurate homography matrix in the next step for filtering out possible errors. After estimation of final homography matrix H, back projection needs to be implemented from current frame to the previous frame using X1 = H 1 X2 to calculate corresponding point in the previous frame. Using Eq. (5), region which has large difference with current pixel value can be estimated. The result is shown in Fig. 6(b). Red points correspond to the region which may contain foreground. f (X2 ) f (X1 ) > θ, X1 F rameprevious (5) Post processing is needed after this step. First, eroding and dilating operations are applied in order to filter out too small errors. Second, contour searching algorithm is further applied on the result image from first step to extract possible shape information of the object. Third, several conditions are further checked for each detected contour. These conditions are summarized as below: If the size of the contour is too small or too big, then it is filtered out. If the ratio between area of contour and area of the minimum enclosing circle of that contour is too small, then the contour is filtered out. Fig. 7 shows the result of ego-motion based algorithm. Fig. 7(a) shows the analyzed result of optical flow; blue points are the outliers calculated by the RANSAC algorithm. Most of the outliers are on the boundary of the image and vehicle itself. Fig. 7(b) shows the middle result after back projection of homography matrix H. There are a lot of red points on the boundary of the vehicle because of the vibration. Fig. 7(c) shows the final result where possible contour is colored as white and false contour is colored as gray. The test video used in Fig. 7 consists of 198 frames, where there are in total 76 people appeared around the machine. Our Ego Motion algorithm successfully detects humans 65 times with 2 times false detection, which means the precision

9 (a) Optical Flow (b) Foreground (c) Final Result Fig. 7.

6 EXPERIMENTS Currently, there is no public benchmark dataset available for fish-eye camera, therefore, we have made ground truth by ourselves.

Area (Rq Rg ) >0.5 Area (Rq Rg ) (6) Matching criteria from Eq. (6) is used to compare two rectangles for the detection algorithm.

9 9 (a) Optical Flow (b) Foreground (c) Final Result Fig. 7. Ego Motion Result and recall rate of the proposed algorithm on this video are about 97% and 86%, respectively. 6 EXPERIMENTS Currently, there is no public benchmark dataset available for fish-eye camera, therefore, we have made ground truth by ourselves. There are in total 430 frames in this test sequence. Test results from designed human detection algorithm and ego-motion based algorithm are shown in Table 1. Area (Rq Rg ) >0.5 Area (Rq Rg ) (6) Matching criteria from Eq. (6) is used to compare two rectangles for the detection algorithm. The overall recall rate of the detection algorithm is 55.47%, but it becomes 70.46% when the distance between human and the camera is in the range of 1 3m because of minimal distortion in this range in the transformed image. Here the distance describes the journey between object and the point which go through the center point of camera vertically.

10 10 Fig. 8 shows the recall rate of human detection algorithm in different ranges more in detail. The ranges used in the test are 0 1m, 1 2m, 2 3m, 3 4m, 4 5m, 5 8m. Normally, there is large distortion in the transformed image in the near distance range; on the other point, if distance is too far away from camera, then objects are observed too small to be detected. This is why the recall rates in the range from 0m to 1m and 5m to 8m are very low. The recall rate inside range from 3m to 4m is relatively lower than other ranges of before and after because there were two humans appeared in this range, where occluded human was counted as a missing object in this case. Ego-motion based detector is designed in order to increase the recall rate within the range where distortion is very large so that system could still detect humans while normal human detection algorithm would not work well. The recall rate of Ego-motion based detector reaches 91.38% in the test set. The correctness of this algorithm is based on the estimated homogeneous matrix H as already shown in Eq. 4. In this test set, there were totally 429 homogeneous matrix H computed where 37 of them were falsely estimated. This happens because of abrupt change of sunlight, shadow of the vehicle and vibration of the vehicle. Detection (Overall) Detection (1m - 3m) Ego Motion Recall 55.47% 70.46% 91.38% False Positive Per Frame Average Time Per Frame ms ms ms Table 1. Test Results Fig. 9 is kind of visual result of designed system. After detection of humans, distance from vehicle and height of each detected human will be estimated using camera calibration information as in Fig.9(a) and then the results were drawn in the original image as illustrated in Fig. 9(b). Fig. 9(c) is showing the radar information of which objects around vehicle can be perceived. 7 CONCLUSIONS AND FUTURE WORK A safety system for commercial vehicle using fish-eye camera is proposed. The significant contribuitons of this paper can be summerized as follows. This is the first research that employs fish-eye camera to solve the safety issues for the commercial vehicles. Classifier and ego-motion based detection apporaches are respectively introduced, implemented and compared as well inbetween based on experimental results. In order to promote the detection rate on commercial vehicles, a fusion of both methods could be the next target in the scope as well as the object learning from different viewing angles.

11 11 Fig. 8. Recall Rate Analysis In Different Range Acknowledgements This research has been supported by European Union Seventh Framework Program (FP7/ ) under grant agreement number and German Federal Ministry of Education and Research (BMBF). References 1. Administration, F.M.C.S.: Motor carrier safety progress report (as of september 30, 2013). Technical report 2. Teixeira, T., Dublon, G., Savvides, A.: A survey of human-sensing: Methods for detecting presence, count, location, track, and identity. (2010) 3. Bosch: Fourth-generation long-range radar 4. Chang, S., Mitsumoto, N., Burdick, J.: An algorithm for uwb radar-based human detection. In: Radar Conference, 2009 IEEE. (May 2009) Tutusaus, M., Koponen, S.: Evaluation of Automotive Commercial Radar for Human Detection. (2008) 6. Braun, T., Szentpetery, K., Berns, K.: Detecting and following humans with a mobile robot. In: Proceedings of the EOS Conference On Industrial Imaging and Machine Vision. (June 2005) 7. Fuerstenberg, K.: Pedestrian protection using laserscanners. In: Intelligent Transportation Systems, Proceedings IEEE. (Sept 2005) Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In Schmid, C., Soatto, S., Tomasi, C., eds.: International Conference on Computer Vision & Pattern Recognition. Volume 2., INRIA Rhône-Alpes, ZIRST-655, av. de l Europe, Montbonnot (June 2005) Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vision 57(2) (May 2004)

12 (a) (b) (c) Fig. 9. Result Show 10. Scaramuzza, D., Siegwart, R.: Appearance-guided monocular omnidirectional visual odometry for outdoor ground vehicles.

: Parametric ego-motion estimation for vehicle surround analysis using an omnidirectional camera. Machine Vision and Applications 16 (2005) 85 95 12. Scaramuzza, D.

In: Proceedings of the Fourth IEEE International Conference on Computer Vision Systems. ICVS 06, Washington, DC, USA, IEEE Computer Society (2006) 45 13. Wu, J.

12 12 (a) (b) (c) Fig. 9. Result Show 10. Scaramuzza, D., Siegwart, R.: Appearance-guided monocular omnidirectional visual odometry for outdoor ground vehicles. Robotics, IEEE Transactions on 24(5) (2008) Gandhi, T., Trivedi, M.: Parametric ego-motion estimation for vehicle surround analysis using an omnidirectional camera. Machine Vision and Applications 16 (2005) Scaramuzza, D., Martinelli, A., Siegwart, R.: A flexible technique for accurate omnidirectional camera calibration and structure from motion. In: Proceedings of the Fourth IEEE International Conference on Computer Vision Systems. ICVS 06, Washington, DC, USA, IEEE Computer Society (2006) Wu, J., Tan, W.C., Rehg, J.M.: Efficient and effective visual codebook generation using additive kernels. J. Mach. Learn. Res (November 2011) Shi, J., Tomasi, C.: Good features to track. In: 9th IEEE Conference on Computer Vision and Pattern Recognition, Springer-Verlag (1994)

Multi-Object Tracking Based on Tracking-Learning-Detection Framework

Multi-Object Tracking Based on Tracking-Learning-Detection Framework Songlin Piao, Karsten Berns Robotics Research Lab University of Kaiserslautern Abstract. This paper shows the framework of robust long-term