A Vision-based Safety Driver Assistance System for Motorcycles on a Smartphone

Transcription

1 014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC) October Qingdao China A Vision-based Safety Driver Assistance System for Motorcycles on a Smartphone Chiung-Yao Fang Wei-Hong Hsu Chung-Wen Ma and Sei-Wang Chen Abstract In this study we develop a vision-based safety driver assistance system (DAS) for use with motorcycles. DASs have been studied by researchers and developed by car companies as a tool to avoid traffic accidents. However existing studies have seldom focused on motorcycle riders. In Taiwan the prevention of motorcycle accidents cannot be ignored when developing safety DASs since they are involved in a significant percentage of traffic accidents. This paper proposes a safety DAS for motorcycles which can obtain an input sequence from a smartphone detect the distance between the owner s motorcycle and an obstacle ahead and output warning messages if necessary. The study is divided into two parts. Firstly transformation matrices of different camera angles are calculated and stored in a transformation matrix database. Secondly a kernel system is developed. The system can turn each input frame into a top-view image using a suitable transformation matrix from the database. Further the system can detect obstacles in the top view image and calculate the distance between the owner s motorcycle and the obstacles. Concurrently the GPS embedded in the smartphone can be used to obtain the speed of the motorcycle in order to calculate the safe distance. Finally the system gives the rider suitable warning messages after comparing these two distances. The experimental results show that the proposed real-time system is robust and efficient. Moreover the system is also suitable for deployment in cars. I. INTRODUCTION Recently many safety driving assistance systems (DAS) have been studied by researchers and developed by car companies as a tool to avoid traffic accidents [1]. These have achieved a reduction in fatal car accidents. In recent years the number of people killed in car accidents is decreasing in Taiwan []. However the number of people killed due to motorcycle accidents has not decreased in line with car accidents []. Therefore to develop a safety DAS for use with motorcycles is necessary and imperative. This study presents a vision-based safety DAS for motorcycles which can detect obstacles in front of a motorcycle and estimate the distance between the motorcycle and the nearest obstacles. This data can then be used to decide whether the current situation is safe or not. The system will output a warning message to the rider to warn them of a dangerous situation. Figure 1. The hardware setup of the safety DAS for motorcycles. Figure. Illustration of pitch () and roll () angles. In comparison to the development of vision-based DAS for cars many different factors require consideration when developing a vision-based DAS for motorcycles. Firstly the activity degree of the motorcycle handlebar is higher than that of the car wheel. This means the camera mounted on the motorcycles will have pitch and roll rotations. These rotations increase the difficulty of obstacle detection in the videos obtained from the camera and therefore the DASs currently used in cars cannot be applied directly to motorcycles. Secondly the equipment used on a motorcycle should be sunproof waterproof and theft proof. Therefore a smartphone is used in this study since they are commonly available. They have sufficient computational ability and contain various embedded functional apps. An increasing number are also sunproof and waterproof. Moreover smartphones are easily mounted and dismounted from a motorcycle. A rider may dismount their smartphone when they leave their motorcycle thus they are also theft proof. Figure 1 shows the hardware setup of our safety DAS for motorcycles. The smartphone is positioned on an X type mobile holder mounted on the motorcycle handlebar. In this study the camera embedded in the smartphone is used to capture the video and the orientation sensor obtains the initial pitch and roll angles of the smartphone simultaneously. This data is used in the top view image translation. Further the GPS is used to determine the current position of the motorcycle. This data can be used to obtain the speed of the motorcycle. II. SYSTEM FLOWCHART As mentioned previously the camera mounted on the motorcycle may have pitch and roll rotations. Before Chiung-Yao Fang is with the Department of Computer Science and R.O.C. ( violet@csie.ntnu.edu.tw). Wei-Hong Hsu is with the Department of Computer Science and R.O.C. ( icegm00@gmail.com). Chung-Wen Ma is with the Department of Computer Science and R.O.C. ( chungweng@hotmail.com). Sei-Wang Chen is with the Department of Computer Science and R.O.C. ( schen@csie.ntnu.edu.tw) /14/$ IEEE 38

2 Figure 3. System flowchart. introducing the system flowchart two angles and related to the pitch and roll rotations should be defined. Figure illustrates these two angles where the dark blue area indicates the road region and the rectangle represents the camera. The green dotted lines are horizontal and the symbols and indicate the pitch and roll rotation angles of the camera respectively. The values of these two rotation angles are very important for the system to transform the input frames into top-view images. Figure 3 shows the system flowchart. The input frames from the video sequence are transformed into top-view images. The values of the angles and should be obtained before this transformation. These two values can be obtained initially from the smartphone and adapted using the information extracted from the input frames. The system can transform the input frames into top-view images quickly by using these two values as indexes to extract the transformation matrix stored in the database. Obstacles can be detected in the top view images by their shadow information and the nearest obstacle distance d obs between the motorcycle and the nearest obstacle ahead can be calculated. The system calculates only the distance d obs since the other obstacle distances are larger. If d obs is larger than the safety distance then the motorcycle is safe[3]. If d obs is smaller than the safety distance the system will output the warning messages. A. Top-view image transformation matrix Inverse Perspective Mapping (IPM) [4][5] can be used to transform the input frames into the top-view images. A pixel p at ( u v) of an input frame can be transformed into its corresponding pixel at in a top-view image by a transform matrix M. The positions of these two pixels can be represented by Homogeneous coordinates where their relationship is given as x u a b c u y M v d e f v 1 g h 1 1 where is a scaling constant. From Eq. (1) the following three equations can be obtained p' ( x y) gu hv 1 Figure 4. The flowchart for the TMD creation. Moreover au bv c x gu hv 1 du ev f y gu hv 1 x ua vb1c 0d 0e 0 f xug xvh y 0a 0b 0c ud ve1f yug yvh Given the positions and of one pair of pixels and respectively substitution for and ( x y) in Eqs. (5) and (6) gives two new equations. Once n pairs of pixels are given then the linear system p 1 p 1 ' ( 1 u 1 v ) ( 1 x 1 y ) ( u v) a u1 v x1u1 x1v1 x1 b u 1 v1 1 y1u1 y1v1 y 1 c u v xu xv x d u v 1 yu yv y e f un vn xnun xnvn xn g un vn 1 ynun ynvn yn h can be constructed. To solve this linear system one can obtain the parameters of the transformation matrix. It should be noted that at least four pairs of pixels should be used to solve this linear system. Moreover the input frames captured by different and angles require different transformation matrices. To calculate the transformation matrix online is time consuming and wastes CPU resources. In this study a transformation matrix database (TMD) is created in advance. The stored transformation matrices are indexed by and angles. In practice when the system estimates and online it can extract a corresponding transformation matrix from the TMD quickly. In the next section we will introduce the method to select the four pairs of pixels to find the transformation matrices and create the TMD. B. Transformation matrix database creation Figure 4 shows the flowchart for the TMD creation. Firstly a white square of cardboard with a length of one meter is placed in the road ahead of the camera on the smartphone. The distance between the white cardboard and the camera is six meters. When a frame is input into the system as shown in Figure 5 (a) for example the system automatically detects the white region from the input frame. The white region shown in Figure 5 on the image plane is projected by the white cardboard square. The four corners of the white region the 39

3 (a) (c) Figure 5. An example of region detection and corner extraction. (a) The input frame. The result of the region detection. (c) The result of the corner extraction. Figure 6. An example of coordinate correction. (a) Figure 7. A further example of coordinate correction. (a) green pixels shown in Figure 5 (c) are regarded as the four pixels whose positions can be used to solve Eq. (7). It should be noted that the positions of their corresponding pixels in the top-view images are predefined. The system reads values from the accelerometer in the smartphone and translates them to and defined in this study. The translation is necessary since the coordinate system of the smartphone is rotated when it is mounted on the motorcycle. The coordinates of the four points (c ru c ro c lu c lo ) will be classified into classes whose index is the translated and. (c ru c ro c lu c lo ) indicates a four-corner vector where c ru c ro c lu and c lo represent the right upper right lower left upper and left lower corners respectively. Since the input sequence contains many input frames and these frames are captured with the same and many vectors (c ru c ro c lu c lo ) will be classified into the same class C. They can be used to correct each other in the same class and some sets can be regarded as noise to be removed. Finally the transformation matrix of each class can be calculated and stored in the TMD. Coordinate correction: The coordinates of the four corners of the white region should be corrected since the camera may suffer from vibrations due to the outdoor environment. When the angle of the camera is zero two criteria are used to correct the coordinates in this study. (1) If = 0 the y coordinates of the two right corners c ru and c ro should be the same with those of the two corresponding left corners c lu and Figure 8. The coordinate system of the smartphone. c lo respectively. The average of the y coordinates of the two upper corners c ru and c lu are regarded as the new coordinate after y-coordinate correction. The correction of the two lower corners c lu and c ro is similar. () The coordinates of the corners on the same positions (e.g. the left-upper corner c lu ) in the same class should be the same. Therefore the average of these coordinates is used to represent the ( cru clu cro clo) coordinates of this class. Figure 6 shows an example of coordinate correction. The blue diamond s indicate the positions of the corners detected from the input frame. One can observe that the y coordinates of c ru and c ro are not equal to those of c lu and c lo respectively. The red squares indicate the positions of the corners after the y-coordinate correction and the green triangles indicate the average coordinates of the corners in the same class. Figure 7 shows another example of coordinate correction. In these cases the angle of the camera is non-zero. The system uses two corresponding frames in order to correct the coordinates. These two frames are captured by the same but different angles. Further the absolute values of these two angles are the same. For example the frames shown in the upper row of Figure 7 are the corresponding frames. and f (585) are captured by two situations ( 5 f (585) 85 and ( ) respectively. We let their corner vectors be and ( c ' ru c' lu c' ro c' lo ) respectively. In this case two criteria are used to correct the coordinates in this study. (1) The left upper corner of the frame and 5 85 ( cru clu cro clo) the right upper corner c' ru c lu of the frame f (585) f (585) ) should have the same y-coordinate. Therefore the average of the y coordinates of the two corners is regarded as the new coordinate after y-coordinate correction. The corrections of the other three corresponding corners c lo and c ro c ro and c lo c ru and c lu are similar. () The center of the left upper corner of the frame frame (585) f (585) and the right upper corner c' ru c lu of the f should be at the center line of the image frame where x = 180 in this example. Therefore the system shifts the coordinates of the corners to satisfy the criterion. Figure 7 shows the positions of the corners of the input frames before coordinate correction. In Figure 7 (a) the blue diamond s indicate the positions of the corners of the frame f while the red triangles indicate the positions of the (585) corners of the frame f (585). In Figure 7 the positions of the corners after coordinate correction are shown. The blue diamonds and the red triangles indicate the corrected positions of the corners of the frames f respectively. f (585) and (585) α and β translation: As mentioned previously the and angles relate to the pitch and roll rotations respectively of the camera mounted on the motorcycle. Figure 8 (a) shows the coordinate system of the example smartphone. When the smartphone rotates around its x-axis the pitch angle is changed. Similarly when the smartphone rotates around its y- axis the roll angle is changed. It should be noted that the 330

4 (a) Figure 9. An example of top-view image transformation. (a) The input frame. The corresponding top-view image. (a) Figure 10. An example of Sobel vertical edge detection and vertical line segment detection. (a) The result of the Sobel vertical edge detection. The result of vertical line segment detection. smartphone is mounted on the motorcycle handlebar horizontally as shown in Figure 1. This means the x-axis y- axis and z-axis are rotated as shown in Figure 8. The pitch angle given by the accelerometer of the smartphone is shown in Figure 8 not as defined in this study. Therefore the system can obtain the value of using arcsin(sin ' /sin ') Similarly can be obtained by arcsin(sin '/ cos ') Given the and angles the transformation matrix can be extracted from the TMD and the system can transform the input frame into a top-view image quickly. Figure 9 shows an example of the top-view image transformation where Figure 9 (a) is the input frame and Figure 9 shows its corresponding top-view image after transformation. C. Online top-view image transformation After the database is created the system should obtain and angles online in order to extract the suitable transformation matrix. In this study the angle is obtained from the input image while the angle is obtained from the accelerometer directly. Here we introduce the method used to obtain the angle. Given an input gray image A applying a Sobel vertical mask with Sobel edge detection can be used to detect the vertical edges and the edge image G v is obtained G v 0 A Figure 10 (a) shows an example of Sobel vertical edge detection. Based on the result of the Sobel vertical edge detection the system can apply a Hough transform and preserve the line segments corresponding to the vertical line. It should be noted that the vertical line is defined as = 90 ( is the angle between the line segment and x-axis of the image) in this study. Therefore the system preserves line segments with in the interval. An example is shown in Figure 10 where the red line segments show the vertical lines. Finally the system can estimate the angle by a weighted vote of s. If the current time is t the weight of each line segment is related to the previous angle at time t 1. The closer the angle of a line segment at time t and the previous angle at time t 1 the higher weight this line segment has. A Gaussian function is used to model this concept. This mechanism can stable the angles obtained from successive frames. In the example shown in Figure 10 = 90. As mentioned the angle is obtained from the accelerometer directly. However for stability the average of several previous angles is used as the current angle. Given and angles the system can perform an online transform of the input frames into top-view images using the transformation matrix extracted from the TMD. D. Obstacle detection Since the obstacles on the road may also have shadows sufficient detection of their shadows is also able to identify their location. Otsu s method [7] is a binarization method and can be used to detect the shadows of the obstacles in the image. Given a frame whose size is pixels each pixel has a gray level L. The following Otsu algorithm can calculate a threshold to extract the shadow features. 1. First construct the normalized histogram of the input image. Let n i indicate the number of pixels whose gray level is i. The normalized histogram value p i can be calculated as ni pi MN where M N i 01 L 1 These values can be regarded as the occurrence probabilities of gray levels in the frame.. Given a threshold k the pixels in the image can be divided into two classes: C 1 and C. Let C 1 represent the class of pixels whose intensity is in and C represent the class of pixels whose intensity is in. The occurrence probabilities of class C 1 and C can be calculated by the equations [ k 1 L 1] P ( k 1 ) k i0 p i L 1 p i ik1 [ 0 k] P ( k ) respectively. 3. Compute the sum of the occurrence probabilities of gray levels in the range [0 k]. k m( k) ip i i0 4. Compute the sum of the occurrence probabilities of gray levels in the range [0 L-1]. 5. Compute L 1 m G ip i i0 [ mg P1 ( k) m( k)] B ( k) P ( k) P ( k) 6. Repeat steps to 5 and find a threshold maximizes B 1 k which k arg max ( k) 0kL1 B 331

5 (a) (c) (d) Figure 11. An example of obstacle detection. (a) An input image. The top-view image transformation result. (c) The binarization result. (d) The obstacle detection result. Using as a threshold the system can binarize the topview images. Figure 11 shows an example of obstacle detection. Figures 11 (a) and are the gray input frame and its corresponding top-view image respectively. Figure 11 (c) shows the binarization result using Otsu s method. In Figure 11 (c) the white pixels whose intensity is in are regarded as the shadow areas. In the experiments the system only tests the threshold values between 10 and 19 ( ) to obtain k. This is to save computational time. Further the lowest pixels at each column are extracted to represent the lower bound of the obstacles shown as the green pixels in Figure 11 (d). Finally only the obstacle within the two red lines is selected as the obstacle in front of the motorcycle. The system is then able to calculate the shortest distance between the owner s motorcycle and the obstacles ahead. k [ 0 k] 10 k 19 E. Safety Distance Estimation The safety distance is estimated by the formula suggested by the Research Center for Traffic Accident Authentication National Cheng Kung University Taiwan [6]. The total stopping distance is the sum of the perception-reaction distance and the braking distance and is given as Moreover the perception-reaction distance can be calculated by t 1 s S d S d s s' s' 1 t1va0 3.6 s' V a0 where is the driver reaction time and indicates the motorcycle speed. The braking distance can be calculated as " 1 ' t Va s ( t 0 ) Va amax where indicates the maximum deceleration when braking a max ' t is the time from the braking system activating to " when it actually works and t is the time from the braking system working to it gaining the maximum deceleration a max. Using these equations the system can estimate the safety distance the motorcycle requires. In our experiments the motorcycle speed V a0 at time t can be calculated by st st' Va0 t where t t t' indicates the time interval. The symbols s t and s t' indicate the positions obtained from the GPS embedded in the smartphone at time t and t respectively. Moreover the values of the other parameters are set to t TABLE I. t ' 0. THE OBSTACLE DETECTION RESULTS. No. of input frames t " 0.7 No. of correct frames No. of incorrect frames and. III. EXPERIMENTAL RESULTS Precision rate Class 1: Tunnel % Class : Turning % Class 3: Speed % km/hour Class 4: Other cases % Total % In total 16 sequences (6061 frames) are captured by the smartphone (SONY XPERIA acro S) and used to test the performance of the proposed system. They are divided into four different classes. The sequence in class 1 is obtained when the motorcycle is driving in a tunnel environment. An example of class 1 is shown in Figure 1 (a). It should be noted that only six frames of this sequence are sampled and shown. The sequence in class is obtained when the motorcycle is turning left or right (Figure 1 ). The sequence in class 3 is obtained when the speed of the motorcycle is lower than 10 km/hour and is shown in Figure 1 (c). The final sequence is classified into class 4. Figures 1 (d) (e) and (f) are some examples of class 4. Figure 1 also shows the experimental results of these sequences. The value on the left upper corner of each frame indicates the current speed of the motorcycle. The value on the right upper corner of each frame indicates the safety distance calculated by the system. One can observe that the higher the speed of the motorcycle the larger the safety distance is. The area under the two pink lines of each frame represents the area ahead of the motorcycle. The obstacle ahead if any is circled in red within the pink boundaries. The system will send a warning message to the rider if any obstacle appears in this area and the distance between this obstacle and the motorcycle is shorter than the safety distance. Table I shows the obstacle detection results of our system. The precision rates of classes 1 3 and 4 are 7.1% 98.61% 99.33% and 99.71% respectively. The average precision rate is 98.49%. One can observe that the precision rate is lower in class 1 (only 7.1%) since the variation of the lighting in the tunnel is extremely large. Figure 13 shows an example of incorrect detection results in class 1. In this example the shadow of the tunnel is regarded as an obstacle by the system as can be seen in the second and third frames of Figure 13. IV. CONCLUSION In this study we have proposed a rider safety assistance system for motorcycles. The system detects the obstacle ahead of a motorcycle by the top-view images transformed from the input frames. A transformation matrix database has been created. The transformation indexes are the pitch and roll rotation angles of the smartphone. A Sobel vertical edge detection and Hough transform are used to determine the pitch rotation angle α and the orientation sensor of the smartphone is used to obtain the roll rotation angle β. From the top-view 33

6 (a) (c) (d) (e) Figure 1. Some examples of testing sequences. (f) Figure 13. An example of incorrect obstacle detection. image the system can estimate the distance between the obstacle ahead and the motorcycle and check if the distance is shorter than the safety distance. The safety distance is calculated by the speed of the motorcycle obtained from the GPS in the smartphone. The system sends a warning message if the distance between the obstacle ahead and the motorcycle is too short. In order to increase the obstacle detection precision rate more obstacle features should be detected and tested in the future. Using a smartphone as the hardware to embed the proposed system in suffers from some problems. Firstly the battery of the smart phone is easily depleted. Secondly the system may not be able to execute in real-time if the computation ability of the smartphone is insufficient. Thirdly only some types of smartphone are sun-proof waterproof and theft proof. However we believe the capabilities of smartphones are likely to be improved in the future. ACKNOWLEDGMENT This work was supported by the National Science Counci l Republic of China under Contract NSC E and NSC 10-1-E REFERENCES [1] L. Li and S. Zhu Design Concept and Method of Advanced Driver A ssistance Systems Proceedings of IEEE International Conference on Measuring Technology and Mechatronics Automation (ICMTMA) pp Hong Kong 013. [] Ministry of Transportation and Communications R.O.C. Statistical S ummary of Traffic Accidents - class A1 (Fatal Accidents). otc.gov.tw/ mocdb/stmain.jsp?sys= [3] Taiwan Area National Freeway Bureau MOTC R. O. C. Driving safe ty information =516& p= [4] M. Bertozzi A. Broggi and A. Facioli Stereo inverse perspective ma pping: theory and applications Image and Vision Computing Vol. 16 pp June [5] M. Nieto L. Salgado F. Jaureguizar and J. Cabrera Stabilization of I nverse Perspective Mapping Image based on Robust Vanishing Point E stimation Proceedings of the 007 IEEE Intelligent Vehicles Symposi um Istanbul Turkey pp June 007. [6] Research Center for Traffic Accident Authentication National Cheng Kung University The Analysis of Braking System t.edu.tw/~ccchang/yesnewtraffic/c/b/b01-a.htm 013. [7] R. C. Gonzalez and R. E. Woods Digital Image Processing Third Editi on Prentice-Hall New Jersey