Semi-automatic Parking System Recognizing Parking Lot Markings

Transcription

1 Proceedings of AVEC 06 The 8 th International Symposium on Advanced Vehicle Control, August 20-24, 2006, Taipei, Taiwan AVEC Semi-automatic Parking System Recognizing Parking Lot Markings Ho Gi Jung 1,2, Chi Gun Choi 1, Pal Joo Yoon 1, Jaihie Kim 2 1 MANDO Corporation Central R&D Center 413-5, Gomae-Dong, Giheung-Gu, Yongin-Si, Kyonggi-Do , Republic of Korea Phone: (82) Fax: (82) Yonsei University, School of Electrical and Electronic Engineering 134, Sinchon-Dong, Seodaemun-Gu, Seoul , Republic of Korea Phone: (82) This paper describes a semi-automatic parking system, which can designate target position by recognizing parking lot markings. Driver is able to designate or refine the target position by drag&drop based novel user interface. Peak pair detection and clustering in Hough space recognize marking lines. Specially, one-dimensional filter in Hough space is developed to incorporate a priori knowledge about the characteristics of marking lines in bird s eye view edge image. Utilizing the structure of parking lot marking line-segments, system recognizes guideline and dividing marking line-segments. By dragging the inside of target position on touch screen, driver can translate the target position. By dragging the outside of target position on touch screen, driver can rotate the target position. Experiments show that the proposed algorithm successfully recognizes parking lots even when adjacent vehicles occlude parking lot markings severely. Furthermore, with the proposed novel user interface, driver is able to set target position in two or three handlings. Topics / Computer Vision, Driver Assistance Systems, Intelligent Vehicle 1. INTRODUCTION Semi-automatic or full automatic parking system is one of the most interesting driver assistant systems. Generally, novice, female and old driver feels constraint in parking backward between vehicles. J. D. Power s 2001 Emerging Technology Study, which found that 66% of consumers were likely to purchase parking aid, is a good proof [1]. Many upper class cars adopt ultrasonic parking assist system, which warns the driver of close distance to obstacle. Recently, car and component manufacturers started to provide vision based parking assist system. Toyota and Aisin Seiki introduced Back Guide Monitor, which helps the driver by projecting the predicted driving course on the image of a rear view camera [2,3]. Aisin Seiki s next generation is expected to include circumstance recognition function to provide an optimistic view to the driver [4]. They use wheel speed sensor, structure from motion technology and virtual camera, i.e. IVR (Intermediate View Reconstruction) technology, to make a virtual rendered image from an optimistic viewpoint. Automatic parking system automates parking operation with automatic steering control and automatic braking control. Automatic parking system consists of three components: path planning including the localization of target position, automatic steering and braking system used to implement the planned trajectory, HMI (Human Machine Interface) used to receive the driver s input and provide the visual information of ongoing parking process. The localization of target position can be implemented by various methods, e.g. fully manual designation [2], GPS infrastructure [5] and the vision based localization of free parking site [6,7]. Toyota s IPA (Intelligent Parking Assist) is a semiautomatic parking system, which leaves the braking control as driver s responsibility. Toyota s IPA developed the localization of target position by HMI, which shows a potential target position on the image of a rear view camera and enables the driver to change the target position with direction control buttons such as up, down, left, right and rotation [2]. Although semiautomatic parking system becomes commercialised, fully manual designation is too tedious and complicated for daily usage. Therefore, it is natural that the need of the vision-based localization of free parking site is increasing rapidly. Nico Kaempchen developed stereo vision based pose estimation of parking lots, which uses feature based stereo algorithm,

2 template matching algorithm on a depth map and 3D fitting to the planar surface model of vehicle by ICP (Iterative Closest Point) algorithm [6]. The vision system uses the disparity of vehicles but ignores all the information of parking site marking. Jin Xu developed color vision based localization of parking site marking, which uses color segmentation based on RCE neural network, contour extraction based on least square method and inverse perspective transformation [7]. We have been trying to develop economically practical vision solution for semi-automatic parking assist system. It is supposed that binocular stereo vision and motion stereo vision are too expensive to be adopted by early mass-produced parking systems. Furthermore, monocular vision system, which recognizes only parking lot marking, should be able to manage occlusions by adjacent vehicles. This paper proposes a monocular vision-based target position designation, which can recognizes parking lot markings and provides drag&drop based user interface. 2. PARKING LOT MARKING RECOGNITION 2.1 Bird s Eye View Edge Image generated from the bird s eye view image using Sobel edge detector. In this research, 3x3 horizontal and vertical Sobel edge detector is used. Fig. 1(d) shows the resultant edge image. 2.2 Hough Transform of Edge Image Hough Transform is one of the most popular methods for the detection of line in binary image. A pixel (x,y) in binary image is transformed into a set of parameters (θ,d), each of which is corresponding to a line passing the pixel in the binary image. If all pixels in a binary image are transformed into corresponding parameter sets and contributions of the parameter sets are accumulated in Hough space, a line in binary image forms a peak in Hough space. Therefore, peak detection in Hough space can recognize lines in binary image [10]. In general, Hough transform uses normal vector direction φ and distance ρ as Hough space axes. In this research, for the sake of easier interface with another components of vision system, somewhat different axes are used: orientation angle θ instead of normal vector direction φ, signed distance d instead of positive distance ρ. With the new axes, the distance between two lines can be calculated simply by subtracting their signed distances d. In the case of the general axes, calculating distance between lines should consider the range of normal vector directions. Equation (1) shows the definition of θ and d. (a) Input image (b) Undistorted image y = a x+ b= tanθ x+ b d = b cosθ (1) (c) Bird s eye view image (d) Edge image Fig. 1 Construction of edge image Rear view image captured with wide-angle lens is transformed into bird s eye view edge image. First, input image is undistorted by radial lens distortion model. If r d denotes the distance between a point P d (x d,y d ) and image centre O d (O xd,o yd ) in the input image coordinate system and r u denotes the distance between a point P u (x u, y u ) and image center O u (O xu, O yu ) in the undistorted image coordinate system, then r d can be converted to r u with 5th order polynomial and vice versa [8]. Second, the undistorted image is transformed into bird s eye view image with homography H, which defines the one-to-one relation between undistorted image coordinate system and bird s eye view image coordinate system [9]. In our system, camera is supposed to have constant height and tilt angle. Therefore, homography is measured once during camera calibration procedure. Fig. 1(c) shows the bird s eye view image of input image. Third, edge image is If there are many lines in edge image, there will be corresponding peaks in Hough space. Because the height of a peak in Hough space depends on the length of corresponding line-segment, it is impossible to detect valid peaks with a fixed threshold. Furthermore, because contributions from several lines are overlapped in Hough space, interference between peaks is unavoidable [11]. To overcome the difficulties, various methods have been developed. Among them, methods using butterfly pattern provide consequent mathematical understanding about Hough transform and show robust detection performance [10]. The expression butterfly pattern is from the fact that the distribution around a peak looks like a butterfly. Using butterfly pattern, it is also possible to detect the start point, end point and thickness of a line-segment. However, because analysing distributions around all peaks requires enormous computational load and time, such kind of methods are hard to be applied to real time applications. To overcome the drawback of methods using butterfly pattern, distribution pattern specified for parallel line-segment pair is devised. It is found that grasping the characteristics of distribution pattern can reduce search range, consequently eliminates the effect of noise and improves the computational speed. Fig. 2 shows that a parallel line-segment pair with fixed

3 distance W is transformed into characteristic distribution pattern in Hough space. Two peaks corresponding to respective line-segments have the same θ value and are distant by W in d axis. Furthermore, the heights of peaks are all the same and there is a deep valley between the peaks. Fig. 3 Structure of 1D filter (a) Edge image (b) Hough transform Fig. 2 Hough transform of parallel line-segment pair 2.3 Marking Line-segment Recognition 1D(one dimensional) filtering and clustering in Hough space are developed to detect peak pairs, whose two peaks have the same θ value and fixed between-distance W in the d axis of Hough space. There is no general characteristic of the peak pair in θ axis direction, because the wing width and starting point of butterfly pattern are various depending on the location and length of line-segment. Therefore, 1D filtering in d axis direction, which incorporates a priori knowledge about the peak pair in Hough space, detects candidates of peak pair. Consecutively, candidate clusters are detected by morphological dilation and connected component detection. It is certified that the centroid of each cluster is a good estimate of a marking line parameter corresponding to the peak pair. Equation (2) and Fig. 3 show the structure of designed 1D filter. HS(,d) is the value of a Hough space element at (,d). The designed filter is based on the fact that if the investigated coordinates in Hough space is (,d) and it is a valid marking line-segment, HS(,d), HS(,d-W) and HS(,d+W) are supposed to be valleys, simultaneously, HS(,d-W/2) and HS(,d+W/2) are supposed to be peaks. P A, P B denotes respectively each value of two peakness-testing coordinates. To reduce the effect of assumed width W and improve detection robustness, peakness-testing finds a maximum value within a predefined range in d axis direction. S(,d) is the application result of the designed filter. Only when S(,d) is positive, L(,d) is defined as a likelihood ranging between 0~1. When the valley values are all zeros, S(,d) has its maximum value, P A +P B. Therefore, Dividing S(,d) by P A +P B normalizes L(,d) for it to be between 0~1. In addition, multiplying the ratio, minimum peak value over maximum peak value, encourages the case when the two peaks have the same heights. If L(,d) is greater than the threshold dual peak, the coordinates (,d) in Hough space becomes a candidate of marking line-segment. S( θ, d) = HS( θ, d W) + P HS( θ, d) + P HS( θ, d + W) A B P = max HS( θ, d i) A i= 1~1 i= 1~1 W 2 P = max HS( θ, d + i) B W 2 min( PA, PB) S( θ, d), S( θ, d) > 0 L( θ, d) = max( PA, PB) PA + PB 0, S( θ, d) 0 (2) (a) Detected peak pair 1, L( θ, d) > θ C( θ, d) = 0, L( θ, d) θ dual peak dual peak (b) In edge image (c) In undistorted image Fig. 4 Recognized marking lines

4 Although almost candidates belonging to a line-segment form a cluster in Hough space, there exist some candidates that are not connected to the cluster depending on the thickness and length of the line-segment. To compensate the drawback of binarization, dilation using 5x5 rectangular kernel is used to robustly detect candidate clusters. Fig. 4(a) shows detected candidate clusters and their centroids. Fig. 4(b) and (c) show lines designated by detected centroids in edge image and undistorted image. It certifies that the designed 1D filter can successfully detect peak pairs and the centroids of candidate clusters are good estimates of line parameters. Marking line-segment is recognized as a section of detected marking line, which satisfies the condition of marking line-segment. In this case, a priori knowledge used in peak pair detection is used again. In other words, a marking line-segment consists of two parallel line-segments with fixed distance W in edge image. Introduction of hysteresis in the procedure improves system s robustness against noise. To determine the start point, likelihood should be greater than startchecking threshold continuously for W/2. Inversely, to determine the stop point, likelihood should be less than stop-checking threshold for W/2. Therefore, thresholding with hysteresis prevents possibilities that short-term variation of likelihood disturbs the recognition of start point and stop point. P stop, u(p start,p stop ). Modified distance from a point P to a line-segment S, D(P,S), is defined like equation (3) and a line-segment with minimum modified distance is selected as a guideline. Fig. 6 shows the detected guideline. ( start start stop ) D( PS, ) = distance( PS, ) upp (, ) up (, P ) (3) Fig. 6 Recognized Guideline Using the difference between the intensity of pixel on marking and the intensity of pixel off marking along the guideline, marking line-segment dividing parking lots can be detected. Parking lot markings consist of one guideline separating parking lots from roadway and dividing marking line-segments normal to the guideline. By detecting a position where the intensity difference becomes small definitely, T -shape junctions between the guideline and dividing marking line-segments can be recognized successfully. Furthermore, because T -shape junction is searched along a whole line extending the guideline, additional dividing line-segments, which are not detected during peak pair detection because they are located far and blurred, can be detected. Fig. 5 Recognized marking line-segments 2.4 Parking Lot Recognition Among the recognized marking line-segments of parking lot markings, guideline is selected using an assumption that it is near from the camera and likely to be normal to the gaze direction. Guideline is the most important line-segment as the reference of parking lot recognition, because it is the border between parking lots and roadway. In accordance with the foot of a perpendicular, distance between a point and a line-segment is defined as one of two values: distance between the point and a line extending the line-segment, minimum of two distances between the point and two endpoints of the line-segment. How well a marking line-segment is normal to the gaze direction can be measured by the inner product of two unit vectors: a unit vector from the camera position P to the start point P start, u(p,p start ) and a unit vector from the start point P start to the stop point Fig. 7 Recognized Parking Lot Markings Proposed method for the recognition of parking lot markings successfully operates in the situation when the adjacent lots are occupied by vehicles. Distortions unsatisfying the constraint will be removed through the peak pair detection in Hough space and the line-segment recognition. If vehicles are normally parked in the adjacent lots beside target position, distortions in bird s eye view image caused by these vehicles are supposed to locate beyond the guideline along the perspective direction. Consequently, in spite of the adjacent

5 vehicle s distortion, proposed method can detect the guideline successfully. Once the guideline is recognized correctly, T -shape junction searching can detect dividing marking line-segments and removes the rest noise. Even if the illumination is too dark and the most portion of parking lot markings are undetectable, guideline tends to appear as a long and distinguishable edge pair. Therefore, guideline can be certainly detected. Once the guideline is recognized, dividing line-segment recognition using weak condition can detect dividing marking line-segments that are missed by the marking line-segment recognition. 3. USER INTERFACE Driver can designate or refine target position using drag&drop based user interface. Captured scene of parking site is displayed on small touch screen monitor and current target position is displayed. The inside of region of rectangular target position is used as a moving cursor. The outside region of rectangular target position is used as a rotating cursor. User-input position in input image coordinate system is translated into bird s eye view image coordinate system using fisheye model and homography. Once system acquires consecutive user-input positions, it tests whether they are inside or outside of target position rectangle. If the user-inputs are in the target position, the difference of consecutive user-input positions is treated as a translation vector. If the user-inputs are out of the target position, the angular difference of consecutive user-input positions with respect to target position s centre is treated as a rotation angle. Fig. 8 shows the basic idea of moving and rotating operation. the multi-turn parking trial situation yet. System checks whether it can track the planned path before the start of parking operation. Fig. 9 Path planning If the path generation is successful, it is converted from Cartesian coordination to vehicle travel steering angle space for steering control. Pose estimation uses popular Ackerman (or bicycle) model, assuming that no slip of wheels occurs due to low speed. The reference point of the vehicle motion is assumed to be located at the centre of rear axle. When the system guides vehicle to park, the operation mode of R-EPS changes from normal power assist mode to active steering control mode. During vehicle progresses to the target position, steering angle is controlled by the R-EPS. Parking controller receives vehicle travel pulse from transmission and steering angle from R-EPS. The steering angle command at the moment is determined by referring to the vehicle travel steering angle profile. With a serial link such as CAN channel, parking controller sends steering command to R-EPS controller, which performs angular position control. Although longitudinal control of vehicle is driver s responsibility, visual and acoustic guidance informs the driver of over-speed warning and the completion of parking operation. 5. EXPERIMENTAL RESULTS (a) Moving the target (b) Rotating the target Fig. 8 Drag&Drop based user interface 4. VEHICLE GUIDANCE SYSTEM After the localization of target position, parking controller generates parking path for semi-automatic guidance. Planned path basically consists of two lines: line extending current vehicle s rear direction, line extending the central line of target position. A circular segment whose radius is the smallest rotating radius of installed steering system connects the transition part of two lines as depicted by Fig. 9. Although proposed semi-automatic parking assist system can implement both perpendicular (or garage) parking and parallel (or street) parking situation, this paper concerns only perpendicular case. Furthermore, current semi-automatic parking assist system does not consider (a) Rear view camera (b) Touch screen Fig. 10 Test environment Fig. 10 shows our test environment. A monocular CMOS camera is installed at the backend of test vehicle and touch screen is installed at radio position. Vision and control algorithm are fulfilled on an industrial PC. Fig. 11 Moving target position by drag&drop

6 Proposed drag&drop based user interface is tested. Various kinds of persons have tried our semi-automatic parking system using the user interface. Fig. 11 shows the situation when driver is moving target position by dragging the inside of target rectangle in bird s eye view coordinate system. Through various test, we become to ensure that with the proposed drag&drop user interface general driver can designate target position in two or three handlings in almost situations. Fig. 12(a) shows that the proposed system can successfully recognize parking lot markings in spite of adjacent vehicles. Fig. 12(b) shows that the proposed system can manage harsh illumination condition such as capturing against the sun and dark shadow. In the input image of Fig. 12(c), the ground surface is shining and causes a lot of noise in edge image. However, filtering in Hough space and structure analysis successfully recognize parking lot markings. (a) Case with occlusion by adjacent vehicles (b) Scene captured against the sun (c) Scene with shining ground surface Fig. 12 Result of parking lot marking recognition Table 1. Accuracy of parking control Dimension of Width 2.76m parking slot Depth 1.64m Deviation of Average 19.49cm final position Standard Deviation 5.99cm Deviation of Average 1.0 orientation Standard Deviation 0.4 To evaluate the feasibility and the performance of the semi-automatic parking system, test vehicle was built on a mid-size sedan. Experiment was executed for the perpendicular parking case on a plane surface. Deviation of final position between target and result was measured in various parking situations. Direction errors are also measured for all test cases. As a result, the average of position deviation is about 20cm and the standard deviation is about 5cm. The average deviation of the orientation is about 1.0 degree. Table 1 shows the parking control result. 6. CONCLUSION This paper proposes a semi-automatic parking system. Specially, 1D filtering in Hough space and drag&drop user interface are proposed for the target designation method. Experiments show that the proposed method provides comfortable user interface and verifies the possibility of semi-automatic parking system. REFERENCES [1] Randy Frank, Sensing in the ultimately safe vehicle, SAE Paper No. : , Society of Automotive Engineers, [2] Masayuki Furutani, Obstacle detection systems for vehicle safety, SAE Paper No. : , Society of Automotive Engineers, [3] Shoji Hiramatsu, et al., Rearview Camera based parking assist system with voice guidance, SAE Paper No. : , Society of Automotive Engineers, [4] K. Fintzel, et al., 3D vision system for vehicles, Proceedings of IEEE Intelligent Vehicle Symposium 2003, pp , [5] Massaki Wada, et al., Development of advanced parking assistance system, IEEE Transaction on Industrial Electronics, Volume 50, No. 1, pp. 4-17, [6] Nico Kaempchen, et al., Stereo vision based pose estimation of parking lots using 3D vehicle models, Proceedings of IEEE Intelligent Vehicle Symposium 2002, pp , [7] Jin Xu, et al., Vision-guided automatic parking for smart car, Proceedings of IEEE Intelligent Vehicle Symposium 2000, pp , [8] Jean-Yves Bouguet, Camera calibration toolbox for Matlab, ndex.html [9] Ho Gi Jung, et al., Stereo vision based localization of free parking site, LNCS Vol (CAIP 2005), pp , Sep [10] Yasutaka Furukawa, and Yoshihisa Shinagawa, Accurate and robust line segment extraction by analyzing distribution around peaks in Hough space, Computer Vision and Image Understanding 92 pp. 1-25, [11] Jongwoo Kim, Raghu Krishnapuram, A robust Hough transform based on validity, 1998 IEEE World Congress on Computational Intelligence, May 1998, pp vol. 2, 1998