Towards an Automatic Vehicle Access Control System: License Plate Location

Transcription

1 Towards an Automatic Vehicle Access Control System: License Plate Location Pedro R. Mendes Júnior, José Maria R. Neves, Andréa I. Tavares, David Menotti Computing Department Federal University of Ouro Preto (UFOP) Ouro Preto, Minas Gerais, Brazil {pedrormjunior, jmneves, andrea.iabrudi, Abstract An automatic vehicle access control system (AVACS) can be divided into three steps: vehicle location, vehicle license plate (VLP) location, and VLP recognition. This paper presents a new method for VLP location based on the horizontal gradient, morphological operations, connected components analysis, and statistical measures. First, the horizontal gradient is acquired and a mean filter is applied on it. Then, morphological operations are applied on the image composed of integer numbers in order to perform a filtering, such that saliences referring to VLP region are kept and the other ones are darkened. Finally, the image is binarized, then a connected component analysis is performed, and statistical measures are used for deciding among the VLP candidates. In the experiments, in 9.55% of the cases, in a database of 7 images, our method is able to correctly locate the VLP. Index Terms Vehicle license plate location; mathematical morphology. I. INTRODUCTION The Federal University of Ouro Preto (UFOP) has undergone changes recently caused mainly by REUNI (Support Programme for the Restructuring and Expansion of Federal Universities). This program has caused profound changes in various sectors of our university from education to physicalstructural. The number of vehicles currently in the parking exceeds the places properly flagged. Thus, it is necessary to study the expansion of this sector in the university. One way to perform this study is using an AVACS. An AVACS can be divided into three steps: vehicle location, VLP location, and VLP recognition. The development of such a system presents the biggest bottleneck or it is less robust in the VLP location. For the VLP location task, in the literature, we can find methods that process binary images [], [], [3], monochrome images [4], [5], and color images [6], and methods that work with more than one kind of image [7]. This paper presents a new VLP location method using digital images for the construction of an AVACS. Our method is the product of an in-depth study of some methods in the literature. Some operations in the proposed method are derived from methods such as [], [], [3]. VLP location methods based on mathematical morphology [8] aim to extract patterns from vehicle image using operations such as bottom- or top-hat [] and horizontal gradient [], and then to binarize it. The morphological operations are used in order to keep the VLP region and to remove the other ones. Such an approach may eliminate the VLP region in the binarization step. If there are other regions more emphasized in the initial operation, the binarization threshold obtained may not maintain the VLP region or only a part of it remains. Aiming to solve this problem, we propose a method as follows. Initially, it is computed the horizontal gradient image and performed a mean filter on it. Then morphological operations are performed in this image to further perform the binarization. A connected component analysis is used in order to build VLP region candidates. Statistical measures are used for deciding among candidates. For the validation of the method, we used two image databases: 377 images acquired by a digital camera at the gate of the UFOP campus and 345 images of Greek vehicles provided by [7]. In these databases, the proposed method achieved successful VLP location rates of 9.8% and 95.4%, respectively. We used 5-fold cross-validation to estimate the empirical parameters of the methods (the proposed method and the other ones implemented for comparison.) The remainder of this paper is organized as follows. In Section II, it is presented the steps of the proposed method. Our results and comparisons with results obtained by other methods in the literature are shown in Section III. Finally, in Section IV, we present possible future works that complement the current one. II. PROPOSED METHOD To solve the VLP location task by means of digital images, a method that performs morphological operations on the horizontal gradient image of the original image is proposed. For calculating the horizontal gradient the Sobel operator [8] is used. Note that vertical edges are detected by using a horizontal gradient operator followed by a binarization operation to detect high values of the gradient [8]. The proposed method can be decomposed into several steps as shown in the diagram in Fig., and they are detailed in the following subsections. A. Horizontal gradient In an image containing the original image edges, it is noticeable that the front and rear of a vehicle are mainly composed of horizontal lines [9], while the VLP has a clear predominance of vertical lines in relation to the horizontal ones.

2 in the VLP region there is a great concentration of pixels with high values (Fig. ). Fig.. Steps of the vehicle license plate location method. The first processing performed on the original image is the application of the Sobel operator for horizontal gradient detection. Fig. shows the horizontal gradient image obtained after applying this operator to the original image in Fig.. In this work, the expected VLP height and width are the average of VLPs heights and widths of the images used for training in the cross-validation process. A mean filter with height and width in pixels, of % of the expected VLP height and width, is performed on the horizontal gradient image [3]. This operation is accomplished aiming to highlight the VLP region more than any other region, because B. Filtering Morphological operations are performed on the image resulting from mean filter application aiming to maintain saliences related to the VLP region and darken the saliences of the regions that are not of interest. Morphological operations are performed in a similar manner to operations in [], []. It differs in the fact that those works have operations primarily on binary images. In order to darken small salient regions, which do not belong to the VLP region, an opening operation by a column SE of size equal to the constant MINHEIGHTCHAR (minimum height in pixels of a VLP character) is used (Fig. ). This opening applied to the mean filtered horizontal gradient image generates a variation between more and less salient vertical saliences in the VLP regiion. The more exalted vertical saliences are then merged. A closing by a line SE of size of the mean distance in pixels between a character and another one is used to restore the gaps in this region. Thus, the VLP region remains with less variation between the pixel values. In this study, the mean distance between characters is the average distance between the characters in the VLP images used as training. A resulting image of an opening operation contains pixel values less than or equal to the corresponding pixel values of the original image. A top-hat operation of an image by a given SE consists of subtracting from it the image obtained by applying an opening by the SE of that image. Thus, by applying a top-hat by a column SE equal to the constant MAXHEIGHTCHAR (maximum height in pixels of a VLP) to the image, we can obfuscate saliences larger than the maximum height of the VLP, as shown in Fig. 3. The morphological operations previously performed on the image may highlight less the separating region of letters and numbers in the VLPs. It happens mainly in Greek VLPs. This may produce, after the the binarization step (Section II-C), an undesired effect. That is the method may treats the VLP as two distinct objects, obtaining only the location of a part of the VLP or failing to locate it. A line SE is used to unite the VLP parts (most salient areas highlighted referring to the letters and numbers) in a closing operation. The SE size is the mean distance between the end of the last letter and the beginning of the first number. In this study, the mean distance between the end of the last letter and the beginning of the first number is obtained by calculating the average distances between the previous and the latter in the VLP of the training images. For ending the morphological operations sequence used to maintain the VLP region and darken the other ones, it is performed an erosion operation followed by a dilation with different SEs. Both SEs are line SEs; the SE size of the erosion operation is twice the width of the mean filter used

3 Fig.. Steps of the vehicle license plate location method: Original image; Horizontal gradient; Mean filter on horizontal gradient; Small saliences darkened. Fig. 3. Steps of the vehicle license plate location method: Big saliences darkened; Erosion and dilation operations; Binarization; Finding the vehicle license plate. in Section II-A and the SE size in dilation is times the mean filter width (Fig. 3). The aim of these last two operations is to remove some saliences which remained at the VLP sides. These saliences when incorporated to the VLP, and remained until the end of the location process, become its height greater than the height required to include all characters. They appear on the horizontal gradient computation when, for instance, the VLP region is on a black part of the vehicle. C. Adjustment to the VLP It is necessary, at this time, the binarization of the image obtained after the filtering step, in order to generate the potential VLP regions. At the end of morphological operations, in some situations, the VLP has low horizontal gradients despite the darkening of the non-vlp salient regions. Thus, in order to maintain the VLP region after the binarization, the threshold for binarization is automattically computed using the Otsu s method [] (Fig. 3). In the connected components analysis, disproportionate objects (it is considered disproportionate objects these in which the height is greater than or equal to the width), small objects, and objects reaching the image margin are removed. Objects with width smaller than the constant MINWIDTHVLP (minimum VLP width) are considered small objects and therefore are disregarded to the next step. Objects reaching the image margin are also disregarded because some saliences regarding the background reach the image margin after binarization. It is important to note that this decision does not require that the VLP is located away to the image margin. Although the VLP can be near to the image margin, after applying the operations of the filtering step (Section II-B), the object referring to the VLP region will be not reaching the image margin. In the resulting image, we find the objects respecting the constraints. This image contains rectangular objects that can be seen as the bounding box of the analyzed objects. The rectangular objects are called candidates, i.e., they are objects regarding regions candidate to be considered as VLP. As a result of previous operation,there may be objects that are joined to each other when they are taken as rectangular objects. If so, these objects are disregarded. That decision exists because the saliences regarding background of an image are often disorganized, producing the union of objects at the moment of having them as rectangles. Similar events do not involve the VLP, since it is basically composed of horizontal gradients (vertical edges) and the vehicle has mainly vertical gradients (horizontal edges). Also, saliences resulting from sticker and brand names of vehicles are not located close enough to the VLP. Then, they could not be joined to the VLP. In the candidates image, it is performed a dilation by a square SE (typically of size 5 5) treating the image when there is intersection between candidates (now, we do not disregard the candidates that intersect). The reason for this dilation is mentioned just ahead. For each candidate, it is selected the corresponding region in the image obtained at the end of Section II-B (image in Fig. 3) to binarize it with the threshold determined by the

4 Otsu s method. It is performed the dilation in the candidate images such that the threshold detected do not be high enough to lose interested sub-regions in the binarization step. That is, the candidate images are dilated so that pixels with lower values are taken into account when determining the threshold for each candidate. After binarization, each candidate (rectangular regions) contains sub-regions. Similar to what is done in Section II-B, an erosion and then a dilation operation (operations are now in binary images) with different SEs are performed in order to remove invalid regions and remove sub-regions that does not belong to VLP. The SE of the erosion operation is a line SE of size of 5% of the expected width of VLP. The SE of the dilation have height and width of % of the expected height and width of VLP, respectively. The height and width of the sub-region is limited by the candidate. At this moment, the upper, bottom and sides limits of each candidate are updated. The i T coordinate referring to the upper limit of the candidate is incremented until some sub-region is found. The i B coordinate regarding to the lower limit is decreased until some sub-region is found. The j L coordinate referring to the left limit is increased and j R coordinate referring to the right limit is decreased, similarly, until some sub-region is found (Fig. 4). After updating the coordinates, the candidate dimensions, i.e. the height and width of the region, are checked again to verify whether they satisfy the minimum VLP height and width or not. If the constraints are not satisfied, the candidate is discarded (unless it is the unique candidate, in this case the upper, lower and sides limits of the candidate are maintained without the update). If all candidates are disregarded, or if there is not any, it is performed a histogram equalization in the original image, so that there is a contrast improvement in the image (and in the VLP region as well), and the whole process is repeated, i.e., starting from obtaining the horizontal gradient. D. Decision among candidates This section describes a way to decide among candidates what is the real VLP. The results presented in Section III are not considering this step. That is, in the experiments, the candidate that best fits the VLP is chosen. We take this decision because two methods in the literature used for comparison ([4], []) do not explain how the decision is made among candidates. To decide among candidates which is the real VLP, it is calculated the mean and standard deviation of the pixels of the image obtained at the end of Section II-B (image in Fig. 3) corresponding to the active pixels of the subregions of each candidate at the end of the adjustment step (previous section). Knowing that the coefficient of variation [] is given by CV = σ µ, a measure attributed to the candidate, V is computed as i.e., V = µ CV, V = µ σ where µ and σ stand for the mean and standard deviation mentioned earlier. Thus, the candidate whose value V is the largest is considered as the VLP (Fig. 3). Once the candidate is chosen, in the binary image, it is performed a dilation by a SE of size 7 7 so that no character belonging to the VLP is cut off. III. RESULTS We conducted comparisons of results with three methods found in literature [], [4], []. The methods were implemented in MATLAB and the source code, including ours, is available in [3]. The runtime of our method is.73 ±.8 seconds for each image. We believe that if optimized and implemented in C/C++ the proposed method can be used in real-time applications. The remainder of this section is organized as follows. First, we describe the methodology and metrics used for comparison. After, the image database used is introduced. And finally, we present and analyze the results. A. Validation methodology To perform an automatic evaluation of methods, all images have been labeled with a minimum bounding box that includes all the characters and with a minimum bounding box that includes the entire VLP (see Fig. 6.) The labeled images are also available in [3]. Results are presented considering the localized area, i.e., Fig. 4. Operations in a candidate regarding the vehicle license plate: Region corresponding to the image obtained at the end of Section II-B; Region binarization; Erosion and dilation; Coordinates update. Fig. 6. Examples of labeling of images: Minimum bounding box that includes all the characters; Minimum bounding box that includes the entire vehicle license plate.

5 Fig. 5. Examples of errors in the location of the vehicle license plate: Location vehicle brand name; Location of brand symbol; Location of extra region; Location of only part of vehicle license plate. area(regionlab char regionmet ) area(regionlab char ) and the excessive area (), i.e., la = ea = area(regionlab vlp regionmet ), area(regionlab vlp ) () () where regionmet is the region found by the method, regionlab char is the characters labeled region (Fig. 6), regionlab vlp is the VLP labeled region (Fig. 6), and area() is a function that obtains the area, in pixels, of a given region. With the subjective analysis of the results, we notice that by locating at least 85% of VLP area, we do not lose the information of the characters, despite being able to confuse the recognition. Also subjectively we infer that by obtaining a rate lower than the % of excessive area, there is no excessive region which can make more difficult the recognition task. In these cases, the location is considered successful. We present the average number of candidates generated by each method, although we did not take into account during validation. In the results reported here, it is considered as the VLP the best candidate that achieves the higher values for ea and la, since two methods in the literature [4], [] do not explain how the decision is made among candidates. B. Image database To validate the proposed method, we used two image databases: 377 images (size 8 6) acquired at the gate of UFOP campus (Brazilian vehicles) using a digital camera and 345 images (93 of 8 6 pixels and 5 of pixels) of Greek vehicles available by [7]. These 7 images are available in [3]. In the first image database, the images were acquired in order to simulate typical conditions of an AVACS. In this image database, the expected size for the VLP is 7.6 ±.44 tall and 88. ±. wide (based on the image of characters labeled) with minimum and maximum height and width of 3, 8, 73, and, respectively. In the Greek image database the expected size for the VLP is 6.84 ± 6.6 tall and 7.43 ±.56 wide with minimum and maximum height and width of 5, 5, 7, and 8, respectively. The proportional values based on the expected VLP size used throughout the proposed method were obtained empirically using 5-fold cross-validation scheme. Each partition of the cross-validation technique contains images of the two databases, i.e., the methods were parameterized and executed based on the two image databases. The values of the MINHEIGHTCHAR, MAXHEIGHTCHAR, and MINWIDTHVLP constants are constraints of the method, i.e., when you set these values free, the possibility of method failure is higher. It is important to note that the images from the database used in this work satisfy the MINHEIGHTCHAR, MAXHEIGHTCHAR, and MINWIDTHVLP constraints and are equal to, 5 and 7, respectively. C. Analysis of results Table I presents results obtained for the proposed method and other methods in the literature [], [4], []. In this table, we present the results obtained from our experiments for the Brazilian and Greek databases separately and joining their images. From the left to right columns, we show the description, the optimum location (la > 85% and ea < %), the excessive location (la > 85% and ea %), the location error (la 85% and ea < %) ), the location (la > ) rates, and the mean number of candidates generated of/by each method. Note that both the values in the table are obtained by a 5-fold cross-validation scheme, and we present their mean and standard deviation values, i.e., µ±σ. Figure 7 summarizes the results obtained by each method for each image in graphs of located area versus excessive area. If we use the simpler location rates (la > ) presented in Table I for comparison purposes, our method obtains the highest rates for all databases used in the experiments, where the rates are slight higher than the others. These rates are quite similar to the ones reported by the authors of [4], [], even the databases are different. However, if we observe the optimum location rate (la > 85% and ea < %), which takes into account a tradeoff between location and excessive area, the values reported by the method presented in [4], [] decrease drastically. From this result, we suggest that the methods described in [4], [] are not able to become part of an AVACS, since the located VLP by these methods does not contain enough information for vehicle identification. Moreover, we can observe that the method proposed in [] produce a number of VLP candidates quite bigger than the other methods, and our method, together with the one in [],

6 TABLE I RESULTS optimum location excessive location location error simpler location candidates number la > 85% and la > 85% and la 85% and Brazilian database ea < % ea % ea < % la > # Suryanarayana et al. method ([]) 93.% ±.8%.% ±.% 6.37% ±.5% 94.3% ±.45%.4 ±.7 Vargas et al. method ([4]) 34.74% ±.3% 3.79% ±.43% 39.53% ± 3.6% 54.64% ±.97% 4.43 ± 4.7 Wang et al. method ([]) 54.63% ±.78% 7.5% ±.4% 3.6% ±.38% 98.4% ±.4% 5. ±. Proposed method 94.43% ± % 9% ± 5%.% ± 4% 96.% ±.64%.57 ±.75 la > 85% and la > 85% and la 85% and Greek database ea < % ea % ea < % la > candidates number Suryanarayana et al. method ([]) 93.85% ±.8%.87% ±.9% 5.9% ±.88% 98.55% ±.34% 3.4 ±.48 Vargas et al. method ([4]) 64.64% ±.7% 5.% ±.8% 7.% ±.9% 86.95% ±.4% 6.4 ± 4.96 Wang et al. method ([]) 6.3% ± 9%.3% ±.45% 39% ±.66% 95.7% ±.3% 9.4 ± 7.43 Proposed method 96.5% ±.8%.87% ±.%.6% ± % 99.3% ±.9% 3 ±. la > 85% and la > 85% and la 85% and The two database ea < % ea % ea < % la > candidates number Suryanarayana et al. method ([]) 93.46% ±.95%.4% ±.9% 5.85% ±.94% 96.9% ±.89%.6 ±. Vargas et al. method ([4]) 49.3% ±.8% 9.7% ±.43% 33.59% ±.44% 7.8% ±.46% 5.38 ± 4.6 Wang et al. method ([]) 58.3% ±.9% % ±.69% 7.4% ± 8% 96.68% ±.%.44 ± 9.33 Proposed method 95.43% ±.%.5% ±.33%.35% ±.34% 97.5% ±.4%.7 ±.6 produces the smallest number of VLP candidates. As a final result, our method obtains successful VLP location rates of 9.8% ±.65% considering the decision algorithm (Section II-D) among candidates and 95.43% ±.% (la > 85% and ea < %) without this decision. Our method is produced as an extension of the method proposed in [] together with some insights of other methods, e.g., []. Due to that, the rates presented by our method are similar, nonetheless higher than, the ones of []. In the next two paragraphs, we present some statements about the low optimum location rates obtained for the methods presented in [4], []. And finally, some observations about the proposed method is made. The main cause of errors in the method proposed by [4] is in the initial stage where are performed vertical and horizontal projections of a preprocessed image. The image database used contains images with great variation in the background. This fact hinders the choice of robust constants. The method proposed in [] is not robust to some changes in scale, that is, it is scale variant. In the step 6 of that method, a rectangle is built under the supposition that its width is greater than or equal to the maximum width of the plate (more specifically to the characters). But if the plate has a width less than the maximum width of the plate, in the step 7, it is difficult to make the adjustment of the detected regions. If the method is parameterized so that in the step 6, the rectangle that represents a candidate has a width less than the width of the plate in question. Then there is a failure in the method because it has no lateral growth in this step. As stated earlier, in Section III-B, the database contains images where the VLP width varies from 7 to 8. The main cause of the errors of the proposed method is in the stage of decision among candidates. There are regions in the vehicle (e.g., bus identification number) that may have greater horizontal gradient and size close to the VLP size. These regions, in the step of deciding among candidates, can be selected as the VLP region. Fig. 5 gives some examples of failures in the VLP location. IV. CONCLUSIONS AND FUTURE WORKS In this paper, we proposed a new VLP location method based on the horizontal gradient, morphological operations, connected component analysis, and statistical measures. We obtained in average a successful VLP location rate over than 95%. The method is scale invariant at some extend. That is, if the VLP dimensions respect minimum and maximum constants, the method can accurately located the VLP. Another contribution of this work is the methodology introduced for evaluate the VLPs located. This methodology allow us to perform a quantitative and qualitative comparison among methods in the literature. Studying and implementing methods for vehicle tracking in videos and VLP character recognition are proposal for future work, because, besides being fundamental parts of an AVACS, they are complementary to this work. With the first method, aiming to extract the frame which best represents the vehicle and its VLP, we can obtain the width, in pixels, of the vehicle to be analyzed, and so estimated the constants based on this width. That is, the proposed VLP location method here becomes robust to the scale changes. With the second method, we will be able to better decide among candidate regions, since we can recognize the characters of each candidate and analyze if they are a VLP. Also we can find more than one VLP in an image, and this capability is required in some AVACS. ACKNOWLEDGMENT The authors would like to thank the PIP/UFOP (Program for Initiation in Research of UFOP) for the scholarship granted for this work. REFERENCES [] F. Martín, M. García, and J. L. Alba, New methods for automatic reading of VLP s (Vehicle License Plates), in Proc. of IASTED Int. Conf. on Signal Processing, Pattern Recognition, and Applications (SPPRA), Jun., pp

7 3 P. V. Suryanarayana et al. 3 M. Vargas et al Y. R. Wang et al. 3 Proposed method Fig. 7. Results (located area excessive area): Method []; Method [4]; Method []; Proposed method. [] P. V. Suryanarayana, S. K. Mitra, A. Banerjee, and A. K. Roy, A morphology based approach for car license plate extraction, in Proc. of the IEEE INDICON, Dec. 5, pp [3] B. Hongliang and L. Changping, A hybrid license plate extraction method based on edge statistics and morphology, in Proc. of the Int. Conf. on Pattern Recognition (ICPR), vol., Aug. 4, pp [4] M. Vargas, S. L. Toral, F. Barrero, and F. Cortés, A license plate extraction algorithm based on edge statistics and region growing, in Proc. of the Int. Conf. on Image Analysis and Processing, vol. 576, Aug. 9, pp [5] V. Abolghasemi and A. Ahmadyfard, An edge-based color-aided method for license plate detection, Image and Vision Computing, vol. 7, no. 8, pp. 34 4, Jul. 9. [6] S.-L. Chang, L.-S. Chen, Y.-C. Chung, and S.-W. Chen, Automatic license plate recognition, IEEE Trans. Intell. Transp. Syst. (ITS), vol. 5, no., pp. 4 53, Mar. 4. [7] C.-N. E. Anagnostopoulos, I. E. Anagnostopoulos, I. D. Psoroulas, V. Loumos, and E. Kayafas, License plate recognition from still images and video sequences: A survey, IEEE Trans. Intell. Transp. Syst. (ITS), vol. 9, no. 3, pp , Sep. 8. [8] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 3rd ed. Prentice Hall, 7. [9] M. Sarfraz, M. J. Ahmed, and S. A. Ghazi, Saudi arabian license plate recognition system, in Proc. of the Int. Conf. on Geometric Modeling and Graphics (GMAG), Jul. 3, pp [] N. Otsu, A threshold selection method from gray-level histograms, IEEE Trans. Syst., Man, Cybern. (SMC), vol. SMC-9, pp. 6 66, Jan [] Y.-R. Wang, W.-H. Lin, and S.-J. Horng, A sliding window technique for efficient license plate localization based on discrete wavelet transform, Expert Systems with Applications, vol. 38, no. 4, pp , Apr.. [] D. C. Montgomery and G. C. Runger, Applied Statistics and Probability for Engineers, 3rd ed. John Wiley & Sons, Inc., 3. [3] P. R. M. Júnior, D. Menotti, and J. M. R. Neves, Vehicle license plate location (VLPL) algorithms. [Online]. Available: