Deep Automatic Licence Plate Recognition system

Transcription

1 Deep Automatic Licence Plate Recognition system Vishal Jain Soma Biswas ernet.in Zitha Sasindran Harish S Bharadwaj harishsb9490@gmail.com Anoop Rajagopal Myntra Designs anoop.kr@myntra.com K R Ramakrishnan krr@ee.iisc.ernet.in ABSTRACT Automatic License Plate Recognition (ALPR) has important applications in traffic surveillance. It is a challenging problem especially in countries like in India where the license plates have varying sizes, number of lines, fonts etc. The difficulty is all the more accentuated in traffic videos as the cameras are placed high and most plates appear skewed. This work aims to address ALPR using Deep CNN methods for real-time traffic videos. We first extract license plate candidates from each frame using edge information and geometrical properties, ensuring high recall. These proposals are fed to a CNN classifier for License Plate detection obtaining high precision. We then use a CNN classifier trained for individual characters along with a spatial transformer network (STN) for character recognition. Our system is evaluated on several traffic videos with vehicles having different license plate formats in terms of tilt, distances, colors, illumination, character size, thickness etc. Results demonstrate robustness to such variations and impressive performance in both the localization and recognition. We also make available the dataset for further research on this topic. and identifying the LP area in whole traffic image can be quite challenging due to many intrinsic factors such as diversity in character pattern, font, shape, size, color, plate dimensions as in Figure 1. Also, capturing viewpoint, blurring and uneven illumination causes character distortion. External elements like complex background, text signs & boards, color similarity between background and vehicle body, reflection from the surrounding objects further enhance complexity of the problem. Sample frames are shown in Figure 2 Figure 1: Left: Single line plates. line plates Right : Double CCS Concepts Computing methodologies Computer vision; Object detection; Object recognition; Neural networks; Keywords Convolution Neural Network; Spatial Transformer Network; Image Transformation Pursuit; Automatic License Plate Recognition 1. INTRODUCTION Vehicle License plate (LP) detection and recognition is an important applied area in traffic surveillance, management, parking management, vehicle recognition, highway toll collection, tracking vehicle for security purposes. Retrieving Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ICVGIP, December 18-22, 2016, Guwahati, India c 2016 ACM. ISBN /16/12... $15.00 DOI: Figure 2: video. Sample frames from CCTV recorded The system as in Figure 3 is typically developed in stages with first being detection of LP in the traffic image. Detection includes localization of LP and generating suitable bounding box over them. Next stage is recognition or identification of characters inside the bounding box generated. This sequential process emphasizes each stage to be accurate and robust for the next stage to be successful. For the detection part, feature based approaches is mostly used [10]. Features of the LP derived from its edge, color, texture, or a combination of two or more of these are used

2 The rest of paper is organized as follows. Related work is discussed in Section 2. The detection and recognition modules of our framework are described in Section 3. Experiments performed on test videos and results obtained are summarized in Section 4. Finally, conclusions are drawn and some comments in general are made in Section 5. Figure 3: Typical stages of an ALPR system. to identify the potential regions. The proposals generated have many false positives. These false positives are generally eliminated using geometrical properties of the LP [6] viz. area, aspect ratio, dimensions of plate etc. LP detection is followed by recognition of the characters involving segmentation and extraction of individual characters from the LP. The segmentation methods may use pixel connectivity, projections of characters, character contours or their combination. Character recognition is performed on the segmented characters with template matching and learning based methods. Each process involves pre-processing to enhance performance of that module. In this work, we adopted LP edge feature based scheme to generate candidate proposals for the LP regions. The false positives are minimized using a convolution neural network which perform binary classification on the candidate plate regions. This integration of calculative and learning approach ensures high recall and precision. As deep networks requires large training data we build a new dataset with Indian LP which have a lot of diversity in plate form factor, fonts, number of lines etc. For recognition of the extracted LP we use another CNN built for characters recognition. This CNN model has 11 parallel classifiers at the output to localize and encode individual character of the character sequence of LP. As the extracted LP usually have variations in form of tilt, non-uniform brightness a spatial transformer layer (ST layer) [8] is used at the input layer of the CNN. This way it enhances the geometric invariance of CNN in a flexible, data-driven manner without much training overhead. Most of the ALPR frameworks work well only in controlled environment with lots of assumptions. Some demand specific viewpoint for image capturing, some require vehicle to be parked or static and some may work for license plates of a specific region. Challenge lies in developing a system that works well in natural real-traffic images. Our major contributions are dataset creation for license plates with different formats, at different distances, in diverse background and character style. a CNN framework for end-to-end LP detection and recognition from video frames without segmentation in uncontrolled conditions. The system works on realtraffic videos and rapidly processes large frame area of High-Definition images where amount of computation and delay is high, optimizing different implementation tools employed. 2. RELATED WORK In this section, we present a brief description of related work on LP detection and recognition. 2.1 License plate detection This stage analyzes the input image and proposes LP regions. The detection of license plates accurately in open environment is challenging as the view-point, illumination, direction of vehicle can be arbitrary. Existing approaches are broadly classified based on features like edge, texture, color character information they have used [10]. The edge-based approaches [3] finds regions with higher vertical edge density. Morphological operations are performed on the edge map to remove smaller and bigger regions of image. Binarization followed by connected component analysis gives candidate license plates filtered for false positives with geometric filtering. Edge based approach is simple and fast but produces lots of false positives. Texture-based approaches [4] detect the desired region according to its pixel intensity distribution as intensity pattern of the LP region differs from the background. The methods examine local features of an image by calculating irregularity of local plate region in the texture of the image at different scales. It works well in case of deformed plate boundaries and are robust to color, size and position of plate region. But high computational complexity for HD complex images limits its use for real-time vehicle-traffic images. Color-based approaches [14, 10, 18] make use of the fact that color of the LP is different from the other regions of the image. In [14], the image is segmented into different regions with different colors using mean-shift algorithm. From these candidates, LP regions are distinguished using dimension, aspect ratio and vertical edge information. The method can detect deformed or tilted license plates as well with the downsides being sensitivity to illumination changes and noise sensitivity. Character-based approaches [10] consider LP as sequence of alphanumeric characters, and search for the character sequences in the image. The method is robust to variations of LP characters font, color, size, tilt and language-script. But computation time is high and detection errors occur in case of background texts. In our work we use edge-based features, discussed above, as it performs fairly well in detecting almost all LP candidates with less computation time. Later a CNN based binary classification is performed for filtering out false-positives. 2.2 License plate recognition The detected LP regions are then recognized for their content. The general approach has mostly been segmenting the character-sequence of LP into individual characters using pixel connectivity, projections, character contours and template matching. This is followed by recognition of the segmented characters using suitable Optical Character Recognition (OCR) technique. Pixel connectivity based methods label each pixel of the

3 binarized LP image based on 4 or 8 neighborhood connectivity. If characters are separated from each other without joins, they can be separated based on connectivity with neighboring pixels. The method is simple, fast and works for rotated plates but, the plain-vanilla usage results in false positives, joined or broken characters, plate edges, arbitrary spots. This issue is addressed in [16] based on some geometrical rules. In projection-based method [11], binarized LP images is projected horizontally to get top and bottom boundaries of the characters and vertically to get left and right endpoints of each character. This method works for different character positions and number of characters but performs poorly in case of rotated images. The contour-based method [17] determines character boundary using statistical boundary shape models and uses character template for segmentation and recognition. Exact boundaries can be extracted from this method but noisy images result in incomplete and distorted contours. However, the segmentation is affected under tilt, broken characters and non uniform brightness. The segmentation also fails to extract all the characters when there are joined or broken. Template matching based methods [12] measure similarity between character and template and chooses class of the best matching template as the predicted class. The method is fast and simple but limited by variations of font, rotation, size, etc. of characters. Learning based method [5] are robust to variations of segmented characters. Features are extracted from character images, such as edge density, gradient, Local Binary Patterns, etc. which are classified discriminatively with learning models like SVM, neural networks, etc. Plate/non-plate CNN classifier previously used in [19, 6] reject false positives and distinguish LP from background objects including image text, giving high precision. However, these approaches adopt a sliding window approach for locating LP making it computationally expensive. In view of the issues with past approaches, as discussed above, we adopt CNN based recognition on whole LP image, avoiding challenging task of image segmentation into each character. 3. DEEP LPR SYSTEM Our ALPR system shown in Figure 4 is a cascade framework constituting three parts: (i) Generation of LP candidates, (ii) Filtering of false positives with binary license plate/non-license plate classifier, (iii) Recognition of characters of extracted LP, with 37 class CNN with 11 parallel classifier at output. 3.1 LP Detection License plate candidates generation We adopt LP edge feature based scheme for candidate proposals. The algorithm detects multiple LPs under different image capture conditions and extracts them using edge statistics. Morphological operations are used to extract vertical edges of the LP regions while removing background. The process is divided into two sub-parts, candidate generation for possible LP regions and candidate verification to eliminate false positives based on geometric parameters. Below we outline the LP candidate generations steps. Original color image of traffic-scene is converted to gray image. Black and white intensity values in LP Figure 4: Illustration of our ALPR framework regions is filtered out with median filtering as it eliminates and minimizes unwanted noise and spots in the image. Histogram equalization is performed to deal with the illumination variations. Sobel vertical edge filter is applied to obtain vertical edge map as LP regions are seen to have higher vertical edge density. Mean filter is used to emphasize LP region with the mask size of order of LP size eliminating edges of car mirror, windows, LP border and other surrounding objects. It also smooths out the edge intensity maps removing high-frequency components pertaining to noise. Morphological filtering is performed to remove smaller and larger objects than the LP dimensions with opening and Mexican top-hat filtering. Structuring Elements (SE) have size of MINHCHAR (minimum character height), and MAXHCHAR (maximum character height) respectively. We convert the resulting image to binary image by Otsu thresholding so that candidate regions are separated from complex background. Candidate verification is done based on criteria that candidate regions should follow: a) width of the region to be greater than height for a LP b) height has to be greater than minimum LP height MINHCHAR c) region should not touch image boundary d) two regions should not overlap e) aspect ratio should be between parameters LP MIN RATIO and LP MAX RATIO. Parameters along with size of structuring elements are selected after testing on multiple images from the desired location of application. The above steps are visualized in Figure 5 CNN detection Candidate generation technique described above detects all LP regions along with many non-license plate regions that are to be filtered out. CNN classifiers show discriminative ability among different classes. We adopt binary license plate/non-license plate CNN for filtering. We use 5-layer CNN model for classifying candidate regions generated from previous stage into license plate or non-license plate regions. Configuration of the model is shown in Table 1. All images are resized to 32 96, range normalized to [0, 1] for passing to input layer of CNN. We experiment with different number of layers in CNN model and obtain highest validation-set ( 20% of training set) accuracy for 5-layer model. Accuracy numbers for different layer configurations are shown in Table 2.

4 Figure 5: Candidate generation steps for LP detection Table 1: Configuration of CNN model for LP detection Layer Type Parameters Softmax 2 classes Fully connected #neurons: 2 Dropout Ratio : 0.5 Fully connected #neurons: 128 Convolution #filters: 48, kernel : 3 3, stride : 1 Convolution #filters: 64, kernel : 3 3, stride : 1 Convolution #filters: 64, kernel : 5 5, stride : 1 Input pixels gray-scale image Table 2: LP detection accuracy with different number of CNN layers. CNN Model Validation-set Accuracy 4-layers 94.8% 5-layers 98.7% 6-layers 97.2% 3.2 LP Recognition Traditional approach for LP character recognition has been to segment characters and recognize each separately. To overcome these difficulty in character segmentation and recognition, we perform string recognition on whole LP image. This method avoids the challenging task of character segmentation. We use character-sequence encoding CNN model [8] for alphanumeric character recognition in LP images. We assume maximum character length to be a fixed number N. Hence It has N parallel classifiers so that one classifier would classify one of N characters of LP. A NULL (#) character class is used to account for lesser number of characters than the maximum length. We also include Spatial Transformer (ST) module introduced by [8] which can make CNN spatially invariant to large variations of input data. We include ST module as first layer in recognition CNN. We use character-sequence encoding CNN with 16 layers. We consider maximum length of character sequence to be 11. So there are 11 different classifiers at output of the CNN to predict each digit. As described earlier, we assume NULL character in place of non-existing places of LP image. Total 37 character-classes are 0,..., 9, a,..., z, NULL. Configuration of CNN layer is described in Table 3. Configuration of the localization network of ST layer is shown in Table 4. Table 3: Configuration of CNN model for recognition with ST Layer Layer Type Parameters Softmax 11 parallel layers with 37 classes Fully connected 11 parallel layers, each has #neurons: 37 Fully connected #neurons: 1024 Convolution #filters: 256, kernel : 3 3, stride : 1 Convolution #filters: 128, kernel : 3 3, stride : 1 Convolution #filters: 64, kernel : 5 5, stride : 1 ST Layer Input pixels gray-scale image 4. EXPERIMENTS We contribute a new dataset for training CNN for ALPR. We use video recordings from traffic monitoring 3 megapixel HD camera. It captures moving traffic footage with 20 fps, resolution of from a distance of 7-8 meters and height of 4 meters. Few sample frames are shown in Figure 2.

5 Table 4: Configuration of ST layer s localization network. Layer Type Parameters Fully connected #neurons: 6 Fully connected #neurons: 50 Convolution #filters: 32, kernel : 5 5, stride : 1 Convolution #filters: 48, kernel : 5 5, stride : 1 Input pixels gray-scale image and for filtering false positives with CNN is 30ms per frame. Thus total time taken for LP extraction per frame is 180ms. So our LP detection algorithm is fast and accurate. Confusion matrix on the test set is shown in Table 5. Table 5: Confusion matrix of CNN classifer for LP detection Confusion matrix License plate Non-license plate License plate 99.36% 0.64% Non-license plate 0.32% 99.68% Some LP extraction results are shown in fig LP detection The localization of LP regions described above, involve several parameters used in morphological operations and candidate verification. These are arrived at, based on the plate dimensions in the frame images, and further tuned by testing on more images. On an average, 2.45 non-lp regions or false positives are generated per frame-image which are filtered out with CNN. We use python and OpenCV library for image processing tasks. For training CNN, data is generated using the LP candidate proposals from the video frames. Our dataset has 8570 negative samples and 5981 positive examples, i.e. total of 14, 551 images. Sample positive and negative training examples are shown in fig 6 Figure 7: Detected license plates. 4.2 LP recognition Figure 6: Top: Positive training samples. Below: Negative training examples. We experiment with different combinations of CNN parameters like initial learning rate, step size, batch-size. We train with a batch size of 32 using Digits [2, 15] on NVIDIA GeForce GTX 660 GPU (2GB memory and 960 CUDA cores), Intel i7 8core CPU machine. We do batch-processing for reducing per-frame-processing time. Also, single-line and double-line type plates have different character arrangements, so a single model cannot learn the character arrangement for both types of plates. Thus, we build separate models for single-line and double-line plates. We perform a 80:20 split of our dataset for training and testing. We obtain a test set accuracy of 99.02%. Positive sample accuracy is 98.46% and negative sample accuracy is 99.3% perhaps because number of negative samples are more in training set. Thus CNN based false-positive rejection gives very high test accuracy ensuring high precision. Average time for candidate generation is 150ms per frame Using the detection algorithm, which is fast and accurate, we generate crops of License Plates from frames. We use around 7, 400 of these crop images for the training set out of which 4, 400 are single-line type and 3000 are double-line type. Test set has total 2, 080 images; 1, 800 of single-line type and 280 of double-line type. We manually annotate the ground truth labels for both train and test images. As we do not have sufficient training samples for other states license plates for training the CNN, we synthetically generate LP images as in [7] for balanced representation. Figure 8 represents the semantic division of the Indian LPs. By permuting over possible content-options for each semantic part helps in generating many synthetic plates. We thus have 28, 280 single-line type synthetic images and 17, 580 doubleline type synthetic images. Some synthetic LP images are shown in Figure 9. Data augmentation of the training set is performed using [9] for obtaining better variability. We use 19 different transformations of scaling, rotation, homography and JPEG compression. These are illustrated for a sample LP image in Figure 10. All images are in grayscale and resized to pixels for training. The data is normalized by subtracting the mean over all training data and range-normalized to [0, 1]. For double line LP we bisect them horizontally. We then horizontally stack the lower part with the upper part, to make double-line images similar in appearance and dimensions to single-line images. Figure 11 illustrates this pre-processing step. We implement the model using theano[13] library in python. Average per image processing time is around 15ms-18ms for

6 Table 7: Character-level accuracy for LP. Model type Number of Correctly Accuracy characters (A) classified (B) (Bx100/A) Single-line % Double-line % Figure 8: License plate semantic parts. Figure 12: ST layer output at different epochs. Figure 9: Synthetic license plates. Figure 10: Transformations used for the license plate images. Figure 11: Bisection and joining of double-line plates. recognition. Table 6 shows accuracy on test set for both single-line and double-line models. Character-level accuracy for single-line type plates are shown in Table 7. Figure 12 shows ST layer output at different epochs. It can be noticed that ST layer applies affine transformation on different LP images to get normalized orientation. As the number of training epoch increases, spatial variation of the training samples reduces underlying the prominence of the ST layer. Table 6: Accuracy for LP recognition. Model type Total Images Fully correct 1 character error 2 character error >2 character error Single-line (57%) 430 (24%) 146 (8 %) 206 (11%) Double-line (79%) 18 (7%) 9 (3%) 31(11%) 4.3 Framework Evaluation We integrate LP detection and recognition to get a unified cascade framework for ALPR as shown in Figure 4. There is no uniform way to evaluate performance of ALPR system [6]. In this work, we use recall and precision rate to quantify LP detection results. Recall rate is the number of true positives detected out of the total number of true positives. High recall rate implies robustness of detection algorithm to various occurrences of the object to be detected. Precision rate is the number of true positives detected out of the total number of detected objects. High precision rate implies discriminative ability of algorithm between true positives and negatives. For evaluating LP recognition technique, we calculate platelevel accuracy and character-level accuracy. In plate-level accuracy, we calculate percentage plates with all characters correctly classified, percentage plates with one character wrongly classified, percentage plates with two characters wrongly classified and percentage plates with more than two characters wrongly classified. In character-level accuracy, we calculate percentage of all the characters, of all the license plates, correctly recognized. We test the ALPR system on 4 traffic-video clips that have 173 cars; 149 with single-line type LP and 24 with doubleline type. Total time taken by the system for processing one frame is around 200ms on average, i.e. system s speed is 5 fps. Table 8 shows LP detection module s recall and precision rate. The LP recognition module s plate-level accuracy results are summarized in Table 9 and character-level accuracy results are summarized in Table 10. Plate type Table 8: LP detection results. Number of plates Plates Detected (B) False positives (C) Recall (Bx100/A) Precision (Bx100)/ (B+C) Single-line % 96.2% Double-line % 92% Figure 13 shows samples where our LP detections fails. It can be seen that the characters which are highly blurred are not correctly recognized. Figure 15 shows some qualitative

7 Table 9: LP - Plate accuracy. Plate-type Detected Fully 1 char. 2 char. >2 char. plates correct error error error Single-line (80.2%) 18 (14.3%) 5 (4.0%) 2 (1.5%) Double-line (65.2%) 6 (26.1%) 2 (8.7%) 0 Table 10: LP recognition - Character accuracy. Plate-type Number of Correctly Accuracy characters (A) classified (B) (Bx100/A) Single-line % Double-line % results of our ALPR on traffic-scene images. These images demonstrate the accuracy and robustness of our ALPR system under diverse conditions of vehicle direction, LP color, font, format, size, character-arrangement, etc. We also qualitatively compare our results with the publicly available OpenALPR [1] library in Figure 16. We see that our framework fares better as we do character recognition holistically whereas OpenALPR involves character segmentation followed by OCR. 5. CONCLUSION In this work, we have presented an automatic LP recognition system based on CNN. We integrated several concepts and techniques for designing the system. Results have shown that the system successfully extracts the multiple LP region from the traffic image and correctly recognizes most of the characters under variety of conditions. We have also shown that incorporating spatial transformer layer provides better representation of data for recognition. The processing time per frame is around 200ms, suitable for real-time applications. We also compare the superiority of our framework over publicly available OpenALPR system. Current framework cannot handle LP of two wheelers and generating a dataset for two-wheelers is left for future. Further, we also want to incorporate video based information like optical flow and temporal information in our work for generating better and fewer candidate regions for LP proposals. References [1] Automatic license plate recognition library. github.com/openalpr/openalpr, [2] Nvidia digits. github.com/nvidia/digits, Figure 13: Sample LPs where our method failed 4.4 Performance for other locations Though we have designed our ALPR system for one cameralocation, taking into consideration the properties of image captured like viewpoint, dimensions, LP measurements, vehicledirection, etc., we test it for two other locations as well. Some sample outputs are shown in Figure 14. For these locations, we miss some license plates while detection and also get few characters wrongly classified. However the system is still able to detect many plates and recognize the characters acceptably in spite of motion blur, occlusion and tilt indicating the CNN framework has good generalization ability. [3] R. Chen and Y. Luo. An improved license plate location method based on edge detection. Physics Procedia, 24: , [4] X. He H. Zhang, W. Jia and Q. Wu. Learning-based license plate detection using global and local features. In Internation Conference on Pattern Recognition, volume 2, pages , [5] Q. Ye J. Jiao and Q. Huang. A configurable method for multi-style license plate recognition. Pattern Recognition, 42(3): , [6] H. Li and C. Shen. Reading car license plates using deep convolutional neural networks and lstms. arxiv preprint arxiv: , [7] A. Vedaldi M. Jaderberg, K. Simonyan and A. Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. arxiv preprint arxiv: , [8] K. Simonyan M. Jaderberg and A. Zisserman. Spatial transformer networks. In Advances in Neural Information Processing Systems, pages , [9] M. Paulin, Z. Harchaoui F. Perronnin C. Revaud, Jérôme, and Schmid. Transformation pursuit for image classification. In IEEE Conference on Computer Vision and Pattern Recognition, pages , [10] M. Shehata S. Du, M. Ibrahim and W. Badawy. Automatic license plate recognition (alpr): A state-of-the-art review. IEEE Transactions on Circuits Systems Video Technology, 23(2): , Figure 14: Sampe results for different viewpoint [11] O. Katai H. Kawakami S. Nomura, K. Yamanaka and T. Shiose. A novel adaptive morphological approach for degraded character image segmentation. Pattern Recognition, 38(11): , 2005.

8 Figure 15: Results of our ALPR framework. Figure 16: Comparison of our framework with OpenALPR system [12] O. Ishaq S. Rasheed, A. Naeem. Automated number plate recognition using hough lines and template matching. In Proceedings of the World Congress on Engineering. and Computer Science, volume 1, pages 24 26, [13] Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arxiv e-prints, abs/ , May tomatic license plate recognition system. In IEEE Conference on Systems, Man, and Cybernetics, pages , [17] L. Bai Y. Zhang, Z. Zha. A license plate character segmentation method based on character contour and template matching. In Applied Mechanics and Materials, volume 333, pages , [14] H. Zhang W. Jia and H.Huaifeng. Region-based license plate detection. Journal of Network and Computer Applications, 30(4): , [18] H. Guo-Jiang Z. Wei-gang and J. Xing. A study of locating vehicle license plate based on color feature and mathematical morphology. In International Conference on Signal Processing, volume 1, pages , [15] J. Donahue Y. Jia, E. Shelhamer et al. Caffe: Convolutional architecture for fast feature embedding. arxiv preprint arxiv: , [19] S. Yang Z. Zhao and X. Ma. Chinese license plate recognition using a convolutional neural network. In PACIIA, volume 1, pages IEEE, [16] H. Yoon Hosub Y. Yoon, KD Ban and J. Kim. Blob extraction based character segmentation method for au-