M-SIFT: A new method for Vehicle Logo Recognition

01 IEEE International Conference on Vehicular Electronics an Safety July 4-7, 01. Istanbul, Turkey M-SIFT: A new metho for Vehicle Logo Recognition Apostolos Psyllos, Christos-Nikolaos Anagnostopoulos, Eleftherios Kayafas Abstract - In this paper, a new algorithm for Vehicle Logo Recognition is propose, on the basis of an enhance Scale Invariant Feature Transform (Merge-SIFT or M-SIFT). The algorithm is assesse on a set of 1500 logo images that belong to 10 istinctive vehicle manufacturers. A series of experiments are conucte, splitting the 1500 images to a training set (atabase) an to a testing set (uery). It is shown that the M- SIFT approach, which is propose in this paper, boosts the recognition accuracy compare to the stanar SIFT metho. The reporte results inicate an average of 94.6% true recognition rate in vehicle logos, while the processing time remains low (~0.8sec). M I. INTRODUCTION any vision-base Intelligent Transport Systems for etecting, tracking or recognizing vehicles in image seuences are referre in the literature. Vehicle type classification is a task that has been aeuately aresse in the literature [1-6]. However, Vehicle Manufacturer Recognition (VMR) is a crucial subject, with relatively limite research reporte in the fiel. This is ue to the fact that manufacturer recognition is an inherently har problem ue to the wie variety in the appearance of vehicles. This task becomes even more ifficult since image acuisition is performe in outoor environment where illumination is ambient. Scale Invariant Feature Transforms (SIFT) were introuce by Lowe [7-9] an they are invariant to rotation, translation an scale variation between images an partially invariant to affine istortion, illumination variance an noise. Research relate to fully invariant features has been publishe by Brown an Lowe [10], Mikolajczyk an Schmit [11]. Image matching is a funamental problem in computer vision which occurs in many computer vision applications regaring a variety of fiels, incluing image retrieval for security enforcement an robot navigation. A common approach to accurate image matching is known as the invariant keypoint or the extraction of the point of interest from the images compare. It involves ientifying Manuscript receive June 1, 01. A. Psyllos is with NTUA, Sc. of Applie Mathematical an Physical Science, Athens, Greece (email: psyllos@central.ntua.gr). C. N. Anagnostopoulos is with University of the Aegean, Mytilene, Greece. (Corresponing author: +305103664; fax: +3051036637; e- mail: canag@ct.aegean.gr E. Kayafas is with NTUA, Sc. of Electrical & Computer Engineering, Athens, Greece (email: kayafas@cs.ntua.gr). the points that can be reliably extracte from ifferent images of the same object or the same category of objects. Earlier research into invariant keypoints inclues the Harris corner etector [1] an the keypoints invariant to rotation an translation [13], [14]. Dlangenkov an Belongie [15] utilize Scale Invariant Feature Transform (SIFT) features, using a vehicle atabase of rear-view vehicle images. Čonos [16] eals with a vehicle moel recognition problem of frontal view images, proposing a SIFT-base escriptor for feature extraction. The rawback of the latter approach is the extensive computational cost reporte in the respective paper. Petrovic an Cootes [17] presente an interesting approach base on ege graient an match refinement algorithm for vehicle moel recognition an verification. A comparative knowlege acuisition system, consisting of several object recognition moules for the rear view of vehicles appears in Maemoto et al. [18]. The VMR problem is resolve through vehicle logo recognition (VLR) an the recognition task reuires the successful extraction of the small logo area from the original vehicle image. Usually, the process involves a license plate location moule, followe by coarse to fine methos to ientify the logo area using symmetry an/or ege statistics in the image. Then logo recognition is either performe through neural networks [19-1] or template matching. Wang et al. [3] presente a metho for logo recognition using template matching an ege orientation histograms with goo results. In this paper, a SIFT-base Vehicle Logo Recognition schema is escribe, whose aim is to obtain reliable Vehicle Make Recognition from the logo image. An enhance SIFT matching moule etects an extracts the points of interest (keypoints) in the uery logo image, escribes them an matches them with keypoints store in a logo image atabase. The whole process is fine-tune by clustering the matche keypoints, using Generalise Hough Transform [4], an geometric verification by means of an affine transformation. The propose metho is assesse on a atabase containing 1500 logo images, which were processe using our previous work escribe in [1], [], [5] using the Meialab Vehicle Database [6]. II. VEHICLE LOGO RECOGNITION SIFT is the state-of-the-art in the fiel of image recognition an is use in a wie range of image retrieval 978-1-4673-0993-6/1/$31.00 01 IEEE 61

applications. It is base on the iea of representing images by a set of invariant keypoint escriptors using graient orientation histograms. The invariant features (points of interest), or just keypoints, are istinguishe as local peaks in the image scale-space representation an filtere to preserve only those that are likely to remain stable over transformations. For each invariant feature a SIFT keypoint escriptor is calculate having the following properties: a) smooth changes in location, orientation an scale o not cause raical changes in the feature vector, b) it is fairly compact, expressing an orientation histogram from a patch of pixels in the neighborhoo of the invariant feature, using a 18 imensional vector, c) it is resilient to affine eformations, such as those cause by weak perspective effects (for small angles of eviation), an thus it coul be efficient for logo recognition in non-controlle conitions. In this paper, in the initial stage of computation, scale an orientation invariant features are etecte an extracte from the uery image, exploring the scale-space structure of the image as escribe in [7] an [7]. In the next sections, our contribution in eveloping an enhance Scale Invariant Feature Transform (Merge-SIFT or M-SIFT) is escribe, pointing out the feature matching an hypothesis valiation proceures in the caniate points. A. Feature Matching For each feature i in the uery image, the escriptor is use to fin its Nearest-Neighbor (NN) matches among all the features store from all the images in a atabase calle NN atabase (see Figure 1). The nearest neighbors are ientifie minimizing an L Eucliean metric: N Qi D i(j) = (k k ) (1) k= 1 where Q i (i.e. 1,..., N ) is the i th escriptor for the uery image, D i (j) (i.e D 1,D...,D N ) is the i th escriptor for image j an N=18. The total number of nearest neighbors is calculate from Euation (1), where carinality is the number of elements in a set an γ is an appropriate threshol value. NN i= carinality( arg Qi D i(j) < ) () Therefore, the number of NNs in the atabase for each feature (keypoint) epens on the threshol selection. In Figure, the parameter τ is plotte versus threshol (γ), where τ is escribe by the euation (3): τ = NNi KP KP where NN i is the number of nearest neighbors foun from Euation () for each feature i in the uery image, KP is the number of keypoints in the uery image an KP b is the total number of keypoints in NN atabase. b (3) B. Hypothesis Valiation 1) Feature Clustering A typical logo image contains approximately 100 features, numerous of which are not important to the matching proceure. This is ue to the fact that some features correspon to ambient illumination reflections, shaows or noise. To maximize the performance of VLR for small, illcapture or noise logos, logo ientification shoul occur with the fewest possible feature matches. It was foun that it is possible to have reliable recognition with as few as 3 feature matches. This step etermines the reasonable NN matches foun in the previous step an verifies whether the uery image represents a logo image store in the atabase. Nearest Neighbor features are clustere using Generalise Hough Transform (GHT) [4], [8], in orer to efine the parameters for a similarity transformation between the uery an the atabase features. Then, an approximate affine transformation follows, which maps each NN from the uery image onto the matche NN feature from the atabase image. Fig. 1. Keypoint Nearest - Neighbour Matching Scheme. For every invariant feature in the atabase there is a recor that contains the image from which the feature comes from as well as the feature parameters position, orientation, scale an the 18-imensional escriptor. Query Q an atabase D feature vector similarity transformation geometry is shown in Figure 3. In this proceure, λ is a normalization coefficient in orer to achieve a reasonable size for the vector length compare to the image size (efault λ=10). 6

threshol, this pair participates in a Hough similarity transformation an a vote is ae to the corresponing cell in the Hough array. The atabase logo image with the highest number of votes is consiere to be the most similar with the uery image an it is forware to the geometrical consistency check with affine transformation (homography). Fig.. Reuce Nearest Neighbour parameter (τ) plotte versus istance threshol (γ). ' ' (x, )=(x + λscos θ, + λssinθ ) (4) ' ' (x, y )=(x + λs cos θ, y + λs sinθ ) (5) In orer to be consistent with a similarity transformation, points (x, y ), (x, y ), (x, y ), (x, y ) in Euations (4) an (5) shoul satisfy the matrix relationship (homography) shown in Euation (6). x 1 0 cos x x 0 1 sin y (6) x 1 0 Tx x x 0 1 Ty y where Λ is the scale change, φ is the rotation angle an T x, T y are the translation values of the similarity transformation. GHT ientifies clusters of features with a consistent interpretation by using each feature to vote for all the logo images that are consistent with the feature. Therefore, a four imensioznal hash table (accumulator) entry can be create, preicting T x, T y, Λ an φ parameters from the match hypothesis, through the similarity transformation, escribe in Euation (6). T x an T y get values between [-a, +a] an [-b +b], where axb is the image imension. In aition, Λ an φ get values in [-4, +4] an [-π,+π] ras, respectively. Bins size of Δφ= π/6 ras, ΔΛ=, ΔTx =a/4 an Δ Ty =b/4 are use for orientation, scale an translations, respectively. The above bin sizes are rather broa an allow clustering even in the case of substantial geometric istortion, ue to a change in 3D point of view. This preiction has large error bouns, as the similarity transformation implie by these 4 parameters is only an approximation to the full 6 egrees-of-freeom posespace for a 3D object an also it oes not account for any nonrigi eformations. When clusters of features are foun to vote for the same pose of a vehicle logo, the probability of the interpretation being correct is much higher than for any single feature. Therefore, when the NN istance between a pair of uery an atabase feature is less than a certain ) Geometrical Verification In this step, the locations of the keypoints in the uery an the atabase image are verifie for geometrical consistency. Those keypoints that fit well to a homography transformation are calle inliers, while those that are inconsistent to it are calle outliers. A keypoint is consiere to be an inlier or outlier accoring to its conformity to an affine geometrical transformation, which takes into account small ifferences in translation, rotation an scaling between the two images. Euation (7) efines the affine transformation for a set of uery an ata points. 1 1 1 x 0 0 1 0 x a 1 1 1 1 0 0 x 0 1 y a x 0 0 1 0 x a 3 (7) 0 0 x 0 1 y a4.................. V... x N N N x 0 0 1 0 x N N Vy N 0 0 x 0 1 x In the above euation, a i (i = 1,, 3, 4) are the warping parameters an V x, V y are the translation parameters of the affine transformation. If Euation (7) is consiere as a system of A*b=c, then solving the respective euations, there is a least suare solution for the parameters b as seen in Euation (8). This euation minimizes the sum of the suare istances from the uery to the atabase image keypoint positions. T 1 T b = [A A] A c (18) Fig. 3. Query an moel keypoint vector similarity transformation geometry. 63

In orer to select which caniate points shoul be use for affine transformation, the RANSAC metho [9] is applie to a pair of images by ranomly choosing three pairs of matche points of interest, since this is the minimum reuire for an affine transformation. These three pairs of points of interest are fitte to an affine transformation, which is then applie to the rest of the points of interest (keypoints) of the two images. Inliers are ientifie by selecting those feature points in one image that are within a certain threshol istance from its matche feature of the other image an minimizing the fitting error. This process was repeate until the inlier count reaches a maximum or the fitting error is below a fixe value. The transformation with the highest inlier count is labele as the best one to map the uery image onto the image of the atabase. The total fitting error, ρ, between uery an logo atabase images uner affine transformation is: Ab c ρ = (9) w where w are the number of keypoints in matrix A, so if ρ in e. (9) excees a threshol, we reject the respective match. Here we have use a threshol of value ρ = 0.05*L x where L x is the maximum imension of the uery image. The above proceure was repeate, for all clusters foun in the Generalise Hough Transform an the atabase image with the maximum number of inliers was selecte. When two or more matches have the same number of inlier points, in orer to choose the best match, the atabase image with the minimum number of outliers is selecte. However, if the maximum clusters size is less than three points, the unconitional match hypothesis is aopte. In this case, the moel with the maximum value of feature matches in the uery image ecies for the logo recognition. The whole process for feature etection, matching an hypothesis valiation is shown in Figure 4, while an inicative example of the NN matches is given in Figure 5. C. Experimental Part 1) Database set The propose metho is assesse on a set of 1500 logo images that have been successfully segmente using our previous work escribe in [1], [] using the Meialab LPR Database [6]. The Meialab LPR Database contains frontal views of vehicles, both stationary an in motion, capture by a Nikon Coolpix 885 ajuste to acuire 104x768 pixel images. From the total of 1500 logo images, 400 have been selecte to form the logo atabase as in []. These images correspon to 10 classes of selecte vehicle manufacturers. Each class contains 40 images an for each one, the keypoints have been etecte an the escriptors have been store. Then, a reference image was selectively chosen by an expert an the remaining 39 samples were registere accoring to that reference view using the homography calculate by the RANSAC algorithm. Only areas belonging to the common parts of the images were selecte an the keypoint escriptors were re-reference to the new position, scale an orientation. In this way, the number of keypoints for every manufacturer logo is substantially increase by merging the keypoints, thus making the recognition process more robust in illumination conitions. Finally, a atabase containing merge keypoints for every manufacturer is create. Fig.4. SIFT Match Hypothesis Testing Flow Chart. ) Query set The remaining 1100 logo images form the uery set for our experiments. For every image, the keypoints an their respective escriptors have been calculate following the SIFT proceure. Then, these escriptors were matche against the escriptors of the atabase set, with the parameter γ set to γ=1. These values correspon to the turning point in the curve shown in Figure. For Eucliean NN search, a KD-Tree ata structure is employe with search time complexity shown in Euation 10. k 1 ( ) O(kN k ) (10) where k is the tree imension an N the number of tree noes [30],[31]. For the experiments in this paper, the typical values are: k = 18 an N ~ 40000 (400 atabase images with approximately 100 keypoints per image). The image in the atabase whose escriptors get the maximum 64

number of votes in Hough Transformation array, is consiere to be the one most similar to the uery image an is then checke with RANSAC for geometric consistency. The recognition statistics are given analytically in Table I. There is an overall 94.6% correct recognition an 5.4% false recognition. The number of Hough-clustere keypoints is approximately 6% of the total keypoints etecte, while the number of inliers is about 50% of the Hough-clustere, as presente in Table II. The recognition spee is shown for comparison with stanar SIFT an M-SIFT, in Table III. A correct match occurs when the two keypoints correspon to the same physical location an a false match when the two keypoints come from ifferent physical locations. The total number of corresponences for the given ataset is known a priori an has been calculate by estimating the homography between each pair of images, for an affine transformation. Total corresponences are the total groun truth matches which are greater that the total matches (false + true) foun. Using 1-Precision versus Recall, we can evaluate the total recognition performance, for a range of experimental conitions, varying the threshol value γ, for NN matching. The results from these experiments using the stanar SIFT features an the enhance (merge) ones, are plotte together for comparison in Figure 6. every moel was substantially increase, thus making the recognition process more robust in outoor conitions, such as low-contrast, inaeuate lighting, reflections, clouy weather etc. TABLE I M-SIFT RECOGNITION RATE Manufacturer True False Alfa Romeo 105 5 Aui 104 6 Bmw 105 5 Citroen 103 7 Fiat 10 8 Peugeot 106 4 Renault 103 7 Seat 104 6 Toyota 106 4 Volkswagen 103 7 Total 1041 59 Average (%) 94.6% 5.4% TABLE II KP SIZE DETECTED VS. HOUGH CLUSTER SIZE & RANSAC INLIERS Query Image Total Hough RANSAC NN Cluster Size Inliers Alfa Romeo 51 4 18 Aui 43 30 14 BMW 30 4 9 Citroen 3 17 7 Fiat 39 15 Peugeot 45 0 14 Renault 47 40 1 Seat 56 16 16 Toyota 37 6 1 VW 1 10 Total 393 49 17 TABLE III VEHICLE LOGO RECOGNITION RATE Methoology Total Recognition Time (ms) SIFT 850 M-SIFT 100 Fig. 5. Database Moel NN matches for a sample uery image. III. CONCLUSIONS This paper provies a escription of a practical application for vehicle make recognition, base on an extening well known image processing routines, such as SIFT. Specifically, a moification of SIFT has been use, where the keypoints from similar objects were merge after homography transformation to the same coorinates an scales. With this techniue, the number of keypoints for Fig. 6. Logo recognition performance for merge SIFT features an stanar SIFT, respectively. 1-Precision an Recall calculate with various threshols for NN matching. 65

The logo recognition system emonstrate goo performance, yieling approximately 95% recognition rate, when applie to a atabase of alreay segmente logo images. One thousan an one hunre (1100) uery logo images were processe with the enhance SIFT-base approach, which gives better recognition accuracy compare to the stanar SIFT proceure. The propose experiments performs better compare to our previous work, in which a Probabilistic Neural Network (PNN) was use [0], [1] with 87% true recognition rate an extens the work escribe in []. The recognition spee is rather fast (~1 sec), which is suitable for real-time applications. To further boost performance an robustness, there is an ongoing research to combine logo recognition with vehicle moel recognition (e.g. Aui A3). A short time vieo that emonstrates some preliminary results in real-time execution can be ownloae from [33]. Moreover, another extension of this work might be the possibility of ealing with a wier range of viewpoints or 3-D recognition, as well as recognition of more complex scenes incluing many vehicles an uner a wier variety of illumination conitions. REFERENCES [1] Weber M., Welling M., Perona P. Unsupervise Learning of Moels for Recognition, Lecture Notes in Computer Science 184, Springer- Verlag, pp. 18 3, 000. [] Kato T., Ninomiya Y., an Masaki I., Preceing vehicle recognition base on learning from sample images, IEEE Transactions on Intelligent Transportation Systems, vol. 3(4), pp. 5 60, 00. [3] Lai A.H.S., Yung N.H.C., Vehicle-type ientification through automate virtual loop assignment an block-base irection-biase motion estimation, IEEE Trans. Intelligent Transportation Systems, 1(), pp. 86-97, 000. [4] Lai A.H.S., Fung G. S. K. an Yung N.H.C., Vehicle Type Classification from Visual-Base Dimension Estimation, 001 IEEE Intelligent Transportation Systems Conference, Oaklan (CA), USA, pp. 01-06, 001. [5] S. Gupte, O. Masou, R. F. K. Martin, N. P. Papanikolopoulos, Detection an Classification of Vehicles, IEEE Trans. on Intelligent Transportation Systems, Vol. 3, No. 1, March 00, pp. 37-47. [6] J. W. Hsieh, S. H. Yu, Y. S. Chen an W.F. Hu, Automatic traffic surveillance system for vehicle tracking an classification, IEEE Trans. on Intelligent Transportation Systems, Vol. 7, No., June 006, pp. 175-187. [7] Lowe D., Object recognition from local scale-invariant features, International Conference on Computer Vision, Corfu, Greece (September 1999), pp. 1150-1157 [8] Lowe D.,, Local feature view clustering for 3D object recognition, 001 IEEE Computer Society Conference on Computer Vision an Pattern Recognition (CVPR'01), vol. 1, pp. 68-688, 001. [9] Lowe D., Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, vol. (60), pp. 91 110, 004. [10] Brown M. an Lowe D., Invariant features from interest point groups, In British Machine Vision Conference, BMVC 00, Cariff, Wales (September 00), pp. 656-665. [11] Mikolajczyk K. an Schmi C. An Affine Invariant Interest Point Detector, European Conference on Computer Vision, vol 1, pp. 18-14, 00. [1] Harris C. an Stephens M., A combine corner an ege etector, Fourth Alvey Vision Conference, Manchester, UK, pp. 147-151, 1988. [13] Siggelkow S., Feature Histograms for Content-Base Image Retrieval, PhD Thesis, Albert-Luwigs-University Frieiburg, December 00. [14] Schulz-Mirbach H., Invariant Features for Gray Scale Images, DAGM Symposium I, pp.1-14,1995. [15] Dlagnekov L., Belongie S., Recognizing Cars, University of California San Diego, Tech. Rep. CS005-0833, 005. [16] Čonos M., Recognition of vehicle make from a frontal view, Diploma Thesis, Faculty of Electrical Engineering, Czech Technical University, 006. [17] Petrovic V. S. an Cootes T. F., "Analysis of Features for Rigi Structure Vehicle Type Recognition", British Machine Vision Conference, vol., pp. 587-596, 004. [18] Maemoto H., Okuma S., Yano Y., Parametric Vehicle Recognition Using Knowlege Acuisition System, IEEE International Conference on Systems, Man an Cybernetics, vol. 4, pp. 398 3987, 004. [19] Lee H. J., Neural Network Approach to Ientify Moel of Vehicles, ISNN 006, Lecture Notes in Computer Science 3973, 006, Springer-Verlag, pp. 66 7. [0] Psyllos A., Anagnostopoulos C.N., Kayafas E., Loumos V. Image Processing & Artificial Neural Networks for Vehicle Make an Moel Recognition, 10th International Conference on Applications of Avance Technologies in Transportation, May 8-30, Athens, Greece, 008. [1] Psyllos A., Anagnostopoulos C.,N., Kayafas E., Vehicle Authentication from Digital Image Measurements, 16th IMEKO TC4 Symposium & 13th Workshop on ADC Moeling an Testing Florence, Italy, September - 4, 008. [] Psyllos A., Anagnostopoulos C.,N., Kayafas, Vehicle Logo Recognition using a SIFT-base Enhance Matching Scheme IEEE Trans on Intelligent Transportation Systems, vol. 11, no., 010, pp. 3-38. [3] Wang Y., Liu Z., Xiao F., A fast coarse-to-fine vehicle logo etection an recognition metho, IEEE International Conference on Robotics an Biomimetics, pp. 691-696, Sanya, China, Dec. 007 [4] Ballar, D., Generalizing the Hough transform to etect arbitrary patterns, Pattern Recognition, 3( ), pp. 111-1,1981. [5] Anagnostopoulos C., Anagnostopoulos I., Loumos V., Kayafas E. (006), "A license plate recognition algorithm for intelligent transportation system applications", IEEE Transactions on Intelligent Transportation Systems, 7 (3), 377-39. [6] The Meialab License Plate Recognition Database: http://www. meialab.ntua.gr/research/lpratabase.html last accesse March 01. [7] Lineberg T., Scale-space theory: A basic tool for analysing structures at ifferent scales, Journal of Applie Statistics, 1(), pp. 4-70. [8] Grimson, E., Object Recognition by Computer: The Role of Geometric Constraints, The MIT Press: Cambrige, MA, 1990. [9] Fischler M., Bolles R., Ranom Sample Consensus: A Paraigm for Moel Fitting with Applications to Image Analysis an Automate Cartography, Comm. of the ACM, vol. (4), pp. 381 395, 1981. [30] Arya S., Mount D., Approximate Nearest Neighbor Searching, Proceeings 4th Ann. ACM-SIAM Symposium on Discrete Algorithms} (SODA), pp. 71 80, 1993. [31] Frieman J., Bentley J., Finkel, R., An algorithm for fining best matches in logarithmic expecte time, ACM Transactions on Mathematics Software,vol. 3(3), pp. 09 6, 1977. [3] Y. Ke, R., Sukthankar, PCA-SIFT: A More Distinctive Representation for Local Image Descriptors, Conference on Computer Vision an Pattern Recognition, vol. pp. 506-513, 004. [33] Vieo Demo: http://www.meialab.ntua.gr/research/lpratabase/ semo.mpg, last accesse March 01. 66