Auto-Detector High-performance Mobile License Plate Recognition System Giovanni B.Garibotto, Paolo Castello, Enrico Del Ninno, Giovanni Borghero, Pietro Pedrazzi, Gianfranco Zan Elsag spa, Genova, Italy Abstract A new mobile License Plate recognition system is described, aimed to automatically detects and read the license plates of the vehicles that appear in the field of view of its micro cameras. The system has been designed for a variety of applications in the field of traffic control and security. In fact all recognized license plates are immediately compared on board, against one or more lists of selected number-plates correspopnding to different levels or priorities of investigation. The paper will refer some of these applications with a few comments from the first year of direct experimentation in the field. Actually this advanced solution has been selected by the main italian forces (Carabinieri and Road Police) to provide an effective monitoring of the territory and increase the level of security for the citizens. Experimental results from the available pilot installations are referred. Keywords: Optical Character Recognition, Computer Vision, License Plate Recognition, Security, Traffic Control 1 Introduction Auto-Detector is a new Mobile On-board License Plate Recognition system, able to detect and recognize the license plates of parked as well as moving vehicles, within the field of view of its micro video cameras. The recognition process for moving vehicles from a moving platform is much more complex than for fixed urban or highway traffic control installations, and represent the collection of all possible problems in Computer Vision. In fact all critical situations are concurrently present, with constant, rapid movements and changes of lighting conditions during patrols (transition from sunlit to shadow areas, tunnels, night patrols, etc.). Elsag s O 2 CR technology, developed around the embedded Intel X-Scale platform, plays a key role in solving these problems. It performs constant recognition at video frequency, achieving performance of about 25 license plates read a second, even in complex, constantly changing situations like those recorded by a moving sensor. The vision sensor is coupled with a pulsed infrared LED illuminator and integrated in an extremely compact package for easy installation on patrol vehicles without requiring any radical modifications to existing equipment. This document describes the functions, technical characteristics and performance of the Auto-Detector Mobile License Plate Recognition system, designed for use during patrols in urban, extra-urban and highway contexts to identify the license plates of vehicles encountered. The system is integrated with localisation and control equipment for communications with the central data collection unit and to permit consultation of the licence plate data-base and geographical positioning information. The Auto-Detector system can also be integrated with a video-recording device connected to a colour micro video camera mounted under the rear view mirror, in order to achieve a panoramic view of the scene, if needed. 1
Section 2 summarises the product s characteristics and the basic component integration architecture. Section 3 describes how the system works in real life patrol situations. Section 4 reports on performance results achieved during the initial test phase in various operating conditions (speed cruise, type of road, etc.). 2 Auto-Detector system characteristics The block diagram in Fig. 2.1 illustrates the two main components: the onboard license plate recognition, alarm detection and video-recording unit the central data update, consultation and alarm management unit. The onboard unit integrates license plate recognition functions (O 2 CR-FPM) with the vehicle s onboard navigator system, which is used to manage online communication with the control centre, to display license plates recognised during patrols on the existing onboard display and to signal alarms (blacklisted licence plate detected). There is logical separation at control centre level between the radio positioning station (which hosts the Accentra and GEODI radio positioning and GSM vehicle communication applications) and the licence plate consultation station. Operations Centre Radio positioning station License plate control station GSM Modem ACCENTRA GEODI Vehicle communications Vehicle radio positioning LAN TRG Entry and deletion of blacklisted license plates Alarm reception Onboard vehicle GSM Modem RS 232 Navigation System Wireless- LAN RS 232 O 2 CR Field- Processing Machine FPM Left camera Right camera Fig.2.1 Auto-Detector functional block diagram 2
2.1 Database consultation and query Operations Centre personnel can consult the database of licence plates recognised and stored during previous missions at any time and in a number of ways in order to: search for a licence plate in the database (both complete and partial matches) and report where and when (if at all) it was encountered. The software running on the license plate control station uses a cartographic system to enable operators to identify license plate recognition sites simply by selecting the transit record in question. It is also possible to display the image of the vehicle corresponding to the recognised licence plate, plus the map position of the site in which it was detected. Additional viewing functions include: o zoom o image enhancement re-examine the vehicle transit images that triggered alarms to check for correctness and identify the type of vehicle if possible. update the license plate blacklist by adding or removing license plate strings and entering comments on the type of alarm (stolen vehicle or investigation in process, etc.). Password protection ensures that only authorised personnel can consult the system. Fig.2.2 Licence plate consultation and blacklist update screens 2.2 Imaging Sensor integration A key issue was the configuration of a miniaturised video camera to allow the unit to be mounted on patrol vehicles without altering their functional characteristics. To ensure effective image acquisition and recognition, a solution was needed to the issues raised by wide, unpredictable variations in lighting conditions, from just a few lux in areas of shadow, tunnels and at night, to more than 100 Klux in full sunlight. The integrated acquisition system comprises a digital micro-camera coupled to a programmable high-speed pulsed LED illuminator. To deliver maximum acquisition efficiency, the pulses of light are synchronised with the camera s acquisition system so that they occur at the same time and have the same duration as the optical sensor s exposure time. The beam of light emitted by the LED is in the near infrared spectrum (solutions using LEDs in the range 730 to 810 nanometres are available) to limit ambient light interference. This makes it possible to: compensate for low light conditions attenuate the interference caused by shadows thrown on license plates in full sunlight. 3
The achieved degree of miniaturisation, as depicted in fig. 2.3, has been possible using high level LED integration (more than 88 LED-on-chip) inside a 2.5 cm diameter cylinder and the use of special digital micro-cameras (progressive scan). Fig.2.3. The level of miniaturisation achieved means the license plate reading sensor can easily be adapted for use in a broad range of tailor-made applications. 2.3 Licence plate processing and recognition unit Fig. 2.5 illustrates a block diagram of the O 2 CR-FPM (Field Processing Machine) image processing architecture installed in patrol vehicles baggage compartments. It performs the following main functions: continuous reading of the two digital video streams delivered by the micro-cameras, with the identification of all license plates detected in the scene and recognition performance in excess of 15 consecutive readings a second recognition time integration to distinguish between transits and avoid signalling the same licence plate more than once comparison with the license plate blacklist (uploaded from the operations centre at the start of patrols and with the possibility of in-patrol updates over a GSM channel if required) data exchange management with the onboard navigator to signal alarms and receive vehicle positioning data. To achieve these performance levels across two independent channels (left and right cameras) requires considerable processing capacity combined with low consumption (much lower than that offered by standard industrial components) to avoid excessive onboard battery drain. The solution adopted implements INTEL X-Scale embedded technology: INTEL X-Scale integer processor (880 MHz, 32 bits, 64 MB RAM) 100 Mb/s network connection low consumption and 12-24 V supply extremely compact easy remote link for maintenance and upgrades. As illustrated in fig. 2.4, the system acts as a fully-fledged network server. Communications with the operations centre are handled over a wireless-lan link at the start/end of patrols and the system is already configured to accept the latest telecoms technology developments 4
based on GPRS or UMTS connections. Thanks to the high level of miniaturisation achieved, it has been possible to mount all the instrumentation in the compartment between the baggage compartment and the rear seat. Micro camera + IR Flash Smart-Reader and controller License plate reading unit Wireless LAN bridge Micro camera + IR Flash Colour Micro camera Smart-Reader and controller Ethernet LAN Ethernet Frame-Vision Store and harddisk Videorecording (optional) Navigation System and Display Composite PAL Video RS232 serial link Fig. 2.4. FPM image processing and onboard license plate recognition unit block diagram This solution concentrates all the processing and control electronics in a hidden compartment that can be accessed from inside the vehicle, without taking up any relevant space in the baggage compartment, which remains completely available for service equipment. 3 Onboard Licence Plate Reading system operation The paragraphs below describe how the system works and user operations. 3.1 Data and blacklist downloads Data and blacklist downloads are performed from the Operations Centre using a PC workstation server with a user interface and applications software to update and consult the licence plate database. The system is connected over a network to the geographical information system units. Data exchange from the central station to the onboard vehicle reading unit (hereafter FPM) installed in the baggage compartments is handled by a wireless-lan link in a special data exchange area (e.g. inside the police force car-park or workshop). This minimises human intervention and makes the data exchange process as automatic as possible. 3.2 Start of patrol Patrol officers are not required to perform any supplementary operations when starting their vehicles and all data update operations are performed completely automatically over a wireless-lan link. After receiving the updated license plate blacklist from the operations centre, the system is ready to start monitoring license plates on patrol. 5
At power on, the FPM system communicates its status to the onboard navigator, which signals whether or not the connection is active by means of a green indicator lamp on the onboard display. During uploads the two units constantly monitor progress and signal any malfunctions. On completing the upload the FPM is ready to acquire, recognise and store license plate numbers detected during the patrol. 3.3 Licence plate reading during patrols During patrols, officers receive and send messages from and to the Operations Centre using the onboard navigator terminal, which in addition to its usual functions, also offers the following services: the licence plate recognition unit stores data referring to vehicle transits in the lanes to the right and left of the patrol vehicle. The string of the licence plate most recently recognised by each camera is updated continuously on the onboard navigator s display on the left or right side according to the camera that recognised it. This is illustrated in the figure below and provides officers with a check that the recognition system is up and running. Transit data recorded by the onboard unit contains: the license plate string recognised, a compressed image of the vehicle and the date, time and position (geographical position data from the onboard navigator). When a license plate on the blacklist is recognised, the system sounds an alarm and the acquired digital images are immediately displayed on the screen to provide the human operator an evidence of the positive detection. The alarm signal is automatically sent also to the Operations Centre through the onboard communication system (based on standard channels GSM-GPRS) in real time. A marker is automatically entered on the onboard navigator map to display the position (on the navigation map) where the license plate was detected and recognised. During patrols, the Operations Centre can also send the onboard navigator supplementary license plates to monitor in addition to those stored in the onboard blacklist. New data strings are entered into the system by a programme running on the license plate control station and linked to the Accentra software package which can broadcast data to all onboard systems. When the onboard navigator receives a message containing a new license plate to monitor, it sends the appropriate string to the onboard FPM system which enters it into the blacklist. 3.4 End of the mission On returning from patrols, the system automatically uploads the data it has collected to the central unit over the wireless-lan link and then shuts the system down. Vehicle transit data 6
and images are stored onboard in encrypted form and can be consulted only from the central license plate consultation unit using special decryption and display software. 4 Results and performance The license plate reading system is designed to recognise standard approved license plate formats. Recognition performance can be affected by non-standard formats and damaged or dirty plates. The performance figures given below are based on the principle of subjective readability by a human observer sitting in the patrol vehicle next to the driver. 4.1 Camera orientation The micro-cameras can be suitably oriented inside the sensor assembly mounted on the patrol vehicle s roof. Various possibilities were considered to establish the optimum arrangement as shown in figure 4.2 below. Studies were based on the following vehicle position categories around the patrol vehicle: A. vehicles moving in the same direction as the patrol car in the lanes to the right and left (overtaking). This situation is typical on urban throughways and highways (limited to the lanes immediately adjacent to the patrol vehicle. B. vehicles moving in the opposite direction to the patrol car in the oncoming lane to the left. This situation is typical in narrow urban contexts with two-directional traffic. C. vehicles parked in the direction of traffic flow to the right of patrol vehicles, and also to the left in narrow one-way streets. This situation is typical in all urban areas (main arteries, narrow city lanes, etc.). D. diagonally parked vehicles to the right and left depending on the type of road (oneway or two-way). This situation is typical in some urban contexts to optimise available parking space and is very frequent in large factory, commercial and airport carparks. The angle of vehicles varies considerable, from the minimal angling used in airport carparks to nearly ninety degrees in some crowded, unregulated carparks. E. all other possible categories, including vehicles parked arbitrarily (even perpendicularly to the traffic flow), on unpaved surfaces and elsewhere. During testing we performed a preliminary study to estimate a priori the probability of encountering vehicles falling into the aforementioned categories. The estimated percentages given in figure 4.1. provided a useful starting point for system capacity planning, but clearly need to be adapted to real patrol requirements and objectives. There are no theoretical limits to the onboard sensor training configuration, which can be adapted to specific user requirements. Our own particular configuration, illustrated in figure 4.2, represents a compromise to collect the largest possible number of license plates during a patrol mission. Only vehicles in category E below cannot be handled by this configuration. All other categories can be handled with differing degrees of success as briefly set out in the paragraph below. 7
25% 15% 30% 20% 10% A B C D E Fig. 4.1 Position categories assumed for vehicles encountered during a patrol. See text for category descriptions. 6 m. c.a. 3. m c.a. 3. m. c.a. Fig. 4.2 Onboard camera training configuration 4.2 Experimental results Auto-Detector has been integrated into a series of experimental pilot systems since March 2002 fo the Italian police forces (Polstrada) as well as for the investigation security forces (Carabinieri), in a variety of typical patrol missions, downtown (in Rome and Genova) as well as along the main highways. At the beginning of 2003 a first series of patrol cars have been already installed and put into service in the main Italian cities. Recognition performance reported below refers to typical license plate conditions (level of dirt build-up, shadows and reflections, colour, etc.). The system can handle images in all environmental conditions thanks to the characteristics of the micro-cameras and the use of optical filters in the near infrared (IR) spectrum to ensure average attenuated luminosity even in conditions of full sunlight and greater stability for automatic reading purposes. Performance figures were calculated by comparing the number of license plates read by the onboard system to the number of license plates considered readable (and counted manually) by an observer sitting next to the driver. This means that license plates which cannot be framed by the sensor system are not considered to be readable. The following average results were recorded: 8
motorway traffic: at an average cruising speed of 110 km/h (fig. 4.3) the average percentage of successful recognitions vs. readable license plates by an onboard operator was > 80% Fig.4.3 Motorway (daytime) Fig.4.4 Parked vehicle (daytime) vehicles parked at the roadside (in the direction of traffic flow) along urban road sections (fig. 4.4): at an average speed of about 40 km/h, the average percentage of successful recognitions vs. readable license plates by an onboard operator was approximately 85% parked vehicles: at an average speed of about 60 km/h, the average percentage of successful recognitions vs. license plates readable by an onboard operator was approximately 75% diagonally parked vehicles: at an average speed of about 30 km/h, the average percentage of successful recognitions vs. license plates readable by an onboard operator was approximately 90% Fig.4.5 Diagonally parked vehicles Fig.4.6 Night time. Night time recognition is affected only by very dirty or damaged plates. Otherwise the near infrared LED illumination produces images that are even more favourable for recognition than at daytime by strongly attenuating all objects without retroreflectivity characteristics like license plates. there was no appreciable performance reduction in rain or overcast conditions. In these cases environmental conditions make the images acquired more uniform, like those at dusk or night 9
the error percentage for all license plates recognised by the system is extremely low (and always lower than 1%), despite the highly variable lighting and direction conditions of vehicles encountered during patrols. This result has been achieved thanks to the explicit use of geometrical context constraints and license plate number syntax, in agreement with the rules established by Italian certification authorities. 5 Conclusions This document describes the technical and functional characteristics of the mobile license plate recognition system (Auto-Detector) and reports on the preliminary results of tests. The decision to adopt fixed camera positions is a compromise to acquire as much data as possible (license plate strings and images) during patrols and has already demonstrated it can deliver very high performance as described in chapter 4. This capillary data collection system can rapidly alert patrols (through the onboard navigation system) and the operations centre when blacklisted numbers are detected. The system has been designed for low consumption to fit with existing power supply onboard. It integrates perfectly with existing on-board systems without adding any extra effort to the workload of patrol personnel, who is free to operate as usual in his/her patrol missions. It can be easily defined as an auxiliary officer on-board of the car, with a reading and recognition performance that highly exceeds human capabilities, being able to detect immediately any appearance of selected number plates in the neighbourhood of the patrol car. As such it strongly enhances the capabilities of the patrol crew. 6 References [1] www.utad.pt/numberplate/ Vehicle Number Plate Recognition Systems. [2] Traffic Technology International 2003, UK and International Press, traffic@ukintpress.com [3] G.Garibotto, et al. Dynamic Vision for License Plate Recognition, chap 6.6, Multimedia Vision-based Surveillance Systems, Kluwer Academic Publisher, 1999. [4] G.Garibotto, et al. Speed-Vision: speed measurement by License Plate reading and tracking, 2001 IEEE Intelligent Transportation Systems Proceedings, pp 585 590. 10