Image Retrieval by Example: Techniques and Demonstrations

Transcription

1 Image Retrieval by Example: Techniques and Demonstrations Elena Stringa, Paul Meylemans 1 and João G.M. Gonçalves Joint Research Centre, Ispra (VA), ITALY 1 Euratom Safeguards Office, LUXEMBOURG Abstract Image retrieval systems have been developed in order to manage large sets of images: the user selects a reference image, and such systems automatically extract from the considered data set a subset of images similar to the example picture. This paper shows that image retrieval techniques can be helpful in managing Safeguards images and reducing the amount of data to be analysed by inspectors. Moreover, an architecture for an efficient retrieval of relevant Safeguards images is proposed. 1. Introduction A time-lapse video surveillance system records typically 25,000 images every three months. In order to reduce the work of inspectors, acquired images are initially selected by a scene change detection system, extracting between 500 and 1,500 pictures. All these images must be analysed by trained inspectors in order to monitor the type of activities that took place in a given plant. Experience shows that from this already reduced dataset, only 10 to 20% of the images are Safeguards relevant, amounting to between 100 to 150 images. This reduction factor is the primary motivation to find new reliable tools for further reducing (or filtering) the number of pictures to be analysed by inspectors, or, at least, for qualitatively improving the management of this large amount of visual data. Image retrieval systems have been developed for managing large sets of images contained in a single or distributed database, for example in Internet-based applications. The main peculiarity of such systems is the possibility of presenting to users a subset of images whose content is similar to the one of a given visual example. For the application of image retrieval to Safeguards, it is necessary that the following requirements are satisfied: Flexibility: for an easy system setting up. The user interface should allow an easy modification of parameter setting. Reasonably, performance are maximized if the system parameters are tuned depending on the kind of monitored nuclear reactor. Performance: all relevant images must be retrieved and presented to the inspector; even if this means to have some false positive results. Effectiveness of result presentation: the user interface should allow an easy analysis of the selected images, including access to all the information related to the images (e.g. acquisition time and place). Image management: the system must allow an easy image management (image moving, copying, bookmarking, browsing, etc.) In this paper we are presenting different systems and techniques available in the field of image retrieval. More precisely, section 2 introduces the basic concepts of image retrieval; section 3 provides an overview of the systems commercially available; section 4 presents some preliminary results performed with real Safeguards images, showing the feasibility of the considered application; in section 5 we are proposing an architecture of a surveillance system with image retrieval capabilities. Finally, in section 6 some conclusions are discussed. 2. Image Retrieval by Example: Basic Concepts Systems for image retrieval by visual examples involve the application of advanced image processing tools and pattern recognition techniques for a fast, efficient and helpful indexing and management of large sets of images. Commercial systems are being mostly developed for Internet-based search applications. The proposed idea is to use image retrieval technologies for extracting Safeguards relevant images from a surveillance database constituted by a stream of still pictures. In order to manage image sets, two operations must be performed: database creation/updating (indexing phase) and database interrogation (query phase). The core of image retrieval is the way in which an image is represented for being stored in a database: it has to be identified by a numerical vector (index) representing its visual content as faithfully as possible. In the literature, two main approaches for image feature creation (i.e., indexing) can be found. The first approach considers images as a global entity, thus image indexes are created by extracting global image descriptors (e.g. mean luminance value of the entire image, colour distribution histogram, etc) [1][2]. The second approach divides each image as a set of regions/objects (i.e.,

2 segmentation) and the indexes are composed by a sequence of sub-indexes related to the extracted regions; in this case features (typically colour and shape) are extracted for each image portion [3][4][5][6]. When querying the database, the first approach uses a whole example image whereas an object or an area is used in the second approach. In each case, the result of the query is a subset of images ranked by decreasing degree of similarity with respect to the example. It is interesting the fact that during the image analysis e.g. after the retrieval images can be divided in relevant and irrelevant, according to their content, without being copied/moved from the source device. Images belonging to a certain category, e.g. the category of relevant images, can be re-analysed in a second time, if necessary, without having to perform again the query. Moreover, a class of image retrieval systems presents the interesting possibility of performing iteratively the retrieval of images: after each iteration, the user provides feedback by indicating the correct and wrong results (relevance feedback). The set of selection parameters is trimmed and a new retrieval made. It can be said that such a system incorporates some degree of learning ability. 3. Image retrieval systems available on the market The first step of our feasibility study was to look for what the market has to offer in the field of image databases. Despite the large amount of research in this field, proved by the huge number of research prototypes described in the scientific literature [7], there are extremely few products commercially available. Commercial systems can be divided in two classes: development kits and closed systems. Image retrieval development kits provide software libraries containing image database management functions; the advantage of such systems is that one can design and develop an application ad hoc for a given problem; the drawback is that the designing and development of professional image retrieval applications is still a tough and time consuming job, even with the availability of the development kit. Such systems are Excalibur Visual RetrievalWare SDK [8], IBM DB2 Universal Database - Image extender [9], that includes the functionality of the so called QBIC SK system [1][10], and the Virage VIR Image Engine SDK [11]. The second group is composed by proper application packages. By means of an appropriate user interface, it is possible to easily create and query image databases. The drawback of these systems is that one cannot extend the pre-programmed functionality. Such applications are Attrasoft ImageFinder and IMatch 2. IMatch 3 is a system somewhere in between the two above mentioned classes. It represents a trade-off between a closed system and a software development toolkit, because it has a Plug-In interface allowing the addition of user algorithms and matching modules to the IMatch. This means that it is possible to extend the image retrieval functionality, by adding new image descriptors or changing the matching technique, without having to develop a new system. Details on systems commercially available can be found in [12], containing information updated up to the first semester of year It is noteworthy that none of the available systems presents the relevance feedback functionality. For our research, we have performed experiments with QBIC, Kingfisher [3], and IMatch [5]. 4. Results from preliminary tests In this section we describe the experiments performed with large sets of images. The experiments had two goals. First, we wanted to verify whether if image retrieval techniques are capable to extract relevant Safeguards images from large data sets, i.e. if they are able to discriminate images of scenes from the same view point taken from fixed digital cameras. Second, we wanted to be sure that image retrieval functionality effectively make easier the task of Safeguards inspectors. Therefore we performed two kinds of preliminary evaluations: the first one based on image retrieval performance with realistic data by using available systems [13]. The second one is an evaluation of the functionality of different image retrieval system Performance evaluation In this section we present the results of preliminary tests performed at ESO on realistic images. Two different systems were tried, presenting the two main approaches implemented in image retrieval (based on the global image or based on regions). The two systems used were QBIC and Kingfisher. For the evaluation, we first performed a classification of the images forming our test set, that was composed of 538 greyscale pictures, each of 496x288 pixels. Images were divided in relevant and irrelevant; the relevant class was further analysed and divided in not mutually exclusive sub-class depending on the occurring relevant event. Table 1 shows the test set characterization, reporting the number of images for each of the individuated classes.

3 For each class of relevant events we count the number of relevant images retrieved in the group of the best 50. The results obtained with QBIC and Kingfisher are reported in table 2, where we have considered as image/region descriptor the intensity mean value and the luminance distribution. The last feature is equivalent of the colour histogram for greyscale images. This result is extremely encouraging: even though not all the relevant images are returned after the first query, it is possible to refine the search by changing the example image. It can be seen that a training set of 3 images is enough to have the performance close to 100%. Class ID Content description Class size C 0 Irrelevant 244 C 1 C 2 C 3 C 4 C 5 Container shell over decontamination area Container vertical over the pond or over the truck Container vertical moving from/to the decontamination area Container in the decontamination area or entering/exiting Container laying on the truck Figure 1. percentage of images retrieved iteratively for class C 1 (Kingfisher system, intensity distribution feature). Table 1. Safeguards test set characterization QBIC Kingfisher mi id mi id C 1 10% 10% 10% 50% C 2 30% 56% 56% 50% C 3 10% 50% 40% 50% C 4 5% 32% 7% 35% C 5 12% 24% 10% 15% Table 2. Percentage of relevant images retrieved by QBIC and Kingfisher as result of the first query by using the mean intensity value (mi) and the image intensity distribution (id) features. Results are reported for the relevant classes indicated in the left column. From preliminary results, it is noteworthy that the two different approaches provide different results. In general, the region-based method provides more relevant results. In order to increase the performance it is necessary to perform again the query, by taking as example image one of the correct results of first query. By considering the relevant event C 1, Figure 1 shows how performance increases after each iteration, by analysing 50 images and changing the example each time (example related to Kingfisher, by selecting the intensity distribution feature). Temporal performance In Safeguards, large image databases have to be built every time a surveillance dataset is available (in general, one dataset every three months from each monitored installation). Temporal performance is then important in both the phases of database creation and querying. QBIC, Kingfisher and IMatch were evaluated from the computational time viewpoint. We have performed experiments with a set of 83 color images whose size is 325x260 pixels. For what concern the query, all the systems are very fast and provided similar performance: in fractions of seconds the query results are available. Temporal performance varies meaningfully for what concern the database creation, as reported in table 3. The results were obtained on a Pc with processor Pentium II 400MHz.and RAM of 128 Mbyte. Indexing time QBIC Kingfisher IMatch 40 sec 420 sec 10 sec Table 3. Temporal performance: time to build an image database from a dataset of 83 colour images (325x260 pixels)

4 4.2. Functional Evaluation This section presents an evaluation of the functionality of tested systems that are the QBIC commands, the Kingfisher and IMatch. As mentioned in section one, an image retrieval system must satisfy some requirements for being useful in Safeguards. Here we are describing such requirements and in table 4 we report the evaluation of image retrieval system based on the functionality point of view. Parameter setting Presentation of results Image management QBIC Kingfisher IMatch LEGEND: Empty field: functionality not present; : partly implemented functionality or with a poor user interface; : functionality with a good user interface; : functionality with a very good user interface. Table 4. functional evaluation of QBIC, Kingfisher and IMatch systems Parameter Setting Tools The image retrieval procedure will be applied to sets of surveillance images acquired in different nuclear installations. For each image set, a database should be created. Images from different plants and surveillance cameras show a different environment, therefore the content, from the image processing point of view, can be very different. Differences in the environment are reflected in the values of the image descriptors used for the database creation. For instance, the colour distribution could be very different for two images coming from different places. In order to guarantee good results for all the databases and to minimise the number of queries to be performed by the inspector, it is necessary to individuate a set of default parameter settings adopted to each database (i.e. monitored environment). This operation can be easily performed if the image retrieval system has a userfriendly interface dedicated to the tuning of the parameters. QBIC and Kingfisher systems allow the selection of all the combinations of available features. Moreover, Kingfisher presents a query interface that makes easy the feature selection operation. IMatch system presents the maximum flexibility because during the setting phase it is possible to modify the importance assigned to each image descriptor. The user interface is good because it is possible to save the different settings so that they appear in a menu in the query interface. This means that, during the query, the inspector indicates the type of installation s/he is analysing, whereas in QBIC and Kingfisher the inspector must know the correspondence between the parameters to be selected and the specific database. Another nice property of IMatch is that the inspector, during the query phase can select an area of interest into which the system focus the attention, without considering the activity outside that area. Moreover, a good user-interface allows an extremely easy and flexible choice of the example images, and to save the current parameter setting for further queries. Presentation of results In Safeguards image analysis, it is important to have a good display of the relevant images. QBIC commands display on the screen the list of the retrieved images with the related value of their distance from the example; in order to analyse them, it is necessary to open them with a specific application or to develop an appropriate user-interface. Kingfisher displays the retrieved images reduced in size, making their analysis very hard. Information on similarity with respect to the example is printed on the interface, but results cannot be saved. IMatch provides a good display of the results and the possibility of viewing the results in real size (figure 2); moreover, some image processing tools are available in order to make the analysis easier, e.g. by improving the image contrast. Some information about the retrieved images can be also exported as a text file. Image management In order to make image retrieval more effective, it is useful to have tools allowing an easy image management. In particular, it would be useful to be able to copy the relevant images, or to assign them to a special category inside the database, or to remove from the database the irrelevant images erroneously retrieved, etc. QBIC commands allow to update databases, by adding or removing images, but the image management is not easy because of the lack of user interface. Kingfisher allows the insertion and/or removal of images in the database, but it is impossible to copy images in a separate folder and the concept of image category is totally absent. IMatch presents good image management properties. Through its interface, it is easy to add, remove, copy images and to organize them in categories. The last property is extremely useful, because whenever a set of relevant images is found, it can be re-displayed without

5 having to perform the query again, therefore without the extraction of false positive. Moreover, the relevant images can be easily selected as new examples for iterative queries. Reference image Analysed Figure 2. Result window of IMatch system with images from Ispra laboratory. The right area of the user interface shows the thumbnails of the retrieved images; in the upper-right area there is the reference image in the real size, and the lower-left area shows the image indicated by the user (shadowed selection in the right area) with its real size. 5. Architecture of the image retrieval system for Safeguards applications This section presents the proposed architecture of the system for Safeguards image retrieval. It has been designed by taking into account the properties of image retrieval systems commercially available and the requirements of Safeguards inspectors. The proposed system (figure 3) is composed by three main modules: system set-up, image management and retrieval modules. The system setting-up is performed by a system expert with the assistance of a Safeguards inspector. By means of training data-sets, characterized by the inspector, indexing and query parameter sets are defined and saved in order to optimise the system performance. A relevance feedback operation is included in order to refine the parameter settings. After a query, the user indicates which are the corrected images and which are irrelevant. On the basis of this information, the relevance feedback module re-computes the set of query parameters optimised for the considered query. The image management module aims at managing images to be inserted in two different databases: the reference database and the surveillance database. The reference database contains example images representing relevant events occurred in the monitored installation. The inspector will select images from this database as visual examples for his queries. More precisely, the inspector selects from this database one or more images representing a precise event of interest, in order to ask of retrieving all the images showing a similar event. The reference database can be updated with the addition of new examples or the removal of not useful ones, but this updating is rarely performed. The surveillance database is formed from images selected on the basis of scene change detection. A new surveillance database is created each time a new image set arrives; this happens more or less every three months for each monitored nuclear plant. In order to make easier the query phase, when retrieved, irrelevant images

6 can be removed from the database through the database controller. The inspector could be interested in making copies of the relevant retrieved images. For this reason, the image management module is also enabled to export images to the computer drives. Set-up commands Set-up Set-up module module Relevance Relevance feedback feedback Set-up commands SYSTEM SET-UP Results information Human-computer interface Indexing parameters Query parameters Indexing commands Image management commands Query command Retrieval results Images fro m CD Indexing Indexing Indexes Surveillance module module database Indexes Retrieval Retrieval module module Reference images Indexes Reference database Database Database controller controller RETRIEVAL IMAGE MANAGEMENT Reference index Figure 3. Proposed architecture for an image retrieval system for Sageguards. The indexing module physically provides the database creation according to the indexing parameters passed by the set-up module. These parameters are related to image descriptors used for creating image indexes. The retrieval module performs the matching between the example image and the images in the surveillance database. It has the following inputs: - The set of query parameters from the set-up module; these parameters are divided in two groups: the first one is related to the image descriptors enabled for the specific query, and the second one is related to the matching strategy (i.e. the kind of distance used for the matching and the threshold value limiting the number of results); - Indexes of the images in the surveillance databases; - Indexes related to the reference images (selected by the user through an image management command); - A command indicating to perform the retrieval; The retrieval module returns the subset of images whose content is similar to the one presented in reference image, i.e. showing an equivalent interesting event. Results are ranked in decreasing order of similarity with

7 respect to the example. 6. Conclusions and next activities The application of image retrieval techniques in Safeguards is feasible and helpful. A ready-to-use application package presenting both the required functionality and a satisfactory reliability of results is not yet commercially available. Therefore, it is necessary to adapt an image retrieval system allowing the introduction of additional image descriptors and matching algorithms. Next activities are focused on a rigorous performance evaluation of the tested systems for individuating the image processing techniques (image descriptors, matching algorithms, etc) that have to be implemented in the retrieval system for Safeguards. 7. Acknowledgements Authors wish to thank Dr. Juha Pekkarinen (Euratom Safeguards Office, Luxembourg) for his support during the experiments on real data, and Prof. Silvana Dellepiane (University of Genoa, Italy) for kindly providing the possibility of testing Safeguards databases with the Kingfisher image retrieval system. References [1] [2] R. Brunelli and O. Mich, COMPASS: an Image Retrieval System for Distributed Databases, Proceedings of ICME 2000, IEEE International Conference on Multimedia and Expo, New York City, July 30 - August 2, [3] R. Vaccaro, S. Dellepiane, Image Database Management with Content-based Techniques, Technologies for the Information Society: Development and Opportunities, J-Y. Roger et al. (Eds.), IOS Press, 1998, pp [4] http: // / rapportsactivite / RA2000 / imedia/logic_logiciels.html [5] [6] [7] A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 12, December 2000, pp [8] [9] http: // www-4.ibm.com / software / data / db2 / extenders / image.htm [10] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petrovic, D. Steele, and P. Yanker, Query by Image and Video Content: The QBIC system, IEEE Computer, [11] [12] C.C. Venters, and M. Cooper A Review of Content-Based Retrieval Systems, July 2000, [13] E. Stringa, J.G.M. Gonçalves, and P. Meylemans, Image Retrieval by Examples: a Feasibility Study for Applications to Safeguards Tasks, Internal document, November 2000.

8