Content-Based Image Retrieval of Web Surface Defects with PicSOM

Transcription

1 Content-Based Image Retrieval of Web Surface Defects with PicSOM Rami Rautkorpi and Jukka Iivarinen Helsinki University of Technology Laboratory of Computer and Information Science P.O. Box 54, FIN-25 HUT, Finland {rami.rautkorpi, Abstract This paper describes the application of PicSOM, a content-based image retrieval (CBIR) system based on selforganizing maps, on a defect image database containing 24 images from a web inspection system. Six feature descriptors from the MPEG-7 standard and an additional shape descriptor developed for surface defect images are used in the experiments. The classification performance of the descriptors is evaluated using K-Nearest Neighbor (KNN) leave-one-out cross-validation and PicSOM s built-in CBIR analysis system. The KNN results show good performance from three MPEG-7 descriptors and our shape descriptor. The CBIR results using these descriptors show that PicSOM s SOM-based indexing engine yields efficient and accurate retrieval of similar defect images from our database. I. INTRODUCTION The development of technology required to produce high quality digital images and the increasing capacity of digital data storage devices have made possible the creation of huge digital image databases. This in turn has necessitated the development of a system to efficiently manage and search such databases. Traditional text-based search methods can be applied to image databases, if the images in a database have been annotated by a human with keywords describing the content and nature of the images. However, this approach becomes unpractical as the size of the database increases. Also the annotations are subjective judgments by the annotator, which means that if several people are involved in annotating a single database, the annotations, and thus any search results, may be inconsistent, and thus the annotations may not reflect the needs of a particular end user of the database. A more efficient approach is content-based image retrieval (CBIR), where low level visual features are extracted from the images, stored and indexed in order to speed up searching [], [2]. The problem is to develop descriptors that capture the essential features of an image, and a system that makes it possible to match these features with the semantic concepts of content and significance that a human user sees in the images. One way to implement such capabilities in a system is the use of relevance feedback provided by the user, enabling the system to refine the search criteria according to the user s preferences. Web surface inspection systems produce large quantities of defect image data, and the classification of such defect images is a significant challenge for CBIR systems. Manual checking of all these defect images is too time-consuming, so some automatic tool is necessarily needed. In this paper we propose to use PicSOM[3], [4], a SOM-based CBIR system, to handle this task. In our previous work we have been dealing with defect images from a paper web inspection system[5], [6] but in this paper we deal with a new kind of defect images, namely metal defect images. So, experiments are conducted with a defect image database containing 24 images from a metal web inspection system. Six feature descriptors from the MPEG-7 standard [7], [8] and the additional shape descriptor developed for surface defect images [9] are first evaluated with a simple K-Nearest Neighbor (KNN) classifier. The best features are then implemented in PicSOM, and finally the retrieval performance of PicSOM is evaluated with these defect images. II. PICSOM RETRIEVAL SYSTEM PicSOM is a content-based image retrieval system for large, unannotated databases, developed at the Laboratory of Computer and Information Science at Helsinki University of Technology [3], [4]. It uses the self-organizing map (SOM) [] as a means of indexing feature data from images. The SOM provides not only an efficient indexing engine, but it also provides an automatic clustering of defect images that is very important in our case. This clustering can be used to reveal the main defect types and also to help to detect more rare defects. The retrieval process is implemented using relevance feedback. When training PicSOM, the first step is to calculate a number of features from the images in a database. These feature vectors are then used to train tree-structured SOMs (TS-SOM) []. The TS-SOM is a hierarchical structure that has a SOM at each level, with the map sizes increasing towards the bottom. The tree-structure speeds up the training and searching of the SOMs. After the training is complete, the distribution of the map units in the feature space reflects the distribution of the feature data. The feature vectors can then be associated with the nearest map unit in the feature space, i.e. the best matching unit (BMU). The maps in the TS-SOMs now serve as two-dimensional indexes to the feature data. The feature vectors that have the same BMU are very similar, and the map units surrounding the BMU represent less similar feature vectors. The division of the

2 map units into clusters in the feature space can be interpreted as a reflection of natural class divisions in the original image data. Searching in a database is iterative, beginning with the system presenting an initial set of images to the user, who can then select the images that best match the type of image that is being searched for. The system then assigns a positive relevance score to each map unit according to the number of selected images that have the unit as their BMU. A negative score is assigned to map units for all the images that were not selected by the user. The relevance values are then spread into the surrounding map units with low-pass filtering. Previously unseen images from the highest scoring map units are selected as candidates for the next phase. The scores of these images from each TS-SOM are summed, and the highest scoring images are shown on the next iteration. The selection process is repeated on each iteration, allowing the system to learn the search criteria based on the user s feedback. Since each TS-SOM is trained with a different feature, the relevance patterns on the maps are different. If the images selected by the user form a cluster in a certain map, the scores for the images in the associated units cumulate, resulting in the map unit, and thus the associated images, having a higher score than any unit in a map where the selections are spread out. This way the relative importance of each feature in the search is automatically weighted. The PicSOM user interface is depicted in Figure. On the top are the bottom levels of the four TS-SOMs (one for each feature set) and then the images selected by the user. Below these are the 2 best-matching images returned by PicSOM. III. FEATURE DESCRIPTORS A. The MPEG-7 standard descriptors The MPEG-7 standard, ISO/IEC 5938, formally named Multimedia Content Description Interface [2], [7], [8], provides standardized descriptions of streamed or stored images or video, to be used in searching, identifying, filtering and browsing images or video in various applications. The standard defines several still image descriptors. The descriptors used in this paper are: Color Layout (CL) specifies a spatial distribution of colors. The image is divided into 8 8blocks and the dominant colors are solved for each block in the YCbCr color system. Discrete Cosine Transform is applied to the dominant colors in each channel and the DCT coefficients are used as a descriptor. Color Structure (CS) slides a structuring element over the image. The numbers of positions where the element contains each particular color are stored and used as a descriptor. Scalable Color (SC) is a 256-color histogram in HSV color space, which is encoded by a Haar transform. Edge Histogram (EH) calculates the amount of vertical, horizontal, 45 degree, 35 degree and non-directional Fig.. The PicSOM user interface. edges in 6 sub-images of the picture, resulting in a total of 8 histogram bins. Homogeneous Texture (HT) filters the image with a bank of orientation and scale tuned filters that are modeled using Gabor functions. The first and second moments of the energy in the frequency domain in the corresponding sub-bands are then used as the components of the texture descriptor. Region-based Shape (RS) utilizes a set of 35 Angular Radial Transform (ART) coefficients that are calculated within a disk centered at the center of the image s Y channel. The descriptors were calculated using the MPEG-7 Experimentation Model (XM) software versions 5.5 and 5.6. B. The Simple Shape Descriptor The Simple Shape Descriptor (SSD) was developed for surface defect description in our earlier project [9]. It consists of several simple descriptors calculated from an object s contour. The descriptors are convexity, principal axis ratio, compactness, circular variance, elliptic variance, and angle. The descriptors are not very efficient individually, but the combination of them has been shown to produce good results with low computational costs [3].

3 Fig. 2. Example images from the metal database classes. A. The image database IV. EXPERIMENTS The image database contained 24 defect images from an online metal web inspection system. The database was preclassified into 4 different classes, with each class containing from up to 65 images. Example images from each class are shown in Figure 2. All images were gray-scale with 256 gray levels, dimensions ranging from less than pixels up to over pixels. Each image was supplied with a segmentation mask, indicating the defect areas to be distinguished from the surface background. The images and the segmentation masks were provided by ABB Oy. Some example images and their segmentation masks are shown in Figure 3. B. KNN leave-one-out cross-validation The performance of the descriptors was first evaluated with the K-Nearest Neighbor leave-one-out cross-validation. All calculations used Euclidean distances and a value of 5 for K. The results in Table I show that the best descriptor is Color Structure, at an overall success rate of 63%, taken as an average weighted with the number of images in each class. The next best descriptors are Edge Histogram (49%), Homogeneous Texture (57%) and the Simple Shape Descriptor (42%). The remaining features performed considerably more poorly, with success rates less than 32%. The effects of using several descriptors in the classification were evaluated by determining the classes of the 5 nearest neighbors for each descriptor and choosing the classification according to the class with the largest total number of occurrences. Using all seven descriptors in the classification increased the performance considerably, with the overall success rate being 76%, which is 3% more than with the best individual descriptor, Color Structure. Using only the four best descriptors achieved the same overall classification rate, so the remaining descriptors were excluded from further experiments. C. CBIR performance The image retrieval performance of PicSOM was evaluated using a built-in testing system, which emulates a human user making queries and giving feedback on the retrieved images in order to find images belonging to a specific class. The results are recall and precision values for each iteration of image retrieval. is the percentage of images belonging to the desired class that have been retrieved so far. A recall of % means that all desired images have been found. Precision is the percentage of desired images from all the images retrieved so far. Precision should be higher than the a priori probability of the desired class, otherwise the system s

4 Fig. 3. Example images and their segmentation masks from the metal database. The numbers are corresponding class labels. TABLE I KNN CLASSIFICATION RESULTS Classification success rates (%) of classes avg CS HT EH SSD SC CL RS All best best retrieval performance is worse than random selection. Figure 4 shows precision/recall graphs for the metal defect database, illustrating the retrieval performance as the query progresses. The average graph shown is a result of calculating for each iteration the weighted average of the precision and recall values over all classes. All three graphs show an increase in precision at the beginning of the query, which reflects the system s ability to refine the search based on relevance feedback from the user. After this, the precision stays relatively stable, until a significant portion of the desired images has been retrieved. The easy class exhibits this behavior very clearly. Precision stays above 8% until almost % of the desired images have been retrieved, and then slopes down very abruptly. Typically the phases of the process are not so clearly defined, as is seen in the smoothly curving graphs for the difficult class and the average over all classes. Figure 5 shows recall/iteration graphs for the metal defect database. The graphs show how the system is able to retrieve the most relevant results very efficiently in the first few iterations. The last few remaining images in the desired class are typically very hard to retrieve. Since 2 images are retrieved on each iteration, and nearly all classes have between 4 and 6 images, an ideal system would achieve % recall after 8 iterations. Thus examining the system s recall values at 8 iterations gives a good idea of the system s retrieval performance. Using more iterations will improve the results, but not in proportion to the number of additional iterations. Figure 6 shows the recall values at 8 and 6 iterations for each class and as a weighted average over all classes. On the average 62% recall is achieved after 8 iterations. The differences in performance among the classes are consistent with the results of the KNN crossvalidation tests. Using the additional 8 iterations increases recall approximately 25%. V. CONCLUSIONS A content-based image retrieval system called PicSOM was applied to a database consisting of defect images obtained from a real metal web inspection system. The MPEG-7 descriptors Color Structure, Edge Histogram, and Homogeneous Texture, and our own Simple Shape Descriptor were used as features. Their goodness was determined with the KNN classifications. These four feature sets were then implemented in PicSOM. The CBIR experiments showed that the SOMbased indexing engine of PicSOM was able to retrieve similar defect images accurately and efficiently from our database. The obtained recall and precision values show good performance that is also comparable to the ones obtained with a simple KNN classifier.

5 Easy class (4) Difficult class (8) Average Precision.. Fig. 4. Precision/recall graphs using the four best features.. Fig iterations 6 iterations avg Classes s at 8 and 6 iterations using the four best features.. Easy class (4) Difficult class (8) Average Iterations Fig. 5. /iteration graphs using the four best features. ACKNOWLEDGMENTS The authors wish to thank Mr J. Pakkanen and the Pic- SOM group (J. Laaksonen, M. Koskela, E. Oja) at Helsinki University of Technology, and our industrial partner ABB Oy (J. Rauhamaa). The financial support of the Technology Development Centre of Finland (TEKES s grant 42/3) is gratefully acknowledged. REFERENCES [] Y. Rui, T. S. Huang, and S.-F. Chang, Image retrieval: Current techniques, promising directions, and open issues, J. of Visual Communication and Image Representation, vol., no., pp , 999. [2] A. Del Bimbo, Visual Information Retrieval. Morgan Kaufmann Publishers, Inc., 999. [3] J. Laaksonen, M. Koskela, S. Laakso, and E. Oja, PicSOM - contentbased image retrieval with self-organizing maps, Pattern Recognition Letters, vol. 2, no. 3-4, pp , 2. [4], Self-organising maps as a relevance feedback technique in content-based image retrieval, Pattern Analysis and Applications, vol. 4, no. 2+3, pp. 4 52, 2. [5] J. Iivarinen, J. Pakkanen, and J. Rauhamaa, Content-based image retrieval in surface inspection, in Proceedings of 7th International Conference on Control, Automation, Robotics and Vision, Singapore, Dec , pp [6] J. Pakkanen, A. Ilvesmäki, and J. Iivarinen, Defect image classification and retrieval with MPEG-7 descriptors, in Proceedings of the 3th Scandinavian Conference on Image Analysis, ser. LNCS 2749, J. Bigun and T. Gustavsson, Eds. Göteborg, Sweden: Springer-Verlag, June 29 July 2 23, pp [7] MPEG-7, MPEG-7 visual part of the experimentation model (version 9.), ISO/IEC JTC/SC29/WG N394, 2. [8], MPEG-7 multimedia content description interface part 3 visual, ISO/IEC JTC/SC29/WG W373, 2. [9] J. Iivarinen and A. Visa, An adaptive texture and shape based defect classification, in Proceedings of the 4th International Conference on Pattern Recognition, vol. I, Brisbane, Australia, Aug , pp [] T. Kohonen, Self-Organizing Maps. Berlin: Springer-Verlag, 995. [] P. Koikkalainen and E. Oja, Self-organizing hierarchical feature maps, in Proceedings of 99 International Joint Conference on Neural Networks, vol. II, San Diego, CA, 99, pp [2] B. S. Manjunath, J.-R. Ohm, V. V. Vasudevan, and A. Yamada, Color and texture descriptors, IEEE Transactions on Circuits and Systems for Video Technology, vol., no. 6, June 2. [3] J. Iivarinen, M. Peura, J. Särelä, and A. Visa, Comparison of combined shape descriptors for irregular objects, in Proceedings of the 8th British Machine Vision Conference, vol. 2, University of Essex, UK, Sept , pp