DEFECT IMAGE CLASSIFICATION WITH MPEG-7 DESCRIPTORS. Antti Ilvesmäki and Jukka Iivarinen

Transcription

1 DEFECT IMAGE CLASSIFICATION WITH MPEG-7 DESCRIPTORS Antti Ilvesmäki and Jukka Iivarinen Helsinki University of Technology Laboratory of Computer and Information Science P.O. Box 5400, FIN HUT, Finland In this paper the visual content descriptors defined by the MPEG-7 standard are applied to defect image retrieval. The applicability of the descriptors is evaluated by extracting MPEG-7 feature vectors for a pre-classified defect database of approx paper surface defect images. The MPEG-7 feature vectors are then used to perform a KNN classification of the images. The results are generally satisfactory, especially the Color Structure and Homogeneous Texture descriptors seem to perform well. 1 INTRODUCTION An increase in the amount of image data that has to be stored, managed and searched has spurred the research and development of systems that automatically compare and classify images on the basis of their content. Technological advances, such as the proliferation of digital cameras, inexpensive disk space and fast network connections have made huge databases of digital images possible, but finding the right image from such a database is a complex problem. Content-based image retrieval (CBIR) systems [10, 1] aim to solve this problem by extracting features that describe the relevant aspects of the images, such as color or shape in a compact manner. Unlike the original images, the features can be fairly easily compared and meaningfully classified because they represent a higher level of abstraction. The potential application area of CBIR systems is vast: face recognition, fingerprint authentication, Internet search engines, trademark databases and satellite image analysis are a few examples of areas where CBIR can be used. Classification of defects occurring in an industrial production process and controlling the production process accordingly is another problem that can be approached using CBIR. This paper examines the suitability of features defined by the visual descriptors of the MPEG-7 standard for classifying paper web defect images. It extends on previous research done in the Laboratory of Computer and Information Science at Helsinki University of Technology in the fields of defect image retrieval [4, 5] and MPEG-7 descriptors [6]. This paper describes preliminary experiments with MPEG-7 descriptors using a KNN classifier and a small, pre-classified database. We intend to later use the descriptors with a generic CBIR system, PicSOM, and expect that the descriptors will perform similarly with larger defect image databases. The classification of paper defect images is a real-world problem that would benefit from a fast, efficient CBIR system capable of evaluating the images. A single paper web inspection system may produce hundreds or even thousands of defect images in a day, but only the most severe ones require immediate action. Sorting through the images manually would be extremely time-consuming and expensive.

2 2 MPEG-7 STANDARD The MPEG-7 standard [3], ISO/IEC 15938, formally named Multimedia Content Description Interface, defines a comprehensive, standardized set of audiovisual description tools. The scope of the standard encompasses both human users and automatic systems that process audiovisual information. The standard is not aimed at any one application in particular, instead it supports a broad range of descriptors for different types of multimedia information. MPEG-7 is expected to find use in digital libraries, broadcast, journalism, surveillance and numerous other areas. The aim of the standard is to facilitate quality access to content, which implies efficient storage, identification, filtering, searching and retrieval of media. In addition to the still image descriptors demonstrated in this paper, the standard includes various descriptors for video and audio content. Furthermore, the standard enables the definition of new types of content descriptors. The existing MPEG-7 descriptors provide a framework for both high level semantic abstractions and low level abstractions such as shape, color and movement. For paper defect image retrieval only the low level visual descriptors that can be extracted without human interaction are of interest. 2.1 MPEG-7 visual descriptors The feature vectors corresponding to the MPEG-7 visual descriptors were extracted with the MPEG-7 Experimentation Model (XM) software version 5.5. A total of six different MPEG-7 visual descriptors could be extracted and used for KNN classification. Dominant Color descriptor represents the most dominant colors in the image. The original MPEG-7 XM descriptor is variable-sized, but to simplify the distance calculation only the two most dominant colors were used. If only one dominant color was found it was duplicated. Scalable Color descriptor is a 256-bin color histogram in HSV color space, which is encoded by a Haar transform. Color Layout descriptor specifies a spatial distribution of colors. The image is divided into 8 8 blocks and the dominant colors are solved for each block in the YCbCr color system. Discrete Cosine Transform is applied to the dominant colors in each channel and the DCT coefficients are used as a descriptor. Color Structure descriptor captures both color content and the structure of this content. It does this by means of a structuring element that is slid over the image. The numbers of positions where the element contains each particular color is recorded and used as a descriptor. As a result, the descriptor can differentiate between images that contain the same amount of a given color but the color is structured differently. Edge Histogram descriptor represents the spatial distribution of five types of edges in 16 sub-images. The edge types are vertical, horizontal, 45 degree, 135 degree and nondirectional, and they are calculated by using 2x2-sized edge detectors for the luminance of the pixels. A local edge histogram with five bins is generated for each sub-image, resulting in a total of 80 histogram bins. Homogeneous Texture descriptor filters the image with a bank of orientation and scale tuned filters that are modeled using Gabor functions. The first and second moments of the

3 energy in the frequency domain in the corresponding sub-bands are then used as the components of the texture descriptor. For more in-depth explanations and definitions of the different descriptors, their intended range of use and the precise algorithms see [7, 2, 9, 8]. The MPEG-7 standard also defines similarity matching functions for each descriptor. However, for most of the descriptors the similarity matching is simply defined as L1-norm (with minor variations in some cases). In the KNN classification experiments in this paper euclidean distance calculations are used to compare the feature vectors. Some experiments were also done using L1-norm but the difference in performance was insignificant. In addition to the descriptors listed above, the Texture Browsing, Contour-Based Shape and Region-Based Shape descriptors defined by MPEG-7 could also be of use in classifying paper defect images, but they were not tested due to practical problems. The Contour-Based Shape descriptor produces vectors with varying amounts of components and uses a very different and unique method for similarity matching. The implementation of the descriptor would have required a considerable additional effort and was not considered worthwhile for the moment. XM 5.5 had problems calculating the Texture Browsing descriptor correctly. Because the Homogeneous Texture descriptor represents essentially the same qualities of an image (directionality, coarseness and regularity) but with more precision, the issues with Texture Browsing were not further examined and the descriptor was omitted. The Region Shape descriptor could not be extracted at all, since it relies on a mask image to indicate the shape of the defect and these masks were not available for the database. 3 EXPERIMENTS 3.1 Feature extraction A pre-classified database of approx grayscale images was used to test the descriptors. The database, supplied by ABB Oy, contained defect images of varying size and type (spots, holes, wrinkles, streaks, tears, slime residues etc.) from a real, online process. No segmentation masks were available for this database, so the feature extraction was done on the whole image. The results would presumably be somewhat better if the images had segmentation masks to differentiate the actual defects from the intact paper. The XML output mode of XM was utilized to extract the features defined by the MPEG-7 visual descriptors for the database. Because the defect images are gray scale bitmaps, some of the descriptors produced feature vectors with a lot of redundant color information. The Dominant Color descriptor uses three components for each dominant color. As only one component is needed to represent the shades of gray the three components were replaced with a single one in order to speed up the calculations. Another consequence of the images containing only shades of gray was that the feature vectors calculated by the histogram-based Color Structure descriptor had many components that were systematically zero for all defect images. Such components were left out before the vectors were used.

4 Figure 1: Example images from the different defect classes. The images have been cropped and resized to make them fit on the page. Notice how classes 11 and 12 look almost identical. 3.2 KNN cross-validation of the database The database had been pre-classified into 14 distinct classes, with a 100 images in 12 of the classes, 76 images in class number 11 and 32 images in class number 12, adding up to a total of 1308 images. Example images from each class are depicted in fig. 1. The classes are based on the cause and type of a defect, and can therefore contain images that are visually dissimilar in many aspects. On the other hand, some classes are very hard to tell apart. This makes their classification with general-purpose visual descriptors an extremely challenging problem. Even a human cannot always differentiate between the images. For example, class 7, oil spots, contains many images that are practically impossible distinguish from class 9, clean holes. The performance of the descriptors with the pre-classified database was evaluated by performing a leave-one-out k-nearest neighbor (KNN) cross-validatory classification. The functions provided by SOM Toolbox 1 for Matlab were utilized for this task. Euclidean distance and K=5 was chosen for all calculations. Some experiments were also performed using L1-norm as the distance, but no significant difference in the performance was detected. Each classification was performed nine times and the mean was used as a result. Each descriptor was also tested with the different normalization algorithms provided by SOM Toolbox. On average, normalization was found to have a very small effect on the results. Color Structure was the only case where using normalization was noticeably beneficial: using discrete histogram normalization improved the results about 5%. It is clear that some of the descriptors perform clearly better than others. Homogeneous Texture and Color Structure make the least errors in classifying the images, with Edge Histogram and Color Layout following. Scalable Color and Dominant Color perform quite appallingly, as was expected. No descriptor is the best in all classes. 1

5 Figure 2: Classification results for all the descriptors and classes When the results for the different classes in fig. 2 are examined, we see that the descriptors seem to be working as they should. For example, the color descriptors are very good at distinguishing class 1, which contains only dark defects. The edge histogram works very well with class 10, which contains vertical stripes. Class 4, which is totally different from all the other classes in almost all aspects, is recognized reasonably well by all descriptors. Classes 11 and 12 are recognized miserably because of several reasons: the size of the classes is smaller than the others, they are very similar with each other and many images in them also look a lot like the images in classes 3 and 5. The outcome of all this is that their performance is sometimes worse than random guessing. A simple KNN combination of all the descriptors was also used. This was done by retrieving the classes of the five nearest images for each descriptor and the winning class was determined by counting the total occurrences of each class. This method was able to perform better than any single descriptor alone. The results are in fig. 3. Using a combination of the best three descriptors, Homogeneous Texture, Color Structure and Edge Histogram, the average performance was slightly better. To get a better understanding of the errors that are made in classifying the images and to see which classes are confused with which, a table showing the different classification percentages with the best three descriptors (Homogeneous Texture, Color Structure, and Edge Histogram) was calculated (table 1). A row in the table represents a class of images, and the distribution of percentages on the row shows how the KNN classifier identified the images in the class. We can see that class 2 is frequently identified as class 14, and that class 3 is often mistaken for class 13. The descriptors seem to have very hard time in differentiating class 5 from classes 11 and 14. Class 9 is sometimes erroneously identified for class 7. Class 11 is usually mistakenly identified as class 5, and class 12 is frequently classified as 13 or 3. We can see that the classes that are being confused with each other are indeed often very similar in many aspects.

6 Figure 3: Results of a KNN combination of all the descriptors and the combination of Color Structure, Edge Histogram and Homogeneous Texture. 4 CONCLUSIONS In this paper six different MPEG-7 visual descriptors were applied to paper defect image classification using the MPEG-7 experimentation Model software and a k-nearest neighbor classifier. The experiments conducted suggest that the descriptors, especially Homogeneous Texture and Color Structure, can be successfully used for defect image classification. Bearing in mind that the KNN experiments were done without segmentation masks and that the images can be difficult to differentiate even for a human the results are quite satisfactory with the exception of classes 11 and 12. ACKNOWLEDGMENTS The authors wish to thank the PicSOM group (J. Laaksonen, M. Koskela, S. Laakso, E. Oja) at Helsinki University of Technology and our industrial partner ABB Oy (J. Rauhamaa). The financial support of the Technology Development Centre of Finland (TEKES s grant 40397/01) and the Academy of Finland (project New Information Processing Principles, 44886) is gratefully acknowledged. REFERENCES [1] Alberto Del Bimbo. Visual Information Retrieval. Morgan Kaufmann Publishers, Inc., [2] Miroslaw Bober. MPEG-7 visual shape descriptors. IEEE Transactions on Circuits and Systems for Video Technology, 11(6), June [3] Shih-Fu Chang, Thomas Sikora, and Atul Puri. Overview of the MPEG-7 standard. IEEE Transactions on Circuits and Systems for Video Technology, 11(6), June 2001.

7 Table 1: Classification percentages for the different image classes with the best three descriptors (Homogeneous Texture, Color Structure, and Edge Histogram). A row represents the images of one class and the distribution of percentages on a row shows how the images were classified. classes [4] Jukka Iivarinen. Texture Segmentation and Shape Classification with Histogram Techniques and Self-Organizing Maps. Acta Polytechnica Scandinavica, Mathematics, Computing and Management in Engineering Series No. 95. D.Sc.(Tech.) thesis, Espoo, [5] Jukka Iivarinen and Jussi Pakkanen. Content-based retrieval of defect images. In Proceedings of Advanced Concepts for Intelligent Vision Systems, pages 62 67, Ghent, Belgium, September [6] Markus Koskela, Jorma Laaksonen, and Erkki Oja. Self-organizing image retrieval with MPEG-7 descriptors. In Proceedings of Infotech Oulu International Conference on Information Retrieval, Oulu, Finland, September [7] B.S Manjuanth, Jens-Rainer Ohm, Vinod V. Vasudevan, and Akio Yamada. Color and texture descriptors. IEEE Transactions on Circuits and Systems for Video Technology, 11(6), June [8] MPEG-7. MPEG-7 multimedia content description interface part 3 visual. ISO/IEC JTC1/SC29/WG11 W3703, [9] MPEG-7. MPEG-7 visual part of the experimentation model (version 9.0). ISO/IEC JTC1/SC29/WG11 N3914, [10] Yong Rui, Thomas S. Huang, and Shih-Fu Chang. Image retrieval: Current techniques, promising directions, and open issues. Journal of Visual Communication and Image Representation, 10(1):39 62, 1999.