Framework for industrial visual surface inspections Olli Silvén and Matti Niskanen Machine Vision Group, Infotech Oulu, University of Oulu, Finland

Transcription

1 Framework for industrial visual surface inspections Olli Silvén and Matti Niskanen Machine Vision Group, Infotech Oulu, University of Oulu, Finland ABSTRACT A key problem in using automatic visual surface inspection in industry is training and tuning the systems to perform in a desired manner. This may take from minutes up to a year after installation, and can be a major cost. Based on our experiences the training issues need to be taken into account from the very beginning of system design. In this presentation we consider approaches for visual surface inspection and system training. We advocate using a nonsupervised learning based visual training method. Keywords: visual inspection, defect detection, learning, training, tuning 1. INTRODUCTION Visual inspection systems are now routine instruments on electronics manufacturing lines, in paper and steel mills, ceramics and wood products industries, just to name a few. Most of the inspection tasks are very specialized resulting in systems that employ very different imaging arrangements and methodological solutions. An unfortunate outcome is that experience from solving one visual inspection problem is difficult to transform into expertise in another. As system implementers frequently learn, the effort needed in acquiring the necessary application understanding can be tremendous. For example, the problem of checking solder joints on a printed circuit board appears to have similarities with detecting and recognizing defects from wooden slabs: in both applications blobs that are somehow detected are classified into various categories. However, in system implementations domain specific knowledge of the tasks needs to be built in. This often amounts to the design of specialized features and even hand-coded analysis procedures that are useless in another application, or even with a variant of the original one. At minimum, domain specific knowledge is acquired by training an inspection system using samples from the application. Based on our experience the way this is carried out may have a substantial impact on the whole system and its usability. We have witnessed and participated in several failed and successful attempts to develop industrial solutions for various applications, including 2-D inspection of printed wiring, 3-D inspection of steel strip, sorting of food particles, grading of lumber boards 2, and characterization of paper 4. The experience acquired underlines the importance of effortless and easy to understand means for tuning the systems, even at the cost of sacrificing some accuracy. In this paper we present a methodological framework that avoids manual selection of samples and enables tuning class boundaries without expertise on the internal parameters of the system. The key ideas are in employing non-segmenting defect detection in which the image is divided into non-overlapping regions for which features are calculated, and non-supervised clustering of regions based on their features, while the boundaries between defects and background as well as boundaries between defect categories are determined by human operator using a 2-dimensional projection of the feature space. The last part in the approach is remarkably similar to the user interface of an early successful bi-chromatic electronic coffee bean sorting machine. However, here we consider wood and other similar surface inspection applications.

2 2. SURFACE INSPECTION SYSTEM TRAINING AND TUNING Surprisingly few industrial surface inspection systems employ the sample based training approach, while many successful ones do completely without samples. Instead, they often rely on manual tuning of numerical detection and classification parameters as explained by Figure 1. Interestingly, industrial inspection systems using this kind of primitive methodology are often considered to be fairly accurate and satisfy the needs of the applications in question. Fig. 1. Typical organization of a visual surface inspection system. An explanation is that on a production line most changes that are made to a visual inspection system are small adjustments, e.g., the boundary between two defect classes or sound material is slightly modified. In this kind of situation changing a parameter value requires much less effort from a production person than retraining using samples. A shortcoming is that the adjustments of numerical values may not have simple relationships with the visual results, so the accuracy depends greatly on the experience of the operator. Fuzzy classifiers have been proposed to replace numerical parameters with linguistic ones, and they may have contribution in guiding the steps by which one takes knowledge in a linguistic form and casts it into discriminant functions. 3. Unfortunately, this doesn t solve for the actual problem. The combined visual classifier and user interface described in this paper provides for the necessary connection. 3. SELECTION OF SAMPLES Manual selection and labeling of samples is necessary at least at an early stage of any inspection methodology development to find discriminating features. This process is illustrated in Figure 2. Fig. 2. Obtaining labeled samples for training is a two stage process. Figure 3 shows the two basic approaches to sample region selection from images. Pixel based selection (b) assumes that a human is able to correctly pinpoint pixels belonging to defects in the image and pixels that are from sound background. In region based selection (c) she roughly selects the regions that contain a defect or defects, but also have a substantial portion of sound background. We advocate region based selection, because the characteristics of the transition regions from background to defect may be important. For instance, the grain around a suspected defect (d) on a lumber board helps in discriminating loose bark particles from small knots. background transition defect (a) (b) (c) (d) Fig. 3. a) a defect (b) pixel based selection (c) region based selection (d) example of a knot.

3 However, as a subjective procedure manual selection may result in samples that are not representative of the detections of inspection algorithms. If this is the case, no training procedure or classifier will result in good defect recognition performance. We may conclude that training should be done using actual detection results as samples. Many inspection systems use fast defect flagging algorithms as their front end, often based on adaptive thresholding, to cut down the amount of data subjected to further analysis. The samples could be selected from among these detections. However, the simplistic fast defect flagging often results in partial detections and error escapes. If low error escape rates are pursued, the detection sensitivity needs to be lowered, resulting in more alarms from sound regions. Also, changing the detection sensitivity on-line may require re-training the system from new samples. This is a serious problem even with the approach presented in Fig. 1, as a change in detection parameters may create a domino effect that propagates to the classifier. Figure 4 shows how minor changes in detection parameters may change the flaggings. Neither of the results have good correspondence with any selections shown in Figure 3. Clearly, better detection approaches are needed. (a) (b) Fig. 4. Impact of changing defect detection sensitivity: (a) underdetection (b) overdetection. Our approach is a compromise that eliminates the manual selection of samples. Both in training and in defect detection the image is divided into non-overlapping, e.g., 32-by-32 pixel sub-images, as shown for wooden boards in Figure 5. Features are calculated for the sub-images and defect detection can done using a classifier. We call this a nonsegmenting method as no meaningful regions are extracted using pixel level image analysis. Fig. 5. Wood images divided into 32-by-32 pixel rectangles.

4 3.1 FEATURE EXTRACTION Implementation cost has been the main argument against selecting other than crude techniques for the defect detection stages of industrial surface inspection systems. However, the complexity of the feature extraction algorithms is actually limited by the number of accesses that can be done to image data at real-time speeds. Based on application experience, we know that there are classes of features that combine low computational complexity and high descriptive power. Interestingly, the best ones we have found are based on distributions. For relatively non-textured materials, such as softwood, color histogram based features perform very well 2 and require only one access per pixel to the image data. In other words, the cost is in the same range as that of thresholding techniques. For textured materials, such as paper and textiles, the LBP Local Binary Patterns 1 method is a promising one due to its computational efficiency and excellent discrimination accuracy. Due to regular addressing patterns and fixed sized regions, the non-segmenting defect detection approach provides a good basis for implementing these techniques efficiently, both in hardware and software. For example, LBP features can be calculated by software at rates faster than 10Mpixels/s LABELLING OF SAMPLES Labeling samples is a rather error prone task that may seriously limit the accuracy of automatic inspection. An illustrating example is in Figure 6 that contains examples of sound and dry knots from pinewood boards. Clearly, both within and between class variations have wide range, making human decisions on categories very difficult. Furthermore, a real world wood inspection system is unlikely to produce well detected defects such as below, resulting in training difficulties described earlier. Sound knots Dry knots Fig. 6. Examples of sound and dry knots. An assortment of pixel wood regions is shown in Figure 7. The difficulty of consistently labeling these samples encourages to consider non-supervised methods to cluster similar samples together. If the features are reasonably good, the result should enable us to easily discriminate between sound wood and other regions, and we can assign labels for clusters of similar regions, rather than for individual samples. Fig. 7. A palette of 32-by-32 pixel regions of pinewood.

5 For non-supervised clustering and combined visualization a number of methods exist of which Self-Organizing Maps 7 (SOMs) has been found to be a good compromise 6. Figure 8 shows regions of pinewood on a 2-dimensional SOM, organized based on their color histogram features. Each rectangular region is a node to which similar samples are clustered. At this point we notice that using small rectangular regions in the defect detection stage simplifies visualization, and we can actually draw a boundary between sound wood and other regions on the map. The SOM can then be used as a classifier to detect regions of interest. Fig. 8. Regions of pinewood on a Self-Organizing Map. The regions of interest are a starting point for further analysis that is application dependent. In our wood inspection solution, connected components analysis (CCA) is performed for the detected regions and additional features are extracted. As the volume of data has reduced to a small fraction of the original, computational complexity is no longer a major issue. Typically, the regions formed via CCA contain knots and other defects. 5. COMBINED CLASSIFIER AND USER INTERFACE The industrial users prefer inspection systems that are never retrained due to the painstaking characteristics of this task. Instead, all changes should be carried out by tuning, and this process should be quick and simple to understand. For instance, the appearances of dry and sound knots on lumber boards depend on the growing conditions of the trees, and it is frequently necessary to adjust the boundaries between knot categories. The approach shown in Figure 9 combines a visual classifier and user interface. The connection between the appearance of defects and classification parameters is direct, so the boundaries between categories can be adjusted simply on screen. The same applies to the detection stage (Figure 8), too. Fig. 9. Principle of visual classifier and adjustment of class boundary between sound and dry knots (dashed line).

6 So far this approach has been exploited in wood and steel inspection systems with very good success. More applications are on their way. Interestingly, combined visual user interface and classifier is not a completely novel concept. Figure 10 presents the user interface principle of a bi-chromatic coffee bean sorter from 1950's. As the beans are dropped through the measurement channel, red and green color signals are generated and fed to the horizontal and vertical deflection units of the cathode ray tube. A third photosensor is mounted in front of the tube and generates a reject signal to a pneumatic ejector whenever it detects the beam on the display. The accept/reject boundary is tuned by modifying the paper mask between the tube and the sensor using scissors and adhesive tape. This user interface principle has evolved into a modern computerized form in a tri-chromatic sorter 8. Our main contribution in that respect appears to be in employing methods to deal with multidimensional data product flow red color filter photo cell vertical deflection bright spot opaque mask photo cell green color filter horizontal deflection cathode ray tube to reject control Fig. 10. A coffee bean sorter from 1950 s using a combined visual user interface and classifier. 6. SUMMARY We have concentrated on the problem of training and tuning industrial visual surface inspection systems. Nonsupervised learning based techniques that support combined visual user interfaces and classifiers have been found to be a very promising approach. In principle, these techniques can be introduced into many already existing systems, but the biggest benefits are received from new designs. REFERENCES 1. T. Ojala, M. Pietikäinen, and T. Mäenpää, Multiresolution gray-scale and rotation invariant texture classification with Local Binary Patterns, IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7): , O. Silven, M. Niskanen, and H. Kauppinen, Wood inspection with non-supervised clustering, Machine Vision and Applications, 2003, in press. 3. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classification. John Wiley & sons, New York, second edition, M. Turtinen, M. Pietikäinen, O. Silven, T. Mäenpää, and M. Niskanen, Texture-based paper characterization using non-supervised clustering, Proc. 6th International Conference on Quality Control by Artificial Vision (QCAV 2003), Gatlinburg, Tennessee, T. Ojala, T. Mäenpää, J. Viertola, J. Kyllönen, and M. Pietikäinen, Empirical evaluation of MPEG-7 texture descriptors with a large-scale experiment, Proc. 2nd International Workshop on Texture Analysis and Synthesis, Copenhagen, pp , M. Niskanen, and O. Silvén, Comparison of dimensionality reduction methods for wood surface inspection, Proc. 6th International Conference on Quality Control by Artificial Vision (QCAV 2003), Gatlinburg, Tennessee, T. Kohonen, Self-organizing Maps, Springer-Verlag, Berlin, Elexso AG,