Highlight detection with application to sweet pepper localization

Transcription

1 Ref: C0168 Highlight detection with application to sweet pepper localization Rotem Mairon and Ohad Ben-Shahar, the interdisciplinary Computational Vision Laboratory (icvl), Computer Science Dept., Ben-Gurion University of the Negev, Beer-Sheva, Israel. Abstract One of the most fundamental aspects of dealing with illumination effects in computer vision is the detection of specular highlights. On one hand, these bright reflections of light may become a significant source of error to many visual analysis algorithms (e.g., edge detection, segmentation etc.). On the other hand, their unique appearance may be exploited to detect, characterize and model objects on which they form. This is particularly the case under challenging viewing conditions, where conventional features used for the detection of visual objects are often unreliable or insufficient. The detection and analysis of specular highlights are therefore advantageous for improving the performance of a wide range of purposes. In this paper, we present a novel method to detect specular highlights with a specific application to localize sweet pepper fruits in natural, cluttered scenes. Indeed, due to their physical properties, sweet pepper fruits almost always produce reflections of the light source illuminating them. We suggest a model of specular highlights based their luminance, saturation and approximately anisotropic structure. The proposed model uses no prior knowledge of lighting direction or any calibration for its estimation but rather it is based on a particular relationship between highlights and image gradients. In particular, highlights are expressed as unique signatures of gradient distributions, visually observed as local pinwheels, in the orientation map of the image. A model of these pinwheel signatures is considered at each candidate location to compute a confidence measure for this location to belong to a specular highlight. Experimental results show that the model greatly enhances the reliability of detecting specular highlights compared with simply relying on their luminance and saturation properties. Keywords: agricultural vision, specular highlight, automation, robotics, environment. 1 Introduction Theoretically, when a light source illuminates an object in a visual scene, part of the light immediately reflects back, while the remainder infiltrates the object. Of the infiltrating light, some would pass through the object and some would reflect back onto its surface and into the air. The immediately reflected light is called specular, while the light reflected after penetrating the object is called diffuse. The physical properties of the illuminated object determine the specular and diffuse components of the light reflected from it. Many common materials exhibit a mixture of both components. However, specular reflections are more characteristic of glossy or shiny objects. In a two dimensional projection of a visual scene, at viewing angles in which the reflection dominates, these reflections would often appear as bright spots of light called specular highlights (Beckmann and Spizzichino, 1963). Proceedings International Conference of Agricultural Engineering, Zurich, /7

2 Specular highlights are often regarded as nuisance for practical computer vision applications. A main reason lies in their characteristic high intensity and low saturation values, which are rendered as intense white regions in the scene image. Often, these regions hinder image processing algorithms that are based on color information and decision thresholds (e.g., segmentation and edge detection). Additional difficulties rise from the viewpoint dependent appearance of specular highlights, which interferes with image registration and subsequent image processing tasks (e.g., stereo matching and object recognition). Naturally, these properties of specular highlights have been used in order to remove them from acquired visual data and improve the performance of a wide range of purposes. Several methods used the viewpoint dependent appearance of highlights in order to detect them either by acquiring scene images from multiple views (Nayar et al., 1997; Lin et al., 2002; Lee and Bajcsy, 1992, January) or by changing the light source direction (Lin and Shum, 2001; Park and Tou 1990, June; Sato and Ikeuchi, 1994). These approaches, however, are not always applicable due to requirement to modify the general setting of the scene. Removal of specular highlights without using visual cues related to their view dependent appearance (i.e., using a single image) is more challenging. Several methods rely on an image of the diffuse component of the scene image, generated according to a reflection model and the parameters of the acquisition device (Tan and Ikeuchi, 2005; Malick et al., 2005; Malick et al., 2006; Shen and Cai, 2009). Other approaches for highlights removal analyze the distributions of image colors within a color space (Tanand Ikeuchi, 2005, June; Tan et al., 2006). These methods, however, are not capable for real-time applications and focus on the removal of specular highlights without their explicit detection. 2 Motivation While their removal may be beneficial to some applications, specular highlights may also be regarded as informative visual cues. E.g., specular highlights were used in order to detect shiny and transparent objects (Osadchy et al., 2003) and exploited as unique signals to estimate the pose of objects (Netz and Osadchy, 2011). Indeed, under challenging viewing conditions, where conventional (e.g., color, brightness) features are often unreliable or insufficient for the detection of visual objects, the unique appearance of specular highlights may be exploited to detect, characterize and model the objects on which they form. This is especially the case for the task of automatic detection of sweet pepper, which is at the focus of this work. Indeed, due to their physical properties, sweet pepper fruits almost always produce reflections of the light source illuminating them. Considering the abundance of visual information in natural, cluttered scenes of sweet pepper fruits, specular highlight could be a significant signal that can be exploited for their detection. In this paper, we present a novel method to detect specular highlights with a specific application to localize sweet pepper fruits. Its simplicity and effectiveness allows for real-time detection of specular highlights no sweet pepper fruit. Our method is based on a model of specular highlights that exploits a particular relationship between highlights and image gradients. The model uses no prior knowledge of lighting direction or any calibration for its estimation. Experimental results show that the model greatly enhances the reliability of detecting specular highlights compared with simply relying on their luminance and saturation properties. 3 The proposed method Specular reflections tend to exhibit high luminance and low saturation as well as somewhat anisotropic structure. In our method to extract specular highlights, we exploit these properties in two subsequent phases described below. Proceedings International Conference of Agricultural Engineering, Zurich, /7

3 3.1 Detection of Candidate Specular Regions Candidate specular regions are regions in which specular highlights are more likely to be detected. As in previous methods, we rely on the high luminance and low saturation values that characterize specular highlights in order to extract candidate regions (See Fig. 1). Given an input RGB image of the scene, I, we convert it into its HSV representation and combine its saturation S and luminance V channels to obtain a binary classification of pixels in candidate specular regions. Specifically, for each pixel x in I we compute the following: I x 1, 0, V x S x t otherwise (1) Where 1 indicates a pixel in a candidate specular region and 0 indicates non-candidates. The threshold value, t, was empirically chosen to be 0.8. Figure 1.d shows the resulting binary representation of candidate specular regions for a sample scene of sweet fruit pepper. (a) (b) (d) Figure 1: A sample scene image of sweet fruit pepper (a) and its luminance (b) and saturation (c) channels. In panel d, candidate specular regions are overlaid on the grayscale versin of the input image for demonstration. After combining and thresholding the luminance and saturation channels, candidate regions are further refined by removing those that are two large or non-solid. As the large amount of false positives indicates, high luminance and low saturation are not exclusive properties of specular highlights. The coarse estimation of specular regions is further refined by considering the area of each region (in terms of pixels) and its solidity. The latter is defined as the proportion of pixels in the region that are also in its convex hull. Too large or non-solid cadidates are ignored. The classification of candidate specular regions mainly serves to reduce the computational resources required for the following computation, in which candidate regions are selected as specular highlights. (c) Proceedings International Conference of Agricultural Engineering, Zurich, /7

4 3.2 A Gradient Based Model of Specular Highlights High luminance and low saturation are not exclusive properties of specular highlights (see figure 1). In order to distinguish specular highlights from among the candidates obtained by the previous phase, we rely on their characteristic anisotropic structures. To detect these structures we follow a previous method to estimate the orientation of the scene image using the eigenvectors of the second moment matrix (DelPozo and Savarese, 2007). The results is a two dimensional map encoding the estimated orientation in the range at each location, based on the grayscale image of the scene. In the obtained orientation map of the image, the anisotropic structures of specular highlights are expressed as unique signatures of gradient distributions, which can be observed as local pinwheels (see Figure 2.b). Visually, the orientation values of a pinwheel signature seem to be organized radially around its center. We use these observations and model a pinwheel signature as a group of concentric circles of increasing radii, C 1, C2,..., Cn. Each circle encodes the expected local orientation values along its perimeter. Thus, the expected local orientation values of a signature are encoded at all discrete angles and at several distances from the center of the signature (see Figure 2.c). (a) (b) (c) Figure 2: A sample close up of a sweet pepper fruit with two specular highlights (a). The highlights are expressed as unique signatures (marked by white circles) that can be visually observed as local pinwheels (b). A model of concentric circles of increasing radii is used in order to model these signatures. Each circle encodes the expected local orientation values along its perimeter (c). The binary classification of candidate specular highlights and the computed orientation map are both fed into the last stage in which we use the model of a pinwheel signature to indicate the occurence of specular highlights. The model is considered at each candidate location in the orientation map to compute a confidence measure for this location to belong to a specular highlight. For a specific candidate location, the orientation map values are compared with the model orientation values. In order do consider the circularity of oerientation values, we compute the distance between two orientations based on their complex number representation (so that shorter distance is computed for orientations that are radially proximate. E.g., the distance between 0 and 360 is 0). Thus, the distance between an estimated orientation value, and an orientation value, the model encodes is: 2 d, cos cos sin sin (2) 2 Whenever the distance between an actual orientation and its corresponding model orientation value is smaller than a threshold, T d, the orientations are considered to agree. The agreement percentage per circle in the model determines the support it provides to the central pixel belonging to a specular highlight. A circle with support larger than a threshold, T, votes positively. The accumulated votes from all circles are used to classify the central c Proceedings International Conference of Agricultural Engineering, Zurich, /7

5 pixel as either belonging to a specular highlight or not according to a third threshold, T h. Finally, an estimation of a pepper fruit size is used to place filled circles at locations that were classified as belonging to specular highlights. 4 Results and Discussion To evaluate our model, we used a dataset of 11 images of sweet pepper fruit in their natural, cluttered scenes. The images include both red and green colored fruits. However, we relied only on specular highlights as visual cues for detection. Figure 3 demonstreates the potential of our model, using two typical scene images of the dataset we used. The original input and the binary classification of candidate specular highlights are shown in columns a and b, respectively. The input to the voting process based on our model is the binary map shown in column b and the orientation estimation map computed for the greyscale version of the input image. At each pixels that was voted with significant confidence to belong to a specular highlights, a circular region was extracted. These circular regions, indicating the locations of specular highlights, are presented in column c. (a) (b) (c) Figure 3: (a) typical images of sweet pepper fruits in their natural scene. (b) Candidate specular regions obtained based on the luminance and saturation properties of specular highlights. (c) Regions from the input image that were preserved as specular highlights according to the voting process based on our model. Although the results show false detections as well as false alarms which cannot be ignored, the significance of our model is encouraging. As noticable by comparing columns b and c, the consideration of the structure of specular highlights, based on their relationship to the gradient distribution is essential. Candidate regions that were obtained relying on the luminance and saturation properties of specular highlights alone are significantly reduced while true specular highlights are preserved. The main limitations of the current model arise from its generic circular structure, which does not always correspond with the form of specular highlights. In addition, as can be noticed in Figure 2.b, the expected orientation values that the model encodes may change according to the pose of the fruits in the scene. Proceedings International Conference of Agricultural Engineering, Zurich, /7

6 5 Conclusions High luminance and low saturation are well known properties of specular highlights. However, as shown above, these properties may be common to other elements in the scene. In this work, we pointed at a unique relationship between highlights and image gradients as a possible signal for the detection of specular highlights. Using a very simple model, we have demonstrated the significance of this relationship in distinguishing highlights from among regions of high luminance and low saturation. More importantly, we showed the potential of using specular highlights as significant signals for automatic detection of sweet pepper fruits. Based on this potential, we believe that incorporating an active light source for robotic harvesting of sweet-pepper fruits can benefit automatic detection based on specular highlights. We hope that the simplicity of our model and the encouraging preliminary results would motivate its further development. 6 Acknowledgements This research was funded in part by the European Commission in the 7th Framework Programme (CROPS GA no ), partially supported by the Helmsley Charitable Trust through the Agricultural, Biological and Cognitive Robotics Center of Ben-Gurion University of the Negev and the Israeli Ministry of Agriculture. The authors also thank the generous support of the Frankel fund.. 7 References Tan, R. T., & Ikeuchi, K. (2005). Separating reflection components of textured surfaces using a single image. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27(2), Shen, H. L., & Cai, Q. Y. (2009). Simple and efficient method for specularity removal in an image. Applied optics, 48(14), Mallick, S. P., Zickler, T. E., Kriegman, D., & Belhumeur, P. N. (2005, June). Beyond lambert: Reconstructing specular surfaces using color. In Computer Vision and Pattern Recognition, CVPR IEEE Computer Society Conference on (Vol. 2, pp ). Ieee. Mallick, S. P., Zickler, T., Belhumeur, P. N., & Kriegman, D. J. (2006). Specularity removal in images and videos: A PDE approach. In Computer Vision ECCV 2006 (pp ). Springer Berlin Heidelberg. Shen, H. L., & Cai, Q. Y. (2009). Simple and efficient method for specularity removal in an image. Applied optics, 48(14), Tan, R. T., & Ikeuchi, K. (2005, June). Reflection components decomposition of textured surfaces using linear basis functions. In Computer Vision and Pattern Recognition, CVPR IEEE Computer Society Conference on (Vol. 1, pp ). IEEE. Tan, P., Quan, L., & Lin, S. (2006). Separation of highlight reflections on textured surfaces. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on (Vol. 2, pp ). IEEE. Beckmann, P., & Spizzichino, A. (1987). The scattering of electromagnetic waves from rough surfaces. Norwood, MA, Artech House, Inc., 1987, 511 p., 1. Proceedings International Conference of Agricultural Engineering, Zurich, /7

7 Osadchy, M., Jacobs, D., & Ramamoorthi, R. (2003, October). Using specularities for recognition. In Computer Vision, Proceedings. Ninth IEEE International Conference on (pp ). IEEE. Netz, A., & Osadchy, M. (2011, June). Using specular highlights as pose invariant features for 2D-3D pose estimation. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp ). IEEE. DelPozo, A., & Savarese, S. (2007, June). Detecting specular surfaces on natural images. In Computer Vision and Pattern Recognition, CVPR'07. IEEE Conference on (pp. 1-8). IEEE. Lee, S. W., & Bajcsy, R. (1992, January). Detection of specularity using color and multiple views. In Computer Vision ECCV'92 (pp ). Springer Berlin Heidelberg. Lin, S., Li, Y., Kang, S. B., Tong, X., & Shum, H. Y. (2002). Diffuse-specular separation and depth recovery from image sequences. In Computer Vision ECCV 2002 (pp ). Springer Berlin Heidelberg. Nayar, S. K., Fang, X. S., & Boult, T. (1997). Separation of reflection components using color and polarization. International Journal of Computer Vision, 21(3), Lin, S., & Shum, H. Y. (2001). Separation of diffuse and specular reflection in color images. In Computer Vision and Pattern Recognition, CVPR Proceedings of the 2001 IEEE Computer Society Conference on (Vol. 1, pp. I-341). IEEE. Sato, Y., & Ikeuchi, K. (1994). Temporal-color space analysis of reflection. JOSA A, 11(11), Park, J. S., & Tou, J. T. (1990, June). Highlight separation and surface orientations for 3-D specular objects. In Pattern Recognition, Proceedings., 10th International Conference on (Vol. 1, pp ). IEEE. Proceedings International Conference of Agricultural Engineering, Zurich, /7