PHOTOMETRIC STEREO FOR NON-LAMBERTIAN SURFACES USING COLOR INFORMATION

Transcription

1 PHOTOMETRIC STEREO FOR NON-LAMBERTIAN SURFACES USING COLOR INFORMATION KARSTEN SCHLÜNS Fachgebiet Computer Vision, Institut für Technische Informatik Technische Universität Berlin, Franklinstr. 28/29, D-1587 Berlin and OLIVER WITTIG Fachgebiet Computer Graphics, Institut für Technische Informatik Technische Universität Berlin, Franklinstr. 28/29, D-1587 Berlin ABSTRACT One robust method to reconstruct shape is photometric stereo (PMS), which reconstructs surface orientation using the Lambertian reflection properties of the surface material. To increase the applicability to non-lambertian surfaces, we extend this method using a twostage process without introducing additional light sources or assuming a known micro facet distribution. In the first step, the glossy reflection is discarded, taking the dichromatic reflection model as a basis. The introduced spherical chromaticity space has been found as a good tool for deriving the matte vector. The method is applied to multi-colored objects. The simulations use spectral distributions of real data. In the second step, we apply a conventional PMS to the derived matte images. 1. Introduction The reconstruction of visible surface orientations is often useful to understand the threedimensional structure of a scene captured in one or more images with respect to the camera. Many methods exist to solve this task using different kinds of object and environment features. One of these features is the reflection characteristic of the surface material. A general formulation of the shape from shading problem was given by Horn 3. The solution using a general reflectance map needs to solve a system of differential equations. The advantage of this method is that from only one image the shape of an object with known reflectance can be recovered. The main drawback is that the mathematical tools used, constrain the object shape to be smooth. A method that is independent of the object geometry was developed by Woodham 14. The photometric stereo method (PMS) recovers shape by illuminating a scene with three distant point light sources and a constant camera position.

2 2. PMS Methods for Non-Lambertian Surfaces A PMS method for the reflectivity extreme of perfect specularity was developed by Ikeuchi 4. This PMS uses distributed light sources. The light comes from a Lambertian surface illuminated with linear lamps (fluorescent tubes). The reflectance maps are built with knowledge of the geometry of the lamps, the size of the Lambertian surface and the assumption of a perfect specular object. Coleman and Jain 1 extended Woodham's PMS by introducing a fourth light source. A theory of photometric stereo for analyzing a larger class of non-lambertian objects under point illumination was developed by Tagare and defigueiredo 13. They show the uniqueness and the completeness of PMS using their m-lobed reflectance maps. The authors give some results for a synthetic sphere using the reflectance maps of the Torrance-Sparrow model. In 12 they used eight light sources to recover the shape of real objects. 3. The Two-Stage Process Until now, all reviewed methods neglected the spectral component of the scene radiance to handle non-lambertian reflection. The idea to detect glossy regions using a spectral signal is motivated by the observation, that highlights are mostly influenced by the illumination color. Furthermore, there exist methods to detect/eliminate highlights in color images, for example 2, 5, 8. One way is to detect highlights and to perform PMS in the rest of the image. This is interesting only in images with small highlighted parts. Our approach is to combine the classical PMS with such a method without omitting the highlighted regions The Dichromatic Reflection Model To apply spectral reflection features to the shape recognition task, we have to choose an adequate reflection model: An additive composition of diffuse and glossy reflection describes many materials in a good way. The glossy reflection is caused by the interaction between the environment and the surface of the object. This class of reflection is called interface or surface reflection. Another part of the light penetrates the interface and enters the material. If the pigments in the interior are distributed randomly, no preference light direction can be perceived. This type of reflection is called body reflection. Both types of reflection depend on the spectral and the geometrical properties of the object. The spectral dependence is described by the wavelength λ. The geometrical dependence can be described by the surface normal N, the light direction S and the direction V to the viewpoint. L(λ, N, S, V) = L b (λ, N, S, V) + L s (λ, N, S, V)

3 This model holds for the description of inhomogeneous, dielectric, opaque surfaces. Assuming an independence of the spectral and geometrical properties the following simplifying separation may be used 11 : L α (λ, N, S, V) = c α (λ) m α (N, S, V), with m α (N, S, V) and α = b, s c α is the spectral power distribution of the reflected light, which is the product of the spectral power distribution of the light and the spectral reflectance of the surface. m α is a geometrical scaling factor. m b is modeled by the Lambertian cosine law. Several possible models for m s are known, for example: the Phong model, the Pentland model or the Torrance-Sparrow model, cp. 9. Assuming three sensors being sensitive in the spectral channels which are characterized by the color names red, green and blue, the sensor response in vector notation is: s x,r s x,g = s x,b c x,sr c x,sg c x,br m x,s (N, S, V) + c x,bg m x,b (N, S, V) resp., c x,sb c x,bb s x = c x,s m x,s (N, S, V) + c x,b m x,b (N, S, V). This equation defines the dichromatic plane (DCP) in a coordinate system spanned by the primary colors (sensor space), see Figure 1. B c x,s c x,b G R Fig. 1. A dichromatic plane defined in terms of c x,s and c x,b. Pixels belonging to pure body reflection lie on a straight line defined by c x,b (matte axis). The position on this line depends on the photometric properties of the scene location corresponding to the pixels. The structure of the cluster lying in the DCP is discussed in detail by Novak and Shafer 9.

4 3.2. Highlight Removal Using the Karhunen-Loève-Transform It is possible to generate decomposed (intrinsic) images containing the separated body and interface reflection components using the dichromatic reflection model 5. The aim of this work was to achieve a geometry independent segmentation. Our aim is to use the body reflection image in cooperation with a classical PMS, as applied to our PMS extension to non-static scenes 1. To yield the intrinsic images for a scene with only one surface material, the following procedure 5 is possible: DCP-normal estimation by means of the Karhunen-Loève transform (KLT), fitting a convex hull to the rotated (two-dimensional) data using a recursive line splitting method, rough classification of the pixels into specular and matte ones, for each class recalculation of the KLT, estimation of the vectors c s and c b by means of the eigenvector with the greatest eigenvalue Highlight Removal Using a Direct Approach Our implementation of this algorithm yields quite good results. The KLT is an elegant utility to produce uncorrelated features in unknown feature clusters. The following disadvantages concerning PMS have motivated us to find a more direct way to build a matte image: The above algorithm needs complex computations and is not suitable for multi material extensions. Because we know that the cluster is approximately a plane passing through the origin of the coordinate system, the KLT is not specific enough to produce high quality results. As a result, we have introduced a two-dimensional parameter space, which will be used to find the base vectors spanning the DCP The spherical chromaticity space It is sufficient that the space is independent of the intensity. Such color components are in general called chromaticities. We suggest using the slant/tilt angles of the spherical coordinates to represent the chromaticities. Therefore, we will call this the Spherical Chromaticity Space (sc-space). This representation reduces the dimension of the search space from three, in the case of a RGB histogram, to two dimensions, in the case of the sc-space, dropping the intensity component. The transformation from RGB to the sc-space is calculated, as follows: C = ( R G B ) T is a color vector in RGB-space. This vector is transformed to sc-space: t C = t(c) =( ϑ(c) ϕ(c) ) T, with

5 ϑ(c) = tan -1 R 2 +G 2 and ϕ(c) = tan -1 G R. B The vectors in the RGB-space c s and c b are represented in the sc-space as vectors t(c s ) and t(c b ), respectively Properties Consider any color vector C of the RGB-space. C is a member of the following set M: M = { v v = ν c s + µ c b, ν, µ }. If the vectors t(c s ) and t(c b ) are understood as being diagonal corners then they define a rectangle in sc-space. All vectors t(c) fall in this rectangle. Consider f(µ) = c s + µ c b. Then, the curve t(f(µ)) is a monotonic function. Consider g(ν) = c b + ν c s. Then, the curve t(g(ν)) is a monotonic function, too Finding the body reflection vector These properties tell us, that t(c b ) is one of the end points of the curve segment in scspace. Fitting a bounding box by minmax testing, limits the number of possible points to two. If c s is known, then c b can be found directly. In the case of unknown c s the number of vectors, which are mapped to the end points of the curve segment, are sufficient to distinguish between both base vectors. The reason of this is, that the number of colors are very rare in maximal highlight regions Finding the specular vector The structure of color clusters on the DCP highly depends on the shape of the highlight parts. In contrast to the matte part of the color signal the specular part cannot in general be modeled by a single rule. Among other things the shape of the specularity depends on: 1. the material of the object (above all on its roughness), 2. the geometry of the object, 3. the angle between S and V. Due to the variant shapes of the highlight structures it is not easy to model the specularity of the material. The consequence of this is, that there are many different methods known for modeling the glossy reflection. It is the material that complicates the calculation of the highlight vector: the rougher the surface the larger the area of the highlight on the DCP. For the purpose of image segmentation a rough estimation of the highlight vector may yield fair results. The method described in section 3.2 yields errors in the angle between an averaged interface reflection vector and an independent measure up to ten degrees or more 6. Compared with image segmentation methods, making use of photometric properties need better matte images. Since PMS methods use known lighting conditions, the assumption of a

6 known illumination vector is not very restrictive. The illumination color can be found in a robust manner when the sc-space representation of a white matte sphere is used Multi-colored objects Using the sc-space as an extension to multi-colored objects is easy: Figure 2(a) shows the color histogram of an object having two specular and one diffuse materials with different colors. The line segments connected with the origin represent the body reflection vector c b. Both highlight clusters have the same direction c s, since all object parts are illuminated from a single light source. For this object, Figure 3(a) shows the sc-space representation together with a relative frequency distribution. Each cluster represents one body color and the specular component. The maximal peaks lying at the end of the clusters indicate the direction of c b. The point where the curve segments would intersect, indicates the direction of c s. Note that the clusters are not necessary linear. If a cartesian representation is used, an estimation of c s can be found by means of line intersections Results For generating 3-colored spheres the neutral interface reflection model 7 in conjunction with the Blinn model, was utilized. The sphere has the same colors as the Macbeth ColorChecker samples Red, Orange and Green. The red and green surface patches are specular, each of different roughness, while the orange patch is matte. The sphere is illuminated with three collimated bright violet point light sources. The color histogram of sphere 1 is shown in Figure 2(a). A 3D-luminance plot of sphere 1 is illustrated in Figure 2(b). The result of highlight elimination for sphere 1 using the direct approach is shown in Figure 3(b). The matte images of sphere 1, 2 and 3 are used as input to PMS. The needle map is shown in Figure 4. It doesn't appear as a complete sphere, since there are object areas that cannot be illuminated with all light sources together. The highest angular difference of an orientation compared with the model is about Blue Re d Green Fig. 2. (a) color histogram sphere 1, (b) 3D-luminance plot of sphere 1. (a) (b)

7 4. Conclusions After reviewing existing methods to apply PMS to non-lambertian surfaces we have suggested a two-stage process to eliminate the highlight effects. The first stage was a refined highlight separation process yielding good results for synthetic objects. In the second stage an ordinary PMS method for Lambertian surfaces was used. An advantage of this approach is that there is no need of a new PMS implementation. Additionally, no new light sources that reduce the recoverable image area have to be introduced. There are no assumptions on the roughness of the material and the approach can be extended to recover multi-colored surfaces in an easy manner. Since the measured values for each color channel are available, an error minimization is possible without using more light sources. The results for the spheres and these possibilities of error reduction are encouraging enough to continue work in this direction (a) Fig. 3. (a) histogram of the spherical chromaticity space of sphere 1, (b) 3D-luminance plot of the body reflection component of sphere 1. (b) Fig. 4. needle map as a result of the two-stage PMS process.

8 5. References 1. E.N. Coleman, R. Jain: Obtaining 3-Dimensional Shape of Textured and Specular Surfaces Using Four-Source Photometry. CGIP 18, (1982). 2. R. Gershon, A. D. Jepson, J. K. Tsotsos: Highlight Identification Using Chromatic Information. Proc. 1st ICCV, London, England, (1987). 3. B.K.P. Horn: Understanding Image Intensities. Artificial Intelligence, 8(2), (1977). 4. K. Ikeuchi: Determining Surface Orientation of Specular Surfaces by Using the Photometric Stereo Method. IEEE Trans. on PAMI, 3(6), (1981). 5. G.J. Klinker: A Physical Approach to Color Image Understanding. Ph.D. Thesis, Technical Report CMU-CS , Carnegie Mellon University, Computer Science Department (1988). 6. G.J. Klinker, S.A. Shafer, T. Kanade: A Physical Approach to Color Image Understanding. IJCV, 4, 7-38 (199). 7. H.-C. Lee: Illuminant Color from Shading. SPIE Vol. 125 Perceiving, Measuring and Using Color, (199). 8. S.W. Lee, R. Bajcsy: Detection of Specularity Using Color and Multiple Views. Proc. ECCV '92, Santa Margherita Ligure, Italy, (1992). 9. C.L. Novak, S.A. Shafer: Anatomy of a Histogram. Technical Report CMU-CS-91-23, Carnegie Mellon University, Computer Science Department (1991). 1. K. Schlüns: Eine Erweiterung des Photometrischen Stereo zur Analyse nichtstatischer Szenen. Proc. 14. DAGM-Symposium Mustererkennung 1992, Dresden, Germany, (1992). 11. S.A. Shafer: Using Color to Separate Reflection Components. COLOR research applications, 1(4), (1985). 12. H. D. Tagare, R. J. P. defigueiredo: Simultaneous Estimation of Shape and Reflectance Maps from Photometric Stereo. ICCV '9, Osaka, Japan, (199). 13. H. D. Tagare, R. J. P. defigueiredo: A Theory of Photometric Stereo for a Class of Diffuse Non-Lambertian Surfaces. IEEE Trans. on PAMI 13(2), (1991). 14. R. J. Woodham: Photometric Stereo: A Reflectance Map Technique for Determining Surface Orientation from Image Intensity. SPIE Vol. 155 Image Understanding Systems & Industrial Applications, (1978).