Shape from Texture: Surface Recovery Through Texture-Element Extraction

Transcription

1 Shape from Texture: Surface Recovery Through Texture-Element Extraction Vincent Levesque 1 Abstract Various visual cues are used by humans to recover 3D information from D images. One such cue is the distortion of textures due to the projection of the 3D world onto a D image plane. Blostein and Ahuja developed a technique that estimates the tilt and slant of a textured plane by analyzing the distortions of texture elements. The technique first locates candidate texture elements with a multiscale region detector. A model of the texture deformations is used to estimate the likelihood of various surface orientations given the area and position of the candidate texture elements. The search determines the most plausible tilt and slant for the surface. This paper revisits the theory proposed by Blostein and Ahuja, and presents the results of experimentations with a simplified version of their technique. and motion. Even though we know this to be true, we tend to perceive a surface covered with identical dots at a particular orientation with respect to the camera. The human perceptual system is thus capable of recognizing the distortion that results from the projection of a textured surface onto a D image. Index Terms Shape from texture, shape recovery, surface orientation, texture-element extraction, multiscale, surface estimation. (A) (B) I. INTRODUCTION The image formation process projects an image of the 3D world onto a D surface. This process is impossible to invert perfectly since information is invariably discarded by the projection. Our own experiences and psychophysical experiments show, however, that the human visual perceptual apparatus is capable of recovering detailed information about the 3D world from D images using various visual cues. It is believed that the human visual system is capable of deducting information about the 3D world from cues such as shading, contour, texture, stereo and motion. In this paper we are interested in the recovery of the tilt and slant of a textured surface through the analysis of distortions in the shape and size of texture elements. The slant is an angle between 0 and 90 o indicating the angle between the surface and the image plane [1]. The tilt is the angle between 0 and 360 o corresponding to the direction in which the surface normal projects in the image []. Informally the tilt indicates the direction in which the distance to the plane increases fastest and the slant indicates how fast the distance increases. Figure 1 illustrates the distortion of texture element on planes. The four pictures show elliptical dots of various size and shape in a D image. Super and Bovik [6] point out that this interpretation is reinforced by visual cues such as stereo vision 1 Vincent Levesque is a M.Eng. student at the Center for Intelligent Machines at McGill University. vleves@cim.mcgill.ca. (C) (D) Figure 1: Synthetic Textured Planes A: slant 5, tilt 70 ; B: slant 50, tilt 90 C: slant 60, tilt 5 ; D: slant 60, tilt 90 (Source: [1]) Blostein and Ahuja [1] proposed an algorithm that recovers the tilt and slant of a textured plane in a two-step process. First candidate texture elements (texels) are extracted from the image using a multiscale region detector originally presented in []. The region detector uses a set of filters of different sizes to locate circular regions of relatively uniform graylevel. Overlapping regions are combined to form candidate texels. The second step is to search for the most plausible surface orientation given the set of candidate texels and a model of the image formation process. A fitness function is used to evaluate the likelihood of various plane orientations. The plane with highest fit rating is selected as the most plausible plane. In this paper the theory proposed by Blostein and Ahuja in [1] and [] is revisited and results of experimentation with a simplified version of their algorithm are presented. Section II gives a brief overview of the shape from texture research over the past decades. Section III discusses in details the theory

2 proposed by Blostein and Ahuja. Section IV provides some details about the implementation used for the experimentation. Section V presents the experimental results obtained with the method. II. BACKGROUND Artists have known the importance of texture as a visual cue for the recovery of shape information for centuries. The first psychological theories of shape from texture are generally attributed to Gibson who insisted on human s reliance on gradients of features. Many researchers have since then proposed shape from texture techniques using a wide range of assumptions and approaches. Bajcsy and Lieberman [3] proposed a simple and elegant method to determine the orientation of a receding plane. Their method analyses texture characteristics using the Fourier transform of different windows. The wavelength observed inside a window is used as a measure of the texel size. A simple projection model is used to infer the relative distance between objects on the receding plane. Witkin [] argues that texel localization is not justified and can be avoided. He proposes instead a measure of texture uniformity using the distribution and orientation of edges. Statistical methods are then used to evaluate the likelihood of various plane orientations. The search for the most likely plane is very similar to Blostein and Ahuja s technique. As we will see later however, Blostein and Ahuja argue that Witkin s technique is bound to fail because of the noise produced by subtexture edges. Aloimonos [5] presented a novel projection model and various techniques to recover the orientation of a textured plane. It is shown that the plane orientation can be recovered if texels can be found but also under the less restrictive assumption that texel boundaries can be found. Super and Bovik [6] propose an approach that doesn t require texel localization. Gabor filters are used to analyze the image in the frequency domain and to determine the most plausible shape. This technique recovers the shape of a curved surface using closed form solutions and thus requires no surface fitting. These papers illustrate the variety of approaches to the problem of shape from texture. The various approaches differ mostly in their degree of reliance on texel extraction, their scope and their assumptions. texture element extractors have prompted many researchers to explore shape from texture methods that do not necessitate texel extraction. An example of this is Witkin s [] method based on the distribution of edges. Blostein and Ahuja argue that texel extraction is an essential part of any shape from texture algorithm. A texture is often formed of a hierarchy of texture elements and subtextures. Subtextures are clearly visible on parts of the plane close to the viewer but are blurred away from the viewer due to the limited resolution of the imaging process. More details are thus visible close to the viewer. Distinguishing texture elements from subtextures is thus impossible. Techniques that rely on the localization of texel boundaries cannot distinguish between texel and subtexture boundaries. Techniques that rely on edges are bound to suffer from the detection of edges within a texel. Blostein and Ahuja thus believe that texel extraction must be an integral part of a robust shape from texture algorithm. They propose the use of a multiscale region detector developed originally in []. The method initially locates circular regions of relatively uniform graylevel at different scales and then groups the disks to form texels. Blostein and Ahuja define an ideal disk as an image for which the graylevel is C within a circular region of diameter D and zero elsewhere. A closed form solution can be found at the center of the disk for the convolution of the image with a G filter (laplacian-of-gaussian) and with a (d/d) G filter of the same dimension. This result can be used to obtain an expression for the diameter D and contrast C of the disk as shown in equations 1 and. (1) D = G * I ( G * I ) + D /8 () C = e ( G * I ) πd The region detector requires the convolution of the image with six pairs of G and (d/d) G filters. The filters have standard deviations,, 3,, 5 and 6. The corresponding diameters are, 8, 1, 16, 0 and pixels. The filters are defined by equations 3 and. Convolutions were computed in the frequency domain by Blostein and Ahuja but were computed in the spatial domain in the current implementation. III. BLOSTEIN AND AHUJA A. Candidate Texel Extraction A texture typically consists of a repetition of similar patterns referred to as texture elements. Unsupervised identification and localization of texture elements is a complex task that requires assumptions to be made. The limitations of available (3) () r r / G( r) = e 6r r ( / ) G( r) = 7 e r /

3 3 The extrema of the response to the G filters are computed by examining the 8 surrounding pixels. Maxima are considered as candidates for positive contrast disks and minima for negative contrast disks. Equation 1 is used to compute the diameter of the disks. Only disks with a diameter differing from the corresponding filter diameter by at most pixels are accepted. Others are rejected in the hope that a better disk will be found at a different scale or, in other words, with a different filter. The contrast of accepted disks is computed with equations. Overlapping disks are then joined together to form a candidate texel. There is no way to know a-priori whether two overlapping disks are part of the same texels or part of the boundary between two adjacent texels. Blostein and Ahuja use a notion of concavity to determine if a texel should be split into multiple texels. If the concavity of two circles along the boundary of a texel falls within a given range, the texel is split in two. Both the single unified texel and the resulting two texels are considered as candidates. A mutual exclusion mechanism is used to prevent the two possibilities from contributing simultaneously to the surface fitting process. As explained in the next section texel splitting was not fully implemented. The result of this step is a set of candidate texture elements with an associated position, area and contrast. The contrast is computed as the weighted average of the contrast of the disks composing the region. B. Plane Fitting The texel extraction step produces a set of candidate texels. The plane fitting step searches through a large number of possible spatial configurations to determine the plane orientations that agrees with as many candidate texels as possible. Blostein and Ahuja derive equations 5 and 6 relating the area of a texel at a given position in the image plane to the tilt and slant of the plane. In these equations X and Y indicates the position of the texel in the range from 1 to 1. T indicates the tilt of the plane and S the slant. The r/f ratio is a measure of the field of view of the camera lens (r is the physical width of the film and f is the focal length). A c denotes the area of the texel at the center of the image and A i the expected area of the texel under consideration. (5) θ = tan 1 (( X cost + Y sint )( r / f )) (6) A = A ( 1 tanθ tan S) i c The actual area of the texel and its position are known. The r/f ratio is assumed to be known. The area of the centered texel (A c ) is not known. The plane fitting is thus done by considering a large number of values for the tilt (T: 0 o, 0 o,, 80 o, 300 o ), slant (S: 0 o, 5 o,, 75 o, 80 o ) and area (A c : 10, 0, 0, 80, 160, 30, 60). The fit rating, as computed with equations 7 and 8, is used to assess the agreement of the spatial configuration with the set of candidate texels. The spatial configuration with the best fit rating is taken to be the most plausible plane orientation. (7) 1(region fit rating = (region areas) region contrast e all regions max(expected area, actual area) (8) region fit = min(expected area, actual area) IV. IMPLEMENTATION fit) / The algorithm presented by Blostein and Ahuja was implemented in C++ on a personal computer with 18MB of memory and a 50MHz P3 processor. All test images were 51x51 56-bit grayscale images taken from the electronic copy of [1] or from the VisTex texture database [7]. The multiscale region detector was implemented in the spatial domain with filters of size corresponding to the filter diameter. Execution time is in the order of minutes. The region detector detects both disks of positive and negative contrast. For simplicity only positive contrast disks were taken into account in this implementation. Texels were formed by grouping overlapping disks together. Blostein and Ahuja propose a vaguely defined measure of concavity as a criterion for the splitting of texels. The exact way in which this concavity is measured could not be found. An alternate definition involving the tangents to adjacent circles at the point of intersection was devised. Integrating it into the algorithm requires the determination of which disks lie on the exterior boundary of the texel. Blostein and Ahuja also propose to keep both the initial unified texel and the resulting two texels as mutually exclusive texels. The splitting of the texels is an essential part of the algorithm. It does, however, complicate the implementation significantly and could not implemented for the experimentation shown here. Neglecting to split texels has no impact on images in which the texels are well spaced. It does, however, cause the method to break down on real images in which texels often overlap. It was observed that low contrast disks often join different true texels of high contrast into a very large texel. A simple heuristic was thus developed to split texels based on this observation. Whenever two disks overlap, a disk with a contrast lower than 30% of that of the other disk is deleted. Once regions are formed, any disk with a contrast lower than 30% of the maximum in the texel is also deleted. This tends to break down large texels without deleting true texels of low contrast. The plane fitting method was implemented following Blostein and Ahuja s guidelines. The plane fitting was limited to a single search and thus provides a precision of 5 degrees for the 3

4 slant and 0 degrees for the tilt. The results can be improved by recursively searching in limited ranges close to the current best plane estimate. This was not implemented. Blostein and Ahuja assume that the r/f ratio is known for the test images. In the following experiment the exact r/f ratio is not known. The best plane was thus computed for r/f ratios of 0.1, 0.5, 1, 1.5 and. V. RESULTS A. Simple dot patterns The algorithm was initially tested with the simple artificial dot patterns shown in Figure 1. These pictures were taken from the electronic copy of [1]. Figure shows the response of the different filters to the image shown in Figure 1, part D. Each filters captures only part of the original dot pattern. Figure 3 shows the result of combining all the disks together. The resulting image is almost identical to the original image. 0.5 and that the algorithm functions correctly for simple synthetic textures. Table 1: Result of plane fitting (true plane: T=90, S=60) r/f A c T S rating B. Tiles Figure : Tile Images (left: Tile.000.pgm, right: Tile.0001.pgm) Figure : Disks of positive contrast (input: fig. 1D) From left to right, top to bottom: =,,3,,5,6. Figure shows two real pictures of tiled floors. These textures are more challenging than the synthetic dot patterns of Figure 1 but still easier to analyze than natural textures. Notice that the picture on the left has large high contrast tiles whereas the one on the right has low contrast tiles divided in columns. The exact plane orientations are not known but we can assume that the picture on the left is close to the plane shown in Figure 1D (tilt=90 o, slant=60 o ) and the picture on the right to Figure 1B (tilt=90 o, slant=50 o ). 1) Tile.000.pgm Figure 3: Detected Regions (original: see fig. 1D) The texels are formed by joining overlapping disks. Since the dots are well spaced, the true texels do not overlap and the candidate texels correspond to single dots. The plane fitting then determines the most plausible plane orientation. Table 1 shows the result of the plane fitting for the plane shown in Figure 1D. The rating is highest for a r/f ratio of 0.5 and the corresponding tilt and slant are perfect at 90 o and 60 o respectively. Similarly the algorithm produces the correct tilt and slant for all three other planes of Figure 1 with a r/f ratio of 0.5. This strongly suggests that the true r/f ratio is close to Figure 5 shows the response of the six filters to the image on the left of Figure. The first image of Figure 6 shows the grouping of all disks together. The boundary of high contrast texels is well represented but the middle of the texels is not completely filled and some low contrast texels are found between the high contrast texels. The second image shows the candidate texels. It isn t clear from the pictures but the background of the first picture is almost entirely covered with low contrast disks. As a result most of the disks are connected to each other and a very large texel occupying the entire image is detected. This problem can be partially solved with the thresholding technique described in the previous section. The third picture of Figure 6 shows the candidate texels after thresholding. Most true texels are now disjoint and some of the true low contrast texels in the back of the plane were saved. The fourth picture shows the texels that contribute to the plane fitness rating for a r/f ratio of 1. The brightness of the texels is proportional to their contribution to the fit rating of the plane.

5 5 are reasonable. The last two ratios produce inconsistent results. Sadly the highest fit rating doesn t correspond to the best plane orientation. Table : Result of plane fitting (Tile.000.pgm) r/f A c T S rating ) Tile.0001.pgm Figure 7 shows the result of the algorithm on the second image of Figure. The image on the left shows the candidate texels. Notice that the texels include diamond-shaped high contrast tiles and triangle-shaped low contrast tiles. In this case thresholding is not necessary to split the texels. The picture on the right shows the subset of the texels that contributed to the fit rating of the best plane for a r/f ratio of 1.0. Table 3 shows the results for different r/f ratios. Again the plane orientation varies considerably depending on the r/f ratio. The best plane corresponding to a r/f ratio of 0.1 is somewhat reasonable with a tilt of 70 o and a slant of 80 o. Other results are not very good. Figure 5: Disks of Positive Contrast (original: see fig. left) From left to right, top to bottom: =,,3,,5,6. Figure 7: Result of Algorithm (Tile.0001.pgm) Left: Candidate texels. Right: Contributing texels (r/f = 1.0) Table 3: Result of Algorithm (Tile.0001.pgm) Figure 6: Result of Algorithm (Tile.000.pgm) Top left: All disks. (graylevel proportional to disk contrast) Top right: Texels. (graylevel proportional to texel contrast) Bottom left: Texels after thresholding (graylevel proportional to texel contrast) Bottom right: Contributing texels for r/f=1 (graylevel proportional to fitness) The search was extended to allow areas of up to 560 pixel since the tiles are very large. Table shows the result of the algorithm. The first three results for ratios of 0.1, 0.5 and 1.0 r/f A c T S rating C. Natural Textures Figure 8, Figure 9 and Figure 10 show three examples of natural textures: two pictures of the sea and a picture of a grass field. The images on the right are the detected disks for each of the three textures. Numerous disks are detected in each case 5

6 6 and the density of disks is highest for parts of the plane farthest from the viewer. As a result larger texels form in parts of the planes away for the viewer where texels should in fact be smaller. The algorithm thus tends to invert the tilt of the plane. Even without this problem, however, the algorithm would not do well. The detected disks are very close to each other and thus it is very difficult to determine which subsets should form texels. The current implementation based on thresholding with respect to the contrast of the disks would not be sufficient to break large texels into smaller, true texels. The texel splitting method of Blostein and Ahuja based on concavity would probably do better although the results might still not be satisfactory. plausible plane orientation as well as the true texels out of a set of candidate texels. The texel extraction implementation was simplified by neglecting to split large texels based on the concavity of their boundary. A thresholding based on the contrast of disks was used instead. This change weakened the algorithm significantly but provided satisfactory results in some restricted cases. The algorithm was shown to produce perfect results when analyzing synthetic textures with simple dot patterns. The algorithm was also shown to produce reasonable results with two pictures showing simple patterns of tiles. The algorithm failed on natural textures. The implementation is thus much weaker than the original implementation that was meant for such natural textures. The assumption that r/f ratio is known also caused significant problems. The exact ratio was not known for any of the test images and the result of the algorithm varied significantly for ratios in the range 0.1 to.0. Figure 8: Water.0000.pgm (left: image; right: disks) The results obtained are sufficient to show the potential of the algorithm on synthetic textures and on simple man-made textures. A faithful implementation of the texture element extraction algorithm should be sufficient to provide good results on natural textures. ACKNOWLEDGMENT The author would like to express his gratitude to the authors of the papers discussed in this paper for sharing their results with the academic community. VI. CONCLUSION Figure 9: Water.000.pgm (left: image; right: disks) Figure 10: GrassLand.0001.pgm (left: image; right: disks) In this paper the shape from texture method proposed by Blostein in Ahuja in [1] was revisited and results of experimentation with a simplified implementation of their algorithm were presented. This shape from texture method differs from other methods that preceded it in its acknowledgement of the importance of texture element extraction. A multiscale texel extraction method is integrated with a surface estimation technique to choose the most REFERENCES [1] D. Blostein and N. Ahuja, Shape from texture: integrating texture-element extraction and surface estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 1, pp , December [] D. Blostein and N. Ahuja, A Multiscale Region Detector, Computer Vision, Graphics and Image Processing, vol. 5, no. 1, pp. -1, Jan [3] R. Bajcsy and L. Lieberman, Texture gradient as a depth cue, Computer Graphics and Image Processing, vol. 5, pp. 5-67, [] A. P. Witkin, Recovering surface shape and orientation from texture, Artificial Intelligence, vol. 17, pp. 17-5, [5] J. Aloimonos, Shape from texture, Biological Cybernetics, vol. 58, pp , [6] B. J. Super and A. C. Bovik, Shape from texture using local spectral Moments, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no., pp , april [7] VisTex Texture Database: x.html 6