Dense 3D Reconstruction from Specularity Consistency

Transcription

1 Dense 3D Reconstruction from Secularity Consistency Diego Nehab Microsoft Research Tim Weyrich Princeton University Szymon Rusinkiewicz Princeton University Abstract In this work, we consider the dense reconstruction of secular objects. We roose the use of a secularity constraint, based on surface normal/deth consistency, to define a matching cost function that can drive standard stereo reconstruction methods. We discuss the tyes of ambiguity that can arise, and suggest an aggregation method based on anisotroic diffusion that is articularly suitable for this matching cost function. We also resent a controlled illumination setu that includes a air of cameras and one LCD monitor, which is used as a calibrated, variable-osition light source. We use this setu to evaluate the roosed method on real data, and demonstrate its caacity to recover high-quality deth and orientation from secular objects. 1. Introduction A oular class of systems for determining the 3D shae of real-world objects relies on the notion of multi-view consistency. These systems may be thought of as hyothesizing 3D oints, then evaluating whether their rojections into two or more viewoints are consistent with images taken from those views. In the simlest case, the consistency is evaluated based on color. More sohisticated systems might evaluate the consistency of windows of ixels (as is the case in many stereo systems), or might consider temoral variation (as in temoral active stereo or structured light systems). Any of these consistency criteria may be used to construct a matching cost function, which is then used during triangulation to find geometry (ossibly augmented by an additional stage to enforce global consistency or smoothness). With few excetions, however, such systems have focused on reconstructing diffuse objects and scenes. In this aer, we consider the roblem of reconstructing secular objects, ositioned in a scene or environment of known geometry and aearance. Because the images of these objects consist entirely of reflections of the scene around them, simle consistency metrics based on color, ixel windows, or temoral variation will not be effective. Instead, we rely on a criterion that considers the reflection of the scene on a surface element. Secifically, we consider the mirror reflection of a camera ray off a surface atch with some hyothesized osition and normal; the roosed osition/normal tule exhibits secularity consistency if the observed ixel is consistent with the intersection of the reflected ray and the (known) scene. Given only a single camera osition, we find that an infinity of osition/normal tules will necessarily satisfy the consistency condition: for any roosed osition, we can find a normal such that the reflected ray hits any desired scene oint. Two camera ositions, however, rovide disambiguating information, and in most cases restrict us to a single allowed osition/normal (excetions are discussed later in the aer). Note that, in general, a oint is reconstructed not because different cameras observe the same art of the scene reflected from it: they observe different reflections that are both consistent with the hyothesized osition/normal. This secularity consistency condition has been reviously identified and exloited in other contexts [23, 5]. We roose to use the constraint to define a matching cost function for two camera views, and demonstrate that the condition may be used for dense stereo reconstruction of secular 3D objects. We analyze our system on synthetic imagery, and show real-world results. Our contributions include: defining a stereo matching cost function based on the secularity consistency constraint; roosing a novel normal-based anisotroic diffusion scheme for the matching cost, which strengthens matches lying on a continuous surface; resenting a theoretical analysis of the ambiguities that can arise from the secularity consistency constraint; obtaining dense and accurate 3D reconstructions of secular objects, by directly exloiting this constraint. 2. Related work Secular surfaces have been widely studied in the literature. By imosing continuity and smoothness constraints, deth can be indirectly obtained from normal estimates [12, 29]. Normal estimates for glossy surfaces can be comuted for examle by fitting of exlicit or arametric models [13, 9, 18], or by direct detection of highlights [8]. On the other hand, most work on stereo reconstruction in the resence of secularities attemts to avoid view deendent effects. Aroaches that coe with secular highlights 1

2 s 2 s 2 n 1 = n 2 n c 1 n 1 n 2 x 1 x 2 Figure 1. The secularity consistency. An incorrect match x 2 is likely to roduce normal hyotheses n 1 and n 2 that do not agree, allowing us to reject the match. In contrast, the correct match will always result in consistent normal estimates by the two cameras. include caturing multile views [2, 16], eliminating highlights with olarization filters [31, 19], treating them as occlusions [15], or removing them from the catured images rior to matching [28, 16, 34, 32]. Naturally, for the class of objects we are considering (i.e., smooth, glossy materials), view-deendent information rovides the most useful of constraints. A wide range of methods derive local shae information from the identification and/or tracking of the distorted reflections of lightsources and secial known features [3, 4, 36, 22, 24]. These methods tend to roduce sarse reconstructions. Dense measurements can be roduced, for examle, by the general framework of light-ath triangulation [14]. Within this framework, our method can be described as a novel technique for 2, 1, 1 -triangulation. Conversely, Bonfort et al. [6] resent a method for 1, 1, 2 -triangulation that requires just one viewoint, but two known reference oints er ray. Outside of this framework, shae has been recovered by rotating an object under known or unknown illumination [33, 1], and by exloring the Helmholtz recirocity [35]. Recently roosed matching cost functions that are esecially suited to secular surfaces include light transort constancy [10] and scatter-trace hotograhy [17]. The idea of checking for normal consistency across different views has been originally roosed by Sanderson et al. [23], in the context of feature matching of secular highlights, and later used by Bonfort and Sturm [5], as art of a voxel carving method for recovering the shae of secular objects. In contrast, by rehrasing this constraint as a matching cost function, we can leverage decades of research on stereo reconstruction [25]. In articular, we can take advantage of efficient global matching algorithms that roduce satisfactory dense reconstructions even when the matching cost functions are not articularly discriminant. 3. Triangulation by secularity consistency The reconstruction method we roose relies on the ability to densely identify environmental scene oints reflected s 1 x 2 c 2 X by a secular surface. This can be achieved by illuminating the secular object with a dense set of controllable light sources. Various setus have been roosed to create a temorally encoded lighting environment using monitor ixels as light sources [37, 30, 6]. Using a suitable encoding scheme, the source osition s of light reflected by a secular surface oint toward a camera can be decoded from the camera s intensity observations over time. In section 6, we resent a similar rototye system to evaluate our aroach. With the light source osition s known, the normal n at a oint IR 3 can be easily comuted as the bisector n = (l + v)/ l + v between the lighting and viewing directions: l = (s )/ s and v = (c )/ c, with c the center of rojection of the observing camera. In the context of stereo reconstruction, however, deths and hence surface oint ositions are not known in advance. Instead, a candidate oint corresonding to a disarity hyothesis has to be tested for consistency. Consider figure 1. A oint on the secular surface rojects to ixel x 1 as seen by camera 1. Assume we also know the light source osition s 1 that casts a highlight through x 1. We want to find the corresonding ixel x 2 to which rojects at camera 2, so we can triangulate for its osition. Naturally, we must be able to distinguish the wrong candidate matches x 2 from the correct match x 2. The figure shows an examle of incorrect match x 2. By triangulation, this match determines an incorrect osition along the line of sight through x 1. Given and s 1, camera 1 hyothesizes a normal direction n 1. Camera 2, on the other hand, makes an indeendent normal direction hyothesis n 2. This hyothesis comes from and the light source osition s 2 that casts a highlight through x 2. Recall that, in the figure, does not roject to x 2. Some other surface oint does, with an indeendent normal direction n. This is the oint resonsible for the light source osition s 2 that is visible through x 2. The indeendence between oints and is likely to cause the normal direction hyotheses to disagree, in other words n 1 n 2. In contrast, the correct corresondence x 2 actually images the oint. Triangulation roduces the correct deth, and the light source osition s 2 visible through x 2 really is reflected from. Since both x 1 and x 2 observe the same surface oint, the normal direction hyotheses n 1 and n 2 will agree. Our framework for dense stereo reconstruction exloits this fact. 4. Dense stereo framework Dense stereo reconstruction generally requires a metric that assesses a disarity hyothesis d for a given image oint (x, y). This is tyically exressed by assigning a matching cost value C(x, y, d) to this hyothesis. A stereo matching algorithm then assigns minimum-cost disarities to all ixels in a reference image, say, the image catured by

3 camera 1. Algorithms differ in whether disarity minimization takes lace locally or globally, and in how an overall minimum is defined and comuted [25]. However, matching costs are universal, and arbitrary cost functions can be directly used with most existing algorithms Matching Costs Stereo matching cost functions are usually based on image intensities and consider squared or absolute differences between candidate matching ixels in the image airs. That is, the cost of matching ixel (x, y) in image 1 with ixel (x + d, y) in image 2 can be exressed, e.g., as C(x, y, d) = I 1 (x, y) I 2 (x + d, y), (1) with I i (x, y) denoting the intensity at a ixel (x, y) in the image catured by camera i. In contrast to this, our reconstruction framework is based on differences in normal hyotheses rather than intensities. A given disarity hyothesis d for a ixel (x, y) corresonds to a candidate oint in sace. As exlained in section 3, roduces two normal estimates n i () from the two cameras. We use the angular difference δ(x, y, d) = cos 1( n 1 () n 2 () ) (2) between the normal estimates as a corresondence measure for a hyothesis (x, y, d). If exactly coincides with a secular surface, δ is exected to be zero. In a realistic setting, however, there will be a slight error in the normal estimate due to noise and to satial and intensity quantization, that is, the measured δ will deviate from the ideal δ 0 by δ = δ 0 + δ. Assuming the error term δ to be meanfree and normal-distributed with standard deviation σ, the likelihood of δ(x, y, d) denoting a match is P(x, y, d) = 1 σ δ(x,y,d) 2 2π e 2σ 2. (3) In our exeriments, we choose σ to be between 5 and 8, deending on the data quality; using much different values leads to a oor discrimination of matches. We therefore set the matching cost function C(x, y, d) = 1 P(x, y, d). This cost function can now be used with a variety of existing stereo reconstruction algorithms. Section 7 shows examle reconstructions using different alternatives Normal aware cost aggregation In traditional (intensity-based) stereo, the cost comutation is often extended to a window region around the oint of interest. This increases the reliability of the cost assessment for textured regions. Cost aggregation within a window can be exressed as a 2D or 3D convolution C (x, y, d) = (w C)(x, y, d) (4) with a window function w [25]. This convolution can alternatively be exressed as a ossibly anisotroic diffusion rocess [26]. When w is anisotroic, it introduces a bias for a certain sloe during the surface reconstruction stage. In articular, if w is a 2-dimensional kernel over x and y (a common choice), the convolution corresonds to the integration over a window in the image domain, introducing a bias for image-arallel surface sloes. We exloit the additional knowledge of normal directions to secifically bias the surface reconstruction toward the surface orientation corresonding to the normals. The intersection of the object surface with the eiolar (xd-)lane of y corresonds to a minimum-cost ridge in the cost function C(x, y, d). Any surface reconstruction algorithm has to trace this ridge. In order to imrove the recondition of the surface reconstruction, we erform anisotroic diffusion of the symmetrized cost function F y (x 1, x 2 ) C(x 1, y, x 2 x 1 ), aggregating costs along the ridge direction redicted by the normal estimates. As shown in [20], given a air of rectified cameras, the unit tangent direction t to the matching ridge at (x 1, x 2 ) can be exressed as t (n 1, n 2 ) (5) where n and i are, resectively, the 2D rojections of the normal and osition of the surface oint at (x 1, x 2 ), on the eiolar lane of y relative to camera i. Following equation (4), we convolve F y with a satiallyvarying, oriented Gaussian filter kernel g: where F y (x 1, x 2 ) = (g F y )(x 1, x 2 ), (6) g(u, v) = e x V 1x, x = (u, v), (7) is controlled by a satially-varying variance matrix V IR 2 2, deendent on the location of the filter alication (x 1, x 2 ): V = r aa I + r d t i t i + F y (x 1, x 2 ) t i t i. (8) i {1,2} Here t i is the unit vector in the direction of the sloe, according to the normal estimate n i, and t i is unit length and orthogonal to t i. Each summand after the summation sign imlements the variance matrix of an oriented Gaussian along t i, each converging to an isotroic Gaussian as the local matching cost reaches one. Adding the variance matrices corresonds to convolving the resective Gaussians. The additional summand r aa I rovides an anti-aliasing refilter to guarantee a faithful discretization of g. In our imlementation we use an anti-aliasing radius of r aa = 0.4 and a diffusion seed of r d = 1.41; deviation from these values tyically leads to a less stable cost aggregation, or to a

4 Source F y F y after 4 iterations F y after 16 iterations Figure 2. Consistency values within an eiolar lane (red line) through a roblematic region with many false matches. High confidence values are dislayed in red; the black line shows the result of a disarity otimization using dynamic rogramming. Anisotroic diffusion according to the normal estimates attenuates false ositives. The images show (from left to right) unmodified consistency values F y(x 1, x 2) and F y(x 1, x 2) after 4 and 16 diffusion stes, resectively. slow convergence, resectively. Finally, in order to ensure energy reservation, each filter kernel is normalized after discretization to add u to one. By iteratively alying equation (6), minimum-cost ridges are emhasized where both normal estimates agree with the matching ridge, while surious ridges are attenuated. Figure 2 shows an examle of a roblematic eiolar lane through the quarter dataset: the comlex leaf attern on the coin, and the non-secular background object introduce many surious matches that are gradually smoothed out by the diffusion rocess. 5. Ambiguities Even disregarding interreflections, the secularity constraint can lead to ambiguities. An observation by camera i can be described by a air (x i, s i ), where x i is the ixel coordinate and s i the light source osition that roduces a highlight through x i. Here, for simlicity, and without loss of generality, we assume that the light source lies along the baseline between the two cameras (as in figure 3). Ideally, for any surface, and for each observation (x 1, s 1 ) by camera 1, there would exist only one corresonding consistent observation (x 2, s 2 ) by camera 2. Unfortunately, this is not the case. We can distinguish between two tyes of ambiguity. A week ambiguity occurs when at least two different observations, (x 2, s 2 ) and (x 2, s 2 ), are consistent with (x 1, s 1 ). A strong ambiguity occurs when every observation (x 2, s 2 ) is consistent with (x 1, s 1 ) Weak ambiguities Consider a ixel x 2 from camera 2 that does not corresond to x 1 (figure 3). By triangulation, we can comute the osition of the non-existent corresonding surface oint: (x 1, x 2 ) = (X, ) x 1 T T = ( x 1 x, 2 x 1 x ), (9) 2 where T is the baseline distance between the two cameras. From and s 1, we can comute the normal direction exected by camera 1 at. This normal direction is uniquely determined by the intersection oint n 1 of the normal ray at oint with the baseline between the two cameras. To find the intersection, we use the angle bisector theorem on the triangle formed by oints, c 1, and s 1 to get n 1(x 1, s 1, x 2) = s 1 X X (X s 1) (10) If (x 2, s 2 ) is consistent with (x 1, s 1 ), then the light from s 2 must reflect at towards the ray through x 2. Using and n 1, and the angle bisector theorem on the triangle formed by oints, c 2, and s 2, we get s 2 (x 1, s 1, x 2 ) = n 1 2 (T 2X ) + (T 2n 1)(X ) n T(X n 1 ). (11) (X ) Even though the ray through x 2 does not hit a surface oint at (this oint would occlude, which is by assumtion visible through x 1 ), the ray may intersect the surface at another deth. For each ossible oint = (X, ) = (x 2 + T, ) along the ray through x 2, we can calculate a normal direction that reflects s 2 towards x 2. Once again, we use the angle bisector theorem on the triangle formed by oints, c 2, and s 2 to obtain n 2 (x 1, s 1, x 2, ) = T (s 2 X ) s 2 (X T) (s2 X ) (12) (X T) 2 + 2

5 x 1 x 2 x 2 x 1 x 2 x 2 s 2 c 1 s 1 c 2 n 2 Figure 3. Weak ambiguities. Given a surface oint, observed as (x 1, s 1) by camera 1, and a false candidate match x 2, the surface cannot go through the intersection oint. However, for every oint along the ray through x 2, there is a surface orientation that results in an observation (x 2, s 2), consistent with (x 1, s 1). In summary, the rays through x 1 and x 2 observe indeendent surface oints, and intersect at a non-existent hyotetical oint (equation (9)). Camera 1 exects to have normal n 1 (equation (10)). For each x 2, we can comute a light source osition s 2 that causes (x 2, s 2 ) to be consistent with (x 1, s 1 ) (equation (11)). Then, for each oint along the ray through x 2, we can comute a normal direction n 2 that reflects light from s 2 towards the ray through x 2 (equation (12)). Whenever the surface goes through a oint with the aroriate normal direction n 2, there is weak ambiguity Strong ambiguities Consider figure 4. A oint generates an observation (x 1, s 1 ) by camera 1. Take now an arbitrary oint = (X, Y ) being observed by camera 2 through ixel x 2. The ray through x 2 intersects the ray through x 1 at a oint that we can obtain from equation (9). Following equations (11) and (12), we can obtain a normal direction at that makes the observation at x 2 consistent with the observation at x 1. In other words, each observation by camera 1 imoses a normal field on the X-lane. Starting from any oint in the X-lane (i.e., an initial condition), any trajectory that resects this normal field (i.e., solves the associated differential equation) will generate a strong ambiguity curve. The curves in figure 4 were generated by numerical integration. If the surface follows one of these curves, every observation by camera 2 will be consistent with (x 1, s 1 ) by camera 1. These are the strong ambiguity curves. In theory, weak and strong ambiguities are extremely unlikely. In ractice, finite recision on the light source osition estimation, as well as noise and discretization artifacts allow the henomenon to manifest itself as wrong matches. These can be articularly roblematic if the surface resembles a strong ambiguity curve. n 1 X c 1 s 1 c 2 Figure 4. Strong ambiguity curves. An observation (x 1, s 1) by camera 1 imoses a normal field on the X-lane. If a surface conforms to this normal field, every observation (x 2, s 2) by camera 2 will be consistent with (x 1, s 1). In other words, we have a strong ambiguity. 6. Acquisition Secular triangulation requires a camera air and an aaratus to illuminate an object with a dense set of calibrated light sources. To that end, the cameras acquire images of the object being lit by a temorally encoded light attern. In rincile, we can use any light attern that allows us to find the identity of a single light source L i in a secular reflection observed by a camera at c i. Efficient encoding schemes, such as gray-code atterns, exist; however, secular reflection is subject to a convolution with a secular lobe. Accordingly, the chosen attern has to be robust under convolution. A very robust encoding would be to trigger one light source at a time. For each surface oint, the light source L i max is determined that roduces a maximum intensity resonse under observation from camera i. The temoral location of the maximum resonse is comaratively stable even under convolution with a secular lobe. For a dense set of many light sources, however, this rocedure is highly inefficient. In order to reduce acquisition times, we use multile linear light sources instead of a dense set of single oint light sources [11]. During acquisition, the linear light sources are swet through sace. For each surface oint, the maximum resonse during each swee is determined. In a good aroximation, these maxima occur when the linear light source covers L i max. Accordingly, we intersect the locations of the corresonding illuminating lines to obtain the osition s i of L i max Acquisition Procedure We have imlemented a system that erforms the linear light source swees by dislaying horizontal and vertical stries on an LCD monitor. The stries are 0.5 cm wide and are sequentially shifted by one strie-width on each frame. Scanning times could be imroved by multilexing the stries with Hadamard atterns [27], but we give reference to quality over seed in the acquisition system in order X

6 Figure 6. (To) Temoral resonse measurements at different oints of a coin. (Bottom) Corresonding measured intensity rofiles. Measurements are shown in blue, and the fit of a single Gaussian lobe is shown in green. Figure 5. Photograh of our exerimental setu. The object is laced in front of the monitor, facing the camera air. The cameras cature image airs while the monitor dislays sweeing linear light sources. The small rojector visible above the cameras is used for comarison with structured-light reconstructions. to avoid additional sources of error. Figure 5 shows our exerimental setu. Both the camera air and the monitor s location with resect to the cameras are calibrated. Knowing the light source locations si, secular triangulation can be erformed as described in section 3. In ractice, the crucial art is a reliable estimation of Limax from the intensity variation over time of each observed oint. Camera noise and the discretization of the line swee revent us from directly icking the strie location with the maximum intensity resonse. Instead, we obtained good results by fitting a Gaussian mixture model I(t) = n X ai e (t µi )2 σ2 i (13) i to the data, and taking the maximum intensity oint in time as µi of the narrowest Gaussian lobe (i.e., for i with minimum σi ). The use of multile Gaussians allows the fit to aroximate diffuse and ambient reflectance contributions by wider lobes, with the secular eak corresonding to a single, narrow lobe. A common choice for fits of Gaussian mixture models is the Exectation Maximization algorithm. In our case, however, it is not alicable, as the suort of the measurements is truncated. Instead, we use a general non-linear fit to equation (13). In exeriments with real data, we found that, for most secular materials, the eak can reliably be detected with even a single lobe. See figure 6 for samle temoral reflectance rofiles of measurements of a coin. In the roosed setu, the maximum-resonse eaks µh and µv of the horizontal and vertical swee, resectively, directly corresond to a monitor location (x, y) (µh, µv ), which in turn allows us to determine si Proerties The aroach requires free lines of sight between the surface oint and the two cameras, as well as from the oint to the two light sources in the reflection direction. This imoses constraints on the object geometry. Similar constraints are also resent with other hotometric reconstruction techniques, such as hotometric stereo. Unlike hotometric stereo, which suffers from normal bias in the case of (artial) self-shadowing, secularity consistency only requires a narrow free cone of sight to the lighting environment, and is hence more robust against such bias. A ractical limitation of the roosed setu, however, is the relatively small solid angle covered by the monitor. In order for a oint to be reconstructed, the reflection of (otentially different) areas of the monitor must be visible by both cameras observing the oint. Only oints with normals falling within a narrow range satisfy this requirement. Normals outside of this range result in gas in the reconstruction. We are currently investigating alternative layouts that will reduce this roblem by covering a larger solid angle. 7. Results Using the setu described in the revious section, we catured two real datasets: a quarter and a mirrored shere. In order to evaluate our method in ideal conditions, we also generated synthetic data for a Greek anel and for an analytic shere, using the same calibration arameters as the real scanner. The simulated datasets were rocessed by the same ieline as the real datasets, starting with simulated camera images instead of real images. In order to demonstrate the versatility of the roosed cost function, we use two different stereo reconstruction algorithms. Figure 7 shows a reconstruction of the simulated shere dataset using Markov Random Fields stereo reconstruction [7]. All other reconstructions were obtained using dynamic rogramming. Results are shown in figure 8. In general, we have found that our triangulation stage

7 a meaningful manner, biasing the stereo reconstruction toward the measured surface orientation. In addition, the normal information can be used to refine the outut of the stereo algorithm, yielding low noise and detailed reconstructions. We have demonstrated the alicability of the roosed criterion, using a simle acquisition setu based on two cameras and a monitor. References Figure 7. Rendering from discrete matches roduced by Markov Random Fields stereo reconstruction [7], evaluated on a simulated dataset of a shere using our secularity constraint. consistently roduces high-quality normals, but can generate incorrect deth estimates (comare the first and second columns of figure 8). Due to the geometry of the caturing setu, an incorrect match can result in a large error in the reconstructed deth, whereas the normal estimate comutation tends to be more robust to such errors. Nevertheless, normal and deth are not indeendent, and we therefore use the measured normals to imrove the quality of the deth estimates. To this end, we use the otimization method described in [21]. Recall the idea is to formulate an energy minimization roblem whose solution attemts to reconcile both measurements. Since the method exects to correct relatively small deth variations, we iterate the rocess to roduce larger corrections. The third column of figure 8 shows the result of such otimization, which roduces great imrovements. The mirrored shere shows the entire range of orientations that can be reconstructed by our system. As discussed in section 6.2, reconstruction is only ossible for oints within the overlaing images of the monitor as seen by both cameras. For further illustration, figure 8(h) shows the reflection of the monitor in the shere as seen from one of the cameras. For the quarter dataset, we also attemted to obtain geometry with a structured light scanner. The rojector, used exclusively for this exeriment, is visible in figure 5. Figure 8(d) shows the resulting reconstruction. The artifacts demonstrate the difficulties arising from scanning secular objects with active triangulation. Attemts to obtain the shae of the mirrored shere using structured light rojection consequently failed comletely. 8. Conclusions We have demonstrated the use of a secularity consistency criterion, based on consistency of osition/normal tules, for dense stereo matching in secular scenes. The fact that normals are closely related to the geometry being reconstructed allows us to aggregate matching costs in [1] Y. Adato, Y. Vasilyev, O. Ben-Shahar, and T. ickler. Toward a theory of shae from secular flow. In ICCV, [2] D. N. Bhat and S. K. Nayar. Stereo and secular reflection. IJCV, 26(2):91 106, [3] A. Blake. Secular stereo. In IJCAI, volume 2, ages , [4] A. Blake and G. Brelstaff. Geometry from secularities. In ICCV, ages , [5] T. Bonfort and P. Sturm. Voxel carving for secular surfaces. In ICCV, ages , [6] T. Bonfort, P. Sturm, and P. Gargallo. General secular surface triangulation. In ACCV, ages , [7] Y. Boykov, O. Veksler, and R. abih. Markov random fields with efficient aroximations. In CVPR, ages , [8] T. Chen, M. Goesele, and H-P. Seidel. Mesostructure from secularity. In CVPR, volume 2, ages , [9] E. N. Coleman and R. Jain. Obtaining 3-dimensional shae of textured and secular surfaces using four-source hotometry. CGIP, 18: , [10] J. Davis, R. Yang, and L. Wang. BRDF invariant stereo using light transort constancy. In ICCV, ages , [11] A. Gardner, C. Tchou, T. Hawkins, and P. Debevec. Linear light source reflectometry. SIGGRAPH 2003, 22(3): , [12] M. A. Halstead, B. A. Barsky, S. A. Klein, and R. B. Mandell. Reconstructing curved surfaces from secular reflection atterns using sline surface fitting of normals. In SIG- GRAPH 1996, ages , [13] K. Ikeuchi and B. K. P. Horn. Numerical shae from shading and occluding boundaries. AI, 17(1 3): , [14] K. N. Kutulakos and E. Steger. A theory of refractive and secular 3D shae by light-ath triangulation. In ICCV, ages , [15] Y. Li, S. Lin, H. Lu, S. B. Kang, and H-Y. Shum. Multibaseline stereo in the resence of secular reflections. In ICPR, volume 3, ages , [16] S. Lin, Y. Li, S. B. Kang, X. Tong, and H-Y. Shum. Diffuse-secular searation and deth recovery from image sequences. In ECCV, volume 3, ages , [17] N. J. W. Morris and K. N. Kutulakos. Reconstructing the surface of inhomogeneous transarent scenes by scatter-trace hotograhy. In ICCV, [18] S. K. Nayar, K. Ikeuchi, and T. Kanade. Determining shae and reflectance of hybrid surfaces by hotometric samling. IEEE TRA, 6(4): , [19] S. K. Nayar, X-S. Fang, and T. Boult. Stereo and secular reflection. IJCV, 21(3): , 1997.

8 Real Quarter Dataset (a) Catured normals (b) Catured ositions (c) Refined ositions (d) Traditional scanner Real Mirrored Shere Dataset (e) Catured normals (f) Catured ositions (g) Refined ositions (h) Photograh Simulated Greek Panel Dataset (i) Catured normals (j) Catured ositions (k) Refined ositions (l) Ground truth Figure 8. Final reconstruction results. [20] D. Nehab, S. Rusinkiewicz, and J. Davis. Imroved subixel stereo corresondences through symmetric refinement. In ICCV, ages , [21] D. Nehab, S. Rusinkiewicz, J. E. Davis, and R. Ramamoorthi. Efficiently combining ositions and normals for recise 3D geometry. SIGGRAPH 2005, 24(3): , [22] M. Oren and S. K. Nayar. A theory of secular surface geometry. IJCV, 24(2): , [23] A. C. Sanderson, L. E. Weiss, and S. K. Nayar. Structured highlight insection of secular surfaces. PAMI, 10(1):44 55, [24] S. Savarese, M. Chen, and P. Perona. Local shae from mirror reflections. IJCV, 64(1):31 67, [25] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo corresondence algorithms. IJCV, 47(1/2/3):7 42, [26] D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. IJCV, 28(2): , [27] Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeur. A theory of multilexed illumination. In ICCV, volume 2, ages , [28] S. A. Shafer. Using color to searate reflection comonents. Color research & alications, 10(4): , [29] J. E. Solem, H. Aanaes, and A. Heyden. A variational analysis of shae from secularities using sarse data. In 3DPVT, ages 26 33, [30] M. Tarini, H. P. A. Lensch, M. Goesele, and H-P. Seidel. 3D acquisition of mirroring objects using stried atterns. Grahical Models, 67(4): , [31] L. B. Wolff and T. E. Boult. Constraining object features using a olarization reflectance model. PAMI, 13(7): , [32] K.-J. Yoon and I.-S. Kweon. Corresondence search in the resence of secular highlights using secular-free two-band images. In ACCV, ages , [33] J. Y. heng and A. Murata. Acquiring a comlete 3D model from secular motion under the illumination of circularshaed light sources. PAMI, 22(8): , [34] W. hou and C. Kambhamettu. Binocular stereo dense matching in the resence of secular reflections. In CVPR, ages , [35] T. ickler, P. N. Belhumeur, and D. J. Kriegman. Helmholtz stereosis: Exloiting recirocity for surface reconstruction. IJCV, 49(2/3): , [36] P. isserman, A. Giblin and A. Blake. The information available to a moving observer from secularities. Image and Vision Comuting, 7(1):38 42, [37] D. E. ongker, D. M. Werner, B. Curless, and D. H. Salesin. Environment matting and comositing. In SIGGRAPH 1999, ages , 1999.