Surfaces with Occlusions from Layered Stereo

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 8, AUGUST Surfaces with Occlusions from Layered Stereo Michael H. Lin and Carlo Tomasi Abstract We roose a new binocular stereo algorithm that estimates scene structure as a collection of smooth surface atches. The disarities within each atch are modeled by a continuous-valued sline, while the extent of each atch is reresented via a ixelwise artitioning of the images. Disarities and extents are alternately estimated in an iterative, energy minimization framework. Exerimental results demonstrate that, for scenes consisting of smooth surfaces, the roosed algorithm significantly imroves uon the state of the art. Index Terms Binocular stereo vision, energy minimization, grah cuts, hybrid system, smooth surfaces, surface fitting, boundary localization, shar discontinuities, quantitative comarison. 1 INTRODUCTION æ THE foundations of stereo are corresondence and triangulation. Given two images, if one can find a air of left and right image oints that corresond to the same world oint, geometry readily yields the three-dimensional osition of that world oint. It is the search for such corresonding airs that is the central art of the stereo roblem. There are several constraints that hel to solve corresondence. Given geometric calibration, the eiolar constraint reduces the search for ossible oint matches from two dimensions to one. Given hotometric calibration, color constancy further narrows the ossibilities to oints that look alike. Marr and Poggio [15] roosed two additional heuristics that mitigate the ill-osedness of stereo: uniqueness, which states that each item from each image may be assigned at most one disarity value, and continuity, which states that disarity varies smoothly almost everywhere. Of these four constraints, the former two are relatively straightforward, but the alication of the latter two varies greatly [2], [9], [18]. We roose a three-axis categorization of binocular stereo algorithms according to their interretation of continuity and uniqueness. In the following sections, we list last, for all three axes, that category which we consider to be the most referable.. M.H. Lin is with Acuity Technologies, 3475 Edison Way, Building P, Menlo Park, CA michelin@cs.stanford.edu.. C. Tomasi is with the Deartment of Comuter Science, Levine Science Research Center, Section D, Duke University, PO Box 90129, Durham, NC tomasi@cs.duke.edu. Manuscrit received 11 June 2002 revised 6 May 2003 acceted 15 Set Recommended for accetance by L. Quan. For information on obtaining rerints of this article, lease send to: tami@comuter.org, and reference IEEECS Log Number Continuity The first axis describes the modeling of continuity of disarities within smooth surface atches. Constant. Every oint within any one smooth surface atch is assigned the same disarity value. This value is usually chosen from a finite, redetermined set of ossible disarities, such as the set of all integers within a given range or the set of all multiles of a given fraction (e.g., 1=4 or 1=2) within a given range. Examles of rior work in this category include traditional sum-of-squareddifferences (SSD) correlation, as well as [5], [10], [12], [13], [15]. Discrete. Disarities are again limited to discrete values, but with multile distinct values ermitted within each surface atch. Surface smoothness in this context means that, within each surface, neighboring ixels should have disarity values that are numerically as close as ossible to one another. Examles of rior work in this category include [3], [11], [17], [21]. Real. Disarities within each smooth surface atch vary smoothly over the real numbers. Various interretations of smoothness can be used most try to minimize local first or second-order differences in disarity. Examles of rior work in this category include [1], [4], [19], [20]. 1.2 Discontinuity The second axis describes the treatment of discontinuities at the boundaries of surface atches: The enalty for a discontinuity is examined as a function of the size of the jum of the discontinuity. Free. Discontinuities are not secifically enalized. These methods often fail to resolve the ambiguity caused by eriodic textures or textureless regions. Examles of rior work in this category include traditional SSD correlation, as well as [12], [15], [21]. Infinite. Discontinuities are enalized infinitely, i.e., they are disallowed. The recovered disarity ma is smooth everywhere. Examles of rior work in this category include [16], [19]. Convex. Discontinuities are allowed, but a enalty is imosed that is a finite, ositive, convex function of the size of the jum of the discontinuity. The resulting discontinuities often tend to be somewhat blurred because the cost of two adjacent discontinuities is no more than that of a single discontinuity of the same total size. Examles of rior work in this category include [11], [17], [20]. Nonconvex. Discontinuities are allowed, but a enalty is imosed that is a nonconvex function of the size of the jum of the discontinuity. The resulting discontinuities often tend to be fairly clean because the cost of two adjacent discontinuities is generally more than that of a single discontinuity of the same total size. Examles of rior work in this category include [3], [6], [7], [10]. 1.3 Uniqueness The third axis describes the alication of uniqueness, esecially to occlusions. One-Way. Uniqueness is assumed within a chosen reference image, but not considered within the other. That is, each location in one image is assigned at most one disarity, but the disarities at multile locations in that image may oint to the same location in the other image. Tyically, each location in one image is assigned exactly one disarity, with occlusion relationshis being ignored. Examles of rior work in this category include traditional SSD correlation, as well as [4], [6], [12]. Asymmetric Two-Way. Uniqueness is encouraged for both images, but the two images are treated unequally. That is, reasoning about occlusion is done and the occlusions that accomany deth discontinuities are qualitatively recovered, but there is still one chosen reference image, resulting in asymmetries in the reconstructed result. Examles of rior work in this category include [1], [5], [15], [20], [21]. Symmetric Two-Way. Uniqueness is enforced in both images symmetrically detected occlusion regions are marked as being without corresondence. Examles of rior work in this category include [3], [10], [11], [13]. 1.4 Overview In this aer, we roose an algorithm (described more fully in [14]) that lies in the most referable category along all three axes. To the authors knowledge, ours is the first such algorithm for binocular stereo. We contend that, for scenes consisting of smooth surfaces, our algorithm imroves uon the state of the art, achieving both more accurate localization in deth of surface interiors via subixel disarity estimation and more accurate localization in the image lane of surface boundaries via the symmetric treatment of images with roer handling of occlusions /04/$20.00 ß 2004 IEEE Published by the IEEE Comuter Society

2 1074 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 8, AUGUST 2004 Section 2 describes our mathematical model of the stereo roblem. Sections 3 and 4 describe surface fitting and boundary localization, while Section 5 summarizes the overall algorithm. Finally, Sections 6 and 7 resent some exerimental results and a few concluding remarks. 2 PRELIMINARIES In this section, we develo a mathematical abstraction of the stereo roblem in a continuous domain. Discretization for comutational feasiblity will be discussed in subsequent sections. 2.1 Mathematical Abstraction We use a layered model [8] to reresent ossible solutions to the stereo roblem. We follow the common ractice of assuming that inut images have been normalized for both hotometric and geometric calibration. In articular, we assume that the images are rectified. Let I¼f ¼ðx y tþg ¼ ðr R f}left} }RIGHT}gÞ be the sace of image locations and let I : I 7! R m be the given inut image air. Tyically, m ¼ 3 for color images, and m ¼ 1 for grayscale images. Note that I is defined on a continuous domain in ractice, it is interolated from discrete ixels. Our abstract model of a hyothesized solution consists of a labeling (or segmentation) f, which assigns each oint of the two inut images to zero or one of n surfaces, lus n disarity mas d½kš, each of which assigns a disarity value to each oint of the two inut images: ½segmentationŠ f : I 7! f Ng ½disarity maš d½kš : I 7! Rfor k in f Ng: The segmentation function f secifies to which one of n surfaces, if any, each image location belongs. We take belonging to mean the existence of a world oint which 1) rojects to the image location in question and 2) is visible in both images. For each surface, the signed disarity function d½kš defines the corresondence (or matching) function m½kš between image locations: m½kš : I 7! I m½kšðx y tþ ¼ x þ d½kšðx y tþ y:t where : }LEFT} ¼ }RIGHT} and vice versa. The interretation of this model is: for all : fðþ ¼k with k>0 ) corresonds to m½kšðþ fðþ ¼0 ) corresonds to no location in the other image: 2.2 Desired Proerties Using this abstraction, how can we evaluate a hyothesized solution? We roose three roerties that together characterize a good solution: consistency, smoothness, and nontriviality. Consistency. Corresondence should be bidirectional: If and q are images of the same world oint, then each corresonds to the other otherwise, neither corresonds to the other. It cannot be that corresonds to q but that q does not corresond to. This translates into a constraint on each m½kš, equivalent to a constraint on each d½kš, and additionally into a constraint on f: for all k : m½kšðm½kšðþþ ¼ ð1þ for all : fðþ ¼k with k>0 ) fðm½kšðþþ ¼ k: ð2þ Ideally, consistency should be exact, but comutationally, it is merely maximized. Smoothness. Continuity dictates that a recovered disarity ma should consist of smooth surface atches searated by cleanly defined, smooth boundaries. Thus, both d½kš and f should be smooth. The disarity mas d½kš are continuous-valued functions, so we take the smoothness of d½kš to mean differentiability, with the magnitude of higher derivatives being relatively small. The segmentation function f is iecewise constant, so we take the smoothness of f to mean simlicity of the boundaries searating those ieces, with the total boundary length being relatively small. Nontriviality. Good solutions should exlain the inut as much as ossible. For examle, any two images could be interreted as views of two distinct surfaces, each shown to one camera such a trivial interretation would be valid but undesirable. Moreover, although color constancy can be violated, a solution that does so excessively would also be undesirable. That is, we exect that a corresondence exists, and that color constancy holds, for most image locations: for most : fðþ > 0 for most where fðþ > 0: I m½fðþšðþ IðÞ: 2.3 Energy Minimization We formalize the stereo roblem in an energy minimization framework. We formulate six energy terms, corresonding to each of the three desired roerties, alied to both disarity mas over surface interiors and segmentation via surface boundaries total energy is the sum thereof. 3 SURFACE FITTING In this section, we consider the subroblem of estimating the disarity mas d½kš, suosing that the segmentation f is known. Using this context, we formulate and discuss the minimization of the three energy terms that encourage surface nontriviality, smoothness, and consistency. 3.1 Surface Model We model the disarity ma of each surface as a bicubic B-sline. The control oints of the sline are laced in each image on a fixed, uniform rectangular grid (5 5 in our exeriments). The resulting sline surface can be thought of as a linear combination of shifted basis functions, with shifts constrained to the grid. Mathematically, we restrict each d½kš as follows: d½kšðx y tþ ¼ X ij D½kŠ½i j tšbðx in y jnþ where b is the bicubic basis function, D is the lattice of control oints, and n is the sacing thereof. 3.2 Surface Nontriviality This energy term, often called the data term in other literature, enalizes any deviation from color constancy. We quantify such deviation using a scaled sum of squared differences: E match I ¼ X ( gðiðm½kšðþþ IðÞ AðÞÞ if fðþ ¼k with k>0 ð4þ where gðv AÞ ¼v T A v, and where AðÞ is a measure of certainty, defined as follows: Let I be the m m identity matrix and x 2 be shorthand for the outer roduct xx T. Let G I reresent the convolution of I with a Gaussian of width. Then, for all, we define ð3þ

3 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 8, AUGUST h A ¼ I þ G ði 2 Þ ðg IÞ 2i 1 EðfÞ ¼ X V q ðf f q ÞþX D ðf Þ ð5þ ðqþ2n 2P where and are small constants. Intuitively, AðÞ serves to normalize the local contrast of I around. Note that this and the next two energy terms should be defined as integrals over all 2I, but, for comutational convenience, we merely take a finite sum over discrete ixel ositions. 3.3 Surface Smoothness Although our sline model already ensures some degree of surface smoothness, this inherent smoothness is limited to a satial scale not much larger than that of the sline control oint grid. Because we would also like to have smoothness on a more global scale, we imose an additional energy term which, loosely seaking, is roortional to the global variance of the surface sloe: E smooth d½kš ¼ smooth d X rd½kšðþ meanðrd½kšþ 2 where the summation and the mean are both taken over all discrete ixel ositions, indeendent of the segmentation f. This energy term attemts to quantify deviations from global lanarity. 3.4 Surface Consistency For erfect consistency, a surface should have left and right views that coincide exactly with one another, as secified in (1). However, with left and right views simultaneously constrained each to have the form of (3), exact coincidence is generally no longer ossible. Therefore, to allow but discourage any noncoincidence, we roose the energy term E match d½kš ¼ match d X 2 m½kšðm½kšðþþ which, intuitively, measures the distance between the surfaces defined by the left and right views. Again, the sum is taken over all discrete ixel ositions, indeendent of the segmentation f. 3.5 Surface Otimization Given a articular k, this section s subroblem is to minimize total energy by varying d½kš while holding f and the remaining d½jš constant. Total energy is a sum of six terms, three of which were shown in this section to deend smoothly on d½kš. In Section 4, the remaining three terms are shown either to deend only on f or to deend smoothly on d½kš. Hence, the total energy as a function of d½kš is differentiable and can be minimized with standard gradientbased numerical methods. For convenience, we use MATLAB s otimization toolbox the secific algorithm chosen is a trust region method with a 2D quadratic subroblem. For each k>0, we call minimizing total energy over d½kš, a surface-fitting ste. 4 SEGMENTATION In this section, we consider the subroblem of estimating the segmentation f, suosing that the disarity mas d½kš are known. Using this context, we formulate and discuss the minimization of the three energy terms that encourage segmentation nontriviality, smoothness, and consistency. 4.1 Segmentation by Grah Cuts Boykov et al. [6] showed that certain labeling roblems can be formulated as energy minimization roblems and solved efficiently by finding minimum-cost cuts of associated network grahs. Formally, let L be a finite set of labels, P be a finite set of items, and NPPbe the set of interacting airs of items. The methods of [6] find a labeling f that assigns exactly one label f 2Lto each item 2P, subject to the constraint that an energy function of the form be minimized. Individual energies D can be arbitrary, while interaction energies V q should be either semimetric or metric. This generic formulation of an energy-minimizing labeling roblem mas to our formulation of the stereo roblem as follows: The labels are the integers 0...N that are the ossible values of the segmentation function f and the items are the ixels of each inut image. This is in contrast to [6] in which the items are the ixels of a single reference image and to [13] in which the items are airs of otentially corresonding ixels. In our formulation, the individual energies stem from testing color constancy at varying disarities and the interaction energies stem from the exectations of smoothness and consistency. Our algorithm rohibits ixels from being slit satially among several surfaces, instead constraining surface boundaries to lie on ixel boundaries. Thus, in reresenting the continuous-domain segmentation function f with a finite number of unknowns, we erform nearest-neighbor interolation on a discrete grid of ixels F, defined on an integer lattice: fðx y tþ ¼F ðroundðxþ roundðyþ tþ: 4.2 Segmentation Nontriviality The rimary goal of the segmentation subroblem is to assign each ixel to the surface it fits best. This is accomlished by minimizing E match I, defined in (4) here, we consider it as a functional of f with m½kš being constant, instead of vice versa. However, note that, since gðþ is nonnegative, E match I is trivially minimized by fðþ 0. To discourage solutions with a large number of unassigned ixels, we add a fixed enalty for each unassigned ixel: E unassigned ¼ X ( unassigned if fðþ ¼0 0 otherwise: Thus, the underlying segmentation roblem, for the moment ignoring smoothness and consistency, is to find the labeling f that minimizes the sum E match I þ E unassigned. Put into the form of (5), this corresonds to the following definition of individual ixel references: D ðf Þ¼g Iðm½kŠðÞÞ IðÞ AðÞ for f > 0 D ð0þ ¼ unassigned: 4.3 Segmentation Smoothness A secondary goal is to encourage a simle segmentation with smooth boundaries of surface extents. There are several attributes which can formalize this notion we choose to minimize boundary length because it is relatively simle to otimize and works fairly well in ractice. In addition to this reference for simle boundaries, there is also an exectation that boundaries will generally be correlated with monocular image features (called static cues in [6]). To estimate edge likelihood, we use a function of gradients and local contrast. This measure of edge likelihood at each oint is then used to adjust the cost er unit length of boundaries assing through that oint. There is one more issue to consider: Which boundaries do we want to minimize? Intuitively, minimizing the length of a boundary will tend to shorten or remove any rotrusions or indentations that are long and thin. This makes sense for regions that corresond to surfaces, but not for regions that corresond to occlusions because occlusion regions are in fact usually long and thin. To encourage a simle segmentation, we thus define this energy term for each surface k>0:

4 1076 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 8, AUGUST 2004 X w E smooth f½kš ¼ s ð qþ if fðþ ¼k xor fðqþ ¼k We use a modified version of the exansion algorithm of [6]. adjacent to q This algorithm is built from exansion moves and gets its ower from the generality of such moves: An exansion on a label k finds where adjacency is according to 4-connectedness within each the best configuration reachable by relabeling any subset of ixels image and where with k. We recede each exansion of any label k with a w s ð qþ ¼ smooth f 1 þ e ðjrijt AjrIjÞ= contraction of the same label (by first relacing all instances of k with the satially nearest label that is not k) which strictly enlarges the set of reachable configurations. We call such a contractionexansion air on any one label a segmentation ste. where smooth f and are constants and ri and A are both evaluated at ð þ qþ=2. Put into the form of (5), E smooth f½kš corresonds to this enalty function: V q ðf f q Þ¼w s ð qþ X T f ¼ k xor f q ¼ k k>0 8 >< 0 if f ¼ f q ¼ w s ð qþ 1 if f 6¼ f q with f ¼ 0orf q ¼ 0 >: 2 if f 6¼ f q with f > 0 and f q > 0 for adjacent to q, where T ðþ equals 1 if its argument is true and equals 0 otherwise. 4.4 Segmentation Consistency For exact consistency, the segmentation f should satisfy (2) everywhere. To quantify and discourage any inconsistencies, we formulate an energy term for each surface k>0: E match f½kš X match f if fðþ ¼k xor fðm½kšðþþ ¼ k which aroximates the area of regions where (2) does not hold. Ideally, as with those in Section 3, this term should be defined with an integral, but, in this case, a naive finite sum is not an adequate substitute when subixel disarities are allowed, as exlained in [14]. Instead, we take E match f½kš ¼ ( X match f ^h jm½kšðþ qj if fðþ ¼k xorfðqþ ¼k q where and q are on conjugate eiolar lines, and where 8 1 < 2 for j4dj 1 2 ^hð4dþ ¼ 3 j4dj 4 2 for : 1 2 < j4dj < for j4dj 3 2 : Note that our imlementation modifies ^h by rounding its corners (at j4dj ¼ 1 2 and j4dj ¼3 2 ) so that total energy remains differentiable with resect to d½kš. Put into the form of (5), E match f½kš corresonds to this enalty function: V q ðf f q Þ¼ X w c ½kŠð qþt f ¼ k xor f q ¼ k k>0 for and q in corresonding scanlines, where w c ½kŠð qþ ¼ match f ^h m½kšðþ q þ ^h m½kšðqþ and match f is a constant. 4.5 Segmentation Otimization This section s subroblem is to minimize total energy by varying f while holding all d½kš constant. Total energy is a sum of six terms, two of which are indeendent of f. In this section, the remaining four terms are written in the form of (5) moreover, our V q can be verified to be metric. Hence, the total energy as a function of f can be otimized with grah cut methods [6]. 5 OVERALL OPTIMIZATION In this section, we consider the roblem of simultaneously determining surface shae in the form of disarity mas and surface suort in the form of segmentation, when both are unknown. Our algorithm is built from the surface-fitting and segmentation stes of Sections 3 and 4. Since each of these stes reduces total energy, given a reasonable initial hyothesis, iterating these stes until convergence might give a reasonable final solution. However, during the course of such comonent-wise otimization, it is quite ossible to reach a local minimum. These undesirable configurations are generally of two tyes: those in which one hyothesized surface sans several actual surfaces and those in which several hyothesized surfaces san one actual surface. Our algorithm currently cannot reliably extract itself from the former tye of local minima and therefore relies uon careful initialization to avoid getting into such situations. The initial hyothesis is formed by lacing one fronto-arallel surface at every integer disarity within the secified range of ossible disarities all ixels are initially unassigned (with f 0). The latter tye of situation is more easily handled. Often, when several hyothesized surfaces san one actual surface, one hyothesized surface will eventually come to dominate and the others will naturally be driven to extinction. When this is not the case, a merge ste will generally remedy the situation. To take a merge ste, we first save a snashot of the current state. We then forcefully remove one surface. The orhaned ixels are relabeled with f ¼ 0, but are immediately redistributed among the remaining surfaces by a series of segmentation stes. Further surface fitting and segmentation stes are then taken until either the total energy falls below that of the saved snashot, in which case the merge succeeds and the snashot is discarded, or the total energy lateaus above that of the snashot, in which case the merge fails and the snashot is restored. The comlete algorithm is as follows: 1. Initialize hyothesis with surfaces at integer disarity. 2. Reeat: a. Alternately aly segmentation and surface fitting stes until rogress is negligible. b. For each surface, attemt to merge it until one merge succeeds or all merges fail. until all merges fail. 3. Otionally fill in unmatched regions, using neighboring matched regions (see [14]). 6 EXPERIMENTAL RESULTS We have imlemented our algorithm using a combination of MATLAB and C, and tested it on several nonsynthetic stereo airs available online [4], [14], [18]. In this section, we resent the results of our exeriments on a reresentative subset of these images and comare them to those achieved by other algorithms. Comlete results can be found in [14].

5 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 8, AUGUST Fig. 1. (a) Venus and (b) Tsukuba stereo airs: estimated disarities, disarity errors, and error distributions. 6.1 Quantitative Results To evaluate the accuracy of our algorithm, we use the general framework roosed by Scharstein and Szeliski [18], who rovide four samle stereo airs with ground truth, describe a metric for comaring results against ground truth, and tabulate results for 20 algorithms. They evaluate overall results by measuring the fraction of unoccluded ixels where estimated and ground truth disarities differ by more than one ixel in articular, they ignore the estimated disarity at occluded ixels. In contrast, we measure the fraction of all ixels (both occluded and unoccluded) where disarity error exceeds a threshold. We also consider a range of thresholds, and lot the fraction of bad ixels as a function thereof. In these lots, we comare our algorithm with the four that aear to be the most accurate among the others tabulated in [18]. Due to sace limitations, we only show results for two of the four stereo airs used in [18]. Venus. This color stereo air (Fig. 1a) shows five slanted lanes, including some regions with virtually no texture. Two of the surfaces are joined by a crease edge the remaining boundaries are all ste edges. Our algorithm does extremely well, roducing about five times fewer bad ixels than the nearest cometition in [18] for a significant range of disarity error thresholds. Our largest error occurs at the corner of the V-shaed deth discontinuity, where our enalty for boundary length causes the ti of the V to be missed. This tye of behavior is a tyical result of minimizing boundary length without regard for boundary curvature and junctions. Tsukuba. This color stereo air (Fig. 1b) shows a laboratory scene consisting of various lanar, smoothly curved, and nonsmooth surfaces. Several long and thin structures are resent our algorithm tends to oversimlify the boundaries thereof. It is notable, however, that, while the given ground truth reresents all surfaces as being fronto-arallel at integer disarity, our algorithm roduces curved surfaces with subixel disarities. In articular, our algorithm models the entire head as one curved surface, with a disarity range of aroximately one half ixel. 6.2 Qualitative Results Clorox. To verify both that our algorithm can recover crease edges and also that it needs neither color nor dense texture, we tested it on two of the grayscale stereo airs used in Birchfield and Tomasi [4], but, due to sace limitations, we only show results for one. The original version of this stereo air shows minor hotometric variations between the left and right images, as well as minor geometric distortion. Here (Fig. 2a), we use a modified version from which the hotometric variations have been mostly removed. Note that the geometric distortion was left in lace this manifests itself in the aarent curvature of the floor, as recovered by our algorithm. This stereo air is well aroximated by five slanted lanes. Not all disarity edges are well marked by intensity edges and some distracting intensity edges do not accomany disarity edges. Birchfield and Tomasi s multiway cut algorithm [4] struggles with these images, deceived by the misleading intensity edges into mislacing the crease edges there. Our algorithm does not have this roblem, but instead makes a different error: The Clorox box is reresented with only one surface. Umbrella. To verify that our algorithm can reconstruct curved surfaces without dense texture, we also tested it on a stereo air of our own creation. This stereo air (Fig. 2b) shows five surfaces. Three are essentially lanar but strongly slanted among these, the floor has low-contrast, fine-grained texture, while both checkerboard atterns have high-contrast, coarse-grained texture. Additionally, the rear surface is a large, somewhat wared, essentially textureless sheet of cardboard and the strongly curved umbrella is comosed of large, essentially textureless anels joined together by high-contrast color edges that are not disarity edges. The simultaneous combination of all these disarate features makes this stereo air articularly challenging. Although we do not have results by other algorithms for this stereo air, we note that few of the algorithms in [18] are caable of reresenting smoothly curved surfaces with subixel disarity

6 1078 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 8, AUGUST 2004 Fig. 2. (a) Clorox and (b) Umbrella stereo airs: inut images, grayscale estimated disarities, and color-coded estimated disarities. (All trademarks remain the roerty of their resective owners. All trademarks and registered trademarks are used strictly for educational and scholarly uroses and without intent to infringe on the mark owners.) values and, among those, fewer still readily reroduce shar discontinuities in the disarity ma. Our algorithm correctly segments the scene into five smooth surfaces, each of which is reresented by a real-valued disarity ma that contains no kinks or creases, even in the resence of strong color edges that might suggest otherwise. Our algorithm laces boundaries accurately at crease edges as well as at edges accomanied by significant occlusion regions and qualitatively recovers the curvature of both the background and the umbrella with very little hel from texture. 7 DISCUSSION AND FUTURE WORK The quantitative and qualitative results resented suggest that, for scenes consisting of smooth surfaces, our algorithm roduces very accurate reconstructions, with subixel disarity values and exlicit and recise localization of boundaries, whether the surfaces are lanar or curved, textured or untextured, highcontrast or low-contrast, color or grayscale. Desite this achievement, however, there is nonetheless much room for imrovement. The most limiting asect of our current imlementation is its model of surfaces. Although our model works quite well for most of the results that we have resented, it can be overly restrictive for scenes whose surfaces are less smooth. To be able to handle surfaces with more shae detail, our imlementation should use a much finer grid for the control oints of the slines that define surface shae. This would likely require a more refined model of surface smoothness. Finally, we note that many of the arameters of our algorithm, controlling such things as coarseness of segmentation and amount of surface shae detail, do not have to be constant, but can in fact vary from surface to surface and even within the same surface. If, in addition to the disarity mas and segmentation, these arameters themselves could be estimated searately and adatively for each surface, we believe that our algorithmic framework would be caable of roducing accurate results for a much wider variety of scenes. [3] P.N. Belhumeur, A Bayesian Aroach to Binocular Stereosis, Int l J. Comuter Vision, vol. 19, , [4] S. Birchfield and C. Tomasi, Multiway Cut for Stereo and Motion with Slanted Surfaces, Proc. Int l Conf. Comuter Vision, , 1999, htt://vision.stanford.edu/~birch/multiwaycut/. [5] A.F. Bobick and S.S. Intille, Large Occlusion Stereo, Int l J. Comuter Vision, vol. 33, no. 3, , [6] Y. Boykov, O. Veksler, and R. Zabih, Fast Aroximate Energy Minimization Via Grah Cuts, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, Dec [7] I.J. Cox, S.L. Hingorani, S.B. Rao, and B. Maggs, A Maximum Likelihood Stereo Algorithm, Proc. Comuter Vision and Image Understanding, vol. 63, no. 3, , [8] T. Darrell and A. Pentland, Cooerative Robust Estimation Using Layers of Suort, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 5, , May [9] U.R. Dhond and J.K. Aggarwal, Structure from Stereo A Review, IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 6, , [10] D. Geiger, B. Ladendorf, and A. Yuille, Occlusions and Binocular Stereo, Int l J. Comuter Vision, vol. 14, no. 3, , [11] H. Ishikawa and D. Geiger, Occlusions, Discontinuities, and Eiolar Lines in Stereo, Proc. Euroean Conf. Comuter Vision, vol. 1, , [12] T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adative Window: Theory and Exeriment, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 9, , Set [13] V. Kolmogorov and R. Zabih, Comuting Visual Corresondence with Occlusions Using Grah Cuts, Proc. Int l Conf. Comuter Vision, , [14] M. Lin, Surfaces with Occlusions from Layered Stereo, PhD thesis, Stanford Univ., 2002, htt://robotics.stanford.edu/~michelin/layered _stereo/. [15] D. Marr and T. Poggio, Cooerative Comutation of Stereo Disarity, Science, vol. 194, , [16] T. Poggio, V. Torre, and C. Koch, Comutational Vision and Regularization Theory, Nature, vol. 317, , [17] S. Roy and I. Cox, A Maximum-Flow Formulation of the N-Camera Stereo Corresondence Problem, Proc. Int l Conf. Comuter Vision, , [18] D. Scharstein and R. Szeliski, A Taxonomy and Evaluation of Dense Two- Frame Stereo Corresondence Algorithms, Int l J. Comuter Vision, vol. 47,. 7-42, 2002, htt:// [19] R. Szeliski and J. Coughlan, Sline-Based Image Registration, Int l J. Comuter Vision, vol. 22, no. 3, , [20] H. Tao, H.S. Sawhney, and R. Kumar, A Global Matching Framework for Stereo Comutation, Proc. Int l Conf. Comuter Vision, , [21] C.L. Zitnick and T. Kanade, A Cooerative Algorithm for Stereo Matching and Occlusion Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 7, , July REFERENCES [1] S. Baker, R. Szeliski, and P. Anandan, A Layered Aroach to Stereo Reconstruction, Proc. Comuter Vision and Pattern Recognition, , [2] S. Barnard and M. Fischler, Comutational Stereo, Comuting Surveys, vol. 14, , 1982.