Hierarchical Volumetric Multi-view Stereo Reconstruction of Manifold Surfaces based on Dual Graph Embedding

Transcription

1 Hierarchica Voumetric Muti-view Stereo Reconstruction of Manifod Surfaces based on Dua Graph Embedding Aexander Hornung and Leif Kobbet Computer Graphics Group, RWTH Aachen University Abstract This paper presents a new voumetric stereo agorithm to reconstruct the 3D shape of an arbitrary object. Our method is based on finding the minimum cut in an octahedra graph structure embedded into the voumetric grid, which estabishes a we defined reationship between the integrated photo-consistency function of a region in space and the corresponding edge weights of the embedded graph. This new graph structure aows for a highy efficient hierarchica impementation supporting high voumetric resoutions and arge numbers of input images. Furthermore we wi show how the resuting cut surface can be directy converted into a consistent, cosed and manifod mesh. Hence this work provides a compete muti-view stereo reconstruction pipeine. We demonstrate the robustness and efficiency of our technique by a number of high quaity reconstructions of rea objects. 1. Introduction The faithfu and consistent reconstruction of three dimensiona rea word objects from images or video sequences remains a great chaenge in computer vision. One particuary promising approach aowing for reconstructions of very high quaity is the combination of voumetric scene representations with combinatoria optimization [16, 22]. In genera such approaches are based on a discretized voume in which the object to be reconstructed is embedded. Then for each voumetric eement (voxe) the ikeihood of being intersected by the object surface is estimated, e.g., by computing its photo-consistency based on coor variances [7, 18], or normaized cross-correation [22]. Earier reconstruction methods [18] then extracted a voxebased approximate surface by a threshoding approach, i.e., the set of most photo-consistent voxes represents the object surface. Whie this is often sufficient for appications such as new view synthesis [14], the resuting scattered set of voxes does not support any goba smoothness or topoogica constraints due to the purey oca abeing of voxes. This makes it especiay difficut to extract a consistent surface for further processing of the geometry. Recenty severa papers, e.g. [2], have shown how the probem of computing proper segmentation surfaces inbetween the voxes of a voumetric scene representation can be casted into a combinatoria optimization probem for graph cuts. By constructing a specificay designed graph structure, the gobay optima soution to such a probem can be efficienty computed as the minimum cut of the respective graph. In [22] it is shown how this approach can be appied to voumetric muti-view stereo reconstruction, yieding 3D reconstructions of very high quaity. However, one drawback of these combinatoria approaches is that the accuracy of the soution is bounded by the resoution of the domain discretization which cannot be too high due to restricted memory capacities and computation time. Moreover, when deaing with more compex objects and imperfect acquisition conditions, we woud ike to use a highy redundant set of images to obtain a robust reconstruction. This increasing amount of input data obviousy increases the need for more efficient agorithms. Furthermore there exists a subte but important issue in using the above mentioned segmentation approaches for voumetric 3D object reconstruction. A photo-consistency measure generay integrates, for a specific region in space, the ikeihood of being intersected by the surface [5, 7, 18]. Hence it is we defined ony for a proper integration domain, e.g., geometricay vaid entities such as voxes. The above segmentation based approaches however generate a graph structure which associates graph nodes with voxe centers and graph edges with voxe faces. This poses the question of propery defining the edge weights of the embedded graph: If we compute photo-consistency vaues for the voxes, these vaues have to be re-mapped to the graph edges, e.g., by taking the average consistency of two face connected voxes. This, however, is equivaent to appying a ow-pass fiter to the voxe consistency vaues and hence reduces the effective resoution of the reconstruction.

2 In [22] the photo-consistency is computed for the graph edges instead of voxes. But whie the projection of a voxe and hence the respective integration domain for the photoconsistency estimation is we defined, there is no such definition for projecting graph edges into images and computing their photo-consistency. A simpe integration of point sampes over 1D edges or 2D faces woud introduce a directiona bias in the 6-connected grid, and the geometrica interpretation becomes even more uncear if one wants to extend this approach to arger voxe neighborhoods to aow for better surface integra approximations [2, 22]. So whie these approaches are generay we suited for probems where voxes have to be segmented into different casses by a contour or surface at voxe boundaries, they do not support the extraction of a proper surface intersecting the interior of voxes, nor is it cear how the photo-consistency of the corresponding graph edges shoud be defined. In this paper we present a new muti-view stereo approach for efficient, high quaity voumetric 3D reconstruction of soid objects, which resoves the above mentioned probems. Our technique is based on a new octahedra graph structure which is embedded into the voumetric grid, such that voxe photo-consistency vaues are propery associated with graph edges. The minimum cut of this graph yieds a manifod surface, which is gobay optima in terms of an energy functiona of the photo-consistency and the area of the reconstructed surface, supporting smooth and consistent object reconstructions of high quaity. Moreover, our specific graph design supports a hierarchica surface extraction, which aows us to efficienty process even high voumetric resoutions and a arge number of input images. Finay, to provide a compete reconstruction pipeine, we show how to extract a manifod triange mesh directy from the set of intersected surface voxes yieding high quaity meshes which are directy avaiabe for further geometry processing. 2. Reated Work Our method is inspired by a number of important resuts from the fieds of voumetric 3D reconstruction as we as efficient energy minimization techniques using graph cuts. Based on the principes of voxe cooring presented by Seitz and Dyer [18], Kutuakos and Seitz [12] formuated the N-view reconstruction probem of an arbitrary Lambertian shape in their theory of space carving. This approach has shown to aow for highy efficient impementations using, e.g., hardware acceerated soutions [14], which makes it especiay suitabe in the context of new-view synthesis and image-based rendering techniques. However, one probem in common with most techniques reated to voxe cooring is the fact, that they cannot guarantee geometric constraints such as spatia coherence of the reconstructed object surface, since a cassifications are made purey oca at voxe eve. Different methods based on deformabe poygona surfaces have been proposed to directy reconstruct a proper manifod. [6] present a eve-set method which enabes automatic changes of the surface topoogy due to its representation as an impicit function. [8] use a stochastic refinement of the visua hu, such that it approximatey satisfies the photo-consistency constraint. [13] improve the convergence by integrating 3D and 2D data into the optimization process. Reconstructions of very high quaity from high resoution input images based on deformabe modes have been presented by [5]. Athough these gradient based methods are abe to yied high quaity reconstructions of consistent object surfaces, they often have a arge computationa compexity on the one hand, and they are in genera sensitive to oca minima in the target functiona. Hence, especiay in the presence of noise, they cannot guarantee convergence to a gobay optima surface, which best satisfies a constraints such as photo-consistency, smoothness, or a minima surface area. Recenty, efficient methods for energy minimization based on graph cuts were proposed for severa vision reated probems, some of which can guarantee finding the goba optimum of the invoved energy functiona. A soution to the voxe occupancy probem from visua hus based on a segmentation using binary abeing was proposed by [20]. [3] showed how to extend the optimization of binary abes to an arbitrary number of discrete abes by iterated graph cuts with appications in disparity based stereo. Generaized muti-camera stereo reconstruction using graph cuts was presented by [11]. The types of energy functionas which can be minimized using iterated graph cuts was anayzed in [10]. [2] show how to extend the graph construction to compute gobay minima geodesic contours or surfaces for arbitrary Riemannian metrics. Appications of graph cut based energy minimization to voumetric stereo reconstruction were presented in [16, 19, 22]. [16] present a voumetric reconstruction technique for computing gobay optima and smooth surfaces in specific stereo setups. They show resuts for severa stereo setups with remarkaby detaied 3D reconstructions. However, the method is restricted to one rectanguar surface patch and cannot be generaized to other surface topoogies. For exampe, for the reconstruction of a cosed surface, they have to join separatey computed patches by which they ose the optimaity near the patch boundaries. [19] present a graph cut based reconstruction approach which combines strict enforcement of sihouette constraints with photo-consistency constraints. However this approach currenty does not support objects of arbitrary genus, and furthermore does not respect concavities that are not present in the sihouette, but which coincide with rim curves. The approach presented in [22] is cosest reated to our work: It extends the Riemannian minima surface idea [2]

3 Figure 1. Given a set of input images (a) our hierarchica 3D reconstruction agorithm works on an adaptivey refined grid. It first computes an approximation to the visua hu of the object (b) at a user specified resoution eve (1283 in this exampe). Then on successive refinement eves, photo-consistency vaues are computed for each voxe within a crust around the estimated surface position (c-e) (refinement eves 1283, 2563, and 5123 respectivey). A new graph based agorithm is used to extract the gobay optima surface on each eve, which is then used as a surface proxy on the next refinement eve. At the desired target resoution our method supports the direct extraction of a cosed and manifod triange mesh (f). to genera, muti-view voumetric stereo. Starting with an approximate surface, e.g., the visua hu, they compute the visibiity and photo-consistency of scene points in the proximity of this surface. Using graph-based surface extraction they are abe to reconstruct a smooth, detaied surface of very high quaity by segmenting the voume into inside and outside voxes. However, due to their specific graph structure, this method poses a number of open questions concerning the computation of edge weights within the embedded graph (c.f. Sect. 1). Finay, most of the above mentioned techniques do not expoit hierarchica agorithms for increased efficiency, but perform a computations directy at the voumetric target resoution, such that it coud be time consuming in practice to appy them at high voumetric resoutions with many input cameras. A first description and anaysis of hierarchica graph cuts based on 6-connected grids for fast segmentation has recenty been presented in [15]. 3. Hierarchica Voumetric Reconstruction Our work targets at reconstructing objects from arbitrary video sequences, for exampe created by using a turn-tabe setup or simpy a hand-hed camera. Simiar to [22] the input to our agorithm is a number of caibrated and foreground segmented input images I0... In 1, e.g., obtained by using structure from motion [1] and interactive segmentation techniques [17]. Instead of extracting the surface directy at the voumetric target resoution we present a hierarchica method using a sequence of adaptivey refined voxe grids V (Fig. 1). The basic idea is to compute a voxeized, intermediate surface proxy Sopt V on each refinement eve, which is then refined to a new proxy S +1 on the next eve to constrain the voumetric region for the surface computation in V +1. This eads to a significanty reduced space and time compexity in comparison to previous work. We start with an initia bounding cube V 0 containing the object to be reconstructed. As mentioned in [5, 7, 22] it is important to compute an initia surface proxy, e.g., to estimate the visibiity of each voxe in every image. Hence our agorithm starts by computing a voxeization of the object s visua hu: Beginning with = 0 we iterativey refine the voume V and test for each voxe v V if it projects to the background in one of the input images. This is done up to a eve 0, where a reevant object features such as thin structures or hoes in the target mode are reasonaby approximated (Fig. 1, 5, 6). We typicay set 0 = 7 corresponding to a resoution of 1283 voxes. However, if necessary as for exampe for the Dragon mode (Fig. 7), the effect of choosing a different eve 0 can be easiy tested by the user since the computation of the visua hu generay is a matter of a few seconds ([7], Tabe 1). We then define our initia voxeized surface proxy S 0 as the outer boundary voxes of the voxeized visua hu, corresponding to the so caed base surface described in [22]. Our hierarchica reconstruction agorithm then proceeds as foows: At a given eve we define a crust Vcrust of voxes around the current surface proxy S Vcrust V, which is supposed to contain the actua object surface, and which spits the voume V into three components: the crust itsef, an exterior, and an interior component (Fig. 2 a). This can easiy be computed by appying a number crust Vcrust d of morphoogica diation steps to S. In our experiments we generay set d = 2 for a eves. This is justified by assuming an approximation error of the surface proxy S 1 on the previous eve up to the size of one voxe. The ony 0 contains a exception is the initia eve 0, where Vcrust 0 non-background voxes of V since the visua hu of an object can be quite far away from the true object surface, e.g., in concave regions [5, 22]. Based on the current sur-

4 face proxy S we then compute the visibiity for each voxe v Vcrust in each image I j using a hardware-acceerated technique described in [7]. Aternativey this can be done as described in [22], or by using simpe ray-casting techniques. Athough our hierarchica method aows for iterative improvements of the visibiity information, our experiments showed, that an initia visibiity estimation at eve 0 is sufficient as ong as a reevant features such as hoes in the object are present. The next centra step is to compute the photo-consistency φ(v) for a voxes v Vcrust. For maxima fexibiity and efficiency we appy the method described in [7] which estimates the photo-consistency per voxe based on an iumination invariant voxe-supersamping approach for Lambertian surfaces and geometricay unconstrained camera setups. Basicay this approach creates a number m of object space sampes p 0 to p m 1 and a camera-weight w j for each voxe v and image I j. For each of these sampes p i a (normaized) coor vaue ĉ i j in each image I j is computed. The fina photo-consistency is then computed as the sum of weighted, normaized coor variances per sampe: ( 2 φ(v) = 1 φ(p i ), φ(p i ) = w j(ĉ i j) 2 w jĉj) i. m i (1) Since φ(v) measures the photo-consistency based on coor variances, it has vaues cose to zero for photo-consistent voxes, whie it increases for inconsistent voxes. Athough the approach presented in [7] eads to the best resuts in our hierarchica reconstruction setting, techniques based on normaized cross-correation of image patches as described in [5, 22] or even simpe coor variances as originay proposed in [18] aso ead to acceptabe resuts, depending on the properties of the input mode and the images. Having consistency vaues for a voxes within crust Vcrust we appy our new graph-cut based surface extraction agorithm to refine the current surface proxy S to an optimized approximation Sopt, which better fufis photo-consistency and smoothness constraints on the current eve. The detais of this step are described in Sect. 4. If is our desired target resoution, we are now abe to extract a manifod triange mesh from Sopt as described in Sect. 5. Otherwise we refine the adaptive grid for a voxes v Sopt to a new surface proxy S +1, and start a new iteration by buiding a new crust Vcrust. +1 This iterative, hierarchica reconstruction setting aows us to efficienty process arge input data in terms of voumetric resoution and number of input images, since the photo-consistency as we as the extraction of S have to be done ony within a thin crust of voxes instead of the whoe voume, whie the gobay optima properties of the fina reconstruction are preserved. j j Figure 2. This image iustrates the graph based contour computation in 2D. (a) shows a voxe grid of consistency vaues where the darker coors indicate higher photo-consistency vaues. Typicay these consistency vaues have many oca minima and maxima due to coor ambiguities in homogeneousy textured object regions, image noise, erroneous visibiity estimation, or imperfect camera caibration. In (b) the dua 2D graph is embedded into the voxe grid. Green coors correspond to ow edge weights whie red edges have higher weights. The source and sink node are connected to the inner and outer boundaries respectivey. Despite the oca inconsistencies the gobay optima graph cut correcty reconstructs surface S opt as shown in (c). 4. Surface Reconstruction by Graph Cutting In this section we wi present a new graph structure for our hierarchica surface reconstruction approach, which estabishes a we defined reationship between voxe photoconsistency vaues, the embedded graph, and the reconstructed surface. We show that this reation is achieved by representing each voxe by its dua octahedra subgraph. We wi drop eve indices for a simpified notation, since a voxes v Vcrust are at the same refinement eve. Our goa is to find a surface S opt within the current crust V crust which approximates the true but unknown object surface. We computed the ikeihood for each voxe to be intersected by this surface by assigning photo-consistency vaues φ(v) to each voxe v. Hence the reconstructed surface S opt has to be defined by a set of surface-intersected voxes S opt V crust. As aready discussed there is an important difference of this surface definition to graph based segmentation approaches such as [2, 22], since these methods generate graph edges (and hence the segmentation boundary) in-between voxes (or pixes in the 2D case). This specific graph structure does not aow for a we defined assignment of voxe photo-consistency vaues to the graph edges. We wi introduce a new embedded graph structure with the properties that 1) the resuting surface S opt is optima with respect to an energy functiona (Eq. 2) of the integrated photo-consistency and surface area and hence cosey approximates the actua object surface, 2) edge weights are we defined by the photo-consistency of a corresponding voxe, and 3) a minimum cut through this graph directy yieds a cosed, 2-manifod surface represented by a set of surface-intersected voxes S opt. Simiar to previous work the energy functiona we want

5 Figure 3. Spitting the faces of a cube into an exterior and an interior part is equivaent to a graph cut in the dua octahedron (a),(c). Simpe configurations with 4 or 6 cut-edges correspond to ocay panar cut surfaces in the geometric embedding (b) whie compex configurations with 8 or more cuts correspond to ocay curved cut surfaces (d). to minimize for a surface S is a weighted sum of the integrated photo-consistency and the area of the surface: E(S) = φ(x) dx + S a ds (2) S The desired optima surface S opt shoud minimize this functiona for a good approximation of the true object surface. Within our voxe grid V consisting of cubica voxes with square faces, the crust V crust propery separates the interior component V int of V from the exterior component V ext (Fig. 2), since V crust is a face-connected (6- neighborhood) set of voxes. Suppose now we have an arbitrary cosed surface S opt. For each voxe in V crust we can abe its faces as interior or exterior depending on which side of the surface they ie. Faces that are intersected by the surface are abeed as interior by defaut. The important observation now is that if we want to separate the interior faces from the exterior faces for a singe voxe, we have to cut aong a sequence of edges of the voxe (Fig 3 a,c). Based on this observation we buid the foowing graph structure. For each voxe face in V crust we define a node in the graph. Within each voxe v we connect the six nodes corresponding to the six faces in an octahedron-fashion and assign the consistency vaue of the voxe pus a surface area constant φ(v) + a to a tweve edges. Due to the duaity of the cube (voxe) and the octahedron, we have a one-toone correspondence between the edges of both. Hence the above voxe cut (aong edges) which separates interior from exterior faces is equivaent to a graph cut in the octahedron which separates the corresponding nodes (Fig 3 b,d). The goba graph embedded in V crust consists of the subgraphs of a voxes. The graph source is connected to a nodes, whose associated faces ie at the interface to the exterior component V ext, whie a nodes at the interior component V int are connected to the sink (Fig 2 b). Computing the minimum cut of this graph yieds a set of cut edges C which minimizes the sum of edge weights, and which defines a cosed surface that separates the graph into two components by spitting those voxes S opt V crust which are most ikey intersected by the true object surface. Besides pure consistency maximization the geometric smoothness of the resuting cut surface is enforced by two aspects of the graph embedding: First, within each voxe, a configuration which cuts as few octahedron edges as possibe is preferred (since they a have the same weight). These preferred configurations cut either four or six edges and correspond to panar cuts whie more compex cuts with eight or more edges correspond to curved configurations (Fig 3). Second, since the cost of a separating cut is the sum of the weights of a edges that are spit, the optima soution might rather contain inconsistent voxes if this eads to a better goba soution than summing over a arger number of voxes with better consistency. Since the edges are embedded fairy uniform in the voxe space, the sum over a edge weights φ(v)+a can be considered as a decent, discretey samped surface integra approximation of the consistency function and the surface area. Therefore a cut through this graph minimizes the discretization of the energy functiona (2): E(C) = φ(e) + a (3) e C e C E(C) discretizes E(S) up to a constant scaing factor which depends on the voxe size. The trade-off between oca consistency and goba surface area minimization can be controed to a certain extend by appying a suitabe transfer function φ (x) = φ(x) s with a smoothness factor s > 0, and by changing the surface area constant a. Since the graph compexity depends ony quadraticay on the current voumetric resoution due to our hierarchica approach, different choices of these parameters can be evauated by the user with ony short deays. However our experiments reveaed that when using consistency measures such as [5, 7, 22], which generate a reativey distinct consistency maximum (Fig. 1 c-e), the energy functiona is dominated by the photo-consistency vaues. This aows us to keep these two parameters constanty set to s = 4 and a = 10 5 for a resoution eves and experiments. This graph-based surface reconstruction supports our initiay formuated requirements concerning the minimized energy functiona, the definition of edge weights, and the extraction of surfaceintersected voxes S opt. 5. Manifod Extraction After the graph-cut agorithm has found the set of surface-intersected voxes S opt, we have to extract this manifod surface and convert it into a triange mesh to aow for further processing of the reconstructed 3D mode. One soution woud be to appy existing iso-surface reconstruction techniques such as [9]. This however woud

6 Figure 4. (a) shows a set of surface voxes from Sopt, which are spit by the graph cut into inside (dark) and outside (bright) faces. The cut-edges within each voxe define a oop of spit-edges (b). Each oop defines a poygona face of the output surface. By trianguating these poygons we can extract the fina output mesh (c). impy a conversion of our voxe based surface into a 3D scaar fied, and the resuting surface woud not necessariy respect the surface topoogy defined by the cut edges C anymore. Instead we derive a simpe and efficient agorithm for the manifod extraction directy from the geometric interpretation of our graph embedding and the computed cut. Consider a voxe v Sopt which contains some cutedges after the cut surface has been computed for the octahedra photo-consistency graph (Fig. 3). Because of the one-to-one correspondence between the octahedron edges and the voxe (cube) edges, the set of cut-edges in the octahedron defines a oop of spit-edges in the cube, spitting it into inside and outside faces (Fig. 4 a,b). Athough one coud create a proper mesh directy by trianguating either the set of inside or outside faces, this approach woud not generate a minima number of vertices and trianges. Instead we can consider each of these oops of spitedges as one (non-panar) poygona face of the output manifod. Coecting a these oops for each intersected voxe yieds a poygona mesh representation Mp of the fina surface whose vertices ie at the voxe corners. By trianguating each of these poygons, e.g., using simpe triange fans, we obtain the desired triange mesh Mt (Fig. 4 c). Since the vertices v Mt are initiay paced at the voxe corners the resuting reconstruction shows discretization artifacts (Fig. 5 b). We eiminate these artifacts in a postprocessing step by appying a mesh smoothing fiter [21] v v + v to the mesh vertices, with the operator being the discrete Lapacian defined on the mesh (Fig. 5 c). By restricting this smoothing to a maximum distance of one voxe the origina reconstruction error is preserved. This method is guaranteed to efficienty extract a smooth, cosed and manifod mesh. As a side note we woud ike to mention that one can aternativey generate a poygona mesh dua to Mp, which paces a vertex at each voxe center instead at voxe corners. In this case we have to generate a poygon for each 23 bock of voxes, that contains at east 3 cut voxes. Whie this approach creates a reduced number of vertices, it is significanty more invoved to impement for hierarchica grids. Figure 5. The Leo mode was reconstructed using 86 images captured with an uncaibrated turn-tabe setup (a). (b) shows the mesh extracted from the graph cut surface. Discretization artifacts due to the underying voxe grid can be effectivey eiminated by appying a Lapacian smoothing fiter (c). Invisibe parts of a mode ike the bey or the feet of the Leo (d) are smoothy cosed by the computed cut surface. A reconstruction (e) of the head from the same image set shows, that even very fine detais such as the nostris and the concavities of the ears are propery reconstructed. Textured and reighted versions of the mode are shown in (f). 6. Resuts In this section we wi present the resuts of our method appied to severa rea-word data sets. Our reference system was a Linux-based Inte Pentium 4 CPU with 3.2 GHz and 2 GB of main memory. We captured video streams of the Warrior, the Leo, and the Dragon mode with an uncaibrated turn-tabe setup and an image resoution of The Bahkauv statue was captured using a handhed video camera with a resoution of However the effective image size of the objects was often significanty smaer because of the necessity to incude caibration objects within the images (e.g., Fig. 5). We caibrated and segmented the video streams using standard structure from motion [1] and image segmentation techniques[17]. Tabe 1 presents the quantitative resuts for each of the modes, incuding the compexity of the input data, timings, and the resuting mesh compexity. Our experiments show that the photo-consistency estimation is the dominating computationa factor. This fact underines the benefit of our hierarchica approach since it significanty reduces the number of processed voxes, keeping the computation times acceptabe even for high voumetric resoutions. The Bahkauv (Fig. 6) was reconstructed from ony 27

7 Figure 6. Reconstruction of the Bahkauv statue from ony 27 images (a). The images were captured using a simpe hand-hed video camera on a roughy circuar path. Despite the specuar surface and other iumination artifacts we are abe to reconstruct 3D meshes of fairy high quaity (b), (c). However, some regions with a significant deviation from the Lambertian mode in the input images ead to sighty noisy surface properties (d). (e) shows a textured and reighted version of the mode. Figure 7. The dragon mode is particuary difficut to reconstruct because of the fine detais such as the caws as we as partiay specuar and transparent surface areas. We used 141 images (a) to increase the robustness of our technique and succeeded in reconstructing a fairy detaied mode (b). However, our Lambertian photo-consistency measure eads to visibe artifacts, e.g., for the fire which is made from coored gass and hence causes compex ight scattering effects. A reighted and textured version of the mode is shown in (c). Mode Bahkauv images. Despite the difficut acquisition conditions with non-lambertian, weaky textured surface properties and other iumination artifacts we achieve quite an acceptabe reconstruction with ess than 10 minutes overa computation time. Previous non-hierarchica approaches generay report reconstruction times in the range of one to severa hours for comparabe input compexity and reconstruction quaity on simiar reference systems. For the reconstruction of the Leo mode (Fig. 5) we used 86 images. Athough a quite acceptabe reconstruction can aready be obtained with about images, we used a higher number of images to better reconstruct detais such as the Leo s ears. With ess cameras they became sighty truncated due to their reativey sma image size and our imperfect camera caibration. The Dragon mode represents the most compex reconstruction in terms of object structure and surface properties. We used 141 input images for increased robustness and a starting eve 0 = 8 corresponding to a voumetric resoution of 2563 voxes to better carve out fine detais such as the caws in the initia surface proxy. Fig. 8 shows a reconstruction of the Warrior at a high resoution of Despite the reativey ow image resoution in comparison to the voumetric resoution our method is abe to reconstruct fine surface detais. For ground truth evauation we computed the Hausdorff distance [4] of our reconstruction to a aser scan, resuting in a very ow mean (max) error of 0.1% (1.9%) with respect to the bounding box diagona. Since we expicity generate the graph data structure for the surface computation, we used a machine with 4 GB of main memory and a reduced crust thickness for the fina surface computation step. Despite this high resoution the computa- Leo Warrior Warrior Dragon Images Leve 0 7(1283 ) 7(1283 ) 7(1283 ) 7(1283 ) 8(2563 ) Target eve 9(5123 ) 9(5123 ) 9(5123 ) 10(10243 ) 9(5123 ) Visua hu Visibiity Consistency Graph cut Meshing Overa 1.9 s 2.9 s 5.4 m 1.3 m 45.0 s 9.5 m 2.4 s 6.5 s 9.7 m 41.9 s 17.5 s 12.8 m 2.2 s 5.9 s 7.0 m 37.4 s 25.5 s 10.5 m 2.2 s 5.9 s 19.1 m 3.2 m 2.6 m 27.8 m 10.5 s 35.9 s 13.6 m 1.6 m 1.2 m 20.4 m Vertices 639 K 298 K 388 K 1570 K 575 K Tabe 1. The time and space compexity of each reconstruction is mainy infuenced by the number of input images, the initia eve 0, and the desired target resoution. We provide accumuated timings in seconds and minutes respectivey for each of the agorithm s main phases. The overa reconstruction time additionay incudes user interaction and other invoved processing steps such as norma computation and the crust generation. The number of mesh vertices at the target resoution is given in the ast row. tion times are sti quite acceptabe. However, one important goa of our future work wi be the reduction of the memory requirements of our method. Our used input data and resuts are avaiabe at 7. Concusion We presented a new agorithm for the reconstruction of a 3D shape from a set of video frames based on graph cut minimization. Our octahedra graph structure estabishes a we defined reationship between the photo-consistency of a voxe and the edge weights of an embedded octahedra

8 Figure 8. The warrior mode was reconstructed from 71 images (a) at a high voumetric resoution of The resuting mesh has a compexity of 1570 K vertices (b). (c) shows the textured and reighted mode. subgraph. Our agorithm is efficient, especiay if the number of input images and/or the voumetric resoution is arge, because our hierarchica procedure effectivey reduces the amount of voxes that have to be processed. This is achieved by using a coarse scae reconstruction to predict the region where the surface ies on the finer eve. The fina resut of the graph cut computation can easiy be converted into a cosed and guaranteed manifod triange mesh. The current main imitation of our method to compute resoutions higher than is the fact that we expicity generate the octahedra graph structure within the voxe grid, eading to a noticeabe memory overhead. This overhead coud be reduced by expoiting the specia structure of our graph, e.g., by using some indexing scheme in the voxe grid. The reconstruction quaity is furthermore imited by our currenty used Lambertian photo-consistency measure. Athough our method aows us to effectivey expoit a high number of input images to achieve robust reconstructions of difficut surfaces, we pan to investigate more sophisticated photo-consistency measures to reconstruct such surfaces more robusty. Finay we pan a more detaied evauation of our graph construction in comparison to the current standard approaches (e.g, [2]) in terms of graph compexity, accuracy of the reconstruction, and metrication artifacts. 8. Acknowedgements We woud ike to thank Martin Habbecke for his hep during the generation of the presented data sets. References [1] P. A. Beardsey, A. Zisserman, and D. W. Murray. Sequentia updating of projective and affine structure from motion. IJCV, 23(3): , [2] Y. Boykov and V. Komogorov. Computing geodesics and minima surfaces via graph cuts. In ICCV, pages 26 33, [3] Y. Boykov, O. Vekser, and R. Zabih. Fast approximate energy minimization via graph cuts. In PAMI, voume 23 (11), pages , November [4] P. Cignoni, C. Rocchini, and R. Scopigno. Metro: Measuring error on simpified surfaces. Comput. Graph. Forum, 17(2): , 1998, [5] C. H. Esteban. Stereo and Sihouette Fusion for 3D Object Modeing from Uncaibrated Images Under Circuar Motion. PhD thesis, Ecoe Nationae Supérieure des Téécommunications, May [6] O. Faugeras and R. Keriven. Variationa principes, surface evoution, PDE s, eve set methods and the stereo probem. In Image Processing, voume 7, pages , [7] A. Hornung and L. Kobbet. Robust and efficient photoconsistency estimation for voumetric 3d reconstruction. In ECCV, pages , [8] J. Isidoro and S. Scaroff. Stochastic refinement of the visua hu to satisfy photometric and sihouette consistency constraints. In ICCV, pages , [9] L. Kobbet, M. Botsch, U. Schwanecke, and H.-P. Seide. Feature sensitive surface extraction from voume data. In SIGGRAPH, pages 57 66, [10] V. Komogorov and R. Zabih. What energy functions can be minimizedvia graph cuts?. PAMI, 26(2): , [11] V. Komogorov, R. Zabih, and S. J. Gorter. Generaized muti-camera scene reconstruction using graph cuts. In EMMCVPR, pages , [12] K. N. Kutuakos and S. M. Seitz. A theory of shape by space carving. IJCV, 38(3): , [13] M. Lhuiier and L. Quan. Surface reconstruction by integrating 3D and 2D data of mutipe views. In ICCV, pages , [14] M. Li, M. Magnor, and H.-P. Seide. Hardware-acceerated rendering of photo hus. Computer Graphics Forum, 23(3): , September [15] H. Lombaert, Y. Sun, L. Grady, and C. Xu. A mutieve banded graph cuts method for fast image segmentation. In ICCV, pages , [16] S. Paris, F. Siion, and L. Quan. A surface reconstruction method using goba graph cut optimization. In ACCV, [17] C. Rother, V. Komogorov, and A. Bake. grabcut : interactive foreground extraction using iterated graph cuts. In SIGGRAPH, pages , [18] S. M. Seitz and C. R. Dyer. Photoreaistic scene reconstruction by voxe cooring. In CVPR, pages , [19] S. Sinha and M. Poefeys. Muti-view reconstruction using photo-consistency and exact sihouette constraints: A maximum-fow formuation. In ICCV, [20] D. Snow, P. Vioa, and R. Zabih. Exact voxe occupancy with graph cuts. In CVPR, June [21] G. Taubin. A signa processing approach to fair surface design. In SIGGRAPH, pages , [22] G. Vogiatzis, P. Torr, and R. Cipoa. Muti-view stereo via voumetric graph-cuts. In CVPR, pages , 2005.