EVen though remarkable progress has been made in

Transcription

1 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST Dene Semantic 3D Recontruction Chritian Häne, Chritopher Zach, Andrea Cohen, and Marc Pollefey, Fellow, IEEE Abtract Both image egmentation and dene 3D modeling from image repreent an intrinically ill-poed problem. Strong regularizer are therefore required to contrain the olution from being too noiy. Thee prior generally yield overly mooth recontruction and/or egmentation in certain region while they fail to contrain the olution ufficiently in other area. In thi paper, we argue that image egmentation and dene 3D recontruction contribute valuable information to each other tak. A a conequence, we propoe a mathematical framework to formulate and olve a joint egmentation and dene recontruction problem. On the one hand knowing about the emantic cla of the geometry provide information about the likelihood of the urface direction. On the other hand the urface direction provide information about the likelihood of the emantic cla. Experimental reult on everal data et highlight the advantage of our joint formulation. We how how weakly oberved urface are recontructed more faithfully compared to a geometry only recontruction. Thank to the volumetric nature of our formulation we alo infer urface which cannot be directly oberved for example the urface between the ground and a building. Finally, our method return a emantic egmentation which i conitent acro the whole dataet. Index Term Volumetric Recontruction, Semantic Labeling, Convex Formulation, Multi-Label Segmentation, Semantic 3D Modeling 1 INTRODUCTION EVen though remarkable progre ha been made in recent year, both image egmentation and dene 3D modeling from image remain intrinically ill-poed problem. The tandard approach to addre thi ill-poedne i to regularize the olution by introducing a repective prior. Traditionally, the prior enforced in image egmentation approache are tated entirely in the 2D image domain (e.g. a contrat-enitive patial moothne aumption), wherea prior employed for image-baed recontruction typically yield piece-wie mooth 3D urface, a their olution. In thi paper we demontrate that joint image egmentation and dene 3D recontruction i beneficial for both tak. While the advantage of a joint formulation for egmentation and depth etimation have already been oberved and utilized in [21], our main contribution i the introduction of a rigorou mathematical framework to formulate and olve a joint optimization for dene 3D recontruction and cla egmentation. We extend volumetric cene recontruction method, which egment a volume of interet into occupied and free-pace region, to a multi-label volumetric egmentation framework aigning emantic clae or a free-pace label to voxel. Thu, interface of the volumetric egment correpond to urface of the recontructed object. Thereby the interface between free-pace and the emantic clae decribe the viible, obervable urface and interface between different emantic clae, for example between ground and building, decribe the hidden non- C. Häne i with the Department of Electrical Engineering and Computer Science, Univerity of California, Berkeley, United State of America, chaene@eec.berkeley.edu C. Zach i with the Cambridge Reearch Laboratory of Tohiba Reearch Europe, Cambridge, United Kingdom, chzach@crl.tohiba.co.uk A. Cohen i with the Department of Computer Science, ETH Zürich, Switzerland, andrea.cohen@inf.ethz.ch M. Pollefey i with the Department of Computer Science, ETH Zürich, Switzerland and with Microoft, Redmond, United State of America, E- mail: marc.pollefey@inf.ethz.ch Manucript ubmitted May 31, 2016, revied September 8, 2016 Fig. 1. (Top) from left to right: Example of input image, bet label image egmentation reult, depthmap. (Bottom) our propoed joint optimization combine cla egmentation and geometry reulting in an accurately labeled 3D recontruction obervable urface. On the one hand, uch a joint approach i highly beneficial ince the aociated appearance (and therefore a likely emantic category) of urface element can influence the patial moothne prior. Thu, a cla-pecific regularizer guided by image appearance can adaptively enforce patial moothne and preferred orientation of 3D urface. Thi allow our method to correctly recover weakly oberved urface and even etimate non-obervable interface between two different emantic clae, for example between ground and building. On the other hand, denely recontructed model induce image egmentation

2 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST which are guaranteed to correpond only to geometrically meaningful object in 3D. Hence, the egmentation reult are by contruction conitent acro multiple image. In a nuthell, we propoe to learn appearance likelihood and cla-pecific geometry prior for urface orientation from training data in an initial tep. Thee data-driven prior can then be ued to define unary and moothne potential in a volumetric egmentation framework, complementary to the meaured evidence acquired from depth map. While optimizing over the label aignment in thi volume, the image-baed appearance likelihood, depth map from computational tereo, and geometric prior interact with each other yielding an improved dene recontruction and emantic labeling. Thi paper i baed on our earlier publication [13]. Compared to the original publication of the method, we propoe a lightly modified data term in Section 3, which enhance the quality of the reult. We extended the line egment Wulff hape (c.f. Section 4.3) to be aymmetric, allowing u to better repreent the geometric prior. We furthermore conduct experiment with a econd emantic claifier (c.f. 4.1) compared to [13], which how that our method i not bound to a pecific image baed emantic claifier. Additional detail on how the numerical optimization algorithm i applied are included. Moreover, all the reult in thi paper are generated from cratch and are qualitatively coniderably better than the earlier reult from [13]. Alo, additional convergence experiment and many cloe-up of the recontructed model howing how our method i able to recover weakly and unoberved urface, have been added to the manucript. The remainder of thi ection give an overview of the related work. Section 2 explain how the multi-label formulation from [41] i applied to form our framework for dene emantic 3D recontruction. The data term i explained in Section 3. The core of our method, the clapecfic geometric prior, are explained in Section 4. The numerical minimization i explained in Section 5 and a thorough experimental evaluation i given in Section Related Work There i a vat literature on dene 3D modeling from image. Here we ketch only a mall ubet of related literature, and refer e.g. to the Middlebury MVS evaluation page [35] for a broader urvey. Given a collection of depth image (or equivalently denely ampled oriented 3D point) the method propoed in [24], [40], [44] eentially utilize the urface area a regularization prior, and obtain the final urface repreentation indirectly via volumetric optimization. One main difference between [24] and [40], [44] i the utilization of a combinatorial graph-cut formulation in the former, wherea [40], [44] employ a continuouly inpired numerical cheme. The regularization prior in thee work i iotropic, i.e. independent of the urface normal (up to the impact of the underlying dicretization), correponding to a total variation (TV) regularizer in the volumetric repreentation. The work of [18] utilize an aniotropic TV prior for 3D modeling in order to enforce the conitency of the urface normal with a given normal field, thu better preerving high frequency detail in the final recontruction. All of the above mentioned work on volumetric 3D modeling from image return olely a binary deciion on the occupancy tate of a voxel. Hence, thee method are unaware of typical cla-pecific geometry, uch a the normal of the ground plane pointing upward. Thee method are therefore unable to adjut the utilized moothne prior in an object- or cla-pecific way. Thi obervation led to the initial motivation for the preent work. More pecifically, it i notoriouly difficult to faithfully recontruct weakly or indirectly oberved part of the cene uch a the ground, which i uually captured in image at very lanted angle (at leat in terretrial image data). [16] propoe to extend an adaptive volumetric method for urface recontruction in order not to mi important part of the cene in the final geometry. The aumption in their method i that urface with weak evidence are likely to be real urface if adjacent to trongly oberved free-pace. A key property of our work i that weakly upported cene geometry can be aited by a cla-pecific moothne prior. If only a ingle image i conidered and direct depth cue from multiple image are not available, aigning object categorie to pixel yield crucial information about the 3D cene layout [15], [34], e.g. by exploiting the fact that building facade are uually vertical, and the ground plane i typically horizontal. Thee relation are generally not manually encoded, but extracted from training data. Such known geometric relation between object categorie can alo be helpful for 2D image egmentation, e.g. by auming a particular layout for indoor image [26], a tiered layout [9] or cla-pecific 2D moothne prior [38]. Utilizing appearance-baed pixel categorie and tereo cue in a joint framework wa propoed in [21] in order to improve the quality of obtained depth map and emantic image egmentation. In our work, we alo aim on joint etimation of 3D cene geometry and aignment of emantic categorie, but ue a completely different problem repreentation which i intrinically uing multiple image and olution method. [3], [31] alo preent joint egmentation and 3D recontruction method, but the determined egment correpond to individual object (in term of an underlying mooth geometry) rather than to emantic categorie. Concurrently to the original reearch [13] that lead to thi paper, a method uing emantic information for dene object recontruction in form of hape prior [1] ha been developed. Alo [17] wa publihed hortly after the earlier verion of our work. They ue RGB-D data a an input and cat the recontruction problem a emantic labeling of voxel. Compared to our method, they only aign emantic clae to voxel on the viible urface wherea our approach dene label the whole recontruction volume. In the remainder of thi ection, we dicu ome work which were publihed after the initial verion of our approach. The approach of [19] recontruct treet cene from a moving monocular camera uing a dicrete graph-baed formulation over a voxel grid. The method in [39] generate a truncated igned ditance field from depth map uing voxel hahing [28] and ubequently aign emantic label to the voxel around the urface uing an efficient optimization uing a conditional random field formulation. Follow up work of our method extend our method to work with a ray potential data term [32], [33] and on octree [2].

3 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST x 0 = 1 y 0,2 y 0,1 x 1 = 1 y 1,2 y 0,1 x 3 = 1 x 2 = 1 y 0,3 Fig. 2. Illutration of the formulation in the 2D cae. 2 JOINT FORMULATION Ω free pace building ground vegetation clutter In thi ection we decribe the underlying energy formulation for our propoed joint urface recontruction and claification framework and it motivation. Similar to previou work on global urface recontruction, we lift the problem from an explicit urface repreentation to an implicit volumetric one. The increaed memory conumption i compenated by the advantage of allowing arbitrary but cloed and oriented topology for the reulting urface. Our final energy i a variant of the continouly inpired convex multi-label formulation preented in [41]. We firt introduce the initial continou formulation and then tate the dicretized verion of the energy ued in thi manucript. 2.1 Continuou Formulation We cat the ultimate goal of emantically guided hape recontruction a a volumetric labeling problem, where one out of L + 1 label i aigned to each location z Ω in a continuou volumetric domain Ω R 3. In the following we will ue indice i and j for label. Allowed label are free/empty pace (with numeric value 0) and occupied pace with an aociated emantic category (value from {1,..., L}). The label aignment will be encoded with L + 1 indicator function x i : Ω [0, 1], i {0,..., L}: x i (z) = 1 iff label i i aigned at z Ω. Note that in the following, the dependence of all quantitie on the 3D location z will be indicated with a ubcript to be more conitent with the later dicrete formulation, i.e. x i (z) = x i z. With thi notation in place, the convex relaxation, originating from [5], [30], of the labeling problem in a continuou volumetric domain Ω read a E cont (x, y) = ρ i zx i z + φ ij z (yz ij ) dz, (1) Ω i i,j:i<j where y ij : Ω [ 1, 1] 3, i {0,..., L} with i < j are tranition gradient. The tranition gradient decribe the boundarie between label pair. They are vanihing in the interior of (emantic) egment and are otherwie normal to the interface boundary. A viual illutration of the variable type for a 2D example i given in Figure 2. To enure that the label indicator function agree with the label tranition gradient they need to atify the following marginalization contraint z x i = y ij y ji. (2) ρ i j:i<j j:j<i : Ω R encode the local preference for a particular label. Note that the moothne term in Equation 1 i an extenion of the tandard length/area-baed boundary regularizer to Finler metric (ee e.g. [8], [27], [38], [41]) and the infiniteimal length function φ ij z : R 3 R + 0 are naturally extended from S 2 to R 3, rendering φ ij z a convex and poitively 1-homogeneou function. Such choice of φ ij z generalize the notion of total variation to location and orientation dependent penalization of egmentation boundarie. In addition to the marginalization contraint in Equation 2, the function x i alo need to atify the normalization contraint, i xi = 1, and non-negativity contraint. See [41] for a detailed derivation and theoretical interpretation of thi energy. A minimizer (x, y) induce a partition of Ω into free-pace and repective object categorie. The boundarie between the individual region form the 3D urface of interet. 2.2 Dicretized Formulation A diadvantage of thi continuou energy formulation i that the cla of moothne prior φ ij i retricted to metric under reaonable aumption (ee e.g. [23]). Conequently, we focu our attention on dicrete lattice (i.e. regular voxel grid) a underlying domain where thee retriction do not apply (ee [41]). Hence, Ω denote a finite voxel grid with voxel Ω in the following. A dicrete verion of the continuou energy in Equation 1 not requiring a metric prior read a (c.f. [41]) E dicr (x) = ( ρ i x i + ) φ ij (x ij x ji ) (3) Ω i i,j:i<j ubject to the following marginalization, normalization and non-negativity contraint, x i = j x i [0, 1], (x ij ) k, x i = j x i = 1, i (x ji e k ) k k {1, 2, 3} x ij 0. (4) e k R 3 denote the k-th canonical bai vector and ( ) k i the k-th component of a vector. The dicrete marginalization contraint above follow from Equation 2 by employing a forward finite-difference cheme for the patial gradient. The variable appearing in Equation 3 have the following interpretation in the context of joint urface recontruction and egmentation tak: x i [0, 1] encode whether label i (i.e. free-pace or one of the olid object categorie) i aigned at voxel, x ij ρ i x ji [ 1, 1] 3 repreent the local urface orientation (i.e. the boundary normal) if it i nonzero, i the unary data term encoding the meaured evidence, i.e. the preference of voxel for a particular label i. Thi data term capture the evidence from

4 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST two ource: firtly, the meaurement from a et of depth map, and econdly, appearance-baed claification core from the input image a obtained from previouly trained claifier. Section 3 decribe in detail how thi unary term i modeled. Finally, φ ij i the location and direction-dependent moothne prior indicating the local compatibility of a boundary between label i and j. Hence, thee prior encode the previouly mentioned clapecific geometric prior. Of highet importance i the directly obervable boundary between free-pace and any of the object categorie. Modeling φ ij from training data i explained in Section 4. We will retrict ourelve to homogeneou prior in the following, i.e. the local moothne contribution φ ij (x ij x ji ) doe not depend on, and the objective in Equation 3 lightly implifie to E dicr (x) = ( ρ i x i + ) φ ij (x ij x ji ). (5) Ω i i,j:i<j The rationale behind the patial homogeneity aumption i that only the orientation of a boundary urface and the affected label are of importance, but not the precie location. While thi hold for general outdoor cene conidered in thi work, it i not the cae anymore when looking at individual object. In [12] we preented how patially varying moothne can be ued to define cla pecific 3D object hape prior. The method in [12] wa inpired by the reult we got with the method preented in thi paper and wa formulated within the ame framework. Once the value of ρ i are determined and the moothne prior φ ij are known, the tak of inference i to return an optimal volumetric labeling. Since we employ a convex problem tated in Equation 3, any convex optimization uitable for non-mooth program can be utilized. In Section 5 we briefly outline the numerical cheme ued in our experiment. 3 THE RAY LIKELIHOOD AND ITS APPROXIMATION In thi ection we decribe how available depth map (with potentially miing depth value) and appearance-baed cla likelihood are converted into repective unarie ρ i for joint volumetric recontruction and claification a decribed in the previou ection. A completely ound graphical model relating image obervation with occupancy tate of 3D voxel require obervation likelihood correponding to clique potential with entire ray in 3D forming clique (e.g. [25]). In the following we argue that, under uitable moothne aumption on the olution, we can approximate the higher-order clique potential by uitable unary one. We aim on factorizing the clique potential into only unary term uch that the induced (higher-order) cot of a particular boundary urface i approximated by the unarie. Additionally, we employ the uual aumption of independence of obervation acro image. Thi mean, that the unary potential decribed below baed on color image (and aociated depth map) are accumulated over all image to obtain the effective unary potential. In the following we conider a particular pixel p in one of the input image (repectively depth map, ince we aume that depth image are etimated w.r.t. color image a their reference view). The pixel p induce a ray in 3D pace, which lead to a et of travered voxel ray(p) and the correponding latent variable x i and their aociated unary potential ρ i. Recall that i indexe one of the L + 1 emantic categorie {0, 1,..., L} with 0 correponding to ky (i.e. free-pace) and i indicating object category i, repectively. Our tak i to (approximately) model the likelihood ( ) P ˆd(p), Â(p) {x i } ray(p), (6) where ˆd(p) i the oberved depth at pixel p (which may be miing), and Â(p) encapulate the local image appearance in the neighborhood of p. Note that in term of a graphical model the repective potential, log P ( ˆd, Â {x i ) } ray(p), depend on the entire clique { : ray(p)}. Clearly, for a particular ray the likelihood of oberving ˆd(p) and Â(p) only depend on the firt croing from free-pace to occupied pace. Neverthele, proper handling of voxel viibility link all voxel along the ray to form a clique. For notational convenience we will drop the dependence on the pixel p, and alo index voxel along ray(p) by their depth d with repect to the current view. We will ubtantially implify the potential (and therefore the inference tak) by conidering the cae where we have a depth meaurement and the cae where ky i the mot likely label. The rational behind thi ditinction i that having a trong repone for the ky label in the appearance baed claifier indicate that the likelihood for the whole ray being freepace hould increae. Thi i the cae regardle of the preence of a depth meaurement. However, an actual ignal of where the urface hould be, can only be extracted if a depth meaurement i given. 3.1 Oberved depth Thi i the cae when ˆd in the depth map i valid (i.e. not miing). In thi cae we aume that Equation 6 factorize into P ( ˆd, Â voxel d i firt croing to i ) = P ( ˆd d)p (Â i), where P ( ˆd d) capture the noie for inlier in the depth ening proce and i uually a monotonically decreaing function of d ˆd. P (Â i) i induced by the confidence of an appearance-baed claifier for object category i. For thi cae we only take into account P (Â i) for olid clae i, meaning i > 0. In the following we will firt tate the unary potential and then explain the reaon for the pecific choice. We only define non-zero unarie for voxel along the ray near the oberved depth. Aume that the inlier noie of depth etimation i bounded by δ, and we denote by ˆd±δ the voxel along the ray with ditance δ and δ, repectively. We et the unary potential L ρ iˆd+δ = σ cla i, ρ 0ˆd+δ i=1 = σ cla i { L 0 for i = 0 ρ i d = η( ˆd d) for i > 0. (7)

5 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST β 0 weight weight β 0 ˆd δ ˆd δ ˆd urface ˆd σ cla 1 ˆd + δ σ cla 2 Fig. 3. Unarie aigned to voxel along a particular line-of-ight for two different olid clae. for voxel d near the oberved depth, i.e. voxel d cloer to ˆd than δ. Here σ cla i = log P (Â i). The function η : [ δ, δ] R i independent of the object category i and reflect the noie aumption of ˆd. We chooe η( ˆd d) = β gn( ˆd d) for β > 0, correponding to an exponentially ditributed noie for depth inlier. Recall that for a depth inliner the recontructed urface i inide the band [ δ, δ] around the oberved depth ˆd. Inerting unarie only near the oberved depth correpond to truncating the cot function, hence we aume exponentially ditributed inlier and uniformly ditributed outlier depth value. See Figure 3 for an illutration of unarie along the ray. Since we enforce patial moothne of the labeling (i.e. multiple croing within the narrow band near ˆd are very unlikely), we expect three poible configuration for voxel in [ ˆd δ, ˆd + δ] decribed below. The firt configuration correpond to the cae where the input depth value i conidered an inlier. The econd and third cae correpond to a depth outlier. For thee two cae we want to avoid that the information from the emantic claifier add a bia toward either free or occupied pace to the olution. We now tate for each configuration the contribution of unary term for the particular ray to the complete energy. ˆd + δ 1) In the labeling of interet we have that free-pace tranition to a particular object cla i at depth d. Hence, x 0 = 1 for [ ˆd δ, d) and x i = 1 for [d, ˆd + δ]. Summing the unarie according to Equation 7 over the voxel in [ ˆd δ, ˆd + δ] yield σ cla i + η( ˆd d ), d [d, ˆd+δ] L i=1 σ cla i L. i.e. the negative log-likelihood of oberving the appearance category i in the image and the one correponding to the depth noie aumption. Note that the econd term, d η( ˆd d ) will be nonpoitive and therefore lower the overall energy. Thi beneficial term i not appearing in the other cae below. 2) If all voxel in the particular range [ ˆd δ, ˆd + δ] are free-pace (x 0 = 1 for all the voxel in thi range), we have an outlier depth meaurement. The contribution to the total energy in thi cae i 3) All voxel in the range are aigned to object label i (i.e. x i = 1 for [ ˆd δ, ˆd + δ]. Thu, the contribution to the energy i σ cla i in thi cae. Overall, our choice of unarie will faithfully approximate the deired true data cot in mot cae. Since camera center are in free-pace by definition, we add a light bia toward free-pace along the line-of-ight from the repective camera center to the oberved depth (i.e. voxel in the range [0, ˆd δ]). Thi ha alo a poitive influence on the convergence peed. However, thi bia need to be choen very mall becaue it doe not handle outlier in the depth map. If a depth meaurement i an outlier with a much larger depth than the true urface a lot of free-pace weight can be added to the grid which i not compenated by any correct meaurement. Remark 1. Often the input depth map are computed baed on (multi-view) tereo matching and hence the inlier noie ditribution could more accurately be defined on invere depth (diparitie) intead of depth. Due to the voxel reolution thi introduce an additional parameter which enure that the uncertainty i big enough with repect to the voxel ize in order to enure that the weight get pread into multiple voxel (ee [14]). In our experiment we refrained from uing thi more complex model becaue we were able to achieve better reult uing a fixed ize uncertainty window defined on the depth. 3.2 Sky If the bet cot label for a particular pixel p i ky we add an additional contribution to the unary term preferring freepace along the whole ray. Thi contribution to the data cot i helpful to avoid bleeding of building etc. beyond their repective ilhouette in the image: ρ 0 = γ min { 0, σ ky min i ky σ i }, (8) with γ > 0 a uitable weight, and ρ i = 0 for i > 0 for all voxel along ray(p). In addition to the data dependent unary term it i beneficial to ue a light prior for voxel being occupied in the whole oberved pace, i.e. the voxel which lie inide the viewing frutum of at leat one depth map. The motivation for thi i that uually the whole free pace can be oberved by uitable camera location, but for the occupied pace thi i not poible in the preence of large object. For example a building that can only be oberved from the outide. Remark 2. The choice for the data term in our propoed joint fuion can be een a an extenion of the data term ued for the TV-Flux fuion from [40]. Leaving out the emantic apect from the data term i.e. only taking into account the cae for oberved depth and leaving out the emantic claifier core reult in the ame data term a we ue in our implementation of the TV-Flux fuion (c.f. [40]). 4 TRAINING THE PRIORS In thi ection, we will explain how the appearance likelihood ued in the unary potential ρ i and the cla-pecific geometric prior φ ij are learned from training data. While the appearance term are baed on claification core of a tandard claifier, training of geometric prior from labeled data i more involved. We firt tart decribing the training of the appearance likelihood before dicuing the training procedure for moothne prior.

6 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST Appearance Likelihood In order to get claification core for the label, a uitable claifier need to be trained on a manually labeled training dataet. Our method doe not rely on a pecific claifier a long a pixel-wie likelihood for the label can be extracted from the input image. Conequently, we conducted experiment with two different claifier. The firt claifier i a booted deciion tree claifier from the STAIR Viion Library [10]. In a firt tep, the training image are egmented into uper-pixel uing the mean hift egmentation algorithm 1. Feature are extracted for each uperpixel. We ue the default parameter a implemented by [10], reulting in 225 dimenional feature vector baed on color, intenity, geometry, texture and location. It hould be noted that the geometry and location feature are extracted by uing 2D information on the image (uper-pixel ize, hape, and relative poition in the image) and they are not related to the 3D geometry of the cene. The econd claifier i from the Automatic Labeling Environment (ALE) [20]. It ue TextonBoot [36] feature jointly with a egment-baed feature repreentation of [22]. TextonBoot are pixel-wie, context-baed feature repreented a a bag-of-word in a et of randomly ampled rectangle around a pixel. The main drawback of uch a context-baed feature repreentation i that it doe in general not nicely follow object boundarie. The pixel-wie, context-baed feature vector i augmented with a feature vector over uper-pixel. Thi lead to claifier repone that nicely follow object boundarie and at the ame time are able to benefit from the context beyond a ingle uper-pixel. Both claifier are trained uing manually labeled ground truth image over 5 clae: ky, building, ground, vegetation, and clutter. We deigned a training dataet by taking 76 image from the CamVid dataet [4] and 101 image from the MSRC dataet [36]. We alo added 34 image taken at treet level of different building. Thee building are not part of the evaluation data et. Once the claifier i trained, it can be ued to obtain core for each pixel of the image. Thee core repreent the log-likelihood of each cla for each region of the image. 4.2 Cla-Specific Geometric Prior We ue a parametric model for the function φ ij appearing in the moothne term of Equation 5. A already mentioned, we retrict ourelve to patially homogeneou function, and thu there i no dependency on the location. Note that the energy formulation in Equation 5 naturally correpond to a negative log-probability. Hence, the function φ ij will be alo interpreted a negative logprobabilitie. Let ij denote a tranition event between label i and j, and let n ij denote an oriented tranition event (with unit urface area) between label i and j with (unitlength) normal direction n. Intead of modeling φ ij directly, we ue 1. OpenCV implementation P (n ij ) = P (n ij ij )P ( ij ), (9) where we applied the homogeneity aumption, i.e.p ( ij ) = P ( ij). The conditional probability, P (n ij ij ) i now modeled a a Gibb probability meaure P (n ij ij ) = exp ( ψ ij (n ij ) ) /Z ij, (10) for a poitive 1-homogeneou function ψ ij. Z ij i the repective partition function, Z ij def = n S exp ( ψ ij (n ij ) ) dn, 2 and S 2 i the 3-dimenional unit phere. Conequently, φ ij in Equation 5 i now given by φ ij (n) = ψ ij (n) + log Z ij log P ( ij ) (11) for a unit vector n S 2. For the next tep we aume that the probabilitie and hence the function φ ij are parametrized by parameter θ. The particular choice of parameter i given in Section 4.3. Maximum-likelihood etimation i ued to fit the parameter to available training data, formally θ = arg max θ P (n ij ij, θ)p ( ij θ), (12) k i,j where the product goe over all training ample k and ψ ij and Z ij are function of the parameter θ ij which are gathered in θ = {θ ij i, j {0,..., L}}. In our implementation, we etimate the dicrete probabilitie P ( ij ) of oberving a tranition ij upfront by counting the relative frequencie N ij / i,j N ij of the repective type of boundarie from training data. Etimating firt P ( ij ) ha the advantage that the ML-etimation in Equation 12 decouple into independent etimation problem of the form N ij θ ij = arg min ψ ij (n ij θ ij k ; θij ) + N ij log Z ij (θ ij ), (13) k=1 where the ummation goe over all the N ij tranition ample n ij k between label i and j. Since for many choice of ψij the partition function cannot be olved analytically, we ue Monte Carlo integration to obtain an etimate for Z ij. Given the low dimenionality of θ ij (up to 4 component, ee Section 4.3 below) and the neceity of Monte Carlo integration for the partition function, we ue a imple grid earch to find an approximate minimizer θ ij. A training data we ue building extracted from a 3D cadatral city model. We egmented the building into the individual tranition (ee Figure 4) which enable u to train ψ ij for the tranition ground free-pace, ground building and building free-pace. The tranition between ground building i chooen to be the part of the building within the ground. Thi choice help to faithfully recontruct cae where for example, ouide taircae going down to baement can be oberved. In our experiment the data from 5 building wa ufficient for the training thank to the mall number of parameter that need to be etimated. Label tranition unoberved in the training data are defined manually. At thi point we need to addre two mall technical iue: Remark 3. φ ij i only pecified for unit vector n S 2, but the argument in the energy model Equation 5 are uually non-normalized gradient direction y ij def = x ij x ji [ 1, 1] 3. The difference between φ ij and the regularization in Equation 5 can be een a the difference between the integral over the urface and the integral over the volume

7 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST Fig. 4. (Top row) A ection of the cadatral 3D city model. (Bottom row) A egmented example building ued for the training of the geometric prior. (Left) building free-pace (overground part of the building), (middle) ground building (underground part of the building), (right) ground free-pace (part of the ground not covered by the building). (c.f. [8]). Remember that ψ ij i a convex and poitively 1- homogeneou function. Together with the fact that the area of the urface element (in finite-difference dicretization) i captured exactly by y ij 2 (i.e. the total variation), we derive the contribution of y ij to the regularizer a y ij 2 φ ij ( y ij / y ij 2 ) = φ ij (y ij ) (14) by the 1-homogeneity of φ ij. Therefore, the extenion of φ ij a given in Equation 11 to arbitrary argument y R 3 i φ ij (y) = ψ ij ( (y) + y 2 log Z ij log P ( ij ) ). (15) }{{} def =C ij Conequently, our moothne prior φ ij will alway be compoed of an aniotropic, direction-dependent component ψ ij and an iotropic contribution proportional to C ij = log Z ij log P ( ij ). Thi alo implie that there i no need to explicitly model any iotropic component in ψ ij. Remark 4. The function φ ij given in Equation 15 above i poitively 1-homogeneou if ψ ij i, but convexity can only be guaranteed whenever C ij = log Z ij log P ( ij ) 0 or P ( ij ) Z ij. Thi i in practice not a evere retriction, ince for a ufficiently fine dicretization of the domain the occurrence of a boundary urface i a very rare event and therefore P ( ij ) Choice for ψ ij We need to retrict ψ ij to be convex and poitively 1- homogeneou. One option i to parametrize ψ ij (n) = ψ ij (n; θ) in the primal and to limit θ uch that the reulting ψ ij ha thee propertie, but thi may be difficult in general. We chooe a lightly different route and parametrize the convex conjugate of ψ ij, ( ψ ij), ( ψ ij ) (p) = max n { p T n ψ ij (n) } = ı Wψ ij (p), i.e. the indicator function for a (convex) hape W ψ ij called Wulff hape (c.f. [8], [43], [45]). We find it eaier to model parametric convex Wulff hape W ψ ij rather than ψ ij directly. Below we decribe the utilized Wulff hape and it parametrization. Which Wulff hape i picked for ψ ij (in addition to it continuou parameter) i part of the ML etimation. The decription below i for Wulff hape in a canonical poition, ince any ψ induced by a rotated hape can be expreed uing a canonical one, ψ(n; R) = max p T n = max (Rp) T n = ψ(r T n; I). p R W ψ p W ψ Given Remark 3 above, there i no need to model the Wulff hape with an iotropic and an aniotropic component (i.e. a a Minkowki um of a phere and ome other convex hape). The Wulff hape decribed below are deigned to model two frequent urface prior encountered in urban environment: one prior favor urface normal that are in alignment with a pecific direction (e.g. ground urface normal prefer to be aligned with the vertical direction), and the econd Wulff hape favor urface normal orthogonal to a given direction (uch a facade urface having generally normal perpendicular to the vertical direction). In order to obtain a dicriminative prior, we aume that the vertical direction of the input data i provided and we align the poitive z-axi with the upward pointing vertical direction. Thi in turn mean that the prior hould be invariant to rotation around the z-axi. Therefore we only allow choice for R which preerve thi invariance. A graphical illutration of the Wulff hape i given in Figure Line Segment Thi Wulff hape W ψ ha two parameter l 1 0 and l 2 0, it i an aymmetric line egment in z-direction centered at the origin with endpoint (0, 0, l 1 ) T and (0, 0, l 2 ) T. With defining n = (n 1, n 2, n 3 ) T, thi hape tranlate to a function { l1 n 3 if n 3 0 ψ(n) = (16) l 2 n 3 otherwie.

8 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST and maximized with repect to the dual variable ν, µ, λ. The update of the primal and dual variable are traightforward: gradient tep are followed either by projection to the repective feaible domain (x i 0, µ ij W ψ ij ) or the following proximity tep, Fig. 5. Viualization of the 2D verion of the ued Wulff hape (line egment and half-phere plu pherical cap). The red line depict the Wulff hape. The black line are polar plot of the function ψ. The ditance of a point on the black curve to the origin i the value that the function ψ(n) attain for a normal vector n in the direction of the point (viualized in blue) Half-phere plu pherical cap Thi Wulff hape W ψ conit of a half-phere with radiu r centered at the origin in oppoition to a pherical cap with height h. The correponding function ψ favor direction pointing upward and iotropically penalize downward pointing normal. ψ can be computed in cloed form (with n = (n 1, n 2, n 3 ) T ), r n ( ) ( ) if n 3 0 ψ(n) = n r 2 2h + h 2 n r 2 3 2h h 2 if ( ) r ( n1 n 2 ) otherwie, where ( ) i n 3 > 0 and n 3 (h 2 + r 2 ) > n (r 2 h 2 ). By contruction W ψ i convex (and therefore alo ψ) a long a r 0 and h [0, r]. 5 INFERENCE The objective in Equation 5 together with the linear contraint from Equation 4 form a nonlinear and non-mooth convex program and can be minimized e.g. by proximal plitting method. We ue the primal-dual approach [6], [29], and introduce Lagrange multiplier, ν for the contraint i xi = 1, (λ i ) k for x i = j (xij ) k and (λ i ) k for x i = j (xij e k ) k, k {1, 2, 3}, correponding to the three coordinate axe. We further partially dualize ψ ij via ψ ij (x ij x ji ) = max µ ij W ψ ij ( µ ij ) T ( x ij x ji thereby introducing additional dual variable µ ij. Overall, the utilized addle-point formulation read a E S-P (x, ν, µ, λ) = ( )) ( i ρi x i + ν i xi 1 +,i,j:i<j + ( λ i ),i,k ( (µ ij ) T ( x ij k x ji ) + C ij x ij ( x i ) j (xij ) k ), x ji ) 2 + ( λ i ) ( x i k ) j (xij e k ) k, (17),i,k ubject to x i 0, x ij 0 and µ ij W ψ ij. The addlepoint energy i minimized with repect to the primal variable x ij prox τf ( x, x ji ) = arg min x ij,x ji 1 2τ x ij + C ij x ij x ij τ x ji x ji 2 + ı{x ij x ji 2 2 0, x ji 0}. Intead of olving thi proximity tep, we lightly modify the objective E S-P a follow: ince w.l.o.g. ome minimizer of E dicrete (Equation 5) will atify the complementarity of x ij and x ji (i.e. (x ij ) T x ji = 0, ee [41] for a detailed explanation), we may replace C ij x ij x ji 2 in E S-P with ( C ij x ij ) 2, x ji leading to a much impler ubproblem ij prox τf ( x, x ji ) = 1 arg min x ij,x ji 2τ x ij x ij τ ( ) + C ij x ij x ji + ı{x ij 2 x ji x ji 2 2 0, x ji 0}, which correpond to firt truncating the negative part of the variable to 0 and ubequently executing a hrinkage tep in R 6 correponding to the proximity operator of the l 2 norm. 6 EXPERIMENTS In thi ection we preent the reult obtained on four challenging real world dataet. We compare our geometry to a tandard volumetric fuion (in particular TV-Flux [40]). In our implementation, thi correpond to leaving out the emantic part from the dataterm and regularizing the depthonly dataterm with an iotropic urface area penalization. We alo illutrate the improvement of the cla egmentation compared to a ingle image bet-cot egmentation. In general the claifier repone from ALE [20] are of better quality than the one obtained uing the STAIR Viion Library [10]. Therefore, mot of our experiment are conducted uing the ALE claifier. Reult uing the STAIR Viion Library are retricted to Section 6.3. Further, we illutrate the evolution of the iterative optimization procedure and we analyze how the convergence time change with the voxel reolution. 6.1 Overview of the Reult We ue the dataet Catle P-30 from [37] and three additional urban dataet (Southbuilding, Providence and Catania). Camera poe where obtained with the publicly available tructure from motion pipeline [42]. The depth map are computed uing our publicly available plane weep tereo matching implementation [11], for each of the image with zero mean normalized cro correlation (ZNCC) matching cot. Up to 20 image are matched to the reference view imultaneouly with bet K occluion handling uing the

9 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST Fig. 6. Reult for 4 dataet. (From left to right) Catle-P30 [37], Southbuilding, Providence, Catania. (From top to bottom) Example input image, example depth map, raw image labeling, our propoed joint fuion reult, TV-Flux fuion reult; The different cla label are depicted uing the following color cheme: building red, ground dark gray, vegetation green, clutter light gray. bet two matching core. In order to reduce the influence of the fronto-parallel bia of the plane weeping we run the plane weeping in multiple different direction and ue ub-pixel interpolation on the per-pixel bet direction. To get rid of the noie, the raw depth map are filtered by dicarding depth value with a ZNCC matching core above 0.4. The cla core are obtained by uing the context baed claifier a explained in Section 4.1. To align the voxel grid with the cene, we ue the approach decribed in [7]. The final number of depth map in each dataet are a follow: Catle P-30 contain 30 depth map, Southbuilding 127, Providence 194 and Catania 126. The reolution of the recontruction volume i , , and voxel, repectively. We ue a multi-threaded C++ implementation to find a minimizer of Equation 5. A tet platform we ued a computer equipped with 64GB of RAM and four 12-core AMD Opteron 6175 CPU. Figure 6 illutrate the reult for all 4 dataet uing the ALE claifier repone. A expected, computational tereo in particular truggle with faithfully capturing the ground, which i repreented by relatively few depth ample een on a very hallow angle. Conequently, depth integration method with a generic urface prior uch a TV-Flux eaily remove the ground and other weakly oberved urface (due to the well-known hrinking bia of the employed boundary regularizer). In contrat, our propoed joint optimization lead to more accurate geometry, and at the ame time defect in the image egmentation are improved over

10 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 a greedy bet-cot cla aignment. Noie in the image egmentation i regularized out and even tructure which are milabeled a ky are recovered with a correct egmentation. Mot importantly, in ambiguou cae our approach output a conitent labeling for the whole dataet. The third row in Figure 6 how the raw image labeling reult. Note that the mot probable cla label according to the trained appearance likelihood ometime confue ground, building, and clutter categorie in ambiguou cae and i prone to labeling building top a ky. Another difficult cae for the image baed claifier i the diambiguation between building and vegetation (c.f. Figure 1). Our joint fuion i able to correctly diambiguate even in thi difficult cae. The joint determination of the right moothne prior alo enable our approach to fully recontruct ground and all the facade completely a een in Figure 6, 4th row. The ground i conitently miing in the TV-Flux reult, and partially the facade and roof tructure uffer from the generic moothne aumption (Figure 6, 5th row). We elected a weighting between data fidelity and moothne in the TV-Flux method uch that uccefully recontructed urface have a (viually) imilar level of moothne to the reult of our propoed method Fig. 8. (Top) TV-Flux fuion, (Bottom) our joint recontruction and egmentation reult. Uing a generic moothne prior, the ground and part of the facade get removed. Uing our joint recontruction and egmentation approach building are naturally extended to the ground. Comparion to Geometry Only Recontruction Our volumetric formulation for joint 3D recontruction and cla egmentation naturally infer urface which are hidden in the input data uch a the ground underneath a building or the building facade hidden behind vegetation. While thee urface are purely inferred baed on the viible data and the learned prior of the urface direction, an exact recovery cannot be expected. However, the preence of thee urface allow u to eaily egment the recontruction into the individual emantic clae. It alo enable u to anwer quetion uch a what i the volume of a building directly, which would be a very challenging tak baed on jut the viible geometry. Even if a emantic egmentation of the viible urface wa available, an additional optimization would need to be carried out to fill in the hole. An illutration of how our emantic model are compoed i given in Figure 7. The facade which i partially hidden behind the vegetation get fully recontructed. Note that the bottom part, which i fully hidden behind the buhe, get correctly extended traight down to the ground. Geometry only recontruction that ue a generic urface area penalization cannot properly recover the ground. In thee recontruction, the building facade are alo affected, epecially if they are oberved by very few image or are partially occluded. Hence, a faithful recontruction i not feaible and part of the facade get removed by the regularization. An example of thi behavior i depicted in Figure 8. With our propoed approach, which take into account the cla pecific geometric prior, the ground get recontructed correctly and the building are naturally tanding on the ground. It i important to note that overhanging tructure are not prohibited by our method a long a the input data indicate their preence. Additionally, vegetation growing cloe to the building facade lead to problem even when mot of the facade i oberved and only mall part are occluded. Often, the vegetation get wrongly connected to the building facade, gluing Fig. 9. (Top) TV-Flux fuion, (Bottom) our joint recontruction and egmentation reult. Note how the vegetation cloe to the building facade get connected to the building in the TV-Flux fuion. In our joint recontruction thi defect get reolved. the two object together. An example of thi can be een in Figure 9. Uing our joint recontruction and egmentation approach the two object get properly diconnected in uch cae. 6.3 Influence of the Semantic Claifier In Section 4.1 we introduced the two different emantic claifier ued a input for our joint fuion. In thi ection, we compare the influence of different emantic claifier core. We exchange the ALE claifier [20] that wa ued for the experiment in all other evaluation of our approach with the STAIR Viion library [10]. A viual comparion i given in Figure 10. Looking purely at the bet cot label baed on a ingle image, the ALE claifier produce moother reult than the STAIR Viion library. Thi i mainly due to the context-baed feature repreentation. Uing feature purely baed on uper-pixel, the STAIR Viion library can eaily confue white wall with clutter or ky, or the ground with clutter. However, by fuing appearance likelihood over multiple image and incorporating the urface geometry, our approach almot perfectly diambiguate the aigned object clae. The difference between the final

11 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST Fig. 7. (Top Row) TV-Flux fuion and our joint recontruction and egmentation reult, (Bottom Row) Our reult divided into three component: ground, building, vegetation plu clutter turn off the geometric prior. Thi i done by uing an identical iotropic moothne term between all emantic clae. Formally, we ue the following moothne term P ji ij i,j:i<j C x x 2, with C the ame contant for all the tranition. In a econd experiment we turn on the aniotropy of the moothne term but do not take into account the relative tranition frequencie by uing a contant probability for all the tranition, i.e. P ( ij ) = P ( ) i, j. We how the reult on the Catania dataet in Fig. 11. Not uing a geometric prior at all lead to a emantically egmented model but the weakly and unoberved geometry i motly not recovered. Epecially the ground i motly miing. When uing the aniotropy the ground and all the weakly oberved urface get recovered the difference to the full model are rather mall and retricted to area which are ambiguou. Thi how that mot of our geometric prior i captured in the aniotropy and not in the relative frequencie. 6.5 Fig. 10. Comparion uing different emantic claifier: (Top row) example input image, (middle row) ALE reult (raw labeling uing per pixel bet repone (left) and 3D model (right)), (bottom row) STAIR Viion library reult (raw labeling uing per pixel bet repone (left) and 3D model (right)) reult generated with the two different emantic claifier are minimal, which how that our method i robut with repect to the choice of image-baed claifier. 6.4 Influence of the Geometric Prior In order to evaluate the influence of the geometric prior we ran our recontruction pipeline with leaving out part of the formulation. In the mot retrictive etting we completely Convergence Analyi In thi ection, we analyze the convergence behavior of our method. Thi ection give ome additional information on how the method behave in practice. We how how the quality and runtime of the method change with grid reolution and how the 3D model evolve during the optimization. In a firt experiment we analyze the convergence behavior and running time with different reolution. Thi evaluation i done with the Southbuilding dataet uing three different reolution. Each reolution ubdivide the voxel ide length by a factor of two leading to eight time a many voxel. The three reolution are a follow lowre = = voxel, midre = = voxel and highre = = voxel. For each of the reolution we found the globally optimal olution by

12 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST Fig. 11. Evaluation of the geometric prior on the Catania dataet. (from left to right) no geometric prior, contant relative tranition frequencie, full propoed model Fig. 12. The three different reolution ued for the convergence analyi of the Southbuidling dataet. (Left to right) lowre, medre and highre iter time[min] iter time[min] iter time[min] lowre medre highre lowre medre highre Iteration Avg. Sqr. Dit. to Solution Avg. Sqr. Dit. to Solution TABLE 1 Evaluation of convergence on the Southbuilding dataet. Number of iteration and elaped time until the ditance to the olution get maller than the indicated threhold lowre medre highre Time [min] Fig. 13. Convergence behavior for three different reolution. (Left) ditance to converged olution with repect to number of iteration paed, (right) ditance to converged olution with repect to elaped time. running the primal-dual algorithm until full convergence (c.f. Figure 12 for the correponding recontruction). In order to meaure the ditance to the converged olution it i enough to ue the per node indicator variable xi a they fully determine the final label aignment. A a ditance meaure we ue the per voxel average quared ditance to 1 P i i 2 the converged olution x i, defined a Ω (x,i x ). The ditance i evaluated every 50 iteration while running the optimization. In Figure 13 we depicted the evolution of the ditance once with repect to the number of iteration executed and once with repect to the elaped time. Another way to quantify convergence i to meaure how long it take until the ditance to the olution i below a given threhold. In Table 1, we give uch meaurement for the number of iteration paed and the elaped time for three different threhold. We oberve i that by plitting the ide length of the voxel by two the number of iteration roughly double. For the elaped time, we need to take into account that each iteration now ha to do computation on eight time a many voxel. We oberve that the low down i a bit le than the expected 16 time longer running time. To viually how how the model evolve during the iterative optimization, we extracted intermediate reult with different number of iteration. The evolution of the olution for the Catania dataet i depicted in Figure 14. Note that the location with trong datacot already get recontructed faithfully from the beginning. While mot of the model i already viible after 2000 iteration, the information need to propagate through the volume in order to alo fully recontruct the weakly oberved urface. After 5900 iteration (18.02 hour of running time) all the urface are recontructed. The ditance to a converged olution a defined above i jut below at thi point. 7 C ONCLUSION We preent an approach for dene 3D cene recontruction from multiple image and imultaneou image egmenta-

13 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST Fig. 14. Evolution of the optimization for the Catania dataet after 50, 2000, 3950 and 5900 iteration. tion. Thi challenging problem i formulated a a joint volumetric inference tak over multiple label, which enable u to utilize cla-pecific moothne aumption in order to improve the quality of the obtained recontruction. We ue a parametric repreentation for the repective moothne prior, which yield a compact repreentation for the prior and at the ame time allow to adjut the underlying parameter from training data. We demontrate the benefit of our approach over tandard moothne aumption for volumetric recontruction on everal challenging data et. Future work need, in particular, to addre the calability of the method. A a volumetric approach operating in a regular voxel grid, our method hare the limitation in term of patial reolution with mot other volumetric approache. Adaptive repreentation for volumetric data can be a potential olution. Thi ha already been done a a follow up of our work in [2]. Another till open problem in calability i the number of emantic label. We plan to extend the number of object categorie to obtain a finergrained egmentation. The difficulty i that a quadratic number of variable in term of label need to be inferred for each voxel. Note that not all pairwie tranition between label in 3D are equally important or even occur in practice. Thi fact can be utilized to improve the computational efficiency of our propoed formulation. ACKNOWLEDGMENTS We would like to thank L ubor Ladický for providing the output of hi emantic claifier and Roland Angt for hi help on the earlier verion [13] of thi paper. Furthermore, we gratefully acknowledge the upport of the 4DVideo ERC tarting grant # and V-Charge grant # both under the EC FP7/ and the Swi National Science Foundation project # REFERENCES [1] Sid Yingze Bao, Manmohan Chandraker, Yuanqing Lin, and Silvio Savaree. Dene object recontruction with emantic prior. In Conference on Computer Viion and Pattern Recognition (CVPR), [2] Maroš Bláha, Chritoph Vogel, Audrey Richard, Jan D. Wegner, Thoma Pock, and Konrad Schindler. Large-cale emantic 3d recontruction: an adaptive multi-reolution model for multi-cla volumetric labeling. In Conference on Computer Viion and Pattern Recognition (CVPR), [3] Michael Bleyer, Carten Rother, Puhmeet Kohli, Daniel Schartein, and Sudipta Sinha. Object tereo - joint tereo matching and object egmentation. In Conference on Computer Viion and Pattern Recognition (CVPR), [4] Gabriel J Brotow, Julien Fauqueur, and Roberto Cipolla. Semantic object clae in video: A high-definition ground truth databae. Pattern Recognition Letter (PRL), 30(2):88 97, [5] Antonin Chambolle, Daniel Cremer, and Thoma Pock. A convex approach for computing minimal partition. Technical report, Ecole Polytechnique, [6] Antonin Chambolle and Thoma Pock. A firt-order primaldual algorithm for convex problem with application to imaging. Journal of Mathematical Imaging and Viion (JMIV), 40(1): , [7] Andrea Cohen, Chritopher Zach, Sudipta N Sinha, and Marc Pollefey. Dicovering and exploiting 3d ymmetrie in tructure from motion. In Conference on Computer Viion and Pattern Recognition (CVPR), [8] Selim Eedoglu and Stanley J Oher. Decompoition of image by the aniotropic rudin-oher-fatemi model. Communication on Pure and Applied Mathematic (CPAM), 57(12): , [9] Pedro F Felzenzwalb and Olga Vekler. Tiered cene labeling with dynamic programming. In Conference on Computer Viion and Pattern Recognition (CVPR), [10] Stephen Gould, Olga Ruakovky, Ian Goodfellow, Paul Baumtarck, Andrew Y. Ng, and Daphne Koller. The STAIR Viion Library [11] Chritian Häne, Lionel Heng, Gim Hee Lee, Alexey Sizov, and Marc Pollefey. Real-time direct dene matching on fiheye image uing plane-weeping tereo. In International Conference on 3D Viion (3DV), [12] Chritian Häne, Nikolay Savinov, and Marc Pollefey. Cla pecific 3d object hape prior uing urface normal. In Conference on Computer Viion and Pattern Recognition (CVPR), [13] Chritian Häne, Chritopher Zach, Andrea Cohen, Roland Angt, and Marc Pollefey. Joint 3d cene recontruction and cla egmentation. In Conference on Computer Viion and Pattern Recognition (CVPR), [14] Chritian Häne, Chritopher Zach, Jongwoo Lim, Ananth Ranganathan, and Marc Pollefey. Stereo depth map fuion for robot navigation. In International Conference on Intelligent Robot and Sytem (IROS), [15] Derek Hoiem, Alexei A Efro, and Martial Hebert. Recovering urface layout from an image. International journal of computer viion (IJCV), 75(1): , [16] Michal Jancoek and Tomá Pajdla. Multi-view recontruction preerving weakly-upported urface. In Conference on Computer Viion and Pattern Recognition (CVPR), [17] Byung-Soo Kim, Puhmeet Kohli, and Silvio Savaree. 3d cene undertanding by voxel-crf. In Conference on Computer Viion and Pattern Recognition (CVPR), [18] Kalin Kolev, Thoma Pock, and Daniel Cremer. Aniotropic minimal urface integrating photoconitency and normal information for multiview tereo. In European Conference on Computer Viion (ECCV), [19] Abhijit Kundu, Yin Li, Frank Dellaert, Fuxin Li, and Jame M Rehg. Joint emantic egmentation and 3d recontruction from monocular video. In European Conference on Computer Viion (ECCV) [20] L ubor Ladický, Chri Ruell, Puhmeet Kohli, and Philip Torr. Aociative hierarchical CRF for object cla image egmentation. In International Conference on Computer Viion (ICCV), [21] L ubor Ladický, Paul Sturge, Chritopher Ruell, Sunando Sengupta, Yalin Batanlar, William Clockin, and Philip Torr. Joint optimiation for object cla egmentation and dene tereo recontruction. In Britih Machine Viion Conference (BMVC), [22] L ubor Ladický, Bernhard Zeil, and Marc Pollefey. Dicriminatively trained dene urface normal etimation. Conference on Computer Viion (ECCV), In European

14 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST [23] Jan Lellmann and Chritoph Schnörr. Continuou multicla labeling approache and algorithm. SIAM Journal on Imaging Science (SIIMS), 4(4): , [24] Victor Lempitky and Yuri Boykov. Global optimization for hape fitting. In Conference on Computer Viion and Pattern Recognition (CVPR), [25] Shubao Liu and David B Cooper. A complete tatitical invere ray tracing approach to multi-view tereo. In Conference on Computer Viion and Pattern Recognition (CVPR), [26] Xiaoqing Liu, Olga Vekler, and Jagath Samarabandu. Orderpreerving move for graph-cut-baed optimization. Tranaction on Pattern Analyi and Machine Intelligence (TPAMI), 32: , [27] John Melonako, Eric Pichon, Sigurd Angenent, and Allen Tannenbaum. Finler active contour. Tranaction on Pattern Analyi and Machine Intelligence (TPAMI), 30(3): , [28] Matthia Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. Real-time 3d recontruction at cale uing voxel hahing. ACM Tranaction on Graphic (TOG), 32(6):169, [29] Thoma Pock and Antonin Chambolle. Diagonal preconditioning for firt order primal-dual algorithm in convex optimization. In International Conference on Computer Viion (ICCV), [30] Thoma Pock, Antonin Chambolle, Daniel Cremer, and Hort Bichof. A convex relaxation approach for computing minimal partition. In Conference on Computer Viion and Pattern Recognition (CVPR), [31] Long Quan, Jingdong Wang, Ping Tan, and Lu Yuan. Image-baed modeling by joint egmentation. International journal of computer viion (IJCV), 75(1): , [32] Nikolay Savinov, Chritian Häne, L ubor Ladický, and Marc Pollefey. Semantic 3d recontruction with continuou regularization and ray potential uing a viibility conitency contraint. In Conference on Computer Viion and Pattern Recognition (CVPR), [33] Nikolay Savinov, L ubor Ladický, Chritian Häne, and Marc Pollefey. Dicrete optimization of ray potential for emantic 3d recontruction. In Conference on Computer Viion and Pattern Recognition (CVPR), [34] Ahutoh Saxena, Sung H Chung, and Andrew Y Ng. 3-d depth recontruction from a ingle till image. International journal of computer viion (IJCV), 76(1):53 69, [35] Steven M Seitz, Brian Curle, Jame Diebel, Daniel Schartein, and Richard Szeliki. A comparion and evaluation of multi-view tereo recontruction algorithm. In Conference on Computer Viion and Pattern Recognition (CVPR), [36] Jamie Shotton, John Winn, Carten Rother, and Antonio Criminii. Textonboot: Joint appearance, hape and context modeling for multi-cla object recognition and egmentation. In European Conference on Computer Viion (ECCV), [37] Chritoph Strecha, Wolfgang von Hanen, L Van Gool, Pacal Fua, and Ulrich Thoenneen. On benchmarking camera calibration and multi-view tereo for high reolution imagery. In Conference on Computer Viion and Pattern Recognition (CVPR), [38] Evgeny Strekalovkiy and Daniel Cremer. Generalized ordering contraint for multilabel optimization. In International Conference on Computer Viion (ICCV), [39] Vibhav Vineet, Ondrej Mikik, Morten Lidegaard, Matthia Nießner, Stuart Golodetz, Victor A Priacariu, Olaf Kähler, David W Murray, Shahram Izadi, Patrick Pérez, and Philip H. S. Torr. Incremental dene emantic tereo fuion for large-cale emantic cene recontruction. In International Conference on Robotic and Automation (ICRA), [40] Chritopher Zach. Fat and high quality fuion of depth map. In International Sympoium on 3D Data Proceing, Viualization and Tranmiion (3DPVT), [41] Chritopher Zach, Chritian Häne, and Marc Pollefey. What i optimized in convex relaxation for multilabel problem: Connecting dicrete and continuouly inpired map inference. Tranaction on Pattern Analyi and Machine Intelligence (TPAMI), 36(1): , [42] Chritopher Zach, Manfred Klopchitz, and Marc Pollefey. Diambiguating viual relation uing loop contraint. In Conference on Computer Viion and Pattern Recognition (CVPR), [43] Chritopher Zach, Marc Niethammer, and Jan-Michael Frahm. Continuou maximal flow and wulff hape: Application to mrf. In Conference on Computer Viion and Pattern Recognition (CVPR), [44] Chritopher Zach, Thoma Pock, and Hort Bichof. A globally optimal algorithm for robut TV-L 1 range image integration. In International Conference on Computer Viion (ICCV), [45] Chritopher Zach, Liang Shan, and Marc Niethammer. Globally optimal finler active contour. In Sympoium of the German Aociation for Pattern Recognition (DAGM), Chritian Häne received hi BSc, MSc and Dr. c. (2016) from ETH Zürich, Switzerland. Currently he i a potdoctoral cholar at the Univerity of California, Berkeley in the Department of Electrical Engineering and Computer Science. He i the recipient of an Early Potdoc.Mobility Fellowhip from the Swi National Science Foundation. Hi reearch interet include convex method for dene 3D recontruction and the application of thee method to challenging cenario, uch a emantic 3D recontruction and 3D recontruction from ingle image. Chritopher Zach (PhD 2007 TU Graz) i currently leading the computer viion group at Tohiba Reearch Europe. Previou to that he had pot-doctoral and enior reearcher poition at UNC-Chapel Hill ( ), ETH Zürich ( ) and Microoft Reearch Cambridge ( ). Hi main reearch interet are convex method in computer viion, tructure from motion, dene recontruction from image, and computer viion on graphic proceing unit. Andrea Cohen tudied at National Univerity of Tucuman, Argentina and at Univerity of Technology of Belfort-Montbeliard, France, from which he received her Computer Science Engineer Diploma a well a her Mc in Computer Science. She i currently a PhD candidate at ETH Zürich in the Computer Viion and Geometry Group. Her reearch interet include the ue of ymmetrie and emantic cue to improve urban cene recontruction. Marc Pollefey i Director of Science at Microoft HoloLen and full profeor in the Dept. of Computer Science of ETH Zurich ince Before that he wa on the faculty at the Univerity of North Carolina at Chapel Hill. He obtained hi PhD from the KU Leuven in Belgium in Hi main area of reearch i computer viion, but he i alo active in robotic, machine learning and computer graphic. Dr. Pollefey ha received everal prize for hi reearch, including a Marr prize, an NSF CAREER award, a Packard Fellowhip and a European Reearch Council Grant. He i the author or co-author of more than 250 peer-reviewed publication. He wa the general chair of ECCV 2014 in Zurich and program chair of CVPR He i a fellow of the IEEE.