PROJECTIVE RECONSTRUCTION OF BUILDING SHAPE FROM SILHOUETTE IMAGES ACQUIRED FROM UNCALIBRATED CAMERAS

PROJECTIVE RECONSTRUCTION OF BUILDING SHAPE FROM SILHOUETTE IMAGES ACQUIRED FROM UNCALIBRATED CAMERAS Po-Lun La and Alper Ylmaz Photogrammetrc Computer Vson Lab Oho State Unversty, Columbus, Oho, USA -la.138@osu.edu, -ylmaz.15@osu.edu http://dpl.ceegs.oho-state.edu/ Commsson III/2 KEY WORDS: Dgtal Photogrammetry, Computer Vson, Reconstructon, Slcng, Urban Plannng, Vsualzaton ABSTRACT: Recoverng the three-dmensonal (3D) object shape les as an unresolved and actve research topc on the cross-secton of computer vson, photogrammetry and bonformatcs. Although varous technques have been developed to tackle the shape recovery problems, the computatonal complexty and the constrants ntroduced by the other algorthms have lmted the applcablty of these methods n real world problems. In ths paper, we propose a method that s based on the projectve geometry between the object space and slhouette mages taken from multple vewng angles. The approach elmnates the requrements of dense feature matchng and camera calbraton that are generally adopted by other reconstructon method. The object s reconstructed by settng a set of hypothetcal planes slcng the object volume and estmatng the projectve geometrc relatons between the mages. The expermental results show that satsfactory 3D model can be generated by applyng mnmal constrants. 1. INTRODUCTION The growng demands on usng 3D models for plannng and analyss make 3D object shape recovery a prevalng area of research n the felds of dgtal photogrammetry and computer vson. Numerous research efforts have been extended to recover the 3D object shape from mages. The most commonly adopted methods requre calbratng the camera pror to the shape recovery by measurng certan features on a calbratng box whch s preset n the object space. The 3D object shape s then reconstructed by tradtonal trangulaton technques and bundle block adjustment (Blosten and Huang, 1987). Alternatvely, the calbraton can be accomplshed by explotng the projectve geometry whch relates the dfferent vews of a scene to each other (Hartley et al., 1992), (Koch et al., 2000), and (Hernandez et al., 2007). In general, these methods estmate ether the fundamental matrx or the homography between the mages and the 3D object shape s recovered up to an unknown scale factor. Whle havng calbrated cameras s desrable for generatng satsfactory result, the calbraton process s usually not ntutve. Hence, t becomes necessary to develop technques to recover the object shape wthout calbraton. Another problem observed s that most 3D object shape reconstructon algorthms depend on the extracton of feature ponts whch poses a lmtaton n cases when the number of feature ponts s nsuffcent or the mage s of poor qualty. The deformaton of the object shape due to the projecton from 3D to 2D also ncreases the dffculty of fndng the correspondences between mages. In addton, the object occluson, whch s commonly observed durng the magng process, can obstruct the recovery of the object shape. In the recent years, varous papers whch are dedcated to the problem of 3D object shape recovery have utlzed the propertes of the homography transformaton and the slhouette mages to solve the aforementoned problems. The homography transformaton provdes a strong geometrc constrant and s comparatvely smple. The mpled 3D scene nformaton can be retreved from 2D mages by the homography transformaton and s utlzed for use n many applcatons, ncludng but not lmted to trackng people (Khan and Shah, 2006), shadow removal, and detectng low-lyng objects (Kelly et al., 2005). These methods are dversfed from the planar homography to the nfnte homography, and from a sngle mage to multple mages. Although some technques have conceptually been proven to be successful n certan cases (Zhang and Hanson, 1996) (Wada et al., 2000) (Zhang et al., 2003) (Yun et al., 2006), n real-world problems ther use s lmted due to specfc requrements or assumptons such as postonng the cameras on a crcle enclosng the object. The computatonal complexty also hnders them from beng practcal. In ths paper, we explot a new method for the metrc reconstructon of 3D object shape usng the concept of slcng planes. Our method s nspred by the affne recovery technque proposed n (Khan et al., 2007). However, n our method the constrants used theren are elmnated, and metrc shape recovery s obtaned. We represent a 3D object by a set of parallel planes ntersectng wth the object n the object space. These hypothetcal planes are related by the homography transformaton. Assume four conjugate ponts lyng on a base plane n the object space can be dentfed n all mages, they are suffcent for constructng the homography relaton between the mages. To generate the vew of another hypothetcal plane that s parallel to ths base plane, we utlze the concept of a vanshng pont (Hartley and Zsserman, 2004). The vanshng pont of a reference drecton s computed from the mage of a par of parallel lnes, after that any new set of four ponts whch forms a plane parallel to the base plane can be derved along the reference drecton by settng an ncrement value. For every set 103

The Internatonal Archves of the Photogrammetry, Remote Sensng and Spatal Informaton Scences. Vol. XXXVII. Part B3b. Bejng 2008 of ponts, the homographes between a reference mage and all other mages are computed. After warpng all mages onto the reference mage by the homography transformaton, ther ntersecton provdes the object shape. The mert of the proposed approach can be descrbed n terms of effcency, flexblty and practcablty. Frst of all, ths method requres no camera calbraton or the estmaton of the fundamental matrx; hence, t reduces the computatonal complexty by elmnatng the requrement for abundant conjugate ponts. The object s reconstructed usng the contours enclosng the ntersected regons, whch are suffcent for revealng the complete object surface wthout the necessty of estmatng vsual hulls. Second, the formulaton provdes three dfferent settngs for fndng the hypothetcal planes, usng ether a par of parallel lnes, or any four randomly located features wth known heghts n the object space. Furthermore, the level of detal n the reconstructed object s easly modfed by changng the number of mages used or the densty of the planes to fnd a best balance between the computaton tme and the smoothness of the recovered 3D object shape. Snce no dense pont correspondences are needed and the mssng nformaton can be recovered from other mages of dfferent vews, the use of object slhouettes automatcally elmnates the problems related to occluson. Another noteworthy advantage s that we adopt full homography, so that true metrc reconstructon s accomplshed by the proposed method. The paper s organzed as follows. In order to better manfest the core technques adopted by our method, we brefly revew two mportant concepts of projectve geometry n Secton 2.1. The proposed approach for fndng consecutve parallel planes s descrbed n Secton 2.2. In Secton 2.3 the technques for recoverng the 3D object shape usng the slces are delneated. Two sets of experments are conducted to verfy the applcablty n close-range and aeral photogrammetry. The setup and results are dscussed n Secton 3. Fnally we conclude the paper n Secton 4. 2. OBJECT SHAPE RECOVERY one onto the other va mappng equatons. In order to provde basc prncples of the mappng equatons used n ths paper, we frst provde an ntroductory dscusson to the projectve geometry as t pertans to 3D shape recovery whch wll be detaled later n the secton. 2.1 The Projectve Geometry The projectve geometry descrbes the physcal characterstcs of the cameras and the relatonshps between the mages. The projecton of a pont X w n the object space to a pont x n the mage space usng a projectve camera s expressed n terms of a drect lnear mappng n homogeneous coordnates as: X Y Z 1 [ p p p p ], λx PX w 1 2 3 4 where λ s the scale factor due to projectve equvalency of (kx; ky; k) (x; y; 1), P s a 3 x 4 camera projecton matrx and p the th column of P. Note that, throughout the paper, we use homogeneous coordnates for ponts both n the mage and object spaces. In the homogenous representaton, the last row of the vectors beng equal to 1 reflects that the pont les on the mage plane. Let s consder the case when the pont n the object space les on the ground plane such that Z 0, then the lnear mappng gven n (1) wll reduce to the planar homography (1) X sx HX [ 1 2 4] w p p p Y, (2) 1 where H s the homography matrx whch maps ponts lyng on a plane n the object space across dfferent mages. Ths formulaton ntroduces another scalng factor, s, to the mappng equaton whch stems from settng Z to 0. One thng to be kept n mnd s that the ground plane s not necessarly the real ground n object space but can be any vsble plane n the scene. Snce general 3D modelng rarely requres absolute geodetc coordnates, a local axs system usually fts well n most cases. In the case when we have multple mages of a scene, an ntutve consequence of the homography transform from the ground plane to the mage s the exstence of a drect mappng between the two mages: Fgure 1: Homography transformaton from ground plane to mages and across mages The relaton between the 3D object space and the 2D mage space can be expressed n terms of the projectve geometry, whch defnes a set of nvarant propertes when object space s projected onto an mage space. These nvarants n turn provde capabltes to relate two or more vews of a scene by projectng x H w X w H w (H 1 wj x ) H x. (3) The H j matrx s the homography descrbng the projectve transformaton between the mages and j. The estmaton of ths transformaton up to a scale factor requres a mnmum of four ponts lyng on the plane. Ths s exemplfed n Fgure 1, where the correspondences between ground plane πand mages I1 and I2 are related by two dfferent homographes. When j j j 104

The Internatonal Archves of the Photogrammetry, Remote Sensng and Spatal Informaton Scences. Vol. XXXVII. Part B3b. Bejng 2008 warpng one mage onto the other, only the pxels n the area that map the ground plane concde, whle other pxels wll create dscrepances dependng on ther Eucldean dstances to the plane π. We should state that t s these dscrepances that wll let us estmate the 3D shape of the object. The perspectve effects durng the magng process causes n 3D objects appearng as stretched out nto nfnty. Consderng a par of parallel lnes L 1 and L 2 n the object space, the ntersecton pont s defned to le at the nfnty whch s represented by [X, Y, Z, 0] T. When these parallel lnes and ther ntersecton at the nfnty are maged, they are mapped nto two non-parallel lnes and a vsble ntersecton pont between them whch s referred to as the vanshng pont v. The vanshng pont can be computed from the cross product of the correspondng lne par l 1, l 2 n the mage space (see Fgure 3), v l. 1 l (4) 2 be denoted as x and x, 1,2,3,4 n the mage space. In order to automatcally estmate the new pont set x drectly from the orgnatng planeπwe need to apply an addtonal constrant. Assume x 1 and x 3 can be observed on the mage and the heght Z s known, then x 2 and x 4 can be computed by explotng the ntrnsc propertes of the vanshng ponts. The procedure can be descrbed as follows. Frst, the vanshng pont of the parallel lnes n πs computed usng v ( x x ) ( x ). (5) 1 2 3 x4 The vanshng pont v z n the drecton of the normal of plane π s obtaned smlarly usng a par of lnes that s orthogonal to the plane π n the object space. The next step s to establsh the relatonshp between x and x. Rewrtng equaton (1) as: A vanshng pont depends only on the drecton of the lnes, whch means despte ther postons all parallel lnes wth the same drectons ntersect at one sngle pont. Though parallelsm s not preserved after projecton, the nformaton about orentaton mpled by the vanshng pont stll plays the role of a key to acheve camera calbraton and 3D object shape reconstructon. 2.2 Generatng the Slcng Planes X λx 1 2 4 + 1 [ p p p ] Y p Z. 3 (6) Fgure 3: Vanshng ponts of parallel lnes. The ponts x consttute the reference plane. By fndng x 1, x 3, v and v z, any plane that s parallel to the reference plane can be determned. Fgure 2: A set of hypothetcal planes ntersect the object volume and create slces n the object space. The tools used n our method can be consdered as the extensons of the projectve geometry concepts delneated n the prevous secton. In a nutshell, the basc dea behnd the proposed approach s to dvde the object space nto a set of planes parallel to each other as shown n Fgure 2. These planes and the homography transform of each of them onto the mages generate slhouette coherency maps whch provde the 3D shape nformaton. Let s assume four ponts X, 1, 2, 3, 4 le on a par of parallel lnes n π. Imagne that these ponts are moved up a dstance Z vertcally n the drecton of the plane normal generatng four new postons X, 1, 2, 3, 4. By defnton, the new pont set consttute a new plane π whch s parallel to π. When the projectve camera maps the orgnal four ponts and the new four ponts onto the mage, the resultng ponts respectvely can The column vector p 3 corresponds to the vanshng pont n the drecton of the Z axs or the normal of the ground plane. By substtutng p 3 wth v z and combnng wth equaton (2) results n: λ x s x + v Z. (7) z The unknowns of ths lnear equaton are λ and s whch are descrbed n equatons (1) and (2). Estmatng both λ and s can be acheved by solvng λ T (A A ) s 1 where A [ x x ] and b Z. v z T A b, (8) Once s s computed, estmaton of any mage pont along the 105

The Internatonal Archves of the Photogrammetry, Remote Sensng and Spatal Informaton Scences. Vol. XXXVII. Part B3b. Bejng 2008 lne x v s acheved by settng dfferent Z to dfferent set of z values. In the case when only x01 and x03 are dentfable on the mage, as shown n Fgure 3, x 2 and x 4 can also be estmated from the property of vanshng pont, such that, all parallel lnes n the object space whch are n the same drecton ntersect at the vanshng pont. If v s computed from equaton (5), then x 2 and x 4 are obtaned by x x 2 4 ( x ( x 2 4 v z ) ( x1 v). (9) v ) ( x v) z Assume multple mages of a scene are provded and the frst mage s chosen as the reference mage, such that the other mages are warped onto ths reference mage by I j H j I j, where the subscrpt j ndcates that the warpng s from th to j th mage. In the sequel of a segmentaton method, the object slhouettes extracted n these mages hghlghts of the slcng planes when they ntersect wth the object volume. Let s consder the object slhouette s defned by settng the mage pxels nsde the object to 1. The hghlghts of the slcng planes when they ntersect the object volume are determned by warpng all the slhouettes onto the reference mage: 3 3. RESULTS AND DISCUSSION In order to verfy the proposed method, we have performed two sets of experments. In the frst experment, as shown n Fgure 4(a), we placed a toy, whch contans rregular shape, on the ground plane. The ground plane contans squares whch provde us wth four measures requred to compute the homography transform from one mage to the other and from the reference mage to the object space. Two pens orented n the normal drecton of the plane are placed to estmate the vanshng pont n the drecton of Z axs as well as the scale factor s. We took 11 mages of the object and the vertcal features from dfferent vewponts around the toy. We should note that, no length measurements are performed and the lengths of the vertcal features are set to be a unt length. Snce no knowledge of absolute ground truth s consdered, we assume all tles are squares and the coordnates of four corners are defned to resde at unt dstances from the orgn. The 3D shape s reconstructed by settng the dstance ncrements Δ Z n the vertcal drecton to 0.5 and computng correspondng Z values used to generate slcng planes. In order to generate fne 3D models, one can set Δ Z to lower values. I ntersecton I n 1 + I 1 2 n, (10) where n s the number of mages. Next, we wll dscuss how these warped slhouettes can be used to recover the object shape. 2.3 Recoverng the 3D Object Shape In equaton (10), thresholdng the accumulaton of the warped slhouettes provdes a mask mage. Ths mask s the mage of the ntersecton between the slcng planes and the object volume. Hence, usng these masks, we can generate the outlnes of the object shape whch corresponds to the surface of the object volume. The back-projecton of masks generated from equaton (10) can be acheved n varous ways. Gven some feature ponts wth known absolute object coordnates, the relaton between the object space and the mage space for back-projectng the ntersecton mages onto the object space and create the exact 3D model. If a specfc feature such as a box or a buldng wth known dmensons or relatve length rato s recognzed n the mages, we can assume a local Eucldean coordnate frame n the object space and the metrc shape recovery s acheved up to a scale. By selectng an arbtrary local coordnate frame, we can also recover the object shape n the case of dstortons on the slhouettes. Theoretcally, one can pck up any measurable feature on the reference plane even though the axes are not orthogonal, but deally a square s preferred for achevng metrc reconstructon. On the extreme case the coordnate nformaton of the object space s not avalable, the 3D object shape of a specfc vewng angle can stll be represented, as wll be shown n the next secton. Fgure 4: Experment on recoverng the 3D shape of a toy. (a), (b) and (c) show the orgnal mages taken from dfferent aspects. The contour of the darkest regon n (d) s used for generatng 3D ponts. Two dfferent vews of the reconstructed 3D shape are shown n (e) and (f). The scales and lengths of three axes are not absolute, and as a consequence, the scale of reconstructed object may vary. In order to accelerate the process, only edge pxels of the ntersectng slhouettes (Fgure 4(d)) are nvolved n computaton. Over 24,000 densely dstrbuted 3D surface ponts 106

The Internatonal Archves of the Photogrammetry, Remote Sensng and Spatal Informaton Scences. Vol. XXXVII. Part B3b. Bejng 2008 are generated n ths test. Among the pctures the toy has several self-occluded parts. The result shows that the effect of occluson s compensated wth nformaton provded by other vews and the reconstructed model s ntact. Detals such as shapes of the toes and hands are able to be observed from the rebult model shown n Fgure 4(e) and 4(f). 4. CONCLUSION The proposed approach n ths paper reconstructs the 3D object shape by explotng slhouette mages taken from uncalbrated cameras. The reconstructon s a metrc recovery up to a scale factor whch can be determned f object space measurements are provded. The slhouette mages are allowed to contan occlusons and dstortons as long as some other vews of the object reveal the occluded regons. The projectve geometrc relatons between the mages provde an easy to mplement algorthm. Compared to other algorthms, whch requre generaton of the convex hull or estmaton of the fundamental matrx, the proposed approach bears lower computatonal complexty. The requrement of havng abundant feature correspondences n other prevalng technques s also removed to ncrease the computatonal effcency. The optmzed balance between runnng tme and accuracy s determned by the number of mages and number of slcng planes. Addtonal post processng whch has not been appled n ths paper can be used to further mprove resultng 3D surfaces. The expermental results show the applcablty of our method for buldng 3D models from close-range or aeral mages. A varety of applcatons such as urban and rural surface modelng and glacer and polar cecap montorng can be realzed by our method. REFERENCES Blosten, S. and Huang, T., 1987. Quantzaton error n stereo trangulaton. In: IEEE Int. Conf. on Computer Vson. Hartley, R. and Zsserman, A., 2004. Multple Vew Geometry n computer Vson-second edton. Cambrdge Un. Press. Hartley, R., Gupta, R. and Chang, T., 1992. Stereo from uncalbrated cameras. In: IEEE Conf. on Computer Vson and Pattern Recognton. Fgure 5: Experment on recoverng the 3D shape of a buldng. (a), (b) and (c) are the east, north and west vews of the buldng. The brghtest regon n (d) ndcates the slce mage on the reference plane. Two dfferent vews of the reconstructed 3D buldng shape are shown n (e) and (f). The second experment takes 4 screenshots from the webste of http://maps.lve.com (Fgure 5(a), 5(b) and 5(c)). The ground plane s set on the top of a buldng and 4 conjugate ponts are measured on each mage. The same procedure as n the frst experment s performed except that no object coordnate frame s presumed and the object shape s reconstructed wthout back-projecton. The object space s set to be dentcal as the reference mage. All the contours of slce mages are warped by homography onto ths space wth preset Z values that are used n computng the hypothetcal planes. The result suggests that, n the case when no ground truth s avalable and true metrc reconstructon s not necessary, we can stll recover the 3D shape up to a scale factor (Fgure 5(e) and 5(f)). Ths experment also demonstrates that for an object of a relatvely regular shape, mages from 4 vews are suffcent for achevng satsfactory reconstructon. Hernandez, C., Schmtt, F. and Cpol, R., 2007. Slhouette coherence for camera calbraton under crcular moton. In: IEEE Trans. on Pattern Analyss and Machne Intellgence. Kelly, P., Beardsley, P., Cooke, E., O Connor, N. and Smeaton, A., 2005. Detectng shadows and low-lyng objects n ndoor and outdoor scenes usng homographes. In: IEE Internatonal Conference on Vsual Informaton Engneerng. Khan, S. and Shah, M., 2006. A multvew approach to trackng people n crowded scenes usng a planar homography constrant. In: European Conf. on Computer Vson. Khan, S., Yan, P. and Shah, M., 2007. A homographc framework for the fuson of mult-vew slhouettes. In: IEEE Int. Conf. on Computer Vson. Koch, R., Pollefeys, M. and Gool, L., 2000. Realstc surface reconstructon of 3d scenes from uncalbrated mage sequences. In: Vsualzaton and Computer Anmaton. Wada, T., Wu, X., Toka, S. and Matsuyama, T., 2000. Homography based parallel volume ntersecton: Toward realtme volume reconstructon usng actve cameras. In: IEEE Internatonal Workshop on Computer Archtectures for Machne Percepton. 107

The Internatonal Archves of the Photogrammetry, Remote Sensng and Spatal Informaton Scences. Vol. XXXVII. Part B3b. Bejng 2008 Yun, Y., Km, S., Lee, S., Km, D. and Cho, J., 2006. Threedmensonal reconstructon of an object usng three-planar homography of a sngle mage. In: Optcal Engneerng, Vol. 45. Zhang, Q., Wang, H. and We, S., 2003. A new algorthm for 3d projectve reconstructon based on nfnte homography. In: Machne Learnng and Cybernetcs. Zhang, Z. and Hanson, A., 1996. 3d reconstructon based on homography mappng 108