ON THE ACCURACY ASSESSMENT OF LEAST-SQUARES MODEL-IMAGE FITTING FOR BUILDING EXTRACTION FROM AERIAL IMAGES

Transcription

1 ON THE ACCURACY ASSESSMENT OF LEAST-SQUARES MODEL-IMAGE FITTING FOR BUILDING EXTRACTION FROM AERIAL IMAGES Sendo WANG and Y-Hsng TSENG Department of Surveyng Engneerng, Natonal Cheng Kung Unversty No. Unversty Road, Tanan 7 Tawan, R.O.C. KEYWORDS: Least-squares Model-mage Fttng, Accuracy Assessment, Model-based Buldng Extracton, Constructve Sold Geometry, Dgtal Photogrammetry ABSTRACT: Model-based buldng extracton from aeral mages has been an ntensve research topc n the feld of dgtal photogrammetry snce the last decade. Based on our prevous research work, the prncple of Constructve Sold Geometry (CSG s appled to model varous buldngs. Each buldng s represented by a combnaton of 3D prmtves and each prmtve s assocated wth a set of shape and pose parameters. Buldng reconstructon s mplemented by adjustng the model parameters to ft model wth mages. We proposed a Least-squares Model-mage Fttng (LSMIF algorthm to obtan the optmal ft between model and mages. In ths paper, the performance of LSMIF s nvestgated. Frst, the convergence rate and pull-n range of the algorthm s analyzed. Then, how to use constrants to ncrease the convergence rate s ntroduced. Fnally, sets real data were tested to assess the theoretcal accuracy of parameter determnaton. In order to assess the emprcal accuracy, the results are also compared wth manually measured data usng an analytcal plotter. Ths study reveals that the functon of LAMIF s table and can generate qualfed 3D nformaton of buldng comparable to manually measured data.. INTRODUCTION 3D spatal nformaton s essental for a wld range of applcatons, such as cty plannng, archtecture desgnng, tourst gudng, or personal navgatng system. Although each applcaton requres dverse spatal nformaton, buldngs are the very requred objects n common (Gülch, 996. To effcently extract buldngs from aeral mages, several full-automated or sem-automated approaches have been proposed by experts both n photogrammetry and computer vson domans (Förstner, 999; Grün,. Model-based buldng extracton, extractng buldngs by fttng pre-defned volumetrc models wth mages, has been recognzed as a convncng approach to effcent buldng extracton that could be mplemented n practce (Braun, et al., 995; Chapman, et al., 99; Lang and Förstner, 996; Veldhus, 998. Model-based buldng extracton reles on a model-mage fttng algorthm to obtan the optmal ft between model and mages. Attempts to solve the problem of model-mage fttng date back to the work of Sester and Förstner (Sester and Förstner, 989. By fttng projected model to mage, the transformaton parameters of a buldng model are determned usng a clusterng algorthm followed by a robust estmaton of model parameters. Ths buddng research work has marked an mportant step toward model-based buldng extracton (MBBE, although the algorthm s restrcted to ft a model to sngle mage rather than multple mages. Concurrently developed n the feld of computer vson for model-based vson, Lowe (Lowe, 99 proposed a least-squares model-mage fttng to solve for projecton and model parameters that wll best ft a 3D model to matchng D mage features. Lowe s study set up the fundamental theory of the least-squares model-mage fttng (LSMIF for generc applcatons. Ths rgorous fttng algorthm has been recognzed as a key to deal wth MBBE (Veldhus, 998; Vosselman, 999. However, to apply LSMIF for MBBE, we need to talor the generc LSMIF theory to meet the specfc stuatons of buldng modelng and model parameters. Vosselman and Veldhus (Vosselman and Veldhus, 999 modfed Lowe s algorthm to mprove pull-n range, but they dd not clearly defne the buldng models and the fttng functons. Based on our prevous research work (Tseng and Wang, ; Wang and Tseng, ; Chou, et al., ; Ln, et al.,, the prncple of constructve sold geometry (CSG s appled to model varous buldngs. Each prmtve s assocated wth a set of model parameters that can be categorzed nto shape and pose parameters. Buldng reconstructon s mplemented by adjustng the model parameters to ft model wth mages. Therefore, an LSMIF algorthm was proposed to ft a prmtve to the correspondng buldng part n the mage, whch s based on the mnmzaton of the squares sum of the dstances from the edge pxels extracted from mages to the correspondng edge of the projected prmtve. The motvaton of ths study s to evaluate the performance of the LSMIF algorthm. Ten varous buldngs, whch can be represented by a combnaton of box and gable-roof prmtves, were constructed usng the algorthm for testng. Based on the test data, we frst nvestgate the convergence rate and pull-n range to see how often the Ph.D. student, TEL: ext 834/835, E-mal: sendo@sv.ncku.edu.tw Assocate professor, FAX: , E-mal: tseng@mal.ncku.edu.tw

2 algorthm can delver correct results. In order to ncrease the convergence rate, the method to add some constrants nto the system s also ntroduced. The accuracy assessment s performed n both the theoretcal and emprcal ways. Theoretcal accuracy can be estmated by the least-squares theory and emprcal accuracy s obtaned by comparng the results wth manually measured data usng an analytcal plotter.. LEAST-SQUARES MODEL-IMAGE FITTING. Defnton of Prmtves A prmtve s a pre-defned smple sold model, whch determnes the ntrnsc geometrc property of an object part, and s assocated wth a set of parameters that can be categorzed nto shape parameters and pose parameters. The shape parameters descrbe the shape sze of the prmtve, e.g., a box prmtve has three shape parameters: length (l, wdth (w, and heght (h. Changng the values of shape parameters elongates the prmtve n the three dmensons and keeps ts shape as a rectangular box. Each prmtve would be assocated wth dfferent shape parameters, e.g., a gable-roof house prmtve has an addtonal shape parameter roof s heght (rh. The pose parameters descrbe the poston and atttude of a prmtve n the object space. It s adequate to use 3 translaton parameters (dx, dy, dz to depct the poston and 3 rotaton parameters, tlt (t around Y-axs, swng (s around X-axs, and azmuth (α around Z-axs to represent orentaton of a prmtve. However, walls of the buldng are supposed to be vertcal n most crcumstances, the tlt and swng angles can be omtted. Unlke the shape parameters, the four pose parameters are sutable for all knds of prmtves. Fgure shows the parameters of a box prmtve, the X -Y -Z coordnate system s the model space, and the X-Y-Z coordnate system s the object space. The shape parameters of a gable-roof house prmtve are llustrated n Fgure. v 8 v 7 v 9 v Z v 5 v 6 F 5 v 4 F F 6 F 4 F 3 Y Z' F 5 F 3 Y' F v F 4 v 3 Z' v v dx dz X' Y' F dy 9 α - v v X' X Fgure : Shape and pose parameters of a box prmtve. Fgure : Shape parameters of a gable-roof prmtve. A model can be descrbed by a polyhedral structure. Polyhedron s a composte of faces and each face conssts of several vertces. For example, the box prmtve, whch s a cubc sold n the model space, conssts of 6 faces (F, F, F 3, F 4, F 5, F 6 and 8 vertces (v, v, v 3, v 4, v 5, v 6, v 7, v 8. The model coordnates of the vertces are: v =(,,, v =(,,, v 3 =(,,, v 4 =(,,, v 5 =(,,, v 6 =(,,, v 7 =(,,, v 8 =(,,. The gable-roof house prmtve shown n Fgure can be descrbed n the same manner.. Coordnate s The algorthm performs the fttng n the photo coordnate system. A prmtve, however, s defned n the model space. It s necessary to transform a prmtve from model space to object space by ntroducng a set of shape and pose parameters, and then to project the object model onto the photo coordnate system wth the known exteror orentaton parameters. On the other hand, edge pxels extracted from the mages should be transformed to the photo coordnate system for matchng. Fgure 3 shows the transformaton steps of a box from model to photo coordnate system and the edge pxels from mage to photo coordnate system. w v 8 F 6 F 8 F 9 F 7 v 5 v 6 F 4 w v 7 Z Y Shape & Pose Parameter Z Y E.O. Parameters y I.O. Parameters r X Model Coord. Model to Object Transformaton X Exteror Orentaton Object Coord. Photo Coord. Fgure 3: Coordnate systems nvolved n the fttng algorthm and ther relatonshp. x Interor Orentaton c Image Coord. The prmtve defned n the model space s a smple unt sold, e.g. a box prmtve s a unt cube whch wdth, length, and heght are all equal to. The shape parameters wll elongate or shorten the box to the correct sze, and the pose parameters wll rotate and move the box to the correct atttude and poston n the object space. Table lsts the change of coordnates of the 8 vertces. Each vertex n the object space can be projected to the photo by the collnearty condton equatons wth the known exteror orentaton elements:

3 r( X X r ( Y Y r3 ( Z Z r( X X r ( Y Y r3 ( Z Z x = f ; y = f ( r ( X X r ( Y Y r ( Z Z r ( X X r ( Y Y r ( Z Z 3 where (x, y are the photo coordnates of a vertex (X, Y, Z are the object coordnates of a vertex (X, Y, Z are the object coordnates of the perspectve center f s the focal length of the camera r, r, r 3, r, are coeffcents of the rotaton matrx defned by the exteror orentaton Table : Vertex coordnates from model space to object space. Model Space Multply After Rotaton After Translaton Vertex No. Coordnate Shape Parameters (Object Space Coordnate v (,, (,, (,, (dx, dy, dz v (,, (w,, (wcosα, wsnα, (wcosαdx, wsnαdy, dz v 3 (,, (w, l, (wcosα-lsnα, wsnαlcosα, (wcosα-lsnαdx, wsnαlcosαdy, dz v 4 (,, (, l, (-lsnα, lcosα, (-lsnαdx, lcosαdy, dz v 5 (,, (,, h (,, h (dx, dy, hdz v 6 (,, (w,, h (wcosα, wsnα, h (wcosαdx, wsnαdy, hdz v 7 (,, (w, l, h (wcosα-lsnα, wsnαlcosα, h (wcosα-lsnαdx, wsnαlcosαdy, hdz v 8 (,, (, l, h (-lsnα, lcosα, h (-lsnαdx, lcosαdy, hdz.3 Edge Feature Extracton The transformed model edges are supposed to ft wth the edge pxels extracted from the correspondng mages. In ths paper, edge pxels are extracted usng the algorthm of the polymorphc feature extracton module (FEX developed by Förstner and Fuchs (Förstner, 994, Fuchs, 995, Fuchs and Förstner, 995, whch can extract nterest ponts, edges, and regons smultaneously. The extracton module conssts of three steps: ( Classfcaton of the pxels nto the three feature types, ( Localzaton of the features, (3 Approxmaton of lnes and blob boundares by analytc functons. By provdng y the known nteror orentaton parameters of each mage, edge pxels extracted by FEX can be transformed to the photo coordnate system. v Extracted Pxel T ( x, y The user nterface of our sem-automatc approach allows the operator to resze, rotate, and move a model to ft the correspondng buldng mages approxmately. Ths procedure would offer a set of approxmate values for the shape and pose parameters, so that the dscrepancy between the projected edges of the ntal prmtve and the extracted edge pxels should be small. Therefore, t s reasonable that the fttng algorthm uses only the edge pxels dstrbuted n a buffer zone of the projected edges shown as Fgure 4..4 Objectve Functon and Least-squares Adjustment That the fttng condton we are lookng for s the projected edge exactly falls on the buldng edges n the mage. It means that the dstance d s consdered as a dscrepancy and s expected to be zero. Therefore, the objectve of the fttng functon s to mnmze the dstance between the extracted edge pxels T and the projected edge v v. The projected edge s composed of the projected vertces v (x, y and v (x, y. Suppose there s an extracted edge pxel T (x t, y t located nsde the buffer, the dstance d from the pont T to the edge v v can be formulated as the followng equaton: d ( y = y x t ( x ( x x x y t ( y ( y x y y x 3 where s the ndex of extracted edge pxels. ( The photo coordnates v (x, y and v (x, y are functons of the shape, pose, and exteror-orentaton parameters, where only the exteror-orentaton parameters are known. E.g., a box prmtve wll have 7 unknowns: w, l, h, α, dx, dy, and dz. Every extracted edge pxel n the buffer may form an objectve functon as Equaton (. Snce d represents the dscrepancy between model and mage, the objectve of fttng s to mnmze the squares sum of the dstances: Σd = Σ[F ( w, l, h, α, dx, dy, dz] mn. (3 Applng the least-squares adjustment to the fttng functon, the dstance d s consdered as the resdual errors v. Therefore, Equaton ( should be rewrtten as follows: v = F ( w, l, h, α, dx, dy, dz (4 Equaton (4 s nonlnear and the unknowns cannot be calculated drectly. In order to solve the unknowns usng Newton-Rapheson method, Equaton (4 should be dfferentated wth respect to the unknown parameter. Then, t may be expressed n the lnearzed form as: Buffer 33 d Projected Edge Fgure 4: Dstance from the extracted pxel to the projected edge and the effectve buffer zone. v t t x

4 F = α F F v w l h dx dy dz w l h dx dy dz α In Equaton (5, F s the approxmaton of the functon F evaluated by the gven ntal value of the 7 unknowns - shape and pose parameters. F, F, etc., are the partal dervatves of F wth respect to the ndcated w l unknowns evaluated at the ntal approxmatons. Δw, Δl, Δh, etc., are ncrements of the unknowns appled to the ntal approxmatons. Each extracted edge pxel can form such an equaton for the correspondng projected edge. Therefore, a vsble edge may have a group of observaton equatons. All of these equatons can be summarzed and expressed by the matrx form: V=AX-L, where A s the matrx of partal dervatves and X s the matrx of ncrements of shape and pose parameters. Once an edge pxel s added, t adds a row to matrx A, V, and L. A box prmtve, whch has up to 9 edges are vsble, may have 9 groups of equatons for one sngle photo. It s easy to solve the matrx X by the matrx operaton: X=(A T PA - A T PL. If the weghts of all unknowns are assumed equal, P s an dentcal matrx. The soluton of X s the ncrements of shape and pose parameters. If any element of matrx does not meet the requrement of the threshold, addng the ncrements on the prevous parameters to regenerate the new matrx A and L, then mplement the adjustment agan. The teraton contnues untl the ncrements converge below the thresholds or dverge beyond the lmts. When the teraton converges, the prmtve parameters are determned and the model s ftted to the mage. If a buldng s vsble on multple photos, t can provde more observaton equatons. These redundant observaton equatons sometmes are very useful especally when an edge s occluded on one photo but vsble on another photo. All of the observatons on dfferent photos jon the least-squares adjustment to solve the model parameters smultaneously 3. CORRECTNESS AND ACCURACY ASSESSMENT In order to study the performance of the LSMIF algorthm, we choose varous buldngs n the NCKU campus from the dgtzed aeral photos. Except buldng s modeled by one sngle prmtve, the others are composed of two or more than two prmtves. For accuracy assessment, an experenced photogrammetrc operator measured all of the vsble buldng corners. 3. Pull-n Range Test The pull-n range s the range from the ntal value to the convergent value of each parameter. It can be taken as a reference of the maxmum error that the fttng functon can tolerate, or the least accuracy the ntal value should acheve. The part of buldng 5 (Fgure 5 was chosen for the pull-n range test. A box prmtve s selected to ft the buldng part, and the shape and pose parameters have been solved by LSMIF. The pull-n-range of each parameter s tested on by one by addng a certan amount of error onto the correct number untl the teraton dvergent. The pull-n ranges of the 6 parameters are shown n Fgure 6. F (5 Fgure 5: The buldng and the box prmtve for the pull-n range test o dx dy dz l w h 9 o -α o Fgure 6: The pull-n range of the each parameter. Due to the nfluence of rrelevant nformaton to the buldng or self-occluson, the pull-n ranges could be case by case. In ths case, one can fnd that the two parameters: dz and h, have better pull-n ranges. It s because the 4 edges on the roof are extracted clearly and successfully, whch provdes a good constrant to the parameter h, especally n the postve drecton. There are also bottom edges are clearly vsble n the left photo, whch provdes the constrant to dz and h both n the negatve drecton. The pull-n range of dz n postve drecton s not as well as h s due to the nose pxels extracted on the roof of part of the same buldng (lower-rght n the photos. The parameter dx has a narrowest pull-n range among other parameters. It s because some trees occlude the bottom edge on the rght sde of the buldng, and the boundary of trees s extracted as edge pxels. When the dx ncreases, the rght edge of the box prmtve wll be ftted to the boundary of trees and results n the wrong soluton. 3. Correctness Analyss and Constrants Because the LSMIF algorthm fts the model to the extracted edges n mages, the correctness of fttng depends on the correctness of the edge extracton. Any edge pxels do not formed by the buldng may cause ncorrect fttng results. Fgure 7 llustrates some of these crcumstances.

5 (a (b (c (d Fgure 7: Four examples of ncorrect fttng results caused by (athe texture of the roof, (bthe shadow of the buldng, (cthe balcones, and (dthe gable-roof s not symmetrc. There are two approaches to fx ncorrect fttngs. The frst approach s to gve a set of very accurate ntal parameters wth the use of a narrow buffer. However, t may not work f the real edge s occluded or ambguous. Besdes, the use of narrow buffer tends to cause LSMIF dvergent f the buffer does not cover the wanted edge pxels. The second approach s to add constrants to the LSMIF. If a certan parameter has been known from other data source, one can treat the parameter as an observaton and add an observaton functon to the least-squares adjustment. By rasng the weght of the observaton, the parameter wll tends to be fxed. Ths s especally useful n the stuaton of nformaton loss. For example, shadows or other buldngs have occluded the bottom edges of the prmtve. If the nearby ground heght can be measured, the parameter of dz can be determned and fxed. Fgure 8 shows an nstance n whch the bottom edges are nvsble but the fttng can be carred out successfully by addng constrants to h and dz. Therefore, the buldng can be extracted completely. Ths s one of the great benefts that make MBBE superor to the tradtonal stereo pont-by-pont measurement. In the case study, the buldngs can be decomposed nto 3 buldng parts and each can be modeled ether by box or gable-roof prmtves. The ftted prmtve s projected onto the mages as a wre-frame for checkng. Table lsts the checkng results. The extracton of buldng 4 was ncorrect because the bottom and the wall edges are almost nvsble. But ths can be mproved by ntroducng the constrants of dz and h nto the LSMIF. The percentage of correct results s about 89%, whch proves the flexblty of LSMIF. Fgure 8: The fttng results wth constrants. 3.3 Accuracy Assessment The fttng results have been compared wth the manual measurement data. For comparson, the 3D vertex coordnates of each prmtve are derved from the shape and pose parameters. After elmnatng the ncorrectly ftted edges, the coordnate dfference between LSMIF and the manual measurement represents the emprcal accuracy of fttng. The mean values and standard devatons of the dfferences are lsted n Table 3. Accordng to the standard devatons of the dfferences, the emprcal accuracy n each coordnate wll be: 33cm(X, 8cm(Y, and 3cm(Z. Fgure 9 depcts the dstrbuton of the dfferences on the X-Y plane. The mean values are close to zero, so that no systematc errors are expected. The emprcal accuracy n X-Y plane s about 4cm. Z has a larger standard devaton because most of the bottom edges of the buldngs are nvsble. Therefore, the emprcal accuracy n 3D was downgraded to about m. Table : The correctness check of LSMIF. Buldng Part Prmtve Vertex LSMIF Results ID ID Type Number Correct Incorrect Gable-roof 6 6 Box 4 4 Box Box 5 5 Box Box 6 4 Gable-roof Box 4 4 Box Box 4 4 Box Gable-roof 6 6 Box 8 Box 3 Box Gable-roof 5 5 Box Box 4 4 Box Box 5 5 Box 4 4 Box Box 4 4 Total Percentage 88.54%.46%

6 Table 3: The statstcs of dfferences from manual measurement to LSMIF results. X Y Z X Y X Y Z Maxmum (m Mnmum (m Average of Absolute Values (m Average (m Standard Devaton (m ΔY ΔX Fgure 9: The error dstrbuton on the X-Y plane. 4. CONCLUSIONS A least-squares model-mage fttng algorthm s ntroduced and evaluated for the use of model-based buldng extracton from aeral mages. Ths algorthm teratvely fts the pre-defned prmtve to the aeral photos and solves the optmal model parameters wth known orentaton parameters of photos. It s able to use multple photos smultaneously, whch s helpful when some edges are vsble n one mage and others are vsble n the second or other mage. By addng constrants, such as ground heght and roof heght, the buldng can be entrely extracted even ts bottom s occluded. Furthermore, the LSMIF extracts buldngs object by object, so that t would be more effcent than pont-by-pont measurements. The performance of the LSMIF algorthm s evaluated by analyzng the pull-n range of parameters, the correctness and the emprcal accuracy of the fttng. In the pull-n range test, t shows that there s at least a ±m pull-n range for each parameter n average. It means that the operator does need to provde very accurate ntal approxmaton of parameters for fttng n common stuatons. Accordng to the results of study cases, the fttng algorthm s functonng very well and effcent. The emprcal horzontal accuracy s comparable to manual measurements. The vertcal accuracy s low due to shadows and occluson. The accuracy and relablty would be able to be mproved f more overlapped photos are used or more constrants are ntroduced. In general, the LSMIF delvers satsfyng results and has a convncng potental to be appled for MBBE. REFERENCES Braun, C., T. H. Kolbe, F. Lang, W. Schckler, V. Stenhage, A. B. Cremers, W. Förstner and L. Plumer, 995. Models for Photogrammetrc Buldng Reconstructon, Computers & Graphcs, 9(, pp Chapman, D. P., A. T. D. Deacon, A. A. Hamd and R. Kotowsk, 99. CAD Modelng of Radoactve Plant - The Role of Dgtal Photogrammetry n Hazardous Nuclear Envronments, Proceedngs of The XIXth Congress of The Internatonal Socety for Photogrammetry and Remote Sensng, Vol. XXIX, pp Chou, H., Y.-H. Tseng and S. Wang,. Least-squares CSG Model-mage Fttng, Proceedngs of the th Conference on Surveyng Academy and Applcatons, Chungl, Tawan, Vol.. Förstner, W., 994. A Framework for Low Level Feature Extracton, Computer Vson, Sprnger-Verlag, pp Förstner, W., D-Cty Models: Automatc and Semautomatc Acquston Methods, Photogrammetrc Week '99, Stuttgart, Vol. 47, pp Fuchs, C., 995. Feature Extracton, Second Course n Dgtal Photogrammetry, IFP Unversty Bonn, pp. p.3-~p Fuchs, C. and W. Förstner, 995. Polymorphc Groupng for Image Segmentaton, The 5th ICCV'95, Boston, pp Grün, A.,. Sem-automated Approaches to Ste Recordng and Modelng, Proceedngs of The XIXth Congress of The Internatonal Socety for Photogrammetry and Remote Sensng, Amsterdam, Netherlands, Vol. XXXIII, pp Gülch, E., 996. Extracton of 3D Objects from Aeral Photographs, Report No. Grant 5TT9536, BMBF/DARA GmbH, Bonn. Lang, F. and W. Förstner, D-Cty Modelng wth a Dgtal One-eye Stereo, Proceedngs of The XVIIIth Congress of The Internatonal Socety for Photogrammetry and Remote Sensng, Venna, Vol. XXXI, pp Ln, W., Y.-H. Tseng and S. Wang,. On the Model Establshment and Operaton for CSG Model-based Buldng Extracton, Proceedngs of the th Conference on Surveyng Academy and Applcatons, Chungl, Tawan, Vol.. Lowe, D. G., 99. Fttng Parameterzed Three-Dmensonal Models to Images, IEEE Transactons on Pattern Analyss and Machne Intellgence, 3(5, pp Sester, M. and W. Förstner, 989. Object Locaton Based on Uncertan Models, Mustererkennung 989, Sprnger Verlag, pp Tseng, Y.-H. and S. Wang,. CAD-Based Photogrammetry for 3D Cty Model Reconstructon, The Congress of The Chnese Geographc Informaton Socety, Tanan, Tawan, Vol. pp Veldhus, H., 998. Performance Analyss of Two Fttng Algorthms for the Measurement of Parametersed Objects, Internatonal Archves of Photogrammetry and Remote Sensng, Vol. 3, pp Vosselman, G., D Measurements n Images Usng CAD Models, Proceedngs of the 5th Annual Conference of ASCI, Hejden, Netherlands, pp Vosselman, G. and H. Veldhus, 999. Mappng by Draggng and Fttng of Wre-Frame Models, Photogrammetrc Engneerng & Remote Sensng, 65(7, pp Wang, S. and Y.-H. Tseng,. Least-squares Model-mage Fttng for Model-based Buldng Extracton, Proceedngs of the Asa GIS, Tokyo, Japan, pp. -8.