ERSO - Acquisition, Reconstruction and Simulation of Real Objects 1

Transcription

1 ERSO - Acquisition, Reconstuction and Simulation of Real Objects 1 Holge Diene, Ullich Köthe Faunhofe-nstitut fü Gaphische Datenveabeitung Joachim-Jungius-Staße 11 D-1859 Rostock, Gemany holge@egd.igd.fhg.de ulli@egd.igd.fhg.de Benhad Ristow Faunhofe-nstitut fü Gaphische Datenveabeitung Rundetumstaße 6 D Damstadt, Gemany istow@igd.fhg.de Abstact A basic system fo acquisition, econstuction and simulation of eal objects (ERSO) is pesented. Ou appoach is to combine the knowledge of diffeent eseach aeas as photogammetic, compute gaphics, and compute vision to develop new techniques fo geneating thee-dimensional models fom images. The fist two paagaphs give an intoduction and oveview of the system achitectue. The following paagaphs descibe in moe detail the diffeent pats of the econstuction pocess.. NTRODUCTON n many application aeas, such as achitectue, vitual scenes, medicine, and photogammety, the acquisition of an abstact desciption of thee-dimensional objects is an impotant topic. A common appoach to this poblem is based on photogammetic methods, which equie a lage amount of manual wok. Since efficient visualization and animation systems ae inceasingly available, the effot necessay fo data acquisition tuns moe and moe into a limiting facto fo the use of these systems. n the ERSO poject ou goal is the development of a basic system that integates data acquisition with the econstuction of scenes and objects. The system uses gaphical inteactive data handling methods and simulation tools to help achitects, developes of vitual wolds o simila use goups to geneate a 3D model fom images of a eal object and to use this model in thei own applications. Within the scope of the ERSO poject new methods fo the automation of data handling and acquisition pocesses will be developed. Fo this, automatic and inteactive algoithms will be combined in ode to maximize the efficiency of futue intelligent vision systems. Macus Scheye nstitut fü Photogammetie und Katogaphie TU Damstadt Petesenstaße 13 D Damstadt, Gemany macus@gauss.phg.vem.tu-damstadt.de Ulf Stelbe Zentum fü Gaphische Datenveabeitung e.v. Rostock Joachim-Jungius-Staße 11 D-1859 Rostock, Gemany ust@ostock.zgdv.de. SYSTEM ARCHTECTURE To geneate thee-dimensional models fom images an adequate pool of data is necessay, and it is impotant to get geometic data of the object as well as specific suface attibutes (e.g. textue). Theefoe, we developed a knowledge base to enable non-expets in photogammety to take pictues sufficient fo the econstuction pocess. Thee ae no specific hadwae equiements fo the camea. Amateu cameas can be used as well as pofessional measuement cameas. With the camea calibation module it is possible to calculate camea paametes (inne image oientation) and eliminate distotion in the images caused by non-pofessional cameas. Reconstucting the 3D coodinates of a single object point needs at least two obseved images of the object point and the oientation between these images. n ou case the oientations between two abitay images ae unknown. So the fist step in the econstuction pocess is to estimate the (exteio) image oientations. This is possible if thee ae moe than eight coespondences of peviously detected image points. Befoe stating the estimating pocess coespondences ae found by a semi-automatic matching pocedue. 1 The poject is sponsoed by the Volkswagen-Stiftung Fig. 1. System achitectue Diene / 1 Pape d: SS-2/4

2 @ - E D?, C >.., To get moe pimitives fo featue matching each image is segmented into points, edges and egions and thei topological connections ae stoed in featue adjacency gaphs. The constuction of a coase 3D-model is possible by matching these gaphs. The next step in the econstuction pocess ceates othophotos (images that would be obtained by an othogonal pojection) and depth maps fom diffeent oiented views which include the exact 3D geomety of the object. Now it is possible to geneate a polygonal model and textue mappings to display and contol the econstuction esult. Many applications need semantic infomation, theefoe we ae developing tools to suppot use defined selection and gouping within the polygonal model.. CAMERA CALBRATON USNG THE PHOTO- GRAMMETRC BUNDLE METHOD The objective of camea calibation is to detemine the inteio geometic quantities of a camea used fo image acquisition. Among othes this is a main pemise in ode to achieve high accuacy and pecision fo econstucting objects. No matte of using digital o analog devices thee can usually be two appoaches distinguished to calibate a camea: Test ange calibation: n this case one o multiple pictues ae taken of a specialized taget aea consisting of well known signal object points. Since this method equies an expet based taget aea and leads to an additional image scene beside the main application data this method has become quiet unpopula. Self-calibation: The pinciple of this method is to include the task of calibation within the job of evaluating all the object points which ae necessay to econstuct the elevant object of the application. Usually this method leads to a highe accuacy in efeence to the object points at the cost of a small set of additional paametes which have to be estimated. This method is widely used in many applications in industial close ange photogammety and shall be consideed as the favoite appoach fo camea calibation in ou poject. The backbone of the calibation pocedue is the photogammetic bundle method which has been fully developed fo the case of metic cameas. t plays a citical ole in the exploitation of the best potential accuacy and pecision in photogammetic applications such as object econstuction. The basic idea behind it is to detemine a set of inteio as well as exteio camea paametes and a set of object points based on the elationship of multiple ovelapping photogaphs. t is based on a mathematical camea model that descibes the elationship between the 3Dobjectspace and the 2D-imagespace. A. Mathematical Camea Model The functional model consists of the so called collineaity equations which ae deived fom the pespective tansfomation. They ely on the basic assumption that the object point, the pespective cente of the camea and the coesponding image point fom a staight line. n efeence to each obseved image point ( X, Y ) of an image ) co- Fig. 2.: Collineaity pinciple esponding to an object point ( 8, 9, : ) this model is witten in the following fom: + + = = ( ) ( ) ( + + ) "#" $%$ &'& () () ()! * (+) + + =, = -/...,, (3.1) ( "#" ( $%$ ) ( &'& ) + (+), + (, ) ), with the paametes of the inteio and exteio oientation of the image ) :, and C1 : nteio oientation (Pincipal point and X Y pincipal focus),, : Cente of pespective of the exteio : oientation,, : Angles of the othogonal otation matix 2 = ( R176 ) of the exteio oientation X and Y epesent geometic coection tems that ω34353 φ κ image O AB R R F GPJLQJO FHGKJLMJLN JO ae applied to the image coodinate measuements to model and compensate the systematic eos due to the lens, solid state sensos, film defomation etc. These tems ae essential in the case of camea calibation and usually diffeent types of independent additional functions ae applied to model the systematic eos. The set of paametes used fo these functions ae called the additional paametes (APs) of the bundle method. n ode to make the bundle method applicable to a wide ange of amateu cameas we decided to povide the following thee independent types of functions to model systematic distotions. Radial Lens Distotion: To descibe adial lens distotion which has usually the geatest impact we pefeed to use a simplified odd powe polynomial poposed by Fye & Bown [4] which leads to the following components in the image coodinate system: 8 ; 9: ( ) <= ( ) (3.2) = + + X X X + R + R + R = + + Y Y Y + R + R + R n this appoach efes to the adial distance between the pincipal point and the image point. K 3, K 5 and K 7 epesent the additional paametes of the adial distotion function. Decenteing Lens Distotion: n this case we also followed a simplified fom deived by Fye & Bown [4] object Diene / 2 Pape d: SS-2/4

3 t u ~ d ~ with the additional paametes 1 and 2 : 2 2 S/T UHVWV 1 ( 2 ) X/Y ZH[W[ ( 2 2 ), (3.3) X\] ^H_W_ and Y\] ^H_W_ ae the distotion values in X = R + X + X Y Y = X Y + R + Y whee the image coodinates espectively and with its components x and y is the adial distance to the pincipal point. CCD Aay Distotion: The application of a digital senso equies an additional tem to compensate a shea and tilt in the CCD aay. n this context we followed an AP set suggested by Beye [1] that has poven to be effective: à`cb eaecf = ( ) + ( ) X X X S Y Y A Y = X X A, (3.4) whee a epesents a scale in x and Sg a shea of the CCD-aay. n summay we have expanded the conventional bundle method by thee additive sets of distotion tems with the additional paametes , 5, 7 of the adial distotion,, of the decenteing distotion and Sg and a of the 1 2 CCD aay distotion. B. Paamete Estimation The fist step in estimating the unknown oientation paametes as well as the object point coodinates is to establish the pai of equations fomulated in (3.1) fo each pai of image coodinates ( Xhi5hi, Y ) obseved on each image )j. Due to the non-lineaity of the equation (3.1) a lineaization is necessay and each obsevation which is accompanied by a measuement eo gets an additional coection tem Vk. By assuming initial values fo the unknown paametes and uncoelated image coodinate obsevations we obtain to the following Gauss-Makov Model: l l l l l7m m l2 1 L + V =! DX and Σ = σ (3.5) The design matix A is of dimension N M (n is the numbe of obsevations, m the numbe of unknown paametes) with N M and ank(!) = M. P descibes the weight matix, Σn/oo the covaiance matix and σn the standad deviation of unit weight fo the obsevations vecto v Lp. The iteative pocess is descibed by the index k. By applying the least squae minimization the unknown coection paametes ae given by the nomal equations: qhs uvu w w = with x. = xy! x! and z N = z{! z L (3.6) The pecision of the estimated paametes is contolled by the covaiance matix: Σ /}} σ σ with (3.7) V V =!! = N M To obtain the solution fo the desied paametes this leads to the Gauss-Newton iteation whee in each iteation equation (3.6) has to be solved: v v v X X D X K + 1 = +, =, 1,..., (3.8) v with D X deived fom (3.6). The iteation is stopped if the nom of the coection vecto falls below a cetain constant value close to zeo. V. SEGMENTATON AND MATCHNG FOR OR- ENTATON OF THE MAGES n the pocess of econstuction each image have oientation paametes in objectspace (exteio oientation) and cameaspace (inteio oientation) to pefom the pespective tansfomation. We assume that these paametes ae unknown. Theefoe, they have to be estimated in a sepaate pocedue. n this pocedue we detect cones in the images and find thei coespondences in othe images. Finally, with all coespondences the oientation of the images can be estimated. A. Cone Detection Thee ae seveal image filtes to detect cones in images. We decided to use two types of cone detectos which ae basically diffeent. The pinciple idea of the SUSAN opeato [13] is to compae the intensities of all pixel in a cicula mask to the intensity of the cente pixel. n the case of a cone most of the pixel in the mask diffe fom the cente pixel. One majo popety of the SUSAN cone detecto is the good pefomance in the pesence of noise. Othe cone detectos make use of the image deivatives which ae stongly influenced by noise. The second cone detecto is a statistically motivated appoach [3]. Theefoe, the aveage of the gadient image ae used to maximize the following function: ƒ det #( X) G G G max with # ( X ) 2 = tace #( X) 2 G G G (4.1) Once we have detected some cones in the images with one of the two methods descibed above we ty to match them semi-automatically. B. Point Matching Algoithm When woking on dense image sequences the coss coelation is successfully used to match featues between images. The coss coelation consides the similaity of the gay values in a window aound the cones. n the case of wide baselines between the images, as in ou case, the Fig. 3. The SUSAN Pinciple Diene / 3 Pape d: SS-2/4

4 coss coelation between the cones in diffeent images can not be used. The pespective tansfomation changes the gay values in a window so that only a tanslation of a window does not detemine the coespondence in an othe image (see Fig. 4). Theefoe, we use a semi-automatic matching pocedue. n a fist step we select manually the necessay numbe of points to estimate the epipola geomety. Based on this estimation we use the A* algoithm to find additional coespondences. The cost function is a combination of the distance to the epipola line and the diection of the flow vecto in compaison to the flow vecto of the neaest found coespondence. C. Estimation of the oientation Fist the elative oientation between all image pais is estimated using the eight point algoithm [2]. The object point and the pojected object points in image one and two define a plane. This plane intesects the image plane. The intesection line is called the epipola line. This fact can be expessed by the following fomula: U! 42! U = U & U = (4.2) The matix A contains the paametes of the pespective tansfomation, T and R the paametes of the oientation in space. With at least eight coespondences we can solve a least squae minimization poblem to estimate the paametes of the exteio oientation. With all elative oientations we build a econstuction tee and ty to find the path with the minimum eo of back pojection. Using this path all elative oientations ae tansfomed into one coodinate system. Finally, all inteio and exteio paametes ae optimized in a bundle block pocedue. V. 2D-SEGMENTATON AND CREATON OF A COARSE 3D-MODEL A. Segmentation So fa, all pocessing steps wee solely based on points in 2D and 3D. Howeve, this is not sufficient to obtain the highe level desciptions we want to geneate. We must detect a iche set of featues within each image and stoe a desciption of thei diffeent popeties. A featue adjacency gaph contains points, edges, and egions and epesents thei topological and geometical elationships. Thee is a wealth of segmentation algoithms in the liteatue. Theefoe, we fist had do study seveal algoithms to assess thei suitability fo the ERSO system. Somewhat to ou supise, elatively simple algoithm outpefomed much moe complicated ones. Nevetheless, no single algoithm is sufficiently accuate to be used in isolation. To combine seveal algoithms, we use an impoved vesion of the method descibed in Köthe [7]. The basic algoithms ae the following: Diffeence of Exponential Edge Detecto: We convolve an image with an exponential filte E and take the diffeence to the oiginal: (, ) = (, ) ( )(, ) $O% X Y ) X Y % ) X Y with Fig. 4. Windows aound a cone in two diffeent images ( σ ) (, ) = exp ( + ) /. % X Y. X Y whee σ denotes the scale of the opeato. n this image, zeo cossings ae detected to yield an edge image. Since the edges in this image still contain gaps, they will only be used to initialize a egion gowing algoithm. Scale selection: Using a scale selection pocedue as descibed in Köthe [8], we locally compute plausible scales fo edges and egions. By compaing the edge detection esults at diffeent scales with the local appopiate scales, we can mege multiple edge images into one. Seeded Region Gowing: Stating fom a set of seed egions, in this case obtained by inveting and eoding the edge image, each pixel not aleady labeled is assigned to the egion into which it fits best. By using a global pioity queue, it is ensued that good fits ae pocessed fist, so that two egions will always meet at pixels that don't fit well into eithe one. The obtained segmentation is subjected to cetain topoplogical tansfomations to avoid well known connectivity poblems. Fom this we finally extact the featue adjacency gaph which epesents the position and neighbohood infomation of the featues as well as any additional infomation (such as gay levels) that will be useful in the subsequent featue matching step. B. Featue Matching To combine the featue adjacency gaphs of all images we decided to use the gaduated assignment algoithm fo gaph matching by Gold and Rangaajan [5]. This algoithm compaes popeties of nodes and edges in a gaph to calculate an appoximate solution of the matching poblem in an iteative pocess and is descibed biefly. The esults of compaing nodes a and i fom diffeent gaphs and all thei adjacent edges ae stoed in the element W ˆ of the weight matix W. The appoach of the algoithm is to minimize the cost function 1 2 Š = % - W - M Š with a doubly stochastic matix - Š (5.1) = ( M ˆ ) that means the elements of evey ow and evey column of M ae nonnegative and sum up to 1. An element M ˆ gives the pobability of a coect coespondence between the node a and the node i. As witten in (5.1) the compaing esults ae combined with M and ae calculated as follows: Œ = + Ž Ž Ž Œ Ž (, ) (, ),(, ) W - # A M # A B J with (a,b) the edge fom a to b, nb(a) all neighbo nodes Diene / 4 Pape d: SS-2/4

5 ž ž Ÿ Ÿ of a, # and # the compae functions of nodes and edges espectively. The development of a Taylo seies of (5.1) shows that minimizing the cost function E means to maximize the function = š /š /š % - W - M (5.2) with a fixed matix ( ) -. This is an assignment poblem and it is solved fo each iteation by the softassign algoithm descibed in [5]. The inne loop fo solving (5.2) is eithe stopped on convegence of M o afte a fixed numbe of iteations. n the oute loop a contol paamete fo the softassign algoithm is inceased and the pocess is stopped afte a given numbe of iteations. V. CREATON OF ORTHOPCTURES AND THE EX- ACT 3D-GEOMETRY The objective of Facets Steeo Vision (FAST-Vision), a method developed by Wobel 1987 [14], is to econstuct simultaneously the object suface Z(X,Y) and its optical density G(X,Y) with X, Y as independent suface coodinates of a egula quadatic XY-aste. Usually, G(X,Y) can be egaded as an othophoto of the object. The elationship between a point on a suface (sufel) and its pixel epesentations in the images P, P ;... can be descibed with egad to geometic and adiometic chaacteistics. The geometic elation between a sufel and its imagepoints is given by the well-known pespective equations aleady descibed in (3.1). To descibe the adiometic elationship between the object and its images we chose to use linea tansfe functions T, T,... that have poven to be effective: '( 8, 9, :) = 4 (' ( X, Y )) = 4 (' ( X, Y )) =K (6.1) with G, G,... as the density values of the images P, P,... Sensos that epesent diffeent camea models can be chosen instead of (3.1) and the tansfe function does not necessaily has to be linea. Since (6.1) includes the object models as non-linea paametes the development of a Taylo seies to the fist ode is applied which leads to: 4 ' X Y ' 8 9 D' 8 9 ( (, )) = (, ) + (, ) + with 8œ = ' D: (, ) ' ( 8, 9 ) + 8 D: : : and 9 = 9 9 : :. (6.2) This equation is fundamental in the pocess of FAST- Vision. t shows on the left hand side the density of an imagepoint coesponding to a cetain sufel and povides a linea elationship to the impovement of the suface model dz and the density model dg. To educe the numbe of unknown paametes the quadatic XY-aste is independently esolved in a gid consisting of quadatic facets fo the geometic and the density model. Each facet within the aste defines the egion of validity fo a local intepolation function used fo the desciption of the geometic model and the density model espectively. This leads to the following diffeentials fo the models: D: 8 9 A 8 9 D: 8 9 D' D' 8 9 ž/ÿ (, ) =, (, ) Fig. 5. FAST-Vision concept ž/ÿ (, ) = α, (, ) (6.3) So fa we pefe to use a bilinea intepolato but basically all intepolation functions that epesent the geometic o density model in an appopiate way can be chosen fo (6.3). The unknown paametes that basically have to be estimated ae the nodes of the facets belonging to the intepolation functions dz and dg espectively. n ode to econstuct an object with the deived method at least a second image is equied. The values associated with image P in (6.2) have to be eplaced with those associated with the images P,P etc. The evaluation of (6.2) fo all pixels fom all pictues leads to a linea Gauss- Makov model analog to (3.5). This ovedetemined poblem is again solved by a least squae minimization of the esiduals which leads to the well-known nomal equations aleady descibed in (3.6). The non-linea natue of FAST- Vision with its lineaization in (6.2) equies a Gauss- Newton iteation to obtain the solution fo the geometic model and the density model simultaneously. V. SEGMENTATON OF THE 3D-MODEL AND N- TEGRATON OF SEMANTC NFORMATON Fist we have to geneate a polygonal model of the object geomety which is defined by seveal oiented depth images. A. Geneating a Polygonal Model Depth images ae typically consideed as a set of points, we conside the ange images as patially defined sufaces. By intepolating between the gid points the suface can easily be completed continuously. Now we use the implicit function H( X) = fo defining the suface [11] whee X is a point in 3D space. Fo the definition of H( X) we define Diene / 5 Pape d: SS-2/4

6 fo each depth image F a function G X which measues the signed distance between a point in space and the intepolated suface in F U V (, ). The coesponding point in the depth image is found by pojecting the point X onto the paamete gid of the ange image. A positive distance indicates that the point lies between the obseve and the suface, a negative indicates that it lies below the suface. The synthesis of seveal depth images is achieved by combining the functions G X as follows: H X { G ( X )} = max (7.1) The point X is called visible if H( X). This citeion is used fo each voxel in a volume which contains the complete object. Afte the geneation of the volumetic model, the maching cube algoithm with a look-up table that esolves ambiguous cases (see Fig. 6) can be applied to geneate a polygonal epesentation [9] [1]. The accuacy of this polygonal mesh is impoved by moving the vetices of the mesh onto the suface implicitly defined by the egisteed ange images. The point is moved between the voxels of the cube until the zeo cossing of the signed distance function is found. The exact coodinates of the moved vetices ae intepolated in the ange images. This step is simila to a ay casting algoithm which is combined with the visibility citeion. This way, the full accuacy of the scanning device is exploited at all vetices of the mesh, and concave object shapes ae also modeled coectly. The amount of polygons can be significantly educed by applying a polygon eduction [12] [6]. B. ntegation of Semantic nfomation Fig. 6. Tiangulation of a cube by applying Maching-Cubes and impoving the mesh The polygonal model does not contain any semantic infomation such as this polygon belongs to a window of a house. Theefoe pats of the polygonal model must be clusteed and epesented by a semantic desciption. t is necessay to use a scene desciption which allows a semantic tee whee the nodes descibe semantic pats of the object. These nodes consist of the semantic infomation such as doo and the geometic epesentation (a subset of polygons) in the polygonal model. Because of the complexity of semantic infomation we offe some tools to goup pats of the geometic model and give them a semantic label. Thee ae manipulatos which allow to goup polygons manually o to select one and seach automatically fo othe polygons which ae simila to the selected. The similaity of the polygon may be the topological infomation, e.g. seach fo all quadangles, the geometic infomation, e.g. seach fo all polygons with the same aea, o the textual infomation, e.g. seach fo all ed aeas. V. CONCLUSON The ERSO poject will be sponsoed until May Cuently the modules ae integated in the use inteface and a demonstation application is pepaed. X. REFERENCES [1] H.A. Beye, "Geometic and adiometic Analysis of a CCD camea based on a Photogammetic Close ange system.", Thesis, nstitute fo Geodesy and Photogammety, Zuich, 1992, pp [2] O. Faugeas, Thee Dimensional Compute Vision. The MT-Pess Cambidge, Massachusetts, London, England, [3] W. Foestne, "A Featue Based Coespondence Algoithm fo mage Matching", nt. Ach. Photogamm. Remote Sensoing, vol. 26, 1986, pp [4] J. Fye and D. Bown, "Lens Distotion fo Close- Range Photogammety", Photogammetic Engineeing and Remote Sensing, 52(1), 1986, pp [5] S. Gold and A. Rangaajan, "A gaduated assignment algoithm fo gaph matching", EEE Tans. Patten Analysis and Machine ntelligence, vol.18, no.4, 1996, pp [6] H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, and W. Stuetzle, Mesh optimisation, in Poc. of SGGRAPH 93, Anaheim, Califonia (1993), pp [7] U. Köthe, "Pimay mage Segmentation", in Sagee et al. "Musteekennung 1995", Poc. of 17. DAGM Symposium, Spinge 1995 [8] U. Köthe, "Local appopiate scale in mophological scale-space", in Poceedings 4. Euopean Confeence on Compute Vision, Spinge 1996 [9] W.E. Loensen and H.E. Cline, Maching cubes: A high esolution 3d suface constuction algoithm, in Poc. of SGGRAPH 87, Anaheim, Califonia (1987), pp [1] C. Montani, R. Scateni, and R. Scopigno, "A modified look-up table fo implicit disambiguation of maching cubes", Visual Compute 1 (1994). [11] P.J. Neugebaue, "Reconstuction of Real-Wold Objects via Simultaneous Registation and Robust Combination of Multiple Range mages", ntenational Jounal of Shape Modeling, Vol.3, No. 1&2 (1997), pp [12] W. Schoede, J. Zage, and W. Loensen, Decimation of tiangle meshes, in Poc. of SGGRAPH 92, Chicago, llinois (July 1992), pp [13] S.M. Smith and J.M. Bady, A New Appoach to Low Level mage Pocessing. DRA Technical Repot TR95SMS1c, 1995 [14] B. Wobel, "Facets Steeo Vision (FAST Vision) - A New Appoach to Compute Steeo Vision and to Digital Photogammety", FPPD 1987 Diene / 6 Pape d: SS-2/4