Exteror Orentaton usng Coplanar Parallel Lnes Frank A. van den Heuvel Department of Geodetc Engneerng Delft Unversty of Technology Thsseweg 11, 69 JA Delft, The Netherlands Emal: F.A.vandenHeuvel@geo.tudelft.nl Abstract In ths paper an effcent drect soluton for the determnaton of the exteror orentaton parameters of an mage s presented. The method requres two sets of parallel lnes n obect space that are coplanar (a parallelogram. The need for ths method arses n applcatons where parallelograms or rectangles are common obect features. Ths s especally true for 3D-modellng of (most buldngs, currently a popular applcaton of photogrammetry and computer vson. The procedure conssts of two steps. In the frst step the ratos of the dstances from the proecton center to the corner ponts of the parallelogram are computed usng only parallelty nformaton. Wth the results of the frst step model coordnates (3D-coordnates n the camera system at arbtrary scale can be computed. In the second step an obect coordnate system s ntroduced n order to determne the exteror orentaton parameters of the mage n ths system. The advantages of ths procedure are the absence of sngulartes (as long as the four ponts are not on (or close to one lne n the mage and ts effcency. Ths makes t a sutable procedure for pose estmaton n real-tme applcatons. The frst step of ths method s successfully appled for 3D-reconstructon of (parts of a buldng from sngle mages. 1 Introducton The exteror orentaton procedure descrbed n ths paper has been developed as a part of a research proect n the feld of close-range photogrammetry appled to 3Dmodelng of buldngs. The need for computer models of buldngs s apparent n Computer Aded Archtectural Desgn (CAAD. Archtects want to show ther desgns and how they ft n the exstng envronment usng computer anmatons [4]. Computer models are not only used n archtectural desgn but for mantenance and renovaton purposes as well. Of most of the exstng buldngs there s no computer model avalable mostly because they were not desgned wth the help of a CAD-system. Here the demand for reverse engneerng arses. Ths holds for detaled models of one or a few buldngs as well as for models that cover several blocks of houses or even complete ctes. These cty models are often derved from aeral photographs and are usually restrcted to ".5D".e. for each poston n the ground plane exactly one heght s avalable. There s a dfference n requred precson between the two types of models. Archtectural models usually requre a precson n the order of 1 to several centmeters whle for cty models the precson s at a level between 0.1 and 1 meter. The applcablty of the method dscussed n ths paper s not restrcted to modelng for archtectural purposes but s sutable for all applcatons where polyhedral obects wth parallel edges are nvolved. As the approxmate orentaton of aeral mages s qute straghtforward, ths method s prmarly developed for close-range photogrammetry and computer vson. Especally n archtectural photogrammetry the obects (buldngs show many straght edges. As these edges or parts of them are often well vsble n the mages, lnes are used more and more n the photogrammetrc applcatons where they appear. Buldngs are often desgned wth the use of only a few basc shapes. The shape that s occurrng n almost every buldng s the rectangle. It s the most common shape of wndows, doors and of course of the buldng tself. The rectangular shape of buldng features s an example of geometrc nformaton that can ad n the 3D-reconstructon. One of the bottle-necks of the 3D-reconstructon process s the ntal orentaton of the mages. Therefore the drect soluton for the exteror orentaton parameters that s presented n ths paper has been developed. Ths soluton explots avalable knowledge or assumptons about the obect geometry. The nformaton requred for ths method conssts of two sets of (coplanar parallel lnes n obect space for the reconstructon of model coordnates n the camera coordnate system (step 1 of the exteror orentaton and at least 7 parameters for the defnton of Reference: F.A. van den Heuvel, 1997. Exteror Orentaton usng Coplanar Parallel Lnes. Proceedngs of the 10th Scandnavan Conference on Image Analyss, Lappeenranta, ISBN 951-764-145-1, pp.71-78
the obect coordnate system n order to compute the parameters of the exteror orentaton (step. The detecton and precse postonng of the lnes n the mage that correspond to parallel edges of the obect s assumed to be successfully accomplshed, let t be manually or automatcally through mage analyss technques. The detecton of lnes n the mage that are parallel n obect space s equvalent to vanshng pont detecton. On ths topc qute some research has been undertaken [13]. Ths s even more true for edge detecton that can be appled for the precse postonng of the lnes n the mage. Before gong nto detal we gve a short overvew of related research on exteror orentaton and pose estmaton n the next secton. Then the frst step of the exteror orentaton procedure s dscussed. An alternatve to the method of the frst step s presented n secton 4 and the results of the two methods are proven to be dentcal. The second step of the procedure n whch the exteror orentaton parameters are determned follows. In secton 6 t s shown that the method for exteror orentaton can be modfed for relatve orentaton of two mages. The paper fnshes wth a dscusson of several tests where the frst step of the procedure s appled to 3D-reconstructon n the camera coordnate system. Related research Exteror orentaton n photogrammetry and pose estmaton as t s mostly called n computer vson s a popular research subect. The exteror orentaton problem can be descrbed n a general way as follows: What s the poston and orentaton of an mage n space relatve to (parts of an obect, based on the perspectve proecton of the obect n the mage and geometrc obect nformaton? In computer vson the problem s often stated the other way around. Startng wth a known poston and orentaton of the mage one s searchng for the poston and orentaton of the obect relatve to the mage. Ths s called pose estmaton. There s no fundamental dstncton between the two perspectves. The research n ths feld ams at a drect soluton to the problem as t s usually dffcult to fnd approxmate values needed for an teratve approach. A secondary goal s to requre only a mnmum amount of obect nformaton. The geometrc nformaton of the obect can be dverse. Most of the research has been drected towards a soluton for the problem wth a mnmum set of three ponts of whch the poston s known n space. Ths s called the perspectve three pont problem. An extensve revew of sx dfferent (but sometmes qute smlar drect solutons of whch the frst dates back to 1841, s gven n [8]. Most of the solutons show sngulartes and sometmes consderable numercal nstablty. Usually addtonal nformaton (read ponts are needed to select the correct soluton from the set of computed solutons. Some research has concentrated on the perspectve n-pont problem wth n greater than 3. Mostly the perspectve four pont problem s attacked (see [9] and [1]. Constrants on the poston of the ponts n space can be mposed, lke n [7] and [1] where the exteror orentaton s based on a rectangle n space. It s clear that addtonal ponts or constraned postons of the ponts n space make a consderable smplfcaton of the formulaton possble wth all ts benefts wth respect to numercal stablty and the reducton n the number of solutons possble. The obect nformaton can be specfed at a hgher level than coordnates of ponts namely n the form of lne nformaton whch s especally sutable for polyhedral obects. And at an even hgher level n the form of prmtves lke spheres or cylnders (see [5]. In these studes lne-to-lne correspondence (lne photogrammetry replaces pont-to-pont correspondence. Lne nformaton can consst of a full specfcaton of the poston and orentaton of the lnes n space lke n [14] and [11] or consst of relatons between the lnes lke perpendcularty and parallelsm as n [5]. In the latter case the exteror orentaton can only be completed after the ntroducton of an obect coordnate system. Especally n the feld of computer vson often the pose estmaton s combned wth the estmaton of nteror orentaton parameters (focal length and prncpal pont and camera calbraton parameters (lens dstorton lke n []. In our approach to the space resecton problem the obect nformaton conssts of two sets of lnes that are coplanar and parallel n obect space. The four lnes make up a parallelogram and thus our space resecton problem s a specal case of the perspectve four pont problem. Lens dstorton s assumed to be absent (or corrected for and nteror orentaton parameters are known unless stated otherwse. Frst the dstance ratos from the proecton center to the corner ponts of the parallelogram are derved from the perspectve proecton of the four lnes of the parallelo-
gram. Ths s the frst step n the exteror orentaton procedure. In the second step the exteror orentaton parameters can be computed wth the ntroducton of an obect coordnate system. 3 Step 1: computaton of the dstance ratos The measurements that defne 4 lnes n the mage are the nput for the frst step of the exteror orentaton procedure. These lnes are assumed to correspond to lnes n obect space that are coplanar and parallel, n other words they form a parallelogram. The mage coordnates of the 4 corner ponts of the parallelogram are nput to the algorthm. To arrve at these coordnates manual measurements can be performed or some knd of (straght edge detecton can be appled, followed by lne ntersecton. The detecton of the edges that make up the parallelogram can be performed manually, although research n automatng ths task has been undertaken. As nteror orentaton and camera calbraton are assumed to be known, we wll not dfferentate between mage and camera coordnates. In other words the vector that s bult from the mage coordnates (x,y and the focal length (c. s assumed to correspond to a drecton vector n obect space. In the camera system ths s the vector (x,y,-c. The algorthm for the computaton of the dstance ratos s based on [1]. We call t the volumetrc approach because t nvolves the computaton of volumes of the tetrahedrons that consst of 3 of the 4 ponts of the parallelogram and the proecton center (see fgure 1. The volumes of the 4 tetrahedrons (the 4 combnatons of 3 corner ponts can be computed n two ways: 1. From the 3 vectors defned by the mage coordnates, the focal length and the dstances to the 3 corner ponts (the focal length wll cancel out n the computaton of the dstance ratos; see below.. From obect nformaton.e. the area (A of the trangle formed by the 3 corner ponts and the dstance (h of the proecton center to the plane of ths trangle. Elmnaton of A and h results n 3 equatons n whch only the 4 dstances between the proecton center and the corner ponts are unknown. From these equatons the 3 dstance ratos can be derved. Fgure 1: Parallelogram,,k,l and volume of a tetrahedron. Volume of the tetrahedron (V k computed from vectors of mage coordnates (x., focal length (c and dstances (d. : k V k = d d d [,,k] wth: 6 k [,, k] = det(,, k x = = ( x, y,-c Notaton: In formulas where the letters,, k and l are used, a cyclc permutaton of the 4 ponts s allowed. For nstance the dentfer +1 can be substtuted by (and l+1 by, etc.. The cyclc order of the ponts s not to be changed. Matrces and vectors are prnted n bold. For coordnate(vectors n obect space captals are used. The volume of the same tetrahedron can be computed from ts area n the plane of the parallelogram (A k and the dstance from ths plane to the proecton center (h: = ( V k A h k 3 (
A k s the area of trangle k and could only be computed f obect coordnates of the corner ponts were avalable. But as we wll see n the sequel n step 1 of the exteror orentaton there s no need to ntroduce an obect coordnate system. Ths s due to the fact that we assume the 4 lnes to buld a parallelogram n obect space. Because of the parallelty a dagonal of the parallelogram always splts ts area n equal parts. And thus the areas A k of the 4 trangles n the plane of the parallelogram are all equal. As the same holds for the dstance h of the proecton center to the plane of the parallelogram t follows that the volumes of the 4 tetrahedrons are dentcal. Combnng (1 and (: [ [ ( Ah = d d d k,,k] = d d k d l,k,l] k k l [ [ ] = d k d l d k,l,] = d l d d l,, k l l These are the 3 equatons where only the four dstances d are unknown. The dstance ratos C can be wrtten as follows: C = d +1 + +3 d = [ k,l,] = [ x, x, x ] ( [,k,l] [ x, x, x ] +1 + +3 The followng remarks have to be made: 1. In the computaton of the determnants the focal length s a constant factor. In the dvson of two determnants t wll cancel out. Ths s why the dstance ratos are ndependent of the focal length as stated before.. The sgn of the determnant depends on the order of the corner pont vectors. The sgn s postve f they are ordered clockwse as seen from the proecton center. Otherwse t wll be negatve. In (4 the sgn cancels out n the same way as the focal length. 3. The determnants may not be (close to zero. In other words the proectons of the corner ponts may not be (close to one lne n the mage. It s obvous that the precson of the dstance ratos ncreases wth an ncreasng area of the parallelogram n the mage. The computaton of dstance ratos s the frst step n the dervaton of the exteror orentaton parameters of the camera. In the second step a 3D-coordnate system s ntroduced wth (at least 7 coordnates of the corner ponts of the parallelogram. Then the 6 parameters of the exteror orentaton are computed. Ths step can be seen as the absolute orentaton of a mono model. Model coordnates (3 (4 ( X m can be wrtten as a functon of one dstance d that defnes the scale of the model: X X m m d x x d = x ( = = λ = λc x (5 4 Step 1: the non-volumetrc approach The results of the prevous secton can be verfed wth a dfferent approach whch we call the non-volumetrc approach. Ths method uses the fact that the drecton of two parallel lnes n the camera system s defned by the perspectve proecton of the two lnes n the mage [3]. Ths s depcted n fgure for two sets of parallel lnes. The normal vector of the plane of the parallelogram (n s found wth the two drecton vectors (r 1 and r : r = ( k ( l 1 r = ( ( l k n = r r (6 1 Fgure : The vectors n the non-volumetrc approach The dstances to the corner ponts (d. can be expressed as a functon of the dstance from the proecton center to the plane of the rectangle (h: d = h n n The normal vector resultng from (6 s drected towards (7
the proecton center. In the expresson for the dstance ratos (C the sgn of the dot product n (7 cancels out: C = d 1 d = n = [, r, r ] n [, r, r ] 1 Obvously ths s a more complcated expresson than (4. In the sequel t s proven that the two approaches lead to the same result. Combnng (4 and (8 we get: [ 1 C =, r, r ] [,k,l] = [, r, r ] [,k,l] 1 (9 (8 r 1 and r can be wrtten as the sum of two products [18]: r ( k ( l 1 = = [,k,l] - [,k,] l = [,l, ] k - [,l,k] (10 shape of a parallelogram. The frst parameter s the rato of the dstances between the parallel lnes or the wdth-heght rato. The angle between the two drectons n obect space s the second parameter (see fgure 3. The parameters of the parallelogram depend on the focal length [10]. As a result the focal length can be determned f one of the parameters s known. In practce the focal length wll strongly depend on the nose n the mage coordnates and not be estmable at all f the optcal axs s perpendcular to the plane of the parallelogram. r = ( (l k = [,,k]l - [,,l]k (11 And so the normal vector to the plane of the parallelogram becomes: or: n = r r = [,k,l][,,k]( l 1 [,k,l][,,l]( k + [,k,][,,l](l k n = r r = [,l,k][,,k]( l 1 + [,l,k][,,l]( k + [,l, ][,,k](k l Substtutng (1 and (13 n (8 we get: and: Whch proves (4. n = [,k,][,,l][,l,k] n = [,l, ][,,k][,k,l] n n [,k,l] [,k,l] = (15 (14 5 The parameters of the parallelogram (1 (13 The shape of the parallelogram s defned by the mage coordnates of the four ponts and the dstance ratos between the proecton center and the corner ponts of the parallelogram. Two parameters are needed to descrbe the Fgure 3: The shape parameters of the parallelogram. The dstance between pont and s computed from the model coordnates (5: S = ( m - m m - m X X ( X X = λ ( C x - x ( C x - x Then the wdth-heght ratos S k become: (16 S = S S = ( C x x ( C x x k ( Ck xk C x ( Ck xk C x k (17 The number of possble dstance ratos s 1 (ncludng dagonals. In other words: there are 1 dfferent combnatons of k to compute ths shape parameter. The dstance ratos C do not depend on the focal length but ths does not hold for the model coordnates. So f the dstance rato S /S k s known t s possble to compute the focal length from (17: c = ( ( (( + ( C y - y C x - x + C y - y - S C k k x k - x S S ( 1- C k k - ( 1- C S k k Agan there are 1 permutatons of k possble. From ths formula t s clear that c can not be computed f C =1 and C k =1, that s f the optcal axs s perpendcular to the plane of the parallelogram. The angle between the two drectons n obect space of the parallelogram s computed from the model coordnates as follows:
( X - X ( Xl - X cos α l = X - X X - X l ( C x - x ( Cl xl - x = C x - x C x - x l l (18 From the 8 observatons n the mage ((x,y of 4 ponts the parameters of the shape of the parallelogram can be computed and the 6 parameters of the exteror orentaton as explaned n the next secton. So there s no redundancy for the determnaton of these 8 parameters. If the two drectons of the parallelogram are (assumed to be perpendcular we deal wth a rectangle n obect space. The known angle gves the possblty to determne the focal length. In case of rectangularty the focal length results from: ( ( ( ( l l l l c = C x x C x x + C y y C y y ( C 1 ( C 1 l ( Agan the focal length can not be calculated f the optcal axs s perpendcular to the plane of the rectangle. If the focal length s known (19 can be converted nto a condton equaton: 0 = ( C x -x ( C x - x ( l l If rectangularty s assumed the condton (0 can be used for testng ths assumpton. In case of acceptance the condton s mposed on the observatons. The adusted observatons and the derved dstance ratos are nput to the second step of the procedure: the computaton of the exteror orentaton parameters. 6 Step : exteror orentaton parameters For the second step of the exteror orentaton procedure an obect coordnate system s ntroduced by supplyng at least 7 (sutable obect coordnates. Ths step conssts of the computaton of the poston and orentaton of the mage n the obect system. As we have computed dstance ratos n the frst step, scale s not yet ntroduced and has to be determned n ths step of the procedure. In ths way step s fully dentcal wth a three dmensonal smlarty transformaton for whch 7 parameters have to be determned. In photogrammetrc termnology an absolute orentaton of the model represented by (5 has to be performed. The drect soluton for ths transformaton s based on [6]. where: 0 λ R x = X - X 0 λ R C x = X - X (1 λ scale factor (= λ R C x. X. X 0 rotaton matrx dstance rato proecton center - ponts and from step 1 mage coordnates the avalable coordnates n the obect system the coordnates of the proecton center The second step of the exteror orentaton s splt n parts: 1. computaton of the scale factor λ. computaton of the parameters of the rotaton matrx R and the locaton of the proecton center X 0 The scale factor s computed as the rato of a dstance n obect space and the correspondng dstance n model space: λ (= λ = X X x C x ( Wth (1 and (5 equaton (1 can be wrtten as: m 0 ( I + S X = ( I S( X X (3 Where the rotaton matrx R s wrtten as the product of two skew-symmetrc matrces. S s bult from the normalsed quaternon elements q. : wth: -1 R = ( I - S ( I+ S (4 0 - q q 3 S = q 0 - q 3 1 - q q 0 1 The quaternon elements q can be related to Euler angles. Equaton (3 can be rewrtten as: wth: X X = M q w (5 m The smlarty transformaton can be wrtten as follows:
0 - Z m - Z Y m + Y M = Z m + Z 0 - X m - X, - Y - Y X + X 0 m m q 1 q = q and w = ( I S X q 3 0 (6 Ths s a system of equatons that s lnear n the normalsed quaternon elements q and the auxlary vector w. The vector X 0 can be computed from (6 after w s solved from (5. These parameters can be computed f at least sx obect coordnates X are known. The known coordnates can not be an arbtrary set of obect coordnates as they wll have to defne the parameters. If the number of avalable obect coordnates exceeds 7 the equaton (5 wll become an overdetermned system of equatons. The parameters can then be solved through least squares. In such a case of redundancy the scale factor λ has to be determned teratvely through the use of adusted camera coordnates n ( (assumng the redundancy apples to the scale factor. 7 Relatve orentaton The procedure for the exteror orentaton as descrbed n the prevous sectons can be modfed for relatve orentaton. Then the same parallelogram has to be present n the mages for whch the relatve orentaton s to be determned. The frst step of the procedure, the computaton of the dstance ratos, remans unchanged. In the second step the model coordnates (represented by equaton (5 of the two mages are related through a smlarty transformaton. For the relatve orentaton equaton (1 becomes (superscrpts are used for the mage number: 1 1 1 λ λ C = C 0 R x x X wth λ = (7 λ The scale factor λ s the rato between the unknown scale factors of the two models ( λ 1 and λ. λ can be computed from a dstance n the two model systems: λ = x C x 1 1 1 x C x (8 Because the wdth-heght rato of the parallelogram s redundantly determned by two mages, two (slghtly dfferent values for λ can be computed. For practcal purposes such as ntal (approxmate value computaton the average of the two can be used n the remanng part of the procedure. For an optmum result n terms of precson an adustment can be appled. Therefor a condton equaton can be derved usng equaton (17. Wth the adusted observatons there s only one soluton for λ. The second parameter of the parallelogram, the angle between the two obect drectons, s redundantly determned n the same way. Its correspondng condton equatons can be found wth (19. Once λ s computed the parameters of R and X 0 are determned as n the prevous secton. Here the vector X 0 s defned n the model system of mage and thus defned up to a scale factor ( λ. R and X 0 contan the 5 ndependent parameters of the relatve orentaton. 8 Conclusons The presented method for exteror orentaton s applcable when a perspectve proecton of a parallelogram n obect space s avalable. It s a drect soluton that s free of sngulartes as long as the four corner ponts of the parallelogram are not on or close to one lne n the mage. Apart from the parameters of the exteror orentaton the two shape parameters of the parallelogram can be computed drectly from the observatons. If a rectangle n obect space s used nstead of a parallelogram a constrant has to be mposed on the observatons. Ths leads to an mprovement of the orentaton parameters. If the same parallelogram s present n several mages the relatve orentaton of the mages can be computed wth a method that s very smlar to the one for the exteror orentaton. Then the shape parameters of the parallelogram are redundantly determned and no other obect nformaton apart from the parallelty s requred. References [1] Abd, M.A.; Chandra, T., 1995. A new effcent and drect soluton for pose estmaton usng quadrangular targets: algorthm and evaluaton. IEEE transactons on pattern analyss and machne ntellgence, Vol.17, No.5, pp.534-538 [] Abd, M.A.; Chandra, T., 1990. Pose estmaton for
camera calbraton and landmark trackng. Proceedngs 1990 IEEE nt.conf. Robotcs and Automaton, pp.40-46 [3] Braun, C., 1994. Interpretaton von Enzelbldern zur Gebaeuderekonstrukton. Dssertaton Bonn Unversty, Druckere Schwarzbold, Germany [4] Coyne, R.; McLaughln, S.; Newton, S., 1996. Informaton technology and praxs: a survey of computers n desgn practce. Envronment and Plannng B: Plannng and Desgn, Vol.3, No.5, pp. 515-55 [5] Ferr, M.; Mangl, F.; Vano, G., 1993. Proectve pose estmaton of lnear and quadratc prmtves n monocular computer vson. CVGIP: Image understandng Vol.58, No.1, July, pp. 66-84 [6] Grafarend, E.W., Lohse, P., Schaffrn, B., 1989. Dre-dmensonaler Ruekwaerts-enschntt. ZfV, Vol.114, pp.61-67,17-137,17-175,5-34,78-87 [7] Haralck R.M., 1989. Determnng Camera Parameters from the perspectve proecton of a rectangle. Pattern Recognton, Vol. No.3 pp.5-30 [8] Haralck R.M.; Lee, C-N.; Ottenberg, K.; Noelle, M., 1994. Revew and analyss of solutons of the three pont perspectve pose estmaton problem. Int. Journal of Computer Vson, 13, 3, pp.331-356 PAMI Vol.17, No.11, pp.1090-1095 [14] Petsa, E.; Patas, P., 1994. Formulaton and assessment of straght lne based algorthms for dgtal photogrammetry. Proceedngs ISPRS Commsson V Intercongress Symposum Melbourne pp.310-317 [15] Tan, T.N.; Sullvan, G.D.; Baker, K.D., 1996. Closed-form algorthms for obect pose and scale recovery n constraned scenes. Pattern Recognton, Vol. 9, No. 3, pp.449-461 [16] Tommasell, A.M.G.; Tozz, C.L., 1996. A Recursve Approach to Space Resecton Usng Straght Lnes PE&RS, Vol.6, No.1, pp.57-66 [17] Wen, W.; Yuan, B., 1995. Structure nference and pose estmaton based on symmetry of obect. proceedngs Fronters of computer technology IEEE Regon 10's Annual Int. Conf., Vol., pp.766-770 [18] Wyle, C.R.; Barrett, L.C., 198. Advanced Engneerng Mathematcs. McGraw-Hll, Inc., ffth ed.ton, pp.768-769 [19] Zeng, Z.; Wang, X., 199. A general soluton of a closed-form space resecton. PE&RS, Vol.58, No.3, pp.37-338 [9] Horaud, R.; Cono, B.; Leboulleux, O., 1989. An analytc soluton for the perspectve 4-pont problem. Computer Vson, Graphcs, and Image Processng 47, pp.33-44 [10] Katan, K., 1993. Geometrc Computaton for Machne Vson. Clarendon Press, Oxford, ISBN 0-19-856385-X, pp.63-65 [11] Lee, C-N.; Haralck, R.M., 1996. Statstcal estmaton for exteror orentaton from lne-to-lne correspondences. Image and Vson Computng 14, pp.379-388 [1] Lee, R.; Lu, P-C.; Tsa, W-H., 1990. Robot Locaton Usng Sngle Vews of Rectangular Shapes. PE&RS, Vol.56, No., pp.31-38 [13] McLean, G.F.; Kottur, D., 1995. Vanshng Pont Detecton by Lne Clusterng. IEEE Transactons on