STEREO PLANE MATCHING TECHNIQUE

Transcription

1 STEREO PLANE MATCHING TECHNIQUE Commission III KEY WORDS: Sereo Maching, Surface Modeling, Projecive Transformaion, Homography ABSTRACT: This paper presens a new ype of sereo maching algorihm called Sereo Plane Maching Sereo plane maching adops leas square mehod under he consrain ha all poins in an area specified by a polygon should lie on a common plane This echnique uilizes he fac ha correspondence beween sereo images becomes 2-D projecive ransformaion (homography wihin an area where a common plane is projeced Three corresponding poin pairs on sereo image pair are used for parameerizaion of geomery of a plane, which allows compuaion of homography wihin he polygon The mos powerful feaure of his echnique is ha i can impose geomerical consrain in planar direcion or posiion For example, he arge plane can be fixed in horizonal or verical direcion, as well as in oher specified direcion in 3-D space Anoher feaure is ha i can be applied o unrecified sereo pair, so far as is orienaion parameers are known Experimenal resuls show ha his algorihm can measure oblique roof or verical wall of a building INTRODUCTION Exising sereo maching echniques auomaically measure poins or line feaures, bu mos of hem canno se consrain in 3-D shape of arge objecs, which includes one of he mos simple 3-D srucure, Plane One approach of sereo maching wih planar consrain can be realized by image regisraion by homography (Szeliski, 994 This approach uses he fac ha correspondence beween sereo images becomes 2-D projecive ransformaion (homography wihin an area where a common plane is projeced This approach can impose consrain on he direcion of planes parallel o specified plane (Oda eal,997, bu oher ype of consrain, such as parallel o line direcion, or going hrough a specific poin, canno be aained In his paper, a new ype of sereo maching algorihm called Sereo Plane Maching is presened Sereo plane maching opimizes square difference of pixel value beween sereo image pair under consrain ha all pixels in specified area should be on a common plane The mos powerful feaure of his echnique is ha i can impose geomerical consrain in planar direcion or posiion For example, a arge plane can be fixed in horizonal or verical direcion, as well as in oher specified direcion in 3- D space Anoher eminen feaure is ha sereo plane maching can direcly measure unrecified sereo pair under he condiion ha is orienaion parameers are known Sereo plane maching uilizes he fac ha correspondence beween sereo images becomes 2-D projecive ransformaion (homography wihin a region where a common plane is projeced Three corresponding poin pairs, called conrol pairs, parameerize he arge plane geomery I is a ype of homography wih epipolar consrain, and compued by use of hree conrol pairs and epipoles This paper firs defines he arge problem which sereo plane maching echnique handles, and presens he basic sraegy of sereo plane maching where an evaluaion funcion for leas square mehod is formularized The following secions show deails of sereo plane maching, including how o parameerize a plane in sereo plane maching, how o derive homography beween sereo images, how o execue leas square mehod and how o impose consrain on plane posiion and direcion Some experimenal resuls are also presened o show ha his mehod can measure oblique roof and verical wall in 3-D ciy space 2 PROBLEM DEFINITION AND BASIC STRATEGY 2 Problem Definiion Suppose ha objec A consised of planer faces is phoographed in image and, and polygon A in is measured as he conour of face A of objec A as show in Figure The arge problem is how o deermine hree-dimensional coordinaes of polygon A auomaically by sereo maching, under he consrain ha all he poins are co-planar, and he assumpion ha inerior and exerior orienaion parameers of he sereo pair are known Inerior and exerior orienaion parameers can give he following ses of coordinae ransformaion funcions: Transformaion funcions from 3D coordinaes P( XYZ,, o image coordinaes p( xy, which is projecion of P :

2 p F( P ( χ( D e i ( D 2 (5 In he case of sereo images and, we have wo funcions, F ( P for, and F 2 ( P for e i ( D ( Hp ( i, D ( p i (6 2 Transformaion funcions from Image coordinaes p( xy, o 3D coordinaes P( XYZ,,, under he condiion ha one of he coordinaes ( XYZ,, is known: where summaion in equaion (5 is compued for all pixels in polygon A Similar o funcion (, we have wo ses of he funcions, Gx, Gy, Gz for, and Gx 2, Gy 2, Gz 2 for 3 Transformaion funcions from a sereo pair of image coordinaes p ( x, y and p 2 ( x 2, y 2 o 3D coordinaes P( XYZ,, : P P P Gx( p, X Gy( p, Y Gz( p, Z P G 2 ( p, p 2 Polygon A (2 (3 3 DEFINITION OF PARAMETER D Suppose ha Q, Q 2, Q 3 are projeced ono p, p 2, p 3 in and p 2, p 22, p 23 in, respecively Also suppose ha poins p 2 p 2( are on he epipolar line of p, poins p 22 and p 2( 2 are on he epipolar line of p 2, and poins p 23 and p 2( 3 on he epipolar line p 3 Wih a se of parameers D D D D 2 3, p 2, p 2(, p 22, p 2( 2, p 23 and p 2( 3, p 2, p 22 and p 23 can be described by he following equaions: p 2 ( D p 2 + D p 2 p 22 A se of D deermines hree corresponding poin pairs and hree 3-D poins on plane A can be compued by equaion (3 Since hree non-colinear poins deermine a unique plane, D deermines geomery of plane A Here we call pairs of corresponding poins conrol pairs ( D 2 p 22 + D 2 p 22 p 23 ( D 3 p 23 + D 3 p 23 (7 O Face A Epipolar Line of p Plane A p 2(3 p p p 2( 2(3 (3 p p p 2(3 ( 2( p 2( p (2 p 2(2 p 2(2 p 2(2 22 Basic Sraegy Figure The arge problem Assuming ha plane A can be described by a se of parameers D D D D 2 3 The maching funcion H, which gives maching poin p 2i for a given poin p i wihin a polygon A, depends on parameer D : p 2i Hp ( i, D To deerminae D by sereo maching, leas square mehod can be applied o minimize he following evaluaion funcion χ( D : (4 Face A Plane A Q 3 Q Figure 2 Definiion of Planar Parameer D 4 DERIVATION OF FUNCTION H I is well known ha correspondence beween p ( x, y on and p 2 ( x 2, y 2 on, where Poin Q on plane A is projeced, can be described by projecive ransformaion (or 2-D Q 2

3 homography Therefore, he funcion H in equaion (4 is projecive ransformaion Projecive ransformaion has eigh independen parameers Projecive ransformaion beween homogeneous coordinaes ( x, y, and ( x 2, y 2, can be expressed wih he following equaion: Epipoles E 2 and E 2 are inersecion poins beween image planes and he sraigh line O which goes hrough wo opical cener O and as shown in Figure 4 Noe ha epipoles are he projecion of he poin Q E, where O inersecs wih plane A In oher words, hese epipoles are also corresponding poins for plane A x 2 y 2 a a 2 a 3 a 4 a 5 a 6 a 7 a 8 x y (8 Epipoles are known o have he properies below Epipoles are independen from geomery of plane A and deermined only by epipolar geomery Epipoles can be compued by subsiuing opical ceners for P in equaion (: where means equaliy up o scale facor, ha is: E F ( [ XYZ,, ] [ X, Y, Z ] X : Y: Z X : Y : Z (9 E 2 F 2 ( O ( The marix on he righ side of equaion (8 is called homography marix 2 All epipolar lines go hrough epipoles 3 When an image plane is parallel o O, epipolar lines are also parallel and epipole is a infinie disance 4 Deerminaion of H wih D and Epipoles From hree conrol pairs and a pair of epipoles gives eigh linear equaions for eigh unknown parameers of homography marix These equaions can be wrien in marix syle as follows: p p 2 H M B V ( Plane A where B [ a, a 2, a 3, a 4, a 5, a 6, a 7, a 8 ], M is a 8 8 marix, and V is 8 dimensional vecor Q Figure 3 Homography H for Plane A Homogeneous expression of projecive ransformaion allows handling epipoles in infinie disance For example, when all epipolar lines are parallel o x axis boh in and, homogeneous coordinaes of epipoles are [,, ] Subsiuion of his coordinaes o boh side of equaion (8 leads o: Two linear equaions can be derived by subsiuing a pair of corresponding poins beween and ino equaion (8 This means ha 8 parameers of he projecive ransformaion can be deermined wih four pairs of corresponding poins Three of hose can be given by conrol pairs ha are used in he definiion of he parameer D in equaion (7 Nex we show ha epipolar geomery provides he fourh pair of corresponding poins; epipoles a 4 a 7 5 OPTIMIZATION WITH LEAST SQUARE METHOD (2 We have adoped Gauss-Newon mehod for non-linear leas square opimizaion This mehod revises arge parameers D wih D : D + D D (3 D can be compued by he following equaion: Figure 4 Epipoles J D ( J J e (4

4 e e e 2 e n (5 M V and can be compued wih equaions (7 and (8 Thus all iems in equaion (7 can be calculaed J e D e 2 D where weighs of all poins are assumed o be equal and number of samples e n D (6 n is 6 STEREO PLANE MATCHING UNDER VARIOUS CONSTRAINTS One paricular meri of sereo plane maching is ha various consrains can be implemened by fixing conrol pairs 6 Fixaion of One/Two Poins on Plane in 3-D Space This compuaion is repeaed unil he evaluaion funcion converges on he minimum value χ( D Equaions (5 and (6 shows ha compuaion of e i and is parial differeniaion can realize leas square opimizaion The value e i can be easily compued by equaion (6 The following discussion gives how parial differeniaion can be compued Denoing p i ( x i, y i and p 2i ( x i, y i, and differeniaing equaion (6, we have: e i D x ( I j 2 xi + ( I D i j y i 2 x i I xi 2 + yi y i yi (7 Parial differeniaion of in x and y direcion can be numerically compued wih pixel values of From Equaion (8, xi and yi can be described as following: Fixaion of one or wo poins on he plane in 3-D space means fixaion of one or wo of conrol pairs One-poin fixaion can be implemened simply by seing projecion pair of his poin as p 3 and p 23 Fixaion of wo poins can be also implemened by seing projecion pairs of hese poins as p 3 - and - p 23 p 2 p Consrain in he Direcion of Plane There are wo ypes of consrain in he direcion of a plane One is he specificaion of direcion parallel o he plane, and he oher is he specificaion of direcion perpendicular o he plane 62 Consrain in direcion parallel o he arge plane: In his case, vanishing poins in he specified direcion can be used as a fixed conrol pair as show in Figure 5 This ype of consrain decrease he degree of freedom of D o wo For example, verical walls of buildings in an aerial phoo have a common vanishing poin: he verical poin The Verical poins in boh images can be he conrol pair ha is common in all verical planes Vanishing poins in direcion v can be calculaed by he following equaion: Ev F ( O + v xi x G i y G i G x x G i i x i y G i G a 7 x i + a 8 y i + yi x G i y G i G y x G i i y y G i i (8 Ev 2 F 2 ( + v (2 Vanishing Poin Vanishing Poin Ev Ev 2 Differeniaion of boh side of equaion ( by D j gives: Plane v v Plane I M B + M Dj V (9 Figure 5 Maching beween Vanishing Poins in Direcion v This leads: V M M B (2 622 Consrain in direcion perpendicular o he arge plane: in his case, wo conrol pairs can be fixed by wo pairs of vanishing poins in direcions ha are perpendicular o he speci-

5 fied direcion Thus degree of freedom of parameer D comes down o one However, his implemenaion does no work when sraigh line O is parallel o he arge plane, since an epipole and all vanishing poins align in a common line and M in he equaion ( becomes a singular marix Afer maching operaion, all poins on polygon in he righ image came o he exac maching posiion on he wall as shown in Figure 6 (2 There is a more robus way o calculae homography beween sereo images by he following heorem: Theorem : Suppose ha plane Π and Π are parallel and H Π and H Π are heir homography marices beween sereo images Then, homography marix for any plane parallel o Π and can be described in following form: Π H( D ( D HΠ + D H (22 Π (The polygon which raced he oblique roof in he lef image Appendix A gives he deailed proof of his heorem The equaion (22 can compue homography marix for any planes parallel o Π and Π The opimum parameer D is he value ha minimizes he evaluaion funcion χ( D and can be compued simply by -D search of D (2Resul of -D search based on heorem 7 EXPERIMENTS AND RESULTS Experimens has been performed wih a pair of sereo images wih a scale of /4, which cover he former building of Insiue of Indusrial Science (IIS, Tokyo Universiy Sereo plane maching has been implemened on sofcopy ploer, Geo-Ploer, which has been developed by Asia Air Survey Co, Ld (Sakamoo e al 2 (3Resul of sereo plane maching wihou consrain Figure 7 Tes wih an Oblique Plane 72 A Tes wih an Oblique roof (The polygon which raced he wall in he lef image To demonsrae measuremen abiliy for general oblique plane, a par of roof of he old IIS building was raced as a polygon Figure 7 ( shows he iniial sae jus afer racing he roof in he lef image Afer ha, sereo maching based on heorem searched he horizonal plane which gave he bes mach for he polygon as shown in Figure 7 (2 This gave good iniial esimaion of parameer D Figure 7 (3 shows ha sereo plane maching wihou consrain polygon compued beer resul han simple -D search (2Resul of sereo plane maching Figure 6 Tes wih a Verical Wall 7 A Tes wih a Verical Wall To demonsrae he effec of consrain in direcion of he plane, a wall of he old IIS building was ploed as polygon, whose heigh was iniially fixed o a cerain heigh Figure 6( shows he iniial sae jus afer an operaor raced he wall in he lef image, where he polygon in righ image was no fi on he wall 8 CONCLUSION We have proposed a sereo plane maching echnique o impose consrain on auomaic sereo maching process This echnique is considered as leas square maching under he consrain ha all poins in a specified area are fi on a common plane Parameerizaion wih conrol pairs including epipoles enables implemenaion of geomerical consrain in planar direcion or posiion This echnique can measure planar surface of manmade srucure, which includes verical walls or oblique roofs We plan o uilize his echnology for developing user inerface for sofcopy mapping sysem, where an operaor can measure planar surface easily wihou adjusing heigh

6 REFERENCE Oda, K, Kano, H, and Kanade, T, 997 Generalized dispariy and is applicaions for muli-sereo camera calibraion Opical 3-D measuremen Techniques IV, pp9-6 Sakamoo, M, Uchida, O, Doihara, T, Oda, K, Lu, W, Obaa, M, 2 Geo-ploer - a sofcopy mapping sysem for low cos digial mapping process IAPRS, Amserdam, The Neherlands, XXXII, Par B4, pp Szeliski, R, 994 Image Mosaicing for Tele-Realiy Applicaions, DEC CRL 94/, April APPENDIX A: PROOF OF THEOREM H Π α M R T n Π M z Π H Π α M R T n Π M z Π (25 If n Π n Π n (ie, wo planes are parallel and γ α α, we can define anoher homography marix H Πβ by a linear combinaion of H Π and H Π wih parameer β : H Πβ ( βh Π + β γ H Π α M R βz Π + ( βz Π T + n M z Π z Π (26 Π P Now anoher homography marix HD wih parameer D : HD ( D HΠ + D H (27 Π z Π Equaion (27 can be deformed ino: n C m m T Figure 8 Homography for Plane Π C HD α( ( βh Π + β γ H Π α (( Dγ+ D γ β D (( Dγ+ D (28 Homography marix for plane Π is deermined by geomeric relaionship beween he sereo cameras ( C and C and he plane Π : Comparison beween equaion (28 and equaion (26 reveals ha HD is a homography marix for a plane parallel o Π H Π α M R -----T + n Π M z Π (23 where M and M are marices of inerior orienaion parameers of C and C, R is he roaion marix beween C and C, T is he ranslaion vecor from C o C in C s coordinae sysem, n Π is a normal vecor o he plane Π in C s coordinae sysem, z Π is he deph from he camera C o plane Π in he direcion of n Π, and α is a scale facor which adjus he lower righ elemen of he marix o The marix of inerior orienaion M is a 3 x 3 marix: M F s x u F s v y (24 where F is focal lengh, s x is he size of pixel in x direcion, s y is he size of pixel in y direcion, and ( uv, is coordinaes of he principal poin H Π H Π Assume ha wo homography marices and for wo planes Π and Π are given: