Hee Ju Prk OLOUR IMAGE MATHING FOR DTM GENERATION AND HOUSE EXTRATION Hee Ju PARK, Petr ZINMMERMANN * Swiss Federl Institute of Technology, Zuric Switzerlnd Institute for Geodesy nd Photogrmmetry heeju@ns.shingu-c.c.kr petr@geod.ug.ethz.ch Technicl Session III- KEY WORDS: DTM/DEM/DSM, Imge mtching, Reconstruction, Urn Ojects ABSTRAT Imge mtching plys key role in utomtic DTM genertion nd house extrction. For the house extrction from lrge scle imgery, point mtching nd line mtching complement one nother: Line mtching gives the 3 dimensionl line informtion which supports house reconstruction t ridge lines or roof oundries; dense well distriuted point mtching results contriute to the surfce model genertion of the remining non-rekline regions. We propose new epipolr line eqution, which is determined y orienttion prmeters nd supports oth epipolr line serch nd epipolr imgery genertion. The proposed mtching process is divided into point mtching nd line mtching. To derive highly relile results in point mtching we include lunder suppression sed on the positionl reltionship etween possile corresponding point pirs. Line mtching is supported y the results of point mtching to reduce the numer of possile corresponding line pirs. Regrding similrity comprison for line mtching we use the line shpe, the flnking regions colour, informtion on positionl reltionship nd connectivity etween cndidtes for corresponding lines nd neighouring points nd lines. We tested the proposed method with smple dtset nd show the results. INTRODUTION Imge mtching plys key role in utomtic DTM genertion nd house extrction. Applictions on DTM genertion cn e found in mny current commercil digitl photogrmmetry worksttions, nd methods nd pplictions of house extrction cn e found in [Gruen et l, 997; Henricsson, 996]. For the extrction of houses from lrge scle imgery, oth point mtching nd line mtching re necessry: Line mtching gives the 3 dimensionl line informtion which is useful for oth for rekline detection nd house reconstruction; wheres dense well distriuted point mtching contriutes to the surfce model genertion of the remining non-rekline regions. Mny studies hve een mde relted to this issue, ut still there is generl roust method missing. The im of this study is to find mtching method focussing on this prticulr prolem. In this pper firstly we will descrie the new epipolr eqution with geometricl proof. Secondly we will descrie our study on new mtching method. Finlly the results of our mtching methodwill e descried nd discussed. PROPOSED EPIPOLAR LINE EQUATION Epipolr line geometry is one fundmentl principle in imge mtching domin [Pul R. Wolf, 984]. Within n overlpping imgery pir, the corresponding point of one point lies on the epipolr line which is corresponding to tht point. Here we suggest new epipolr line finding method. The sic principle of the epipolr line derivtion is the condition of coplnrity, the detils re s follows: Let there e couple of cmers which cpture n oject t the sme time. We cll one s left cmer with indices, nother s right cmer with indices. Let O, O e the left nd the right cmer centre. And Let P e point on the oject. By definition O, O, P re on the sme plne clled epipolr plne. Epipolr lines corresponding to P re defined y the intersections of the epipolr plne nd the imge plnes of the left nd right cmer. As the intersection of two plnes is lwys stright line, the resulting epipolr lines re stright lines nd there exist couple Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm. 697
Hee Ju Prk of epipolr line corresponding to P. Let p e point of the left imge corresponding to P. Let k e point on the epipolr line of the left cmer imge, k point on the epipolr line of the right cmer. Then O, O,P, k, k lie on the sme plne, which is clled coplnr condition. Therefore the vectors O O, O p, O k nd O k lie on the sme plne. Let B r e vector of seline, f e the cmer s focl length. Let R nd R e the rottion mtrices of the left cmer nd right cmer. Let the terrestril coordintes of O, O, e (X, Y, Z ), (X, Y,Z ). Let the photo coordintes of p, k, k e, y ) (x,, ( x, y ). (x, y) 3 R = 3, R 3 3 = 3 3 3 3 Then, r B = ( Bx, By, Bz) = ( X X, Y Y, Z Z) () 3 O = ( X, Yp, Z ) = 3( x, y, f ) 3 3 () 3 O k = ( X, Y, Z) = 3( x, y, f ) 3 3 (3) 3 O k = ( X, Y, Z ) = 3( x, y, f ) 3 3 (4) Becuse the vectors B r, O p, O k stisfy the coplnr condition, B r ( O P Ok ) = Therefore, (5) Bx By Bz X Y Z = Bx( YZ Z Y ) + By( Z X X Z ) + Bz( X Y Y X ) = X Y Z (6) From (),(3),(6) { + { + { ( ByZ 3 ( ByZ ( ByZ BzY BzY BzY ) + ) + )) + ( BzY ( BzX 3 ( BzX BxZ p BxZ BxZ ) + ) + 3 ( BxY 3 ) + ( BxY ( BxY ByX ByX )} x ByX )} y )} f = (7) (7) is the epipolr line eqution on the left imge which is corresponding to the point p(x,y) of the left imge. This cn e represented s follows. Ax By + f = + (8) 698 Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm.
Hee Ju Prk A = B 3 3 3 3 T Bz By Bz Bx By Bx 3 3 3 3 xp yp f (9) The epipolr line eqution for the right imge which is corresponding to the point p on left imge is otined s follows: Becuse the vectors B r, p B r ( O P Ok ) = Therefore, O nd O k stisfy the coplnr condition, () Bx By Bz X Y Z = Bx( YZ Z Y ) + By( Z X X Z ) + Bz( X Y Y X ) = X Y Z () So we cn get the following epipolr line eqution for the right imge which corresponds s to p of the left imge. A x + B y + f = A = B 3 3 3 3 T Bz By Bz Bx By Bx 3 3 3 3 xp yp f () (3) When we use pixel coordintes, we hve to know the trnsformtion etween pixel coordintes nd photo coordintes. If we hve pixel coordinte (u, u ) of the left imge, we cn get (x, y ) y trnsformtion etween pixel coordintes nd photo coordintes. The epipolr line eqution (8) lwys psses two points f (, - ), f (-, ), nd lso B A the epipolr line eqution () lwys psses two points f (, - ), f (-, ). We cn get the pixel coordintes of these B A points y trnsformtion etween photo coordintes nd pixel coordintes. Then we cn get the epipolr line eqution represented in pixel coordintes. 3 PROPOSED MATHING METHOD The sic proceeding of the proposed method is composed of two seprte steps: in the first stge we pply point mtching, nd in the next stge we perform line mtching. The positions of mtched points re used s constrint to reduce the numer of possile line mtches. Additionlly we cn get the pproximte differences for ech colour chnnel etween corresponding regions in overlpping imges for the mtched point set. This serves s nother supporting constrint for line mtching. oncerning point mtching the similrity etween corresponding res ner the point is used s locl consistency check, nd the positionl reltion etween possile neighouring mtched points is used s glol consistency check. oncerning line mtching the similrity in shpe, the ngle of the lines, the flnking regions colour serve s locl consistency check, nd the reltion etween possile corresponding lines nd neighouring mtched points serve s glol consistency check. A scheme of the sic flow of the proposed mtching method is shown in Figure. 3. Point Mtching First we generte points for mtching y the Foerstner Interest Opertor [Foerstner et l, 987] for ech imge. As Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm. 699
Hee Ju Prk Figure. Bsic flow of proposed mtching method Figure. Neighouring points in point mtching threshold of Interest Vlue w which is relted with the contrst, we use positive vlue of w lim, for exmple w lim =.. We don t consider the vlue q which is relted with shpe. The reson of this is the line mtching of the next step. In literture mny methods for mtching etween Interest Points cn e found [Foerstner et l, 987;Zhng, 994]. One simple method is the correltion coefficients method. To increse the reliility of mtched points, we perform n dditionl check for the cse when the templte window nd serch window is reversed. If the result for ech point pir mtched is the sme for oth - norml cse nd reversed cse-, it is ccepted. This method is clled ck-mtching [Hnn 989]. To improve the reliility we ccept the cse when oth of the mtched points re Interest Points. Till this stge the mtching is performed with the imges gry vlues ecuse of etter performnce. Through the ove process we get possile mtched points sets. Then we check the correltion coefficients etween ech mtched point pir for ech RGB colour chnnel. If ny of correltion coefficients for ech colour chnnel is less thn.5, tht point pir is rejected. Till this process we check the locl similrity of res ner the points. 3. Blunder suppression to possile mtched point set Bsed only on the locl similrity comprison, voiding lunders is difficult. To solve this prolem we check the glol similrity etween possile corresponding point pir nd its neighouring possile corresponding point pirs. Let two points of possile corresponding pir e point nd point j. We ssume tht there is numer M of neighouring points ner the point nd point j. Suppose point m, nd its possile corresponding point n re ner the point nd point j s shown in Figure. The coordintes of point point point m, point n re ( i x, iy ), ( j x, j y ), ( m, m ),( ) x y n x, ny. We define mesure of Strength of Mtching SM for the pir of point point j s follows : exp( s( dx) / sdv( dx)) + mn ( m, n) + [( d( n)]/ SM = (4) + + [( d( n)]/ ( m, n) : correltion coefficient etween point j nd point j : correltion coefficient etween point m nd point n mn d ( d ( n) dx = ( ix mx ) ( jx nx ) : distnce etween point i nd it s neighouring point m : distnce etween point j nd it s neighouring point n 7 Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm.
Hee Ju Prk This definition hs empiricl chrcter. If the point pir reltion etween other point pirs is similr, SM will e reltively high. If the reltion is wek then SM will e smll compred to other point pirs. Point pirs whose SM is lower thn threshold re eliminted, for exmple the pirs with lowest 5% in SM vlue. 3.3 Line mtching If we cn reduce the numer of cndidte lines to e compred, we reduce the processing time nd increse the possiility of finding corresponding line pirs. In line mtching, firstly we reduce the numer of possile cndidte lines s descried in the following: If the reltion etween ny mtched point nd pir of lines on oth imge is reversed like Figure 5, this cse of line cn e discrded. One line hs two flnking regions see Figure 3. If pir of lines is corresponding, then t lest one flnking region of them is similr in ech colour chnnel. So line pirs without ny similr flnking regions cn e rejected. Becuse we cn know the positions of mny overlpping windows y mtched point sets in previous stge, we know the pproximte difference in ech RGB colour chnnel vlue for whole corresponding regions using the mtched points. We represent the differences s men nd stndrd devition for ech colour chnnel. We ssume tht the following cse rrely hppens for ech colour chnnel. dg E( dg) > 4* sdv( dg) dg: flnk region difference etween line i nd line j E (dg), sdv (dg) : men & stndrd devition of corresponding region difference (5) If oth flnking regions of line pir fulfill the ove cse, then we cn discrd such line pir in the comprison. We consider the line pirs which pss the test s two check points. In the next step we clculte the similrity coefficient for the overlpping prt etween the lines for the locl similrity comprison. There re mny possile definitions for this similrity coefficient. One simple definition is s follows. Suppose the coordintes of point on the line of imge i s ( x, y i i ), nd its corresponding point s ( x, y j j). Then y i = yj without ny rottion except through relief displcement in stereo epipolr imgery. The line similrity coefficient s cn e defined s (6) cov( x x j ) = W W s c n s s c x i x j W : weight for the numer of overlpping flnks. (e.g. W = when one of the line flnks is similr. And W n W =.5 when oth flnking regions re similr for every colour chnnel.) c : weight for the numer of overlpping points etween line i & i.e. the numer which hs sme y coordinte ech other etween line i & j. c This ove definition doesn t consider the effect due to slope in oject spce. This slope my e stright or curve. To simplify comprison, we suppose verge slope in height. Suppose there is ngle q due to verge slope like Figure 6. The igger verge slope in height is, the igger the ngleq is. Figure 3. Flnking region of line Figure 4. Neighouring points nd lines in line mtching Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm. 7
Hee Ju Prk imge imge Figure 5. Suspicious corresponding point nd line reltion Figure 6. Angle due to verge slope in height Insted ofx j we cn use x j which is modified y ngle q. The Angle q my e relted with relief displcement, nd lso my not e relted with relief displcement. So we define s follow s s = s x i cov( x x j) W s exp( q) x j And we reject the line in which the expression c W n (7) cov( x is less thn certin threshold, for exmple <.6., ) s s j exp( q) Furthermore we reject the pirs which hve less overlpping points thn threshold, for exmple <. Then we define the prmeter Strength of mtching SM for line pirs s follows. x i x i x j SM exp( s( dx) / sdv( dx)) shk w( k )) Wp ( + Wn )( s + + + ) ( m, n ) [( d( d( n)]/ h k + [( d( h) k )]/ = + (8) + + [( d ( n)]/ + [( d( h) + d ( k )]/ ( m, n) (m,n): pir of mtched points k: neighouring possile cndidtes for corresponding lines ner line j Wp: weight for neighouring mtched point. For exmple Wp=7 : line shpe similrity coefficient etween line i nd line j s shk : line shpe similrity coefficient etween line h nd line k h k w ( is like co-opertive coefficient in [Gruen et l, 998], which is relted with the proility when possile corresponding pir ( j) nd its neighouring pir ( k) exist t the sme time. In this context we define it s follows. If the reltion etween h nd k is reversed, w ( j; ( = -. And if h is connected nd j is overlpping with oth i nd then w ( = And if k is connected nd i is overlpping with oth k then w ( =. And if h is connected nd k is connected then w ( =. Other cse exp( s ( dx )) w( = + [ d( h) k)] / If we hve set of strength of mtching prmeters { SM } for lines in the left imge, then we cn esily get nother set of strength of mtching prmeters { SM } for lines in the right imge. { SM } nd { SM } re symmetric. When we clculte SM, we only need to find symmetric vlue corresponding to ech SM in the list of { SM }, nd then ssign it. The next step is to find the mtching pir using { SM } nd { SM }. For ech i in SM we determine the pir which hs the highest Strength of Mtching vlue. One line cn hve two or more corresponding lines. So we choose ll possile pirs with highest SMs which re not overlpping ech other. We pply the sme process with { SM }. If 7 Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm.
Hee Ju Prk line pir is corresponding to nother, then it will hve high proility concerning{ SM } nd { SM }. So oth results re consistent, nd we select the pirs nd continue with nother set of pirs. We cll this { P }. Then we discrd P less thn threshold, for exmple the lowest 5%.We receive finl set of mtched line pirs. j j 4 RESULTS OF TEST AND DISUSSION We tested the proposed mtching strtegy with the Avenches dtset descried in [Mson et l, 994]. We checked the results y humn visul inspection with stereoscope. Figure 7 shows one exmple of our tests. We used the SE opertor [Heitger, 995] s edge detection opertor. Our epipolr imgery genertion using the proposed epipolr line eqution ws successful in ll cses. oncerning point mtching there ws no lunder mong 5 mtched point pirs when we set the msk size for the Foerstner Interest Opertor W=5. oncerning line mtching line pirs were mtched nd no lunder ws found. At lest pirs of lines re mtched for ech house. Generlly the ridge line ws mtched. The use of previous mtched point sets supported line mtching useful. Aville mtched points reduced the numer of cndidte line pir in comprison to the cse without previous mtched points. So fr the chnging topology prolem ws not solved though our method lthough there is no lunders in mtched line pirs. For exmple sometimes the roof line nd the ottom line re seen s one line on one imge, ut hs two seprted corresponding lines in n other view. In this cse it cn hppen tht the roof line is mtched to the ottom line or to the roof line. We need more studies to solve this remining prolem successfully. 5 ONLUSIONS In this pper we propose new method which is useful for imge mtching, for DTM genertion nd house extrction. The proposed new epipolr line eqution which is determined y orienttion prmeters is esy to implement nd useful for epipolr line serch nd epipolr imgery genertion. oncerning imge mtching two new ides were proposed. One is lunder suppression to mtched point pirs sed on positionl reltionship etween possile mtched point pirs in point mtching. The other one is the use of mtched point sets s constrints for line mtching in two wys - one to define position constrints nd the other to define colour similrity constrint for overlpping flnking regions. The mtching strtegy of comining line shpe, flnking region similrity, positionl nd connectivity reltionship etween corresponding lines nd corresponding points, corresponding lines nd its neighouring corresponding lines works firly well with our test dtset. But there is still prolem in solving the chnging topology cse. More thorough evlution for vrious imgery nd nd serch for solution for the chnging topology cse will e continued in future studies. REFERENES Foerstner, W., Guelc E., 987. A fst Opertor for Detection nd Precise Loction of Distinct Points, orners of irculr Fetures, Proc. of IFPDD, pp. 8-35. Gruen, A., Wng, X., 998. -Modeler: A topology genertor for 3-D city models, Interntionl Archives of Photogrmmetry nd Remote Sensing, Vol.3, Prt 4, pp.88-96. Gruen, A, Bltsvis E., Henricsson, O., 997, Automtic Extrction of Mn-Mde Ojects from Aeril nd Spce Imges (II), Birkheuser Verlg, Bsel. Hnn M.J, 989. A System for Digitl Stereo Imge Mtching, PE & RS, Vol. 55, No., pp. 765-77. Heitger, F., 995. Feture Detection using Suppression nd Enhncement. Technicl Report TR-63, Imge Science L, ETH-Zuric Switzerlnd. Henricsson O., 996, Anlysis of Imge Structures using olor Attriutes nd Similrity Reltions, PhD Thesis No. 663, Swiss Federl Institute of Technology. Mson, S., Bltsvis M., Stllmn, D., 994. High Precision Photogrmmetric Dt Set for Building Reconstruction nd Terrin Modelling, Internl Report, Institute for Photogrmmetry nd Geodesy, ETH, Zurich. Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm. 73
Hee Ju Prk Wolf P. R., 984. Elements of Photogrmmetry, nd Edition, pp. 385-386. McGrw-Hill Book ompny. Zhng, Z., 994. A Roust Technique for Mtching Two Unclirted Imges Through the Recovery of the Unknown Epipolr Geometry, INRIA Rpport de recherche no. 73. () edge lines nd mtched points set () mtched lines (white line) Figure 7. Test results for Avench dtset 74 Interntionl Archives of Photogrmmetry nd Remote Sensing. Vol. XXXIII, Prt B3. Amsterdm.