Rectangle Region Based Stereo Matching for Building Reconstruction

Transcription

1 1 Rectangle Regon Based Stereo Matchng for Buldng Reconstructon Jng Wang, Toru Myazak, Hrokazu Kozum, Makoto Iata, Jongha Chong, Hroyuk Yagyu, Hdeo Shmazu, Takesh Ikenaga, Member, IEEE, Satosh Goto, Fello, IEEE Abstract Feature based stereo matchng s an effectve ay to perform D buldng reconstructon Hoever, n urban scene, the cluttered background and varous buldng structures may nterfere th the performance of buldng reconstructon In ths paper, e propose a novel method to robustly reconstruct buldngs on the bass of rectangle regons Frstly, e propose a mult-scale lnear feature detector to obtan the salent lne segments on the object contours Secondly, canddate rectangle regons are extracted from the salent lne segments based on ther local nformaton Thrdly, stereo matchng s performed th the lst of matchng lne segments, hch are boundary edges of the correspondng rectangles from the left and rght mage Expermental results demonstrate that the proposed method can acheve better accuracy on the reconstructed result than pxel-level stereo matchng Index Terms Feature Based Stereo Matchng, Buldng Reconstructon, Rectangle Regons P I INTRODUCTION erformance of stereo matchng based D buldng reconstructon n the aeral mage s closely related to matchng features There are manly three knds of stereo matchng methods: ntensty based matchng, area based matchng and feature based matchng [1], [] Intensty based matchng methods drectly study the ntensty profles of to mages to fnd the correspondence [] Area based matchng methods explore the correspondence beteen the mage regons based on some smlarty measures [], [5] Feature based matchng methods ntroduce many hgher level features n the matchng process such as edge element [6] [8], lne segment [9], curve segment [10], corner [11], [1], crcle and ellpse, texture regon, and so on To mprove performance, Manuscrpt receved October, 007 Ths ork as supported by the fund from CREST and JST Jng Wang, Takesh Ikenaga, Satosh Goto are th the Graduate School of Informaton, Producton and Systems, Waseda Unversty, Ktakyushu, , Japan (phone: ; fax: ; e-mal: anjng@rurasedajp, kenaga@asedajp, goto@asedajp ) Toru Myazak, Hrokazu Kozum, Makoto Iata, Hroyuk Yagyu, Hdeo Shmazu are th the System Technology Laboratory, NEC System Technologes Ltd, Ikoma, , Japan (e-mal: myazak-txb@necstneccojp,kozum-hxa@necstneccojp,ata-mxb@nec stneccojp, yagyu@labnecstneccojp, shmazu-hxa@necstneccojp) Jongha Chong s th the Graduate School of Informaton and Communcatons, Hanyang Unversty, Korea (e-mal: jchong@hanyangackr) some researchers also proposed to combne a collecton of feature types For example, Lm and Bnford [1] used a herarchy of features varyng from edges, curves, to surfaces and bodes for hgh-level attrbute matchng We and Hrznger [], [1] proposed to nclude ntensty gradents together th ntensty to perform stereo matchng Although there has been much research on feature based stereo matchng [6] [1], there s not a systematc method to perform D buldng reconstructon n urban scene due to the complexty of the cluttered background and the varous buldng structures Shortcomngs of many current methods nclude ther nablty to exclude features generatng from nose and texture, and shortage n losng features due to shado or occluson In order to address ths challenge, e propose a robust stereo matchng method called Sem-Automatc Rectangle based Dynamc Programmng Stereo Matchng (SAR-DPSM) Frstly, non-contour features are excluded to produce the salent lne segments Secondly, snce the rectangle s the basc polygonal shape for the buldngs, rectangle regons are respectvely extracted n to mages based on the salent lne segments Thrdly, the canddate rectangle regons from the left and rght mage are matched th each other by human observers After that, mproved Dynamc Programmng Stereo Matchng (DPSM) s performed based on these matched lne segments The ntroducton of the rectangle regons can reduce the nose level n reconstructon result On the hole, the proposed method combnes the features of rectangle regon, contour of the rectangles and ntensty together to fulfl the matchng process The rest of the paper s organzed as follos Secton descrbes the proposed rectangle regon based stereo matchng method n detal Expermental results are explaned n Secton Summary of the paper s gven n Secton II RECTANGLE REGION BASED STEREO MATCHING The proposed SAR-DPSM method s manly based on the follong three facts: The boundary of most buldngs n the mage s n polygonal shape or n curved contour hch can be decomposed nto rectangles The area and the heght to dth rato of the rectangles regons correspondng to buldngs are n certan range n

2 the gven aeral mage The rectangle regons correspondng to the buldngs are ntensty homogeneous nsde Based on the above facts, e dvde the rectangle based stereo matchng process nto three stages Frstly, e utlze a mult-scale lne detector to extract salent lne segments on the object contour and suppress lne segments comng from the nose or the texture Secondly, the canddate rectangles are detected based on the local nformaton of salent lne segments and then are fltered accordng to ther heght to dth rato and the proposed salency measure Thrdly, the canddate rectangles from to mages are matched th each other by human observers and DPSM s then mplemented on the bass of matchng lne segment lst Our ork s superor to past ork snce n the proposed method, most of the non-contour lne segments are suppressed before regon analyss and reconstructon Ths can not only mprove the accuracy and effcency of canddate rectangle extracton, but also smplfy the follong regon based stereo matchng In the follong, e explan and state the detals of the proposed method A Mult-scale Salent Lne Segment Detecton In ths stage, e ntend to extract salent lne segments belongng to certan object contours and prepare data for the later canddate rectangle extracton We make use of the scale nvarance of the contour segment to dstngush them from texture edge responses or nose edge responses The approach proposed n the paper [15] s mplemented under a mult-scale frameork Note that our approach s dfferent from them n the follong aspect Pecese lnear segment of object contours are the basc processng elements rather than spatal pror Because e beleve that lne segment s much smpler to descrbe than contour pror but stll keeps the shape nformaton of the contours The mult-scale lne detector s mplemented n three steps Frstly, an mage pyramd s constructed by repeatedly smoothng the mage th a Gaussan flter and then sub-samplng the mage by smply averagng over a k*k pxel regon [16] Ths method s not only rather effcent but also does not lead to any nformaton loss [17] Secondly, a revsed verson of Nevata-Babu edge operator [18] s mplemented at every mage scale to detect lne segments Thrdly, the extracted lne segments at each scale are combned to detect salent lne segments under scale nvarance prncples B Canddate Rectangle Detecton In ths secton, e ould lke to explan ho to detect canddate rectangles on the bass of the salent lne segments Rectangle Detecton from Salent Lne Segments There has been much research n the problem of rectangle detecton For example, Krshnamachar et al [19] proposed a MRF based method hch can estmate the probablty of every boundary lne segment belongng to a contour and then generate potental rectangles n the mage Although the applcaton of the rectangle detecton n [19] s also rooftop detecton, the method cannot be ell appled to our problem because of the follong reasons The MRF based rectangle detecton method manly makes use of the Gestalt prncples lke the proxmty, the parallel property and the perpendcular property among the boundary lne segments Any lne segment that satsfes the above property th ts neghbor lne s kept as a canddate boundary lne of certan rectangle; otherse t s removed Ths dea orks effectvely hen rectangle regons are sparsely dstrbuted But n urban scene, the houses are very close to each other, thus a sngle lne segment may have many neghborng lnes satsfyng the above prncples In ths case, boundary lnes belongng to every potental rectangle cannot be effectvely segregated And then, all the other methods on probablty analyss of lne segments n local regon are not sutable for our current applcaton To address the challenge, e propose the follong rectangle detecton method based on both the Gestalt prncples and the ntensty nformaton Then a salency measure s desgned to evaluate the exstng probablty of every canddate rectangle In the frst step, e produce a perpendcular neghbor lne set N( l 1 ) for every lne segment l 1 When a lne segment l satsfes the follong prncples, t belongs to N ( l 1 ) Frstly, the smallest dstance beteen the endponts of l 1 s under certan threshold Secondly, the angle dfference beteen l 1 should be close enough to 90 degree The above to prncples assure the proxmty and the perpendcular property No e have a lot of L shaped corners of the potental rectangles In the second step, the _ shaped structures are generated from to L shaped corners Assume the endponts of l 1 are ( A, B 1 1), for every to lne segments l n N ( l 1 ), Frstly, l s closer to A 1 s closer to B 1, e, l are respectvely closer to each endpont of l 1 Secondly, a convexty based measure developed from l 1, l s under certan threshold The breaks among l 1, l are measured by a convexty measure descrbed n (1), here l j s the length of the break beteen l and l j, and L s the length of l Then the convexty measure should satsfy () to assure the proxmty among l 1, l

3 l1 + l1 LGR = (1) l1 + l1 + L1 + L + L LGR < threshold lg r () In the thrd step, complete rectangles th four boundary lnes are produced from to _ shaped structure When l are both perpendcular to l 1 to form a _ shaped structure, e search for another lne segment l that can form another _ shaped structure th both l Then l 1 are parallel th each other To declare an effectve rectangle detecton from l 1, l, l, the perpendcular dstance beteen l 1 should satsfy () to assure a compact rectangle λl < d < ( 1+ λ) L, L = max( L, L ),0 < λ < 1 () Here d s the perpendcular dstance beteen l 1 λ s an adjustng parameter λ s emprcally set as 0 n our experments Havng detected the canddate rectangles, e further select the ones hch are most probably correspondng to the buldngs based on a salency measure Three measures such as convexty measure, regon homogenety and boundary gradent are ncorporated nto the salency measure (a) Fg 1 An example to llustrate the salency measure (a) Input mage (b) The most salent 100 rectangles accordng to the proposed salency Frstly, a convexty measure C s defned n (), smlar to the LGR n (1) The smaller the convexty measure s, the better the proxmty among the four lne segments Secondly, the regon homogenety measure H s defned as the varance of the ntensty of the pxels nsde the rectangle regon Smaller H mples smoother regon Thrdly, the boundary gradent measure G s defned as the average gradent of all the pxels on the boundary lne segments When a canddate rectangle has hgher G, t means that the locaton of ts boundary lnes s rather precse All the above three measures are normalzed to the range of [0, 1] and then combned nto the salency measure S defned n (5) Then hgher salency measure M symbolzes more salent canddate rectangle l1 + l1 + l + l C = () l1 + l1 + l + l + L1 + L + L + L S = 1 ( 1 C) + (1 H ) + G, = 1 (5) Besdes the above salency measure, e also set a threshold on the heght to dth rato to exclude rectangles n elongated (b) shape Wth the salency measure and the heght to dth rato, e perform the experment on a test mage In Fg 1(b), the most 100 salent rectangles satsfyng the gven heght to dth rato threshold are shon From the result, e can see that some detected canddate rectangles are not strctly regular rectangles At the same tme, the detected rectangles are overlapped th each other on a buldng To smplfy the processng of rectangle based stereo matchng, e have to take some post-processng on the detected rectangles Post-processng on Detected Rectangles The post-processng on detected rectangles ncludes to parts Frstly, all the detected rectangles are adjusted to strct rectangle shapes Secondly, among the rectangles overlappng th each other on the same spot of a buldng, only the most salent one s kept Fg A fgure to llustrate the shape adjustng process Some canddate rectangles are not n strct rectangle shape because the acceptance range on the formng angle of to perpendcular lne segments s set as from π / - π / 1 to π / + π /1 In order to adjust them nto strct rectangle shapes, the follong steps are mplemented We nvestgate the four corners of a rectangle and pck up the angle most dfferent from π / Then on the most based corner, a ne lne s generated hch can form better perpendcular pars th ts to neghborng lne segments We llustrate ths process n ' Fg l s the nely generated lne segment to replace l TABLE I NUMBER OF CANDIDATE RECTANGLES BEFORE AND AFTER OVERLAPPING RECTANGLE PROCESSING Image Index Before Processng After Processng Reduced Percentage % % % % In urban area, the buldngs are alays very close to each other Accordng to the constructon rule, the neghborng buldngs usually have parallel boundary edges The above phenomena lead to the overlappng detecton of many rectangles on a spot To reduce the redundancy n rectangle detecton, e decde to keep a sngle rectangle on each buldng regon In order to smplfy the computng complexty, e drectly compare the four corners of every to canddate rectangles If all of the four corners of a rectangle are respectvely close enough to the four corners of another

4 rectangle, the less salent rectangle s removed Ths processng s mplemented not only before the shape adjustng of canddate rectangles but also after t Ths processng s rather meanngful for the follong rectangle based stereo matchng snce t saves a lot of redundant computng on the overlappng rectangles To llustrate ths, the numbers of canddate rectangles th and thout ths processng are compared n Table 1 From Table 1, t s clear that the overlappng rectangle processng can greatly reduce the redundancy n rectangle detecton for all the expermental mages C Detected Rectangle Based Stereo Matchng In ths secton, the detals of the rectangle regon based stereo matchng are explaned A sem-automatc method s desgned Frstly, the correspondence beteen the rectangles n the left and the rght mage s marked by the observers Then a refned DPSM process s carred out based on the correspondng lne segments from to corresponded rectangles Throughout rectangle detecton, except geometrcal and ntensty analyss, almost no other a-pror knoledge of buldngs s ntroduced, hch leads to the follong phenomenon Some canddate rectangles are parts of certan buldng regons, and others may be the combnaton of several buldngs close to each other On the spot of a buldng, there may be several canddate rectangles partly overlapped th each other Among them, the observers select only one canddate rectangle that best matches a buldng regon on the boundary Then t s easy to determne the correspondng rectangle n another mage by spatal proxmty In Fg, e llustrate the process of rectangle correspondence Assume that there are N rectangles respectvely n the left L R and rght mage Among them, C and C are respectvely the th rectangle n the left and rght mage Then e can fnd out the correspondence beteen the four boundary lnes of C L R and C In ths ay, e can buld a lst of matchng lne segments Let us assume that any to correspondng lne segments are l l r The left and rght mages are taken under the constrants that the cameras are parallel, hch assures that the correspondng pxels n the to mages le on the same horzontal scan-lne In pxel-level DPSM (PL-DPSM) [], to horzontal lnes th the same y coordnate are frstly ntroduced n to mages and then correspondng pxel pars are searched on the to horzontal lnes In our rectangle based stereo matchng, snce e have already obtaned the correspondence beteen l l r, for any possble y coordnate on both lnes, a par of correspondng pxels (a) (b) Fg Rectangle correspondence n SAR-DPSM (a) Detected canddate rectangles (b) Selected rectangles for correspondence Fg Manual rectangle detecton and correspondence n MR-DPSM on to example mage pars

5 5 P l ( x l and P r ( x r can be frstly obtaned Then stereo matchng can be carred out n to ranges ndependently One range s from the most left pont ( 0, y ) to the correspondng pont, and another s from the correspondng pont to the most rght pont ( 1 Here defnes the dth of the nput mages In ths case, the accuracy of the reconstructed D data s mproved, especally on the buldng boundares Ths can be proved by the expermental data n Secton In DPSM, the normalzed correlaton beteen the pxels from to mages s utlzed to measure the probablty of ther correspondence ( ) ( ) (6) L( xl +, y + j) L R( xr +, y + j) R, j = corr (, ) = ( ) ( ) y xl xr L( xl +, y + j) L R( xr +, y + j) R, j =, j = Here corr y( xl, xr ) defnes the normalzed correlaton beteen one pxel ( x l from left mage and another pxel ( x r from rght mage Durng the computng, the pxels n the neghbour ndo of ( x l and ( x r are also taken nto consderaton L( x, y) and R ( x, y) defne the ntensty of the pxel ( x, y) respectvely n the left and rght mage And L and R are the average ntensty n the respectve neghbour ndos 1 L = (, ) L xl + y + j (7), j = 1 R = (, ) R xr + y + j (8), j = Based on the gven camera parameters and the correspondence beteen pxels, e can obtan the parallax nformaton and then acheve the depth on every pxel III EXPERIMENTAL ANALYSIS To assess the effectveness, the proposed SAR-DPSM method has been appled to 55 pars of aeral mages Each par ncludes a left mage and a rght mage, both n the sze of 60*80 They are mage blocks from a par of large aeral mages Besdes our proposed algorthm, to other approaches are also used n the comparatve experments The frst one s the pxel-level DPSM (PL-DPSM) The second one s the manual rectangle based DPSM (MR-DPSM) method In MR-DPSM method, the rectangle regons are manually marked by the observers In comparson, the SAR-DPSM method can automatcally detect the rectangle regons In Fg, the process of manual rectangle detecton and manual rectangle correspondence n MR-DPSM method s llustrated on to example mage pars In our mplementaton, the reconstructon process after determnng the pxel correspondence s all the same for the above three methods The frst experment ams to compare the accuracy of the reconstructed data by dfferent methods Because of the error n stereo matchng, there s usually heavy nose on object boundary Here e desgn to measures to evaluate the nose level n the reconstructed data Accordng to the stereo matchng process descrbed n Secton, to potental matchng pxels n the left and rght mage alays stay on the same y coordnate Then the nose n the reconstructed data mostly exhbts as the dscontnuty of the depth n horzontal drecton The coarse boundary of the objects reflects ths phenomenon To measure such nose, e mplement Harr-Wavelet transform on the reconstructed data The scalng functon Φ(t) and avelet functon Ψ(t) used n our experment are shon as follos 1, 0 t < 1 (9) Φ( t) = 0, otherse 1, 0 t < 1 (10) Ψ( t / ) = 1, 1 t < 0, otherse The avelet coeffcent n the vertcal component s then normalzed to [0,55] and assgned to each pxel In ths ay, e obtan a nose mage that ncludes the hgh frequency element n the horzontal drecton of the reconstructed data The nose mage s frstly thresholded by 1 to a bnary mage, here the background s set to 0 and the nose s set to 55 The dscontnuty of the depth n horzontal drecton s no reflected by the horzontal bar th several consecutve nose pxels The length of such horzontal bar, e, the number of consecutve nose pxels, can be taken as a measure of the nose level Globally, e compute the average M and the standard devaton D of the length of all the horzontal bars n an mage to evaluate the nose level Smaller M and D means loer nose level and hence better accuracy of the reconstructed data TABLE II PERCENTAGE OF THE ENTIRE DATA WITH BETTER PERFORMANCE COMPARED WITH PL-DPSM PM PM PD PD MR-DPSM 8180% 8909% 9818% 966% SAR-DPSM 77% 7091% 855% 86% TABLE III REDUCED PERCENTAGE OF NOISE LEVEL MEASURES COMPARED WITH PL-DPSM RM RM RD RD MR-DPSM 79% 90% 1660% 175% SAR-DPSM 5% 1% 800% 791% At frst, the to rectangle-based methods, MR-DPSM and SAR-DPSM, are compared to the pxel-level based method PL-DPSM Snce the canddate rectangles provde more nformaton on buldng regon, the to rectangle-based methods, proposed SAR-DPSM method and MR-DPSM method can acheve better performance than PL-DPSM n most of the test mages Table shos that n hat percentage of the entre test mage pars, the to rectangle based methods

6 6 can acheve better performance than PL-DPSM under dfferent nose level measures P M, PM, PD, PD are respectvely the percentage of the entre data hen MR-DPSM or SAR-DPSM can acheve better performance than PL-DPSM on the measure of M, M, D, D Among them, M, M are respectvely the average of horzontal bar length n the left and rght mage D, D are respectvely the standard devaton of horzontal bar length n the left and rght mage In Table, e can fnd out that proposed SAR-DPSM method can acheve better accuracy n at least 70% of the entre test data, hle MR-DPSM method can acheve n at least 81% of all the data We further provde the reduced percentage on the value of dfferent nose level measures for the to rectangle-based methods compared th PL-DPSM In Table, RM, RM, RD, RD defne the reduced percentage on the value of each measure lke M, M, D, D compared th PL-DPSM method On the average, MR-DPSM performs better than SAR-DPSM From Table and Table, t s also clear that for both of the rectangle based methods, mprovement on D, D s much more evdent than on M, It s because the nformaton of M rectangle helps to mprove the accuracy of reconstructon on buldng boundares, the dstrbuton of the nose level value becomes less scattered In order to sho the performance of the three methods n dfferent cases, e also compare the value of M and D on the left mage of 10 example mage pars by three methods n Fg 5 and Fg 6 These left mages are shon n Fg 7 The block ndces, from one to ten, stand for the mage blocks _6, 5_18, 6_6, 6_16, 7_, 8_15, 8_8, 9_17, 10_5, _7 n order Snce every mage s a block cut from a large aeral mage, the block ndex stands for the locaton of the mage block That s to say, _6 block s horzontally the th and vertcally the 6th block n the hole mage Fg 5 Comparson on Devaton of Horzontal Bar Length Fg 6 Comparson on M by three methods Block Index D by three methods PL-DPSM SAR-DPSM MR-DPSM We can see that MR-DPSM performs better than SAR-DPSM n most cases snce there s more human nstructon n MR-DPSM There are to cases hen MR-DPSM and SAR-DPSM do not perform better than PL-DPSM When the rectangle detecton module fals to mark any rectangle regon n a test mage, the performance of PL-DPSM and SAR-DPSM are the same, lke n _6 mage block In the mage block of 5_18, 6_6, 7_, 9_17, _7, MR-DPSM or SAR-DPSM performs orse than PL-DPSM After observng these mages, e fnd out that n 5_18, 6_6, 9_17, there are many small houses In 7_ and _7, there are buldngs n rregular shape These condtons lead to Fg 7 The left mages of the 10 example mage pars, from left to rght and then from the above to belo, they are _6, 5_18, 6_6, 6_16, 7_, 8_15, 8_8, 9_17, 10_5, _7 n order

7 7 rong detecton of rectangles and thus less accurate reconstructed data In Fg 8 and Fg 9, the reconstructed data of the three methods n the mage block _6 and 6_16 s shon TABLE IV MANUAL WORKING TIME IN MR-DPSM AND SAR-DPSM Block Index Tme (Seconds) Reduced Percentage MR-DPSM SAR-DPSM _ % 5_ % 6_ % 6_ % 7_ % 8_ % 8_8 16 6% 9_ % 10_ % _ % Average % TABLE V NUMBER OF CANDIDATE RECTANGLES IN MR-DPSM AND SAR-DPSM Block Index Number of Rectangles MR-DPSM SAR-DPSM _ _ _ _16 1 7_ 10 8_ _8 1 9_ _5 5 _7 6 Accordng to the above analyss, MR-DPSM can acheve better performance than the proposed SAR-DPSM n most cases On the other hand, there s more ork for the observers n MR-DPSM To llustrate the advantage of the proposed SAR-DPSM method n effcency, the second experment s carred out on comparng the manual orkng tme n SAR-DPSM and MR-DPSM For SAR-DPSM method, canddate rectangles are automatcally detected but manually corresponded But for MR-DPSM method, the above to processes have to been fulflled all by the observers For 5 pars out of totally 55 pars of test mages, t takes less tme for the observers hen perform SAR-DPSM method Hence SAR-DPSM method needs less manual ork than MR-DPSM n 955% of all the mage pars The average orkng tme for all the 55 pars n SAR-DPSM method s 196 seconds hle that for MR-DPSM method s 776 seconds The average reduced percentage on the orkng tme for every mage par s 65% That s to say, e can spend only about half of the orkng tme n MR-DPSM method but stll acheve graceful performance by the proposed SAR-DPSM method In Table, the orkng tme of the to methods n 10 example mage pars s compared At the same tme, the number of detected canddate rectangles respectvely n MR-DPSM and SAR-DPSM are also shon n Table 5 The canddate rectangles n MR-DPSM are greatly reduced n SAR-DPSM and then the computng burden can be eased But the performance does not deterorate too much accordng to the above analyss on nose level Block Index TABLE VI TIME COST OF THREE PROCESSES Rectangle Detecton Tme (Seconds) DPSM th Rectangles Fnally, e ll dscuss the computatonal complexty of the proposed SAR-DPSM method All the experments are carred out under Mcrosoft Wndos Server 00 SP1 on a NEC Express 5800 PC equpped th Intel Pentum IV 0GHz processor and 5GB RAM Except the manual rectangle correspondng process, the proposed SAR-DPSM method nclude to parts: rectangle detecton and DPSM th rectangles Table 6 shos the tme cost of the above to processes on 5 example mage pars In comparson, the tme cost of DPSM thout rectangle module n PL-DPSM s also shon Averagely, on the test mage th sze of 60*80, t takes about 106 seconds n rectangle detecton DPSM th rectangles n SAR-DPSM spends about 155 seconds hle DPSM thout rectangles n PL-DPSM spends about seconds DPSM th rectangles has smaller tme complexty because th the lst of matchng lne segments, the searchng range of matchng pont for every pxel s reduced IV CONCLUSION DPSM thout Rectangles _ _ _ _ _ Average The hgh complexty of the urban scene, such as the croded houses, the cluttered background and the varous buldng structures, affects the performance of the tradtonal stereo matchng methods especally on the boundares of buldng regons In ths paper, e propose a novel rectangle-based stereo matchng method, SAR-DPSM to address the above challenge The canddate rectangles potentally correspondng to every buldng regon are detected n both left and rght mages Then after manual correspondence, the stereo matchng s carred out to obtan the D data th loer nose level Comparatve experments sho that the proposed SAR-DPSM can perform better than PL-DPSM n most cases On the other hand, compared th MR-DPSM, SAR-DPSM can save a lot of human orkng tme th graceful performance A draback of the proposed SAR-DPSM method s that t s sem-automatc Then, ho to automatcally buld

8 8 correspondence beteen the rectangles from left and rght mage ll be an mportant ork n the future Moreover, the accuracy of the canddate rectangle detecton greatly nterferes th the afterards stereo matchng Ho to obtan the canddate rectangles that are better matchng th the boundary of buldng regons ll be another focus of our future ork Fg 8 The left and rght mage of mage block _6, and ther reconstructed data by applyng (the nd column) PL-DPSM, (the rd column) MR-DPSM and (rght column) SAR-DPSM The frst ro s the left mage and the reconstructed data on the left mage The second ro s the rght mage and the reconstructed data on the rght mage Fg 9 The left and rght mage of mage block 6_16, and ther reconstructed data by applyng (the nd column) PL-DPSM, (the rd column) MR-DPSM and (rght column) SAR-DPSM The frst ro s the left mage and the reconstructed data on the left mage The second ro s the rght mage and the reconstructed data on the rght mage REFERENCES [1] RA Lane and N A Thacker, Tutoral: overve of stereo matchng research, Tna Memo, vol 1, pp 1-10, 199 [] G We and G Hrznger, Intensty and feature based stereo matchng by dsparty parameterzaton, n Proc ICCV 98, pp , Jan 1998 [] T Poggo, V Torre and C Koch, Computatonal vson and regularzaton, Nature, vol 17, no 6, pp 1-19, 1985 [] M Okutom and T Kanade, A locally adaptve ndo for sgnal matchng, Intl J Computer Vson, vol 7, no, pp 1-16, 199 [5] L D Stefano, M Marchonn and S Mattoca, A fast area-based stereo matchng algorthm, Image and Vson Computng, vol, no 1, pp , 00 [6] PF McLauchlan, Recovery of textured surfaces usng stereo vson, PhD Thess, AIVRU, Unversty of Sheffeld, 1990 [7] S Hongo, N Sonehara and I Yorozaa, Edge-based bnocular stereopss algorthm: a matchng mechansm th probablstc feedback, Neural Netorks, vol 9, no, pp 79-95, 1996 [8] Y Ohta and T Kanade, Stereo by ntra- and nter-scalne search usng dynamc programmng, IEEE Trans on PAMI, vol 7, no, 1985 [9] SH Lee and JJ Leou, A dynamc-programmng approach to lne segment matchng n stereo vson, Pattern Recognton, vol 7, no 8, pp , 199 [10] N M Nasrabad, A stereo vson technque usng curve-segments and relaxaton matchng, IEEE Trans on PAMI, vol 1, no 5, pp , 199 [11] VV Vnod and S Ghose, Pont matchng usng asymmetrc neural netorks, Pattern Recognton, vol 6, no 8, pp , 199 [1] JPP Starnk and E Backer, Fndng pont correspondences usng smulated annealng, Pattern Recognton, vol 8, no, pp 1-0, 1995

9 9 [1] H S Lm and T O Bnford, Stereo correspondence: A herarchcal approach, n Proc Image Understandng Workshop, 1987 [1] GQ We, W Brauer and G Hrznger, Intensty-based and gradent based stereo matchng usng herarchcal Gaussan Bass Functons, IEEE Trans on PAMI, vol 0, no 11, pp , 1998 [15] F J Estrada and J H Elder, Mult-scale Contour Extracton Based on Natural Image Statstcs, n Proc CVPRW 06, pp 18-18, Jun 006 [16] P J Burt, Fast Flter Transforms for Image Processng, Computer Graphcs and Image Processng, vol 16, pp 0-51, 1981 [17] SM Konsh, AL Yulle, and JM Coughlan, A Statstcal Approach to Mult-Scale Edge Detecton, Image and Vson Computng, vol 1, pp 1-10, 00 [18] J Wang, M Iata, H Kozum, H Shmzu, S Goto and T Ikenaga, Mult-scale Fragmented Edges Groupng for Monocular Buldng Extracton, n Proc 0th Karuzaa Workshop, Apr 007 [19] S Krshnamachar and R Chellappa, Delneatng Buldngs by Groupng Lnes th MRFs, IEEE Trans on Image Processng, vol 5, no 1, pp , 1996 Jng Wang receved the BE and ME degrees n the Department of Computer Scence and Technology from Northester Polytechnc Unversty n X'an, Chna respectvely n 00 and 005 From Aprl 005, she s a PhD canddate n the Graduate School of Informaton, Producton and Systems, Waseda Unversty Her research nterests are manly n mage segmentaton, contour extracton and mage understandng Toru Mayazak receved the ME degree n the Department of Informaton and Computer Scences from Toyohash Unversty of Technology n 006 He orks at the System Technology Laboratory, NEC System Technologes, Ltd He s a member of Informaton Processng Socety of Japan Hs research nterests nclude mage processng, computer graphcs, vrtual realty Hanyang Unversty, Korea Currently, he s a Presdent of IEEK (Insttute of Electroncs Engneerng of Korea) Hs current research nterests are n CAD algorthm and desgn methodology for SoC, VLSI desgn for dgtal sgnal and mage processng, vdeo compresson, hgh-speed reless LAN, and dgtal communcaton systems Hroyuk Yagyu receved the BE degree n Electronc Engneerng from Osaka Electro-Communcaton Unversty n 1985 He orks at the System Technology Laboratory, NEC System Technologes, Ltd Hdeo Shmazu s currently the drector of System Technology Laboratory, NEC System Technologes, Ltd He receved hs PhD degree th hs knoledge management research n Meda and Governance from Keo Unversty Hs ork focuses on artfcal ntellgence, nformaton securty, mult-meda processng and ubqutous computng Takesh Ikenaga receved the BE and ME degrees n electrcal engneerng and the PhD degree n nformaton \& computer scence from Waseda Unversty, Tokyo, Japan, n 1988, 1990, and 00, respectvely He joned LSI Laboratores, Nppon Telegraph and Tele-phone Corporaton (NTT) n 1990, here he has been undertakng research on the desgn and test methodologes for hgh-performance ASICs, a real-tme MPEG encoder chp set, and a hghly parallel LSI & System desgn for mage-understandng processng He s presently an assocate professor n the system LSI feld of the Graduate School of Informaton, Producton and Systems, Waseda Unversty Hs current nterests are applcaton SOC for mage, securty and netork processng He s a member of the IPSJ and the IEEE He receved the IEICE Research Encouragement Aard n 199 Hrokazu Kozum receved the ME degree n Informaton and Communcaton Engneerng from the Unversty of Tokyo n 000 He orks at the System Technology Laboratory, NEC System Technologes, Ltd Hs research nterests are mage processng, computer vson and pattern recognton Makoto Iata receved the ME degree n Informaton Engneerng from Nara Insttute of Scence and Technology n 1997 He orks at the System Technology Laboratory, NEC System Technologes, Ltd Hs research nterests are mult-meda processng and pattern recognton He s a member of Informaton Processng Socety of Japan System Satosh Goto as born on January rd, 195 n Hroshma, Japan He receved the BE and ME degree n Electroncs and Communcaton Engneerng from Waseda Unversty n 1968 and 1970, respectvely He also receved the Dr of Engneerng from the same unversty n 1981 He s IEEE fello, member of Academy Engneerng Socety of Japan and professor of Waseda Unversty Hs research nterests nclude LSI system and Multmeda JongWha Chong receved the BS and MS degrees n electronc engneerng from Hanyang Unversty n Seoul, Korea, n 1975 and 1977, respectvely and the PhD degree from Waseda Unversty, Japan, n 1981 n electronc communcaton engneerng From 1979 to 1980, he as th NEC Central laboratory Snce 1981, he has been a Professor and drector of CAD & SoC Desgn Center n the department of electroncs and computer engneerng,