IMAGE-GUIDED NON-LOCAL DENSE MATCHING WITH THREE-STEPS OPTIMIZATION

Transcription

1 IMAGE-GUIDED NON-LOCAL DENSE MATCHING WITH THREE-STEPS OPTIMIZATION Xu Huang a, Yongjun Zhang a, *, Zhaoxi Yue a a Schoo of Remote Sensing and Infomation Engineeing, Wuhan Univesity, Wuhan, Hubei, China Commission Ⅲ, WG Ⅲ/1 KEY WORDS: HOG; Image Guided Matching; Non-oca Dense Matching; SGM; Image Guided Intepoation ABSTRACT: This pape intoduces a new image-guided non-oca dense matching agoithm that focuses on how to sove the foowing pobems: 1) mitigating the infuence of vetica paaax to the cost computation in steeo pais; 2) guaanteeing the pefomance of dense matching in homogeneous intensity egions with significant dispaity changes; 3) imiting the inaccuate cost popagated fom depth discontinuity egions; 4) guaanteeing that the path between two pixes in the same egion is connected; and 5) defining the cost popagation function between the eiabe pixe and the uneiabe pixe duing dispaity intepoation. This pape combines the histogam and an impoved histogam of oiented gadient (HOG) featue togethe as the cost metics, which ae then aggegated based on a new iteative non-oca matching method and the semi-goba matching method. Finay, new ues of cost popagation between the vaid pixes and the invaid pixes ae defined to impove the dispaity intepoation esuts. The esuts of ou expeiments using the benchmaks and the Toonto aeia images fom the Intenationa Society fo Photogammety and Remote Sensing (ISPRS) show that the poposed new method can outpefom most of the cuent state-of-the-at steeo dense matching methods. 1. INTRODUCTION Steeo dense matching, which aims to find pixe-wise coespondences between a steeo pai, has attacted inceasing attention in the photogammety and compute vision communities fo decades. Athough steeo matching is matuing, eseach gaps emain in the aspects eated to impoving the pefomance of cost computation, matching accuacy in homogeneous intensity egions, and dispaity intepoation. Most steeo matching agoithms cuenty consist of fou steps: 1) cost computation, 2) cost aggegation, 3) dispaity computation, and 4) dispaity efinement (Schastein and Szeiski, 2002). 1.1 Review of pevious wok Cost is defined as the measue to descibe the simiaity between coespondences. The pefomance of cost computation methods ae affected by the adiometic conditions of the image. Fo exampe, when the image adiometic condition is good, the absoute diffeence (AD), the insensitive measue of Bichfied and Tomasi (BT), o the gadient measue can achieve accuate matching esuts (Meiet et a, 2011). When the image adiation vaies, zeo-based nomaized coss coeation (ZNCC) and nomaized gadient (Zhou and Bouange, 2012) ae often used to compensate fo inea adiation distotions between coespondences, whie (Zabih and Woodfi, 2005; Jiao et a., 2014; Kodeas et a, 2015), mutua infomation (Pau et a, 1997), and image adiation coection (Jung et a, 2013) ae insensitive to noninea adiation distotion. Hischmue evauated the popua cost computation methods and concuded that and mutua infomation measues can achieve the best matching esuts unde vaying adiometic conditions (Hischmuee and Schastein, 2009). In addition to image adiation, cost computation aso is infuenced by the vetica paaax of the epipoa steeos, wheeby the intensity distibution of the centa pixe may be diffeent fom that of its coespondence in ich textue aeas, theeby making the coesponding cost computation based on oca metics potentiay uneiabe. In ode to make the cost computation moe obust, most steeo matching methods choose to enage the window of cost computation. Howeve, thee is no consensus on the appopiate size. In addition, the un time of cost computation wi incease with the gowing size of the window. The dispaities of the pixes in a age window may not be consistent, which may bu the edges in depth discontinuity egions. Thus, investigating a new cost computation method that can weaken the infuence of vetica paaax is wothwhie. In 2002, Schastein and Szeiski (2002) divided steeo matching methods into two types, oca methods and goba methods, accoding to the cost aggegation patten. In ecent yeas, many eseaches have deveoped oca methods into non-oca methods successfuy. As oca methods and the ate non-oca methods ae essentiay image-guided, and goba methods o semi-goba methods (SGM) ae based on the minimization of enegy function (Taniai et a, 2014; Yang et a, 2009; Hischmue, 2008). Steeo matching methods aso can be divided into image-guided methods and enegy function-guided methods. Image-guided steeo matching methods suppose that a the pixes with simia intensities in the suppot window o homogeneous intensity egions have the same dispaities. The cost aggegation is guided by the image intensity. Loca methods, such as the biatea fite (Yoon and Kweon, 2006) and the image-guided fite (He et a, 2013), have achieved matching esuts as good as those of goba methods since Howeve, the oca methods need to define the window size fist. Lage windows take moe time fo cost aggegation whie sma windows pefom bady in textueess aeas. In ode to avoid the window definition, the non-oca methods, which ae based on ecusion, wee poposed (Yang, 2015; Pham and Jeon, 2013; Ciga and Aantan, 2013; Sun et a, 2014; Cheng et a, 2015), which diffeed fom the oca methods in that the cost aggegation of evey pixe is suppoted by the emaining pixes in the whoe image fo non-oca methods. The suppots fom the emaining pixes depend on the intensity simiaity and the cost aggegation path. The non-oca methods ae fast and pefom we in depth discontinuity egions o textueess egions with consistent dispaities. Howeve, none of the cuent methods conside that the intensities of pixes may be simia but the coesponding dispaities may change smoothy o shapy, in which case the non-oca methods may pefom bady. Thus, the non-oca methods need to be impoved in ode to avoid the pobem of inconsistent dispaities. Image- doi: /ispsannas-iii

2 guided matching woks we in depth discontinuity aeas, and enegy function-guided matching can achieve moe obust matching esuts. If both methods ae combined, moe accuate matching esuts can be achieved. The iteatue indicates that the oca methods and the Goba/SGM methods have been combined successfuy (Mei et a, 2011; Žbonta and LeCun, 2015; Mozeov and van de Weije, 2015). Howeve, combining non-oca methods and Goba/SGM methods has yet to be achieved. Initia dispaity images, afte cost aggegation and dispaity computation, sti have many mismatches that must be eiminated by outie detection, such as a eft-ight consistency check and emova of peaks. The outie detection may invaidate some dispaities and may ead to hoes in the dispaity image, which then needs to be intepoated fo a dense esut. Intepoation methods can be divided into dispaity image-based intepoation and intensity image-guided intepoation. Dispaity image-based intepoation methods cassify invaid dispaities into mismatches and occusions. The intepoation of invaid dispaities is based on the neighbouhood vaid pixes (Mei et a., 2011; Hischmue, 2008). This method is fast and can achieve good intepoation esuts fo good initia dispaity images. Howeve, if the vaid pixes ae insufficient, the intepoation esuts wi be unsatisfactoy. Intensity image-guided intepoation supposes that pixes with simia intensities have consistent dispaities (Yang, 2015). The vaid dispaities ae popagated fom vaid pixes to invaid pixes with simia intensities. Athough thee ae ess vaid pixes, this method can sti acquie satisfactoy intepoation esuts. Howeve, intensity image-guided methods may be invaidated in the homogenous intensity egion with inconsistent dispaities. Instead, it is moe easonabe to aow dispaity changes fo pixes with simia intensities. Thus, a new intensity image guided intepoation method which can satisfy the new hypothesis is needed. 1.2 Contibution of this pape This pape poposes a new image-guided non-oca dense matching method with a thee-step optimization based on the combination of image-guided methods and enegy functionguided methods. The image-guided non-oca method with a thee-step optimization (INTS) consists of the foowing steps: (1) a new non-oca method is adopted to stengthen the cost popagation in the homogenous intensity egions; (2) the semigoba matching method (SGM) (Hischmue, 2008) is used to guaantee cost popagation in the aea with ich textues; and (3) a new intensity image-guided intepoation method is utiized to intepoate invaid dispaities, fom which the fina matching esuts ae acquied. The main contibutions of this pape ae as foows: This pape impoves the histogam of the oiented gadient (HOG) featue, which is capabe of being inea adiometic invaiant. The impoved HOG featue is used to compute costs, and this is the fist time the HOG featue is used as the cost metic, which is abe to educe the infuence of vetica paaax to cost computation. This pape poposes a new image-guided non-oca matching method whee penaty tems ae intoduced duing matching in homogenous intensity egions. A new kene function is poposed, which is moe obust than the Gaussian kene function. The costs popagated fom depth discontinuity egions ae imited. New cost popagation paths ae avaiabe, which can guaantee the connectivity of the path between the homogenous pixes. The new method can avoid the mismatches in homogenous intensity egions with inconsistent dispaities. This pape poposes a new intensity image-guided intepoation method. The new method defines new ues of popagation fom vaid pixes to invaid pixes, which can impove the intepoation accuacy. 2.1 Cost Computation 2. PROPOSED METHOD HOG is a we-known featue descipto fo object detection, which has been successfuy used in image ecognition (Tiggs and Rhone-Aps, 2005). As HOG can descibe object featues accuatey, it aso has been used as a cost metic. Howeve, computing the compete HOG featues fo evey pixe is vey time-consuming. HOG featues aso ae adiometic vaiant, which may be uneiabe due to vaying adiometic conditions. Thus, in this pape, an impoved HOG featue is poposed which is not ony fast but aso is inea adiometic invaiant. This pape supposes that the adiation distotion is inea in a vey sma window, such as a 3 3 neighbouhood. The inea adiation distotion can be expessed as the foowing equation: g ( p ) c g ( p ) t (1) whee, p and p epesent the coespondences in eft and ight images, espectivey; when steeo pais ae epipoa images, the eationship between thei hoizonta odinates is px px d, whee d epesents the dispaity; g and g epesent the intensities of the coespondences espectivey; c and t ae the coefficients of the inea adiation distotion mode. The tansation facto t can be eiminated by gadient computation in a oca sma window. The Sobe opeato is used in this pape. In ode to eiminate the scae facto c futhe, gadient diections ae cacuated as foows: ( p ) ( p ) ( p ) actan Gy ( p ) / Gx ( p ) ( p ) actan Gy ( p ) / Gx ( p ) whee, Gx ( p ) and Gx ( p ) epesent the hoizonta gadients of p and p espectivey; Gy ( p ) and Gy ( p ) epesent the vetica gadients of p and p espectivey; ( p ) and ( p ) epesent the gadient diections of p and p espectivey. The gadient diection is a inea adiometic invaiant metic; and the ange of gadient diections is [0, 360 ). We define a W W window cented at pixe p as a basic desciption ce. A the gadient diections in the ce ae counted, and then a gadient diection histogam is buit, as shown in Figue 1. The ange of gadient diections is divided into 12 bins. The initia count of evey bin is set as zeo. When the gadient diection in the ce beongs to a cetain bin, add 1 to the coesponding count. Figue 1(a) epesents the desciption ce, whee the ectange epesents a pixe in the ce; the diection of the aow epesents the gadient diection; the backgound coous of the pixes coespond one to one with the bins of the gadient diection histogam. Figue 1(b) epesents the gadient diection histogam, whee the numbe in each bin epesents the coesponding count. Finay, the (2) doi: /ispsannas-iii

3 count in each bin is divided by the sum of the pixes in the ce fo nomaization. Accoding to the nomaized gadient diection histogam, a 12 1 vecto can be constucted as the featue descipto of pixe p, as shown in Equation (3). Figue 1. Gadient Diection Histogam V ( p ) ( b, b, b, b, b, b, b, b, b, b, b, b ) (3) HOG T whee, b i (i = 0~11) epesents the vaue in each bin of the nomaized gadient diection histogam; and VHOG ( p ) epesents the featue descipto of pixe p. When compaed with the taditiona HOG featue, the impoved HOG negects the gadient nom duing the constuction of the histogam, which makes it possibe fo the impoved HOG to be inea adiometic invaiant. Aso, the taditiona HOG needs to combine sevea basic desciption ces into a bock to constuct a age featue descipto. As the cost aggegation step of steeo matching can aggegate the cost in a neighbouhood togethe, the impoved HOG ony descibes pixe featues at the eve of ces instead of in bocks, which can avoid epeated cacuations. This pape egads the distance between the HOG featue desciptos of the coespondences as cost metics, as shown in Equation (4). In ode to descibe the pixe featues moe pecisey, this pape combines HOG and as the fina cost metic, as shown in Equation (5). C ( p V p V ep( p (4) HOG HOG HOG whee, ep( p epesents the suspected coesponding pixe of p with d as a dispaity, namey, p ep( p ; CHOG ( p epesents the HOG cost metic between pixe p and p ; V ( p ) HOG epesents the HOG featue descipto of p in the eft image; and VHOG ep( p epesents the HOG featue descipto of p in the ight image. C( p q min C p, ep( p, t / t t HOG (1 q) min C p, ep( p, t HOG HOG whee, C( p epesents the cost of p with d as a dispaity; C epesents the cost cacuated by metics; C HOG epesents the cost cacuated by HOG metics; t, t HOG epesent the tuncation theshods; and q epesents the weight coefficient whose ange is [0, 1]. In ode to make C and C HOG in the same ange, C is scaed by thog / t. (5) The diffeence between the y coodinates of coespondences in steeo epipoa pais is caed vetica paaax. The vetica paaax has a coupt infuence on the cost computation, especiay in the aeas with ich textues. The HOG featues can educe the infuence of vetica paaax to a cetain extent. Each gadient diection computation, except fo the centa gadient, is independent of the centa pixe in the basic ce of HOG; and the vetica paaax of the centa pixe theefoe cannot affect the computation of othe gadient diections. Each bin in the gadient diection histogam aso has a 30-degee ange. When the gadient diection changes because of vetica paaax, the computation of the HOG featues is not affected as ong as the change is no moe than the ange. In Section 3.1, expeiments on steeo epipoa images with vetica paaax ae discussed. 2.2 Image-guided Non-oca Matching In ecent yeas, sevea image-guided non-oca methods wee poposed, of which the basic mathematic modes ae consistent essentiay (Yang, 2015; Pham and Jeon, 2013; Ciga and Aantan, 2013; Sun et a, 2014; Cheng et a, 2015): L ( p C( p T L ( p -1 (6) whee, L( p epesents the aggegated cost of pixe p at dispaity d in the cuent path; epesents the diection of the path; C( p epesents the cost of pixe p at dispaity d; p 1 epesents the pevious pixe in the cuent path; and T epesents the intensity simiaity of pixe p and p 1, which is used to constain the cost popagation between p and p 1. Geneay, T is computed by the Gaussian kene function: T T ( p, p -1) exp g( p) g( p 1) / (7) G whee, TG epesents the constaint tem T which is computed by the Gaussian kene function; g epesents the image intensity; and epesents the smooth tem. The vaue of T is cose to 1 in homogenous intensity egions, which means that the non-oca methods foce dispaities in the homogenous egions to be consistent. It is obviousy impobabe because the suface of textueess aeas in the ea wod may be uneven so the coesponding dispaities shoud be inconsistent. In ode to sove the matching pobem caused by textueess aeas with inconsistent dispaities, this pape intoduces penaty tem P 1 and P 2 into Equation (6), as shown in Equation (8). Compaed to Equation (6), Equation (8) consides the change in dispaities in homogenous intensity egions. When the neighbou dispaity changes smoothy, a owe penaty P 1 is used fo santed o cuved sufaces. When the dispaity changes shapy, a age penaty P 2 is used fo depth discontinuities. L ( p -1, ( -1, 1) 1, L p d P L ( p C( p T min L ( -1, 1) 1, p d P min L ( p -1, k) P2 k When pixe p and p + 1 is ocated in the same homogenous egion, taditiona non-oca methods fuy tust the cost popagated fom p and add a age constaint tem T ( p +1, p ). Howeve, the cost popagated fom the pixe in the same egion (8) doi: /ispsannas-iii

4 may be uneiabe due to the depth discontinuities. The cost computation is uneiabe if the dispaities in the cost metic window ae discontinuous (e. g., object edges). The impope cost of pixes aound depth discontinuity egion is the cause of the fattening pobem. This pape intoduces a new appoach to compute constaint tem T, which can educe the fattening pobem geaty. When the cost is popagated fom p to p + 1, the absoute diffeences of image intensities between pixe p + 1 and the pevious s + 1 pixes ae cacuated, espectivey, as shown in Equation (9). If evey diffeence is sma, the cost popagation p is eiabe and a age T is added. Othewise, if one of the diffeences is age, the cost is uneiabe because the cost may be popagated fom depth discontinuities. A smae T is added to educe the popagation. This pape defines s as haf of the cost metic window in a the expeiments. used when the intensity diffeence is age than 2. When is equa to 5 o 20, espectivey, the pefomances of the Gaussian kene function and the quadatic-based kene function ae shown in Figue 2. 1 T ( 1, ) s q p p g( p 1) g( p 1 i) Q T i1 Tq ( p 1, p 1 t) g( p 1) g( p 1 t) Q t whee, the symbo s means any, namey, that any one of s + 1 pevious pixes aong the cost aggegation path; the symbo means exist, namey, that thee exists at east one of the pevious s + 1 pixes which satisfy the theshod condition. t is decided by the fist one of the pevious pixes that satisfy the theshod condition. Q is the theshod of the intensity diffeence. Tq epesents the constaint tem T, which is computed by a quadatic-based kene function. The quadaticbased kene function is descibed beow. Taditiona image-guided methods adopt the Gaussian kene function to compute constaint tem T, as shown in Equation (7). The Gaussian kene function is a deceasing function, whose absoute sope is aso deceasing. The Gaussian function vaue T G with a sma smooth tem wi decease geaty, when the intensity diffeences ony change sighty aound zeo. In fact, it is impossibe that the intensities of pixes in the same egion ae exacty the same. In ode to stengthen the cost popagation in homogenous intensity egions, taditiona methods choose a age ( =15~25) to acquie a age T (Yang, 2015; Pham and Jeon, 2013; Ciga and Aantan, 2013; Sun et a, 2014; Cheng et a, 2015). Howeve, when the cost is popagated between two diffeent egions, T may emain age with a age, which may bing mismatches in the depth discontinuities. This pape intoduces a new kene function based on a quadatic tem, whose sope is inceasing in the ange [0, 2 ], as foows: 2 a g g 1 2 Tq ( p 1, p) TG ( p, p 1) g 2 g g( p 1) g( p) a ( e 1) / (9) (10) whee, Tq epesents constaint tem T based on the quadaticbased kene function; a epesents the coefficient of the quadatic tem; and epesents the smooth tem. The absoute sope of the quadatic-based kene function is inceasing in the ange [0, + ]. When intensity diffeences change a itte bit, the vaue of T is sti age. When the intensity diffeences change geaty, the vaue of T deceases shapy. The ange of T is [0, 1]. In ode to make the new kene function meet the ange, the Gaussian kene function is Figue 2. Compaison of Gaussian Kene Function and Quadatic-based Kene Function In Figue 2, the hoizonta axis epesents the absoute vaue of intensity diffeence g ; and the vetica axis epesents the vaue of constaint tem T. Tq ( ) is defined as the vaue of T computed by the quadatic-based kene function. TG ( ) is defined as the vaue of T computed by the Gaussian kene function. It can be seen fom Figue 5 that when 5, the quadatic-based kene function can acquie a age Tq (5) with a sma intensity diffeence ( g 5 ). Howeve, the coesponding Gaussian kene function vaue TG (5) deceases shapy. If a age smooth tem (e.g., 20 ) is chosen, the Gaussian kene function can acquie a age T G (20) when the intensity diffeence g is sma. Howeve, it does not mean that the coesponding TG (20) wi become sma when the intensity diffeence is age ( g 10 ). It is aso woth noting that when the intensity diffeence is sma ( g 5 ), the quadatic-based function vaue Tq (5) with 5 is sti age than the Gaussian kene function vaue TG (20) with 20, which shows the obustness of the quadatic-based kene function in the case of sma smooth tem. The matching esuts of the non-oca methods ae eated to the cost aggegation path. This pape cassifies the cost popagation paths into connected paths and unconnected paths. If a the pixes fom the beginning to the end beong to a same egion, the path is caed a connected path; othewise, the path is caed an unconnected path. A good cost aggegation path shoud be connected. Most non-oca methods define paths though hoizonta/vetica scanines (Pham and Jeon, 2013; Ciga and Aantan, 2013; Sun et a, 2014; Cheng et a, 2015). Howeve, the path, which is descibed by hoizonta/vetica scanines, may be unconnected even though the beginning and the end beong to the same egion, as shown in Figue 3. Figue 3. Paths Descibed by Hoizonta/Vetica Scanines doi: /ispsannas-iii

5 In Figue 3, the ed cices epesent pixes; the bue ines epesent paths; and the aows epesent the diections of the cost aggegation. Pixes p1 and p2 beong to the same egion. Pixes p3 and p4 beong to othe egions. If the cost of p2 is popagated to p1, thee ae two paths: p2 p3 p1 o p2 p4 p1. Regadess of the path chosen, passing though the pixe beyond the egion of p1 and p2 cannot be avoided. Thus, the suppot fom p2 to p1 is vey sma, namey, the path is unconnected. In ode to sove the above pobem, this pape poposes a new cost aggegation method based on eight diections, which contain not ony the hoizonta/vetica diections, but fou diagona diections as we. The new method needs two iteations. In the fist iteation, the cost aggegation esuts fom the eight diections ae summed, as foows: 8 S( p C( p L ( p C( p (11) 1 whee, L epesents the aggegated cost cacuated by Equation (8); epesents the path diections, incuding 0, 45, 90, 135, 180, 225, 270, and 315 ; and S epesents the sum of the aggegated cost fom the eight diections. Afte the fist iteation, the path ony exists aong scanines of the 8 diections fo evey pixe. Thee is no path to ink the pixe beyond the scanines. Thus, egad the aggegation esut S in the fist iteation as the new cost in the second iteation, and then aggegate the new cost again aong the scanine. Finay, 2 sum the cost aggegation esuts fom the 8 diections. S is defined as the new aggegation esut in the second iteation. Afte two iteations, thee exists paths between any two pixes in the image. The path is connected o unconnected. The fina cost aggegation esuts may vay geaty in magnitude fo each pixe, which wi make it necessay to 2 nomaize aggegation esut S. Afte two iteations, the path between abitay two pixes is shown in Figue Semi-goba Matching (SGM) based on Non-oca Aggegated Cost The above image-guided non-oca method pefoms we in homogenous intensity egions, but the matching esut is not obust in the textue egions because the non-oca methods do not conside the cost popagation between diffeent egions. As the enegy function-based matching methods pefom we in textue egions, the poposed method combines the two. Fist, the non-oca matching method in Section 2.2 was adopted. 2 Then, the cost aggegation esut S wee egaded as the new cost and the semi-goba method (Hischmue, 2008) based on the new cost was used to guaantee pefomance in textue egions. Finay, the Winne Takes A (WTA) stategy was adopted to achieve the initia dispaity image, and the eft-ight consistency check method was used to eiminate outies. 2.4 Image-guided Dispaity Intepoation Afte the eft-ight consistency check, some invaid pixes wee pesent in the initia dispaity image. In ode to achieve bette matching esuts, these invaid pixes needed to be intepoated. This pape theefoe poposes a new image-guided dispaity intepoation method as we. In this method, the vaid pixes ae egaded as the eiabe pixes, and the invaid pixes as uneiabe pixes. The dispaities of the uneiabe pixes ae intepoated by the eiabe pixes in the same egion. The intepoation method is simia to the non-oca method descibed in Section 2.2. Fist, the cost of evey pixe is computed accoding to the initia dispaity image, as shown in Equation (12). C( p 0 0 min ( p), ( p) 0 0 ( ) d M t if M vaid if M p invaid (12) whee, C( p epesents the cost of pixe p at dispaity d; 0 M epesents the initia dispaity image; and t epesents the tuncation theshod. Then, the cost is aggegated in a manne simia to the non-oca method descibed in Section 2.2. Diffeent fom the non-oca method, the intepoation method makes fu use of the eiabe pixes and uneiabe pixes to constain the cost popagation path. Utiizing the eiabe pixes and uneiabe pixes, this new method defines a new constaint tem T, as foows. Figue 4. Paths afte Two Iteations Figue 4 shows the paths whee the cost is popagated fom pixe p1 to p3. The cices epesent pixes; the geen ines epesent the eight scanines of p1, and the bue ines epesent the eight scanines of p3. The paths of cost popagation ae defined by the intesections between the two sets of scanines, which ae epesented by pupe cices; and the aows epesent the diections of the cost popagation. It can be seen fom Figue 4 that the cost aggegation method based on eight diections coud povide sevea paths fo cost popagation fom p1 to p3, incuding hoizonta paths, vetica paths, and diagona paths. Odinaiy, at east one of these paths must be connected; but it is possibe that none of the paths ae connected in the ing o U-shaped egions. if p eiabe and T ( p, p 1) p +1 eiabe if p uneiabe and T ( p, p 1) p +1 uneiabe T ( p, p 1) if p uneiabe and 0 p +1 eiabe T ( p, p1) if p eiabe and u 1 p +1 uneiabe (13) Equation (13) epesents the vaue of T when the cost is popagated fom p to p + 1. When both p and p + 1 ae eiabe pixes o uneiabe pixes, T is sti computed by Equation (10). When p is an uneiabe pixe but p + 1 is a eiabe pixe, T is set as equa to 0, which cuts off the path between p and p + 1. When p is a eiabe pixe but p + 1 is an uneiabe one, T is stengthened by an exponentia function, which encouages the doi: /ispsannas-iii

6 popagation fom p to p + 1. Odinaiy, the vaue of u shoud be age than 2. The detaied paths duing cost popagation ae shown in Figue 5. (a) Epipoa Image (b) Cost Metic Figue 5. Cost Popagation Path Defined by Reiabe Pixes and Uneiabe Pixes In Figue 5, the geen cices epesent eiabe pixes; the ed cices epesent uneiabe pixes; the aows epesent the diections of cost popagation; the ines epesent the cost popagation paths; and the thickness of the ines is eated to the vaue of T. The ine is thicke fo a age vaue of T. A ed coss on the ine means the cost is fobidden to pass though the path. Afte cost aggegation, the WTA stategy is adopted again to achieve the fina dispaity image. 3. EXPERIMENTS In ode to test the pefomance of the poposed method, a seies of expeiments wee caied out. The expeiments can be divided into five pats: (1) expeiment on the steeo pai with vetica paaax; (2) expeiment on the steeo pai with inconsistent dispaities in homogenous intensity egions; (3) expeiment on the famous Middebuy Steeo Vision data sets; (4) expeiment on the KITTI benchmak; (5) expeiment on the actua aeia imagey of Toonto. The fist expeiments aimed at testing the obustness of the poposed HOG cost metic in steeo pais with vetica paaax. The second expeiments compaed the taditiona non-oca method and the poposed non-oca method. The thid expeiments compaed the poposed method with the state-of-the-at matching methods. The fouth expeiment tested the poposed method in steet view images. The fifth expeiments tested the eiabiity of the poposed method in outdoo images. In these five expeiments, a the matching paametes wee fixed, as shown in Tabe 1. Step Cost Computation Image-guided Non-oca Matching Dispaity Intepoation Paamete Window Size Weighting Coefficient q Smooth Tem Penaty Tem P1 Vaue 5 (pixe) Penaty Tem P Tuncation Theshod t Smooth Tem Function Base u Tabe 1. Matching Paametes (c) HOG Cost Metic (d) +HOG Cost Metic Figue 6. Compaison of Cost Metic Compaing Figue 6(b) and Figue 6(c) indicates seious mismatches, especiay in the foo egion when the cost metic was used, which was due to the foo egion being ich in fine-textues. Theefoe, the cost computation may be wong even though the vetica paaax is sma. On the othe hand, the HOG cost metic was abe to educe the infuence of vetica paaax, which can achieve a bette matching esut. Figue 6(d) shows the esuts of the and HOG combination. The weight of is 0.3, as shown in Tabe 1. The matching esut of the combination metic was bette than that of the metic but was wose than that of the HOG metic. Athough the metic was sensitive to the vetica paaax, it pefomed we in steeo pais with noninea adiometic distotions. Thus, the combination metic of and HOG was used in the poposed new method. 3.2 Epipoa Steeo Pai with Inconsistent Dispaities in Homogenous Region Adiondack data povided by Middebuy Steeo Vision wee used, as shown in Figue 7(a). Both of the egions in the ed ectanges ae homogenous intensity egions. The dispaity changed shapy in Rectange 1. The dispaities in Rectange 2 changed smoothy. Figue 7(b) is the esut of the taditiona non-oca method (Ciga and Aantan, 2013). Figue 7(c) is the esut of the poposed method. In ode to compae the pefomances of both methods soey, ony the initia dispaity images wee given. (a) Epipoa Image 3.1 Epipoa Steeo Pai with Vetica Paaax Existing This pape chose PayTabe data povided by Middebuy Steeo Vision fo expeiments, as shown in Figue 6(a). The aveage vetica paaax was pixes. The maximum vetica paaax was pixes. The cost metic, the HOG cost metic, and a combination of and HOG wee used in the poposed matching method. The coesponding matching esuts ae shown in Figue 6(b), Figue 6(c) and Figue 6(d), espectivey. (b) taditiona method (c) poposed method Figue 7. Compaison of Matching in Homogenous Regions Figue 7(b) ceay shows seious mismatches in both Rectange 1 and Rectange 2 if thei dispaities wee foced to be doi: /ispsannas-iii

7 consistent. The poposed method consides the changes in the dispaities in the homogenous intensity aeas, which achieved a bette matching esut fo Figue 7(a). 3.3 Test on Middebuy Steeo Vision Data Sets The poposed method was used to match steeo pais in Middebuy benchmak. The aveage absoute eo in the pixes was the accuacy metic. The pefomance of the poposed method is shown in Tabe 2, which ists the fina matching esuts of the top 8 agoithms in the Middebuy Steeo Vision Benchmak. In Tabe 2, the fist ow ists the name of evey agoithm, and the second ow ists the genea aveage absoute eo. The cented numbe epesents the matching accuacy, and the supescipt numbe epesents the ank. The thid ow ists the unning time. The fouth ow ist the unning envionment. The coesponding ce is made up of two pats: the eft pat epesents the pocesso (CPU o GPU), and the ight pat epesent the numbe of pocesso coes used fo unning the agoithms (seia o paae). It can be seen fom Tabe 2 that the genea matching accuacy of the INTS method anks top five in the Middebuy Steeo Vision by the end of The INTS method uses a singe i7 CPU coe fo matching, and the aveage unning time is 104s. Athough INTS does not have an obviousy supeio un time, it was the fastest compaed to fou othe top methods when a the un time was conveted to the same unning envionment. In addition, the cost aggegation of eight diections in Sections 2.2 and 2.3 was independent, which woud be easy to impement to paae pocessing fo acceeation. Name optica fow infomation was not used, which has aeady been used in top agoithms in KITTI benchmak; 2. INTS method pefomed bady in backgound egions. Futue wok wi focus on how to impove the matching esuts in backgound egions. 3.5 Test on Aeia Imagey of Toonto This pape appied the INTS method to aeia images of Toonto to test the pefomance of matching outdoo images, as shown in Figue 8(a). The steeo pai was captued by the Micosoft Vexce s UtaCam-D (UCD) camea. The image size is In ode to test the matching accuacy, the coesponding LiDAR point set was used as conto infomation, which was captued by the Optech aibone ase scanne ALTMORION M. The point density is appoximatey 6.0 points/m2. The the matching esut is shown in Figue 8(b). (a) Epipoa Image (b) Reconstuction Resuts Figue 8. Reconstuction of Toonto NTDE MCCNN_Layout MCCNN+RBS SOU4Pnet INTS LCU MC-CNN Mesh Steeo Avg (pixe) Time (s) CPU /GPU Count Gefoce GTX TITAN X i7 1 NVIDIA GTX TITAN X GTX 980 GPU i7 1 E NVIDIA GTX TITAN Back (GPU) i7 8 Tabe 2 Rank in Middebuy Steeo Vision Benchmak ( ) 3.4 Test on KITTI Benchmak INTS method was tested on steeo 2015 data sets povided by KITTI. A pixe is consideed to be coecty estimated if the dispaity is ess than 3 pixes. The pefomance of INTS is shown in Tabe 3. In tabe 3, D1 epesents pecentage of steeo dispaity outies in fist fame; bg epesents pecentage of outies aveaged ony ove backgound egions; fg epesents pecentage of outies aveaged ony ove foegound egions; a epesents pecentage of outies aveaged ove a gound tuth pixes. D1-fg (%) D1-a (%) Eo D1-bg (%) A/A A/Est Noc/A Noc/Est Rank 22 Tabe 3. Accuacy Evauation It can be seen fom Tabe 3 that ony about 7% pixes ae outies fo steet view image matching with INTS method. INTS method pefomed we in foegound egions, but it pefomed bady in backgound egions. It may be caused by deep depth of fied in backgound egions. The ank in KITTI benchmak is ony 22th. The ank is not high fo two easons: 1. It can be seen fom Figue 8(b) that the INTS method can achieve good econstuction esuts in uban aeas. Occusion egions and shadow egions do exist, but the ta buidings wee econstucted we. In addition to the aeia images of Toonto, ISPRS aso povided the coesponding LiDAR point sets which had been egisteed igoousy. In ode to test the actua matching accuacy of INTS in Toonto, the LiDAR point sets wee egaded as tuth vaues and the matching point sets wee compaed to the LiDAR point sets. Howeve, some outies sti emained in the LiDAR point sets and some LiDAR points ay in occusion egions that wee not visibe in the images. Thus, the LiDAR point sets wee fiteed to eiminate the outies and occusion points befoe compaison. Then, the fiteed LiDAR points wee pojected onto the epipoa steeo pais, which acquied a seies of coespondences. Finay, the dispaities fom dense matching wee compaed with the dispaities fom the LiDAR points to evauate the pefomance of the INTS method. In ode to compehensivey evauate the accuacy esuts, the aveage matching accuacy, the pecent of pixes with matching accuacy beow 0.5 pixes, 1 pixe, 2 pixes and 4 pixes wee chosen as the accuacy metics, as shown in Tabe 4. doi: /ispsannas-iii

8 Aveage (pixes) % 0.5 pixes % 1 pixes % 2 pixes % 4 pixes Numbe of LiDAR Points Running Time s Tabe 4. Accuacy Evauation It can be seen fom Tabe 4 that the matching points of INTS ae vey simia to the LiDAR points. The aveage matching accuacy is ony pixes. The matching accuacies of most points (57.1%) ae beow 0.5 pixe, which shows the obustness of the INTS method. The points with matching accuacies above 4 pixes wee egaded as outies, which wee caused by not ony the INTS method itsef, but aso by the LiDAR points because the fiteing method cannot guaantee tota eimination the outies and occusions in LiDAR point sets. The unning time was ow when ony a singe i7 CPU coe was used. Howeve, the unning time coud be expected to be impoved geaty with paae pocessing. 4. CONCLUSION This pape poposed a new image-guided non-oca dense matching method. An impoved HOG featue was intoduced into the cost computation fo the fist time, which can educe the infuence of vetica paaax. A new non-oca cost aggegation stategy was pesented, which guaantees that the paths between any two pixes wee connected. Vaied dispaities in homogenous intensity egions wee consideed, a moe obust kene function was intoduced, and a new dispaity intepoation method was poposed, which defined new popagation ues to guaantee the intepoation accuacy. Expeiments on benchmaks and the actua aeia imagey showed that INTS method pefomed we in matching accuacy and eiabiity. INTS is one of the pesent state-of-the-at matching methods. Its unning time is not exceptiona at this time with ony one singe CPU, but INTS can be acceeated geaty with paae pocessing in futue wok. ACKNOWLEDGEMENTS This wok was suppoted by the Nationa Natua Science Foundation of China (Gant No , ), and the Academic Awad fo Exceent Ph.D. Candidates funded by the Ministy of Education of China (Gant No ). REFERENCES Cheng, F. Y., Zhang, H., Sun, MG., et a, Coss-tees, edge and supepixe pios-based cost aggegation fo steeo matching. Patten Recognition, 48(7), pp Ciga, C., Aantan, A. A., Infomation pemeabiity fo steeo matching. Signa Pocessing-Image Communication, 28(9), pp Daa, N., Tiggs, B., Histogams of oiented gadients fo human detection. In: IEEE Confeence on Compute Vision and Patten Recognition, San Diego, Caifonia, pp He, K., Sun, J., Tang, X., Guided image fiteing. IEEE Tansactions on Patten Anaysis and Machine Inteigence, 35(6), pp Hischmue, H., Steeo pocessing by semigoba matching and mutua infomation. IEEE Tansactions on Patten Anaysis And Machine Inteigence, 30(2), pp Hischmuee, H., Schastein, D., Evauation of steeo matching costs on Images with adiometic diffeences. IEEE Tansactions on Patten Anaysis and Machine Inteigence, 31(9), pp Jiao, J. B., Wang, R. G., Wang, W. M., et a., Loca steeo matching with impoved matching cost and dispaity efinement. IEEE Mutimedia, 21(4), pp Jung, I. L., Chung, T. Y., Sim, J. Y., et a, Consistent steeo matching unde vaying adiometic conditions. IEEE Tansactions on Mutimedia, 15(1), pp Kodeas, G. A., Aexiadis, D. S., Daas, P., Enhanced dispaity estimation in steeo images. Image And Vision Computing, 35, pp Mei, X., Sun, X., Zhou, M., Jiao, S., Wang, H., Zhang, X., On Buiding an Accuate Steeo Matching System on Gaphics Hadwae. In: 2011 IEEE Intenationa Confeence on Compute Vision Wokshops, Baceona, pp Mozeov, M. G., van de Weije, J., Accuate steeo matching by two-step enegy minimization. IEEE Tansactions on Image Pocessing, 24(3), pp Pau, V., Wiiam, M., Wes, III, Aignment by maximization of mutua infomation. Intenationa Jouna of Compute Vision, 24(2), pp Pham, CC., Jeon, J. W., Domain tansfomation-based efficient cost aggegation fo oca steeo matching. IEEE Tansactions on Cicuits and Systems fo Video Technoogy, 23(7), pp Schastein, D., Szeiski, R., A taxonomy and evauation of dense two-fame steeo coespondence agoithms. Intenationa Jouna of Compute Vision, 47(1-3), pp Sun, X., Mei, X., Jiao, SH., et a, Rea-time oca steeo via edge-awae dispaity popagation. Patten Recognition Lettes, 49, pp Taniai, T., Matsushita, Y., Naemua, T., Gaph cut based continuous steeo matching using ocay shaed abes. In: 2014 IEEE Confeence on Compute Vision and Patten Recognition, Coumbus, OH, pp Yang, Q. X., Wang, L., Yang, R. G., et a, Steeo matching with coo-weighted coeation, hieachica beief popagation and occusion handing. IEEE Tansactions on Patten Anaysis and Machine Inteigence, 31(3), pp Yang, Q. X., Steeo matching using tee fiteing. IEEE Tansactions on Patten Anaysis and Machine Inteigence, 37(4), Yoon, K. J., Kweon, IS., Adaptive suppot-weight appoach fo coespondence seach. IEEE Tansactions on Patten Anaysis and Machine Inteigence, 28(4), pp Zabih, R., Woodfi, J., Non-paametic Loca Tansfoms fo Computing Visua Coespondence. Lectue Notes in Compute Science, 801, pp Žbonta, J., LeCun, Y., Computing the steeo matching cost with a convoutiona neua netwok. In: IEEE Confeence on Compute Vision and Patten Recognition, Boston, USA, pp Zhou, X. Z., Bouange, P., Radiometic invaiant steeo matching based on eative gadients. In: IEEE Intenationa Confeence on Image Pocessing, Lake Buena Vista, FL, pp doi: /ispsannas-iii