PM-Huber: PatchMatch with Huber Regularization for Stereo Matching

Transcription

1 PM-Huber: PatchMatch with Huber Regularization for Stereo Matching Philipp Heise, Sebastian Klose, Brian Jensen, Alois Knoll Department of Informatics, Technische Uniersita t Mu nchen, Germany {heise,kloses,jensen,knoll}@intume Abstract Most stereo corresponence algorithms match support winows at integer-alue isparities an assume a constant isparity alue within the support winow The recently propose PatchMatch stereo algorithm [7] oercomes this limitation of preious algorithms by irectly estimating planes This work presents a metho that integrates the PatchMatch stereo algorithm into a ariational smoothing formulation using quaratic relaxation The resulting algorithm allows the explicit regularization of the isparity an normal graients using the estimate plane parameters Ealuation of our metho in the Milebury benchmark shows that our metho outperforms the traitional integer-alue isparity strategy as well as the original algorithm an its ariants in sub-pixel accurate isparity estimation Figure 1: Stereo pair taken from [13] an a point clou create by using the sub-pixel isparity map generate by our algorithm stereo case this problem can occur in homogeneous untexture regions, regions with repeating structures an extreme sampling choices eg normals nearly orthogonal to the iew irection To alleiate these problems an explicit smoothing moel base on the combination of PatchMatch an Particle Belief Propagation resulting in the PMBP Algorithm [6] has been recently propose, leaing to improe results compare to the original algorithm We present an algorithm base on an explicit ariational energy formulation combining the PatchMatch stereo algorithm with regularization of the isparity an normal graients resulting in sub-pixel accurate isparity maps improing the state of the art Our isparity maps are well suite for the creation of point clous without iscretization or staircasing artifacts as shown in figure 1 1 Introuction Most stereo matching algorithms are base on the assumption that the pixels within the matching winow share the same isparity alue Further ery often only iscrete isparity alues are consiere leaing to iscrete epth layers One reason for the wiesprea use of this simplifie moel is that the number of likelihoo ealuations for more precisely sample isparities an the inclusion of iscretize surface orientations quickly becomes intractable On the other sie sub-pixel accurate epth alues are necessary to create plausible an precise meshes or point clous Bleyer et al[7] showe that the PatchMatch algorithm [4, 5] can be applie for stereo matching using slante support winows so that instea of just estimating a single isparity alue for each pixel a complete isparity plane estimation is mae The PatchMatch algorithm oes not try to iscretize the space of the likelihoo function, but rather relies on ranomize sampling an propagation of goo estimates This also results in an implicit smoothing moel, when goo estimates are propagate in the irect neighbourhoo But the implicit smoothing can also lea to problems when wrong or unreliable estimates are propagate In the 11 Contribution In this paper we show that the projections of scene points belonging to the same planar surface in rectifie stereo pairs are fully relate by a linear transformation with three egrees of freeom This has alreay been shown in [7] for planes in the isparity space an is in the following extene to the real scene space of fully calibrate an rec1

2 tifie stereo cameras Our main contribution is an explicit ariational smoothness moel for the PatchMatch algorithm using quaratic relaxation [12, 17] In [17, 14] only the first orer eriaties of the optical flow ectors an isparity-alues hae been consiere, but the propose algorithm allows us to control the smoothness of the first-orer an secon-orer eriaties of the isparities The secon-orer eriaties of the isparities are implicitly etermine by the graient of the normals estimate by the PatchMatch algorithm Instea of performing an exhaustie search as in [17, 14] for the ealuation of the ata term we employ the PatchMatch algorithm Ealuation of the propose metho for stereo pairs of the Milebury benchmark [15] shows its effectieness in estimating sub-pixel accurate isparity maps At the time of writing we are currently ranke at position 1 out of about 145 algorithms for the sub-pixel error threshol 05 2 Metho 21 Slante support winows In [7] the authors showe how planes in the isparity space affect the patch neighbourhoo For completeness we repeat their result Gien an image point p = [ x 0 y 0 1 ] with the isparity alue z 0 an a normal n = [ n x n y n z ] we can calculate the parameter of a plane π = [ n ] with = n x x 0 n y y 0 n z z 0 This follows from π [ x 0 y 0 z 0 1 ] = 0, which must hol if the point lies on the plane π Therefore the isparity alue z of any image point [ x y ] on the plane is gien by z = n xx n y y + (n x x 0 + n y y 0 + n z z 0 ) n z (1) We can reformulate this as a linear transformation assuming that the point in the secon image is gien by p = p [ z 0 0 ] with z being the isparity as 1 + nx p n z = n y n z nx n z x 0 ny n z y 0 z 0 p (2) For the general case we make use of prior knowlege about the camera projection matrices P = K[I 0] an P = K [R t] with the origin set at the first camera Then the plane-inuce homography from the first to the secon camera [10](p 327) is gien by ( H π = K R t ) n K 1 (3) for a plane π = [ n ] with normal n an istance to the origin For a rectifie stereo camera setup the rotation is the ientity I an the translation between the cameras is gien by [ b 0 0 ] with b being the baseline between the cameras Assuming ientical intrinsics K = K ue to the rectification process an K being an upper triangular matrix the resulting homography is ( H π = K I 1 ) [ b 0 0 ] n K 1 (4) = I K 1 b n x b n y b n z K 1 (5) From equation (2) an (5) it follows that in the case of isparity an scene planes the transformation between two rectifie images inuce by a plane has only three egrees of freeom with a being the scaling, b the shearing an c the translation resulting in the matrix with the following structure 1 + a b c (6) The effects on the support winow is shown in figure 2 Figure 2: Illustration of the shearing an scaling transformation inuce by isparity an scene planes To map from the secon image to the first image the inerse of the matrix from equation (2) or (5) can be use 22 Moel Gien two rectifie stereo color images I 1, I 2 : (Ω R 2 ) R 3, a isparity map : Ω R an a normal map n : Ω {x R 2 : x 1} our algorithm is base on minimizing an energy of the form E(, n) = λe ata (, n) + E smooth (, n), (7) consisting of a ata term escribing the similarity between pointwise matches in the stereo pair an a smoothness term faoring similar isparity an normal alues of ajacent pixels In the following n refers to the non oerparametrize representation of the normal containing only two components If neee the normal ˆn with three components can irectly be calculate from n since we only consier the normals from one half of the unit sphere 1 Our ata term is similar to the one use in [7] E ata (, n) = Ω 1 Z q N (p) 1ˆn = [ n x n y 1 n 2 x n 2 y ] w(p, q) ρ(q, T (, n) q) p (8)

3 T is one of the linear transformations parametrize by an n gien in equation (2) or (5) an ρ measures the pixel similarity between the patches: ρ(p, q) =(1 α) min( I 1 (p) I 2 (q) 1, τ col ) + α min( I 1 (p) I 2 (q) 1, τ gra ) (9) I 1 (p) an I 2 (q) in the preious equation are the linearly interpolate pixel color-alues in the respectie stereo images an I is an four channel image containing the image eriaties calculate by the horizontal an ertical Sobel operator an iagonal graients calculate using central ifferences The eriaties are calculate from grayscale ersions of the stereo images The function w in equation (8) computes a weighting mask base on the color similarity between the center pixel p an the other pixels q insie the patch w(p, q) = e γ(p,q) I1(p) I1(q) 1 (10) In our formulation of w the γ alue changes with istance to the center γ(p, q) = γ min + γ raius smoothstep(0, r max, q p ) (11) The reasoning behin the arying γ is that pixels close to the center belong more likely to the same plane an that pixels far away hae to be ery similar in terms of their coloristance to get the same consieration This formulation is ifferent to a ecreasing weighting factor with increasing istance Z is an normalization constant with Z = w(p, q) (12) q N (p) Our regularization term E smooth imposes spatial smoothness on the isparity-alues an the normals n resulting in E smooth (, n) = with ɛ being the robust Huber norm x ɛ = Ω g(p) ɛ + g(p) n ɛn p, (13) { x 2 2ɛ if x ɛ, x ɛ 2 else (14) As epth an normal iscontinuities often occur at strong image graients we introuce the per-pixel weighting function g(p) with g(p) = e ζ I1(p) η (15) 23 Solution Following [3, 12, 17, 14] we use quaratic relaxation to ecouple our ata an regularization term Introucing an auxiliary ector fiel allows us to perform two alternating minimizations approximating the original minimization problem This results in the following auxiliary energy formulation E aux ( u, n u,, n ) = λ E ata ( u, n u ) with Ω + θ 2 (Π Π u ) Σ (Π Π u ) + E smooth (, n ) p, (16) Π w = [ n w w ] (17) an Σ = iag(σ n, σ n, σ ) being a iagonal matrix weighting the square istances of the normals an the isparity alues Forcing θ to infinity ries the ariables Π u an Π together an results in lim E aux E We split the optimization of the E aux into two sub-problems, namely one θ optimization problem inoling Π u with fixe Π an another one with Π an fixe Π u We collect all fixe terms inepenent of argument minimizing ariable in a constant c Fixe Π u, sole for Π For optimization of the energy E aux we make use of a primal-ual formulation of the Huber-ROF moel as escribe by Chambolle et al [8] The Legenre-Fenchel transformation of the weighte Huber norm g x ɛ using a h(x) a h ( p a ) (a > 0) is gien by { } 1 (g x ɛ ) (p) =g sup x g x p x ɛ = ɛ ( ) 1 2 g p p + δ g p, (18) where δ is the inicator function With the preious result the minimization problem of E aux with respect to can be written as arg min E aux = arg min sup E(, p ) (19) p { = arg min sup p ɛ 2 g(p) p p δ + θσ 2 u 2 p + c g(p), p Ω ( 1 ) g(p) p } (20)

4 Figure 3: From left to right: One image of the portal stereo pair from [2], our isparity map after initialisation, isparity map after the 1st iteration, the final isparity map an two images with ifferent iews of a point-clou generate using the final isparity map We take the eriatie of E(, p ) with respect to an p an using the iergence theorem we get E(, p ) =g(p) i p + θσ ( u ) E(, p ) =g(p) p p g(p) proj(x) = (21) (22) The formulation of the Eaux minimization with respect to n is analogous an leas to the following eriaties E(n, pn ) =g(p) i pn + θσn (n nu ) n E(n, pn ) n pn =g(p) n pn g(p) = g(p) t t+1 p g(p) (24) (25) = g(p) i pt+1 θσ (t+1 u ) (26) n t+1 p g(p) n (27) = g(p) i ptn θσn (nt+1 nu ) (28) = g(p) nt an perform seeral inner iterations using the following upate rules t p + β g(p) t pt+1 = proj (29) 1 + β g(p) 1 t + ν (θσ u g(p) i pt+1 ) 1 + ν θσ t pn + βn g(p) nt = proj 1 + βn n g(p) 1 nt + νn (θσn nu g(p) i pt+1 n ) = 1 + νn θσn t+1 = (30) pt+1 n (31) nt+1 x max(1, x ) (33) The projection fulfills the constraint of the ual ariable p 1 The super-script enotes here the iteration number For the step sizes β, ν, βn an νn we use the alues of ALG3 reporte by Chambolle et al [8] Hana et al [9] also gie a goo introuction an further etails to the Legenre-Fenchel transform an its applications (23) with pn being the ual ariable To sole the energy minimization with respect to Π we use graient escent an ascent as in [14] pt+1 pt β t t+1 ν ptn pt+1 n βn t+1 n nt ν where proj projects back onto the unit sphere (32) Fixe Π, sole for Πu Instea of performing an exhaustie search as one in [17, 14] we employ a ariant of the PatchMatch stereo algorithm Gien a set of samples S(p) for each point p, the best sample θ s? = arg min λ Eata (Πu ) + (Πu Π )> Σ (Πu Π ) 2 Πu S(p) (34) is store at Πt+1 u (p) after each iteration We o not follow the sequential pixel processing scheme from [7], but use a completely parallel approach Our set S(p) is efine as S(p) =SN (p) {Π (p)} Srn N (p) Srn (p) Siew (p) Srn? (p) (35) SN (p) contains the 3 3 patch of samples centere aroun p from the preious iteration The set Srn N (p) contains only one particle from Πtu ranomly chosen from the 7 7 neighbourhoo aroun p Srn (p) is one completely ranomly chosen sample The set Siew (p) contains the iew propagate particles Each position p has storage for a few iew particles an particles from the other iew are propagate if storage is still aailable Srn? (p) is an slightly ranomly perturbe particle base on the best particle from S(p)\Srn? (p)

5 In figure 3 ifferent stages of our algorithm are shown for a stereo pair an the corresponing final isparity map together with a generate point clou The ranomize sampling after the initialisation is clearly isible in the image, but alreay after one iteration the first samples hae been successfully propagate in the neighbourhoo 24 Implementation Details We perform the epth an normal map estimation in both images of the stereo pair This allows us to perform the iew propagation of samples an also left-right consistency checking The left-right consistency checking plays an important role in our algorithm, because it allows the remoal of inconsistent results Especially in the occlue areas arbitrary particles with inconsistent isparity an normal alues are ery often persistent Therefore after each Patch- Match iteration - before we apply the Huber-ROF smoothing - we fill the occlue areas with the next non-occlue plane-particle from the same scanline with the more istant epth alue at the occlue position as illustrate in figure 4 This is similar to the post-processing propose in [7] but without the weighte meian filtering step Our occlusion checking not only uses the epth alues but also the plane normals an allows only isparity ifferences up to 05 an normal eiations of 5 For lookup of the plane parameters in the secon image we o not use linear interpolation but nearest neighbour sampling The occlusion-filling is also one for the final result an is the only post-processing step we perform For the initialisation we foun it beneficial s l Figure 4: The occlue gray area in the first iew is fille using the plane parameters from position s l Resulting in isparity alues as inicate by the otte line s r although also isible in both iews is not chosen, because its plane woul result in closer isparity alues to raw normal samples more restrictiely an the normals of the first PatchMatch iteration are within the 05 raius [ n x n y ] 05 To allow propagation an refinement of the particles in the first iterations of the algorithm we perform a few iterations with θ = 0 We control the alues of θ uring the iterations using the smoothstep function Each iteration consists of one PatchMatch iteration followe by seeral inner iterations for smoothing using the weighte Huber-ROF moel s r 25 Runtime Our algorithm has been esigne to be execute on massiely parallel architectures Our PatchMatch sampling strategy is completely parallel in contrast to the original PatchMatch stereo algorithm Also the Huber-ROF subproblem can be sole ery efficiently on parallel architectures The runtime of our algorithm highly aries with the parameter settings an number of iterations For the highquality settings as use for the Milebury benchmark ealuation our algorithm has a runtime of about 2 minutes For the PatchMatch stereo algorithm the authors reporte a runtime of about 1 minute for an aerage Milebury pair [7] Different settings for our algorithm allow the estimation of isparity maps in a few secons Our current GPU implementation is completely unoptimize an seeral obious performance enhancements hae not been exploite yet 26 Metho Parameters In the following we assume that the alues of the stereo image channels are in the range [0, 1] The size of the patch consiere in the ata term is pixels centere aroun the pixel p For setting the α, τ col an τ gra parameters we mainly follow [7] an set them to {α, τ col, τ gra } = {005, 004, 001} The new parameters γ min, γ raius are set to 5 an 39 an r max to The parameters of the weighting function g are set to {ζ, η} = {3, 08} ɛ n an ɛ of the robust Huber norm were both set to 0001 The alue of θ σ n starts at 0 an goes up to 50 with an aitional offset of 5 for the weighte Huber-ROF smoothing of the normals For the intermeiate isparity maps we use a range from 0 to 1, therefore θ σ takes alues between 0 an 50 5 max again with an special offset of max max is the maximum allowe isparity alue For the computation of the ata term we set λ = 50 3 Ealuation For the ealuation of our algorithm we use the Milebury stereo benchmark [15, 1] Our results for the Milebury stereo benchmark were mae using constant parameters as escribe in the preious section The maximum allowe isparity was fixe to 60 an use for all four pairs This shows that our algorithm oes not necessarily nee to know the isparity range in aance Our Milebury benchmark results for the error threshol 05 are shown in table 1 At the time of writing we are currently ranke at position 1 out of about 145 algorithms for the sub-pixel error threshol 05 We achiee results comparable or better than the original PatchMatch stereo implementation [7] an the PMBP metho [6] that also has an explicit smoothing moel The final isparity maps an also the error maps for the 05 error threshol are shown in figure 5 For the error-threshol 1 our algorithm has rank 25 As mentione

6 Tsukuba Ag 1 Our metho 2 SubPixSearch 3 PMF 5 PMBP 10 PatchMatch Venus Tey Cones Rank nonocc all isc nonocc all isc nonocc all isc nonocc all isc Table 1: First three entries from the Milebury stereo benchmark [15] an aitionally the results from PMBP [6] an the original PatchMatch-Stereo [7] algorithm Our algorithm is currently ranke at position 1 out of about 145 algorithms for the error-threshol 05 Subscripts enote rankings in the table Figure 5: From left to right: one of the input images, groun-truth isparity map, our result an the isparity errors > 05 From top to bottom: Milebury stereo pairs [15] Tsukuba, Venus, Tey an Cones before, we i not perform the weighte meian filtering step of the original algorithm an we assume that this leas to the slightly worse results for the 1 threshol To empha- size the sub-pixel accuracy of our algorithm we also create point clous of some Milebury atasets that contain planar an cure surfaces as epicte in figure 6 The hea

7 an the groun-plane of the Art scene are well reprouce by the point clou Also the cure surface of the platform in the Baby1 scene is ery smooth an oes not exhibit staircasing or iscrete epth layer effects For the Phong shae point clous the normals estimate by our algorithm hae been use instea of estimating them using neighbouring ertices Vieos of the point clous can be foun the supplementary material In orer to show that our algorithm also works for more realistic ata we teste it using two rectifie an ownscale images from Strecha et al [18] The resulting point clou is shown in figure 7 Another point clou create from our isparity maps is shown in figure 1 Figure 7: A point clou create from a rectifie stereo pair Images proie by Strecha et al [18] 4 Conclusion We presente an new approach to combine the ranomize sampling of the PatchMatch algorithm with an explicit ariational smoothing metho that gies control of the isparity an normal graients Our ealuation shows that we achiee ery goo sub-pixel results in the Milebury benchmark that make our algorithm well suite for the generation of point clous or meshes In the future we woul like to exten our algorithm to multi-iew, which probably can be one using equation (3) The estimate normals are also maybe useful for epthmap merging an multiiew reconstruction Aitionally we woul like to optimize our current GPU OpenCL implementation towars real-time frame-rates Also a moifie ersion for the estimation of optical flow is alreay planne References Figure 6: Colore an Phong shae point clous of the Milebury atasets Art, Baby1, Cones an Cloth3 [16, 11] [1] Milebury stereo benchmark mileburyeu/stereo/ 5 [2] Portal stereo scene cz/ cechj/gcs/stereo-images/ 4 [3] J-F Aujol, G Gilboa, T Chan, an S Osher Structure-texture image ecomposition moeling, algorithms, an parameter selection Int J Comput Vision, 67(1): , [4] C Barnes, E Shechtman, A Finkelstein, an D Golman PatchMatch: a ranomize corresponence algorithm for structural image eiting ACM Transactions on Graphics (TOG), 28(3):24, [5] C Barnes, E Shechtman, D Golman, an A Finkelstein The generalize patchmatch corresponence al-

8 gorithm Computer Vision ECCV 2010, pages 29 43, [6] F Besse, C Rother, A Fitzgibbon, an J Kautz PMBP: PatchMatch Belief Propagation for Corresponence Fiel Estimation In Proceeings of the British Machine Vision Conference, pages BMVA Press, , 5, 6 [7] M Bleyer, C Rhemann, an C Rother PatchMatch Stereo - Stereo Matching with Slante Support Winows Proc BMVC, pages 1 11, July , 2, 4, 5, 6 [8] A Chambolle an T Pock A first-orer primal-ual algorithm for conex problems with applications to imaging Journal of Mathematical Imaging an Vision, 40(1): , , 4 [9] A Hana, R A Newcombe, A Angeli, an A J Daison Applications of legenre-fenchel transformation to computer ision problems Technical Report DTR11-7, Imperial College - Department of Computing, September [10] R Hartley an A Zisserman Multiple View Geometry in Computer Vision Cambrige Uniersity Press, ISBN: , secon eition, [11] H Hirschmuller an D Scharstein Ealuation of Cost Functions for Stereo Matching In Computer Vision an Pattern Recognition, 2007 CVPR 07 IEEE Conference on, pages 1 8, [12] Y Huang, M K Ng, an Y-W Wen A Fast Total Variation Minimization Metho for Image Restoration Multiscale Moeling & Simulation, 7(2): , Jan , 3 [13] P Monasse Quasi-Eucliean Epipolar Rectification Image Processing On Line, 2011, [14] R A Newcombe, S J Loegroe, an A J Daison DTAM: Dense Tracking an Mapping in Real- Time ICCV 11: Proceeings of the 2011 International Conference on Computer Vision, pages 1 8, Aug , 3, 4 [15] D Scharstein an R Szeliski A Taxonomy an Ealuation of Dense Two-Frame Stereo Corresponence Algorithms Int J Comput Vision, 47(1-3):7 42, , 5, 6 [16] D Scharstein an R Szeliski High-accuracy stereo epth maps using structure light In Computer Vision an Pattern Recognition, 2003 Proceeings 2003 IEEE Computer Society Conference on, [17] F Steinbrücker, T Pock, an D Cremers Large isplacement optical flow computation without warping Computer Vision, 2009 IEEE 12th International Conference on, pages , , 3, 4 [18] C Strecha, R Fransens, an L Van Gool Combine epth an outlier estimation in multi-iew stereo Computer Vision an Pattern Recognition, 2006 IEEE Computer Society Conference on, 2: ,