Stereo Depth Continuity

Size: px

Start display at page:

Download "Stereo Depth Continuity"

Horace Baker
5 years ago
Views:

1 Stereo Depth Contnuty Steven Damond Jessca Taylor March 17, Abstract We tackle the problem of producng depth maps from stereo vdeo. Some algorthms for dong ths usng correspondences produce nosy output. Chan et al. (011) [] use convex optmzaton technques to reduce nose and also smooth out the depth maps over tme. We start wth the nosy output of OpenCV and apply ths technque to smooth t, developng some refnements along the way. Specfcally, we gnore pxels wth the hghest possble depth value (as ths ndcates that no correspondence was found for the pxel), and we make the smoothng less aggressve f the pxels dffer by color. These refnements mprove the smoothng to yeld more accurate depth estmates. We test the algorthm on multple vdeos and fnd that t generally performs well as long as OpenCV s output was somewhat reasonable. Revew of prevous work Gven stereo mages, t s possble to retreve depth maps by fndng correspondences between the left and rght mages. Ths technque s used by OpenCV ( trunk/doc/py_tutorals/py_calb3d/py_depthmap/py_depthmap.html). We ran OpenCV s StereoBM algorthm wth a SADWndowSze parameter of 5, whch ncreases accuracy by matchng smaller regons but ncreases nose. It s straghtforward to fnd depth maps for stereo vdeo by smply fndng maps for each frame ndependently. However, n addton to nhertng all the problems of the orgnal algorthm, the result s usually choppy, lackng smooth changes over tme. Chan et al. (011) [] apply a smoothng algorthm to sttch together frame-by-frame depth maps, whch succeeds n both reducng the nose of each depth map and ncreasng contnuty over tme. They solve the followng optmzaton problem: mnmze µ f,j,k g,j,k +,j,k,j,k β x (f +1,j,k f,j,k ) β y (f,j+1,k f,j,k ) β t (f,j,k+1 f,j,k ) where g s a 3D matrx contanng depth estmates from a frame-by-frame algorthm, f s a 3D matrx of smoothed depth estmates, and µ, β x, β y, and β t are non-negatve parameters. 1

2 Steven Damond, Jessca Taylor 4 SUMMARY OF TECHNICAL SOLUTION 3 Improvements on Prevous Work We found that OpenCV produced very nosy output. Addtonally, many pxels (n some cases the majorty) were labeled as the furthest possble depth value. Usually, these pxels dd not actually have such hgh depths; nstead, they were scattered around the mage at many dfferent depth values. It s lkely that OpenCV found no correspondences for these pxels, and gave them the hghest depth value as a default. Therefore, we adjusted the smoothng algorthm to gnore these pxels depth values. Ths way, the algorthm s free to vary the smoothed depth estmate correspondng to these pxels n order to produce more contnuty. Addtonally, we changed the smoothng to place less emphass on smoothness for pxels wth smlar colors. Ths captures the ntuton that depth changes should happen where color 1 changes happen. To do ths, we created a weght for each pxel proportonal to 1+, where s a vector combnng color dfferences n the x, y, and tme drectons. As explaned later, Chan et al. [] nclude smoothness constrants between pxels at opposte ends of the mage, or between the fnal and frst frame, just as there are smoothness constrants between neghborng pxels or frames. Includng these constrants s necessary to make optmzaton much more effcent. We do not drop these constrants, but we reduce the weght of pxels on edges that correspond to these wrapped constrants (to 1/3 of the orgnal weght) to make these constrants less cumbersome. 4 Summary of Techncal Soluton To obtan a smoothed depth map for the vdeo, we solve a convex optmzaton problem. We regard the vdeo as a tme space volume, where each pxel has an x,y, and t coordnate. The x axs corresponds to the pxel column, the y axs to the pxel row, and the t axs to the pxel frame. Let g be a 3D matrx of pxel depth estmates ndexed by x,y, and t. g s obtaned by estmatng a depth map for each frame of the vdeo ndependently. Some pxels may not be assgned a depth by the frame-by-frame algorthm. We call such pxels unlabeled. Let E be the set of labeled pxels n g. Let f be a matrx varable representng the smoothed pxel depth estmates. Then our optmzaton problem s gven by mnmze µ (,j,k) E f,j,k g,j,k + w,j,k,j,k β x (f +1,j,k f,j,k ) β y (f,j+1,k f,j,k ) β t (f,j,k+1 f,j,k ) w, µ, β x, β y, and β t are non-negatve parameters. The optmzaton problem fnds an f that s close to the orgnal depth estmates g but has fewer sharp changes n depth. Specfcally, the term (,j,k) E f,j,k g,j,k penalzes f s devaton from labeled pxels n g whle the term,j,k w,j,k β x (f +1,j,k f,j,k ) β y (f,j+1,k f,j,k ) β t (f,j,k+1 f,j,k )

3 Steven Damond, Jessca Taylor 5 DETAILS OF TECHNICAL SOLUTION penalzes large depth gradents n f. The balance between these terms s controlled by the parameter µ. The parameter w s a matrx wth a non-negatve weght for each pxel. The weght w,j,k determnes how heavly the gradent at f,j,k s penalzed. To choose w we followed the prncple that neghborng pxels wth smlar colors or ntenstes are lkely to belong to the same surface and thus be at smlar depths. Conversely, neghborng pxels wth very dfferent colors or ntenstes lkely belong to dfferent objects and thus could have very dfferent depths. β x, β y, and β t are global weghts appled to the component of the gradent n x,y, and t dmensons, respectvely. The β parameters allow changes n depth n certan dmensons to be weghted more (or less) heavly. Emprcally, settng β t less than β x and β y mproves performance. Ths makes sense gven that movng a sngle frame typcally makes more of a dfference to depth than movng a sngle pxel. 5 Detals of Techncal Soluton Our optmzaton problem s extremely hgh dmensonal, snce a vdeo contans mllons of pxels and we estmate a depth for every pxel. There s no generc solver for convex optmzaton problems capable of solvng our problem n a reasonable amount of tme. Hence, we mplemented a custom solver for our problem based on the work by Chan et al. (011) []. Lke Chan et al., our solver uses an teratve algorthm where each teraton runs n O(n log n). Here n s the total number of pxels n the vdeo. Ths runtme s an mportant dstncton of our algorthm, snce a generc convex optmzaton algorthm would run n O(n ) or O(n 3 ) []. We obtan an O(n log n) runtme per teraton by explotng specal structure n our problem. 5.1 The ADMM Algorthm Our algorthm uses the Alternatng Drecton Method of Multplers (ADMM). ADMM s an teratve method desgned for convex optmzaton problems too large for standard nteror pont methods [1]. ADMM solves problems of the form mnmze f(x) + g(z) subject to Ax + Bz = c where f, g are convex functons. ADMM operates on the so-called augmented Lagrangan L ρ (x, z, y) = f(x) + g(x) + y T (Ax + Bz c) + (ρ/) Ax + Bz c 3

4 Steven Damond, Jessca Taylor 5 DETAILS OF TECHNICAL SOLUTION where y s the dual varable. ADMM conssts of the followng algorthm: Data: A,B,c Result: Prmal optmal x, z and dual optmal y Intalze x (0), z (0), y (0) ; whle The stoppng condtons are not met do x (k+1) = arg mn x L ρ (x, z (k), y (k) ); z (k+1) = arg mn z L ρ (x (k+1), z, y (k) ); y (k+1) = y (k) + ρ(ax (k+1) + Bz (k+1) c); end The ntalzaton depends on the problem. ADMM wll converge faster f t s ntalzed near the soluton. The stoppng condton s based on the L norms of the prmal resdual and dual resdual r (k+1) = Ax (k+1) + Bz (k+1) c s (k+1) = ρa T B(z (k+1) z (k) ) When r (k+1) and s (k+1) are both small t ndcates that ADMM s near the correct soluton. Boyd et al. [1] recommend usng both a relatve and absolute metrc for measurng convergence. Specfcally, they suggest termnatng when r (k+1) ɛ pr and s (k+1) ɛ dual where ɛ pr = pɛ abs + ɛ rel max { Ax (k), Bz (k), c } ɛ dual = nɛ abs + ɛ rel A T y (k) p s the number of rows n A and n s the number of columns. ɛ abs and ɛ rel are chosen based on the desred accuracy and the problem scale. 5. Applyng ADMM To use ADMM we must reformulate our problem to match the ADMM standard form. Let f R C R F. Denote N = CRF. We defne new varables r R C R F, u (x), u (y), u (t) R N. Let vec(f) denote the vectorzed verson of f. Specfcally, to vectorze f we flatten each frame n row-major order and concatenate the frame vectors. We now defne matrces D x, D y, D t R N N. D x, D y, D t are forward shft operators n the x, y, and t dmensons respectvely. [D x vec(f)] s the dfference between the depth at the pxel one space forward n x from pxel and the depth at pxel. [D y vec(f)] and [D t vec(f)] are defned smlarly. Formally, f pxel occurs at locaton x, y, t: [D x vec(f)] = f(x + 1 mod C, y, t) f(x, y, z) [D y vec(f)] = f(x, y + 1 mod R, t) f(x, y, z) [D t vec(f)] = f(x, y, t + 1 mod F ) f(x, y, z) 4

5 Steven Damond, Jessca Taylor 5 DETAILS OF TECHNICAL SOLUTION We use perodc boundary condtons, meanng that for pxels on the boundares of f we look at the depth dfference between that pxel and the next pxel n the flattened verson of f. For example, f x = C 1 we look at the dfference f(x + 1 mod C, y, t) f(x, y, z) = f(0, y + 1, t) f(x, y, z) These boundary condtons are not natural, but they are necessary for the matrces D x, D y, D t to be block-crculant. A matrx s defned as block-crculant f every kth column of the matrx s the frst column rotated k 1 steps forward. Vsually the matrx has the form below c 0 c n 1... c 1 c 1 c 0... c c c 1... c c n 1 c n... c 0 Let c be the frst column of a block crculant matrx C. Multplyng a vector x by C s equvalent to a crcular convoluton of c wth x [3]: Cx = c x In the Fourer doman, crcular convoluton s equvalent to elementwse multplcaton. Thus applyng the dscrete fourer transform operator F, F(c x) = F(c) F(x) Remark 5.1. We can thus use the DFT to solve lnear equatons Cx = b where C s a blockcrculant matrx n tme O(n log n). Observe, F(Cx) = F(b) F(c) F(x) = F(b) ( ) F(b) x = F 1 F(c) Returnng to our formulaton of the optmzaton problem, we make two more defntons to smplfy notaton u (x) D x u = u (y), D = D y u (t) D t We can now express our optmzaton problem n ADMM form: mnmze µ (,j,k) E subject to r = f g u = Dvec(f) r,j,k + w u (x) u (y) u (t) 5

6 Steven Damond, Jessca Taylor 5 DETAILS OF TECHNICAL SOLUTION The augmented Lagrangan s L ρ (f, u, r, y, z) = µ (,j,k) E r,j,k + w u (x) u (y) u (t) z T vec(r f + g) y (x) + (ρ/) vec(r f + g) yt (u Dvec(f)) + (ρ/) u Dvec(f) where y = y (y) R 3N, z R N are dual varables. y (t) 5.3 Computng ADMM Iteratons We now explan how we compute each teraton of ADMM. The structure of our problem allows us to compute each step of the teraton n at most tme O(n log n) where n s the number of pxels n the vdeo. Wthout takng advantage of structure, an teraton would take tme O(n ), whch s far less feasble gven that we have mllons of pxels. For our problem the ADMM algorthm takes the followng form: Data: g Result: Prmal optmal f, u, r and dual optmal y, z Intalze f (0) = g, u (0) = Dvec(g), r (0) = z (0) = y (0) = 0; whle The stoppng condtons are not met do u (k+1) = arg mn u L ρ (f (k), u, r (k), y (k), z (k) ); r (k+1) = arg mn r L ρ (f (k), u (k+1), r, y (k), z (k) ); f (k+1) = arg mn f L ρ (f, u (k+1), r (k+1), y (k), z (k) ); y (k+1) = y (k) ρ(u (k+1) Dvec(f (k+1) )); z (k+1) = z (k) ρvec(r (k+1) f (k+1) + g); end Intalzng f (0) = g, u (0) = Dvec(f (0) ), r (0) = f (0) g = 0 s useful because startng ADMM near ts eventual soluton decreases the number of teratons needed. There s no obvous startng estmate for the dual varables, so we ntalze them to 0. We next explan how the u,r, and f updates are computed u update Recall the u update s the soluton to the problem u (x) u = arg mn w u (y) u y T (u Dvec(f)) + (ρ/) u Dvec(f) u (t) The u update can be expressed n closed form usng the shrnkage formula defned by L [4]. We defne v (x) = β x D x vec(f) + 1 ρ y(x) 6

7 Steven Damond, Jessca Taylor 5 DETAILS OF TECHNICAL SOLUTION wth v (y), v (t) defned analogously. We further defne v R N such that v = (v (x) ) + (v (y) ) + (v (t) ) for all. Then for each pxel wth gradent weght w the updated value of u (x) s gven by u (x) The updates for u (y), u (t) are analogous r update { = max v w } ρ, 0 v(x) v Recall the r update s the soluton to the problem r = arg mn µ r,j,k z T vec(r f + g) + (ρ/) vec(r f + g) r (,j,k) E For a pxel (, j, k) / E, the value r,j,k does not change. For a pxel (, j, k) E, a smlar shrnkage formula gves a closed form expresson for the r update. We use l to ndcate the ndex of (, j, k) n the vectorzed mage. { r,j,k = max f,j,k g,j,k + 1 ρ z l µ } ( ρ, 0 sgn f,j,k g,j,k + 1 ) ρ z l f update Recall the f update s the soluton to the problem f = arg mn(ρ/) vec(r f + g) + (ρ/) u Dvec(f) + yt Dvec(f) + z T vec(f) f The optmal f satsfes the normal equatons ρ(i + D T D)vec(f) = ρg + (ρr z) + D T (ρu y) ρ(i + D T x D x + D T y D y + D T t D t )vec(f) = ρg + (ρr z) + D T (ρu y) Recall that D x, D y, D t are block-crculant matrces. Ths means Dx T, Dy T, Dt T are also block crculant. The dentty matrx I s block crculant as well. Thus by Remark 5.1 we can solve the normal equatons n tme O(n log n) usng the DFT operator F and the FFT algorthm. Denote M = ρ(i + Dx T D x + Dy T D y + Dt T D t ). Then applyng F to Mvec(f) yelds F(Mvec(f)) = ρ(f(i) + F(D T x ) F(D x ) + F(D T y ) F(D y ) + F(D T t ) F(D t )) F(vec(f)) = F(M) F(vec(f)) 7

Steven Damond, Jessca Taylor 6 EXPERIMENTS Here F(I), F(D T x ), F(D x ), etc., mean the DFT operator appled to the frst column of each matrx.

4 Convergence Mvec(f) = ρg + (ρr z) + D T (ρu y) F(Mvec(f)) = F(ρg + (ρr z) + D T (ρu y)) F(M) F(vec(f)) = F(ρg + (ρr z) + D T (ρu y)) vec(f) = F 1 ( F(ρg + (ρr z) + D T (ρu y)) F(M) We use the

Our stoppng crtera results n more ADMM teratons, but t corresponds much better to the accuracy of the ADMM soluton.

8 Steven Damond, Jessca Taylor 6 EXPERIMENTS Here F(I), F(D T x ), F(D x ), etc., mean the DFT operator appled to the frst column of each matrx. F(M) need only be computed once, before the ADMM loop begns. We use F(M) to solve the normal equatons Convergence Mvec(f) = ρg + (ρr z) + D T (ρu y) F(Mvec(f)) = F(ρg + (ρr z) + D T (ρu y)) F(M) F(vec(f)) = F(ρg + (ρr z) + D T (ρu y)) vec(f) = F 1 ( F(ρg + (ρr z) + D T (ρu y)) F(M) We use the stoppng crtera descrbed n Secton 5.1 wth ɛ abs = 1e 4 and ɛ rel = 1e 4. Ths s dfferent from the crtera used by Chan et al. []. Our stoppng crtera results n more ADMM teratons, but t corresponds much better to the accuracy of the ADMM soluton. We were stll able to get convergence n relatvely few teratons by scalng g so all entres were n [0, 1]. We used a consstent value of ρ = 1 rather than updatng ρ each teraton as done by Chan et al. []. We found that updatng ρ prevented ADMM from convergng. ) 6 Experments We tested ths algorthm on a vdeo also used n [], specfcally the Horse vdeo found at We scaled the vdeo down to be 180 by 10 pxels and used all 140 frames. We optmzed over the 3.09 mllon varables n 980 seconds, or 7 seconds per frame. As there s no ground truth provded for ths dataset, qualtatve comparson s necessary. We compare depth maps of a sngle frame of the vdeo (orgnal, OpenCV, and our algorthm): The smoothed depth map s much less nosy than the orgnal depth map, contanng clearly defned regons of smlar depth. Black pxels n the orgnal depth map (representng unlabeled pxels wth no depth estmate) have been smoothed over wth neghborng depth values, mprovng accuracy. Although t s dffcult to show ths n paper form, the depth maps are choppy when they are not smoothed over tme (only smoothed over pxel locatons). Smoothng over tme reduces choppness wthout sacrfcng accuracy. 8

Steven Damond, Jessca Taylor 7 CONCLUSION We also tred some alternatve datasets. Specfcally, we used stereo vdeos found at http:// hc.wr.un-hedelberg.

The algorthm performed well on some of these vdeos but not others, mostly n lne wth how OpenCV s algorthm dd on these vdeos.

When an occluded car becomes vsble agan, t mantans roughly the same depth that t had orgnally. Addtonally, we tred usng the Indoor 1 dataset found at http://research.mcrosoft.

9 Steven Damond, Jessca Taylor 7 CONCLUSION We also tred some alternatve datasets. Specfcally, we used stereo vdeos found at hc.wr.un-hedelberg.de/benchmarks/document/challengng_data_for_stereo_and_optcal_ Flow/, whch do not contan ground truth. The algorthm performed well on some of these vdeos but not others, mostly n lne wth how OpenCV s algorthm dd on these vdeos. It performed relatvely well on the CrossngCars vdeo: Our algorthm succeeded n fllng n black regons and handlng occlusons over tme. When an occluded car becomes vsble agan, t mantans roughly the same depth that t had orgnally. Addtonally, we tred usng the Indoor 1 dataset found at com/en-us/downloads/9d bce1/ because ths dataset contaned ground truth. Unfortunately, none of the frame-by-frame algorthms we tested produced reasonable results on ths dataset. We tred algorthms other than OpenCV s StereoBM, but none was very accurate for the datasets we used. These algorthms were OpenCV s StereoSGBM, the MRF mnmzaton-based algorthm found at and the Mcrosoft Research StereoMatcher package at (wth a patch from edu/stereo/code/). None of these algorthms produced reasonable depth maps for our datasets. For a more quanttatve evaluaton, we downloaded depth maps from top-ranked algorthms at and ran our algorthm on them. We counted bad pxels n the same way the webste does (proporton of non-occluded pxels that are more than 4 ntensty unts away from the ground truth value) on the teddy mage. Algorthm Orgnal Bad Pxels New Bad Pxels Percent Change TSGO % % % AdaptngBP % % % DoubleBP % % % ADCensus % % % CoopRegon % % % In general, smoothng made very lttle dfference to the accuracy of the depth maps. Ths s most lkely because the output of the top algorthms s already relatvely smooth. 7 Concluson Our mprovements to the smoothng algorthm n [] make t approprate for use wth OpenCV s depth maps. The algorthm succeeds n smoothng depth maps over space and tme as long 9

10 Steven Damond, Jessca Taylor REFERENCES as the orgnal depth maps are reasonable. Our extensons to the algorthm handle unlabeled pxels and mprove performance around edges. We mplement the ADMM algorthm, whch greatly mproves performance to O(n log n) per teraton, allowng entre vdeos to be smoothed. Due to ts qualty and reasonable performance, our algorthm s approprate for practcal use n smoothng nosy but relatvely accurate depth maps. In partcular, our algorthm works well when only a small proporton of pxels (such as only pxels near edges) ntally have depth estmates and others are unlabeled. References [1] S. Boyd, N. Parkh, E. Chu, B. Peleato, and J. Ecksten. Dstrbuted optmzaton and statstcal learnng va the alternatng drecton method of multplers. Foundatons and Trends R n Machne Learnng, 3(1):1 1, 011. [] S. Chan, R. Khoshabeh, K. Gbson, P. Gll, and T. Nguyen. An augmented lagrangan method for total varaton vdeo restoraton. Image Processng, IEEE Transactons on, 0(11): , Nov 011. [3] G. H. Golub and C. F. Van Loan. Matrx computatons, volume 3. JHU Press, 01. [4] C. L. An effcent algorthm for total varaton regularzaton wth applcatons to the sngle pxel camera and compressve sensng. PhD thess, Cteseer,

Wavefront Reconstructor

A Dstrbuted Smplex B-Splne Based Wavefront Reconstructor Coen de Vsser and Mchel Verhaegen 14-12-201212 2012 Delft Unversty of Technology Contents Introducton Wavefront reconstructon usng Smplex B-Splnes