Xbox Knnect: Stereo III Depth map http://www.youtube.com/watch?v=7qrnwoo-8a CSE5A Lecture 6 Projected pattern http://www.youtube.com/watch?v=ceep7x-z4wy The Fundamental matrx Rectfcaton The eppolar constrant s gven by: where p and p are -D coordnates of the mage coordnates of ponts n the two mages. Wthout calbraton, we can stll dentfy correspondng ponts n two mages, but we can t convert to -D coordnates. However, the relatonshp between the calbrated cordnates (p,p ) and uncalbrated coordnates (q,q ) can be expressed as p=aq, and p =A q Therefore, we can express the eppolar constrant as: (Aq) T E(A q ) = q T (A T EA )q = q T Fq = 0 where F s called the Fundamental Matrx. Image par rectfcaton Example: forward moton smplfy stereo matchng by warpng the mages e Apply projectve transformaton H so that eppolar lnes correspond to horzontal scanlnes H e e map eppole e to (,0,0) e try to mnmze mage dstorton Note that rectfed mages usually not rectangular courtesy of Andrew Computer Zsserman Vson I
Correspondence Search Algorthm Match Metrc Summary MATCH METRIC DEFINITION Normalzed Cross-Correlaton (NCC) For = :nrows for j=:ncols best(,j) = - for k = mndsparty:maxdsparty c = Match_Metrc(I (,j),i (,j+k),wnsze) f (c > best(,j)) best(,j) = c dspartes(,j) = k end end end end O(nrows * ncols * dspartes * wnx * wny) Sum of Squared Dfferences (SSD) Normalzed SSD Sum of Absolute Dfferences (SAD) Zero Mean SAD Rank Census These two are actually the same Some Issues Lghtng Condtons (Photometrc Varatons) Orderng Wndow sze Wndow shape Lghtng Ambguty Half occluded regons W(P l ) W(P r ) Ambguty Multple Interpretatons Each feature on left eppolar lne match one and only one feature on rght eppolar lne.
Wndow sze Wndow Shape and Forshortenng Effect of wndow sze W = W = 0 Better results wth adaptve wndow T. Kanade and M. Okutom, A Stereo Matchng Algorthm wth an Adaptve Wndow: Theory and Experment,, Proc. Internatonal Conference on Robotcs and Automaton, 99. D. Scharsten and R. Szelsk. Stereo matchng wth nonlnear dffuson. Internatonal Journal of Computer Vson, 8():55-74, July 998 (Setz) Problem of Occluson Stereo matchng Smlarty measure (SSD or NCC) Optmal path (dynamc programmng ) Constrants eppolar orderng unqueness dsparty lmt dsparty gradent lmt Trade-off Matchng cost (data) Dscontnutes (pror) (From Pollefeys) (Cox et al. CVGIP 96; Koch 96; Falkenhagen 97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV 0) Stereo Matchng wth Dynamc Start (Sldes adapted from Jm Rehg at GA Tech) C(,j) j End Dynamc programmng yelds the optmal path through grd. Ths s the best set of matches that satsfy the orderng constrant Every pxel on each scanlne wll be labeled as matchng, or occluded. CS5A, (Sldes adapted Fall 00 from Jm Rehg at GA Tech) Dynamc Effcent algorthm for solvng sequental decson (optmal path) problems. Cost assocated wth each arc. How many paths through ths trells? Usng Dynamc, can fnd optmal path n O(M T) tme (here M=)
Dynamc for Stereo Effcent algorthm for solvng sequental decson (optmal path) problems. States: Dynamc Used wth Hdden Markov Models, Vterb Algorthm Π =. Π =.0 Π = 6. For Stereo, t can denote pxel coordnates across an eppolar lne n one mage can denote the dsparty to the other eppolar lne CS5A, (Sldes adapted Fall 00 from Jm Rehg at GA Tech) Suppose cost can be decomposed nto stages: CS5A, (Sldes adapted Fall 00 from Jm Rehg at GA Tech) Dynamc Mnmum Cost Path Dynamc Used wth Hdden Markov Models, Vterb Algorthm C t- ()=7. States: C t- ()=5. Π = C t- ()=9.7 What s mnmum cost of reachng node j at tme t? C t ( j) = mnπ j + C t () What s mnmum cost of reachng node j at tme t? ( ) C t ( j) = mn Π j + C t () Mnmum cost of path from t=0 to reach state j at tme t. CS5A, (Sldes adapted Fall 00 from Jm Rehg at GA Tech) CS5A, (Sldes adapted Fall 00 from Jm Rehg at GA Tech) Dynamc Used wth Hdden Markov Models, Vterb Algorthm Dynamc C t- ()=7. States: C t- ()=5. Π = C t- ()=9.7 CS5A, (Sldes adapted Fall 00 from Jm Rehg at GA Tech) ( ) ( Π j + C t ()) C t ( j) = mn Π j + C t () So, b t () = b t ( j) = argmn b t (j) gves prevous state along mnmum cost path So, b t () = b t () = b t () = b t (j) gves prevous state along mnmum cost path CS5A, (Sldes adapted Fall 00 from Jm Rehg at GA Tech) 4
Dynamc Compute Optmal Path Costs ``A Maxmum Lkelhood Stereo Algorthm, Cox, Hngoran, Rao, Maggs, Computer Vson & Image Understandng, 6,, pp. 54-567. Mn cost path. Iteratvely, compute mnmum cost to reach all nodes. Recursvely, startng wth the node at tme t-max, select lowest cost termnal node, and backtrack along path C(,j): Cost of optmal path to match of pxels and j M(,j): Ponter to prevous node along optmal path Back trackng to get optmal path Stereo Matchng wth Dynamc C(,j) s mnmum of. C(-,j-) + match-cost of pxel L() & R(). C(-,j) +occlusonpenalty. C(,j-)+occlusonpenalty Stereo Matchng wth Dynamc Stereo Matchng wth Dynamc Scan across grd computng optmal cost for each node gven ts upper-left neghbors. Scan across grd computng optmal cost for each node gven ts upper-left neghbors. 5
Stereo Matchng wth Dynamc Stereo Matchng wth Dynamc Scan across grd computng optmal cost for each node gven ts upper-left neghbors. Scan across grd computng optmal cost for each node gven ts upper-left neghbors. Backtrack from the termnal to get the optmal path. Stereo Matchng wth Dynamc Once C(,j) s completely calculated: Backtrack from the termnal to get the optmal path. Some Challenges & Problems Photometrc ssues: speculartes strongly non-lambertan BRDF s Surface structure lack of texture repeatng texture wthn horopter bracket Geometrc ambgutes as surfaces turn away, dffcult to get accurate reconstructon (affne approxmate can help) at the occludng contour, lkelhood of good match but ncorrect reconstructon Varatons on Bnocular Stereo Trnocular Eppolar Constrants. Trnocular Stereopss. Helmholtz Recprocty Stereopss These constrants are not ndependent! 6
Helmholtz recprocty θn, φn ^ n Dsparty and Normal Feld θout, φout ^ n θout, φout θn, φn [Helmholtz, 90], [Mnnaert, 94], [ Ncodemus et al, 977] Helmholtz Stereopss Expermental Aparatus Bulldog: Dsparty Second Generaton Rg Bulldog: Normal Feld 7
Plastc Baby Doll: Normal Feld Plastc Baby Doll: Dspartes Surface after ntegratng normal feld Recprocal Images: Typcal Dataset SOURCE VIEW Recprocal Images: Typcal Dataset Recprocal Images: Typcal Dataset SOURCE Conventonal Stereo Constant brghtness No structure n textureless regons SOURCE Conventonal Stereo Constant brghtness No structure n textureless regons VIEW VIEW Photometrc Stereo Needs reflectance model No drect depth estmates 8
Recprocal Images: Typcal Dataset Metrc Reconstructon SOURCE Conventonal Stereo Constant brghtness No structure n textureless regons VIEW Photometrc Stereo Needs reflectance model No drect depth estmates Helmholtz Stereo No assumed reflectance Gves depth and surface normals More on stereo The Mddleburry Stereo Vson Research Page http://cat.mddlebury.edu/stereo/ Recommended readng" D. Scharsten and R. Szelsk. " A Taxonomy and Evaluaton of Dense Two-Frame Stereo Correspondence Algorthms. IJCV 47(//):7-4, Aprl-June 00. PDF fle (.5 MB) - ncludes current evaluaton. Mcrosoft Research Techncal Report MSR-TR-00-8, November 00." Myron Z. Brown, Darus Burschka, and Gregory D. Hager. Advances n Computatonal Stereo. IEEE Transactons on Pattern Analyss and Machne Intellgence, 5(8): 99-008, 00. 9