Today. Stereo: Correspondence and Calibration. Last time: Estimating depth with stereo. Last time: Epipolar geometry. Last time: Epipolar constraint

Transcription

1 Today Steeo: Coesondence and Calibation Mon, Mach 28 Pof. Kisten Gauman UT-Austin Reca: eiola consta Steeo age ectification Steeo solutions Comuting coesondences Non-geometic steeo constas Calibation Examle steeo alications Last te: Estating deth ith steeo Steeo: shae fom motion beteen to vies We need to conside: Info on camea ose ( calibation ) Image o coesondences Last te: Eiola geomety Eiola Plane Eiola Line scene o otical cente age lane Eiole Baseline Eiole Last te: Eiola consta Examle: conveging cameas Potential matches fo have to lie on the coesonding eiola line l. Potential matches fo have to lie on the coesonding eiola line l. Slide cedit: M. Pollefeys Figue fom Hatley & Zisseman CS 376 Lectue 17 Steeo 1

2 Examle: aallel cameas An audio camea & eiola geomety Sheical micohone aay Figue fom Hatley & Zisseman Adam O' Donovan, Ramani Duaisami and Jan Neumann Micohone Aays as Genealized Cameas fo Integated Audio Visual Pocessing, IEEE Confeence on Comute Vision and Patten Recognition (CVPR), Minneaolis, 27 An audio camea & eiola geomety X X Last te: Essential matix T RX [T ] RX x Let E [T x] R X T EX E is called the essential matix, and it elates coesonding os beteen both cameas, given the otation and tanslation. If e obseve a o in one age, its osition in othe age is constained to lie on line defined by above. Note: these os ae in camea coodinate systems. Essential matix examle: aallel cameas R I T [ d,,] E [ T x ]R d d [ x, y, f ] ' [ x', y', f ] age I(x,y) Disaity ma D(x,y) age I (x,y ) E (x,y )=(x+d(x,y),y) Fo the aallel cameas, age of any o must lie on same hoizontal line in each age lane. What about hen cameas otical axes ae not aallel? CS 376 Lectue 17 Steeo 2

3 Today Reca: eiola consta Steeo age ectification Steeo solutions Comuting coesondences Non-geometic steeo constas Calibation Examle steeo alications Steeo age ectification In actice, it is convenient if age scanlines (os) ae the eiola lines. eoject age lanes onto a common lane aallel to the line beteen otical centes ixel motion is hoizontal afte this tansfomation to homogahies (3x3 tansfoms), one fo each inut age eojection Slide cedit: Li Zhang Steeo age ectification: examle Today Reca: eiola consta Steeo age ectification Steeo solutions Comuting coesondences Non-geometic steeo constas Calibation Examle steeo alications Souce: Alyosha Efos Coesondence oblem Multile match hyotheses satisfy eiola consta, but hich is coect? Coesondence oblem Beyond the had consta of eiola geomety, thee ae soft constas to hel identify coesonding os Silaity Uniqueness Odeing Disaity gadient To find matches in the age ai, e ill assume Most scene os visible fom both vies Image egions fo the matches ae sila in aeaance Figue fom Gee & Ciolla 1999 CS 376 Lectue 17 Steeo 3

4 Dense coesondence seach Coesondence oblem Fo each eiola line Fo each ixel / indo in the age comae ith evey ixel / indo on same eiola line in ight age ick osition ith minum match cost (e.g., SSD, coelation) Paallel camea examle: eiola lines ae coesonding age scanlines Adated fom Li Zhang Souce: Ande Zisseman Coesondence oblem Coesondence oblem Intensity ofiles Neighbohoods of coesonding os ae sila in ensity attens. Souce: Ande Zisseman Souce: Ande Zisseman Coelation based indo matching Textueless egions Souce: Ande Zisseman Textueless egions ae non distinct; high ambiguity fo matches. Souce: Ande Zisseman CS 376 Lectue 17 Steeo 4

5 Effect of indo size Effect of indo size W = 3 W = 2 Want indo lage enough to have sufficient ensity vaiation, yet small enough to contain only ixels ith about the same disaity. Souce: Ande Zisseman Figues fom Li Zhang Foeshotening effects Occlusion Souce: Ande Zisseman Slide cedit: David Kiegman Sase coesondence seach Coesondence oblem Beyond the had consta of eiola geomety, thee ae soft constas to hel identify coesonding os Silaity Uniqueness Disaity gadient Odeing Restict seach to sase set of detected featues (e.g., cones) Rathe than ixel values (o lists of ixel values) use featue descito and an associated featue distance Still nao seach futhe by eiola geomety Tadeoffs beteen dense and sase seach? CS 376 Lectue 17 Steeo 5

6 Uniqueness consta U to one match in ight age fo evey o in age Disaity gadient consta Assume ieceise continuous suface, so ant disaity estates to be locally smooth Figue fom Gee & Ciolla 1999 Figue fom Gee & Ciolla 1999 Odeing consta Pos on same suface (oaque object) ill be in same ode in both vies Odeing consta Won t alays hold, e.g. conside tansaent object, o an occluding suface Figue fom Gee & Ciolla 1999 Figues fom Fosyth & Ponce Beyond individual coesondences to estate disaities: Otize coesondence assignments joly Scanline at a te (DP) Full 2D gid (gah cuts) Scanline steeo Ty to coheently match ixels on the entie scanline Diffeent scanlines ae still otized indeendently Left age Right age ensity CS 376 Lectue 17 Steeo 6

7 Shotest aths fo scan-line steeo Left age I Right age I Coheent steeo on 2D gid Scanline steeo geneates steaking atifacts S q t Left occlusion Right occlusion s S ight Can be lemented ith dynamic ogamming Ohta & Kanade 85, Cox et al. 96 Slide cedit: Y. Boykov Can t use dynamic ogamming to find satially coheent disaities/ coesondences on a 2D gid E Steeo matching as enegy minization I 1 I 2 D W 1 (i) W 2 (i+d(i)) D(i) E Edata ( I1, I 2, D) Esmooth ( D) W 2 1( i) W2 ( i D( i data )) i Enegy functions of this fom can be minized using gah cuts Y. Boykov, O. Veksle, and R. Zabih, Fast Aoxate Enegy Minization via Gah Cuts, PAMI 21 Esmooth D( i) D( j) neighbos i, j Souce: Steve Seitz Reca: steeo ith calibated cameas Given age ai, R, T Detect some featues Comute essential matix E Match featues using the eiola and othe constas Tiangulate fo 3d stuctue Eo souces Lo-contast ; textueless age egions Occlusions Camea calibation eos Violations of bightness constancy (e.g., secula eflections) Lage motions Today Reca: eiola consta Steeo age ectification Steeo solutions Comuting coesondences Non-geometic steeo constas Calibation Examle steeo alications CS 376 Lectue 17 Steeo 7

8 Steeo in machine vision systems Examle deth mas (entagon) Left : The Stanfod cat sots a single camea moving in discete incements along a staight line and oviding multile snashots of outdoo scenes Right : The INRIA mobile obot uses thee cameas to ma its envionment Fosyth & Ponce Deth fo segmentation Deth fo segmentation Edges in disaity in conjunction ith age edges enhances contous found Danijela Makovic and Magit Gelautz, Inteactive Media Systems Gou, Vienna Univesity of Technology Danijela Makovic and Magit Gelautz, Inteactive Media Systems Gou, Vienna Univesity of Technology Model-based body tacking, steeo inut Vitual vieo video David Demidjian, MIT Vision Inteface Gou htt://eole.csail.mit.edu/demidji/movie/atic-tacke/tun-aound.m1v C. Zitnick et al, High-quality video vie eolation using a layeed eesentation, SIGGRAPH 24. CS 376 Lectue 17 Steeo 8

9 Vitual vieo video Uncalibated case What if e don t kno the camea aametes? To ossibilities: 1. Calibate ith a calibation object 2. Weak calibation htt://eseach.micosoft.com/ivm/vvv/ Calibating a camea Pesective ojection Comute insic and extinsic aametes using obseved camea data Image lane Focal length Main idea Place calibation object ith knon geomety in the scene Get coesondences Solve fo maing fom scene to age Camea fame Otical axis Scene o Image coodinates Thus fa, in camea s efeence fame only. Camea aametes Extinsic camea aametes Extinsic: location and oientation of camea fame ith esect to efeence fame Intinsic: ho to ma ixel coodinates to age lane coodinates P c R( P T) Refeence fame Camea efeence fame Wold efeence fame Camea 1 fame P c X, Y, Z T CS 376 Lectue 17 Steeo 9

10 Camea aametes Extinsic: location and oientation of camea fame ith esect to efeence fame Intinsic: ho to ma ixel coodinates to age lane coodinates Refeence fame Intinsic camea aametes Ignoing any geometic distotions fom otics, e can descibe them by: x ( x o ) s y ( y o ) s x y x y Camea 1 fame Coodinates of ojected o in camea efeence fame Coodinates of age o in ixel units Coodinates of age cente in ixel units Effective size of a ixel (mm) Camea aametes We kno that in tems of camea efeence fame: c and P T c X, Y, Z Substituting evious eqns descibing insic and extinsic aametes, can elate ixels coodinates to old os: ( x ( y o ) s x x o ) s y y R1( P T) f R ( P T) 3 R 2( P f R ( P 3 P R( P T) T) T) R i = Ro i of otation matix This can be eitten as a matix oduct using homogeneous coodinates: hee: f / sx M f / sy M ext Pojection matix ox o y x y X x Y y M MP ext Z 1 R R R T T T M Calibating a camea Comute insic and extinsic aametes using obseved camea data Main idea Place calibation object ith knon geomety in the scene Get coesondences Solve fo maing fom scene to age: estate M=M M ext When ould e calibate this ay? Makes sense hen geomety of system is not going to change ove te hen ould it change? CS 376 Lectue 17 Steeo 1

11 Weak calibation Want to estate old geomety ithout equiing calibated cameas Achival videos Photos fom multile unelated uses Dynamic camea system Main idea: Estate eiola geomety fom a (edundant) set of o coesondences beteen to uncalibated cameas Fom befoe: Pojection matix This can be eitten as a matix oduct using homogeneous coodinates: hee: f / sx M f / sy M ext ox o y x y M R R R T T T M ext X Y Z 1 Fom befoe: Pojection matix This can be eitten as a matix oduct using homogeneous coodinates: x y M M M M M c ext ext c P X Y Z 1 Fo a given camea: Uncalibated case M So, fo to cameas ( and ight): 1 c, M,, 1 c, ight M,ight, ight c Intenal calibation matices, one e camea 1 c, M,, 1 c, ight M,ight, ight, ight c, c E Uncalibated case Fom befoe, the essential matix E. 1 1 M,ight, ight EM,, 1 M EM, ight,ight,, F F, ight, Fundamental matix Comuting F fom coesondences Each o coesondence geneates one consta on F Collect n of these constas F, ight, Solve fo f, vecto of aametes. CS 376 Lectue 17 Steeo 11

12 Fundamental matix Relates ixel coodinates in the to vies Moe geneal fom than essential matix: e emove need to kno insic aametes Steeo ieline ith eak calibation So, hee to stat ith uncalibated cameas? Need to find fundamental matix F and the coesondences (ais of os (u,v ) (u,v)). If e estate fundamental matix fom coesondences in ixel coodinates, can econstuct eiola geomety ithout insic o extinsic aametes. 1) Find eest os in age 2) Comute coesondences 3) Comute eiola geomety 4) Refine Examle fom Ande Zisseman Steeo ieline ith eak calibation 1) Find eest os Steeo ieline ith eak calibation 2) Match os ithin oxity to get utative matches Steeo ieline ith eak calibation 3) Comute eiola geomety -- obustly ith RANSAC Select andom samle of utative coesondences Comute F using them - detemines eiola consta Evaluate amount of suot - inlies ithin theshold distance of eiola line Choose F ith most suot (inlies) Using coelation seach to get utative matches: noisy, but enough to comute F using RANSAC Puned matches: those consistent ith eiola geomety CS 376 Lectue 17 Steeo 12

13 Summay Rectification: make eiola lines align ith scanlines Steeo solutions: Coesondence: dense, o at eest os Non-geometic steeo constas (e.g., silaity, ode, smoothness) Calibation With calibation object in scene: elate old coodinates to age coodinates Weak calibation: solve fo fundamental matix, elate age coodinates to age coodinates CS 376 Lectue 17 Steeo 13