Epipolar Geometry CSE P576 Dr. Matthew Brown
Epipolar Geometry Epipolar Lines, Plane Constraint Fundamental Matrix, Linear solution + RANSAC Applications: Structure from Motion, Stereo [ Szeliski 11] 2
2-view Geometry How do we transfer points between 2 views? (u 1,v 1 ) (u 2,v 2 )? 3
X<latexit sha1_base64="nietpjtpxuzhajhprtgjay03bgk=">aaab7hicbvbns8naej3ur1q/qh69lbbbu0leqmeif48vtftoq9lsj+3szsbsboqs+hu8efdeqz/im//gbzudtj4yelw3w8y8mbvcg9f9dkobm1vbo+xdyt7+wefr9fikrznmmfrzihlvdalgwsx6hhub3vqhjuobnxbyn/c7t6g0t+sjmayyxhqkecqznvby+2feuonqza27c5b14hwkbgvag+pxf5iwlezpmkba9zw3nufolefm4kzszzsmle3ochuwshqjdvlfstnyyzuhirjlsxqyuh9p5dtwehqhtjomzqxxvbn4n9flthqt5fymmuhjlouitbctkpnnzmgvmiomllcmul2vsdfvlbmbt8wg4k2+ve7av3xprxsp17xmbrfhgc7ghc7bgwy04r5a4amdds/wcm+odf6cd+dj2vpyiplt+apn8wc4fi5i</latexit> <latexit sha1_base64="nietpjtpxuzhajhprtgjay03bgk=">aaab7hicbvbns8naej3ur1q/qh69lbbbu0leqmeif48vtftoq9lsj+3szsbsboqs+hu8efdeqz/im//gbzudtj4yelw3w8y8mbvcg9f9dkobm1vbo+xdyt7+wefr9fikrznmmfrzihlvdalgwsx6hhub3vqhjuobnxbyn/c7t6g0t+sjmayyxhqkecqznvby+2feuonqza27c5b14hwkbgvag+pxf5iwlezpmkba9zw3nufolefm4kzszzsmle3ochuwshqjdvlfstnyyzuhirjlsxqyuh9p5dtwehqhtjomzqxxvbn4n9flthqt5fymmuhjlouitbctkpnnzmgvmiomllcmul2vsdfvlbmbt8wg4k2+ve7av3xprxsp17xmbrfhgc7ghc7bgwy04r5a4amdds/wcm+odf6cd+dj2vpyiplt+apn8wc4fi5i</latexit> <latexit sha1_base64="nietpjtpxuzhajhprtgjay03bgk=">aaab7hicbvbns8naej3ur1q/qh69lbbbu0leqmeif48vtftoq9lsj+3szsbsboqs+hu8efdeqz/im//gbzudtj4yelw3w8y8mbvcg9f9dkobm1vbo+xdyt7+wefr9fikrznmmfrzihlvdalgwsx6hhub3vqhjuobnxbyn/c7t6g0t+sjmayyxhqkecqznvby+2feuonqza27c5b14hwkbgvag+pxf5iwlezpmkba9zw3nufolefm4kzszzsmle3ochuwshqjdvlfstnyyzuhirjlsxqyuh9p5dtwehqhtjomzqxxvbn4n9flthqt5fymmuhjlouitbctkpnnzmgvmiomllcmul2vsdfvlbmbt8wg4k2+ve7av3xprxsp17xmbrfhgc7ghc7bgwy04r5a4amdds/wcm+odf6cd+dj2vpyiplt+apn8wc4fi5i</latexit> <latexit sha1_base64="nietpjtpxuzhajhprtgjay03bgk=">aaab7hicbvbns8naej3ur1q/qh69lbbbu0leqmeif48vtftoq9lsj+3szsbsboqs+hu8efdeqz/im//gbzudtj4yelw3w8y8mbvcg9f9dkobm1vbo+xdyt7+wefr9fikrznmmfrzihlvdalgwsx6hhub3vqhjuobnxbyn/c7t6g0t+sjmayyxhqkecqznvby+2feuonqza27c5b14hwkbgvag+pxf5iwlezpmkba9zw3nufolefm4kzszzsmle3ochuwshqjdvlfstnyyzuhirjlsxqyuh9p5dtwehqhtjomzqxxvbn4n9flthqt5fymmuhjlouitbctkpnnzmgvmiomllcmul2vsdfvlbmbt8wg4k2+ve7av3xprxsp17xmbrfhgc7ghc7bgwy04r5a4amdds/wcm+odf6cd+dj2vpyiplt+apn8wc4fi5i</latexit> 2-view Geometry How do we transfer points between 2 views? (planar case) (u 1,v 1 ) (u 2,v 2 )? Planar case: one-to-one mapping via plane 4
2-view Geometry How do we transfer points between 2 views? (non-planar) X? X? X? (u 1,v 1 ) (u 2,v 2 )? Non-planar case: depends on the depth of the 3D point 5
Epipolar Line How do we transfer points between 2 views? (non-planar) X? X? X? (u 1,v 1 ) (u 2,v 2 )? A point in image 1 gives a line in image 2 6
Epipolar Lines [ R. Cipolla ] 7
Epipolar Lines [ R. Cipolla ] 8
The Epipolar Constraint For rays to intersect at a point (X), the two rays and the camera translation must lie in the same plane 3.10 9
Epipolar Geometry Example: 2-view matching in 3D 10
Raw SIFT matches Epipolar Geometry 11
Epipolar lines Epipolar Geometry? Can use RANSAC to find inliers with small distance from epipolar line 12
Consistent matches Epipolar Geometry 13
<latexit sha1_base64="3myafgfbz85tqxrlykm907s7qug=">aaacs3ichzfnt9wweiadul7sqhc4crg6kuppfsducefc7avhkngfabmkjjnzlbwnsh3eksof5nhb/02dsbflqepihr2az0yznklkwbxx/t+ou/zhfwnza9v7+gln9/ngb3+ii0oxglncfoomoroelza23ai4krxqpbfwndz/bpn1ayjnc3llfixmcjqxpoomghukb09elmccyzrjqvh8sfgqmoaj/nb5tlwizpocv5ozucakszmtchornlhuoqbhqy+chou9cnsuhs1/mj3gxbyvor862jnnx4ohp/i7w28fwyohwtpfppgdpqwrcpcgcar1lpilmdvugc4e2p6vhpkyezqhqzws5qbndbfzbn+1krrnhbjpgtxf/62oaa71ik9spp3xtq+ynvgem1ymo53vxjavacmeg2wvwkba7qfxyhuwixzwuka4nrwzo6oom/bmnl0cwf3ywzejrsqfkcvj4fmp5tq20ch6gr4hgk7qofqfltaymcd3jk7s3lrf3ambuolzqussaw7qk3pzv55szeg=</latexit> <latexit sha1_base64="3myafgfbz85tqxrlykm907s7qug=">aaacs3ichzfnt9wweiadul7sqhc4crg6kuppfsducefc7avhkngfabmkjjnzlbwnsh3eksof5nhb/02dsbflqepihr2az0yznklkwbxx/t+ou/zhfwnza9v7+gln9/ngb3+ii0oxglncfoomoroelza23ai4krxqpbfwndz/bpn1ayjnc3llfixmcjqxpoomghukb09elmccyzrjqvh8sfgqmoaj/nb5tlwizpocv5ozucakszmtchornlhuoqbhqy+chou9cnsuhs1/mj3gxbyvor862jnnx4ohp/i7w28fwyohwtpfppgdpqwrcpcgcar1lpilmdvugc4e2p6vhpkyezqhqzws5qbndbfzbn+1krrnhbjpgtxf/62oaa71ik9spp3xtq+ynvgem1ymo53vxjavacmeg2wvwkba7qfxyhuwixzwuka4nrwzo6oom/bmnl0cwf3ywzejrsqfkcvj4fmp5tq20ch6gr4hgk7qofqfltaymcd3jk7s3lrf3ambuolzqussaw7qk3pzv55szeg=</latexit> <latexit sha1_base64="3myafgfbz85tqxrlykm907s7qug=">aaacs3ichzfnt9wweiadul7sqhc4crg6kuppfsducefc7avhkngfabmkjjnzlbwnsh3eksof5nhb/02dsbflqepihr2az0yznklkwbxx/t+ou/zhfwnza9v7+gln9/ngb3+ii0oxglncfoomoroelza23ai4krxqpbfwndz/bpn1ayjnc3llfixmcjqxpoomghukb09elmccyzrjqvh8sfgqmoaj/nb5tlwizpocv5ozucakszmtchornlhuoqbhqy+chou9cnsuhs1/mj3gxbyvor862jnnx4ohp/i7w28fwyohwtpfppgdpqwrcpcgcar1lpilmdvugc4e2p6vhpkyezqhqzws5qbndbfzbn+1krrnhbjpgtxf/62oaa71ik9spp3xtq+ynvgem1ymo53vxjavacmeg2wvwkba7qfxyhuwixzwuka4nrwzo6oom/bmnl0cwf3ywzejrsqfkcvj4fmp5tq20ch6gr4hgk7qofqfltaymcd3jk7s3lrf3ambuolzqussaw7qk3pzv55szeg=</latexit> <latexit sha1_base64="3myafgfbz85tqxrlykm907s7qug=">aaacs3ichzfnt9wweiadul7sqhc4crg6kuppfsducefc7avhkngfabmkjjnzlbwnsh3eksof5nhb/02dsbflqepihr2az0yznklkwbxx/t+ou/zhfwnza9v7+gln9/ngb3+ii0oxglncfoomoroelza23ai4krxqpbfwndz/bpn1ayjnc3llfixmcjqxpoomghukb09elmccyzrjqvh8sfgqmoaj/nb5tlwizpocv5ozucakszmtchornlhuoqbhqy+chou9cnsuhs1/mj3gxbyvor862jnnx4ohp/i7w28fwyohwtpfppgdpqwrcpcgcar1lpilmdvugc4e2p6vhpkyezqhqzws5qbndbfzbn+1krrnhbjpgtxf/62oaa71ik9spp3xtq+ynvgem1ymo53vxjavacmeg2wvwkba7qfxyhuwixzwuka4nrwzo6oom/bmnl0cwf3ywzejrsqfkcvj4fmp5tq20ch6gr4hgk7qofqfltaymcd3jk7s3lrf3ambuolzqussaw7qk3pzv55szeg=</latexit> Computing F Single correspondence gives us one equation 2 3 2 u1 v 1 1 f 11 f 12 f 13 4f 21 f 22 f 5 4 23 f 31 f 32 f 33 x 1 y 1 1 3 5 =0 Multiply out u 1 x 1 f 11 + u 1 y 1 f 12 + u 1 f 13 + v 1 x 1 f 21 + v 1 y 1 f 22 +v 1 f 23 + x 1 f 31 + y 1 f 32 + f 33 =0 14
Computing F Rearrange for unknowns, add points by stacking rows [ u 1 x 1 u 1 y 1 u 1 v 1 x 1 v 1 y 1 v 1 x 1 y 1 1] [ u 2 x 2 u 2 y 2 u 2 v 2 x 2 v 2 y 2 v 2 x 2 y 2 1] [ u 3 x 3 u 3 y 3 u 3 v 3 x 3 v 3 y 3 v 3 x 3 y 3 1] [ u 4 x 4 u 4 y 4 u 4 v 4 x 4 v 4 y 4 v 4 x 4 y 4 1] [ u 5 x 5 u 5 y 5 u 5 v 5 x 5 v 5 y 5 v 5 x 5 y 5 1] [ u 6 x 6 u 6 y 6 u 6 v 6 x 6 v 6 y 6 v 6 x 6 y 6 1] [ u 7 x 7 u 7 y 7 u 7 v 7 x 7 v 7 y 7 v 7 x 7 y 7 1] [ u 8 x 8 u 8 y 8 u 8 v 8 x 8 v 8 y 8 v 8 x 8 y 8 1] 2 6 4 f 11 f 12 f 13 f 21 f 22 f 23 f 31 f 32 f 33 3 7 5 = 2 3 0 0 0 0 0 60 7 405 0 This is a linear system of the form Af = 0 can be solved using Singular Value Decomposition (SVD) 15
Pairwise Matching Find feature matches and use RANSAC to find inliers to F 16
Structure from Motion Given an (unordered) set of input images, compute camera positions and 3D structure of the scene 17
Sparse Reconstruction 18
Dense Reconstruction Use multiview stereo to compute dense depth (search along epipolar lines for matches) 19
Stereo Matching Stereo cameras are typically arranged by horizontal translation so epipolar lines (thus point matches) are horizontal (a) (b) (c) Stereo algorithms attempt to match along scanlines to find depth [ D. Scharstein ] 20
Disparity and Depth [ D. Scharstein ] 21
Stereo Images Image pair with depth dependent disparity 22
3D TV Most common use field sequential (shutter) glasses Transmit alternate left/right image at 120Hz Glasses free 3D systems send different images directly to each eye 23
Recap: 2-view Geometry Planar geometry: one to one mapping of points u = Hx viewing a plane, rotation full 3D 24
Recap: 2-view Geometry Epipolar geometry: point to line mapping u T Fx =0 moving camera, 3D scene 25
Recap: RANSAC Robust estimation of H/F and consistent matches Raw feature matches RANSAC for F 26
Next Lecture Multiview geometry + optimization 27