Learning Two-View Stereo Matching
|
|
- Arabella Berry
- 5 years ago
- Views:
Transcription
1 Learning Two-View Stereo Matching Jianxiong Xiao Jingni Chen Dit-Yan Yeung Long Quan Department of Computer Science and Engineering The Hong Kong University of Science and Technology The 10th European Conference on Computer Vision Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
2 Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
3 Introduction Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
4 Introduction Stereo Matching between Two Images Input: two wide-baseline images taken from the same static scene, neither calibrated nor rectified. For more general applications, such as robust motion estimation from structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
5 Introduction Related Work Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
6 Introduction Related Work Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
7 Introduction Related Work Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
8 Introduction Related Work Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
9 Introduction Related Work Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
10 Introduction Our Semi-supervised Matching Approach Propose a semi-supervised prospective of the matching problem without training. Utilize all information in an optimization procedure: local, regional and global. More robust to noise: the label vector is affected not merely by one matched pair but by all pairs with weighted paths to it. Capable of handling real number labels which is the inherent requirement of sub-pixel accuracy matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
11 Semi-supervised Matching Framework Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
12 Semi-supervised Matching Framework Three Main Catalogs of Learning Methods Supervised Learning Given that. Now, whether and is or? Unsupervised Learning Given, and, any interesting structure in them? Semi-supervised Learning Jianxiong Xiao et al. (HKUST)? Learning Two-View Stereo Matching ECCV / 45
13 Semi-supervised Matching Framework Three Main Catalogs of Learning Methods Supervised Learning Given that. Now, whether and is or? Unsupervised Learning Given, and, any interesting structure in them? Semi-supervised Learning Jianxiong Xiao et al. (HKUST)? Learning Two-View Stereo Matching ECCV / 45
14 Semi-supervised Matching Framework Three Main Catalogs of Learning Methods Supervised Learning Given that. Now, whether and is or? Unsupervised Learning Given, and, any interesting structure in them? Semi-supervised Learning Jianxiong Xiao et al. (HKUST)? Learning Two-View Stereo Matching ECCV / 45
15 Semi-supervised Matching Framework Three Main Catalogs of Learning Methods Supervised Learning Given that. Now, whether and is or? Unsupervised Learning Given, and, any interesting structure in them? Semi-supervised Learning Jianxiong Xiao et al. (HKUST)? Learning Two-View Stereo Matching ECCV / 45
16 Semi-supervised Matching Framework Notations For p = 1 or 2, q = 3 p: x p (s p 1) c p +t p : coordinate position (s p,t p ) in the p-th image space, s p {1,,r p }, t p {1,,c p },i = (s p 1) c p + t p. X p : Input image with size n p = r p c p pixels ( ) T X p = x p 1,xp 2,...,xp (s p 1) c p +t,...,x p p n. p x q j :a matching point of x p i located at coordinate position (s q,t q ) in the q-th continuous image space, s q,t q R. Label Vector y p i = ( v p i,h p ) T (( i = s 1,t 1) ( s 2,t 2)) T R 2, representing the position offset from the point in the first image to the point in the second image. Label Matrix Y p = ( y p 1,,yp n p ) T and Visibility Vector O p = ( o p 1,,op n p ) T. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
17 Semi-supervised Matching Framework Notations For p = 1 or 2, q = 3 p: x p (s p 1) c p +t p : coordinate position (s p,t p ) in the p-th image space, s p {1,,r p }, t p {1,,c p },i = (s p 1) c p + t p. X p : Input image with size n p = r p c p pixels ( ) T X p = x p 1,xp 2,...,xp (s p 1) c p +t,...,x p p n. p x q j :a matching point of x p i located at coordinate position (s q,t q ) in the q-th continuous image space, s q,t q R. Label Vector y p i = ( v p i,h p ) T (( i = s 1,t 1) ( s 2,t 2)) T R 2, representing the position offset from the point in the first image to the point in the second image. Label Matrix Y p = ( y p 1,,yp n p ) T and Visibility Vector O p = ( o p 1,,op n p ) T. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
18 Semi-supervised Matching Framework Notations For p = 1 or 2, q = 3 p: x p (s p 1) c p +t p : coordinate position (s p,t p ) in the p-th image space, s p {1,,r p }, t p {1,,c p },i = (s p 1) c p + t p. X p : Input image with size n p = r p c p pixels ( ) T X p = x p 1,xp 2,...,xp (s p 1) c p +t,...,x p p n. p x q j :a matching point of x p i located at coordinate position (s q,t q ) in the q-th continuous image space, s q,t q R. Label Vector y p i = ( v p i,h p ) T (( i = s 1,t 1) ( s 2,t 2)) T R 2, representing the position offset from the point in the first image to the point in the second image. Label Matrix Y p = ( y p 1,,yp n p ) T and Visibility Vector O p = ( o p 1,,op n p ) T. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
19 Semi-supervised Matching Framework Notations For p = 1 or 2, q = 3 p: x p (s p 1) c p +t p : coordinate position (s p,t p ) in the p-th image space, s p {1,,r p }, t p {1,,c p },i = (s p 1) c p + t p. X p : Input image with size n p = r p c p pixels ( ) T X p = x p 1,xp 2,...,xp (s p 1) c p +t,...,x p p n. p x q j :a matching point of x p i located at coordinate position (s q,t q ) in the q-th continuous image space, s q,t q R. Label Vector y p i = ( v p i,h p ) T (( i = s 1,t 1) ( s 2,t 2)) T R 2, representing the position offset from the point in the first image to the point in the second image. Label Matrix Y p = ( y p 1,,yp n p ) T and Visibility Vector O p = ( o p 1,,op n p ) T. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
20 Semi-supervised Matching Framework Notations For p = 1 or 2, q = 3 p: x p (s p 1) c p +t p : coordinate position (s p,t p ) in the p-th image space, s p {1,,r p }, t p {1,,c p },i = (s p 1) c p + t p. X p : Input image with size n p = r p c p pixels ( ) T X p = x p 1,xp 2,...,xp (s p 1) c p +t,...,x p p n. p x q j :a matching point of x p i located at coordinate position (s q,t q ) in the q-th continuous image space, s q,t q R. Label Vector y p i = ( v p i,h p ) T (( i = s 1,t 1) ( s 2,t 2)) T R 2, representing the position offset from the point in the first image to the point in the second image. Label Matrix Y p = ( y p 1,,yp n p ) T and Visibility Vector O p = ( o p 1,,op n p ) T. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
21 Semi-supervised Matching Framework Smoothness IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = A graph G = V,E. Two Images = Two Graphs G 1 = V 1,E 1 and G 2 = V 2,E 2. N ( x p ) i : the set of data points in the neighborhood of x p i. Affinity Matrix W p, w p ij is non-zero iff x p i and x p j are neighbors in E p. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
22 Semi-supervised Matching Framework Smoothness IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = A graph G = V,E. Two Images = Two Graphs G 1 = V 1,E 1 and G 2 = V 2,E 2. N ( x p ) i : the set of data points in the neighborhood of x p i. Affinity Matrix W p, w p ij is non-zero iff x p i and x p j are neighbors in E p. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
23 Semi-supervised Matching Framework Smoothness IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = A graph G = V,E. Two Images = Two Graphs G 1 = V 1,E 1 and G 2 = V 2,E 2. N ( x p ) i : the set of data points in the neighborhood of x p i. Affinity Matrix W p, w p ij is non-zero iff x p i and x p j are neighbors in E p. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
24 Semi-supervised Matching Framework Smoothness IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = A graph G = V,E. Two Images = Two Graphs G 1 = V 1,E 1 and G 2 = V 2,E 2. N ( x p ) i : the set of data points in the neighborhood of x p i. Affinity Matrix W p, w p ij is non-zero iff x p i and x p j are neighbors in E p. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
25 Semi-supervised Matching Framework Smoothness IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = A graph G = V,E. Two Images = Two Graphs G 1 = V 1,E 1 and G 2 = V 2,E 2. N ( x p ) i : the set of data points in the neighborhood of x p i. Affinity Matrix W p, w p ij is non-zero iff x p i and x p j are neighbors in E p. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
26 Semi-supervised Matching Framework Semi-Supervised Setting Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data X 1 l,y1 l and X 2 l,yl 2 = Unlabeled Data X 1 u,yu 1 and X 2 u,yu 2. Semi-supervised learning on the graph representation tries to estimate a label matrix Ŷ p that is consistent with: the initial incomplete label matrix, the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
27 Semi-supervised Matching Framework Semi-Supervised Setting Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data X 1 l,y1 l and X 2 l,yl 2 = Unlabeled Data X 1 u,yu 1 and X 2 u,yu 2. Semi-supervised learning on the graph representation tries to estimate a label matrix Ŷ p that is consistent with: the initial incomplete label matrix, the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
28 Semi-supervised Matching Framework Semi-Supervised Setting Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data X 1 l,y1 l and X 2 l,yl 2 = Unlabeled Data X 1 u,yu 1 and X 2 u,yu 2. Semi-supervised learning on the graph representation tries to estimate a label matrix Ŷ p that is consistent with: the initial incomplete label matrix, the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
29 Semi-supervised Matching Framework Semi-Supervised Setting Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data X 1 l,y1 l and X 2 l,yl 2 = Unlabeled Data X 1 u,yu 1 and X 2 u,yu 2. Semi-supervised learning on the graph representation tries to estimate a label matrix Ŷ p that is consistent with: the initial incomplete label matrix, the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
30 Semi-supervised Matching Framework Semi-Supervised Setting Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data X 1 l,y1 l and X 2 l,yl 2 = Unlabeled Data X 1 u,yu 1 and X 2 u,yu 2. Semi-supervised learning on the graph representation tries to estimate a label matrix Ŷ p that is consistent with: the initial incomplete label matrix, the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
31 Semi-supervised Matching Framework Semi-Supervised Setting Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data X 1 l,y1 l and X 2 l,yl 2 = Unlabeled Data X 1 u,yu 1 and X 2 u,yu 2. Semi-supervised learning on the graph representation tries to estimate a label matrix Ŷ p that is consistent with: the initial incomplete label matrix, the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
32 Semi-supervised Matching Framework Consistency Cost with Initial Labeling Given a configuration Ŷ p, consistency with the initial labeling can be measured by C p (Ŷp l,o p) = x p i X p l o p i ŷ p i y p 2 i. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
33 Semi-supervised Matching Framework Consistency Cost with Geometry Consistency with the geometry of the data in the image space, which follows from the smooth manifold assumption, motivates a penalty term of the form C p s (Ŷp,O p) = 1 2 w p ij (o φ p i,op j x p i,xp j X p ) ŷ p i ŷ p j 2, where ( ) φ o p i,op j = 1 2 [ (o p) ( ) ] 2 2 i + o p j penalizes rapid changes in Ŷ p between points that are close, and only enforces smoothness within visible regions. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
34 Semi-supervised Matching Framework Local Label Preference Cost Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
35 Semi-supervised Matching Framework Local Label Preference Cost Local Label Preference Cost The local cost is defined as: C p (Ŷp d,o p) ( = o p i ρp i x p i X p (ŷp i ) + ( 1 o p i ) τ p) i. Similarity cost function ρ p i (y) : similarity cost between the pixel x p i one image and the corresponding point for the label vector y in the other image space. { } Penalty term τ p x p i = max x p j N (xp i ) i x p j : to avoid that every point tends to have zero visibility to escape from cost charging. in Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
36 Semi-supervised Matching Framework Regional Surface Shape Cost Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
37 Semi-supervised Matching Framework Regional Surface Shape Cost Assumption Shape Cues The shapes of the 3D objects surface in the scene are very important cues for matching. Intuitive Approach To reconstruct the 3D surface based on two-view geometry. Unstable, especially when the baseline is not large enough. Piecewise Planar Patch Assumption Since two data points with high affinity relation are more likely to have similar label vectors, we assume that the label vector of a data point can be linearly approximated by the label vectors of its neighbors. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
38 Semi-supervised Matching Framework Regional Surface Shape Cost Assumption Shape Cues The shapes of the 3D objects surface in the scene are very important cues for matching. Intuitive Approach To reconstruct the 3D surface based on two-view geometry. Unstable, especially when the baseline is not large enough. Piecewise Planar Patch Assumption Since two data points with high affinity relation are more likely to have similar label vectors, we assume that the label vector of a data point can be linearly approximated by the label vectors of its neighbors. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
39 Semi-supervised Matching Framework Regional Surface Shape Cost Assumption Shape Cues The shapes of the 3D objects surface in the scene are very important cues for matching. Intuitive Approach To reconstruct the 3D surface based on two-view geometry. Unstable, especially when the baseline is not large enough. Piecewise Planar Patch Assumption Since two data points with high affinity relation are more likely to have similar label vectors, we assume that the label vector of a data point can be linearly approximated by the label vectors of its neighbors. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
40 Semi-supervised Matching Framework Regional Surface Shape Cost Reconstruction Cost The label of a data point can be linearly constructed by its neighbors: y p i = x p j N (xp i ) The reconstruction cost can be defined as C r (Y p ) = y p i x k i X k w p ij yp j. x p j N (xp i ) = (I W p )Y p 2 F tr ((Y p ) T L p Y p) = a p ij y p i y p j 2, x p i,xp j X p w p ij yp j A p = W p + (W p ) T W p (W p ) T : adjacency matrix D p is a diagonal matrix containing the row sums of A p,d p I L p = D p A p : un-normalized graph Laplacian matrix Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45 2
41 Semi-supervised Matching Framework Regional Surface Shape Cost LLE Cost To align the two 2D manifolds (image spaces) to one 2D manifold (visible surface). The labeled data (known matched pairs) are accounted for by constraining the mapped coordinates of matched points to coincide. Ŷc p = Ŷ p l Ŷ p u Ŷ q u X p c, O p c = A p c = = X p l Xu p Xu q, O p l O p u O q u [, A p A p = ll A p ll + Aq ll A p lu A q lu A p ul A p uu 0 A q ul 0 A q uu, A p ul A p lu A p uu ], C p r (Ŷ1,Ŷ 2,O 1,O 2) = (ac p ) ij φ ( (oc p ) i,(o p ) c ) j (ŷ p c ) i (ŷ p 2 c ) j. x p i,xp j X c p Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
42 Semi-supervised Matching Framework Global Epipolar Geometry Cost Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
43 Semi-supervised Matching Framework Global Epipolar Geometry Cost Global Epipolar Geometry Cost For x p i at position (s p,t p ), the epipolar line: ( a p ) i,bp i,cp i = (s p,t p,1)f T pq. Squared Euclidean distance in the image space of the other image: ( a p d p i i (y) = sq + b p i tq + c p ) 2 i ( a p) 2 ( i + b p) 2, i where y = (( s 1,t 1) ( s 2,t 2)) T. Global cost: C p g (Ŷp,O p) = o p i d p i x p i X p (ŷp ). i Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
44 Semi-supervised Matching Framework Symmetric Visibility Consistency Cost Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
45 Semi-supervised Matching Framework Symmetric Visibility Consistency Cost Symmetric Visibility Consistency Cost If x p i in one image has label to match with a point in the q-th image, then there must exist some point in the q-th image to have label to match x p i. C p v (O p,ŷ q ) = β x p i X p (o p i γ p i )) 2 ( ) 2 (Ŷq x p i,xp j X p w p ij o p i o p j γ function indicates whether or not there exist one or more data points that match a point near x p i from the other view according to Ŷ q. Last term enforces the smoothness of the occlusion. β controls the strength of the visibility constraint. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
46 Semi-supervised Matching Framework Symmetric Visibility Consistency Cost Voting for γ ( ) T For each point x q j at position (s q,t q ) in X q with label y q j = v q j,hq j : Place a 2D Gaussian ψ (s,t) on the p-th image centered at the matched position c j = (s p,t p ) T. = Mixture of Gaussian x q ψ cj (s,t) in the voted image space. j Truncate it γ p (s,t) = min { 1, ψ cj (s,t) }. x q j X q Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
47 Iterative MV Optimization Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
48 Iterative MV Optimization Optimization Process The optimization process has two steps: 1 M-step: estimate matching given visibility 2 V-step: estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
49 Iterative MV Optimization M-step: Estimation of Matching Given Visibility Visibility term C v imposes two constraints on Ŷ given O 1 Local Constraint : For each pixel x p i in the p-th image, it should not match the invisible (occluded) points in the other image. 2 Global Constraint: For each visible point in the q-th image, at least one data point in the p-th image should match it. In the M-step, we approximate the visibility term by considering only the local constraint, which can be incorporated into the similarity function ρ p i (y) in C d. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
50 Iterative MV Optimization M-step: Estimation of Matching Given Visibility Visibility term C v imposes two constraints on Ŷ given O 1 Local Constraint : For each pixel x p i in the p-th image, it should not match the invisible (occluded) points in the other image. 2 Global Constraint: For each visible point in the q-th image, at least one data point in the p-th image should match it. In the M-step, we approximate the visibility term by considering only the local constraint, which can be incorporated into the similarity function ρ p i (y) in C d. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
51 Iterative MV Optimization M-step: Estimation of Matching Given Visibility (Con t) Let Y = (( Y 1) T, ( Y 2 ) T ) T. Cost function: ) ( C M (Ŷ = λl C p l + λ s Cs p + λ d C p d + λ r Cr p + λ g C p ) g + ε Ŷ 2, p=1,2 where ε Ŷ 2 is a small regularization term in order to prevent degenerate situations. For fixed O 1 and O 2, the cost minimization is done by setting the derivative with respect to Ŷ to be zero since the second derivative is a positive definite matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
52 Iterative MV Optimization V-step: Estimation of Visibility Given Matching Let O = (( O 1) T, ( O 2 ) T ) T. Cost function: ( C V (O) = λl C p l + λ s Cs p + λ d C p d + λ r Cr p + λ g Cg p + λ v C p ) v + ε O 2, p=1,2 For fixed Ŷ 1 and ^Y 2, the cost minimization is done by setting the derivative with respect to O to be zero since the second derivative is a positive definite matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
53 Iterative MV Optimization System of Linear Equations Solving We can derive that by the way we defined W p and the cost functions, the coefficient matrix is strictly diagonally dominant and positive definite. Hence, Gauss-Seidel and Conjugate Gradient iterations both converge to the solution of the linear system with theoretical guarantee. GPU is helpful to speed up. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
54 Learning the Symmetric Affinity Matrix Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
55 Learning the Symmetric Affinity Matrix Learning the Affinity Matrix Directly define the matrix: no reliable approach for model selection if only very few labeled points are available. Learn the matrix: more reliable and stable. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
56 Learning the Symmetric Affinity Matrix Manifold Assumptions Smooth manifold and linear re-constructable assumption for the manifold in image space. The label space and the image space share the same local linear reconstruction weights. Linear construction weight matrix W p by minimizing the energy function E W p = x p i X p E x p, i E x p i = x p i x p j N (xp i ) w p ij xp j 2 = x p j,xp k N (xp i ) w p ij Gi jk w p ik. ) T (x where G i jk (x = p i x p p j i x p k). To avoid the undesirable contribution of negative weights, we enforce w p ij = 1, w p ij 0. x p j N (xp i ) Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
57 Learning the Symmetric Affinity Matrix Quadratic Programming Objective Function min W p x p i X p x p j,xp k N (xp i ) w p ij Gi jk w p ik + κ ij ( w p ij w p ji ) 2 (1) s.t. x p i X p, x p j N (xp i ) w p ij = 1, w p ij 0. ( ) 2 where ij w p ij w p ji is a penalty term to encourage w p ij and w p ji to be similar. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
58 More Details Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
59 More Details Label Initialization 1 For each image, the occlusion boundaries and depth ordering in the scene are identified using [Hoiem et al. 2007]. 2 SIFT key points and nearest neighbor algorithm to achieve an initial matching. 3 Enforce the constraints of the matching to be one-to-one cross consistency. 4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography transformation. 6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
60 More Details Label Initialization 1 For each image, the occlusion boundaries and depth ordering in the scene are identified using [Hoiem et al. 2007]. 2 SIFT key points and nearest neighbor algorithm to achieve an initial matching. 3 Enforce the constraints of the matching to be one-to-one cross consistency. 4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography transformation. 6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
61 More Details Label Initialization 1 For each image, the occlusion boundaries and depth ordering in the scene are identified using [Hoiem et al. 2007]. 2 SIFT key points and nearest neighbor algorithm to achieve an initial matching. 3 Enforce the constraints of the matching to be one-to-one cross consistency. 4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography transformation. 6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
62 More Details Label Initialization 1 For each image, the occlusion boundaries and depth ordering in the scene are identified using [Hoiem et al. 2007]. 2 SIFT key points and nearest neighbor algorithm to achieve an initial matching. 3 Enforce the constraints of the matching to be one-to-one cross consistency. 4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography transformation. 6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
63 More Details Label Initialization 1 For each image, the occlusion boundaries and depth ordering in the scene are identified using [Hoiem et al. 2007]. 2 SIFT key points and nearest neighbor algorithm to achieve an initial matching. 3 Enforce the constraints of the matching to be one-to-one cross consistency. 4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography transformation. 6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
64 More Details Label Initialization 1 For each image, the occlusion boundaries and depth ordering in the scene are identified using [Hoiem et al. 2007]. 2 SIFT key points and nearest neighbor algorithm to achieve an initial matching. 3 Enforce the constraints of the matching to be one-to-one cross consistency. 4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography transformation. 6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
65 More Details Computing the similarity cost function Our algorithm works with some label data in a semi-supervised manner by the consistent cost C l The local cost C d just works in an axillary manner. Unlike the traditional unsupervised matching, our framework does not heavily rely on the similarity function ρ p i (y). For efficient computation, we just sample some values for some integer combination of h and v to compute ρ p i (y) = exp( xp i xq j 2 2σ 2 ). We normalized the largest sampled value to be 1, and then fit ρ p i (y) with continuous and differentiable quadratic function, i.e. ρ p i (y) = (v v o) 2 + (h h o ) 2 2σ 2, where (v o,h o ) and σ are the center and spread of the parabola for x p i. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
66 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
67 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
68 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
69 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
70 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
71 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
72 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
73 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
74 More Details The Complete Procedure 1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then interpolate the results from sparse matching with depth information to achieve an initial solution. 3 Learn the affinity matrix W 1 and W 2. 4 while (cost change between two iterations threshold): 1 Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data, 2 Compute the parameters for the similarity cost function ρ and epipolar cost function d, 3 Estimate matching given visibility, 4 Compute the γ map, 5 Estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
75 Experiments Outline 1 Introduction 2 Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost 3 Iterative MV Optimization 4 Learning the Symmetric Affinity Matrix 5 More Details 6 Experiments Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
76 Experiments Experiments Set the parameters to be more favor C l and C g for M-step, and C v for the V-step, tune the parameters manually. The intensity value is set to be the norm of the label vector, i.e. y. For visualization, we scale the intensity to the range between 0 and 200, and only visible matching is shown, i.e. o 0.5. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
77 Experiments More Results Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
78 Experiments 3D Reconstruction from 3 Views and Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
79 Experiments 3D Reconstruction from 3 Views and Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
80 Experiments 3D Reconstruction from 3 Views and Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
81 Experiments 3D Reconstruction from 3 Views and Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
82 Experiments 3D Reconstruction from 3 Views and Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
83 Experiments 3D Reconstruction from 3 Views (Con t) Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
84 Experiments Application for Structure from Motion Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
85 Q & A Q & A Q & A Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV / 45
Structure from Motion. Introduction to Computer Vision CSE 152 Lecture 10
Structure from Motion CSE 152 Lecture 10 Announcements Homework 3 is due May 9, 11:59 PM Reading: Chapter 8: Structure from Motion Optional: Multiple View Geometry in Computer Vision, 2nd edition, Hartley
More informationEE795: Computer Vision and Intelligent Systems
EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 14 130307 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Stereo Dense Motion Estimation Translational
More informationMotion Estimation. There are three main types (or applications) of motion estimation:
Members: D91922016 朱威達 R93922010 林聖凱 R93922044 謝俊瑋 Motion Estimation There are three main types (or applications) of motion estimation: Parametric motion (image alignment) The main idea of parametric motion
More informationMultiview Stereo COSC450. Lecture 8
Multiview Stereo COSC450 Lecture 8 Stereo Vision So Far Stereo and epipolar geometry Fundamental matrix captures geometry 8-point algorithm Essential matrix with calibrated cameras 5-point algorithm Intersect
More informationAugmented Reality VU. Computer Vision 3D Registration (2) Prof. Vincent Lepetit
Augmented Reality VU Computer Vision 3D Registration (2) Prof. Vincent Lepetit Feature Point-Based 3D Tracking Feature Points for 3D Tracking Much less ambiguous than edges; Point-to-point reprojection
More informationProject Updates Short lecture Volumetric Modeling +2 papers
Volumetric Modeling Schedule (tentative) Feb 20 Feb 27 Mar 5 Introduction Lecture: Geometry, Camera Model, Calibration Lecture: Features, Tracking/Matching Mar 12 Mar 19 Mar 26 Apr 2 Apr 9 Apr 16 Apr 23
More informationLearning and Inferring Depth from Monocular Images. Jiyan Pan April 1, 2009
Learning and Inferring Depth from Monocular Images Jiyan Pan April 1, 2009 Traditional ways of inferring depth Binocular disparity Structure from motion Defocus Given a single monocular image, how to infer
More informationSubpixel accurate refinement of disparity maps using stereo correspondences
Subpixel accurate refinement of disparity maps using stereo correspondences Matthias Demant Lehrstuhl für Mustererkennung, Universität Freiburg Outline 1 Introduction and Overview 2 Refining the Cost Volume
More informationChaplin, Modern Times, 1936
Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection set] Multi-view geometry problems Structure: Given projections
More informationStructured Light II. Thanks to Ronen Gvili, Szymon Rusinkiewicz and Maks Ovsjanikov
Structured Light II Johannes Köhler Johannes.koehler@dfki.de Thanks to Ronen Gvili, Szymon Rusinkiewicz and Maks Ovsjanikov Introduction Previous lecture: Structured Light I Active Scanning Camera/emitter
More informationMatching. Compare region of image to region of image. Today, simplest kind of matching. Intensities similar.
Matching Compare region of image to region of image. We talked about this for stereo. Important for motion. Epipolar constraint unknown. But motion small. Recognition Find object in image. Recognize object.
More informationMotion Tracking and Event Understanding in Video Sequences
Motion Tracking and Event Understanding in Video Sequences Isaac Cohen Elaine Kang, Jinman Kang Institute for Robotics and Intelligent Systems University of Southern California Los Angeles, CA Objectives!
More informationEpipolar Geometry and Stereo Vision
Epipolar Geometry and Stereo Vision Computer Vision Jia-Bin Huang, Virginia Tech Many slides from S. Seitz and D. Hoiem Last class: Image Stitching Two images with rotation/zoom but no translation. X x
More informationStereo imaging ideal geometry
Stereo imaging ideal geometry (X,Y,Z) Z f (x L,y L ) f (x R,y R ) Optical axes are parallel Optical axes separated by baseline, b. Line connecting lens centers is perpendicular to the optical axis, and
More informationSegmentation and Tracking of Partial Planar Templates
Segmentation and Tracking of Partial Planar Templates Abdelsalam Masoud William Hoff Colorado School of Mines Colorado School of Mines Golden, CO 800 Golden, CO 800 amasoud@mines.edu whoff@mines.edu Abstract
More informationFlow Estimation. Min Bai. February 8, University of Toronto. Min Bai (UofT) Flow Estimation February 8, / 47
Flow Estimation Min Bai University of Toronto February 8, 2016 Min Bai (UofT) Flow Estimation February 8, 2016 1 / 47 Outline Optical Flow - Continued Min Bai (UofT) Flow Estimation February 8, 2016 2
More informationComputational Optical Imaging - Optique Numerique. -- Multiple View Geometry and Stereo --
Computational Optical Imaging - Optique Numerique -- Multiple View Geometry and Stereo -- Winter 2013 Ivo Ihrke with slides by Thorsten Thormaehlen Feature Detection and Matching Wide-Baseline-Matching
More informationStereo and Epipolar geometry
Previously Image Primitives (feature points, lines, contours) Today: Stereo and Epipolar geometry How to match primitives between two (multiple) views) Goals: 3D reconstruction, recognition Jana Kosecka
More informationSUMMARY: DISTINCTIVE IMAGE FEATURES FROM SCALE- INVARIANT KEYPOINTS
SUMMARY: DISTINCTIVE IMAGE FEATURES FROM SCALE- INVARIANT KEYPOINTS Cognitive Robotics Original: David G. Lowe, 004 Summary: Coen van Leeuwen, s1460919 Abstract: This article presents a method to extract
More informationApplication questions. Theoretical questions
The oral exam will last 30 minutes and will consist of one application question followed by two theoretical questions. Please find below a non exhaustive list of possible application questions. The list
More information10/03/11. Model Fitting. Computer Vision CS 143, Brown. James Hays. Slides from Silvio Savarese, Svetlana Lazebnik, and Derek Hoiem
10/03/11 Model Fitting Computer Vision CS 143, Brown James Hays Slides from Silvio Savarese, Svetlana Lazebnik, and Derek Hoiem Fitting: find the parameters of a model that best fit the data Alignment:
More informationStereo Vision. MAN-522 Computer Vision
Stereo Vision MAN-522 Computer Vision What is the goal of stereo vision? The recovery of the 3D structure of a scene using two or more images of the 3D scene, each acquired from a different viewpoint in
More informationCS231A Course Notes 4: Stereo Systems and Structure from Motion
CS231A Course Notes 4: Stereo Systems and Structure from Motion Kenji Hata and Silvio Savarese 1 Introduction In the previous notes, we covered how adding additional viewpoints of a scene can greatly enhance
More informationTwo-view geometry Computer Vision Spring 2018, Lecture 10
Two-view geometry http://www.cs.cmu.edu/~16385/ 16-385 Computer Vision Spring 2018, Lecture 10 Course announcements Homework 2 is due on February 23 rd. - Any questions about the homework? - How many of
More informationStereo matching. Francesco Isgrò. 3D Reconstruction and Stereo p.1/21
Stereo matching Francesco Isgrò 3D Reconstruction and Stereo p.1/21 Structure of a stereo vision system Extract interesting point from each image Determine a set of matching points Compute the fundamental
More informationDense 3D Reconstruction. Christiano Gava
Dense 3D Reconstruction Christiano Gava christiano.gava@dfki.de Outline Previous lecture: structure and motion II Structure and motion loop Triangulation Today: dense 3D reconstruction The matching problem
More informationarxiv: v1 [cs.cv] 28 Sep 2018
Camera Pose Estimation from Sequence of Calibrated Images arxiv:1809.11066v1 [cs.cv] 28 Sep 2018 Jacek Komorowski 1 and Przemyslaw Rokita 2 1 Maria Curie-Sklodowska University, Institute of Computer Science,
More informationSIFT: SCALE INVARIANT FEATURE TRANSFORM SURF: SPEEDED UP ROBUST FEATURES BASHAR ALSADIK EOS DEPT. TOPMAP M13 3D GEOINFORMATION FROM IMAGES 2014
SIFT: SCALE INVARIANT FEATURE TRANSFORM SURF: SPEEDED UP ROBUST FEATURES BASHAR ALSADIK EOS DEPT. TOPMAP M13 3D GEOINFORMATION FROM IMAGES 2014 SIFT SIFT: Scale Invariant Feature Transform; transform image
More informationGeometric Registration for Deformable Shapes 3.3 Advanced Global Matching
Geometric Registration for Deformable Shapes 3.3 Advanced Global Matching Correlated Correspondences [ASP*04] A Complete Registration System [HAW*08] In this session Advanced Global Matching Some practical
More informationEfficient Iterative Semi-supervised Classification on Manifold
. Efficient Iterative Semi-supervised Classification on Manifold... M. Farajtabar, H. R. Rabiee, A. Shaban, A. Soltani-Farani Sharif University of Technology, Tehran, Iran. Presented by Pooria Joulani
More informationStructure from Motion
11/18/11 Structure from Motion Computer Vision CS 143, Brown James Hays Many slides adapted from Derek Hoiem, Lana Lazebnik, Silvio Saverese, Steve Seitz, and Martial Hebert This class: structure from
More informationComputer Vision I. Dense Stereo Correspondences. Anita Sellent 1/15/16
Computer Vision I Dense Stereo Correspondences Anita Sellent Stereo Two Cameras Overlapping field of view Known transformation between cameras From disparity compute depth [ Bradski, Kaehler: Learning
More informationCorrespondence and Stereopsis. Original notes by W. Correa. Figures from [Forsyth & Ponce] and [Trucco & Verri]
Correspondence and Stereopsis Original notes by W. Correa. Figures from [Forsyth & Ponce] and [Trucco & Verri] Introduction Disparity: Informally: difference between two pictures Allows us to gain a strong
More informationThe SIFT (Scale Invariant Feature
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor developed by David Lowe University of British Columbia Initial paper ICCV 1999 Newer journal paper IJCV 2004 Review: Matt Brown s Canonical
More information3D Computer Vision. Structured Light II. Prof. Didier Stricker. Kaiserlautern University.
3D Computer Vision Structured Light II Prof. Didier Stricker Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de 1 Introduction
More informationRegularization and Markov Random Fields (MRF) CS 664 Spring 2008
Regularization and Markov Random Fields (MRF) CS 664 Spring 2008 Regularization in Low Level Vision Low level vision problems concerned with estimating some quantity at each pixel Visual motion (u(x,y),v(x,y))
More informationFundamental matrix. Let p be a point in left image, p in right image. Epipolar relation. Epipolar mapping described by a 3x3 matrix F
Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix F Fundamental
More informationEdge and local feature detection - 2. Importance of edge detection in computer vision
Edge and local feature detection Gradient based edge detection Edge detection by function fitting Second derivative edge detectors Edge linking and the construction of the chain graph Edge and local feature
More informationMulti-stable Perception. Necker Cube
Multi-stable Perception Necker Cube Spinning dancer illusion, Nobuyuki Kayahara Multiple view geometry Stereo vision Epipolar geometry Lowe Hartley and Zisserman Depth map extraction Essential matrix
More informationOther Linear Filters CS 211A
Other Linear Filters CS 211A Slides from Cornelia Fermüller and Marc Pollefeys Edge detection Convert a 2D image into a set of curves Extracts salient features of the scene More compact than pixels Origin
More informationLast lecture. Passive Stereo Spacetime Stereo
Last lecture Passive Stereo Spacetime Stereo Today Structure from Motion: Given pixel correspondences, how to compute 3D structure and camera motion? Slides stolen from Prof Yungyu Chuang Epipolar geometry
More informationWhat have we leaned so far?
What have we leaned so far? Camera structure Eye structure Project 1: High Dynamic Range Imaging What have we learned so far? Image Filtering Image Warping Camera Projection Model Project 2: Panoramic
More informationCS 4495 Computer Vision A. Bobick. Motion and Optic Flow. Stereo Matching
Stereo Matching Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix
More informationReminder: Lecture 20: The Eight-Point Algorithm. Essential/Fundamental Matrix. E/F Matrix Summary. Computing F. Computing F from Point Matches
Reminder: Lecture 20: The Eight-Point Algorithm F = -0.00310695-0.0025646 2.96584-0.028094-0.00771621 56.3813 13.1905-29.2007-9999.79 Readings T&V 7.3 and 7.4 Essential/Fundamental Matrix E/F Matrix Summary
More informationCamera Geometry II. COS 429 Princeton University
Camera Geometry II COS 429 Princeton University Outline Projective geometry Vanishing points Application: camera calibration Application: single-view metrology Epipolar geometry Application: stereo correspondence
More informationCS664 Lecture #21: SIFT, object recognition, dynamic programming
CS664 Lecture #21: SIFT, object recognition, dynamic programming Some material taken from: Sebastian Thrun, Stanford http://cs223b.stanford.edu/ Yuri Boykov, Western Ontario David Lowe, UBC http://www.cs.ubc.ca/~lowe/keypoints/
More informationCombining Appearance and Topology for Wide
Combining Appearance and Topology for Wide Baseline Matching Dennis Tell and Stefan Carlsson Presented by: Josh Wills Image Point Correspondences Critical foundation for many vision applications 3-D reconstruction,
More informationStep-by-Step Model Buidling
Step-by-Step Model Buidling Review Feature selection Feature selection Feature correspondence Camera Calibration Euclidean Reconstruction Landing Augmented Reality Vision Based Control Sparse Structure
More informationCS 4495 Computer Vision A. Bobick. Motion and Optic Flow. Stereo Matching
Stereo Matching Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix
More informationMidterm Examination CS 534: Computational Photography
Midterm Examination CS 534: Computational Photography November 3, 2016 NAME: Problem Score Max Score 1 6 2 8 3 9 4 12 5 4 6 13 7 7 8 6 9 9 10 6 11 14 12 6 Total 100 1 of 8 1. [6] (a) [3] What camera setting(s)
More information3D Computer Vision. Dense 3D Reconstruction II. Prof. Didier Stricker. Christiano Gava
3D Computer Vision Dense 3D Reconstruction II Prof. Didier Stricker Christiano Gava Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de
More informationSegmentation and Grouping
Segmentation and Grouping How and what do we see? Fundamental Problems ' Focus of attention, or grouping ' What subsets of pixels do we consider as possible objects? ' All connected subsets? ' Representation
More informationRectification and Disparity
Rectification and Disparity Nassir Navab Slides prepared by Christian Unger What is Stereo Vision? Introduction A technique aimed at inferring dense depth measurements efficiently using two cameras. Wide
More informationVisual Tracking (1) Tracking of Feature Points and Planar Rigid Objects
Intelligent Control Systems Visual Tracking (1) Tracking of Feature Points and Planar Rigid Objects Shingo Kagami Graduate School of Information Sciences, Tohoku University swk(at)ic.is.tohoku.ac.jp http://www.ic.is.tohoku.ac.jp/ja/swk/
More informationStructure from motion
Structure from motion Structure from motion Given a set of corresponding points in two or more images, compute the camera parameters and the 3D point coordinates?? R 1,t 1 R 2,t R 2 3,t 3 Camera 1 Camera
More informationObtaining Feature Correspondences
Obtaining Feature Correspondences Neill Campbell May 9, 2008 A state-of-the-art system for finding objects in images has recently been developed by David Lowe. The algorithm is termed the Scale-Invariant
More informationStructured light 3D reconstruction
Structured light 3D reconstruction Reconstruction pipeline and industrial applications rodola@dsi.unive.it 11/05/2010 3D Reconstruction 3D reconstruction is the process of capturing the shape and appearance
More informationShape Modeling and Geometry Processing
252-0538-00L, Spring 2018 Shape Modeling and Geometry Processing Discrete Differential Geometry Differential Geometry Motivation Formalize geometric properties of shapes Roi Poranne # 2 Differential Geometry
More informationStructure from Motion
/8/ Structure from Motion Computer Vision CS 43, Brown James Hays Many slides adapted from Derek Hoiem, Lana Lazebnik, Silvio Saverese, Steve Seitz, and Martial Hebert This class: structure from motion
More informationCSE152 Introduction to Computer Vision Assignment 3 (SP15) Instructor: Ben Ochoa Maximum Points : 85 Deadline : 11:59 p.m., Friday, 29-May-2015
Instructions: CSE15 Introduction to Computer Vision Assignment 3 (SP15) Instructor: Ben Ochoa Maximum Points : 85 Deadline : 11:59 p.m., Friday, 9-May-015 This assignment should be solved, and written
More informationLecture 3: Camera Calibration, DLT, SVD
Computer Vision Lecture 3 23--28 Lecture 3: Camera Calibration, DL, SVD he Inner Parameters In this section we will introduce the inner parameters of the cameras Recall from the camera equations λx = P
More informationA Factorization Method for Structure from Planar Motion
A Factorization Method for Structure from Planar Motion Jian Li and Rama Chellappa Center for Automation Research (CfAR) and Department of Electrical and Computer Engineering University of Maryland, College
More informationHomographies and RANSAC
Homographies and RANSAC Computer vision 6.869 Bill Freeman and Antonio Torralba March 30, 2011 Homographies and RANSAC Homographies RANSAC Building panoramas Phototourism 2 Depth-based ambiguity of position
More informationLocal features: detection and description. Local invariant features
Local features: detection and description Local invariant features Detection of interest points Harris corner detection Scale invariant blob detection: LoG Description of local patches SIFT : Histograms
More informationMethods for Representing and Recognizing 3D objects
Methods for Representing and Recognizing 3D objects part 1 Silvio Savarese University of Michigan at Ann Arbor Object: Building, 45º pose, 8-10 meters away Object: Person, back; 1-2 meters away Object:
More informationObject Recognition with Invariant Features
Object Recognition with Invariant Features Definition: Identify objects or scenes and determine their pose and model parameters Applications Industrial automation and inspection Mobile robots, toys, user
More informationDense 3D Reconstruction. Christiano Gava
Dense 3D Reconstruction Christiano Gava christiano.gava@dfki.de Outline Previous lecture: structure and motion II Structure and motion loop Triangulation Wide baseline matching (SIFT) Today: dense 3D reconstruction
More informationEdge detection. Convert a 2D image into a set of curves. Extracts salient features of the scene More compact than pixels
Edge Detection Edge detection Convert a 2D image into a set of curves Extracts salient features of the scene More compact than pixels Origin of Edges surface normal discontinuity depth discontinuity surface
More informationA Patch Prior for Dense 3D Reconstruction in Man-Made Environments
A Patch Prior for Dense 3D Reconstruction in Man-Made Environments Christian Häne 1, Christopher Zach 2, Bernhard Zeisl 1, Marc Pollefeys 1 1 ETH Zürich 2 MSR Cambridge October 14, 2012 A Patch Prior for
More informationComputer Vision Lecture 17
Announcements Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics Seminar in the summer semester Current Topics in Computer Vision and Machine Learning Block seminar, presentations in 1 st week
More informationEstimation of common groundplane based on co-motion statistics
Estimation of common groundplane based on co-motion statistics Zoltan Szlavik, Laszlo Havasi 2, Tamas Sziranyi Analogical and Neural Computing Laboratory, Computer and Automation Research Institute of
More informationCS231A Midterm Review. Friday 5/6/2016
CS231A Midterm Review Friday 5/6/2016 Outline General Logistics Camera Models Non-perspective cameras Calibration Single View Metrology Epipolar Geometry Structure from Motion Active Stereo and Volumetric
More informationFeatures Points. Andrea Torsello DAIS Università Ca Foscari via Torino 155, Mestre (VE)
Features Points Andrea Torsello DAIS Università Ca Foscari via Torino 155, 30172 Mestre (VE) Finding Corners Edge detectors perform poorly at corners. Corners provide repeatable points for matching, so
More informationLocal Feature Detectors
Local Feature Detectors Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr Slides adapted from Cordelia Schmid and David Lowe, CVPR 2003 Tutorial, Matthew Brown,
More informationIntroduction à la vision artificielle X
Introduction à la vision artificielle X Jean Ponce Email: ponce@di.ens.fr Web: http://www.di.ens.fr/~ponce Planches après les cours sur : http://www.di.ens.fr/~ponce/introvis/lect10.pptx http://www.di.ens.fr/~ponce/introvis/lect10.pdf
More informationEpipolar Geometry and Stereo Vision
Epipolar Geometry and Stereo Vision Computer Vision Shiv Ram Dubey, IIIT Sri City Many slides from S. Seitz and D. Hoiem Last class: Image Stitching Two images with rotation/zoom but no translation. X
More informationImage Segmentation. Selim Aksoy. Bilkent University
Image Segmentation Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr Examples of grouping in vision [http://poseidon.csd.auth.gr/lab_research/latest/imgs/s peakdepvidindex_img2.jpg]
More informationImage Segmentation. Selim Aksoy. Bilkent University
Image Segmentation Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr Examples of grouping in vision [http://poseidon.csd.auth.gr/lab_research/latest/imgs/s peakdepvidindex_img2.jpg]
More informationBIL Computer Vision Apr 16, 2014
BIL 719 - Computer Vision Apr 16, 2014 Binocular Stereo (cont d.), Structure from Motion Aykut Erdem Dept. of Computer Engineering Hacettepe University Slide credit: S. Lazebnik Basic stereo matching algorithm
More informationInstance-level recognition
Instance-level recognition 1) Local invariant features 2) Matching and recognition with local features 3) Efficient visual search 4) Very large scale indexing Matching of descriptors Matching and 3D reconstruction
More informationBayesian Methods in Vision: MAP Estimation, MRFs, Optimization
Bayesian Methods in Vision: MAP Estimation, MRFs, Optimization CS 650: Computer Vision Bryan S. Morse Optimization Approaches to Vision / Image Processing Recurring theme: Cast vision problem as an optimization
More informationCS 664 Segmentation. Daniel Huttenlocher
CS 664 Segmentation Daniel Huttenlocher Grouping Perceptual Organization Structural relationships between tokens Parallelism, symmetry, alignment Similarity of token properties Often strong psychophysical
More informationRectification and Distortion Correction
Rectification and Distortion Correction Hagen Spies March 12, 2003 Computer Vision Laboratory Department of Electrical Engineering Linköping University, Sweden Contents Distortion Correction Rectification
More informationStereo. 11/02/2012 CS129, Brown James Hays. Slides by Kristen Grauman
Stereo 11/02/2012 CS129, Brown James Hays Slides by Kristen Grauman Multiple views Multi-view geometry, matching, invariant features, stereo vision Lowe Hartley and Zisserman Why multiple views? Structure
More informationVision par ordinateur
Epipolar geometry π Vision par ordinateur Underlying structure in set of matches for rigid scenes l T 1 l 2 C1 m1 l1 e1 M L2 L1 e2 Géométrie épipolaire Fundamental matrix (x rank 2 matrix) m2 C2 l2 Frédéric
More informationFeature Trajectory Retrieval with Application to Accurate Structure and Motion Recovery
Feature Trajectory Retrieval with Application to Accurate Structure and Motion Recovery Kai Cordes, Oliver M uller, Bodo Rosenhahn, J orn Ostermann Institut f ur Informationsverarbeitung Leibniz Universit
More informationRobust Geometry Estimation from two Images
Robust Geometry Estimation from two Images Carsten Rother 09/12/2016 Computer Vision I: Image Formation Process Roadmap for next four lectures Computer Vision I: Image Formation Process 09/12/2016 2 Appearance-based
More informationWeek 2: Two-View Geometry. Padua Summer 08 Frank Dellaert
Week 2: Two-View Geometry Padua Summer 08 Frank Dellaert Mosaicking Outline 2D Transformation Hierarchy RANSAC Triangulation of 3D Points Cameras Triangulation via SVD Automatic Correspondence Essential
More informationFeature Based Registration - Image Alignment
Feature Based Registration - Image Alignment Image Registration Image registration is the process of estimating an optimal transformation between two or more images. Many slides from Alexei Efros http://graphics.cs.cmu.edu/courses/15-463/2007_fall/463.html
More informationEE795: Computer Vision and Intelligent Systems
EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 10 130221 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Canny Edge Detector Hough Transform Feature-Based
More informationLocal Features Tutorial: Nov. 8, 04
Local Features Tutorial: Nov. 8, 04 Local Features Tutorial References: Matlab SIFT tutorial (from course webpage) Lowe, David G. Distinctive Image Features from Scale Invariant Features, International
More informationComputer Vision I - Algorithms and Applications: Multi-View 3D reconstruction
Computer Vision I - Algorithms and Applications: Multi-View 3D reconstruction Carsten Rother 09/12/2013 Computer Vision I: Multi-View 3D reconstruction Roadmap this lecture Computer Vision I: Multi-View
More informationInstance-level recognition
Instance-level recognition 1) Local invariant features 2) Matching and recognition with local features 3) Efficient visual search 4) Very large scale indexing Matching of descriptors Matching and 3D reconstruction
More informationInstance-level recognition part 2
Visual Recognition and Machine Learning Summer School Paris 2011 Instance-level recognition part 2 Josef Sivic http://www.di.ens.fr/~josef INRIA, WILLOW, ENS/INRIA/CNRS UMR 8548 Laboratoire d Informatique,
More informationImage Transfer Methods. Satya Prakash Mallick Jan 28 th, 2003
Image Transfer Methods Satya Prakash Mallick Jan 28 th, 2003 Objective Given two or more images of the same scene, the objective is to synthesize a novel view of the scene from a view point where there
More informationMulti-View Stereo for Static and Dynamic Scenes
Multi-View Stereo for Static and Dynamic Scenes Wolfgang Burgard Jan 6, 2010 Main references Yasutaka Furukawa and Jean Ponce, Accurate, Dense and Robust Multi-View Stereopsis, 2007 C.L. Zitnick, S.B.
More informationMultiple View Geometry in Computer Vision Second Edition
Multiple View Geometry in Computer Vision Second Edition Richard Hartley Australian National University, Canberra, Australia Andrew Zisserman University of Oxford, UK CAMBRIDGE UNIVERSITY PRESS Contents
More informationNon-linear dimension reduction
Sta306b May 23, 2011 Dimension Reduction: 1 Non-linear dimension reduction ISOMAP: Tenenbaum, de Silva & Langford (2000) Local linear embedding: Roweis & Saul (2000) Local MDS: Chen (2006) all three methods
More informationComputer Vision Lecture 17
Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics 13.01.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar in the summer semester
More informationMotion Estimation (II) Ce Liu Microsoft Research New England
Motion Estimation (II) Ce Liu celiu@microsoft.com Microsoft Research New England Last time Motion perception Motion representation Parametric motion: Lucas-Kanade T I x du dv = I x I T x I y I x T I y
More information