Optical flow and depth from motion for omnidirectional images using a TV-L1 variational framework on graphs

Transcription

1 ICIP Monday, November 9 Optical flow and depth from motion for omnidirectional images using a TV-L1 variational framework on graphs Luigi Bagnato Signal Processing Laboratory - EPFL Advisors: Prof. Pierre Vandergheynst and Prof. Pascal Frossard 1

2 Introduction Problem Formulation Our Solution 2 Motivations Optical flow and dense depth map estimation are typical inverse problems in computer vision. Variational methods: - pro: very good performances - cons: quite heavy BUT efficient (real time) GPU-based implementation is possible for planar images. Omnidirectional vision systems are attractive in many applications (Robotics, 3D reconstruction) - They suffer of rather complex distortions 2

3 Introduction Problem Formulation Our Solution 3 Omnidirectional Vision Systems Catadioptric camera Panoptic camera 104 webcams arranged on a hemispherical aluminum board 3

4 Introduction Problem Formulation Our Solution 4 2-Sphere Representation Every single viewpoint optical system admits a unique mapping on a 2-sphere Omnidirectional image Projection on 2-Sphere Reference: Baker and Nayar. A theory of single-viewpoint catadioptric image formation. Int J Comput Vis (1999) vol. 35 (2) pp

5 Introduction Problem Formulation Our Solution 5 Synthetic Spherical Video Sequence 0 φ 2π θ π 5

6 Introduction Problem Formulation Our Solution 5 Synthetic Spherical Video Sequence 0 φ 2π θ π 5

7 Introduction Problem Formulation Our Solution 6 First Frame I 0 6

8 Introduction Problem Formulation Our Solution 7 Second Frame I 1 7

9 Introduction Problem Formulation Our Solution 8 Image Residual I 1 I 0 8

10 Introduction Problem Formulation Our Solution 9 Optical Flow Field Ground Truth 9

11 Introduction Problem Formulation Our Solution 10 Optical Flow Field - module Ground Truth 10

12 Introduction Problem Formulation Our Solution 11 Spherical Optical Flow y S 2 u = D 1 t s u(y) D(y) optical flow depth map y D P I : R 3 R (radial function) 11

13 Introduction Problem Formulation Our Solution 11 Spherical Optical Flow y S 2 t u = D 1 t s u(y) D(y) optical flow depth map y D P I : R 3 R (radial function) 11

14 Introduction Problem Formulation Our Solution 11 Spherical Optical Flow y S 2 t u = D 1 t s u(y) D(y) optical flow depth map y D P I : R 3 R (radial function) 11

15 Introduction Problem Formulation Our Solution 11 Spherical Optical Flow y S 2 y D u P t u = D 1 t s I : R 3 R u(y) D(y) (radial function) optical flow depth map 11

16 Introduction Problem Formulation Our Solution 11 Spherical Optical Flow y S 2 y D u P t u = D 1 t s I : R 3 R u(y) D(y) (radial function) optical flow depth map Brightness consistency I 0 (y) I 1 (y + u) = 0 11

17 Introduction Problem Formulation Our Solution 11 Spherical Optical Flow y S 2 y D u P t u = D 1 t s I : R 3 R u(y) D(y) (radial function) optical flow depth map Brightness consistency I 0 (y) I 1 (y + u) = 0 Linear approximation I 0 (y) ( s I 1 (y)) T u I 1 (y) = 0 11

18 Introduction Problem Formulation Our Solution 12 TV-L1 Inverse Problem J = Ω ψ( u i ) dω + λ Ω ρ(i 0,I 1, u) dω TV Regularization L1 norm fidelity term, robust to outliers ψ( s u i )= i s u i, i {1, 2} ρ(u) =I 1 (y)+( s I 1 (y)) T u I 0 (x) 12

19 Introduction Problem Formulation Our Solution 12 TV-L1 Inverse Problem J = Ω ψ( u i ) dω + λ Ω ρ(i 0,I 1, u) dω TV Regularization L1 norm fidelity term, robust to outliers ψ( s u i )= i s u i, i {1, 2} ρ(u) =I 1 (y)+( s I 1 (y)) T u I 0 (x) Functional Splitting J = Ω ψ( ui )+ 1 2θ u v 2 + λρ(v)dω v is an auxiliary variable close to u 12

20 Introduction Problem Formulation Our Solution 13 Two Step Algorithm J = Ω ψ( ui )+ 1 2θ u v 2 + λρ(v)dω 13

21 Introduction Problem Formulation Our Solution 13 Two Step Algorithm J = Ω ψ( ui )+ 1 2θ u v 2 + λρ(v)dω 1. Fix u and solve min v { Ω } 1 2θ u v 2 + λρ(v)dω Easy to solve, solution can be found pointwise by a soft thresholding scheme 13

22 Introduction Problem Formulation Our Solution 13 Two Step Algorithm J = Ω ψ( ui )+ 1 2θ u v 2 + λρ(v)dω 1. Fix u and solve min v { Ω } 1 2θ u v 2 + λρ(v)dω Easy to solve, solution can be found pointwise by a soft thresholding scheme 2. Fix v and solve min u { Ω ψ( s u i )+ 1 } 2θ u v 2 dω Classical TV denoising problem We need an efficient discretization scheme 13

23 Our Solution Introduction Problem Formulation 14 Graph Discretization Γ =( V, E, w) w : E R w(u, v) =w(v, u) > 0 Spherical geometry embedded in connectivity. Weights are decreasing with the geodesic distance. Can handle irregular sampling grid. Graph differential geometry ( w w(u, v) f)(u, v) = d(u) f(u) w(u, v) d(v) f(v) (div w F )(u) = w(u, v) (F (v, u) F (u, v)) d(v) u v Gradient (value on edges) Divergence at vertex u Local variation at vertex v w f = u v [( ) ] 2 w f (u, v) Degree at vertex v d(v) = u v w(u, v) Reference: Zhou and Scholkopf. Regularization on discrete spaces. Lect Notes Comput Sc (2005) vol pp

24 Our Solution Introduction Problem Formulation 15 Graph Regularization Γ =( V, E, w) w : E R w(u, v) =w(v, u) > 0 Discrete TV subproblem min u i { u i TV w + 12θ ui v i 2 } i {1, 2} Graph based Chambolle iterations v i = u i θdiv w p i i {1, 2}, p n+1 i = pn i + τ w (div w p n i u i/θ) 1+τ w (divp n i u i/θ) i {1, 2} References: Chambolle. An algorithm for total variation minimization and applications. J Math Imaging Vis (2004) vol. 20 (1-2) pp Peyre et al. Non-local Regularization of Inverse Problems. Computer Vision-Eccv (2008) 15

25 Introduction Problem Formulation Our Solution 16 Optical Flow - our solution Ground Truth Estimated optical flow (module) 16

26 Introduction Problem Formulation Our Solution 17 Optical Flow - planar technique Ground Truth Estimated optical flow (module) 17

27 Introduction Problem Formulation Our Solution 18 Planar-TVL1 GrH-TVL1 Error Module (SSE) Angle (AAE) Planar-TVL GrH-TVL

28 Introduction Problem Formulation Our Solution 19 Catadioptric Video Sequence I 0 I 1 19

29 Introduction Problem Formulation Our Solution 20 Catadioptric Video Sequence Image residual after motion compensation Initial Image residual 20

30 Our Solution Introduction Problem Formulation 21 A new inverse problem: Depth Estimation u t y S 2 u = D 1 t s u(y) D(y) optical flow depth map Z = D 1 optimization variable 21

31 Our Solution Introduction Problem Formulation 21 A new inverse problem: Depth Estimation u t y S 2 u = D 1 t s u(y) D(y) optical flow depth map Z = D 1 optimization variable I 1 (y)+z(( I 1 ) T t s ) I 0 (y) = 0 21

32 Our Solution Introduction Problem Formulation 21 A new inverse problem: Depth Estimation u t y S 2 u = D 1 t s u(y) D(y) optical flow depth map Z = D 1 optimization variable J = Ω I 1 (y)+z(( I 1 ) T t s ) I 0 (y) = 0 Z + 1 2θ (Z K)2 + λ I 1 (y)+z(( I 1 ) T t s I 0 (y)) dω K is an auxiliary variable close to D 21

33 Introduction Problem Formulation Our Solution 22 Depth Estimation - synthetic data Ground Truth Estimated depth map 22

34 Introduction Problem Formulation Our Solution 23 Real 3D Reconstruction From depth map to 3D model Estimated depth map 23

35 Introduction Problem Formulation Our Solution 23 Real 3D Reconstruction From depth map to 3D model Estimated depth map 23

36 24 Conclusions Adapted TVL1 algorithm for omnidirectional optical flow - Valid for all single effective viewpoint devices - Graceful handling of irregular sampling grid - Numerically stable Novel algorithm for dense depth map estimation - no correspondences to solve - test on real sequences are convincing Real time implementation (work in progress)! For more details: L.Bagnato,P.Vandergheynst,P.Frossard. A Variational Framework for Structure from Motion in Omnidirectional Video Sequences. Submitted to IEEE Transactions in Image Processing 24

37 25 Thank you! 25

38 26 References Baker and Nayar. A theory of single-viewpoint catadioptric image formation. Int J Comput Vis (1999) vol. 35 (2) pp Peyre et al. Non-local Regularization of Inverse Problems. Computer Vision-Eccv (2008) Chambolle. An algorithm for total variation minimization and applications. J Math Imaging Vis (2004) vol. 20 (1-2) pp Zhou and Scholkopf. Regularization on discrete spaces. Lect Notes Comput Sc (2005) vol pp

39 Introduction Problem Formulation Our Solution 27 Naive Discretization 27

40 Introduction Problem Formulation Our Solution 28 Structure from Motion Omnidirectional image Input: 2 omnidirectional images from a video sequence Algo: Graph based TVL1 variational approach Output: Depth Map and Ego-motion Projection on 2-Sphere 28

41 Introduction Problem Formulation Our Solution 29 Spherical Optical Flow Field Brightness consistency D y I 0 (y) I 1 (y + u) = 0 Linear approximation I 1 (y + u) ( I 1 (y + u 0 )) T (u u 0 ) I 0 (y) =0 29

42 Introduction Problem Formulation Our Solution 30 Result - our solution 30

43 Introduction Problem Formulation Our Solution 31 - planar technique 31