Solving Vision Tasks with variational methods on the GPU

Transcription

1 Solving Vision Tasks with variational methods on the GPU Horst Bischof Inst. f. Computer Graphics and Vision Graz University of Technology Joint work with Thomas Pock, Markus Unger, Arnold Irschara and the ICG team 1

2 GPU-Speed 2

3 Recent GPU-Hardware Executes hundreds of parallel threads 32/64 bit floating point arithmetic, lots of main memory Rich set of general purpose arithmetic operations Features accessible via C-like languages (Cg, CUDA) (Supercomputer in pocket format) Nvidia GTX 285 Nvidia Tesla S1070 3

4 Embedded Systems Drivers: GPUs on mobile Phones Mobile Games Nvidia Tegra 4

5 What type of algorithms? Highly parallel (parallelizable) Good mathematical properties Measure of quality Variety of problems Variational Methods 5

6 Outline Total Variation Basics Applications Segmentation Optical Flow Stereo & 3D Conclusion 6

7 Computer Vision is Ill-Posed [1] Restrict the space of possible solutions by an a-priori asumption of the solution The Bayesian framework is often used to estimate the unknown quantities Equivalent to the Variational approach [1] J. Hadamard. Sur les problémes aux dérivées partielles et leur signification physique

8 Introduction 8

9 What is the TV norm? The TV norm is the L 1 norm of the L 2 vector norm of the image gradient History of L 1 estimation techniques Galileo Galilei, 1632 Laplace, 1793 Huber,

10 Why does it preserve discontinuities? W u dx = 1.0 W u 2 dx = Total Variation has no bias against discontinuities 10

11 Denoising Model of Rudin Osher and Fatemi Defined as the Variational Problem [3],[4] min u W u dx + l 2 W ( u - f ) 2 dx [3] L. Rudin and S. Osher and E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, 1992 [4] A. Chambolle and P. Lions, Image Recovery via Total Variation Minimization and Applications,

12 First Convex Example Total Variation 12

13 Numerical Methods Euler-Lagrange equations of the ROF model E E u l 2 = u dw + ( u - u0 ) dw 2 W = - Ł u u W + l ł ( u - u ) 0 0 = (Primal Formulation) Problem: EL equation is degenerated in flat regions Replace u by u 2 = u + e 2 e But decreases ability to preserve sharp edges 13

14 Variational Denoising Chambolle s projection algorithm Duality based formulation of the unconstrained formulation u max r p 1 r p u max u r p dw - r 0 p 1 2 W 1 a W r 2 ( p) dw Dual Formulation is quadratic in p, but has a bad constraint. Key observation of A. Chambolle: The Lagrange multipliers for the constrained Euler-Lagrange equation can be eliminated. Simple fixed point iteration scheme. r n 1 r n p + t p - u0 r n+ 1 p = Ła ł 1 r n 1+ t p - u0 Ła ł A. Chambolle. An Algorithm for Total Variation Minimization and Applications. J. Math. Imaging,

15 Primal ROF Primal versus Dual Non-smooth optimization problem Hard to optimize Dual ROF Smooth optimization problem with constraints Easy to optimize What about Primal-Dual? Make use of both, primal and dual More general than pure primal or dual Extremely fast primal-dual algorithm [Pock, Cremers, Bischof, Chambolle, 2009] 15

16 Outline Total Variation Basics Applications Segmentation Optical Flow Stereo & 3D Conclusion 16

17 Variational Image Denoising TV-L2 Denoising u Reconstructed Image f Original images with artifacts λ...regularization parameter Ω Image Domain TV-L1 Denoising 17

18 Variational Image Denoising TV-L2 18

19 Variational Image Denoising TV-L1 19

20 Link from Denoising to Segmentation 20

21 ( ) Proposed Energy Extend weighted TV with spatially varying data term: min u W ( x) u dx + l( x) u - f is provided by the user: f = l( x) = 0... Information not used 0 < l( x) <... Weak constraints l( x) fi... Hard constraints g x can also be modified by the user Functional remains convex Convex formulation of GAC g 0 background 1 foreground M. Unger, T. Pock, W. Trobin, D. Cremers, and H.Bischof. TV-Seg - interactive total variation based image segmentation. BMVC W f dx

22 Evolution of u using hard constraints iterations 22

23 With Color Model 23

24 Brain Segmentation 24

25 25

26 3D Interactive Segmentation 26

27 Texture Features Interactive texture segmentation using HOG features and on-line random forests Features pre-calculated Texture descriptor learned on-line using RF J. Santner, M. Unger, T. Pock, Ch. Leistner, A. Saffari, and H. Bischof. Interactive texture segmentation using random forests and total variation. In BMVC'09,

28 Results A. Saffari, C. Leistner, J. Santner, M. Godec, and H. Bischof. On-line random forests. In 3rd IEEE On-line Learning for Computer Vision Workshop. IEEE,

29 Outline Total Variation Basics Applications Segmentation Optical Flow Stereo & 3D Conclusion 29

30 TV-L 1 Optical Flow We use a robust variant of the Horn-Schunck formulation [8] Total Variation Regularization and L 1 data term Total Variation of Flow Sophisticated optimization techniques are needed! We have developed fast primal-dual schemes Implemented on the GPU L 1 norm of Optical Flow Constraint [8] B.K. Horn and B.G. Schunck. Determinig Optical Flow. Artificial Intelligence,

31 Huber Norm + Anisotropic Replace stair-casing afflicted isotropic Total Variation (TV) by a robust penalty function initially proposed by Huber. Incorporate directional information yielding an anisotropic Huber regularity. Huber Anisotropic M. Werlberger, W. Trobin, T. Pock, A. Wedel, D. Cremers, and H. Bischof. Anisotropic Huber-L1 optical flow. In BMVC'09,

32 Performance Evaluation TV-L 1 Optical Flow Implemented in CUDA 3.0 Computed on nvidia GeForce GTX 280 GPU 50 iterations / pyramid level Image Size Frames per Second 128x x x

33 Evaluation of optical flow methods Input Images Ground Truth Challenging Benchmark Dataset: 33

34 Results 34

35 Optical Flow Benchmark 35

36 Optical Flow for Driver Assistence (A. Wedel, Daimler) 36

37 Tracking using only Flow 37

38 Tracking as 3D Volume Segmentation Use Segmentation in 3D Minimize Volume M. Professor Unger, Horst Bischof T. Mauthner, Horst Cerjak, T. Pock, and H. Bischof. Tracking as segmentation of spatial-temporal volumes Variational by anisotropic Methods weighted & GPU TV. In EMCVPR 2009 pp Springer,

39 Results 39

40 Outline Total Variation Basics Applications Segmentation Optical Flow Stereo & 3D Conclusion 40

41 Total Variation energy functional Total Variation regularization Data term Data term potentially non-convex Defines domain of application Denoising Stereo T. Pock, T. Schönemann, G. Graber, H. Bischof, and D. Cremers. A convex formulation of continuous multi-label problems. ECCV08 41

42 Finding a Convex Representation Original non-convex problem New convex formulation Theorem: Minimizing is equivalent to minimizing 42

43 Convex Relaxation Theory due to Alberti and Bouchitte (2002) Consider the function u(x) as a surface in higher dimensions (functional lifting) Consider the maximal flux of a vector field going through the surface T. Pock, D. Cremers, H. Bischof, and A. Chambolle. An algorithm for minimizing the Mumford-Shah functional. ICCV,

44 Stereo results We applied our method to stereo problems Data term: absolute differences 44

45 Qualitative comparison to Ishikawa Tsukuba data Ground truth 45

46 Quantitative comparison to Ishikawa Tsukuba data set, (size=384x288) Results comparable to 16 connected graph The proposed algorithm is 20 times faster Requires only 3,6% of the memory Can be applied to much larger problems 46

47 3D Reconstruction System Feature Extraction Feature Matching Geometric Verification Track Generation Initial Structure from Motion Bundle Adjustment Dense Matching Range Image Fusion 47

48 Aerial Triangulation (SfM approach) 3.7 Mio. 3D points (SIFT keys) ~4200 measurements / image 48

49 DEPTH map 49

50 DEPTH map 50

51 DEPTH map 51

52 Jakomini Platz 52

53 Depth Map Fusion Task: Compute Digital Surface Model (DSM) from range images High overlap in aerial images One position is seen from 8-15 images Exploit redundancy to improve the DSM Robust fusion of range images to a single DSM using a Total Generalized Variation functional 53

54 Results 54

55 Fusion of real Data 55

56 Conclusion Total Variation is powerful Convex Optimization Variety of Applications Fast using GPUs General Framework, easy to extend Couple different modules 56

57 Videos/Code/Papers see 57

58 Acknowledgments Funding provided by: Austrian Joint Research Project Cognitive Vision under sub-projects S9103-N03 and S9104-N04 Doctoral Program Confluence of Vision and Graphics funded by Austrian Science Found FIT-IT Program funded by BMVIT under Project VMGPU Ludwig Boltzmann Inst. on Forensic Radiology 58