Fraunhofer Institute for Computer Graphics Research Interactive Graphics Systems Group, TU Darmstadt Fraunhoferstrasse 5, Darmstadt, Germany

Transcription

1 Scale Space and PDE methods in image analysis and processing Arjan Kuijper Fraunhofer Institute for Computer Graphics Research Interactive Graphics Systems Group, TU Darmstadt Fraunhoferstrasse 5, Darmstadt, Germany Tel.: +49 (0) Fax.: +49 (0) Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 1/47

2 Rough course outline Biological vision scale space What do we see? Can we mimic human vision? Image structure How can we model structure? What are relevant features? PDE-based imaging PDE are used to evolve images to an optimal structure. How can we design perceptually relevant PDEs? Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 2/47

3 Example 1: Biological vision scale space What is a cloud? Is it a white thing in the sky? Water particles? An outline? Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 3/47

4 Example 2: Image structure, Ridge detection Local coordinates can be used as a detector for ridges, edges, corners, Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 4/47

5 Example 3: PDE-based imaging PDE-based segmentation: Compute the number of cells and their sizes. (transmission and differential interference contrast microscopy) Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 5/47

6 Example 3, cont. Use three evolution phases: Initial expansion with Free expansion Surface wrapping Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 6/47

7 Example 3 - Results Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 7/47

8 Find the differences Dutch army in Uruzgan Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 8/47

9 Lenin 1919, Iran 2008, N Korea. ( ) Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 9/47

10 No hacks J[ u] = E D u dxdy + λ 2 E u u 0 2 dxdy, Scientific alternative: (no pragmatic solutions!) Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 10/47

11 Contents Image analysis & processing deals with the investigation of images the application of specific tasks on them, like enhancement, denoising, deblurring segmentation. Mathematical methods that are commonly used are presented and discussed. The focus will be on the axiomatic choice for the models, their mathematical properties their practical use. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 11/47

12 Image analysis & processing As image analysis and processing is a mixture of several disciplines Physics, mathematics, vision, computer science, engineering, This course is aimed at a broad audience. Only basic knowledge of analysis is assumed. Necessary mathematical tools will be outlined during the meetings. Mathematics Electrical Engineering Computer Science Computer Vision Medicine Human Perception Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 12/47

13 Some key words Images & Observations: Scale space, regularization, distributions. Filtering: Edge detection, enhancement, Wiener, Fourier, Objects: Differential structure, invariants, feature detection Deep structure: Catastrophes & Multi-scale Hierarchy Variational Methods & Partial Differential Methods: Perona Malik, Anisotropic Diffusion, Total Variation, Mumford-Shah. Curve Evolution: Normal Motion, Mean Curvature Motion, Euclidian Shortening Flow. Tools: Level sets, explicit numerical schemes Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 13/47

14 Contents I: Introduction, Axioms, II: Gaussian kernel, regularisation III: Derivatives, deblurring, IV: Features, image geometry, differential structure V: Perona Malik, VI: Total Variation VII: Mean Curvature Motion, VIII: Mumford Shah, Snakes/Active Contours IX: Level Sets, Chan Vese X-XIII : presentations Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 14/47

15 Examination Investigation and public presentation of recent work in image analysis provided at the course: Front-End Vision and Multi-scale Image Analysis, B. M. ter Haar Romeny Optic Flow Color Differential Structure Steerable kernels Handbook of Mathematical Models in Computer Vision, N. Paragios, Y. Chen and O. Faugeras Diffusion Filters and Wavelets Total Variation Image Restoration PDE-Based Image and Surface Inpainting Geometric Level Set Methods in Imaging, Vision, and Graphics S. Osher, Stanley & N. Paragios Adaptive segmentation of vector-valued images Joint image registration and segmentation Fast methods for implicit active contour models A written exam (questions) on contents of the course. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 15/47

16 Introduction, Axioms Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 16/47

17 Introduction Apertures and the notion of scale Observations and the size of apertures Mathematics, physics, and vision We blur by looking A critical view on observations Taken from B. M. ter Haar Romeny, Front-End Vision and Multi-scale Image Analysis, Dordrecht, Kluwer Academic Publishers, Chapter 1 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 17/47

18 Observations and the size of apertures What is a cloud? Observations are always done by integrating some physical property with a measurement device. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 18/47

19 Measurements A typical image: Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 19/47

20 Mathematics, physics, and vision Observations: math vs. physics Objects have a size. Points don t exist in reality. Objects live on a range of various sizes. They contain several scales. Objects are measured by some device. Cameras, the eye, Devices are finite. They have a minimum and a maximum detection range: the inner and outer scale. They determine the spatial resolution. The device measures an hierarchy of structures. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 20/47

21 From Wikipedia: Powers of Ten Powers of Ten is a 1977 short documentary film which depicts the relative scale of the Universe in factors of ten (see also logarithmic scale and order of magnitude). It was written and directed by Charles and Ray Eames. The idea for the film appears to have come from the 1957 book Cosmic View by Kees Boeke. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 21/47

22 We blur by looking Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 22/47

23 The visual system We see multi-scale: The images only contain two values (black and white). We regards them as grey level images, or see structure. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 23/47

24 A critical view on observations Infinite resolution is impossible. We cannot measure at infinite resolution. Take uncommitted observations Allow different scales. View all scales. There is no bias, no knowledge, no memory. We know nothing. At least, at the first stage. Refine later on. There s more than just pixels. There is no preferred size. Noise is part of the measurement. In a measurement noise can only be separated from the observation if we have a model of the structures in the image, a model of the noise, or a model of both. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 24/47

25 Spurious resolution Don t trust the grid size the world isn t blocky. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 25/47

26 You don t see what you see Don t trust the resolution / nearest neighbor interpolation. What does a detector of a 3 pixels circular size detect? Do you see the image as it is? Or did you see it in a modified way and is its intrinsic size different? Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 26/47

27 Summary Observations are necessarily done through a finite aperture. Making this aperture infinitesimally small is not a physical reality. The size of the aperture determines a hierarchy of structures, which occur naturally in (natural) images. The visual system exploits a wide range of such observation apertures in the front-end simultaneously, in order to capture the information at all scales. Observed noise is part of the observation. There is no way to separate the noise from the data if a model of the data, a model of the noise or a model of both is absent. The aperture cannot take any form. An example of a wrong aperture is the square pixel so often used when zooming in on images. Such a representation gives rise to edges that were never present in the original image. This artificial extra information is called 'spurious resolution'. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 27/47

28 Introduction, Axioms Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 28/47

29 Axioms Foundations of scale space Constraints for an uncommitted front-end Axioms of a visual front-end Axiomatic derivation of the Gaussian kernel Scale space from causality Scale space from entropy maximization Derivatives of sampled, observed data Scale space stack Taken from B. M. ter Haar Romeny, Front-End Vision and Multi-scale Image Analysis, Dordrecht, Kluwer Academic Publishers, Chapter 2 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 29/47

30 Constraints for an uncommitted front-end I1 I2 I3 O K Y B L1 F1 A P N L2 F2 Convolution kernel x x x x x x x x x x x Semigroup property x x x x x x x x x Locality x Regularity x x x x x x x x Infinitesimal generator x Max. loss principle x Causality x x x x x Nonnegativity x x x x x x Tikhonov regularization x Average grey level invar. x x x x x x Flat kernel for t to infinity x Isometry invariance x x x x x x x x x x x Homogeneity & isotropy x Separability x x Scale invariance x x x x x x x x Valid for dimension ,2 1,2 1 1 >1 N 1,2 N N N Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 30/47

31 Axioms of a visual front-end Uncommitted assumptions: 1. scale invariance (no preferred scale or size) 2. spatial shift invariance (no preferred location) 3. isotropy (no preferred orientation) 4. linearity (no memory or model) 5. separability (for the sake of computational ease) Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 31/47

32 Axioms of a visual front-end Physical properties: L [candela/meter 2 ] <> x [meters] Intensity <> distance : Dimensions don t match Pi-teorem: Physical laws must be independent of the choice of the fundamental parameters [set =1/x, the frequency, L the observed image and L 0 the real image] 1. Scale invariance L/L 0 = G( ) (or: L/L 0 = G(x/ )) Both sides are dimensionless. Task: Find G! Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 32/47

33 An uncommitted front-end 2. Linear shift invariance Convolution (probing everywhere with a function G): In Fourier domain equal to multiplication: 3. Isotropy Rotationally invariant : set 4. Linearity Which implies Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 33/47

34 An uncommitted front-end 5. Separability Dimensions are independent p = 2 Outer scale (image averages) For, G( ) 0 So 2 < 0, say -1/2 for later convenience. Inner scale (don t do anything) For 0, G( ) 1: OK Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 34/47

35 An uncommitted front-end Back to the spatial domain with the inverse Fourier transform Add a normalization factor 1/ 2, no enhancement of the average data value Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 35/47

36 Scale space from causality Whatever you do on this image, you don t want the introduction of white regions in the black ones. No new level lines are to be created: Causality Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 36/47

37 Scale space from causality Causality: non-enhancement of local extrema. Let L = L xx + L yy L equals the sum of the eigenvalues of the Hessian. Then at a maximum L < 0 and L t < 0 and at a minimum L > 0 and L t > 0 So L L t > 0. Choose L t = a L, a > 0 With a = 1, L t = L intensity x Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 37/47

38 Scale space from causality L t (x,y;t) = L (x,y;t) Obviously, for t -> 0, L (x,y;t) = L 0 The general solution (Greens function) for this diffusion equation is convolution of the original image with an Gaussian: G(x,y;t) = Exp (-(x 2 +y 2 )/(4 t)) / (4 t) Note: one uses rater 4t than 2s 2 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 38/47

39 Scale space from entropy maximization A statistical measure for the disorder of the filter is given by the entropy: 1D for simplicity If it is maximized it states something like there is nothing ordered (we know nothing). Obviously, there are some constraints. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 39/47

40 Scale space from entropy maximization Constraints The function must be normalized; no global enhancement: The mean of the measurement is at the location where we measure, say 0: There is a standard deviation, say : The function is positive; it s a real object: g(x)>0 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 40/47

41 Scale space from entropy maximization Maximize the Euler Lagrange equation Set the variational derivative w.r.t. g(x) equal to zero: So Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 41/47

42 Scale space from entropy maximization 1. x g(x) dx = 0 -> λ 2 = 0 2. x 2 g(x) dx = 2 -> λ 3 = -1/(2 2 ) 3. g(x)>0 -> OK 4. g(x) dx = 1 -> λ 1 = Log[e/ (2 2 )] => g(x) = Exp[-1+1-Log[ (2 2 )]- x 2 /(2 2 )] = Exp[-x 2 /(2 2 )] / (2 2 ) Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 42/47

43 Derivatives of sampled, observed data The Gaussian kernel and all of its partial derivatives form the unique set of kernels for a front-end visual system that satisfies the constraints: no preference for location, scale and orientation, and linearity. It is a one-parameter family of kernels, where the scale is the free parameter. The derivative of the observed data is given by which equals Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 43/47

44 Derivatives of sampled, observed data Derivatives of a Gaussian: The first order derivative of an image gives edges Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 44/47

45 Gaussian scale space D L(x; ) = L0(x) * Exp (- x /(2 ) / Sqrt[ (2 ) ] L(x; ) is called the Gaussian scale space image. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 45/47

46 Summary We have specific physical constraints for the early vision front-end kernel. We are able to set up a 'first principle' framework from which the exact sensitivity function of the measurement aperture can be derived. There exist many such derivations for an uncommitted kernel The assumptions of linearity, isoptropy, homogeneity and scale-invariance; The principle of causality; Minimization of the entropy They all lead to the same unique result: the Gaussian kernel Differentiation of discrete data is done by the convolution with the derivative of the observation kernel. This means that differentiation can never be done without blurring the data somewhat. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 46/47