Scale Space and PDE methods in image analysis and processing. Arjan Kuijper

Transcription

1 Scale Space and PDE methods in image analysis and processing Arjan Kuijper Fraunhofer Institute for Computer Graphics Research Interactive Graphics Systems Group, TU Darmstadt Fraunhoferstrasse 5, Darmstadt, Germany Tel.: +49 (0) Fax.: +49 (0) Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 1/24

2 Summary of the previous week The differential structure of images Differential image structure Isophotes and flow lines Coordinate systems and transformations First order gauge coordinates: (u,w) - system Gauge coordinate invariants: examples Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 2/24

3 Today 1 left over part The differential structure of images Second order structure Principal curvatures Shape index Third order image structure: T-junction detection Fourth order image structure: junction detection Scale invariance and natural coordinates Irreducible invariants Taken from B. M. ter Haar Romeny, Front-End Vision and Multi-scale Image Analysis, Dordrecht, Kluwer Academic Publishers, Chapter 6 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 3/24

4 Second order structure At any point on the surface we can step into an infinite number of directions away from the point, and in each direction we can define a curvature. So in each point an infinite number of curvatures can be defined. When we smoothly change direction, there are two (opposite) directions where the curvature is maximal, and there are two (opposite) directions where the curvature is minimal. These directions are perpendicular to each other, and the extremal curvatures are called the principal curvatures. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 4/24

5 The Hessian matrix and principal curvatures The Hessian matrix is the gradient of the gradient vectorfield. The coefficients form the second order structure matrix. The Hessian matrix is square and symmetric, so we can bring it in diagonal form by calculating the Eigenvalues of the matrix and put these on the diagonal elements. 1 2 L xx L yy L 2 xx 4L 2 xy 2L xx L yy L 2 yy,0, 0, 1 2 L xx L yy L 2 xx 4L 2 xy 2L xx L yy L 2 yy These special values are the principal curvatures of that point of the surface. In the diagonal form the Hessian matrix is rotated in such a way, that the curvatures are maximal and minimal. The principal curvature directions are given by the Eigenvectors of the Hessian matrix. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 5/24

6 The shape index For the two principal curvatures the shape index is given by (k 1 k 2 ) shapeindex ª 2 p arctan k 1 +k 2 k 1 -k 2 k 1 k 2 shapeindex ª 2 p arctan -L xx - L yy L xx L xy 2-2 L xx L yy + L yy 2 The length of the vector defines how curved a shape is, definition of curvedness curvedness ª 1 1 k k 2 L xx 2L 2 xy L2 yy Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 6/24

7 Shape index The shape index runs from -1 (cup) via the shapes trough, rut (k 1 =0), and saddle rut to zero, the saddle, and the goes via saddle ridge, ridge (k 2 =0), and dome to the value of +1, the cap. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 7/24

8 Shape index Example: shape index and curvedness Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 8/24

9 Principal directions The principal curvature directions are given by the Eigenvectors of the Hessian matrix: L xx L yy L 2 xx 4L 2 xy 2L xx L yy L2 yy,1, 2L xy L xx L yy L 2 xx 4L 2 xy 2L xx L yy L2 yy,1 2L xy The local principal direction vectors form locally a frame. We orient the frame in such a way that the largest Eigenvalue (maximal principal curvature) is one direction, the minimal principal curvature direction is p/2 rotated clockwise. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 9/24

10 Principal directions Frames of the normalized principal curvature directions at a scale of 1 pixel. Green: maximal principal curvature direction; Red: minimal principal curvature direction. The principal curvatures have been employed in studies to the 2D and 3D structure of trabecular bone & blood vessels. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 10/24

11 Gaussian and mean curvature The Gaussian curvature K is defined as the product of the two principal curvatures. The Gaussian curvature is equal to the determinant of the Hessian matrix: L 2 xy L xx L yy Left: Gaussian curvature K for an MR image at a scale of 5 pixels. Middle: sign of K. Right: zerocrossings of K. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 11/24

12 Gaussian and mean curvature The mean curvature is defined as the arithmetic mean of the principal curvatures. The mean curvature is related to the trace of the Hessian matrix: 1 2 L xx L yy The directional derivatives of the principal curvatures in the direction of the principal directions are called the extremalities The product of the two extremalities is called the Gaussian extremality, a true local invariant. The boundaries of the regions where the Gaussian extremality changes sign, are the extremal lines. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 12/24

13 A rather complicated expression L 2 2 xy L xxx 2 2L xxx L xyy 3 L xxy 2 L xyy L xxx L xyy L xx L yy 2 2 L xxy 2L xy L xyy L yy 2L xxx L xxy L xy L xx L yy L xx 2L xx L xy L xyy L xxy L yy L xxy 2L 2 xy L 2 yy L yyy L 2 2 xy L yyy L 2 xx 4L 2 xy 2L xx L yy L 2 yy but useful for e.g. brain matching Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 13/24

14 Third order image structure: T-junction detection T-junctions in the intensity landscape of natural images occur typically at occlusion points. Occlusion points are those points where a contour ends or emerges because there is another object in front of the contour. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 14/24

15 Third order image structure: T-junction detection When we zoom in on the T-junction of an observed image and inspect locally the isophote structure at a T-junction, we see that at a T-junction the derivative of the isophote curvature k in the direction perpendicular to the isophotes is high. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 15/24

16 Third order image structure: T-junction detection When we study the curvature of the isophotes in the middle of the image, at the location of the T-junction, we see the isophote 'sweep' from highly curved to almost straight for decreasing intensity. So the geometric reasoning is the "the isophote curvature changes a lot when we traverse the image in the w direction". It seems to make sense to study 1 L2 x L y2 3 L xxy L 5 y L 4 x 2L 2 xy L x L xyy L xx L yy L 4 y 2L 2 xy L x L xxx 2L xyy L xx L yy L 2 x L 2 y 3L 2 xx 8L 2 xy L x L xxx L xyy 4L xx L yy 3L yy L 3 x L y 6L xy L xx L yy L x 2L xxy L yyy L x L y 3 6L xy L xx L yy L x L xxy L yyy 2 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 16/24

17 Third order image structure: T-junction detection To avoid singularities at vanishing gradients through the division by L 2 x + L 2 y 3 6 = L w we use as our T-junction detector: t= k w L w 6 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 17/24

18 Fourth order image structure: junction detection Yet another order higher: Find in a checkerboard the crossings where 4 edges meet. When we study the fourth order local image structure, we consider the fourth order polynomial terms from the local Taylor expansion. The main theorem of algebra states that a polynomial is fully described by its roots. How well all roots coincide, given by the discriminant, is a particular invariant condition. The discriminant of second order image structure is just the determinant of the Hessian matrix, i.e. the Gaussian curvature. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 18/24

19 Fourth order image structure: junction detection The forth order discriminant is slightly more complicated: Lxxxy 4 Lyyyy 2 Lxxxy 3 64 Lxyyy Lxxyy Lxyyy Lyyyy 12 Lxxxx Lxxxy Lxyyy 9 Lxxyy Lxyyy 2 15 Lxxyy 2 Lyyyy Lxxxx Lyyyy 2 6 Lxxxy 2 6 Lxxyy 2 Lxyyy 2 9 Lxxyy 3 Lyyyy Lxxxx Lxyyy 2 Lyyyy 9 Lxxxx Lxxyy Lyyyy 2 Lxxxx 54 Lxxyy 3 Lxyyy 2 81 Lxxyy 4 Lyyyy 54 Lxxxx Lxxyy Lxyyy 2 Lyyyy 18 Lxxxx Lxxyy 2 Lyyyy 2 Lxxxx 27 Lxyyy 4 Lxxxx Lyyyy 3 Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 19/24

20 Scale invariance and natural coordinates The dimensionless coordinate is termed the natural coordinate. This implies that the derivative operator in natural coordinates has a scaling factor: n n x`n Øsn x n Compare L w and the natural L w : Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 20/24

21 Irreducible invariants It has been shown by Hilbert that any invariant of finite order can be expressed as a polynomial function of a set of irreducible invariants. For e.g. scalar images these invariants form the fundamental set of image primitives in which all local intrinsic properties can be described. There are only a small number of irreducible invariants for low order. E.g. for 2D images up to second order there are only 5 of such irreducibles. One mechanism to find the irreducible set are gauge coordinates: Zeroth order First order Second order L Lw Lvv, Lvw, Lww Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 21/24

22 Tensors There are many ways to set up an irreducible basis. In tensor notation, tensor indices denote partial derivatives and run over the dimensions D L ii = i=x When indices come in pairs, summation over the dimensions is implied (the so-called Einstein summation convention, or contraction) Zeroth order L ii = L xx + L yy L First order Li Li (= Lx Lx + Ly Ly, the gradient) Second order Lii (= Lxx+ Ly y, the Laplacian) Li j L ji Li Li j L j (= Lxx Lx y 2 + Ly y 2, the 'deviation from flatness'), (= Lx 2 Lxx+ 2 Lx Ly Lx y+ Ly 2 Ly y, 'curvature') Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 22/24

23 Tensors Each of these irreducible invariants cannot be expressed in the others. Any invariant property to some finite order can be expressed as a combination of these irreducibles. Isophote curvature, a second order local invariant feature, is expressed as k=-l vv L w These irreducibles form a basis for the differential invariant structure. The set of 5 irreducible gray value invariants in 2D images has been exploited to classify local image structure for statistical object recognition. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 23/24

24 Summary Invariant differential feature detectors are special (mostly) polynomial combinations of image derivatives, which exhibit invariance under some chosen group of transformations. The derivatives are easily calculated from the image through the multi-scale Gaussian derivative kernels. The notion of invariance is crucial for geometric relevance; Non-invariant properties have no value in general feature detection tasks. A convenient paradigm to calculate features invariant under Euclidean coordinate transformations is the notion of gauge coordinates (v,w). Any combination of derivatives with respect to v and w is invariant under Euclidean transformations. The second order derivatives yield isophote and flow line curvature, cornerness; the third order derivatives gives T-junction detection in this framework, etc. Scale Space and PDE methods in image analysis and processing - Arjan Kuijper 24/24