Non-Rigid Image Registration III

Non-Rigid Image Registration III CS6240 Multimedia Analysis Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 1 / 47

Variational Approach Basic ideas [Fis05, Mod04]: T denotes template image to be deformed. Represent deformation of T by displacement u(x) of point x, i.e., x in T becomes x+u(x) in the deformed template. T u denotes the deformed template image. (a) Template image T (b) Deformed template image T u. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 2 / 47

R denotes the fixed reference or target object. D(R,T;u) denotes the difference between fixed R and T after deformation u. Find u that minimizes D(R,T;u). This problem is ill-posed; need regularizer (smoothing function) S. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 3 / 47

The problem can be formulated as follows: Given reference image R, template image T, difference measure D, regularizer S, find the displacements u(x) of points x in T that minimizes the error E where α is a regularization parameter. E = D(R,T;u)+αS(u) (1) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 4 / 47

Apply variational calculus method: 1 When E is minimized, u satisfies the Euler-Lagrange equation: αa(u)+f(u) = 0. (2) A(u) is obtained from S(u). Force term F(u) is obtained from D(R,T;u). 2 Apply time-marching method to obtain 3 Apply semi-implicit discretization to obtain u t +αa(u)+f(u) = 0. (3) u(k +1) u(k)+γ[αau(k +1)+F(u(k))] = 0. (4) Au is obtained from A(u). 4 Rearrange to obtain the iterative equation: u(k +1) = (I+γαA) 1 [u(k) γf(u(k))]. (5) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 5 / 47

Types of difference measure Sum-squared difference Cross correlation Mutual information Types of regularizer (smoothing function) Elastic Fluid Diffusion Curvature Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 6 / 47

Sum-Squared Difference D(R,T;u) = 1 [R(x) T(x+u(x))] 2 dx 2 = 1 [R(x) T u (x)] 2 dx 2 (6) This is ok for R and T with similar intensity and contrast. Force F(u) is given by [Fis05, Mod04]: F(u) = (R(x) T u (x)) T u (x). (7) Need to apply d-linear interpolation to compute T u, where d is the number of dimensions of the images. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 7 / 47

Mutual Information P R (r): probability that a point in R has the value r. P Tu (t): probability that a point in T u has the value t. P R,Tu (r,t): joint probability that the corresponding points in R and T u have the values r and t respectively. Mutual information (MI) M(R,T u ) is given by M(R,T u ) = P R,Tu (r,t)log P R,T u (r,t) drdt. (8) P R (r)p Tu (t) MI can also be computed by entropy H: where M(R,T u ) = H(R)+H(T u ) H(R,T u ) (9) H(R) = H(R,T u ) = P R (r)logp R (r)dr P R,Tu (r,t)logp R,Tu (r,t)drdt. (10) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 8 / 47

Distance measure D(R,T;u) = M(R,T u ). Force F(u) is given by [Fis05]: [ 1 P Tu (t) F(u) = P Tu (t) t 1 P R,Tu (r,t) ] P R,Tu (r,t) T u (x) (11) t Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 9 / 47

Notes: Also need to apply d-linear interpolation to compute T u. MI has been widely used for registering images with different intensities and contrasts (e.g., [Vio95, VI97]). MI is not differentiable if the probability densities are non-smooth. Can approximate using Parzen window technique [DH73, Vio95] which is differentiable. MI objective function sometimes reaches its global minimum at an incorrect alignment, even when a correct alignment between the images exists [PMV00]. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 10 / 47

Elastic Registration Elastic Registration Based on physical properties of elastic body. Applying external force to an elastic body results in deformation or strain of the body. Strain is related to tension or stress of the body. The body s shape results from an equilibrium of external force and internal stress. Model the smoothing function S(u) by elastic potential. Allow for stretching or shrinking without tearing. Let u = (u 1,...,u d ) T, x = (x 1,...,x d ) T where d is the number of dimensions of the images. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 11 / 47

Elastic Registration Elastic potential P(u) is given by P(u) = µ 4 d j,k=1 where λ and µ are the Lamé s elastic coefficients. ( uk + u ) 2 j + λ x j x k 2 ( u)2 dx (12) For 2D case (x 1 = x,x 2 = y), [ ( u1 ) 2 ( ) 2 u2 P(u) = µ + + 1 ( u2 + u ) ] 2 1 + λ x 1 x 2 2 x 1 x 2 2 ( u)2 dx (13) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 12 / 47

Elastic Registration Elastic regularizer gives the equation [BK89, Fis05, Mod04]: A(u) = µ 2 u+(λ+µ) u. (14) ( ) T =,..., is the gradient operator. x 1 x d d 2 2 = is the Laplacian operator. x 2 i=1 i 2 u = ( 2 u 1,..., 2 u d ) T. d u i u = is the divergence of u. x i i=1 Typically, λ = 0. If λ 0, stretching in one direction is accompanied by shrinking in the perpendicular direction. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 13 / 47

Elastic Registration Boundary Conditions Explicit boundary conditions can be imposed to further influence how the template is deformed. Without lack of generality, assume that the images are bounded within the internal [0, 1]. The following are possible explicit boundary conditions (2D case): Zero Padding (Dirichlet): u(x) = 0 for image boundary points x. Mirroring (Neumann): u l (x) n(x) = 0 for image boundary points x, and outer normal unit vectors n at image boundary. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 14 / 47

Elastic Registration Sliding: Template image boundary is mapped to reference image boundary, and boundary points can slide along the boundaries. u 1 u 2 u 2 u 1 u 1 (x,x 2 ) = u 2 (x 1,x) = 0, for x = 0,1 and x j [0,1]. u 2 (x,x 2 ) x 1 = u 1(x 1,x) x 2 = 0, Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 15 / 47

Elastic Registration Bending: Corners of template image boundaries are fixed to those of reference image boundaries, and the sides can bend inward or outward. u 1 u 2 u 1 u 2 u 2 (x,x 2 ) = u 1 (x 1,x) = 0, for x = 0,1 and x j [0,1]. u 1 (x,x 2 ) x 1 = u 2(x 1,x) x 2 = 0, Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 16 / 47

Elastic Registration Periodic: The template image is mapped onto a torus. u 2 u 1 u 1 u 2 u 1 (0,x 2 ) = u 1 (1,x 2 ), u 2 (x 1,0) = u 2 (x 1,1), for x j [0,1]. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 17 / 47

Elastic Registration Finite Difference Approximation The iterative algorithm involves computing the inverse of (I+γαA). The matrix A is obtained from finite difference approximation of A(u). For 2D case, A(u) = µ 2 u+(λ+µ) u (λ+2µ) 2 x 2 +µ 2 1 x 2 2 = 2 (λ+µ) x 1 x 2 2 (λ+µ) x 1 x 2 µ 2 x 2 +(λ+2µ) 2 1 x 2 2 [ ] u1 (x) u 2 (x) The above equation can then be approximated by finite difference method into the form Au(x). (15) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 18 / 47

Elastic Registration Notes: The above matrix is for one pixel of a 2-D image. For a d-dimensional image with N voxels, the u vector has dn entries, and the A matrix has dn dn entries! Many of the entries are 0. But there are still many non-zero entries. Direct inversion of (I+γαA) is possible only for small images. The entries in A also depend on the boundary condition used. With periodic boundary condition, FFT method can be used to implement an O(N logn) algorithm. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 19 / 47

Elastic Registration Non-zero patterns of the matrix A [Mod04]: (a) 2D case, 5 7. (b) 3D case, 5 7 4. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 20 / 47

Elastic Registration Example 1: Elastic registration of disk to C [Mod04]: (a) Reference image. (b) Template image. (c) Shaded template. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 21 / 47

Elastic Registration (d) Dirichlet. (e) Neumann. (e) Periodic. (g) Periodic. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 22 / 47

Elastic Registration Example 2: Elastic registration of hand image [Mod04]: (a) Reference image. (b) Template image. (c) Affine registered template. (d) Elastically registered template. Notes: Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 23 / 47

Fluid Registration Fluid Registration Based on simplified model of fluid. Elastic models: spatial smoothing of displacement field. Fluid models: spatial smoothing of velocity field. Let P(u) denote elastic potential. Then, fluid regularizer S(u) = P( u/ t). Fluid regularizer gives the equation [Mod04]: A(u) = µ 2 v+(λ+µ) v, v = u t. (16) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 24 / 47

Fluid Registration Example 1: Fluid registration of disk to C [Mod04]: (a) Elastically registered template. (b) Fluid registered template. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 25 / 47

Fluid Registration Example 2: Fluid registration of hand image [Mod04]: (a) Fluid registration only. (b) Affine + fluid registration. Notes: Fluid registration can produce more deformation than elastic registration. Non-rigid registration without linear pre-registration can cause too much deformation even though the difference is minimized. Linear (affine or similarity) pre-registration is needed to normalize the size, position, and orientation. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 26 / 47

Diffusion Registration Diffusion Registration Based on intensity gradient; non-physical. Diffusion regularizer is given by the equation [FM99, Mod04]: S(u) = 1 2 d l=1 ( u l ) T u l dx = 1 2 d l=1 u l 2 dx (17) with Neumann boundary condition: ( u l (x)) T n(x) = 0 (18) for image boundary points x, and outer normal unit vectors n at image boundary. This regularizer smoothes the deformation while minimizing oscillations of the components of the displacement u. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 27 / 47

Diffusion Registration Diffusion regularizer gives the equation [Fis05, Mod04]: Notes: A(u) = 2 u. (19) Diffusion registration can be regarded as a special case of elastic registration without the u term. Diffusion registration can be extended to velocity-based method like fluid registration. Discrete Cosine Transform method can be used to implement an O(N logn) algorithm. Additive Operator Splitting method can be used to implement an O(N) algorithm. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 28 / 47

Diffusion Registration Example 1: Diffusion registration of disk to C [Mod04]: (a) Elastically registered. (b) Diffusion registered. (c) Diffusion registered. (d) Velocity-based diffusion. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 29 / 47

Diffusion Registration Example 2: Diffusion registration of hand image [Mod04]: (a) Elastically registered. (b) Affine + diffusion registration. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 30 / 47

Demons Registration Demons Registration Diffusion registration is related to Thirion s demons registration [Thi98]. Basic ideas: Place demons at certain locations in the image. Demons decide whether movement of particles of template image reduces difference between reference and template images. Apply optical flow method to compute the required displacement. Apply low-pass (Gaussian) filtering to produce smooth solution. In ideal case, T u (x) = R(x). Apply optical flow method: dt u dt where v = x/ t. Then, = T u t +( T u) T v = 0. (20) ( T u ) T v = T u = T u (x,t) T u (x,t+1). (21) t Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 31 / 47

Demons Registration Template image is deformed to match reference image. So, we can regard T u (x,t+1) as R(x). A solution of the above equation is T u (x) v(x) = (T u (x) R(x)) T u (x) 2. (22) The above solution is undefined when T u = 0. To handle this situation, Thirion suggests including a non-zero term κ in the denominator: where κ = T u (x) R(x). v(x) = (T u(x) R(x)) T u (x) T u (x) 2 +κ 2 (23) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 32 / 47

Demons Registration Relationship to diffusion registration [Mod04]: Can formulate a diffusion registration with appropriate difference measure such that the derived force F(u) = v. Under mild conditions, a solution u(x) of the Euler-Lagrange equation is the convolution of F with a Gaussian kernel, i.e., Demons method. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 33 / 47

Curvature Registration Curvature Registration Elastic, fluid, and diffusion registrations are sensitive with respect to linear pre-registration. Rigid (or affine) pre-registration is needed to roughly align images before non-rigid registration. Curvature registration is less dependent on initial configuration of reference and template images. Difference measure is the sum-squared difference: D(R,T;u) = 1 [R(x) T u (x)] 2 dx. (24) 2 Curvature regularizer is given by S(u) = 1 2 d l=1 ( 2 u l ) 2 dx (25) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 34 / 47

Curvature Registration with Neumann boundary conditions u l = 2 u l = 0 (26) for image boundary points x. Curvature regularizer gives the equation A(u) = 2 2 u. (27) Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 35 / 47

Curvature Registration Notes: ( 2 u l ) 2 can be viewed as an approximation of curvature. Thus, the idea of the regularizer is to minimize the curvature of the components of the displacement. The discretized Acurv = A 2 diff. Discrete Cosine Transform method can be used to implement an O(N logn) algorithm. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 36 / 47

Curvature Registration Example 1: Deformed grid after curvature registration [Mod04]: (a) Reference image. (b) Template image. (c) Fluid registration. (d) Curvature registration. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 37 / 47

Curvature Registration Example 2: Curvature registration of disk to C [Mod04]: (a) Fluid registered. (b) Curvature registered. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 38 / 47

Curvature Registration Example 3: Curvature registration of hand image [Mod04]: (a) Curvature registration only. (b) Affine + curvature registration. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 39 / 47

Summary Summary Elastic registration Based on properties of elastic object. A(u) = µ 2 u+(λ+µ) u. Physically meaningful deformation. Small, local deformation. With periodic boundary condition, FFT method can be used to implement an O(N logn) algorithm. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 40 / 47

Summary Fluid registration Based on simplified properties of fluid. A(u) = µ 2 v+(λ+µ) v, v = u t. Physically meaningful deformation. Can produce very large deformation. With periodic boundary condition, FFT method can be used to implement an O(N logn) algorithm. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 41 / 47

Summary Diffusion registration Based on 1st order spatial derivatives of displacement. Non-physical model. Small deformation. A(u) = 2 u. Can be combined with fluid idea to obtain large deformation. Additive Operator Splitting method can be used to implement an O(N) algorithm. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 42 / 47

Summary Curvature registration Based on 2nd order spatial derivatives of displacement. Non-physical model. A(u) = 2 2 u. Kernel contains affine transformation. Small deformation. Can be combined with fluid idea to obtain large deformation. Discrete Cosine Transform method can be used to implement an O(N logn) algorithm. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 43 / 47

Summary Sample execution time [Mod04]: (a) Elastic. (b) Diffusion & curvature. Further notes: Software toolkit available, e.g., FLIRT [FM03]. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 44 / 47

Summary Further reading: More details on non-rigid registration methods: [Mod04]. Comparing various non-rigid registration methods: [Mod04], Sections 13.3, 13.4. How to choose parameter values: [Mod04], Section 13.5. Extensions of the methods: [Mod04], Section 13.7. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 45 / 47

Reference References I R. Bajcsy and S. Kovačič. Multiresolution elastic matching. Computer Vision, Graphics, and Image Processing, 46:1 21, 1989. R. Duda and P. Hart. Pattern Classification and Scene Analysis. Wiley, 1973. B. Fischer. On non-rigid medical image registration. www.ma.man.ac.uk/~shardlow/moir/fischer.pdf, 2005. B. Fischer and J. Modersitzki. Fast inversion of matrices arising in image processing. Numerical Algorithms, 22:1 11, 1999. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 46 / 47

Reference References II B. Fischer and J. Modersitzki. FLIRT: A flexible image registration toolbox. In J. C. Gee, J. B. A. Maintz, and M. W. Vannier, editors, Proc. 2nd Int. Workshop on Biomedical Image Registration (LNCS 2717), pages 261 270, 2003. J. Modersitzki. Numerical Methods for Image Registration. Oxford University Press, 2004. J. P. W. Pluim, J. B. A. Maintz, and M. A. Vierveger. Image registration by maximization of combined mutual information and gradient information. IEEE Trans. on Medical Imaging, 19(8):809 814, 2000. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 47 / 47

Reference References III J.-P. Thirion. Image matching as a diffusion process: An analogy with Maxwell s demons. Medical Image Analysis, 2(3), 1998. P. Viola and W. M. Wells III. Alignment by maximization of mutual information. Int. Journal of Computer Vision, 24(2):137 154, 1997. P. Viola. Alignment by maximization of mutual information. PhD thesis, MIT, 1995. Leow Wee Kheng (CS6240) Non-Rigid Image Registration III 48 / 47