Image Registration: Warping Without Folding

Transcription

1 Image Registration: Warping Without Folding Jeffrey A. Fessler EECS Department BME Department, Dept. of Radiology The University of Michigan UM BME 500 Oct 7, 2009 Acknowledgments: Se Young Chun, Matt Jacobson James Balter, Marc Kessler 1

2 Introduction Outline Image registration Enforcing / encouraging local invertibility (diffeomorphism) IEEE J. Selected Topics in Signal Processing, Feb Motion-compensated image reconstruction Conventional Model based Temporal regularization Image reconstruction (and registration) toolbox: fessler 2

3 Image Reconstruction Overview object f System L noise measured data y Reconstruction ˆf X-ray CT Sinogram X-ray CT Image 3

4 Image Reconstruction Formulations Static f( r) Dynamic f( r,t) contrast changes object motion (synergy with image registration) Image Reconstruction Research Topics Models for imaging physics Models for measurement statistics Models for object Algorithm formulation Algorithm acceleration Algorithm analysis / evaluation... 4

5 Why FBP (thin slice) MBIR (120 kvp, ma with modulation, 0.4 s rotation, pitch 1.375, slice thickness mm) 5

6 Part 1 Nonrigid image registration ensuring local invertibility 6

7 Image Registration Many applications, e.g., forensics, remote sensing, medicine... rigid transformations nonrigid transformations (warps) Example: Respiratory motion (application: radiotherapy planning) Target Source Inhale Exhale 7

8 Image registration: Overview Given two images (or image volumes): f( r) and g( r), r = (x,y,z), find a spatial transformation T( r), where T : R 3 R 3, such that f( r) is similar to the warped image g( T( r)) Usual steps: parameterize by α the spatial transformation: T( r; α) ( ) choose a similarity measure Ψ f( ),g( T( )) find optimal deformation parameters α numerically: ( ) ˆα = argmax Ψ f( ),g( T( ; α)) α Challenge: want estimated transformation T( r; ˆα) to be plausible. Typically we want it to be diffeomorphic, or topology preserving, or invertible, or at least locally invertible. 8

9 Image registration: Similarity measures sum of squared differences Ψ( f 1, f 2 ) = f 1 (r i ) f 2 (r i ) 2 i correlation ( )( ) Ψ( f 1, f 2 ) = f1 (r i ) f 1 f2 (r i ) f 2 i mutual information... Ψ( f 1, f 2 ) = MI( f 1, f 2 ) = H( f 1 )+H( f 2 ) H( f 1, f 2 ) 9

10 Image registration: B-spline deformations Nonrigid spatial transformation: T( r; α) = r+(d x ( r; α x ),d y ( r; α y ),d z ( r; α z )), }{{} deformation where α = (α x, α y, α z ) denotes unknown deformation coefficients. Tensor-product B-spline deformation model: d x ( r; α x ) = α x i jk β(x/m x i)β(y/m y j)β(z/m z k) i, j,k α y i jk β(x/m x i)β(y/m y j)β(z/m z k) d y ( r; α y ) = i, j,k d z ( r; α z ) = i, j,k α z i jk β(x/m x i)β(y/m y j)β(z/m z k) m x,m y,m z denote the knot spacing in each dimension. These spacings determine the spatial scale of the deformation. 10

11 Cubic B-spline Kernel 2/3 1/

12 B-spline deformations: Benefits differentiable (smooth) local support recursive filters for computations piecewise polynomial hierarchical Nonrigid image registration similarity measures usually have many local maximizers. To help find a good local maximum, one usually uses coarse-to-fine search. This is easy with B-spline deformations. (Thevenaz & Unser, IEEE T-IP, 2000) 12

13 B-spline deformations illustrated knot locations, mx=8 my=16 Knot locations Local support 13

14 B-spline deformations illustrated Invertible Not invertible 14

15 Invertibility When is r T( r) = r + d( r) a locally invertible transformation? By the inverse function theorem, it is sufficient for T to be continuously differentiable, and { } have positive Jacobian determinant: det T( r) > 0 for all r. Jacobian of transformation/deformation ( ) T( r) = r+ d( r) = x dx y dx z dx x dy y dy z dy x dz y dz z dz (mathematical theory vs practice) 15

16 Ensuring local invertibility in image registration We need to estimate B-spline deformation coefficients α subject to some local invertiblity constraint α C: ˆα = argmax Ψ(α). α C Ideal sufficient condition ensuring local invertibility for parametric deformation model: { { } } α C 0 = α : det T( r; α) > 0, r R 3. This condition is very difficult to implement. Conventional relaxed local invertibility condition: C 0 C 1 { { } } α C 1 = α : det T( r; α) > 0, r grid points. This condition does not ensure local invertibility everywhere. It is also computationally demanding. We seek simpler sufficient conditions for local invertibility: C C 0. 16

17 Unconstrained vs constrained optimization Source No constraint Jacobian constraint Target No constraint Jacobian penalty Image registration is an ill-posed problem. Jacobian constraint on grid required > 3 computation as unconstrained case. Nevertheless, some negative Jacobians remain (between grid points) because C 0 C 1. We need a simpler constraint that ensures positive Jacobian determinants everywhere. 17

18 A sufficient condition: Box constraints Simple lower/upper bounds on B-spline coefficients: { α C 3 = α : α x i jk m x K, α y i jk m y K, α z i jk m } z K, i, j,k, where K 2.05 in 2D and K 2.48 in 3D. Choi et al., 2000; Rueckert et al., MICCAI 2006 Fact: C 3 C 0. So constraining α C 3 ensures local invertibility everywhere. Box constraints are particularly simple for optimization. However, C 3 is a very restrictive set of deformations. Maximum displacement is only about half the knot spacing. Precludes even simple (large) global translations. 18

19 Proposed sufficient condition for invertibility Theorem (IEEE JSTSP, Feb. 2009): Suppose 0 k q < 1 for q {x,y,z}. Define the set: 2 C 4 = {α : m x k x α x i+1, j,k α x i, j,k m x K x, m y k y α y i, j+1,k αy i, j,k m yk y, If α C 4, then r R 3 : { } 1 (k x + k y + k z ) det T( r; α) m z k z α z i, j,k+1 α z i, j,k m z K z, α q i+1, j,k αq i, j,k m qk q for q = y,z, α q i, j+1,k αq i, j,k m qk q for q = x,z, α q i, j,k+1 αq i, j,k m qk q for q = x,y, i, j,k}. (1+K x )(1+K y )(1+K z )+(1+K x )k y k z + k x (1+K y )k z + k x k y (1+K z ). Corollary: { } Choosing k x = k y = k z = 1/3 ε ensures that 0 < det T( r; α), r. 19

20 Comparing sufficient conditions 1D example with two coefficients: α 1, α 2, for n = 2 (quadratic B-splines) α 2 2 C 0 = C 4 C α 1 20

21 Limitations of sufficient conditions 2D simulations using augmented Lagrange multiplier approach to enforce the constraint α C 3 or α C 4. Source Box constraint Proposed constraint Target Box constraint Proposed constraint Clearly C 3 C 0 and C 4 C 0. 21

22 Solution: composition of transformations Composing multiple transformations can overcome the limitations of sufficient conditions, e.g., T T α3 T α2 T α1 where α 1, α 2, α 3 C 4. Proposed constraint Proposed constraint Proposed constraint 22

23 Composition for box constraints Requires many more compositions: Rueckert et al., MICCAI 2006 Box constraint Box constraint Box constraint Box constraint Box constraint Box constraint Each of the 30 warps used many augmented Lagrangian iterations. Tradeoff: simplicity of constraint and its flexibility. 23

24 Simplyfing further via regularization Idea: replace constrained optimization ˆα = argmax α C 4 Ψ(α) with simpler unconstrained, but regularized, optimization: ˆα = argmax Ψ(α) γr(α) α where R(α) is zero if α C 4 but is large otherwise. This encourages local invertibility, but does not enforce it strictly. The regularization parameter γ controls the tradeoff between image similarity regularity of the deformation (local invertibility). 24

25 Proposed regularizer proposed quadratic α x i+1,j,k αx i,j,k α y i+1,j,k αy i,j,k α z i+1,j,k αz i,j,k α x i,j+1,k αx i,j,k α y i,j+1,k αy i,j,k α z i,j+1,k αz i,j,k α x i,j,k+1 αx i,j,k α y i,j,k+1 αy i,j,k α z i,j,k+1 αz i,j,k Interval constraints in C 4 replaced by piecewise quadratic penalty function of differences of neighboring B-spline coefficients. cf. conventional quadratic roughness regularization m x = m y = m z = 1,k x = k y = k z = 1/3, and K x = K y = K z = 4/3. 25

26 Regularization tradeoffs As regularization parameter γ # of negative Jacobian determinants so RMS difference between images log of (1 + # of negative Jacobian determinants) Proposed quadratic log of RMS difference between target and deformed images Proposed regularizer: good image similarity, few negative Jacobian determinants. 26

27 Regularization tradeoffs: Details Proportion of negative Jacobian determinants 15 x no constraint Jacobian proposed quadratic kim RMS error of deformed image 27

28 3D registration of CT inhale/exhale scans 3D CT scans of a cancer patient at exhale and inhale, for radiation treatment planning voxels. Source: Coronal Target: Coronal 28

29 CT inhale/exhale scans: Sagittal Source: Sagittal Target: Sagittal 29

30 3D registration results: Unconstrained No constraint: Sagittal No constraint: Sagittal 500 No constraint: Sagittal Relatively small difference image values, but many negative Jacobian determinants 30

31 3D registration results: Jacobian Jacobian penalty: Sagittal Jacobian penalty: Sagittal 500 Jacobian penalty: Sagittal Jacobian penalty based on C 1 (grid) Better behaved warp and reasonable difference image. But slow. 31

32 3D registration results: Proposed regularizer Proposed method: Sagittal Proposed method: Sagittal 500 Proposed method: Sagittal Proposed regularizer based on C 4. Tailored design of constraint/penalty: k x = k y = 1/4, k z = 1/2. Similar warp and difference image, but faster. 32

33 Quantification Method CPU time RMS difference # negative (seconds) (HU) Jacobians Unconstrained Jacobian penalty Proposed penalty Computation time per iteration (in seconds) at the finest level Regularization parameter adjusted empirically in both penalized cases to be the smallest value that yields no negative Jacobian determinants on the voxel grid. 3 multiresolution levels: knot spacings 8 pixels with downsampled images, 8 pixels with original images, 4 pixels with original images. 120 iterations of CG at each level. Work in progress to compose a couple coarse-scale deformations before refining to fine scale to reduce RMS differences. Code on web site: fessler 33

34 Diffeomorphisms: To be or not to be... Source Target Invertibility penalty Sliding at diaphragm / rib cage interface. Enforcing smoothness leads to bone warping. 34

35 Work in progress... Requiring the warp to be smooth everywhere seems suboptimal. One possible solution presented at SPIE Stay tuned... 35

36 Summary Simple condition for ensuring local invertibility everywhere Admits more deformations than conventional box constraints Simple regularizer requires comparable computation as unregularized image registration and much less computation than Jacobian determinant constraints / penalties Rigid structures (bones) Sliding tissue interfaces Parameter selection Computation (GPUs?) Performance characterization Open problems 36

37 Part 2 Model-based image reconstruction with motion compensation 37

38 Motion in image reconstruction Object being scanned: f( r,t) Measured data vector: y = (y 1,...,y M ) Static image reconstruction: Assume f( r,t) = f( r,t 0 ) = f( r) during scan. Estimate f( r) from measurements y. (Ill posed.) Dynamic image reconstruction ( List-mode data model) Assume each data point y i is acquired instantaneously at a corresponding time instant t i Relate y i to object at time t i, e.g., y i = Z a i ( r) }{{} physics f( r,t i )d r + }{{} ε i. statistics More generally: p(y i f(,t i )). Estimate f( r,t) from measurements y. (Even more ill posed!) 38

39 Gated data model Group data into K vectors, e.g., K respiratory phases: y 1,..., y K Assume f( r,t) is stationary during kth phase of data acquisition Relate y k to f k ( r) f( r,t k ) using physics and statistics From K data vectors y 1,..., y K, reconstruct object:? 39

40 Gated data model: Illustration f 1 y 1 f 2 y 2 Imaging System Reconstruction Method ˆf? f K y K 40

41 Gated data: Image reconstruction options Pool all data and ignore motion fast low noise variance motion induced blur Frame-wise: reconstruct each gate/frame separately: y k ˆf k simple high noise variance no motion blur (except within-gate motion) Frame-wise with post-reconstruction averaging (FW-PRA) map each reconstructed frame onto 1st frame, then average averaging should reduce noise should avoid motion blur if registration is accurate registration accuracy limited by noise in the individual gates FW-PRA with motion estimates from a separate modality use another modality (e.g., PET-CT) to estimate motion performance depends on consistency of motion between modalities Thorndyke et al., Med Phys,

42 Frame-wise with post-reconstruction averaging y 1 common target image 1 K ˆα 2 ˆf y 2 1 K ˆα K y K SSD registration 1 K Image Reconstruction Post Registration Consolidation 42

43 Forward model with motion Post-reconstruction averaging assumes an implicit model that relates the frames f 2,..., f K to the first frame f 1. We now make the (motion) model explicit: f k = W(α k ) f 1, k = 2,...,K. W(α k ) is the linear (!) transformation of the image values corresponding to motion α k. This model suggests additional image reconstruction approaches. 43

44 Linear interpolation and Nonrigid deformations B-spline interpolation model for continuous-space image: f( r) = f(x,y,z) = c nml β(x n)β(y m)β(z l) n,m,l Find coefficients c = {c nml } by prefiltering digital image f[n,m,l]. Nonrigid deformation of f : ) g( r) = f( T( r; α) = c nml β(t x ( r; α) n)β(t y ( r; α) m)β(t z ( r; α) l). n,m,l Resample warped image on grid (of target image): g( r j ) = c nml W r j ;n,m,l(α), j = 1,...,N n,m,l W r;n,m,l (α) β(t x ( r; α) n)β(t y ( r; α) m)β(t z ( r; α) l) In matrix-vector form, where g = {g( r j )} and f = { f[n,m,l]}: g = W(α)c, c = W 1 (0) f = g = W(α)W 1 (0) f. 44

45 Forward model with motion f 1 y 1 A ε 1 W(α 2 ) f 1 y 2 A ε 2 W(α K ) f 1 y K A ε K Motion Model Acquisition Process Measured Data 45

46 Gated data: More image reconstruction options Model-based image reconstruction with motion compensation given motion estimates, from FW-PRA or from separate modality, compensate for motion in reconstruction process. Qiao et al., PMB 2006; Taguchi et al., SPIE Model-based image reconstruction jointly with registration Jacobson & Fessler, IEEE NSS-MIC 2003, IEEE SSP 2003, ISBI 2006 Odille et al., MRM 2008 estimate jointly the first frame and the motion from all data Model-based image reconstruction with temporal regularization Mair et al., IEEE T-MI, 2006 estimate all frames and the motion between frames from all data 46

47 Model-based image reconstruction with motion compensation Given: motion estimates ˆα k for k = 2,...,K, from FW approach or from a separate modality, model for system physics / statistics: p(y k f k ) = p(y k W( ˆα k ) f 1 ). Perform penalized-likelihood (aka MAP) estimation of one image: ˆf 1 = argmax f 1 Ψ( f 1 ; ˆα 2,..., ˆα K ; y 1,..., y K ) Ψ( f 1 ; ˆα 2,..., ˆα K ; y 1,..., y K ) K logp(y k W( ˆα k ) f 1 ) βr( f 1 ). k=1 R( f) is optional regularization to control noise in ill-posed image reconstruction problems. For linear model with additive gaussian noise y k = A k f k +ε k : ˆf 1 = argmin f 1 K k=1 y k A k W( ˆα k ) f βr( f 1 ). 47

48 Motion compensated image reconstruction y 1 common source image argmax f1 Ψ( f 1 ; ˆα 2,..., ˆα K ; y 1,..., y K ) ˆα 2 ˆf 1 y 2 ˆα K y K Image Reconstruction SSD registration Post Registration Consolidation 48

49 Joint image reconstruction / registration Previous approach used possibly suboptimal motion estimates: ˆf 1 = argmax f 1 Ψ( f 1 ; ˆα 2,..., ˆα K ; y 1,..., y K ) Alternative: jointly estimate one image and K 1 deformation parameters: ( ˆf 1, ˆα 2,..., ˆα K ) = argmax f 1,α 2,...,α K Ψ( f 1 ; α 2,..., α K ; y 1,..., y K ) K Ψ( f 1 ; α 2,..., α K ; y 1,..., y K ) = logp(y k W(α k ) f 1 ) βr( f 1 ) k=1 Natural optimization strategy is to alternate between: updating image estimate ˆf 1 using current motion parameters, updating motion estimates { ˆα k } using current image estimate. Can initialize motion parameters using frame-wise method. 49

50 Joint estimation illustrated y 1 ˆα y 2 argmax Ψ( f 1 ; α 2,..., α K ; y 1,..., y K ) ˆf 1 y K Goal: find image estimate and motion parameters that best fit all measured data. 50

51 Motion-compensated temporal regularization Previous joint estimation approach: ( ˆf 1, ˆα 2,..., ˆα K ) = argmax f 1,α 2,...,α K Ψ( f 1 ; α 2,..., α K ; y 1,..., y K ) Alternative approach based on temporal regularization: ( ˆf 1,..., ˆf K ; ˆα 2,..., ˆα K ) = argmax f 1,..., ˆf K ;α 2,...,α K Ψ( f 1,..., f K ; α 2,..., α K ; y 1,..., y K ) Ψ( f 1,..., f K ; α 2,..., α K ; y 1,..., y K ) = K logp(y k f k ) βr( f k ) k=1 K γ f k+1 W(α k ) f k 2 k=2 } {{ } temporal regularization with motion effects Pro: no warp in log-likelihood. Con: more unknowns; γ choice? 51

52 Ten-Gate 3D PET Simulation (a) True Image (b) Mean Pre Image of Ungated Data 80K total counts/axial mm and 30% randoms (ECAT HR+), divided across 10 gates. Derived from 17 slices of real thorax anatomy. B-spline deformations (11x14x5x3 control grid), derived from helical CT scans at multiple inspirations (Matt Jacobson, 2006 thesis) 52

53 Sample Reconstructed Images (a) True Activity Image (b) PL with Known Motion (c) Fully Joint Estimation (JEDM) (d) Frame Wise Semi Statistical (FWPR PLC) (e) Frame Wise Post Averaging (FWPR PA) (f) Ungated PL 53

54 Mean Reconstructed Images (a) True Activity Image (b) PL with Known Motion (c) Fully Joint Estimation (JEDM) (d) Frame Wise Semi Statistical (FWPR PLC) (e) Frame Wise Post Averaging (FWPR PA) (f) Ungated PL 54

55 Lesion Recovery Comparison Study Parameters: 80K total counts/axial mm, 10 gates, 30% randoms Known Motion JEDM FWPR PA FWPR PLC Ungated 60 Cumulatively Averaged %Error No. of Realizations 55

56 Motion Tracking Performance Front Back motion Left Right motion Axial motion True Motion JEDM FWPR PLC FWPR PA True Motion JEDM FWPR PLC FWPR PA 20 True Motion JEDM FWPR PLC FWPR PA Lesion Position (mm) Lesion Position (mm) Lesion Position (mm) Gate No Gate No Gate No. 56

57 Regularization Using PET-CT Side Info. Relax regularization strength in neighborhood of lesion. (a) (b) 57

58 Regularization Using PET-CT Side Info. (cont d) (a) Uniformly Penalized Ungated PL (b) Uniformly Penalized JEDM (c) JEDM with Unpenalized ROI Uptake Error = 54.65% Uptake Error = 36.89% Uptake Error = 13.85% Very weak use of boundary side information = robust to mis-registration. 58

59 Summary Several possible methods for motion-compensated image reconstruction Model-based approaches such as joint estimation have potential Repeated motion estimation steps necessitate simple invertibility regularizers More work needed on algorithms, acceleration, evaluation,... 59