DRAMMS: Deformable Registration via Attribute Matching and Mutual-Saliency weighting

DRAMMS: Deformable Registration via Attribute Matching and Mutual-Saliency weighting Yangming Ou, Christos Davatzikos Section of Biomedical Image Analysis (SBIA) University of Pennsylvania

Outline 1. Background 2. Motivations 3. Framework 4. Methods 5. Results 6. Discussions 2

1. Background Definition of Registration Image Registration is the process of finding the optimal transformation that aligns different imaging data into spatial correspondence. [Maintz & Viergever 98, Lester & Arridge 99, Hill 01, Zitova 03, Pluim 03, Crum 04, Holden 08] Source (Subject) Target (Template) S2T (overlaid on T) Transformation (Deformation Field) 3

1. Background Registration Literature Division of Most Registration Methods: Category 1 Landmark/feature-based Category 2 voxel-wise (intensity-based) DRAMMS 4

1. Background Registration Literature (1) Expected: General-purpose registration methods! brain A brain B heart A heart B breast A breast B prostate A prostate B brain heart breast prostate Category 1: Landmark/feature-based methods [Davatzikos 96, Thompson 98, Rohr 01, Johnson 02, Shen 02, Joshi 00, Chui 03,...] Pros: 1) Intuitive; 2) Fast; Not suitable for general-purpose [Figure from Rohl 03]. Cons: 1) Errors in landmark detection & matching; 2) Task-specific: different registration tasks need different landmark detection methods. 5

1. Background Registration Literature (2) Category 2: (Intensity-based) Voxel-wise methods [Christensen'94, Collins'94, Thirion'98, Rueckert'99, Vercauteren'07, Glocker'08,...] Images Under Registration Joint Histogram After Reg. A B A2B white Assumption: A [figure from Rueckert 99] B black black A2B white white B Consistent relationship between intensity distributions black black white B [figures from Papademetris] Pro: General-purpose registration methods (only rely on intensities). Con: 1) 2) => motivations for DRAMMS 6

Outline 1. Background 2. Motivations 3. Framework 4. Methods 5. Results 6. Discussions 7

2. Motivations (1): Why? Challenge 1: No consistent relationship in intensity distributions Histological Image (Prostate) MR Image (Same Prostate) white black black black intensity-based voxel-wise methods (e.g. MI) fail. Matching Ambiguity Black White Black Reason: Matching ambiguity <= characterizing voxels only by intensities. Not Distinctive! 8

2. Motivations (1): How? Proposed Solution to Challenge 1: - To reduce matching ambiguity, 1-dim image intensity => high-dim attribute vector Similarity map (by attributes) High similarity Low similarity Attribute Matching DRAMMS 9

2. Motivations (2) Challenge 2: Partial loss of correspondence Histological Image (Prostate) MR Image (Prostate) Normal Brain Brain w/ lesion Inspiration 2: A continuous weighting mechanism for all voxels: -Weight high for voxels able to establish reliable correspondence; => let them drive the registration -Weight low for voxels not able to establish reliable correspondence. => reduce their negative impact to the registration Mutual-Saliency weighting DRAMMS 10

Outline 1. Background 2. Motivations 3. Framework 4. Methods 5. Results 6. Discussions 11

Framework 2. Mutual-Saliency 1. Attribute-Matching Deformable Registration via A T? B 1. Attribute Matching u T(u) To reduce matching ambiguities 2. Mutual-Saliency weighting To account for loss of correspondence DRAMMS 12

Outline 1. Background 2. Motivations 3. Framework 4. Methods 4.1. Attribute Extraction and Selection 4.2. Mutual-Saliency Weighting 4.3. Implementation 5. Results 6. Discussions 13

4.1.1. Attribute Extraction Ideal Attributes 1) Generally Applicable: to diverse registration tasks; 2) Discriminative: voxels similar iff true correspondence. Recent work [Shen and Davatzikos 01, Liu 02, Xue 04, Verma 04, Wu 07, etc] - Intensity attributes - Edge attributes - Tissue membership attributes (based on segmentation) - Geometric moment invariant attributes - Wavelet attributes - Local histogram attributes 14

4.1.1 Attribute Extraction Gabor Attributes A(x) A (0) (x) x A (1) (x) A (2) (x) A (3) (x) Gabor filter bank (multiscale, multi-orientation) 15

Why Gabor Attributes? Reason 0: Sometimes, maybe one reason is enough Dennis Gabor (1900-1979) Nobel Prize in Physics (1971) 16

Why Gabor Attributes? Reason 1/3: General Applicability brain heart breast prostate Success in Texture segmentation [e.g., Jain 91]; Cancer detection [e.g., Zhang 04]; Prostate tissue differentiation [e.g., Zhan 06]; Brain registration tasks [e.g., Liu 02, Verma 04, Elbacary 06]; 17

Why Gabor Attributes? Reason 2/3: Multi-scale and Multi-orientation. characterize voxels distinctively Original Image scale scale Gabor Attributes orientation 18 orientation

Why Gabor Attributes? Reason 3/3: Suitable for Registration Original Image High Freq. Gabor Attributes Edge maps => relatively independent of intensity distributions Low Freq. Gabor Attributes Smoothed (coarse) version => reduce local minimum in reg.. 19

Gabor Attributes characterize voxels distinctively Special Points Ordinary Points 20

4.1.2. Select Optimal Gabor Attributes Why? 1) Non-orthogonality among Gabor filters redundancy; 2) Attribute vector A( ) being too long computational expensive. How? Step 1: Select training voxel pairs: Mutual- Saliency Step 2: Select attribute on training voxel pairs: Training voxel pairs Mutual- Saliency by iterative backward elimination and forward inclusion. 21

Role of Gabor Attributes and Optimal Gabor Attributes distinctiveness 22

Role of Gabor Attributes and Optimal Gabor Attributes distinctiveness 23

Outline 1. Background 2. Motivations 3. Framework 4. Methods 4.1. Attribute Extraction and Selection 4.2. Mutual-Saliency Weighting 4.3. Implementation 5. Results 6. Discussions 24

Framework 2. Mutual-Saliency 1. Attribute-Matching Deformable Registration via A T? B 1. Attribute Matching u T(u) To reduce matching ambiguities 2. Mutual-Saliency weighting To account for loss of correspondence DRAMMS 25

4.2. Mutual-Saliency weighting Recent work [Bond 05, Wu 07, Mahapha 08] Their approach: Higher weights for more salient regions Their assumption: Salient regions more likely to establish reliable correspondence. Saliency in one image => Matching reliability between two images? A counterexample Our work: saliency in one image => mutual-saliency b/w two images [Anandan 89, McEache 97] Directly measure matching reliability 26

4.2. Mutual Saliency (MS) weighting Idea: True correspondence should similar to each other; not similar to anything else. similarity Delta fun. u Calculation of MS: T(u) Reliable matching High MS value where 27

Role of Mutual-Saliency Map Account for partial loss of correspondence Registration without MS map Source image Target image MS map Registration with MS map 28

Outline 1. Background 2. Motivations 3. Framework 4. Methods 4.1. Attribute Extraction and Selection 4.2. Mutual-Saliency Weighting 4.3. Implementation 5. Results 6. Discussions 29

4.3. Implementation Optimized and regularized by Free Form Deformation (FFD) model [Rueckert 99] Diffeomorphism FFD [Rueckert 06] Multi-resolution to reduce local minima Gradient descent optimization Implemented in C Run on 2.8G CPU, Unix OS 30

Outline 1. Background 2. Motivations 3. Framework 4. Methods 5. Results 5.1. Cross-subject registration; 5.2. Multi-modality registration; 5.3. Longitudinal registration; 5.4. Atlas construction. 6. Discussions 31

5.1. Cross-subject Registrations A2B deformation Brain A (Subject) B (Template) A2B deformation Cardiac Evaluate registration accuracy by mean sq. diff. (MSD) and corr. coef. (CC) between registered image and target image. 32

5.1. Cross-subject Registrations Observations: 1) In images that intensity-based method can register, attribute matching increased registration accuracy considerably; 2) Each of DRAMMS components provides additive improvement for registration accuracy. 33

5.2. Multi-modality Registrations Human Prostate Histology MR Histology2MR Mutual-saliency Histological Joint histogram after registration MR 34

5.2. Multi-modality Registrations Mouse Brain Histology MR Histology2MR Mutual-saliency Joint histogram after registration 35

5.3. Longitudinal Registration 36

5.4. Atlas Construction Images from 30 training subjects template 37

5.4 Atlas Construction (cont.) By intensity-based FFD (mutual-information) By DRAMMS 38

5.4. Atlas Construction (cont.) Lesion Low MS weight 39

5.4. Atlas Construction (cont.) Mean Mutual-Saliency Map in 3D 40

Outline 1. Background 2. Motivations 3. Framework 4. Methods 5. Results 6. Discussions 41

Discussions DRAMMS: a general-purpose registration method; Diffeomorphism Improves MI-based methods, especially when 1) no consistent relationship between intensity distributions; 2) loss of correspondence 42

DRAMMS A bridge between two categories of methods Category 1 Landmark/feature-based Category 2 voxel-wise (intensity-based) DRAMMS 43

DRAMMS bridge 1 DRAMMS Attribute Matching Category 1 Landmark-based Category 2 voxel-wise Every voxel will become a landmark to some extent. Still using all voxels 44

DRAMMS bridge 2 DRAMMS Mutual-Saliency weighting Category 1 Landmark-based Category 2 voxel-wise Weight = 1 for landmarks 0 otherwise Weight = 1 for all voxels 45

Take-home message Deformable Registration via 1. (optimal) Attribute Matching To reduce matching ambiguities 2. Mutual-Saliency weighting To account for loss of correspondence DRAMMS 46

Thank you! Code to be available: (Lab) https://www.rad.upenn.edu/sbia/ (Personal) https://www.rad.upenn.edu/sbia/yangming.ou/ 47