Enhao Gong, PhD Candidate, Electrical Engineering, Stanford University Dr. John Pauly, Professor in Electrical Engineering, Stanford University Dr.

Transcription

1 Enhao Gong, PhD Candidate, Electrical Engineering, Stanford University Dr. John Pauly, Professor in Electrical Engineering, Stanford University Dr. Greg Zaharchuk, Associate Professor in Radiology, Stanford University

2 Prof. John Pauly Dr. Morteza Mardani Prof. Greg Zaharchuk Dr. Jia Guo

3

4 CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs

5 Medical imaging medical intervention interior clinical analysis and visual representation PET fmri Structural Imaging Functional Imaging Image source: Wikipedia, NIVIDA

6 Image Reconstruction From anatomy to image Reconstruction and Restoration Denoising and Super-resolution Pathology Detection From image to labels Tumor Segmentation Pathology Detection Diagnosis Assistance From pathology to diagnosis Prescription and Treatment Prognostic Prediction Scanner Prognosis Analytics Diagnosis Images Enhancement & Augmentation Pathology Treatment Image source: blogs.nvidia.com

7 Magnetic Resonance Imaging ü Great tissue contrast for distinguish normal tissue vs pathology ü No exposure to ionizing radiation q Samples in Fourier domain (k-space) Need some magic transform to convert to image domain Scanner Signal Image

8 Reconstruction of sparse sampled complex signal in Fourier domain (k-space) of images Much harder image restoration tasks Under-sampled k-space Reconstruction Recovered k-space

9 Reconstruction of sparse sampled complex signal in Fourier domain (k-space) of images Harder image restoration tasks Under-sampled k-space Reconstruction Recovered k-space Image Super-resolution Image Restoration

10 Reconstruction of sparse sampled complex signal in Fourier domain (k-space) of images Solved with constrained optimization Under-sampled k-space Reconstruction Recovered k-space Reconstruction Model Image: X Acquisition Model: E Measured Signal: Y=EX Solve inverse problem with optimization Consistent with Signal Model Regularizations (Sparsity, Low-rank, Dictionary) Solving using Iterative Optimization

11 Case #1 Multi-contrast (structure) MRI reconstruction Original Proposed Post-Diamox

12 By Courtesy, Center for Advanced Functional Neuroimaging, Stanford

13 Initial Recon Ground-truth Recon Sequential Jointly+Local PatchàImage Intermediate reconstruction Sequential and independent Train Local deep network for patch-patch regression Jointly improve multicontrast local patch Image synthesis Generate Improved reconstruction

14 Initial Recon Ground-truth Recon Sequential Jointly+Local PatchàImage Intermediate reconstruction Sequential and independent Train Local deep network for patch-patch regression Jointly improve multicontrast local patch Image synthesis Generate Improved reconstruction

15 CNN Sequential Jointly+Local PatchàImage Intermediate reconstruction Sequential and independent Train Local deep network for patch-patch regression Jointly improve multicontrast local patch Image synthesis Generate Improved reconstruction

16 CNN Sequential Jointly+Local PatchàImage Intermediate reconstruction Sequential and independent Train Local deep network for patch-patch regression Jointly improve multicontrast local patch Image synthesis Generate Improved reconstruction

17 CNN Convention Methods Intermediate reconstruction Sequential and independent Using Deep Learning Train Local deep network for patch-patch regression Jointly improve multicontrast local patch Data Augmentation Image synthesis Generate Improved reconstruction

18 method details Convolutional Encoder-Decoder with bypasses O. Ronneberger, et al M. Uecker et al. MRM M. Uecker, et al. BART Convolutional Encoder-Decoder with downsample poolings H. Noh, et al. ICCV215

19 CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs residual learning results Training Initial Residual Predicted Residual Testing Reduced Residual Initial Residual Predicted Residual Reduced Residual

20 CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs residual learning results Train Test Residual Fitting RMSE Residual R2 Norm (Before) Residual R2 Norm (After) Artifact Reduction (%) % 17% Training Initial Residual Predicted Residual Testing Reduced Residual Initial Residual Predicted Residual Reduced Residual

21 clinical results in routine multi-contrast neuro-imaging protocol ü Faster scan with less artifacts ü Accelerate FLAIR/GRE with T1w/T2w scan ü Reconstruction with sharable information ü Acceleration with preserved pathology Improved GRE reconstruction on a hemorrhage subject

22 Case #2 Perfusion MRI reconstruction Original Proposed Post-Diamox

23 Background Arterial Spin Labeling(ASL) ( ) x 1= Blood flow map

24 Detail Methods and innovations Extend conventional spatial-frequency Denoising Non-Local-Mean (NLM) Filtering Extend with multi-contrast information NLM Denoising Train deep network models for image-to-image regression tasks Ground-truth with higher resolution and SNR End-to-end Denoising and super-resolution Use multi-contrast images/patches as inputs More robustness, accuracy and avoid overfitting A. Buades, et.al. CVPR 25

25 Step 1: Nonlinear ASL signal denoising using Non-Local (NLM) and MulC-contrast Guided Filter Nex=6 High SNR Ref ASL Nex=1 Low SNR ASL raw Training w ASL = w ASL = 1 w ASL = 5 w ASL = 2 w ASL = 1 w ASL = 5 original recon more regulariza.on using nonlinear denoising Step 2: Generate patches from High-SNR Ref. ASL, Low-SNR raw ASL, multi-level denoised ASL and anatomical MR images Nex=6 High SNR Ref ASL Nex=1 Low SNR ASL Denoised ASL with different w ASL T2wFSE PDw Step 3: Training deep network to learn the nonlinear image restoration from multi-contrast patches Input Input Patches Patches Deep Convolutonal-Deconvolu4onal Neural Network by-passes connec.ons Multi-contrast patches Step 4: Generate the restored image from stored patches Output Patches More Layers Cost function Output Compare vs. Ref Output: restored high-snr ref High SNR Ref ASL Restored from Low SNR Diff Original Low-SNR Original Diff

26 Step 1: Nonlinear ASL signal denoising using Non-Local (NLM) and MulC-contrast Guided Filter Nex=1 Low SNR ASL raw Applying w ASL = w ASL = 1 w ASL = 5 w ASL = 2 w ASL = 1 w ASL = 5 original recon more regulariza.on using nonlinear denoising Step 2: Generate patches from High-SNR Ref. ASL, Low-SNR raw ASL, multi-level denoised ASL and anatomical MR images Nex=1 Low SNR ASL Denoised ASL with different w ASL T2wFSE PDw Step 3: Training deep network to learn the nonlinear image restoration from multi-contrast patches Input Input Patches Patches Deep Convolutonal-Deconvolu4onal Neural Network by-passes connec.ons Multi-contrast patches Step 4: Generate the restored image from stored patches Output Patches More Layers Cost function Output Output: predicted high-snr patch High SNR Ref ASL Restored from Low SNR Diff Original Low-SNR Original Diff

27 Results

28 Note: RMSE=Root-Mean-Squared-Error (normalized) Results Deep Learning Model Multi-contrast information for regularized de-noising High SNR ASL Synthetic ASL T2 weighted Proton density Low SNR ASL RSME 1% RSME 29% Diff map vs High SNR + 4-fold time reduction 3-fold RSME improvement Diff map vs High SNR

29 Clinical results ü Improved SNR and resolution in perfusion image and transit-delay image ü Pathology is well preserved for pa4ents ü More pa4ents datasets and human ra4ngs are in progress Moyamoya, pre and post Diamox Original Proposed Original Proposed Pre Post-Diamox

30 Case Study: Enhanced MRI Recon with Deep Learning Summary: Improved Efficiency and Accuracy for clinical applications Case #1 Multi-contrast (structure) MRI reconstruction 4x~6x acceleration Preserved pathology in clinical scans More efficient reconstruction for clinical applications Case #2 Perfusion MRI reconstruction 4x~6x acceleration Preserved pathology in clinical scans Better SNR and resolution for clinical settings Original Proposed Post-Diamox

31 How to avoid overfitting? Augmentation with image transform: Cropping + Rotation + Shifting + Flipping Augmentation with image distortion: co-registration + distortion with motion field A. Krizhevsky, et al. NIPS 212 S. Hauberg, et al. Arxiv 215

32 Visualize network parameters How network works Example Kernels: For 3D Residual Space: Extract smoothness For 2D Structure Space: Extract structures Potential Improvements

33

34 4x Acceleration applying spatial-frequency filtering (NLM) Reconstruction Speed: 1~1minà1sec/slice; big advantage over iterative methods!

35 CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs Discussion: Network Design in MRI Reconstruction Results on GAN based CS-MRI on MR Phantom (submitted to NIPS217) [1] K. Hammernik, et al. ISMRM 217 [2] I. Goodfellow, et al. NIPS 214 [3] D. Pathak, et al. CVPR 216 [4] A. Bola, et al. Arxiv 217 [5] M. Mardani, et al. Arxiv 217

36 Conclusion: Enhance multi-contrast MRI reconstruction with Deep Learning and NVIDIA GPUs Deep Learning approach to solve medical imaging reconstruction Deep Learning model learns to remove artifact and fusing multi-contrast information Better efficiency and accuracy for clinical applications and diagnosis NVIDIA GPUs are key for computational acceleration 4x acceleration for Non-Local-Mean Denoising using CUDA 1x acceleration for Model Inference using CUDA, CuDNN Benefit to medical diagnosis Multi-contrast scans: 4x faster acquisition, preserve pathology Single-delay Perfusion (ASL) MRI: 4x faster acquisition, 7.78dB SNR gain Multi-delay Perfusion (ASL) MRI: 6x faster acquisition, 5% better resolution Examples shown in clinical exams for hemorrhage and stroke diagnosis.

37 Showcase of work in the direct big win in the content generation and augmentation Image Reconstruction From anatomy to image Reconstruction and Restoration Denoising and Super-resolution Pathology Detection From image to labels Tumor Segmentation Pathology Detection Diagnosis Assistance From pathology to diagnosis Prescription and Treatment Prognostic Prediction Scanner Prognosis Analytics Diagnosis Images Enhancement & Augmentation Pathology Treatment Image source: blogs.nvidia.com

38 Image reconstruction/restoration Great attention from radiology community

39 Prof. John Pauly Dr. Morteza Mardani Prof. Greg Zaharchuk Dr. Jia Guo