STAR: Spatio-Temporal Architecture for super-resolution in Low-Dose CT Perfusion

Transcription

1 STAR: Spatio-Temporal Architecture for super-resolution in Low-Dose CT Perfusion Yao Xiao 1, Ajay Gupta 2, Pina C. Sanelli 3, Ruogu Fang 1 1 University of Florida, Gainesville, FL 2 Weill Cornell Medical College, New York, NY 3 Northwell Health, Manhasset, NY Abstract. Computed tomography perfusion (CTP) is one of the most widely used imaging modality for cerebrovascular disease diagnosis and treatment, especially in emergency situations. While cerebral CTP is capable of quantifying the blood flow dynamics by continuous scanning at a focused region of the brain, the associated excessive radiation increases the patients risk levels of developing cancer. To reduce the necessary radiation dose in CTP, decreasing the temporal sampling frequency is one promising direction. In this paper, we propose STAR, an end-toend Spatio-Temporal Architecture for super-resolution to significantly reduce the necessary scanning time and subsequent radiation exposure. The inputs into STAR are multi-directional 2D low-resolution spatiotemporal patches at different cross sections over space and time. Via training multiple direction networks followed by a conjoint reconstruction network, our approach can produce high-resolution spatio-temporal volumes. The experiment results demonstrate the capability of STAR to maintain the image quality and accuracy of cerebral hemodynamic parameters at only one-third of the original scanning time. 1 Introduction Computed tomography perfusion (CTP) is one of the most widely used imaging modality for disease diagnosis and therapeutics planning such as stroke and oncology[3,11], especially in emergency situations. Cerebral CTP scans a focused brain region for a prolonged amount of time to quantify the blood flow dynamics in the brain. However, a single 40-second cerebral CTP scan can subject the human body to as much as a year s worth of radiation exposure from natural surroundings [7]. In contrast, a chest x-ray would be on par with about ten days worth of exposure. Also, by repetitively scanning a particular region of the brain, there is always a chance the patient may experience the effects of excessive exposure to radiation. Effects such as hair loss (epilation) and skin reddening (erythema) have been reported in a CT brain perfusion over-exposure incident [13]. Risks such as cancer and congenital disabilities are also within public concern [2]. Solutions such as lowering the radiation dose will increase image noise [8] and optimizing a CT scan system will increase the cost. Significant research continues with the goal of reducing radiation exposure from CTP scans. Corresponding author.

2 2 Y. Xiao, A. Gupta, P. Sanelli and R. Fang In recent years deep learning has achieved significant performance improvement in super-resolution (SR) and image reconstruction [1,5,10,9]. Deep learning models, especially convolutional neural network (CNN) structure, allows the use of learning from low-resolution (LR) image input to reconstruct a high-resolution (HR) output, thus providing a practical solution for image reconstruction. However, most of the super-resolution frameworks using deep learning techniques favor to focus on the 2D natural image SR, since adding the temporal dimension is more challenging, especially with medical images. In this work, we aim to address the challenges in temporal SR and demonstrate the feasibility of our CNN based framework in cerebral CTP for the purpose of increasing scanning intervals and thus reducing the overall scanning time, such that reducing the radiation amount to the patients. Contributions: This paper proposes STAR an end-to-end Spatio-Temporal Architecture for super-resolution, and we validate this framework on the clinical cerebral CTP dataset. The proposed STAR architecture consists of two main components: single-directional networks (SDN) and a multi-directional conjoint CNN. SDNs can capture both spatial and temporal features from CTP slices simultaneously by different cross-section patch representations, and the multidirectional conjoint CNN can integrate various single-directional information to reconstruct the final HR spatio-temporal cerebral CTP sequences. Specifically, the contributions are three-fold: (1) Our patch representation layer extracts CTP features from both spatial and temporal dimension. With the cross-section information, the STAR model can represent both spatial and temporal details for CTP image SR, especially for improving CTP sequence temporal resolution. (2) We integrate multiple SDNs with a conjoined multi-directional network to boost the performance for 3D spatio-temporal CTP data. (3) STAR can reduce the scanning time to only one-third of the current method with comparable image quality and accuracy, regarding peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) of the hemodynamic maps for disease diagnosis. 2 Methodology In this section, we first introduce our patch representation schema for generating 2D spatio-temporal LR inputs. Then, we explain how to improve the spatial and temporal resolution simultaneously for each cross-section by the SDNs. Last but not least, we describe our conjoint model to synthesize multi-directional inputs into a spatio-temporal HR image sequence. Patch Representation. The input 2D LR patches for image SR are generated from 3D cerebral CTP slices (X Y T ). X and Y represent the 2D spatial dimensions and T indicates the temporal dimension of the sequence. We also consider the diagonal (D) direction from X Y as one spatial dimension, where X and Y are equal in our data. We extract 2D patches on the X Y direction as well as on one of the spatial directions with T dimension: X T, Y T, and D T. With these cross-section data, we can re-scale them on spatial direction, temporal direction, or both spatial and temporal directions to create 2D LR patches. For instance, a 2D spatio-temporal patch represents a single spatial

3 STAR: Spatial-Temporal Architecture for super-resolution 3 Fig. 1: Single Directional Network. This model learns the difference between the LR inputs and the HR ground truth image from each cross-section from the high-dimensional medical images. By adding a skip connection between with the input image to the reconstruction layer, the model learns the reside between the LR and HR images. The convolution and ReLU layer occur in pairs and we set 64 filters with size 3 3 for each convolutional layer. vector change through time, re-scale on temporal dimension allows the change of CTP scanning time in a particular ratio. After feeding these LR patches into the convolution layers for learning the spatio-temporal details, HR output will be generated in the testing stage based on the captured features. Single Directional Network. The Single Directional Network takes the input patches from one of the four combinations of spatial and temporal dimensions: X Y, X T, Y T, and D T. Selecting a proper CNN model is a critical component for learning spatio-temporal features for SR problems. We adapt the very deep network for super-resolution (VDSR) [5] with optimized network structure to the SDN (See Fig 1) due to its high performance in 2D natural image SR. With numerous small filters of size 3 3 in the convolution layers the deep net architecture not only captures the detailed image information but also reduces the computational complexity [12]. The convolution layers of SDN exploit the spatio-temporal information over large cross-section regions by cascading small filters many times. A filter is an integral component of the layered architecture. It refers to an operator applied to the entire image which transforms the information encoded in the pixels. We set 64 filters of size 3 3 in the convolutional layers where a filter operates on the 3 3 region of the 2D input patches. The first layer of convolution operates on the spatio-temporal patches directly to obtain the feature maps, while the kernels in middle layers are convolved with the results from the previous layer. The computed intermediate j th feature map f (l) j for the middle layer l are calculated by convolving kernels wkj l (l 1) with the output feature maps f k from the previous layer l 1, that is f (l) j = 1 K K k=1 f (l 1) k wkj l, where is the convolution operator. By padding zeros for every convolutional layer during training, we ensure our output size is the same as the input. The ReLU activation layer is used after each convolutional layer; it ensures only certain features are most relevant and will be passed to the next convolution layer. It s activation function max(0, x) defines the output of a node given an input or set of inputs x.

4 4 Y. Xiao, A. Gupta, P. Sanelli and R. Fang Fig. 2: STAR Architecture. The last layer of SDN is for CTP image reconstruction. A reconstruction function y = φ(p, f (L) ) is responsible for constructing HR outputs. In this function, p denotes the 2D patches result from the middle convolution layers, and φ is the reconstruction function that sums up the predicted residuals and LR inputs to generate the HR outputs. We also set a high learning rate and apply residual learning to accelerate the convergence and ensure a less training time. STAR Architecture. The Single Directional Network only extracts features from one of the directions: Y T, X T, D T or X Y. By simply stacking the output from various cross-sections into a spatio-temporal volume, the subvoxel information and the contextual cues from different planes are missed. Thus we enhance the SDN by integrating different cross-sections together into spatiotemporal volume through a conjoint layer and another CNN architecture, with the goal of preserving the complementary inter-directional information. Fig. 2 visualizes the proposed STAR model. In this model, the left side shows the extraction of 2D patches through the four directions: Y T, X T, D T and X Y. Followed by the arrows, we feed those patches into a single dimensional network S (L) i respectively, where i = 1, 2, 3, 4 is the index of different directional inputs and L indicates the number of convolution layers. After the reconstruction of the perfusion slices P i through a single directional network S (L) i, we calculate the mean M = mean(φ(p i, S (L) i )) of all directions output in the conjoint layer. In the end, we supply another deep neural network for the conjoint learning, and the result from that is the final HR CTP slices. The advantage of combining different direction spatial-temporal features can be seen from two aspects. On one hand, the sub-voxel information and contextual cues from different planes of brain CTP volume can be learned to alleviate the bias that is caused by only learning from one direction. On the other hand, the central anatomical structure of the brain can always be captured from different directions. This ensures the pixels in the same area of the brain will be reinforced after the conjunction; providing complementary details to overcome the blurring caused by the bicubic interpolation for LR patches generation.

5 STAR: Spatial-Temporal Architecture for super-resolution 5 ssvd CBF CBV PSNR SSIM PSNR SSIM Bicubic Spat-SDN Temp-SDN STAR Table 1: PSNR Comparison for Perfusion Maps that are generated by different methods. 3 Experiments and Results PSNR (db) 40.4 Bicubic 3 Layer 20 Layer XT YT DT Single Directional SDN Fig. 3: PSNR comparison between 3 layer CNN and 20 layer CNN in SDNs. Our models are built on top of Caffe, a deep learning framework by the BVLC [4], and trained with a GPU server that contains NVIDIA K40 GPU with 64GB of RAM. The models are evaluated on 22 patients 10,472 CTP slices scanned at four 5mm thickness brain regions with the spatial resolution of 0.43mm. The slices within one sequence are intensity normalized and co-registered over time. We randomly split these slices into two subsets: 7,140 for training (15 patients), 1,428 for validation (3 patients) and 1,904 for testing (4 patients). The size for each sequence is (X Y T ). In order to create more input images to brew a robust model, we clip patches with size pixel and a stride of 21 from four directions, which yields 36,800, 36,800, 73,600, and 62,951 patches in the directions of XY, Y T, DT, and XY. We create the LR patches by using bicubic method re-size to 1/3 on the original 2D HR patches. Experiment on SDNs. We test on two different CNN structures, and the result shows that the basic single directional network outperforms the shallow network in SRCNN (3 layers) [1] at all four cross-sections. This confirms that for both spatial and temporal SR, deeper is better than the shallow net. In Fig. 3, three single directional SDNs that have temporal SR are compared with the bicubic interpolation and SRCNN method, and it shows that the 20 layer model achieves better PSNR on average. Among these three types of single directional networks, Y T direction gives the best PSNR by the 20 layer structure. The XY spatial direction has a lower PSNR value ( db) compared to the temporal cross-sections which yield to about db and db higher than bicubic and SRCNN. Therefore, we choose this 20 layer deep CNN structure for our SDN model. To maintain the output image is the same size of the input, the kernel size, stride size, and pad size of our deep directional CNN is set to be 3, 1, 1 respectively. Except the last layer outputs one feature map, other convolution layers have 64 outputs. With residual learning, the loss function is determined by the sum of estimated residuals between the HR ground truth image and the LR input. The basic learning rate is set to be 0.1, and the weight decay is set to be for faster convergence.

6 6 Y. Xiao, A. Gupta, P. Sanelli and R. Fang Fig. 4: The gray-scale images (first row) and cerebral hemodynamic parameters (second row: CBF, third row: CBV) have achieved a higher resolution through different stages: column a): LR input, column b): SDN spatial, column c): SDN temporal, column d): STAR spatio-temporal, column e): the ground truth image. Furthermore, we also compare the performance based on the different patch representations in the basic model. The result of down-sampling on temporal direction only can be seen in Fig. 3. We also evaluate on SDNs with Y T, XT and DT LR inputs that are scaled down on both spatial and temporal directions, they bring about 2.13 db, 2.98 db and 4.01 db lower PSNR than temporal direction only. However, these SNDs outperform much better results where they achieved a greater improvement on image SR - on average of 0.9 db higher than the differences between the improvement of temporal only SDNs. This experiment indicates that the temporal pattern can be predicted more precisely than the spatial features through the proposed approach. In other words, there is an extensive potential of our basic networks to combine the spatial and temporal learning to produce better performance in an appropriate way, for which the high performance will be explained in our STAR cross-section learning approach. Experiment on STAR. The proposed STAR network is a combination of multiple SDNs from different cross-sections together and is cascaded with another deep convolution network where the convolution occurs in pairs with the ReLU layer. The cascaded layers have the same parameter settings of SDN. In the first convolution layer, filters are convolved on top of the mean outputs

7 STAR: Spatial-Temporal Architecture for super-resolution 7 from the previous spatial SDN and the three temporal SDNs. Thus the sub-voxel information and the contextual cues can be learned through different directions. As can be seen in Fig. 4, the resolution of images from left to right of the columns have been improved gradually. The first row of the figure is the grayscale CTP slice. The following two rows are the cerebral hemodynamic parameters: Cerebral Blood Flow (CBF) and Cerebral Blood Volume (CBV); which are calculated by Perfusion Mismatch Analyzer (PMA, [6]) from the corresponding CTP sequences in the first row. The first row LR images in column a) are downscaled with a ratio of 3 from the ground truth gray-scale images in column e). Column b) and c) are the intermediate results from spatial SR SDN where we only perform spatial SR on the XY direction, and temporal SR SDN where we only perform temporal cross-section SR. The superior outcomes of the proposed STAR method are shown in the column d). As the areas that the arrows are pointing at, the images with LR are missing the details and the rough sketch is blurry. The spatial SR images can show more details than the LR images, but still, the boundaries are not clear enough; and the temporal SR images are similar to the spatial SR images, which with drawbacks in different areas. By combining the spatial and temporal details, we can get a much better SR result: the images present more clearly and with extra details. We also measure the PSNR and SSIM for the perfusion images. The highlighted row in Table 1 shows the best values for our method. The basic models no matter the spatial-only SDN (Spat-SDN) or the temporal-only SDN provides better image quality than the bicubic method in different degrees. Our final STAR model gives the highest PSNR of db on CBF which is 6.86 db higher than the baseline and in CBV calculation, the STAR improves bicubic about db. Other than that, in the SSIM comparison, STAR still achieves the best results of and respectively. These experiments show the perfusion maps at only 1/3 of the original scanning time with comparable perfusion map quality and accuracy through the STAR framework. Our model allows the input with 1/3 of CTP scanning time and provides high-resolution outputs which potentially reduce the possibility of radiation over-exposure. 4 Conclusion In this paper, we have presented STAR, an end-to-end spatio-temporal superresolution framework. The experimental results show that the proposed basic model of single directional network improves both spatial and temporal resolution, while the multi-directional conjoint network further enhances the SR results - comparing favorably with only temporal or only spatial SR. By learning the spatial-temporal features, our approach ensures the ability to maintain the quality of brain CTP slices within one-third of the original scanning time. In the future, we believe that by reducing the scanning time, our approach will provide an applicable solution for improving spatial and temporal resolution and help to lower the possibility of excessive patient radiation exposure, while increasing the potential of assisted clinical diagnosis of cerebrovascular disease with highquality perfusion images. Our plans include applying our pipeline to a variety of

8 8 Y. Xiao, A. Gupta, P. Sanelli and R. Fang imaging modalities, including functional MRI, PET/CT for functionality superresolution and investigating the correlation between the temporal and spatial upscale ratios with super-resolution quality. Acknowledgements This work is partially supported by the National Science Foundation under Grant No. IIS , National Center for Advancing Translational Sciences of the National Institute of Health under Award Number UL1TR000457, National Key Research and Development Program of China (No: 2016YFC ) and by National Natural Science Foundation of China (No: ). References 1. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision. pp Springer (2014) 2. de González, A.B., Mahesh, M., Kim, K.P., Bhargavan, M., Lewis, R., Mettler, F., Land, C.: Projected cancer risks from computed tomographic scans performed in the united states in Archives of internal medicine 169(22), (2009) 3. Hoeffner, E.G., Case, I., Jain, R., Gujar, S.K., Shah, G.V., et al.: Cerebral perfusion ct: technique and clinical applications 1. Radiology 231(3), (2004) 4. Jia, Y., Shelhamer, E., Donahue, J., et al.: Caffe: Convolutional architecture for fast feature embedding. arxiv preprint arxiv: (2014) 5. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR Oral) (June 2016) 6. Kudo, K.: Perfusion mismatch analyzer, version asist-japan web site. http: //asist.umin.jp/index-e.htm, accessed: December 15, Mettler Jr, F.A., Bhargavan, M., et al.: Radiologic and nuclear medicine studies in the united states and worldwide: Frequency, radiation dose, and comparison with other radiation sources Radiology 253(2), (2009) 8. Nelson, T.R.: Practical strategies to reduce pediatric ct radiation dose. Journal of the American College of Radiology 11(3), (2014) 9. Oktay, O., Bai, W., Lee, M., et al.: Multi-input cardiac image super-resolution using convolutional neural networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp Springer (2016) 10. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp (2016) 11. Shrier, D.A., Tanaka, H., Numaguchi, Y., Konno, S., Patel, U., Shibata, D.: Ct angiography in the evaluation of acute stroke. American Journal of Neuroradiology 18(6), (1997) 12. Szymanski, L., McCane, B.: Deep networks are effective encoders of periodicity. IEEE transactions on neural networks and learning systems 25(10), (2014) 13. Wintermark, M., Lev, M.: FDA investigates the safety of brain perfusion ct. American Journal of Neuroradiology 31(1), 2 3 (2010)