Channel Splitting Network for Single MR Image Super-Resolution. Xiaole Zhao, Yulun Zhang, Tao Zhang, and Xueming Zou. PSNR(dB)

Transcription

1 1 Channel Splitting Network for Single R Image Super-Resolution Xiaole Zhao, Yulun Zhang, Tao Zhang, and Xueming Zou arxiv: v [s.cv] 13 De 018 Abstrat High resolution magneti resonane (R) imaging is desirable in many linial appliations due to its ontribution to more aurate subsequent analyses and early linial diagnoses. Single image super resolution (SISR) is an effetive and ost effiient alternative tehnique to improve the spatial resolution of R images. In the past few years, SISR methods based on deep learning tehniques, espeially onvolutional neural networks (CNNs), have ahieved state-of-the-art performane on natural images. However, the information is gradually weakened and training beomes inreasingly diffiult as the network deepens. The problem is more serious for medial images beause laking high quality and effetive training samples makes deep models prone to underfitting or overfitting. Nevertheless, many urrent models treat the hierarhial features on different hannels equivalently, whih is not helpful for the models to deal with the hierarhial features disriminatively and targetedly. To this end, we present a novel hannel splitting network (CSN) to ease the representational burden of deep models. The proposed CSN model divides the hierarhial features into two branhes, i.e., residual branh and dense branh, with different information transmissions. The residual branh is able to promote feature reuse, while the dense branh is benefiial to the exploration of new features. Besides, we also adopt the merge-and-run mapping to failitate information integration between different branhes. Extensive experiments on various R images, inluding proton density (PD), T1 and T images, show that the proposed CSN model ahieves superior performane over other state-of-the-art SISR methods. Index Terms Convolutional neural network, hannel splitting, feature fusion, magneti resonane imaging, super-resolution. I. INTRODUCTION SPATIAL resolution is one of the most important imaging parameters for magneti resonane imaging (RI). In many linial appliations and researh work, high resolution (HR) RI is usually preferred beause it an provide more signifiant struture and texture details with a smaller voxel size [1], thus failitates aurate subsequent analysis and early diagnosis. However, it is limited by several fators suh as X. Zhao is with the Shool of Life Siene and Tehnology, University of Eletroni Siene and Tehnology of China (UESTC), Chengdu, Sihuan , China ( zxlation@foxmail.om). T. Zhang is with the High Field agneti Resonane Brain Imaging Laboratory of Sihuan and Key Laboratory for Neuro Information of inistry of Eduation, Chengdu, Sihuan , China; He is also with the Shool of Life Siene and Tehnology, University of Eletroni Siene and Tehnology of China (UESTC), Chengdu, Sihuan , China ( taozhangjin@gmail.om). X. Zou is with the Shool of Life Siene and Tehnology, University of Eletroni Siene and Tehnology of China (UESTC), Chengdu, Sihuan , China (mark.zou@alltehmed.om). Y. Zhang is with the Department of Eletrial and Computer Engineering, Northeastern University, Boston, A 0115, USA ( yulun100@gmail.om). anusript reeived Otober XX, 018; revised November XX, (a) 10 5 PSNR (db) SRCNN (C1) VDSR (C1) CSN (C1) CSN (C96) Number of parameters (b) RDN (C1) EDSR (C96) Fig. 1. The performane omparison between several typial SISR models on proton density (PD) volumes of IXI dataset for SR. (a) The validation results on 6 PD volumes (576 D slies). (b) The testing results vs the number of model parameters on 70 PD volumes (670 D slies, Biubi: db). The symbols, and represent models with less than 1, 0, and more than 0 parameters respetively. C1 indiates that the training sample is a single slie, and C96 indiates that the model treats 96 slies of a 3D volume as 96 hannels. hardware devie, imaging time, desired signal-to-noise ratio (SNR) and body motion et, and inreasing spatial resolution of magneti resonane (R) images typially redues SNR and/or inreases imaging time []. Image super-resolution (SR) is a typial ill-posed inverse problem in omputer vision ommunity, whih mainly aims at inferring a HR image from one or more low resolution (LR) images. It is a well-studied problem in both natural image (NI) and R image proessing. High resolution means that the pixel density of an image is higher than its LR ounterpart. Thus, HR images an offer more details that may be ritial in various appliations suh as medial imaging [3], [4], aerial spetral imaging [5] and remote sensing imaging [6], [7] and seurity and surveillane [8], where high frequeny details are very important and greatly desired. Up to now, many SR methods have been studied and proposed. Early methods inlude: (i) interpolation methods, e.g., biubi, Lanzos-σ [9]; (ii) modeling and reonstrution methods, e.g., iterative bak projetion (IBP) [10], projetion onto onvex set (POCS) [11] et.; (iii) traditional shallow learning methods, e.g., example learning [1], [13], ditionary learning [14], [15] et. The performane of these methods is inherently limited beause the additional information available for solving this ill-posed inverse problem is also very limited, e.g., the interpolation methods make use of the basi smoothing priori by impliitly assuming that the image signal is ontinuous and bandwidth limited, and traditional learning based methods an learn insuffiient information due to the limited representational ability of these shallow models. In reent years, various advaned SR methods have emerged with the rapid development of deep learning tehniques [16]

2 External Skip Connetion (ESC) Global Skip Connetion (GSC) x x0 xe SB CSB Feature Extrat x 1 xi Global Feature Fusion (GFF) E F R Nonlinear apping (a) The overall arhiteture of the proposed two-way hannel splitting network x n x Conat x R Pixel Shuffle Image Reonstrution C y C 3 3 Conv C 1 1 Conv R ReLU ean Skip Add ~ Split ~ erge x i 1 C R C C R C i ~ ~ C CSB x i 1 C R C C R C (b) The internal struture of eah hannel splitting blok Fig.. The arhiteture of the proposed CSN model and the hannel splitting blok CSB. (a) The overall struture of the proposed CSN model, whih onsists of three typial parts: feature extration, nonlinear mapping and image reonstrution. Notie that GFF [30] is applied to fully utilize hierarhial features, while ESC and GSC are mainly applied to stabilize model training. (b) The hannels of the intermediate feature maps within a CSB are split into two branhes. One is built as a residual struture (residual branh), and the other is built as dense network (dense branh). and greatly promoted the best state of SR performane. Superresolution onvolutional neural network (SRCNN) [17] and fast super-resolution onvolutional neural network (FSRCNN) [18] are two pioneering ontributions that utilize onvolutional neural networks (CNNs) [19] to solve SR tasks. The further improvement based on these pioneering work mainly foused on inreasing model depth or sharing model parameters at the beginning, for example, deeply reursive onvolutional network (DRCN) [0], deep reursive residual network (DRRN) [1], super resolution using very deep onvolutional networks (VDSR) [] and memory network (emnet) [3] et. These methods, however, are mainly aimed at the SR task of natural images, not speially at medial images. The medial image proessing ommunity has notied these advanes and some medial image SR methods based on deep learning have also appeared [4], [5], [6], [7]. For R images, training deep models with a large amount of parameters and extremely deep strutures is usually more diffiult beause high-quality and effetive training samples are relatively sare and unavailable. It is worth noting that the hallenge is not the availability of training data itself, but the aquisition or the quality of relevant annotations/labeling for these data [8]. Therefore, some SR models that aim at NI are likely to fail when diretly trained with R images despite suffiient training samples, e.g., Fig.1 displays the peak signalto-noise ratio (PSNR) performane of several reent single image SR (SISR) models and the proposed hannel splitting network (CSN) on the proton density (PD) volumes of IXI dataset ( for SR, where enhaned deep super-resolution network (EDSR) [9] and residual dense network (RDN) [30] are advaned models on NI. But it is failed to train the EDSR model (the same onfiguration as [9]) with D PD images. This problem of training failure aused by the degradation of sample quality will get worse as the depth (or width) of the network and the number of model parameters inrease. Thus, in the ontext of R image super resolution based on deep learning tehniques, the dilemma has beome more apparent: on the one hand, models with shallow strutures and fewer parameters are easy to train, but their SR performane is usually unsatisfatory; on the other hand, models with deeper strutures and more parameters are promising to improve SR performane, but it is more diffiult for them to be fully trained with R images. An effetive way to alleviate the diffiulty of model training is residual learning, whih is initially proposed for image reognition [31], [3]. It has been widely proved to be helpful for feature reuse and model onvergene, thus making it possible to build extremely deep models. However, residual learning strategy alone is still insuffiient to train the model with a very deep struture and a very large number of parameters in ase of R images, e.g., EDSR [9] with about 43 parameters is a typial residual network, but the original onfiguration an hardly be well trained with D R images in our settings. The problem of training failure an be addressed by onatenating multiple D R images into a single multihannel training sample at the expense of performane, e.g.,

3 ZHAO et al.: CHANNEL SPLITTING NETWORK FOR SINGLE R IAGE SUPER-RESOLUTION 3 we train the original EDSR [9] model by taking all 96 slies of a 3D volume as 96 hannels of a single training sample, as shown in Fig.1 (marked as EDSR (C96)). In this paper, we improve the above dilemma by introduing a deep hannel splitting network (CSN) framework. It assumes that the hierarhial features of deep models have ertain lustering properties, and expliitly disriminating them is benefiial to ease the representational burden of deep models and further improve the SR performane. Therefore, instead of transferring the feature maps of the previous layer ompletely to the next layer, we split the feature maps into two different parts (branhes) with different information transmissions. Eah branh an be strutured differently, e.g., in this work, we use propagation mehanisms similar to residual network (ResNet) [31], [3] and dense network (DenseNet) [33] (or RDN [30]) on eah branh. Besides, the merge-and-run (AR) mapping [34], [35] is also applied to failitate the information integration of different branhes. Thus, our model has two notable harateristis: (1) hannel splitting disriminatorily limits the hierarhial features into different lusters and redues the representational redundany of the model by urtailing the internal onnetions; () the merge-and-run mapping an promote information sharing and integration between the hierarhial features and therefore help to improve the information flow through the entire network. In order to make full use of the hierarhial features of the model, we adopt the global feature fusion (GFF) tehnology [30] (Fig.(a)). oreover, multilevel residual mehanism and onstant saling tehnique [9], [36] are also applied to our models to further stabilize the model training. To verify the effetiveness of the proposed model, a set of standard datasets for the task of single R image SR are generated from the IXI dataset, whih inlude three types of R images (PD, T1 and T) and two image degenerations (biubi downsampling and k-spae trunation). The quantitative and qualitative experiments on the datasets display the superiority of the proposed model over other advaned methods. The rest of this paper is organized as follows. In setion II, we present some previous ontributions related the present work. The proposed method and the experimental results are detailed in setion III and setion IV, respetively. Setion V gives some disussion and future work. Finally, we onlude the whole work in setion VI. II. RELATED WORK A. Super Resolution with Deep Learning Although the work using artifiial neural networks (ANNs) to solve SR problems has emerged as early as 006 [], the pioneering work with deep learning tehniques in the modern sense is SRCNN [17]. Subsequently, some advaned methods based on inreasing network depth and parameter sharing are proposed. Kim et al. [] inreased the network depth by staking more onvolutional layers with global residual learning (GRL) and firstly introdued reursive learning in a deep network for parameter sharing [0]. Another work whih introdued by Tai et al. [1] has utilized reursive bloks to reuse parameters. otivated by the fat that human thoughts have persisteny, a deep persistent memory network (emnet) whih onsists of the so-alled memory blok has also been proposed by the same author [3]. To improve information flow and to apture suffiient knowledge for reonstruting the high frequeny details, Hu et al. [35] proposed a asaded multi-sale ross network (CSCN) in whih a sequene of subnetworks (Fig.3(b)) is asaded to infer HR features in a oarse-to-fine manner. To some extent, these methods promote the design of new strutures for image generation tasks. There is a ommon feature among the above methods: they use the interpolated image of the original LR image as the input of their models. This preproessing is onvenient for keeping the size of the output image onsistent with the target HR image. However, it plaes the inferene of the network in the HR image spae, resulting in great omputation and memory onsumption. There exist two solutions for this issue urrently: deonvolution or transpose onvolution [18] and effiient sub-pixel onvolutional neural network (ESPCNN) [38]. Both of them an effetively solve the above problem by shifting the HR referene to the LR image spae. Benefiting from the nonlinear mapping within the LR image spae, these methods are apable of inreasing the sale of the models and thus boost the SR performane greatly, e.g., EDSR/DSR [9]. Further, Zhang et al. [30] ombined the idea of residual learning [31], [3] with densely onneted DenseNet [33] and proposed a novel residual dense network (RDN) to fully utilize the hierarhial features of deep models. B. R Image Super Resolution The early appliation of SR methods to medial images mainly fouses on the task of multi-frame image super resolution (FSR). For example, IBP [10] was adopted to generate a new R image with inreased spatial resolution from several spatially shifted, single-shot and diffusion-weighted brain R images [39]. Greenspan et al. [40] and Shilling et al. [41] employed IBP and POCS [11] to produe a 3D R volume with isotropi resolution from several D slies, respetively. These methods are usually aompanied by speifi aquisition strategies (e.g., rotation, saling and translation) to simulate the LR image generation. However, reovering a HR image from multiple degraded LR images usually needs to alibrate and fuse these LR images, whih is a very hallenging task in itself. To avoid the diffiulty of alibration and fusion between multiple LR images, Rousseau [4] first proposed to enhane R image resolution with single image SR tehniques. In this method, the extra information was introdued into the reonstrution proess by referring to another HR image. A similar method was proposed by anjón et al. [43]. They also used a HR image as referene but with a different strategy to produe HR images. These methods introdue very limited extra information beause they learn knowledge from only one external HR image. SR methods based on onventional mahine learning, e.g., sparse representation [44], [45] and ompressive sensing [46], are also applied to medial images subsequently. Reently, more advaned methods based on deep learning [16] have also been applied to R image SR

4 4 C 5 5 Conv (a) C 3 3 Conv (b) ean Skip Add Fig. 3. The merge-and-run stage mappings in a building blok. (a) Original mapping [34]. (b) ulti-sale ross (SC) mapping [35]. () The proposed mapping. is the hannel number of feature maps, and g is the growth of a dense onnetion [33]. BN and ReLU are omitted for simplifiation. tasks [4], [5], [6], [7], [47]. These methods, however, simply use deep learning tehniques to deal with the SR tasks of R images without onsidering the differenes between natural images and medial images. Contrarily, the proposed CSN model aims at dealing with the hierarhial features disriminatively and reduing the representational burden of the model to adapt the degradation of R training samples. C. ulti-stream Networks ulti-stream networks are widely adopted by the image SR ommunity to boost the SR performane by assembling the information from different streams (paths). Wang et al. [48] explored an end-to-end CNN arhiteture by jointly training both deep and shallow CNNs, where the shallow network stabilizes model training and the deep one ensures aurate HR reonstrution. Ren et al. [49] proposed a ontext-wise network fusion approah to integrate the outputs of individual networks by extra onvolutional layers. Yamanaka et al. [50] ombined skip onnetion layers and parallelized CNNs into a single CNN arhiteture. CSCN [35] is another multi-stream struture, in whih omplementary information under different reeptive fields is integrated by the merge-and-run mehanism [34] (Fig.3). There are also multi-stream strutures for medial image SR tasks, e.g., Oktay et al. [47] developed a multi-input ardia image SR network, whih is apable of assembling information from different viewing planes to improve the SR performane. These methods are fundamentally different from the proposed CSN model, in that they form the multi-stream struture in the reuse of the preeding features, while our CSN network onstrut the multi-stream struture by splitting the preeding features into different branhes. III. PROPOSED ETHOD A. Overall Network Arhiteture The arhiteture of the proposed CSN model is outlined in Fig.. Like other typial deep SR models, it onsists mainly of three parts: feature extration, nonlinear mapping and image reonstrution. The feature extration sub network (FEN) is first applied to express the input image x as a set of shallow features. These shallow features are then transmitted to the nonlinear mapping sub network (NN), whih is omposed of a series of staked hannel splitting bloks (CSB). Subsequently, the hierarhial features outputted by eah CSB are onatenated together to produe the final output of the g () NN. This operation is also alled global feature fusion (GFF) [30]. Finally, the olleted hierarhial features are fed into the image reonstrution sub network (IRN) to generate the final HR prediation y of the entire model. 1) Feature Extration: The FEN ontains two 3 3 onv layers with a 1 1 onv layer in the middle. Denote F E ( ) as the orresponding funtion of FEN, then the shallow features x E extrated by FEN an be represented as: x E = F E (x), (1) where x is the original LR input. The 1 1 onv layer is the point-to-point linear transformation of the feature maps of the first 3 3 onv layer, whih is used to enhane the robustness of the extrated features. ) Nonlinear apping: The NN is denoted as F ( ), i.e., the output of the NN is given by F (x E ). Supposing we have n hannel splitting bloks in the entire network, the output x i of the i-th CSB an be obtained by: x i = F i (x i 1 ), i = 1,,..., n, () where the funtion F i ( ) orresponds to the operations of the i-th CSB. ore details about F i ( ) will be presented in setion III-B. x 0 = x E is the input of the first CSB (or the whole nonlinear mapping sub network). Therefore, the output of the last CSB an be iteratively formulated as follow: x n = F n (x n 1 ) = F n (F n 1 ( (F 1 (x 0 )) )). (3) The output of the i-th CSB x i is produed by a series of operations (e.g., onvolutional layer, ReLU and onstant saling et.) within the blok, so it is viewed as a set of loal features [30]. These loal features onstitute the final output of our nonlinear mapping sub network. It should be noted that the output of the preeding CSB is diretly used as the input of the next CSB. This is similar to the so-alled ontinuous memory (C) mehanism [30] and ontributes to the information propagation in the network [3]. 3) Image Reonstrution: This phase inludes two related parts: global fusion of the loal features x i and restoration of HR images based on the fused features. To fuse the loal features, the outputs of all CSBs are first onatenated into a single tensor (red retangle in Fig.): x = [x 0, x 1,..., x n ], (4) where [...] implies the onatenation. Next, the global features are extrated by fusing all loal features from all the preeding hannel split bloks. This is ompleted by a omposite funtion F F ( ) orresponding to a 1 1 onvolutional layer followed by a 3 3 onvolutional layer. Finally, the global residual learning (GRL) is utilized to stabilize the training proess [9], [30], whih is simply implemented via a global skip onnetion (GSC): x R = F F (x ) x 0 = F F ([x 0, x 1,..., x n ]) x 0, (5) where x R is the fused features that will be used to reover the HR image y. As for HR image restoration, it is mainly made up of a pixel shuffle layer followed by a 3 3 onvolutional

5 ZHAO et al.: CHANNEL SPLITTING NETWORK FOR SINGLE R IAGE SUPER-RESOLUTION 5 (a) HR image (b) Fourier transform () k-spae trunation (d) Inverse reovery Fig. 4. The image degradation of k-spae trunation for SR 4. Different from biubi downsampling, the generated LR images are sometimes aompanied by Gibbs-ringing artifats. The result in (d) is generated by zero-padding in k-spae and inverse Fourier transform (IFT) for display purpose. layer, and an external residual learning (ERL). Formally, it an be represented as: y = F R (x R ) x, (6) where F R ( ) is the funtion orresponding to the pixel shuffle layer and the following 3 3 onvolutional layer, and x is the original LR input to the model. Note that the pixel shuffle layer is onduted by ESPCNN [38] in the way of [9]. B. Channel Splitting Blok The CSCN model explored by [35] is a multi-stream struture that integrates the omplementary information under different reeptive fields. oreover, it has been proved that residual learning enables feature reuse and dense learning enables new features exploration, both of whih are important for learning good representations [51]. Inspired by this, we present a two-way hannel splitting blok (CSB) to inorporate different information with different propagation mehanisms. As shown in Fig.(b), the distintive features of the proposed CSB are hannel splitting and merging, and fusion of residual learning and dense learning. Besides, loal residual learning (LRL) is also applied to further improve the information flow. It has been shown that LRL is helpful to stabilize the training proess and improve the representation ability of the model, resulting better SR performane [30]. 1) Channel Splitting and erging: As for the i-th CSB, the input tensor x i 1 is first equally split into two tensors along the hannel diretion, x i 1 and x i 1, whih are the inputs of the lower (dense) branh and the upper (residual) branh respetively. It ould be formally expressed as: x i 1, x i 1 = S (x i 1 ), i = 1,,..., n, (7) where S ( ) is the hannel splitting funtion. It an be viewed as a unary operator that splits the input tensors into two parts along the hannel diretion. Through this hannel splitting operation, we an apply different information transmission mehanisms on eah branh. For example, we adopt residual learning and dense learning on the upper and lower branhes respetively. Correspondingly, there exists a hannel merging operation, ( ), at the end of eah CSB: x i 1 = (x i 1, x i 1), i = 1,,..., n. (8) It should be noted that ( ) is a bivariate funtion, while [...] in (4) and (5) is a multivariate operator. The hannel splitting and merging operations are expeted to artifiially interfere with the information flow within the network, whih helps the model to proess the hierarhy features with different properties in a targeted way. In addition, it is also an effetive way to maintain the sale of model parameters and inrease the depth of the network. ) Feature Reuse and New Feature Exploration: A CSB module assembles two residual-like branhes in parallel with a merge-and-run mapping [34] but eah branh has different strutures. Suppose the i-th CSB ontains m stage mappings and eah branh in a stage inludes two onvolutional layers with a ReLU operation in the middle (Fig.(b)). Denote Hi,j ( ) and Hi,j ( ) as the transition funtions of the lower and the upper residual branhes in the j-th stage mapping respetively. Then the transition funtion of the j-th stage mapping an be represented in matrix form as below: [ ] x i 1,j = x i 1,j [ H i,j (x i 1,j 1 ) ] Hi,j (x i 1,j 1 ) 1 [ ] [ ] I I x i 1,j 1, (9) I I x i 1,j 1 where i = 1,,..., n and j = 1,,..., m is are the indexes of the i-th CSB and the j-th stage. x i 1,j 1 and x i 1,j 1 (x i 1,j and x i 1,j ) are the inputs (outputs) of the j-th stage in the i-th CSB, and x i 1,0 = x i 1 and x i 1,0 = x i 1 are the inputs of the lower and the upper branhes respetively. I denotes identity matrix. Therefore, the oeffiient matrix C = 1 [ ] I I, (10) I I is an idempotent transformation matrix of the merge-and-run mapping [34], [35]. The idempotent property an promote the information flow aross the different modules and enourage gradient bak-propagation during model training, similar to identity mapping [3]. It is worth noting that the upper branh is a residual-like struture similar to EDSR [9] and the lower branh is a simplified dense-like struture similar to DenseNet [33] or RDB [30], whih uses only one skip dense onnetion to explore new features. Through merge-and-run mapping, we an effetively integrate the superiority of feature reuse and new feature exploration provided by the residual branh and the dense branh.

6 6 3) Loal Residual Learning (LRL): The feature maps from these two branhes, x m and x m, are merged together after m stages of merge-and-run mappings. Next, a loal residual learning (LRL) [30] is introdued in the CSB to further improve the information flow. The output of this CSB module is given by: x i = L( (x i 1,m, x i 1,m)) x i 1, (11) where L( ) orresponds to a 1 1 onvolutional operation at the end of the CSB, as shown in Fig.(b). Unlike [30], the loal residual features are derived from our CSB module, instead of the densely onneted blok [33]. C. ultilevel Residual ehanism Normally, the LR images and the orresponding HR images share same information to a large extent, whih indiates that a large part of the topologial struture of their high-dimensional manifolds are similar to eah other. Therefore, it is benefiial to expliitly allow the model to learn the residual between the original LR input and the HR output []. However, beause the size of the LR/HR images is different, the residual between them annot be diretly obtained. We adopt a biubi interpolated version of the LR image to math the size of the HR image, and use it to approximate the residual between the original LR image and the HR image. This is implemented by simply adding the interpolated image to the output of the last onvolutional layer of the entire network, whih we term as external skip onnetion (ESC) (Fig.(a)). Thus, (6) should be rewritten as: y = F R (x R ) ˆx, (1) where ˆx is the interpolated version of the original LR input x. Although we use biubi interpolation here, one an use any other interpolation algorithm (e.g., nearest neighbour, bilinear and B-Spline et.). Combined with GSC and LRL, the whole network shows a harateristi of multilevel residual learning. Our experiene shows that this an further stabilize the training proess, and even help to slightly improve the model performane. Beause the degeneration of R training samples makes the model training more unstable, it is espeially helpful for the task of single R image super resolution. D. Training Objetive and Network Depth Our model is a typial end-to-end mapping from LR images to HR images. The estimation of model parameters is ahieved by minimizing the loss between the reonstruted HR images and the ground truth HR images. Given a training dataset D = {x (i), y (i) }, i = 1,,..., D, where D is the total number of training samples, we use l 1 loss for model training: L(θ) = 1 D y (i) F CSN (x (i) ; θ) 1, (13) D i=1 where θ indiates the set of model parameters, and F CSN ( ) is the mapping funtion of the entire CSN model and y (i) is the HR target orresponding to LR input x (i). Despite that C g (a) g g (b) Fig. 5. Several strutures for stage mapping. (a) The baseline struture and CSN-SP. (b) R3R3 and D3D3. () R3D5 and R5D3. Note Fig.3(a) is different from R3R3. For simplifiation, ReLU between the two onvolutional layers is omitted, please refer to Fig.(b) for the overall struture of a CSB. minimizing l loss is generally preferred sine it maximizes peak signal to noise ratio (PSNR), l 1 loss provides better onvergene for model training [9]. This is espeially helpful in ase of the degradation of training samples. The depth of a deep network is usually defined as the longest path from the input to the output. Thus, the depth of the overall CSN model is given by: D = n(m 1) s 6, (14) where s represents the depth of the pixel shuffle layer. Note that s depends on the saling fator [9], i.e., s = 1 for SR and SR 3, and s = for SR 4. IV. EXPERIENTAL RESULTS In this setion, we first introdue the generation of training examples and the implementation details. Then we investigate the impat of different onfigurations of CSB and the whole network on SR performane. Next, our CSN model is ompared with several typial SISR methods under two ommon image degradations: biubi downsampling (BD) and k-spae trunation (TD). We use PSNR and strutural similarity index metri (SSI) [5] as the metris of quantitative evaluation. A. Data Generation The IXI dataset is used to onstrut our SR datasets, whih ontains three types of R images: 581 T1 volumes, 578 T volumes and 578 PD volumes. Firstly, we take the intersetion of these three subsets, resulting in 576 3D volumes for eah type of R images. These 3D volumes are then lipped to the size of (height width depth) to fit 3 saling fators (, 3 and 4). In this work, we only fous on the in-plane SR of D R slies. Therefore, eah R volume ontains 96 training samples with a single hannel. The LR images are generated aording to biubi downsampling and k-spae trunation. As for trunation degradation, the HR images are first onverted into k-spae by disrete Fourier transform (DFT) and then trunated along both height and width diretions (Fig.4). We randomly seleted 500 volumes for training (D), 70 volumes for testing (T ) and the remaining for quik validation (V). For onveniene, we use dataset (modality, degradation) to indiate a speifi dataset, e.g., g g ()

7 ZHAO et al.: CHANNEL SPLITTING NETWORK FOR SINGLE R IAGE SUPER-RESOLUTION CSN-R3D3 CSN-SP baseline (a) CSN-R3D3 CSN-R3R3 CSN-D3D (b) CSN-R3D3 CSN-R3D5 CSN-R5D Fig. 6. The performane omparison between the strutures shown in Fig.5 on V(T1, TD) for SR (biubi = 31.7dB). (a) Channel splitting and mergeand-run mapping. (b) Different branh strutures. () Different kernel sizes. The baseline stage mapping is a single onvolutional layer followed by a ReLU operation, whih emphasizes the impat of hannel splitting. () 10 5 TABLE I THE PERFORANCE OF THE ODELS WITH DIFFERENT STAGE APPINGS ON T (T1, TD) (SR ). THE AXIAL VALUES OF EACH COLUN ARE IN BOLD, AND THE SECOND ONES ARE UNDERLINED. Network Network Network Configuration Parameters Depth PSNR (db) SSI Biubi baseline CSN-SP CSN-R3D CSN-R3R CSN-D3D CSN-R3D CSN-R5D TABLE II THE QUANTITATIVE RESULTS OF THE ODELS WITH DIFFERENT ESC APPROXIATIONS ON T (PD, BD). THE AXIAL VALUES OF EACH ROW ARE IN BOLD, AND THE SECOND ONES ARE UNDERLINED. sale ESC- ESC- ESC- ESC- None NN Bilinear Biubi 41.0/ / / / / / / / / / / / D(T, BD) represents the T training dataset with biubi degradation and T (PD, TD) represents the PD testing dataset with k-spae trunation degradation. At present, the model only targets at the task of single D R image super resolution. Thus, we have = training examples (D slies) in total. The generated dataset an be onveniently applied to develop 3D SR algorithms as eah dimension is lipped to the ommon multiple of, 3 and 4, whih will be a part of our future work. B. Implementation Details The onfiguration of the model is shown in Fig. with n = m = 4. The size of minibath and the number of feature maps are set to 16 and 56 respetively. For the dense branh within a CSB, the growth (g in Fig.3 and Fig.5) is set as 64. If not speified, the kernel size follows the annotations of Fig.. We train the models by using image pathes of size 4 4 randomly extrated from LR slies with the orresponding HR pathes. Data augmentation is simply onduted by random horizontal flips and 90 rotations, as [9] and [30]. All models Unstable Fig. 7. The impat of different ESC approximation on model training and performane. The omparison is arried out on D(PD, BD) for SR. Note that ESC-None shows obvious instability. are implemented (or reimplemented) in TensorFlow and trained on a NVIDIA GeFore GTX 1080 Ti GPU for one million iterations. We use xavier initialization [54] for all model parameters and Adam optimizer [53] to minimize the loss by setting β 1 = 0.9, β = and ɛ = Learning rate is initialized as 10 4 for all layers and halved at every 10 5 iterations (pieewise onstant deay). C. odel Analysis In this setion, we study several omponents of the proposed model, inluding the struture of stage mapping, multilevel residual learning, global feature fusion and building blok utilization. The struture of the entire network and the building blok refers to Fig.. 1) Channel Splitting Blok: The proposed stage mapping an be onfigured in several ways, thus equipping different CSB modules. For omparison, we have studied the struture of stage mapping from the following aspets: If without hannel splitting, four onvolutional layers in a stage mapping orresponds to a single onvolutional layer with almost the same number of parameters (the top row in Fig.5(a)), whih is the referene struture of a stage mapping. We take it as the baseline

8 PSNR (db) (a) Impat of building bloks (n) (b) Impat of stage mappings (m) Number of parameters 10 7 () Comparison on T (T, TD) Fig. 8. The performane omparison between the models with different number of stage mappings and building bloks. (a) and (b) The validation performane of the models on V(T, TD) (SR, Biubi: 31.9dB). () The testing performane of all ompared models on T (T, TD) (SR, Biubi: 33.06dB). To investigate the role of the AR mapping, we removed it from the proposed CSB, resulting in the struture shown in the bottom row of Fig.5(a). We term it as CSN-SP, where S means splitting and P means plain. We also design two stage mappings shown in Fig.5(b) to hek the effet of different branh strutures, whih are termed as R3R3 (top) and D3D3 (bottom). R and D represent residual branh and dense branh, and the number indiates the kernel size. Strutures shown in Fig.5() are used to investigate the impat of different kernel size, whih are termed R3D5 (top) and R5D3 (bottom) orrespondingly. The proposed stage mapping struture (Fig.(b) and Fig.3()) is marked as R3D3. m and n are set to 4 for all experiments in this setion. The performane of the ompared stage mapping strutures on V(T1, TD) for SR are shown in Fig.6, where Fig.6(a) shows the impat of hannel splitting and AR mapping on the performane of the model. It an be seen that both of them an signifiantly improve the SR performane of the model. From Fig.6(b), we an observe the slightly better performane of R3D3 than that of R3R3 and D3D3. It is worth noting that the model parameters R3R3>R3D3>D3D3 and the depth of the networks is the same, whih implies that mixing different branh strutures is indeed helpful to boost the performane, although slightly. Fig.6() shows that R3D5 and R5D3 perform obviously better than R3D3. However, the number of model parameters of them is about 1.6 times of R3D3. Besides, despite R3D5 and R5D3 have the same number of parameters but R5D3 performs slightly better than R3D5. The onlusions are further verified by the results on the orresponding testing dataset T (T1, TD), as shown in Table I. ) External Skip Connetion: To verify the impat of the external skip onnetion (ESC), we build three other models aording to our CSN model, two of whih used nearest neighbor (NN) and bilinear respetively to approximate the residual between the original LR input x and the orresponding HR target y, and the other does not use ESC. They are termed as ESC-None, ESC-NN, ESC-Bilinear, and the one we use is termed as ESC-Biubi. We train these models on D(PD, BD) and the validation performane is plotted in Fig.7. It an be easily observed that ESC-Biubi perform signifiantly better than other models. The orresponding results on T (PD, BD) also illustrate this onlusion (Table II). Another important observation is that ESC helps stabilize model training, no matter whih interpolation method is used. Thus, ESC an redue the possibility of training failure, whih is also benefiial in the ase of the degradation of training examples. 3) The Number of Stages and Bloks: From (14), it an be seen that the network depth D is mainly determined by the number of CSB m and the number of stage mappings n. We examine the impat of these two hyperparameters on the performane of the model. Firstly, we fix m to 4 and hange n from 1 to 4. Fig.8(a) displays the evolution urves of PSNR performane on V(T, TD) for SR. It an be seen that the performane is improved gradually with the inreased number of building bloks, but at the expense of inreased parameters (omputational ost and memory oupation). Next, we fix n to 4 and hange m from 1 to 4. The PSNR urves of the models on the same dataset are plotted in Fig.8(b). We observe a similar trend of the urves as m hanges. The result is unsurprising beause inreasing m or n inreases the network depth D and the number of parameters. Finally, we show the final SR performane of all ompared models on the orresponding testing dataset T (T, TD) in Fig.8(), versus the number of the number of parameters. It is worth noting that the models with t building bloks and 4 stage mappings perform better than the models with 4 building bloks and t stage mappings (t = 1,, 3), despite the former has fewer model parameters. We hoose n = m = 4 for our CSN model, the depth of whih thus is 43 (for SR and SR 3) or 44 (SR 4) aording to (14). D. Comparison with Other ethods To further illustrate the effetiveness of the proposed CSN model, several state-of-the-art SISR methods, inluding NL [55], SRCNN [17], VDSR [], RDN [30], and EDSR [9] are ompared with the proposed CSN model quantitatively and qualitatively. The models are retrained on the generated datasets on all kinds of R images and for all saling fators. As mentioned earlier, the degradation of training samples may lead to training failure of some models, espeially for those with extremely deep struture and large number of parameters,

9 ZHAO et al.: CHANNEL SPLITTING NETWORK FOR SINGLE R IAGE SUPER-RESOLUTION 9 TABLE III QUANTITATIVE COPARISON BETWEEN DIFFERENT ALGORITHS ON T (:, BD). THE AXIAL PSNR (DB) AND SSI VALUES OF EACH ROW ARE IN BOLD, AND THE SECOND ONES ARE UNDERLINED. PD T1 T sale Biubi NL [55] SRCNN [17] VDSR [] RDN [30] CSN EDSR [9] CSN (D) (D) (C1) (C1) (C1) (C1) (C96) (C96) SR 35.04/ / / / / / / / SR / / / / / / / / SR / / / / / / / /0.935 SR 33.80/ / / / / / / / SR / / / / / / / / SR 4 8.8/ / / / / / / / SR 33.44/ /0.97.3/ / / / / / SR / / / / / / / / SR / / / / / / / /0.931 Biubi NL SRCNN-C1 VDSR-C1 RDN-C1 EDSR-C96 CSN-C96 CSN-C1 Ground Truth 8.61/ / / / / / / /0.97 PSNR/SSI 9.98/ / / / / / / / PSNR/SSI Fig. 9. The visual effet of the ompared methods on a T1 image with saling fator SR 3 (top) and a PD image with saling fator SR 4 (bottom). The maximal PSNR (db) and SSI values for eah displayed image are in bold, and the seond ones are underlined. e.g., EDSR [9]. To solve this problem, we train EDSR with the entire 3D volumes by taking the 96 slies as 96 hannels. This an effetively avoid the training failure problem but at the ost of auray redution. We attah C1 and C96 to the model name to indiate if a model is trained slie by slie (SRCNN(C1)) or it treats 96 slies as 96 hannels (e.g., EDSR (C96)). For fair omparison, we train both CSN (C96) and CSN (C1) to math the ases. 1) Biubi Degradation (BD): Biubi degradation is a widely used simulation of LR image generation in image SR settings whih simply shrinks HR images to a smaller size with the biubi kernel. We examine this degradation in this setion first. Table III shows the quantitative results of the ompared methods over the testing datasets of PD, T1 and T R images for SR, SR 3 and SR 4. Overall, all the deep learning based methods (SRCNN [17], VDSR [], RDN [30], EDSR [9] and our CSN) present great advantages over the traditional methods (Biubi and NL [55]). However, the proposed CSN model exhibits the best SR performane on all datasets and SR sales in both C1 and C96 senarios although it has fewer parameters and shallower model strutures than RDN [30] and EDSR [9]. Fig.9 presents the visual effets of the ompared methods under the biubi degradation. The top row displays the results on a T1 image with saling fator SR 3. It an be observed that the edges of the dark groove (pointed by the red arrows) reonstruted by the proposed CSN model are sharper that of other methods, while the traditional methods (Biubi and NL [55]) almost lost this detail information. The bottom row gives the results on another PD image with saling fator SR 4. We an see that Biubi, NL [55], even SRCNN [17] and VDSR [] annot reonstrut a lear and reliable result. Despite RDN [30] gives a signifiantly better result than other methods, the edges in the restored image are still relatively blurry. Instead, our CSN model exhibits a lear and sharp image that suessfully reovers the blak holes at both top and bottom ends of the image path, making it more faithful to the ground truth.

10 10 TABLE IV QUANTITATIVE COPARISON BETWEEN DIFFERENT ALGORITHS ON T (:, TD). THE AXIAL PSNR (DB) AND SSI VALUES OF EACH ROW ARE IN BOLD, AND THE SECOND ONES ARE UNDERLINED. PD T1 T sale Biubi NL [55] SRCNN [17] VDSR [] RDN [30] CSN EDSR [9] CSN (D) (D) (C1) (C1) (C1) (C1) (C96) (C96) SR 34.65/ / / / / / / / SR / / / / / / / / SR 4 8.8/ / / / / / / /0.946 SR 33.38/ / / / / / / /0.97 SR / / / / / / / /0.966 SR / / / / / / / / SR 33.06/ / / / / / / / SR / / / / / / / / SR / / / / / / / / ) Trunation Degradation (TD): The k-spae trunation of the HR image is a proess that simulates the real image aquisition proess where a LR image is sanned by reduing aquisition lines in phase and slie enoding diretions. The missing data, therefore, is in k-spae and the degradation pattern of the LR image is different from simply shrinking the image size in the image domain by, e.g., biubi interpolation [5]. Table IV shows the quantitative omparison between different methods under trunation degradation. Again, our CSN model presents the best SR performane in the ase of C1, followed by RDN [30]. It is noteworthy that the SR performane of biubi, NL [55], SRCNN [17] and VDSR [] is slightly worse than that of these methods on T (:, BD) in the same setting, e.g., SR on T images. On the ontrary, the results of RDN [30] and our CSN on T (:, TD) are better than that on T (:, BD), whih may imply that they are more suitable for the task of R image super resolution beause the image degradation used to generate the LR images are more ompatible with the real R imaging proess. Fig.10 presents the visual effets of the ompared methods on a PD image (top, SR ), a T1 image (middle, SR 4) and a T image (bottom, SR 4). The red arrows (both thik and thin ones) mark the most distint positions in the small image pathes. For the T1 image, other methods fail to reonstrut the white ridge pointed by the thik arrows and the dark line pointed by the thin arrows. However, The proposed CSN model an learly reover these details. Similar omparison an also be observed from the T image. There are two blak holes near to the top end of the small image path, while only our CSN model an give a lear indiation of the potential struture. We an also observe other obvious differenes, e.g., the blak hole at the bottom end of the image path. A. ultiple Branhes V. DISCUSSION AND FUTURE WORK Like the original stage mapping in [34], the stage mapping in our CSB an also be easily extended to more branhes ( 3). The differene is that we branh the network by hannel splitting, instead of feature reuse. In extreme ases, it an be extended to branhes with eah branh oupying one hannel of the input feature. This means that we expliitly differentiate the hierarhial features rather than having the network learn to distinguish between different features. Therefore, when the training samples are degraded and the model is omplex, it helps to ease model training. B. Depth and Width Branhing the network by reusing the entire feature map makes the model wider, like [34], [35]. This signifiantly inreases model parameters when the network depth is the same. Sine EDSR [9] is a typial network with very wide struture and auses training failure, while RDN [30] is a deeper but less wide network and an be suessfully trained. Therefore, we speulate that the width of the model may also be one of the reasons for training failure in ase of training sample degradation. Our work an be regarded as a manner to going deeper with nearly unhanged model width and parameters. C. Branh Struture Currently, we only utilize the strutures similar to ResNet [3], [31] and DenseNet [33] (or RDN [30]) for different branhes. The experimental results show that mixing different branh strutures is helpful to improve the performane of the model, but it is not onspiuous. This is probably beause the strutural differene between the two branhes is relatively small. We onjeture that as the strutural differene of the branhes inreases, so does the performane differene. The further investigation will be a part of our future work. D. 3D Extension The present work only aims at the task of D R image super resolution, and the further extension ould be in 3D ase. However, sine many types of medial images are in 3D format, it is intuitively possible to further enhane SR performane if the 3D strutural information an be reasonably utilized [3], [4], [5], [7]. A prominent problem in the 3D settings is that the number of parameters will inrease dramatially as the network depth inreases, leading model training more diffiult. Our model an deepen the network without signifiantly inreasing the model width and parameters, whih also helps to extend 3D models. E. Information Sharing In this paper, we only deal with the SR task for a single type of D R images and a single saling fator. However, there is evidene that ombining the information from different image types and saling fators is helpful to improve the performane of deep models [], [9]. The medial images SR

11 ZHAO et al.: CHANNEL SPLITTING NETWORK FOR SINGLE R IAGE SUPER-RESOLUTION 11 Biubi NL SRCNN-C1 VDSR-C1 RDN-C1 EDSR-C96 CSN-C96 CSN-C1 Ground Truth 36.16/ / / / / / / / PSNR/SSI 6.56/ / / / / / / / PSNR/SSI 9.43/ / / / / / / / PSNR/SSI Fig. 10. The visual effet of the ompared methods on a PD image (top), a T1 image (middle) and a T image (bottom) with saling fator SR, SR 4 and SR 4, respetively. The maximal PSNR (db) and SSI values for eah displayed image are in bold, and the seond ones are underlined. framework ombined multi-type and multi-sale information is also expeted to further improve the SR performane of deep models. VI. C ONCLUSION A major problem with using deep models to super-resolve R images is the lak of high-quality and effetive training samples, whih probably leads to performane degradation or even training failure of deep models. In this work, we have presented a novel deep hannel splitting network (CSN) for the task of D R image super resolution, whih is primarily made up of a series of asaded hannel splitting blok (CSB). The hierarhial features are split into two branhes with different information propagations (residual branh and dense branh), whih helps the model to disriminate different features expliitly. To improve integrate information, the AR [34] mapping is also applied to merge the feature maps of different branhes. Channel splitting helps to inrease the depth of the network and the diversity of proessing the hierarhial features. We onjeture that the performane improvement of the proposed model benefits from both two parts and additional performane an be further gained by exploring other branh strutures and information fusion strategies. As it improves the dilemma between improving model performane and easing model training to some extent, it also has the potential to deal with other types of medial images, suh as CT, ultrasound and PET et. ACKNOWLEDGENT The work is supported in part by the National Key Researh and Development Program of China (No. 016YFC and 016YFC010080). R EFERENCES [1] E. Carmi, S. Liu, N. Alona, A. Fiat, D. Fiat, Resolution enhanement in RI, agn. Reson. Imag., vol. 4, no., pp , Feb [] E. Plenge, D. H. J. Poot,. Bernsen, et al, Super-resolution methods in RI: an they improve the trade-off between resolution, signal-tonoise ratio, and aquisition time?, agn. Reson. ed., vol. 68, no. 6, pp , Feb. 01. [3] J. Hu, X. Wu, J. Zhou, Single image super resolution of 3D RI using loal regression and intermodality priors, in Pro. Int. Conf. Digit. Image Proess., vol.10033, 016. pp c.