Accelerated very deep denoising convolutional neural network for image super-resolution NTIRE2017 factsheet

Transcription

1 Accelerated very deep denoising convolutional neural network for image super-resolution NTIRE2017 factsheet Yunjin Chen, Kai Zhang and Wangmeng Zuo April 17, Team details Team name HIT-ULSee Team leader name Yunjin Chen Team leader address, phone number, and Address: Room No. 1102, Shidai Building B, Xinye Road No. 8, Qianjiangxincheng, Jianggan District, Hangzhou, Zhejiang Province, China Phone: chenyunjin nudt@hotmail.com Rest of the team members Kai Zhang and Wangmeng Zuo Team website URL (if any) Affiliation Yunjin Chen is with the ULSee Inc., Kai Zhang and Wangmeng Zuo are with School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China. Affiliation of the team and/or team members with NTIRE2017 sponsors (NVIDIA, SenseTime, Twitter, Google) User names and entries on the NTIRE2017 Codalab competitions (development/validation and testing phases) chenyunjin ULSee and cskzh 1

2 Best scoring entries of the team during development/validation phase Bicubic (PSNR/SSIM/GPU runtime (s)) 2: /0.9350/ : /0.8689/ : /0.8137/0.1 Unknown (PSNR/SSIM/GPU runtime (s)) 2: /0.9209/ : /0.8638/ : /0.8043/0.1 Link to the codes/executables of the solution(s) Codalab Key: 704c9cc9-334d-4b84-b2fc-e88b6a32dbb0 2 Contribution details Title of the contribution Accelerated very deep denoising convolutional neural network for image superresolution General method description Since the LR input and desired HR image have different image size. Proper handling is typically required in SISR methods. Generally, there are two basic ways to handle this. The first one is to first interpolate the LR image via Bicubic method to have the desired size and then learn a mapping function from the Bicubic image to predict the latent HR image. Representative methods of this type include A+ [8], SRCNN [2] and VDSR [5]. While this way is easy to implement, it suffers from the computational burden [7]. In addition, this can greatly hinder the growth of receptive filed. The other way is to directly operate on the LR image and then use some techniques to upscale the learned features to have desired size. Representative methods include the subpixel layer proposed in the ESPCN model [7]and the deconvolution layer exploited in [3]. Compared to the deconvolution layer, the sub-pixel layer of ESPCNN is not only much faster but also allows for more convolution operations on LR space. Therefore, we are motivated to incorporate the sub-pixel layer into our previous denoising convolutional neural network [10] for fast and effective image super resolution. The architecture of our proposed SR network is shown in Figure 1. Our proposed network takes both LR image and its bicubically interpolated one as input. However, the computational convolution does not operate on the bicubically interpolated image. Instead, the bicubically interpolated image acts the role in residual learning. In other words, rather than directly predicting the desired HR image, we use residual learning strategy to predict the residual which is the pixel-level difference between the bicubically interpolated LR image and HR ground-truth image. Formally, we adopt the residual learning to train a residual mapping R(I LR ) = I HR I LR s bicubic, where I HR/I LR represents HR/LR image, s bicubic denotes bicubic interpolation with upscaling factor s. We empirically find that predicting the residual can accelerate training. This is probably due to the fact that the residual has a rather simpler distribution compared to the HR image. 2

3 Our proposed SR network takes the color LR image as input. For a SR network with upscaling factor s, we use 128 filters of size to generate 128 feature maps in the first convolution layer, while in the last convolution layer, we use 3s 2 filter of size In the middle layers, we use 128 filters of size In the final sub-pixel layer, 3s 2 LR feature maps are merged into the residual with desired size via the sub-pixel layer. The depth is set to 20. We pad zeros before each convolution to make sure that each feature map of the middle layers has the same size of the input image. We exploit a leaky ReLU function as the activation function. f(x) = max(x, 0.05x). We separately learn a model for each upcaling factor in the two tracks of the NTIRE 2017 SR Challenge. However, our SR network is easy to extend to handle various upscaling factors simultaneously with a single model. Conv+LReLU A stack of Conv+LReLU layers Conv Bicubic LR input Sub-pixel shuffling Residual Image HR Figure 1: Network architecture of our proposed SR model. Description of the particularities of the solutions deployed for each of the challenge competitions or tracks We make use of an unified model setting for all the SR tasks of the challenge competition. The sole difference among the solutions for each of the challenge competitions is training samples. References Please see the References section. Representative image / diagram of the method(s) Please see the Figure 1. 3 Global Method Description Total method complexity: all stages The main computation burden of our proposed deep-cnn based SR method is 3

4 the 20 convolution layers. The filter size is fixed to 3 3. For the case of 2, the number of 2D convolution is N = = ; for the case of 3, the number of 2D convolution is N = = , and for the case of 4, the number of 2D convolution is N = = Note that the input image of our deep-cnn model is the low-resolution image itself. Assuming that the dimension of the input image is m n, then the total computational complexity of our proposed method in terms of float-pointoperation (FLOP) is m n 3 3 N = 9Nmn. Which pre-trained or external methods / models have been used (for any stage, if any) Our deep-cnn based SR models are trained from scratch. Which additional data has been used in addition to the provided NTIRE training and validation DIV2K data (at any stage, if any) Training description We cropped HR/LR sub-image pairs from the provided 800 DIV2K training samples for both the Bicubic and unknown down-sampling case. In our training, the unknown down-sampling case was handled in the exactly same way as the Bicubic down-sampling case. This is reasonable because we find that although in the unknown down-sampling case, the LR image generation way is unknown, all the images are generated in the same way. In the case of 2, 3 and 4, the size of the LR sub-images is chosen as 45 45, and 41 41, respectively. The size of the corresponding HR sub-image is 90 90, , and , respectively. For each case, we extracted approximate 250K sub-images. We trained a CNN model for each individual SR task, namely, we have 6 models for this SR challenge. Our CNN model was initialized by the method [4] and optimized via the Adam [6] algorithm. We made use of a mini-batch size of 128. We trained 50 epochs for our CNN models. The learning rate was decayed exponentially from 10 3 to 10 5 for the 50 epochs. We use the MatConvNet package [9] to train the proposed CNN models. The training was done with a Titan X Pascal GPU. It takes about 24 hours to finish the training of a single model. Testing description We can directly apply the trained CNN models to the corresponding SR test images without any preprocessing and post-processing. Quantitative and qualitative advantages of the proposed solution Due to the great approximation ability of the exploited deep-cnn model, it can 4

5 directly handle the case of unknown down-sampling, where the resulting LR images are very different from those LR images generated from Bicubic downsampling. For example, in the case of 4, the LR images in the unknown case are much more blurred. Nevertheless, our deep-cnn based SR methods successfully handle these unknown cases very well in a implicit way, with just slightly inferior PSNR/SSIM performance compared to the Bicubic cases. The resulting performance on the validation dataset reads as follows: Bicubic (PSNR/SSIM) 2: / : / : / Unknown (PSNR/SSIM) 2: / : / : / Results of the comparison to other approaches (if any) Please see the following aspect. Results on other standard benchmarks such as Set5, Set14, B100, Urban100 (if any) Dataset BSD / / / Urban / / / Table 1: Test performance of our trained CNN based SR models on the BSD100 and Urban100 dataset. The SR results of our previous CNN based model [10], TNRD [1] and VDSR [5] on the BSD100 and Urban100 dataset can be found at com/cszn/dncnn. Novelty degree of the solution and if it has been previously published Our proposed SR solution is an improved version of our previous deep-cnn based model - DnCNN [10]. In this improved version, we significantly accelerate the DnCNN model by introducing the subpixel layer, proposed in [7] for image upscaling in the last layer. Therefore, the input image of the proposed model is the LR image itself, instead of the Bicubic interpolated image, which is the input of the DnCNN model. As a consequence, the computational complexity can be greatly reduced. Meanwhile, this acceleration strategy does not harm the SR performance. 4 Track 1: Bicubic downsampling Any particularities of the deployed solution for the Track 1 competitions ( 2, 3, 4) (if applicable) As mentioned before, our SR solutions make use of an unified model setting. 5

6 5 Track 2: Unknown downsampling Any particularities of the deployed solution for the Track 2 competitions ( 2, 3, 4) (if applicable) 6 Ensembles and fusion strategies Describe in detail the use of ensembles and/or fusion strategies (if any). Our proposed SR solution is a standalone model, and we do not make any fusion strategy. What was the benefit over the single method? What were the baseline and the fused methods? 7 Technical details Language and implementation details (including platform, memory, parallelization requirements) Our solutions are implemented in Matlab with GPU computation. In the training phase, the GPU memory usage is about 6GB. In the test phase, we work in a conservememory way, and for the case of 4, 3 and 2, it takes about 825MB, 1242MB and 1657MB, respectively. Human effort required for implementation, training and validation? Before running a training experiment for a SR task in the Challenge, we have to prepare the training samples. We run a script to extract approximate 250K subimages from the training dataset DIV2K images, and then we run a training experiment. There are some hyper-parameters, such as number of layers, number of feature maps and activation function, etc. We run a few training experiments with different settings, and finally choose the best trade-off between runtime and test performance. Training/testing time? Runtime at test per image. The training was done with a Titan X Pascal GPU. It takes about 24 hours to finish the training of a single model. In the test phase, the average runtime on the test images for the case of 2, 3 and 4 is 0.1s, 0.16s and 0.36s, respectively. Comment the robustness and generality of the proposed solution(s)? Is it easy to deploy it for other sets of downscaling operators? We train different models for 6 different down-sampling cases in this challenge. The trained models are highly tailored to the specialized downscaling operators. For example, the model trained to handle the case of unknown 4 will not work 6

7 for the case of Bicubic 4. It is easy to deploy our solution for other sets of downscaling operators, as long as the downscaling operator is fixed. It does not matter if the downscaling operator is known or unknown. What does matter is that the downscaling operator should keep unchanged. Then, our proposed deep CNN based SR model is able to produce good results. We believe that this property of our proposed SR model originates from the good approximation capability of multi-layer CNN models. Comment the efficiency of the proposed solution(s)? For the test images in this challenge, our models take a fraction of a second on a modern GPU. We think the efficiency of our proposed solutions are moderate. 8 Other details Planned submission of a solution(s) description paper at NTIRE2017 workshop. We do not plan to submit the SR solution described above to the NTIRE2017 workshop. General comments and impressions of the NTIRE2017 challenge. We think it is generally a good SR competition, but the setup of the unknown cases can be improved. Assuming that the LR image is generated in the way that f = DBu, where B is a blurring kernel, D is a decimation matrix. For the unknown cases, the main problem of the current setting is that the down-sampling operators are fixed, even though different scale factors correspond to different blurring kernels. In the case of a specific scale factor, as the down-sampling operator is exactly the same for all images, deep CNN based models can work in a lazy way - exactly following the work flow to handle the Bicubic down-sampling case, just retraining the CNN model with newly extracted training samples. CNN based models can work in this way as the extra blurring effect can be implicitly overcome by its great approximation capability as long as the the down-sampling process keeps unchanged for all images. A better setup for unknown case would be to exploit clearly different downsampling processes for each individual image, e.g., different blurring kernels. Given the large degeneration diversity in the resulting LR images, we believe that a CNN model working for the Bicubic case can not be straightforwardly applied to this circumstance any more. Some extra efforts should be made to properly handle the extra blurring effect. Moreover, we believe that this is a more realistic setting for real SR applications. What do you expect from a new challenge in image restoration and enhancement? Probably blind image denoising and blind image deconvolution. 7

8 Other comments: encountered difficulties, fairness of the challenge, proposed subcategories, proposed evaluation method(s), etc. References [1] Yunjin Chen and Thomas Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, [2] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2): , [3] Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In European Conference on Computer Vision, pages Springer, [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages , [5] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , [6] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv preprint arxiv: , [7] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient subpixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , [8] Radu Timofte, Vincent De Smet, and Luc Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian Conference on Computer Vision, pages Springer, [9] Andrea Vedaldi and Karel Lenc. Matconvnet: Convolutional neural networks for matlab. In Proceedings of the 23rd ACM international conference on Multimedia, pages ACM, [10] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing,