Deep Back-Projection Networks For Super-Resolution Supplementary Material

Transcription

1 Deep Back-Projection Networks For Super-Resolution Supplementary Material Muhammad Haris 1, Greg Shakhnarovich 2, and Norimichi Ukita 1, 1 Toyota Technological Institute, Japan 2 Toyota Technological Institute at Chicago, United States {mharis, ukita}@toyota-ti.ac.jp, gregory@ttic.edu Project Page 1. Overview We provide additional materials for better understanding of our proposed networks. First, we provide the detailed architectures from the variants of DBPN. Second, we present additional analysis of DBPN. Last, we provide additional qualitative results from our networks compare to the stateof-the-arts methods. 2. Implementation Details of Networks Architecture There are six variants of DBPN which is shown in the paper: DBPN-SS, DBPN-S, DBPN-M, DBPN-L, D-DBPN-L, and D-DBPN. The detailed architectures of those networks are shown in Table Additional Analysis 3.1. Sanity Check (R1 and R2) As an additional experiment, suggested by R1 and R2, we compared our network with the same network capacity and training set. We use DBPN-S (our shallow network with 599k parameters) and LapSRN [5], a state-of-the-art network with 812k parameters. The R1 suggested training our network on BSDS200 [1] and T91 [9] dataset. However, the modern deep learning methods should take beneficial on performing in larger dataset, such as DIV2K. Therefore, we should note that the smaller BSDS200 [1] and T91 [9] dataset may be less appropriate for training modern deep super-resolution methods, in particular due to lack of image variance. Moreover, BSDS200 is not intended, and may be ill-suited, for super-resolution training. Instead, larger dataset is used for fair comparison between DBPN-S and LapSRN which is DIV2K [8]. In Table 2, we report preliminary results of training with DIV2K, with total number 800 training images, on the x4 super-resolution task. For reference, we also include original LapSRN trained on BSDS200 [1] + T91 [9] dataset. From the table, it s evident that our network has overall better performance than either version of LapSRN, and in particular significantly better results on BSDS100, Urban100, and Manga109. Please be noted also that the gap is higher on our deeper network such as D-DBPN as shown in the manuscript s Fig. 8 and 9 which shows the efficiency of utilizing deeper network and large dataset Error Feedback As stated in our manuscript, error feedback (EF) is used to guide the reconstruction in the early layer. Here, we analyze how error feedback can help for better reconstruction. For the scenario without EF, we replace up- and downprojection unit with single up- and down-sampling (deconvolution and convolution) layer. We show PSNR of DBPN-S with EF and without EF in Table 3. The result with EF has 0.53 db and 0.26 db better than without EF on Set5 and Set14, respectively. In Fig. 1, we visually show how error feedback can construct better and sharper HR image especially in the white stripe pattern of the wing. The performance of DBPS-S without EF is interestingly 0.57 db and 0.35 db better than SRCNN [2], FSRCNN [3], respectively, on Set5. These results show the effectiveness of our mutual-connected up- and downsampling layers which can demonstrate the LR-HR mutual dependency by mapping LR features to HR space, then project it back to the LR space Convergence Curve In Fig. 2 and Fig. 3, we show the convergence curve of 4 and 8 enlargement from each proposed network in the manuscript. Our proposed networks have fast convergence speed especially for D-DBPN where the results of 50k iteration can outperform the state-of-the-art methods except for EDSR. 1

2 Table 1. Network Architecture of DBPN variants. Feat0 and Feat1 refer to first and second convolutional layer in the initial feature extraction stages. Note: conv(f, n, st, pd) where f is filter size, n is number of filters, st is striding, and pd is padding Scale DBPN-SS DBPN-S DBPN-M DBPN-L D-DBPN-L D-DBPN Input/Output Luminance Luminance Luminance Luminance Luminance RGB Feat0 conv(3,64,1,1) conv(3,128,1,1) conv(3,128,1,1) conv(3,128,1,1) conv(3,128,1,1) conv(3,256,1,1) Feat1 conv(1,18,1,0) conv(1,32,1,0) conv(1,32,1,0) conv(1,32,1,0) conv(1,32,1,0) conv(1,64,1,0) Reconstruction conv(1,1,1,0) conv(1,1,1,0) conv(1,1,1,0) conv(1,1,1,0) conv(1,1,1,0) conv(3,3,1,1) 2 conv(6,18,2,2) conv(6,32,2,2) conv(6,32,2,2) conv(6,32,2,2) conv(6,32,2,2) conv(6,64,2,2) BP stages 4 conv(8,18,4,2) conv(8,32,4,2) conv(8,32,4,2) conv(8,32,4,2) conv(8,32,4,2) conv(8,64,4,2) 8 conv(12,18,8,2) conv(12,32,8,2) conv(12,32,8,2) conv(12,32,8,2) conv(12,32,8,2) conv(12,64,8,2) Parameters (k) Depth No. of stage (T ) Dense connection No No No No Yes Yes Figure 1. Qualitative comparisons of DBPN-S with EF and without EF on 4 enlargement Filter Size We analyze the size of filters which is used in the backprojection stage. In the manuscript, we stated that the choice of filter size in the back-projection stage is based on the preliminary results. For the 4 enlargement, we show that filter 8 8 is 0.08 db and 0.09 db better than filter 6 6 and 10 10, respectively, as shown in Table Luminance vs RGB In the final network (D-DBPN), we change input/output from luminance to RGB color channels. There is no significant improvement in the quality of the result as shown in Table 5. However, it might reduce the complexity and simplify the implementation by avoiding the use of another interpolation techniques, such as Bicubic, to process other channels. Table 2. Comparison of the DBPN-S and LapSRN-DIV2K on 4 enlargement. DBPN-S LapSRN-DIV2K LapSRN [5] Algorithm PSNR SSIM PSNR SSIM PSNR SSIM Set Set BSDS Urban Manga Table 3. Analysis of EF using DBPN-S on 4 enlargement. Red indicates the best performance. Set5 Set14 SRCNN [2] FSRCNN [3] Without EF With EF Table 4. Analysis of filter size in the back-projection stages on 4 enlargement from D-DBPN. Red indicates the best performance. Filter size Striding Padding Set5 Set Table 5. Analysis of input/output color channel using DBPN-L. Red indicates the best performance. Set5 Set14 RGB Runtime Evaluation Luminance We present the runtime comparisons between our networks and 3 state-of-the-art networks: VDSR [4], DRRN [7], and EDSR [6]. The comparison must be done in fair settings. Therefore, we choose only three methods which have the same in nature with our implementation using Caffe. The runtime is calculated using python function timeit which encapsulating forward function in Caffe. For EDSR, we use original author code based on Torch and use timer function to obtain the runtime. We evaluate each network using Nvidia TITAN X GPU

3 Figure 2. Convergence curve for 4 enlargement on Set5. Figure 3. Convergence curve for 8 enlargement on Set5. (12G Memory). The input image size is 64 64, then upscaled into (2 ), (4 ), and (8 ). The results are the average of 10 times trials. Table 6 shows the runtime comparisons on 2, 4, and 8 enlargement. It shows that our SS and S networks obtain the best and second best performance on 4 and 8 enlargement. On 2 enlargement, we did not construct the variants of our proposed network except for D-DBPN. Therefore, we cannot produce the runtime for SS, S, M, and L networks. Compare to EDSR, our final network (D- DBPN) show its effectiveness by having faster runtime with comparable quality on 2 and 4 enlargement. On 8 enlargement, the gap is bigger. It shows that D-DBPN has better results with lower runtime than EDSR. Noted that input for VDSR and DRRN is only luminance channel and need preprocessing to create middle-resolution

4 image. So that, the runtime should be added by additional computation of interpolation computation on preprocessing. Table 6. Runtime evaluation with input size Red indicates the best and blue indicates the second best performance, * indicates the calculation using function timer in Torch, and N.A. indicates that the algorithm runs out of GPU memory ( ) ( ) ( ) VDSR [4] DRRN [7] N.A. *EDSR [6] DBPN-SS DBPN-S DBPN-M DBPN-L D-DBPN [7] Y. Tai, J. Yang, and X. Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 4 [8] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang, B. Lim, S. Son, H. Kim, S. Nah, K. M. Lee, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages IEEE, [9] J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image superresolution via sparse representation. Image Processing, IEEE Transactions on, 19(11): , Additional Qualitative Results In Fig. 4-16, we provide additional results for 8 enlargement to clearly show the effectiveness of our proposed network. The comparisons focus to compare between top-3 current state-of-the-art networks which are LapSRN [5], EDSR [6], and D-DBPN. The complete results on all datasets will be published in our website. References [1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5): , [2] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2): , , 2 [3] C. Dong, C. C. Loy, and X. Tang. Accelerating the superresolution convolutional neural network. In European Conference on Computer Vision, pages Springer, , 2 [4] J. Kim, J. Kwon Lee, and K. Mu Lee. Accurate image superresolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , June , 4 [5] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep laplacian pyramid networks for fast and accurate superresolution. In IEEE Conferene on Computer Vision and Pattern Recognition, , 2, 4 [6] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July , 4

5 Figure 4. Visual comparison for 8 enlargement. D-DBPN is able to separate clearly between the hiragana word and outer stripe pattern. Figure 5. Visual comparison for 8 enlargement. All networks fail to keep the shape consistency from the HR image. However, the correct number of holes in the image is only achieved by D-DBPN. Figure 6. Visual comparison for 8 enlargement. D-DBPN is able to construct shaper eyelashes close to the ground truth. Figure 7. Visual comparison for 8 enlargement. D-DBPN is able to construct sharper edges. However, it also creates soft black stripes in the middle part of the wall.

6 Figure 8. Visual comparison for 8 enlargement. D-DBPN is able to construct sharper edges from the windows. Figure 9. Visual comparison for 8 enlargement. D-DBPN is able to construct more detailed patterns compare to LapSRN and EDSR. Figure 10. Visual comparison for 8 enlargement. D-DBPN is able to preserve the stripe pattern in the wall. Figure 11. Visual comparison for 8 enlargement. D-DBPN is able to construct the white stripes better than LapSRN and EDSR.

7 Figure 12. Visual comparison for 8 enlargement. D-DBPN is able to construct sharper the blue bars pattern. Figure 13. Visual comparison for 8 enlargement. D-DBPN is able to construct sharper pattern of 2 than LapSRN and EDSR. Figure 14. Visual comparison for 8 enlargement. D-DBPN is able to construct the characters sharper than LapSRN and EDSR. Figure 15. Visual comparison for 8 enlargement. D-DBPN is able to construct the bars in the window.

8 Figure 16. Visual comparison for 8 enlargement. D-DBPN is able to preserve the sketch pattern (light black stripes) in the image better than LapSRN and EDSR.