CNN for Low Level Image Processing Huanjing Yue 2017.11 1
Deep Learning for Image Restoration General formulation: min Θ L( x, x) s. t. x = F(y; Θ) Loss function Parameters to be learned Key issues The availability of paired training data The design of learning architecture The definition of loss function 2
General Pipeline Training phase Test phase The problem Architecture design Prepare training data Input: degraded-latent image sample pairs Output: Trained model Model training Degraded Image Trained deep network Restored image Input: Degraded images Output: Restored images 3
Topics Image Super-resolution Image Denoising Image Deraining Image Dehazing 4
Image Super-resolution 5
Super-resolution via CNN (SRCNN) 256 256 (input, bicubic interpolation) 256 256 64 (feature map of Conv1) 256 256 32 (feature map of Conv2) 256 256 (output) f 1 @9 9 1(3), f 2 @1 1 64, f 3 @5 5 32 Dong, Chao, et al. Image super-resolution using deep convolutional networks." IEEE PAMI 38.2 (2016): 295-307. (Its conference version published in ECCV 2014) 6
SRCNN: example feature maps 7
Very deep CNN for SR (VDSR) Training parameters: f@3 3 64 Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. "Accurate image super-resolution using very deep convolutional networks." CVPR, 2016 8
VDSR with and without Residual Learning Performance table (PSNR) for residual and non-residual networks ( Set5 dataset, X2). Residual networks rapidly approach their convergence within 10 epochs. 29.59 9
Insights of VDSR Residual learning is effective The deeper, the better Single network for multiple scaling factors (Top)Our results using a single network for all scale factors. Super-resolved images over all scales are clean and sharp. (Bottom) Results of Dong et al. (SRCNN) ( 3 model used for all scales). Result images are not visually pleasing. To handle multiple scales, existing methods require multiple networks. 10
LapSRN: Laplacian Pyramid SR Network Red arrows indicate convolutional layers. Blue arrows indicate transposed convolutions (upsampling). Green arrows denote element-wise addition operators. f@3 3 64, conv layer depth (d) in each level is 10 for 2, 4 SR, d=5 for 8 SR Lai, Wei-Sheng, et al. "Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution, CVPR, 2017. 11
Main points of LapSRN Robust loss function. Speed. LapSRN is much faster than several CNN based SR methods. This is because it does not require the bicubic interpolation as the input. Progressive reconstruction. It generates multiple intermediate SR predictions in one feed-forward pass through progressive reconstruction using the Laplacian pyramid. 12
Image Denoising 13
DnCNN: Deep Residual Learning Beyond Gaussian Denoising Batch normalization and residual learning are particularly beneficial to Gaussian noise removal Single model for multiple tasks (e.g. super-resolution, deblocking) Training details: f@3 3 64, depth: 17 K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang, "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising," IEEE Trans. on Image Processing, 2017. 14
Effect of Batch Normalization and Residual Learning Learning residual (noise) is easier compared with learning noise free image. 15
Gaussian Denoising Results DnCNN-S: DnCNN model trained by specific noise level. DnCNN-B: DnCNN model trained by multiple noise levels, which could be used for blind Gaussian denoising task. 16
Connecting model based optimization and discriminative learning based methods Model based optimization methods General to handle different image restoration problems The hand-crafted prior may not be strong enough Discriminative learning based methods Data driven end-to-end learning The generality of learned models is limited Can we integrate the model based optimization and discriminative learning to develop a general image restoration method? 17
Half quadratic splitting 18
Image restoration with deep CNN denoiser prior (IRCNN) Plugging the strong CNN denoiser prior into model based optimization Step (a): analytical solution Step (b): deep CNN denoiser K. Zhang, W. Zuo, S. Gu, L. Zhang. "Learning Deep CNN Denoiser Prior for Image Restoration." CVPR 2017. 19
CNN Denoiser s-dconv denotes s-dilated convolution, s = 1, 2, 3 and 4. A dilated filter with dilation factor s can be simply interpreted as a sparse filter of size (2s+1) (2s+1) where only 9 entries of fixed positions are non-zeros. 20
Experiment Results 21
Problem: ADMM solution: Cnn solution: DeepAM: Deeply Aggregated Alternating Minimization for Image Restoration (CVPR 2017) 22
DeepAM: Deeply Aggregated Alternating Minimization for Image Restoration (CVPR 2017) 23
Image Deraining 24
Deep Joint Rain Detection and Removal from a Single Image (CVPR 2017) 25
Results DSC: Y.Luo,Y.Xu,andH.Ji. Removing rain from a single image via discriminative sparse coding. ICCV 2015. LP: Y. Li, R. T. Tan, X. Guo, J. Lu, and M. S. Brown. Rain streak removal using layer priors. CVPR 2016. 26
Removing rain from single images via a deep detail network. CVPR 2017. DSC LP 27
Image Dehazing 28
Single Image Dehazing via Multi-Scale Convolutional Neural Networks (ECCV 2016) Wenqi Ren et al. Single Image Dehazing via Multi-Scale Convolutional Neural Networks, ECCV 2016. 29
Results 30
Open Problems Taking advantage of the classical image restoration methods and applying them to the CNN based restoration. For problems without training-pairs available. e.g. realistic noise removal, underwater image enhancement. Use GAN? Exploring more flexible and light network structures with less parameters to be trained. 31
Thanks! References: Lei Zhang, Image Restoration: From Sparse and Low-rank Priors to Deep Priors. 32
Convolution Layer 33
Convolution Layer 34
Convolution Layer 35
Pooling Layer 36
Non-linear Activation 37
Batch Normalization 38
Loss Function 39 Softmax Softmax Loss http://freemind.pluskid.org/machine-learning/softmax-vs-softmax-loss-numerical-stability/
Questions The rules for designing neural network architecture. Generally, the NN architecture with wider and deeper structure could generate better results. Right? If one NN works well on current problem, could we improve its performance by introducing more parameters, for example, densely connection, make it wider or deeper, introducing some skip connections? The common strategy used for designing NN structure. 40
41