Blind Image Deblurring Using Dark Channel Prior Jinshan Pan 1,2,3, Deqing Sun 2,4, Hanspeter Pfister 2, and Ming-Hsuan Yang 3 1 Dalian University of Technology 2 Harvard University 3 UC Merced 4 NVIDIA
Overview Blurred image captured in low-light conditions 2
Overview Restored image 3
Overview Blurred image 4
Overview Restored image 5
Overview Our goal A generic method state-of-the-art performance on natural images great results on specific scenes (e.g. saturated images, text images, face images) No edge selection for natural image deblurring No engineering efforts to incorporate domain knowledge for specific scenario deblurring 6
Blur Process Blur is uniform and spatially invariant Blurred image Sharp image Blur kernel Noise 7
Blur Process Blur is uniform and spatially invariant Blurred image Sharp image Blur kernel Noise Convolution operator 8
Challenging Blind image deblurring is challenging?? 9
Ill-Posed Problem 10
Ill-Posed Problem 11
Related Work Probabilistic approach Posterior distribution Likelihood Prior on I Prior on k p k, I B p B I, k p I p k Blur kernel k Latent image I Blurred image B 12
Related Work Blur kernel prior Positive and sparse 70 60 50 Most elements near zero P(k) p(b) 40 30 20 A few can be large 10 Shan et al., SIGGRAPH 2008 0 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 k b 13
Related Work Sharp image statistics Fergus et al., SIGGRAPH 2006, Levin et al, CVPR 2009, Shan et al., SIGGRAPH 2008 Histogram of image gradients Log # pixels 14
Related Work Gaussian: -I 2 Log prob - I 0.5 Laplacian: - I - I 0.25 I I Parametric models Derivative distributions in natural images are sparse: log pi ( ) = I α i, α < 1 Levin et al., SIGGRAPH 2007, CVPR 2009 i 15
Related Work MAP I,k framework p k, I B p B I, k p I p k argmax k,i p k, I B (I, k) = argmin k,i {l B I k + φ I + φ k } 16
Related Work The MAP I,k paradox [Levin et al., CVPR 2009] P(, )>P(, ) Latent image kernel Latent image kernel 17
Related Work The MAP I,k paradox [Levin et al., CVPR 2009] sharp blurred i α <? 18 i α
Related Work The MAP I,k paradox [Levin et al., CVPR 2009] 15x15 windows 25x25 windows 45x45 windows simple derivatives [-1,1],[-1;1] FoE filters (Roth&Black) Red windows = [ p(sharp I) >p(blurred I) ] 19
Related Work The MAP I,k paradox [Levin et al., CVPR 2009] P(step edge) d = 1 < k=[0.5,0.5] P(blurred step edge) d1 = 0.5 d2 = 0.5 sum of derivatives: cheaper 0 = 0.5 0.5 1 0.5 + 0.5 = 1. 41 1. 5 P(impulse) < P(blurred impulse) sum of derivatives: d =1 1 1 d 2 = 1 0.5 d = 0.5 d = 0. 5 1 2 0.5 0.5 0.5 + 1 = 2 0.5 + 0.5 = 1. 41 cheaper 20
Related Work The MAP I,k paradox [Levin et al., CVPR 2009] P(sharp real image) < P(blurred real image) 0.5 I i i = 5.8 0.5 I i i = 4.5 cheaper Noise and texture behave as impulses - total derivative contrast reduced by blur 21
Related Work The MAP I,k paradox [Levin et al., CVPR 2009] Maximum marginal probability estimation Marginalized probability [Levin et al., CVPR 2011] Variational Bayesian [Fergus et al., SIGGRAPH 2006] MAP I,k p k B p B k p k = p B, I k p k di I = p B I, k p(i)p k di l p k, I B p B I, k p(i)p k Marginalizing over I 22
Related Work The MAP I,k paradox [Levin et al., CVPR 2009] Maximum marginal probability estimation Marginalized probability [Levin et al., CVPR 2011] Variational Bayesian [Fergus et al., SIGGRAPH 2006] Score Optimization surface for a single variable Maximum a-posteriori (MAP) Variational Bayes Pixel intensity Computationally expensive 23
Related Work Priors favor clear images [Krishnan et al., CVPR 2011, Pan et al., CVPR 2014, Michaeli and Irani, ECCV 2014] E(Clear image) < E(Blurred image) Effective for some specific images, such as natural images or text images Cannot be generalized well 24
Related Work MAP I,k with Edge Selection Main idea E(clear) < E(blurred) in sharp edge regions [Levin et al., CVPR 2009] [Cho and Lee SIGGRAPH Asia 2009, Xu and Jia ECCV 2010, ] Advantages and Limitations Fast and effective in practice Explicitly try to recover sharp edges using heuristic image filters and usually fail when sharp edges are not available 25
Related Work MAP I,k with Edge Selection (Extension) Exemplar based methods [Sun et al., ICCP 2013, HaCohen et al., ICCV 2013, Pan et al., ECCV 2014] Computationally expensive 26
Our Work Dark channel prior Theoretical analysis Efficient numerical solver Applications 27
Convolution and Dark Channel Dark channel [He et al., CVPR 2009] c DI ( )( x) = min I( y) y N( x) c { rgb,, } Compute the minimum intensity in a patch of an image 28
Convolution and Dark Channel Convolution s B(x) = I(x+[ ] - z) k(z) 2 z Ω k Ω k: the domain of blur kernel s : the size of blur kernel [ ] : the rounding operator k(z) 0, k(z)=1 z Ω k 29
Convolution and Dark Channel Proposition 1: Let N(x) denote a patch centered at pixel x with size the same as the blur kernel. We have: B(x) min I(y) y N(x) 54 0 73 141 49 89 183 0 86 32 35 63 27 18 0 72 0 0 142 103 81 73 9 7 0 29 0 9 0 0 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 30 0 0 255 149 163 0 146 A toy example 30
Convolution and Dark Channel Property 1: Let D(B) and D(I) denote the dark channel of the blurred and clear images, we have: D( B)(x) D( I)(x) Property 2: Let Ω denote the domain of an image I. If there exist some pixels x Ω such that I(x) = 0, we have: D( B) > D( I) 0 0 31
Convolution and Dark Channel Average number of dark pixels 60000 40000 20000 0 Blurred images Clear images 0 0.01 0.02 0.03 0.04 0.05 Intensity The statistical results on the dataset with 3,200 examples 32
Convolution and Dark Channel Clear Blurred Clear Blurred Blurred images have less sparse dark channels than clear images 33
Proposed Method Our model Add the dark channel prior into standard deblurring model min I* k B + γ k + µ I + λ DI ( ) Ik, How to solve? 2 2 2 2 0 0 L0 norm and non-linear min operator 34
Optimization Algorithm skeleton min I* k B +γ k k 2 2 2 2 L0 norm min I* k B + µ I + λ DI ( ) I Half-quadratic splitting method Non-linear min operator Linear approximation 2 2 0 0 35
Optimization Update latent image I: min I * k B + µ I + λ DI ( ) I 2 2 0 0 Half-quadratic splitting [Xu et al., SIGGRAPH Asia 2011, Pan et al., CVPR 2014] min I* k B + α I g + β DI ( ) u + µ g + λ u Iug,, 2 2 Alternative minimization 2 2 2 2 0 0 min I* k B + β DI ( ) u + µ I g I 2 2 2 2 2 2 min α β µ λ ug, 2 2 I g + DI ( ) u + g 0 + u 0 36
Optimization Update latent image I: u, g sub-problem α I g β DI u + µ g + λ u 2 2 min + ( ) 0 0 ug, + 2 min β DI ( ) u λ u 0 u 2 min α x g + µ g 0 g u and g are independent! u 2 2 DI ( ), DI ( ) = β g 0, otherwise λ µ I, I = α 0, otherwise Related papers: [Xu et al., SIGGRAPH Asia 2011, Xu et al., CVPR 2013, Pan et al., CVPR 2014] 37
Optimization Update latent image I: I sub-problem min operator min I* k B + β DI ( ) u + µ I g I Our observation 2 2 2 2 DI ( )=MI Let y = argmin z N x I(z), we have 1, z=y, M(x, z)= 0, otherwise. 38
Optimization I sub-problem Compute M M Intermediate image I D(I) M T Visualization of M T u u Toy example 39
Experimental Results Natural image deblurring Specific scenes Text images Face images Low-light images Non-uniform image deblurring 40
Natural Image Deblurring Results Quantitative evaluation Levin et al., CVPR 2009 Köhler et al. ECCV 2012 Sun et al., ICCP 2013 41
Natural Image Deblurring Results 100 100% Success rate (%) Ours 80 Xu et al. Xu and Jia 60 Pan et al. Levin et al. 40 Cho and Lee Michaeli and Irani Krishnan et al. 20 1.5 2 2.5 3 3.5 4 Error ratios Quantitative evaluations on the dataset by Levin et al., CVPR 2009 42
Natural Image Deblurring Results Average PSNR Values 33 30 27 24 21 18 im01 im02 im03 im04 Average Blurred images Fergus et al. Shan et al. Cho and Lee Xu and Jia Krishnan et al. Hirsch et al. Whyte et al. Pan et al. Ours Quantitative evaluations on the dataset by Köhler et al. ECCV 2012 43
Natural Image Deblurring Results Success rate (%) 100 80 60 40 20 0 1 2 3 4 5 6 Error ratios Ours Xu and Jia Pan et al. Michaeli and Irani Sun et al. Xu et al. Levin et al. Krishnan et al. Cho and Lee Quantitative evaluations on the dataset by Sun et al. ICCP 2013 44
Natural Image Deblurring Results Blurred image Cho and Lee SIGGRAPH Asia 2009 Xu and Jia, ECCV 2010 Krishnan et al., CVPR 2011 Ours without D(I) Ours 45
Natural Image Deblurring Results Blurred image Krishnan et al., CVPR 2011 Xu et al., CVPR 2013 Pan et al., CVPR 2014 Ours without D(I) Ours Our real captured example 46
Text Image Deblurring Results Average PSNRs Cho and Lee 23.80 Xu and Jia 26.21 Krishnan et al. 20.86 Levin et al. 24.90 Xu et al. 26.21 Pan et al. 28.80 Ours 27.94 Natural image debluring methods Quantitative evaluations on the text image dataset by Pan et al., CVPR 2014 47
Text Image Deblurring Results Blurred image Xu et al., CVPR 2013 Pan et al., CVPR 2014 Real captured example Ours 48
Saturated Image Deblurring Results Blurred image Xu et al., CVPR 2013 Pan et al., CVPR 2014 Real captured example Ours 49
Face Image Deblurring Results Blurred image Xu et al., CVPR 2013 Pan et al., ECCV 2014 Ours Real captured example 50
Non-Uniform Deblurring Blurred image Krishnan et al., CVPR 2011 Whyte et al., IJCV 2012 Xu et al., CVPR 2013 Ours Our estimated kernels 51
Convergence 0.82 160 Average Kernel Similarity 0.8 0.78 0.76 1 12 23 34 45 Average Energies 120 80 40 1 12 23 34 45 Iterations Iterations Kernel similarity plot Objective function value plot 52
Running Time Running time (/s) comparisons (obtained on the same PC). Method 255 x 255 600 x 600 800 x 800 Xu et al. (C++) 1.11 3.56 4.31 Krishnan et al. (Matlab) 24.23 111.09 226.58 Levin et al. (Matlab) 117.06 481.48 917.84 Ours without D(I) (Matlab) 2.77 15.65 28.94 Ours with naive implementation (Matlab) 134.31 691.71 964.90 Ours (Matlab) 17.07 115.86 195.80 53
Analysis and Discussions Effectiveness of dark channel prior Average PSNR Values 33 31 29 27 25 Ours without dark channel Ours im01 im02 im03 im04 Average Results on the dataset by Köhler et al. ECCV 2012 Success rate (%) 105 95 85 75 Ours without dark channel 1.5 2 2.5 3 Error ratios Ours Results on the dataset by Levin et al. CVPR 2009 54
Analysis and Discussions Existing prior favors clear images [Krishnan et al. CVPR 2011] Energy Values of p(i) 80000000 60000000 40000000 20000000 0 I pi ( ) = I 1 14 27 40 53 66 79 92 105 118 Image index 1 2 Blurred images Clear images Statistics of different priors on the text image deblurring by Pan et al., CVPR 2014. The normalized sparsity sometimes favors blurred text images 55
Analysis and Discussions Dark channel prior favors clear images Average number of dark pixels 80000 60000 40000 Blurred images 20000 Clear images 0 0 0.2 0.4 0.6 Intensity Statistics of different priors on the text image deblurring by Pan et al., CVPR 2014. The dark channel prior favors clear text images 56
Limitations The dark channel of clear image does not contain zero-elements DB ( ) = DI ( ) 0 0 Property 2 does not hold Dark channel prior has no effect on image deblurring 57
Limitations The dark channel of clear image does not contain zero-elements Blurred image Without D(I) Ours Dark channel of clear image Dark channel of blurred image Estimated dark channel Dark channel prior has no effect on image deblurring 58
Limitations Images containing noise Blurred image 59
Limitations Images containing noise Without D(I) 60
Limitations Images containing noise With D(I) 61
Take Home Message The change in the sparsity of the dark channel is an inherent property of the blur process! Code and datasets will be available at the authors websites. 62
More Results Real captured image 63
More Results Our result 64
More Results Real captured image 65
More Results Our result 66
Our Related Deblurring Work Outlier deblurring (CVPR 2016) http://vllab.ucmerced.edu/~jinshan/projects/outli er-deblur/ Object motion deblurring (CVPR 2016) http://vllab.ucmerced.edu/~jinshan/projects/obje ct-deblur/ Text image deblurring and beyond (TPAMI 2016) http://vllab.ucmerced.edu/~jinshan/projects/textdeblur/ 67