Blind Image Deblurring Using Dark Channel Prior

Transcription

1 Blind Image Deblurring Using Dark Channel Prior Jinshan Pan 1,2,3, Deqing Sun 2,4, Hanspeter Pfister 2, and Ming-Hsuan Yang 3 1 Dalian University of Technology 2 Harvard University 3 UC Merced 4 NVIDIA

2 Overview Blurred image captured in low-light conditions 2

3 Overview Restored image 3

4 Overview Blurred image 4

5 Overview Restored image 5

6 Overview Our goal A generic method state-of-the-art performance on natural images great results on specific scenes (e.g. saturated images, text images, face images) No edge selection for natural image deblurring No engineering efforts to incorporate domain knowledge for specific scenario deblurring 6

7 Blur Process Blur is uniform and spatially invariant Blurred image Sharp image Blur kernel Noise 7

8 Blur Process Blur is uniform and spatially invariant Blurred image Sharp image Blur kernel Noise Convolution operator 8

9 Challenging Blind image deblurring is challenging?? 9

10 Ill-Posed Problem 10

11 Ill-Posed Problem 11

12 Related Work Probabilistic approach Posterior distribution Likelihood Prior on I Prior on k p k, I B p B I, k p I p k Blur kernel k Latent image I Blurred image B 12

13 Related Work Blur kernel prior Positive and sparse Most elements near zero P(k) p(b) A few can be large 10 Shan et al., SIGGRAPH k b 13

14 Related Work Sharp image statistics Fergus et al., SIGGRAPH 2006, Levin et al, CVPR 2009, Shan et al., SIGGRAPH 2008 Histogram of image gradients Log # pixels 14

15 Related Work Gaussian: -I 2 Log prob - I 0.5 Laplacian: - I - I 0.25 I I Parametric models Derivative distributions in natural images are sparse: log pi ( ) = I α i, α < 1 Levin et al., SIGGRAPH 2007, CVPR 2009 i 15

16 Related Work MAP I,k framework p k, I B p B I, k p I p k argmax k,i p k, I B (I, k) = argmin k,i {l B I k + φ I + φ k } 16

17 Related Work The MAP I,k paradox [Levin et al., CVPR 2009] P(, )>P(, ) Latent image kernel Latent image kernel 17

18 Related Work The MAP I,k paradox [Levin et al., CVPR 2009] sharp blurred i α <? 18 i α

19 Related Work The MAP I,k paradox [Levin et al., CVPR 2009] 15x15 windows 25x25 windows 45x45 windows simple derivatives [-1,1],[-1;1] FoE filters (Roth&Black) Red windows = [ p(sharp I) >p(blurred I) ] 19

20 Related Work The MAP I,k paradox [Levin et al., CVPR 2009] P(step edge) d = 1 < k=[0.5,0.5] P(blurred step edge) d1 = 0.5 d2 = 0.5 sum of derivatives: cheaper 0 = = P(impulse) < P(blurred impulse) sum of derivatives: d =1 1 1 d 2 = d = 0.5 d = = = cheaper 20

21 Related Work The MAP I,k paradox [Levin et al., CVPR 2009] P(sharp real image) < P(blurred real image) 0.5 I i i = I i i = 4.5 cheaper Noise and texture behave as impulses - total derivative contrast reduced by blur 21

22 Related Work The MAP I,k paradox [Levin et al., CVPR 2009] Maximum marginal probability estimation Marginalized probability [Levin et al., CVPR 2011] Variational Bayesian [Fergus et al., SIGGRAPH 2006] MAP I,k p k B p B k p k = p B, I k p k di I = p B I, k p(i)p k di l p k, I B p B I, k p(i)p k Marginalizing over I 22

23 Related Work The MAP I,k paradox [Levin et al., CVPR 2009] Maximum marginal probability estimation Marginalized probability [Levin et al., CVPR 2011] Variational Bayesian [Fergus et al., SIGGRAPH 2006] Score Optimization surface for a single variable Maximum a-posteriori (MAP) Variational Bayes Pixel intensity Computationally expensive 23

24 Related Work Priors favor clear images [Krishnan et al., CVPR 2011, Pan et al., CVPR 2014, Michaeli and Irani, ECCV 2014] E(Clear image) < E(Blurred image) Effective for some specific images, such as natural images or text images Cannot be generalized well 24

25 Related Work MAP I,k with Edge Selection Main idea E(clear) < E(blurred) in sharp edge regions [Levin et al., CVPR 2009] [Cho and Lee SIGGRAPH Asia 2009, Xu and Jia ECCV 2010, ] Advantages and Limitations Fast and effective in practice Explicitly try to recover sharp edges using heuristic image filters and usually fail when sharp edges are not available 25

26 Related Work MAP I,k with Edge Selection (Extension) Exemplar based methods [Sun et al., ICCP 2013, HaCohen et al., ICCV 2013, Pan et al., ECCV 2014] Computationally expensive 26

27 Our Work Dark channel prior Theoretical analysis Efficient numerical solver Applications 27

28 Convolution and Dark Channel Dark channel [He et al., CVPR 2009] c DI ( )( x) = min I( y) y N( x) c { rgb,, } Compute the minimum intensity in a patch of an image 28

29 Convolution and Dark Channel Convolution s B(x) = I(x+[ ] - z) k(z) 2 z Ω k Ω k: the domain of blur kernel s : the size of blur kernel [ ] : the rounding operator k(z) 0, k(z)=1 z Ω k 29

30 Convolution and Dark Channel Proposition 1: Let N(x) denote a patch centered at pixel x with size the same as the blur kernel. We have: B(x) min I(y) y N(x) /9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/ A toy example 30

31 Convolution and Dark Channel Property 1: Let D(B) and D(I) denote the dark channel of the blurred and clear images, we have: D( B)(x) D( I)(x) Property 2: Let Ω denote the domain of an image I. If there exist some pixels x Ω such that I(x) = 0, we have: D( B) > D( I)

32 Convolution and Dark Channel Average number of dark pixels Blurred images Clear images Intensity The statistical results on the dataset with 3,200 examples 32

33 Convolution and Dark Channel Clear Blurred Clear Blurred Blurred images have less sparse dark channels than clear images 33

34 Proposed Method Our model Add the dark channel prior into standard deblurring model min I* k B + γ k + µ I + λ DI ( ) Ik, How to solve? L0 norm and non-linear min operator 34

35 Optimization Algorithm skeleton min I* k B +γ k k L0 norm min I* k B + µ I + λ DI ( ) I Half-quadratic splitting method Non-linear min operator Linear approximation

36 Optimization Update latent image I: min I * k B + µ I + λ DI ( ) I Half-quadratic splitting [Xu et al., SIGGRAPH Asia 2011, Pan et al., CVPR 2014] min I* k B + α I g + β DI ( ) u + µ g + λ u Iug,, 2 2 Alternative minimization min I* k B + β DI ( ) u + µ I g I min α β µ λ ug, 2 2 I g + DI ( ) u + g 0 + u 0 36

37 Optimization Update latent image I: u, g sub-problem α I g β DI u + µ g + λ u 2 2 min + ( ) 0 0 ug, + 2 min β DI ( ) u λ u 0 u 2 min α x g + µ g 0 g u and g are independent! u 2 2 DI ( ), DI ( ) = β g 0, otherwise λ µ I, I = α 0, otherwise Related papers: [Xu et al., SIGGRAPH Asia 2011, Xu et al., CVPR 2013, Pan et al., CVPR 2014] 37

38 Optimization Update latent image I: I sub-problem min operator min I* k B + β DI ( ) u + µ I g I Our observation DI ( )=MI Let y = argmin z N x I(z), we have 1, z=y, M(x, z)= 0, otherwise. 38

39 Optimization I sub-problem Compute M M Intermediate image I D(I) M T Visualization of M T u u Toy example 39

40 Experimental Results Natural image deblurring Specific scenes Text images Face images Low-light images Non-uniform image deblurring 40

41 Natural Image Deblurring Results Quantitative evaluation Levin et al., CVPR 2009 Köhler et al. ECCV 2012 Sun et al., ICCP

42 Natural Image Deblurring Results % Success rate (%) Ours 80 Xu et al. Xu and Jia 60 Pan et al. Levin et al. 40 Cho and Lee Michaeli and Irani Krishnan et al Error ratios Quantitative evaluations on the dataset by Levin et al., CVPR

43 Natural Image Deblurring Results Average PSNR Values im01 im02 im03 im04 Average Blurred images Fergus et al. Shan et al. Cho and Lee Xu and Jia Krishnan et al. Hirsch et al. Whyte et al. Pan et al. Ours Quantitative evaluations on the dataset by Köhler et al. ECCV

44 Natural Image Deblurring Results Success rate (%) Error ratios Ours Xu and Jia Pan et al. Michaeli and Irani Sun et al. Xu et al. Levin et al. Krishnan et al. Cho and Lee Quantitative evaluations on the dataset by Sun et al. ICCP

45 Natural Image Deblurring Results Blurred image Cho and Lee SIGGRAPH Asia 2009 Xu and Jia, ECCV 2010 Krishnan et al., CVPR 2011 Ours without D(I) Ours 45

46 Natural Image Deblurring Results Blurred image Krishnan et al., CVPR 2011 Xu et al., CVPR 2013 Pan et al., CVPR 2014 Ours without D(I) Ours Our real captured example 46

47 Text Image Deblurring Results Average PSNRs Cho and Lee Xu and Jia Krishnan et al Levin et al Xu et al Pan et al Ours Natural image debluring methods Quantitative evaluations on the text image dataset by Pan et al., CVPR

48 Text Image Deblurring Results Blurred image Xu et al., CVPR 2013 Pan et al., CVPR 2014 Real captured example Ours 48

49 Saturated Image Deblurring Results Blurred image Xu et al., CVPR 2013 Pan et al., CVPR 2014 Real captured example Ours 49

50 Face Image Deblurring Results Blurred image Xu et al., CVPR 2013 Pan et al., ECCV 2014 Ours Real captured example 50

51 Non-Uniform Deblurring Blurred image Krishnan et al., CVPR 2011 Whyte et al., IJCV 2012 Xu et al., CVPR 2013 Ours Our estimated kernels 51

52 Convergence Average Kernel Similarity Average Energies Iterations Iterations Kernel similarity plot Objective function value plot 52

53 Running Time Running time (/s) comparisons (obtained on the same PC). Method 255 x x x 800 Xu et al. (C++) Krishnan et al. (Matlab) Levin et al. (Matlab) Ours without D(I) (Matlab) Ours with naive implementation (Matlab) Ours (Matlab)

54 Analysis and Discussions Effectiveness of dark channel prior Average PSNR Values Ours without dark channel Ours im01 im02 im03 im04 Average Results on the dataset by Köhler et al. ECCV 2012 Success rate (%) Ours without dark channel Error ratios Ours Results on the dataset by Levin et al. CVPR

55 Analysis and Discussions Existing prior favors clear images [Krishnan et al. CVPR 2011] Energy Values of p(i) I pi ( ) = I Image index 1 2 Blurred images Clear images Statistics of different priors on the text image deblurring by Pan et al., CVPR The normalized sparsity sometimes favors blurred text images 55

56 Analysis and Discussions Dark channel prior favors clear images Average number of dark pixels Blurred images Clear images Intensity Statistics of different priors on the text image deblurring by Pan et al., CVPR The dark channel prior favors clear text images 56

57 Limitations The dark channel of clear image does not contain zero-elements DB ( ) = DI ( ) 0 0 Property 2 does not hold Dark channel prior has no effect on image deblurring 57

58 Limitations The dark channel of clear image does not contain zero-elements Blurred image Without D(I) Ours Dark channel of clear image Dark channel of blurred image Estimated dark channel Dark channel prior has no effect on image deblurring 58

59 Limitations Images containing noise Blurred image 59

60 Limitations Images containing noise Without D(I) 60

61 Limitations Images containing noise With D(I) 61

62 Take Home Message The change in the sparsity of the dark channel is an inherent property of the blur process! Code and datasets will be available at the authors websites. 62

63 More Results Real captured image 63

64 More Results Our result 64

65 More Results Real captured image 65

66 More Results Our result 66

67 Our Related Deblurring Work Outlier deblurring (CVPR 2016) er-deblur/ Object motion deblurring (CVPR 2016) ct-deblur/ Text image deblurring and beyond (TPAMI 2016) 67