Digital Image Restoration Blur as a chance and not a nuisance Filip Šroubek sroubekf@utia.cas.cz www.utia.cas.cz Institute of Information Theory and Automation Academy of Sciences of the Czech Republic Prague CMP 2017 1
CMP 2017 2
Image Degradation Model channel original image u noise + n acquired image = z convolution CMP 2017 3
Energy Minimization CMP 2017 4
Energy Minimization Data term CMP 2017 5
Energy Minimization Data term Blur has different shapes Compact support Non-negative Preserve energy CMP 2017 Blur regularization Motion blur Out-of-focus & Abberations 6
Energy Minimization Data term Image regularization CMP 2017 7
Statistics of sharp images Intensity 0 0.2 0.4 0.6 0.8 1 Gradient -0.5 0 0.5 CMP 2017 8
Statistics of blurred images Intensity 0 0.2 0.4 0.6 0.8 1 Gradient -0.5 0 0.5 CMP 2017 9
Regularization CMP 2017 10
Energy Minimization NO-BLUR solution: WHY? Regularization favors blurred images CMP 2017 11
Regularization favors blur 1.5 1 0.5 5 10 15 20 CMP 2017 12
Regularization favors blur 1.5 1 Artificially sparsify images 0.5 5 10 15 20 CMP 2017 13
Ad-hoc steps! To avoid no-blur solution: Artificial sharpening Remove spikes Adjusting priors on the fly Hierarchical approach Chan TIP98 Shan SigGraph08 Cho SigGraph09 Xu ECCV09, CVPR13 Almeida TIP10 Krishnan CVPR11 Zhong CVPR13 Sun ICCP13 Michaeli ECCV14 Perrone PAMI15 Pan CVPR16 CMP 2017 14
Artificial Sharpening Blurred image Shock filter Blur prediction h-step Deconvolution u-step CMP 2017 Cho et al., SIGGRAPH 2009 15
Remove Spiky Objects Xu et al., ECCV 2010 CMP 2017 16
Remove Spiky Objects Mask out small objects Xu et al., ECCV 2010 CMP 2017 17
Remove Spiky Objects Reconstructed image with small objects removed Xu et al., ECCV 2010 CMP 2017 18
Hierarchical Deconvolution Scale N-2 Scale N-1 Scale N image blur CMP 2017 19
Bayesian Paradigm a posteriori distribution unknown likelihood given by our problem a priori distribution our prior knowledge CMP 2017 20
Blind deconvolution with MAP CMP 2017 21
Bayesian Paradigm revisited Marginalize the posterior Maximize the marginalized prob. Then maximize the posterior CMP 2017 22
1 no-blur solution 0.9 0.8 0.7 0.6 0.5-1 -0.5 0 0.5 1-1.2-1 -0.8-0.6-0.4 correct solution CMP 2017 23
How to marginalize? If Gaussian distributions analytic solution exists in the form of Gaussian distribution If not (our case) approximation Laplace approximation Factorization with Variational Bayes CMP 2017 24
Variational Bayes Factorization of the posterior and then marginalization is trivial. Every factor q depends on moments of other variables => must be solved iteratively. Miskin AICA00 Fergus SIGGRAPH06 Tzikas TIP09 Whyte IJCV12 Levin PAMI11 Babacan ECCV12 Wipf JMLR14 CMP 2017 25
Marginalization in blind deconvolution CMP 2017 26
Limitations of BD Gaussian noise original SNR=20dB SNR=50dB CMP 2017 27
Limitations of BD Model discrepancies original 7% discrepancy CMP 2017 28
Automatic Relevance Determination (Students t-distribution) Standard data term replaced by CMP 2017 29
CMP 2017 30
Space-variant Blur Object motion Camera motion Optical aberrations Scene depth CMP 2017 31
Approximation of SV Blur Patch-wise convolution General SV PSF Locally convolution may not hold Parametric model (Blur Basis) More accurate Model may not hold Conversion to space-invariant HW design Local object motion Segmentation Line blur Joshi CVPR08, Sorel ICIP09, Ji CVPR12,Sun CVPR15 Whyte CVPR10, Gupta ECCV10, Hirsch ICCV11, Zhang NIPS13 Levin SIGGRAPH08, Ben-Ezra PAMI04, Tai PAMI10, Wavefront coding Levin NIPS06, Shan ICCV07, Dai CVPR08, Chakrabarti CVPR10, Kim CVPR14 CMP 2017 32
Fast Moving Objects D.R.,J.K.,F.S.,L.N.,J.M. arxiv:1611.07889 FMO moves over a distance exceeding its size within exposure time CMP 2017 33
Formation model Alpha matting CMP 2017 34
FMO Appearance Estimation CMP 2017 35
FMO Localization The pipeline consists of three stages: 1) Explorer 2) Detector 3) Tracker CMP 2017 36
Input FMO Video CMP 2017 37
FMO Tracking + Reconstruction CMP 2017 38
Temporal SR - Table tennis CMP 2017 39
Temporal SR - Darts CMP 2017 40
Rotating FMOs Simple convolution replaced by space-variant convolution is a function of trajectory and angular velocity is a 3D object (sphere) Gradient descent w.r.t Exhaustive search w.r.t angular velocity CMP 2017 41
Temporal SR with Rotation Input video frame Estimated high-speed video CMP 2017 42
Limitations Poor collaboration between tracking and restoration Estimated blur and FMO appearance improve tracking Unstable FMO appearance estimation Combine multiple video frames Track multiple FMOs CMP 2017 43
Thank You For Your Attention CMP 2017 44
Multichannel Model channel K channel 2 channel 1 original image noise acquired images [u u * h k ] + n k = z k CMP 2017 46
Multichannel Model Acquisition model: Optimization problem Data term Gaussian noise L2 norm CMP 2017 Image regularization term Blur Regularization term 47
Multichannel Blur Regularization u z 1 = u * h 1 h 2 * u = z 2 z 1 * = u * h 1 * * h 2 * u = * z 2 0 CMP 2017 48
Camera motion CMP 2017 49
Astronomical images Degraded images Reconstructed image PSF estimation CMP 2017 50
MC Model with Decimation channel K channel 2 channel 1 original image noise acquired images D [u u * h k ] + n k = z k CMP 2017 51
Robustness to misalignment original image PSF degraded image CMP 2017 52
Super-resolution rough registration Superresolved image (2x) CMP 2017 Optical zoom (ground truth) 53
original 8 images SR 2x SR 3x CMP 2017 54
Video Super-resolution Input video Super-resolution CMP 2017 55
Embedded Systems CMP 2017 56
Acquired blurred image CMP 2017 57
Blur estimation CMP 2017 58
Patch-wise Deconvolution CMP 2017 59
Super-resolution JPEG SR CMP 2017 60
Embedded Super-Resolution Input image SR from 5 images SR from 10 images CMP 2017 61
Patch-wise restoration CMP 2017 62
CMP 2017 63
Regularization CMP 2017 64
Adjusting priors Start with an overestimated noise level and slowly decrease it to the correct level. Start with p<<1 and slowly increase it to p=1. CMP 2017 65