FAST SPATIALLY VARYING OBJECT MOTION BLUR ESTIMATION. Yi Zhang and Keigo Hirakawa

Transcription

1 FAST SPATIALLY VARYING OBJECT MOTION BLUR ESTIMATION Yi Zhang and Keigo Hirakawa Department of Electrical and Computer Engineering, University of Dayton, Dayton, OH {zhangy3, ABSTRACT Object motion results in spatially varying image blur. We propose an efficient method to recover a dense estimation of blur kernel. Proposed method takes advantage of the sparse representation of double discrete wavelet transform (DDWT) to simplify the wavelet analysis of blurry image. Our optimal solution includes separating the estimation of blur direction and length by investigating the crosscorrelation; and exploiting mean absolute summation (MAS) function for noise-robust estimation. We demonstrate by experiments the considerable improvement in speed and handling noise. Index Terms Spatially varying blur, double discrete wavelet transform, noise, fast approximation.. INTRODUCTION Object motion during exposure causes pixel sensor to record light stemming from multiple points of the object, causing the moving object to appear blurry in an image. Analysis of such phenomenon is attractive because object motion blur may carry temporal information about the scene that may be useful for application such as vehicle speed estimation and target tracking. Previously, we proposed the notion of double discrete wavelet transform (DDWT, ) that simplifies blur analysis by decoupling blur from the signal. Though DDWT successfully detected spatially varying motion blur and recovered sharp image, computational inefficiency and proneness to noise remain to be drawbacks in its dense blur kernel estimation. In this paper, we aim to address these shortcomings by a three-steps blur estimation method direction estimation, length estimation and spatial smoothing. In particular, direction estimation by DDWT cross-correlation and spatial smoothing by graph cut address the speed bottleneck; length estimation with mean absolute summation (MAS) improves robustness to noise. 2. BACKGROUND 2.. Related Work Recent advances in blind and non-blind image deblurring yielded promising results on images that are corrupted by global blur kernel 2, 3, 4, 5, 6, 7, 8, 9,,. Progress on spatially varying blur case has been slow because the reduced observation data support. In this scenario, it is common to simplify the problem by assuming parametric blur kernel, 2 or detecting blur without kernel estimation 3. Other solutions incorporate image segmentation/matting 4, 5, 6, 7, multiple input images 8, 9, user intervention 7, 2, 2, coded aperture 22, 23, and coded exposure 24 aimed at further constraining the set of feasible solutions. Specifically for estimating parametric dense blur kernel map of object motion, Chakrabarti 2 exploited the statistics of the gradient images and color image segmentation. Our previous work found the autocorrelation of DDWT coefficients has infimum that (a) Input (h) Estimation by proposed Fig.. Example of spatially varying blur image. corresponds to the pixel velocity of the object. Shi 3 estimated blur confidence map by evaluating three dimensional features that are extracted using image gradient, Fourier transform and local filtering. Notwithstanding the successes of the prior art, the performance inefficiency as well as sensitivities to noise are common drawbacks of existing approaches 3, 2,. Hence, addressing such problems is of significant practical value Double Discrete Wavelet Transform Over-complete DDWT is designed to sparsify sharp image and blur kernel simultaneously. Let x : Z 2 R be latent sharp image. Assuming Lambertian reflectance surface, the observation y : Z 2 R corrupted by noise and blur is assumed to be given by: y(n) = {x h n}(n) + ɛ(n), () where ɛ : Z 2 R is measurement noise, n Z 2 is the pixel location index and denotes convolution. The point spread function h n : Z 2 R denotes (possibly local) blur kernel acting at pixel location n. Where understood, the subscript n is omitted from h n(n). DDWT y v ij is defined as application of over-complete wavelet filter twice to the observed image: v ij (n) ={d i d j y}(n) (2) ={d i h} {d j x}(n) + {d i d j ɛ}(n) ={q i u j }(n) + η ij (n), (3) where d i (n) and d j (n) are ordinary discrete wavelet transform (DWT) filters in i and jth subbands, respectively; u j := d j x and q i := d i h are the overcomplete wavelet transforms of x and h, respectively; and η ij := d i d j ɛ is noise in DDWT domain. The relation between (2) and (3) is illustrated in Figure 2. The pipeline in Figure 2(a) and 2(b) are equivalent 2(a) is the direct result of applying DDWT on the observed image, but Figure 2(b) is the interpretation we give to (2). Key is to realize that v ij is the result of convolving sparse signal u j with sparse filter q i, plus noise η ij. Assume for the moment that we have horizontal motion (with contant velocity during exposure). The blur kernel is simplified to a

2 (a) DDWT analysis in (2) (b) DDWT analysis in (3) Fig. 2. The two processing pipelines above are equivalent. Though (a) is the direct result of applying DDWT (di? dj ) to the observed blurry image y, (b) is the interpretation we give to the DDWT coefficients. parametric form: n h(n) = k(n) step n + Define: 2 k(n) step n 2 k(n) o, (4) where step( ) denotes the step function and k(n) is the pixel velocity at location n. When di denotes a Haar wavelet transform, the DWT response q i of blur kernel h in (4) and the corresponding DDWT coefficients v ij in (3) respectively are n o q i (n) = k(n) δ n + 2 k(n) δ n 2 k(n), n o uj n + 2 k(n) uj n 2 k(n) v ij (n) = k(n) + η ij (n). vθij (n) :={di ( )? dj (Mθ )? y(mθ )}(n) ={di ( )? h(mθ )}? {dj (Mθ )? x(mθ )}(n) + {di ( )? dj (Mθ )? (Mθ )}(n) ={qθi? ujθ }(n) + ηθij (n), where ujθ (n) := {dj (Mθ )? x(mθ )}(n) = uj (Mθ n) qθi (n) := {di ( )? h(mθ )}(n) ηθij (n) := {di ( )? dj (Mθ )? (Mθ )}(n). (5) Hence v ij is a noisy observation of the difference of two DWT coefficients uj displaced by k pixels. Figure 3 shows comparison between DWT (Figure 3(a)) and DDWT (Figure 3(b)) coefficients of Figure (a). Based on the intuition that DWT captures edge and texture information and by (5), we expect double edges to appear in DDWT at the moving objects, where the distance between double edges correspond to the velocity in pixels. Indeed, the blur detection is simplified to detecting the displacement of double edges and textures. (7) (8) Owing to the fact that h(mθ n) is a horizontal blur, we may simplify qθi (n) and vθij (n) as: n o qθi (n) = kk(mθ n)k δ n + 2 kθ (n) δ n 2 kθ (n) vθij (n) = j j uθ (n+ k (n)) uθ (n k (n)) 2 θ 2 θ kk(mθ n)k + ηθij (n), (9) where kθ (n) := Mθ k(mθ n) ( r cos θt = r sin θt if θ π/4. otherwise () Unshearing vθij (n) results in the following v ij (n, θ) : = vθ (Mθ n) = k(n) uj n k(n) uj (n+ 2 ) ( 2 ) kk(n)k + η ij (n, θ), () (a) DWT coeffs w (b) DDWT coeffs v Fig. 3. Example of DWT and DDWT coefficients of blurry image. The motion blur manifests itself as a double edge in DDWT, where the distance between the double edges (yellow arrows in (b)) correspond to the speed of the object Arbitrary Motion Direction If the direction of object motion is known, one may use image shearing to skew the image to obtain horizontal direction DDWT vector. Let k(n) = rsin θ, cos θt R2 denote the motion blur parameterized by direction θ π/2, π/2) and length r R+. Let Mθ R2 2 denote a shearing matrix of the form: # " tan θ, if θ π/4 Mθ = ". (6) #, otherwise cot θ ηθij (n). ij where η (n, θ) is the unsheared version of For simplicity, superscripts i and j are omitted in the rest of the paper. 3. PROPOSED WORK In, we exhaustively searched over blur direction and length using autocorrelation that is sensitive to noise, and carry out local smoothness by incorporating bilateral filter. This procedure was very time consuming, and we propose below new ways to improve its speed and robustness to noise. In this paper, we address these problems by fast direction estimation first hence reducing the number of searching directions; utilizing noise-robust blur detector and carrying out spatial smoothing by efficient approach. 3.. Fast Blur Direction Estimation Define DWT w : Z2 R of blurred image y w(n) := {d? y}(n) = {h? u}(n) + ζ(n), (2) where ζ(n) := {d? }(n) is the DWT noise. Consider the following:

3 (a) R wv (b) Radon(R wv) (c) p(θ) Fig. 4. Example of cross-correlation and its application. The brightest pixel in (b) corresponds to the maximum value. The direction candidates set can be reduced by choosing peaks in the histogram of ˆΘ(n). (see (c)) (a) Φ(n, l) with blur (b) Φ(n, l) without blur Fig. 5. Example of Φ(n, l) with blur and without blur. definition. The cross-correlation function R wv : Z 2 R of w and v is defined by the relation: R wv(l) := E w(n l) v(n), (3) where l Z 2 denotes displacement between w and v. Recalling (3) and (2), we expand (3) as follows: R wv(l) = E {h u + ζ}(n l) {q u + η}(n) = E m h(m)u(n l m) m q(m )u(n m ) + {ζ(n l)η(n)} = h(m)q(m )R u(l + m m ) + R ζη (l), m m (4) where R u is the autocorrelation of u and R ζη is the cross-correlation function of ζ and η. With a reasonable assumption that DWT coefficients u is sufficiently de-correlated (i.e. R u(l) δ(l)), (4) reduces to R wv(l) m h(m)q(l + m) + R ζη (l). (5) That is, cross-correlation is simply a summation of few blur kernel h with noise when q is sufficiently sparse. Even if the approximation is not perfect, it is good enough to detect the motion direction via the Radon transform: ˆθ = arg max Radon(Rwv). (6) Θ Indeed, Figure 4(b) shows Radon(R wv) of a small blur patch in Figure 6(k). The maximum occurs at Θ = 37 degrees (Figure 4(b)) that coincides with true blur direction θ. By generalization, one can obtain spatially varying blur direction by localizing cross-correlation by follows: R wv(n, l) := E n Λ(n) w(n l) v(n ), ˆΘ(n) = arg max Radon(R wv(n)). (7) Θ(n) where Λ(n) denotes a local neighborhood centered at n. If finite objects are moving within a scene, we expect that motion direction angles ˆΘ(n) are also finite. As such, we may further reduce variabilities by clustering ˆΘ(n) to a set of motion direction candidates Ω = {θ,..., θ n}. See Figure 4(c) for example Blur Length Estimation The approach in is sensitive to noise because of the correlation of true signal is attenuated by k 2. Such influence grows quadratically as the blur kernel length increases. We propose to replace this with a solution that has linear influence (with respect to k ). Recalling (), we evaluate mean absolute summation (MAS) function: ( Φ(n, l) = E v n + l ) ( 2, Θ + v n l ) 2, Θ { ( =E k(n) u n + (k(n) + l)) u ( n (k(n) l)) +u ( n + (k(n) l)) u ( n (k(n) + l)) +η(n + l, Θ) + η(n l, Θ)}, Θ Ω, n Λ(n), (8) where l = γsin Θ, cos Θ T denotes the candidate blur length. Clearly, MAS function attains minimum when l = ±k(n): ˆk(n) = arg min Φ(n, l) (9) l Figure 5 shows an example of Φ(n, l) for moving along θ and nonmoving location. Minimum occurs at γ = (Figure 5(a)) coinciding with actual motion k(n). Without motion, the minimum is found at a small number γ = L (Figure 5(b)) that L corresponds to the wavelet window size. It is justifiable that feature detector cannot be used to detect features that is smaller than itself. In practice, we set ˆk(n) to zero when the minimum occurs at L Regularizing Blur Estimation Map The blur kernel estimation in 3.2 can be improved by post-processing. We smooth the estimation map by graph cut 25. Such optimization process searches the solution that minimizes following energy: E(k) = n E (k(n)) + λ n,n E 2(k(n), k(n )), (2) where E = δ(k(n) ˆk(n)) is a binary cost that enforces output similar to (9). And E 2 = δ(k(n) k(n )) exp ( n 2 /σ 2) encourages discontinuities to align with image edges 23, 22. Where n 2 denotes image gradient energy at location n and σ is set to twice of the median of n over the entire image.

4 (a) input (b) 3, 67s (c) 2 ( ), 3s (d) ( ), 2532s (e) proposed (3 ), 8s (f) input (g) 3, 2s (h) 2 (9 ), 79s (i) ( ), 83s (j) proposed (2 ), s (k) input (l) 3, 43s (m) 2 ( ), 42s (n) (35 ), 2872s (o) proposed (37 ), 8s (p) input (q) 3, 4s (r) 2 (9 ), 4s (s) ( ), 92s (t) proposed (3 ), 3s Fig. 6. Examples of spatially varying blur image. Estimated noise standard deviations in (a), (f), (k), (p) are 45.6,.24, 8.92, 26.62, respectively. (Recall Nikon D9 has a 2-bit sensor) 4. EXPERIMENT We evaluate proposed algorithm using raw image data captured by Nikon D9. Each image is corrupted by object motion blur and noise. Experiments are implemented in MATLAB 24(a) and executed on PC using Intel(R) Core(TM) i7-26 CPU at 3.4GHz with 6GB memory. We compared our results to state-of-art blur estimation/detection algorithms of, 8, 3. As a rough indication, noise standard deviation estimated by 26 is reported in Figure 6 and execution time is marked under each result (from 2nd to 5th column). Estimated blur directions are also annotated for 2, and proposed method. Rather than densely estimating blur kernel, 3 tries to yield a blur confidence map where brighter pixel corresponds to region with more confidence of blur. Such method is able to extract blur object but may mis-classifies at smooth background (Figure 6(g)) from a low-noise image. It fails with more image noise (Figure 6(b, l)) or multiple moving objects (Figure 6(q)). The weakness of 2 is that the estimated blur directions are not stable (Figure 6(h, m, r)), sensitive to noise (Figure 6(m)) and not capable of handling multiple moving objects (Figure 6(r)). The main drawbacks of are performance inefficiency and sensitive to noise (Figure 6(d, n)). In comparison, proposed work yields comparable estimation when noise is low (Figure 6(j)). It outperforms other competitors with the presence of moderate noise (Figure 6(e, o)) or multiple moving objects (Figure 6(t)). Furthermore, it only takes 8 seconds to analyze a 5 5 pixels color image. 5. CONCLUSION We proposed a fast yet effective spatially varying blur estimation based on improvements to DDWT blur analysis. Such method densely estimates object motion blur kernel by direction estimation, length estimation, spatial smoothing.

5 6. REFERENCES Yi Zhang and Keigo Hirakawa, Blur processing using double discrete wavelet transform, in Computer Vision and Pattern Recognition (CVPR), 23 IEEE Conference on. IEEE, 23, pp L. Lucy, Bayesian-based iterative method of image restoration, Journal of Astronomy, vol. 79, no , A. Levin, Y. Weiss, F. Durand, and W.T. Freeman, Understanding and evaluating blind deconvolution algorithms, in Computer Vision and Pattern Recognition, 29. CVPR 29. IEEE Conference on. IEEE, 29, pp R. Fergus, B. Singh, A. Hertzmann, S. Roweis, and W. Freeman, Removing camera shake from a single photograph, ACM SIGGRAPH, J. Cai, H. Ji, and Z. Shen, Blind motion deblurring from a single image using sparse approximation, CVPR, vol. 62, pp , January M. Ben-Ezra and S.K. Nayar, Motion deblurring using hybrid imaging, in Computer Vision and Pattern Recognition, 23. Proceedings. 23 IEEE Computer Society Conference on. IEEE, 23, vol., pp. I SK Nayar and M. Ben-Ezra, Motion-based motion deblurring, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 26, no. 6, pp , Q. Shan, J. Jia, and A. Agarwala, High-quality motion deblurring from a single image, ACM Transactions on Graphics (SIGGRAPH), T.S. Cho, S. Paris, B.K.P. Horn, and W.T. Freeman, Blur kernel estimation using the radon transform, in Computer Vision and Pattern Recognition (CVPR), 2 IEEE Conference on. IEEE, 2, pp Sunghyun Cho and Seungyong Lee, Fast motion deblurring, in ACM Transactions on Graphics (TOG). ACM, 29, vol. 28, p. 45. Li Xu, Shicheng Zheng, and Jiaya Jia, Unnatural l sparse representation for natural image deblurring,. 2 A. Chakrabarti, Todd Zickler, and William T. Freeman, Analyzing spatially-varying blur, CVPR, 2. 3 Jianping Shi, Li Xu, and Jiaya Jia, Discriminative blur detection features, S. Dai and Y. Wu, Motion from blur, in Computer Vision and Pattern Recognition, 28. CVPR 28. IEEE Conference on. IEEE, 28, pp A. Levin, Blind motion deblurring using image statistics, Advances in Neural Information Processing Systems, vol. 9, pp. 84, S. Chan and T. Nguyen, Single image spatially-variant outof-focus blur removal, ICIP, 2. 7 Chandramouli Paramanand and Ambasamudram N Rajagopalan, Non-uniform motion deblurring for bilayer scenes, in Computer Vision and Pattern Recognition (CVPR), 23 IEEE Conference on. IEEE, 23, pp L. Bar, B. Berkels, M. Rumpf, and G. Sapiro, A variational framework for simultaneous motion estimation and restoration of motion-blurred video, ICCV, J. Chen, L. Yuan, C.K. Tang, and L. Quan, Robust dual motion deblurring, in Computer Vision and Pattern Recognition (CVPR), 28 IEEE Conference on. IEEE, 28, pp A. Levin, P. Sand, T.S. Cho, F. Durand, and W.T. Freeman, Motion-invariant photography, in ACM Transactions on Graphics (TOG). ACM, 28, vol. 27, p T.S. Cho, A. Levin, F. Durand, and W.T. Freeman, Motion blur removal with orthogonal parabolic exposures, in Computational Photography (ICCP), 2 IEEE International Conference on. IEEE, 2, pp Anat Levin, Rob Fergus, Frédo Durand, and William T Freeman, Image and depth from a conventional camera with a coded aperture, in ACM Transactions on Graphics (TOG). ACM, 27, vol. 26, p Ayan Chakrabarti and Todd Zickler, Depth and deblurring from a spectrally-varying depth-of-field, in Computer Vision ECCV 22, pp Springer, Ramesh Raskar, Amit Agrawal, and Jack Tumblin, Coded exposure photography: motion deblurring using fluttered shutter, in ACM Transactions on Graphics (TOG). ACM, 26, vol. 25, pp Yuri Boykov and Vladimir Kolmogorov, An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 26, no. 9, pp , D.L. Donoho and I.M. Johnstone, Adapting to unknown smoothness via wavelet shrinkage, Journal of the american statistical association, vol. 9, no. 432, pp , 995.