Handling Outliers in Non-Blind Image Deconvolution

Transcription

1 Handing Outiers in Non-Bind Image Deconvoution Sunghyun Cho 1 Jue Wang 2 Seungyong Lee 1,2 sodomau@postech.ac.kr juewang@adobe.com eesy@postech.ac.kr 1 POSTECH 2 Adobe Systems Abstract Non-bind deconvoution is a key component in image deburring systems. Previous deconvoution methods assume a inear bur mode where the burred image is generated by a inear convoution of the atent image and the bur kerne. This assumption often does not hod in practice due to various types of outiers in the imaging process. Without proper outier handing, previous methods may generate resuts with severe ringing artifacts even when the kerne is estimated accuratey. In this paper we anayze a few common types of outiers that cause previous methods to fai, such as pixe saturation and non-gaussian noise. We propose a nove bur mode that expicity takes these outiers into account, and buid a robust non-bind deconvoution method upon it, which can effectivey reduce the visua artifacts caused by outiers. The effectiveness of our method is demonstrated by experimenta resuts on both synthetic and rea-word exampes. 1. Introduction Singe image deburring has gained considerabe attention in recent years. Given a burry input image b, existing deburring approaches mode the image degradation as: b k + n, (1) where k is the bur kerne, refers to the underying sharp atent image, n is noise, and is the convoution operator. The task for bind image deconvoution is to infer both k and from a singe input b, which is severey i-posed. Bind deconvoution approaches intensivey use non-bind deconvoution, the process of estimating the atent image given the burred image b and the estimated bur kerne k, whie inferring k and for generating the fina output [7, 5, 16, 12, 4, 10]. This makes non-bind deconvoution a key component in the deburring pipeine. Various non-bind deconvoution approaches have been proposed in the iterature, ranging from cassic Wiener fiter to modern optimization approaches with image priors. However, in practice these methods often produce severe ringing artifacts even when the bur kerne is known or we estimated (Fig. 1). We argue that this is mainy because the inear bur mode in Eq. (1) does not consider non-inear outiers that often exist in rea imaging process. For instance, one common outier is saturated pixes. When we shoot a ow ighting scene with a ong exposure time, a few bright spots in the scene wi be saturated (Fig. 1a). The intensities of these pixes can no onger be modeed using Eq. (1) since a non-inear cipping function has been appied at the corresponding sensor ocations. Other types of outiers incude dead pixes of sensors, hot pixes, non-inear in-camera processing, and so on. These outiers are common in rea imaging systems, but are ignored in previous deconvoution methods. We deveop an agorithm that expicity handes outiers in the deconvoution process. We first anayze how various types of outiers vioate the inear bur assumption and consequenty cause severe ringing artifacts to the resut image. We then propose a new deconvoution method which contains an expicit component for outier modeing. In our approach, we cassify image pixes into two main categories: inier pixes which satisfy the inear bur mode and can be we-recovered using traditiona deconvoution, and outier pixes which cannot be expained by the inear mode. We empoy an Expectation-Maximization (EM) method to iterativey refine the outier cassification and the atent image. To evauate the effectiveness of the proposed method, we compare our approach with the state-of-the-art deconvoution methods on both synthetic and rea-word exampes. The resuts show that by expicity detecting and handing outiers in the deconvoution process, severe ringing artifacts which are common to previous methods can be effectivey reduced in our approach. 2. Reated Work Non-bind deconvoution has been extensivey studied in image processing and computer vision fieds. Cassica methods incude Wiener fitering, Kaman fitering, constrained east squares fitering and the Richardson-Lucy agorithm [14, 13]. We refer the readers to the comprehensive survey [1] for more detais of these methods. 1

2 In order to suppress ringing artifacts and restore image features effectivey, various image priors and reguarization schemes have been proposed for deconvoution, such as tota variation (TV) reguarization [15], sparse image prior [11], and natura image statistics [16]. In previous work, these image priors are usuay combined with an L 2 -norm based data fideity term, which is derived from a Gaussian noise mode. This combination eads to high quaity resuts when the observed image contains ony a sma amount of noise that can be we approximated by a Gaussian distribution. To hande impusive noise such as sat-and-pepper noise, Bar et a. [2] proposed a data fideity term based on an L 1 -norm, which can be derived from the assumption of a Lapacian distribution for noise. However, since these priors and data fideity terms are derived from specific noise modes, they cannot effectivey suppress artifacts caused by other types of noise and outiers. In contrast, our method restores image detais and suppresses ringing artifacts better than these approaches, as we wi demonstrate ater. For handing saturated pixes in non-bind deconvoution, Harmeing et a. [9] proposed a method that detects saturated pixes by threshoding input burry images, and masks out them from the deconvoution process. Whie the method recovers saturated regions better than previous deconvoution methods, using a singe threshod to detect saturated pixes in the input image is erroneous, as there is no guidance on how to find the optima threshod vaue. In contrast, our method does not invove a threshod, and is more reiabe and accurate as demonstrated ater. To suppress ringing artifacts, Yuan et a. [18] proposed a coarse-to-fine progressive deconvoution approach, where biatera reguarization is iterativey appied at each scae for restoring sharp edges whie avoiding ringing. This method thus tries to impicity hande outiers by directy suppressing artifacts. We show that ringing artifacts can be more efficienty removed by expicity modeing the outiers in the deconvoution process. To the best of our knowedge, we are the first to systematicay mode outiers for non-bind deconvoution. 3. Outier Anaysis In this section we anayze how various types of outiers vioate the inear bur mode and cause artifacts in previous approaches. A deconvoution resuts in this section were generated using the sparse prior based method proposed by Levin et a. [11]. Saturated/Cipped pixes As camera sensors have imited dynamic ranges, pixes receiving more photons than the maximum capacity wi be saturated and the corresponding pixe intensities wi be cipped. This is a common scenario when shooting a night image with ong exposure, where the majority of the scene is dark but there are some bright ights. The non-inear cipping vioates the inear bur mode, eading to ringing artifacts around these spots. To demonstrate this, we syntheticay burred an HDR image using a known kerne, and then cipped pixe vaues that are arger than a threshod, as shown in Fig. 1a. Directy appying deconvoution produces severe ringing artifacts shown in Fig. 1b. Note that in this case the bur kerne is accurate and no noise is added. For comparison, we appied the same deconvoution method without cipping the input pixe vaues, which yieds a high quaity resut in Fig. 1c. It ceary suggests that saturated pixes can be a main source of ringing artifacts. Besides saturation, parts of an image, such as shadow areas, coud be underexposed and intensities in those regions coud be cipped to back. Such cipping aso breaks the inear bur mode, and may cause ringing artifacts too. Non-Gaussian noise An image can be degraded by noise from various sources, and they are typicay non- Gaussian [2]. In addition, the camera sensor may contain defective pixes causing bright or dark spots in an image [6]. For ow quaity inputs, there may be compicated compression artifacts. Previous methods often assume a specific type of noise, thus is not robust enough to hande a cases. Fig. 1d shows a synthetic exampe where uniform noise was added to the input image. Severe artifacts can be seen in the deconvoution resut (Fig. 1e), as the deconvoution method assumes a Gaussian noise mode. Noninear camera response curve Digita cameras a have buit-in image processing units, and some of the processing steps are highy noninear. For instance, a camera usuay appies a non-inear response curve to map the scene radiance to pixe intensity. This wi vioate the inear bur mode, especiay around high contrast edges where pixes on the two sides of an edge are separated far away on the response curve. Fig. 1f shows a synthetic exampe of appying a non-inear response curve to a burred raw image, and Fig. 1g shows the deconvoution resut, where artifacts are obvious around strong edges. Fig. 1h shows the deconvoution resut without appying the response curve, which is artifact-free. 4. Deconvoution with Outier Handing Our goa is to deveop a robust deconvoution agorithm which can perform reiaby we when outiers present. Among different types of outiers, noninear in-camera processing can be avoided by using raw camera output, or reduced by first appying an inverse response curve obtained from camera caibration [8] to the input image. However, other outiers are extremey hard to remove using image processing techniques. Our key observation is that athough these outiers vary in nature, their impacts to the deconvoution process are common: First, they cause the inear bur mode to fai. Second, they cause the noise in Eq. (1) to be non-gaussian. We thus propose a deconvoution agorithm

3 (a) (b) (c) (d) (e) (f) (g) (h) Figure 1. Iustrations of deconvoution outiers. (a) Input burred HDR image with high intensity vaue cipping. (b) Deconvoution resut of (a). (c) Deconvoution resut of (a) without appying intensity cipping. (d) Input burred image with added uniform noise. (e) Deconvoution resut of (d). (f) Input burred image with non-inear response curve appied. (g) Deconvoution resut of (f). (h) Deconvoution resut of (f) without appying the response curve. that specificay handes these two vioations. For simpicity we wi derive our agorithm under the assumption that the bur kerne is spatiay invariant. In practice we can easiy extend it to hande non-uniform bur kernes. The key idea of our approach is to cassify observed pixe intensities into two categories: iniers whose formation satisfies Eq. (1); and outiers whose formation does not, which incude cipped pixes and those from other sources. For cassification, we introduce a binary map m such that m x 1 if the observed intensity b x is an inier, and m x 0 otherwise, where the subscript x is the pixe index. To find the most probabe atent image given the burred image b with outiers and the bur kerne k, we excude the outiers from the deconvoution process using the inier map m. Since we do not know the true vaue of m, we propose an EM method which aternatingy computes the expectation of m and performs deconvoution using the expectation. To estabish a more accurate bur mode incuding outiers, we assume that a noise-free burred image is captured by sensors and then the captured intensities are cipped into the dynamic range of a camera. Noise and outiers are added to the cipped burred image within the dynamic range. This mode can be represented as: b c(k ) + n, (2) where c is a cipping function. When u is within the dynamic range, c(u) u; otherwise, c(u) returns the maximum or minimum intensity of the dynamic range. We assume that the additive noise n is spatiay independent, and it foows a Gaussian distribution ony for iniers. For outiers, we assume that the observed intensity at an outier pixe is competey independent of k and, and may have an arbitrary vaue in the dynamic range. Reca that outiers can come from mutipe sources that are hard to mode accuratey, as we discussed in Sec. 3. Therefore we assume a uniform distribution for outiers, without superimposing any strong priors. In the foowing, we formuate our objective function, and derive the detaied process for deconvoution using this bur and noise mode. Mathematicay, finding the most probabe atent image can be formuated as a maximum a posteriori (MAP) estimation probem such that: MAP argmax p( k, b). (3) According to the Bayes theorem: MAP argmax p(b k, )p() argmax p(b, m k, )p() argmax m M m M p(b m, k, )p(m k, )p() (4) where M is the space of a possibe configurations of m. We define the atent image prior p() as p() exp( λφ())/z p, (5) where Z p is a normaization constant. Using the sparse prior [11], we set φ() { x ( h ) x α + ( v ) x α}, where h and v are differentia operators aong the x and y directions, respectivey. We use α 0.8 in our system. Since we assumed that noise is spatiay independent, the ikeihood p(b m, k, ) x p(b x m, k, ). Then, based on our noise mode, we define p(b x m, k, ) as: { N (bx f p(b x m, k, ) x, σ) if m x 1 (6) C if m x 0 where f k, N is a Gaussian distribution, and σ is the standard deviation. C is a constant defined as the inverse of the width of the dynamic range in the input image. For the cassification prior p(m k, ), we assume that m is aso spatiay independent, i.e., p(m k, ) x p(m x k, ) x p(m x f x ). We then define p(m x f x ) based on the vaue of f x as: { Pin if f p(m x 1 f x ) x DR (7) 0 otherwise where DR is the dynamic range, and P in [0, 1] is the probabiity that b x is an inier. For exampe, by setting P in 0.9, we assume that 90% of non-cipped observed pixes b x are iniers. According to our bur mode in Eq. (2), when f x is out of DR, the observed intensity b x cannot be an inier. It is either a cipped vaue or an outier of another type, thus m x shoud be aways 0 in this case. Note that we do not imit the dynamic range of x in Eq. (2), so f x can go beyond DR. In our impementation, we use the normaized range, DR [0, 1].

4 Burred input & Bur kerne Ground truth & Our resut Burred input & Ground truth 1st iteration 2nd iteration 3rd iteration 15th iteration Figure 2. Deconvoution exampe of our method with intermediate resuts. The input image is syntheticay burred and high intensity vaues are cipped. Uniform noise is aso added. The right four coumns show magnified patches of intermediate estimates of and the weights wm computed for the red channe at EM iterations. Due to saturated pixes and noise, the estimate at the first iteration shows severe artifacts, but as iteration goes, artifacts are removed, and detais are recovered. Soving Eq. (4) is chaenging, since the arge number of possibe configurations for m makes marginaizing p(b, m k, ) intractabe. We thus adopt an EM method [3] to sove it. Instead of marginaizing ikeihood p(b, m k, ) with respect to m, our EM method tries to find an optima, which maximizes the compete-data posterior p( b, m, k) p(b, m k, )p(). As we do not know the true vaue of m, we cannot use the compete-data ikeihood p(b, m k, ) when maximizing p( b, m, k). Our EM method thus evauates the expectation of p(b, m k, ) and uses it for finding the optima. The expectation of p(b, m k, ) and the estimate of are updated by performing the E and M steps aternatingy. Specificay, the E step finds the posterior distribution p(m b, k, ) of m using the current estimate o of, and then evauates the expectation Q(, o ) of the compete-data og ikeihood p(b, m k, ) under p(m b, k, ). Note that the evauated Q(, o ) is not a vaue, but a functiona with respect to whose parameters have been determined using p(m b, k, ). The subsequent M step revises the estimate of by maximizing the compete-data og posterior, which is defined as Q(, o ) + og p(). In the foowing, we wi derive the detais of the two steps. E step Using the current estimate o of, the expectation Q(, o ) is defined by: Q(, o ) E[og p(b, m k, )] (8) E[og p(b m, k, ) + og p(m k, )], (9) where E is the expectation under p(m b, k, o ). We put Eqs. (6) and (7) into Eq. (9), assuming if fxo is in or out of DR, so is fx, respectivey. Then, up to a constant, we have: " # X Q(, o ) E mx og N (bx fx, σ) (10) x X E[mx ] x 2σ 2 bx fx 2 (11) where E[mx ] is given by: E[mx ] p(mx 1 b, k, o ). (12) By the Bayes theorem, the posterior p(mx b, k, o ) is p(mx b, k, o ) p(bx mx, k, o )p(mx k, o ), (13) p(bx k, o ) where p(bx k, o ) 1 X p(bx mx, k, o )p(mx k, o ).(14) mx 0 Again, by putting Eqs. (6) and (7) into Eq. (13), we have: ( N (bx fxo, σ)pin if fxo DR o N (b E[mx ] (15) x fx, σ)pin + CPout 0 otherwise where f o k o and Pout 1 Pin. As E[mx ] competey determines Q(, o ) as a functiona of, E[mx ] is the ony vaue that is actuay computed in the E step. In Eq. (6), p(bx mx 0, k, o ) aways foows a uniform distribution. If fxo is out of DR, we coud expect bx to be cose to one bound of DR, which can be better represented by a nonuniform distribution. However, in this case, p(mx 1 k, o ) becomes zero and p(bx mx 0, k, o ) has no effect on E[mx ]. Therefore, we use a uniform distribution in Eq. (6) for a cases of outier pixes, incuding cipped pixes. M step The M step finds the revised estimate n such that n argmax{q(, o ) + og p()}, (16) which is equivaent to minimizing: X wxm bx (k )x 2 + λφ(), x (17)

5 where wx m E[m x ]/2σ 2. To minimize Eq. (17), we use the iterativey reweighted east squares (IRLS) method used in previous deconvoution approaches [11]. First, we introduce pixewise weights w h and w v such that wx h ( h ) α 2 and wx v ( v ) x α 2. Then, Eq. (17) can be approximated by: wx m b x (k ) x 2 + λψ(), (18) where x ψ() x { w h x ( h ) x 2 + w v x ( v ) x 2}. (19) Agorithm 1 EM Deconvoution for Outier Handing procedure DECONVOLUTION(b, k) Let w m x, w h x & w v x 1 for a x Set o by minimizing Eq. (18) for iter 1, N iters do E step updates w m, w h and w v using o M step updates n by minimizing Eq. (18) o n end for return o end procedure For fixed w h and w v, Eq. (18) becomes a quadratic function with respect to, which can be effectivey minimized using the conjugate gradient (CG) method. We thus can minimize Eq. (17) by aternating between updating (w h, w v ) and minimizing Eq. (18). The above EM procedure is easy to understand if we consider the meaning and roe of E[m x ]. If the observed intensity b x is ikey to be an inier, E[m x ] computed in the E step is cose to one, and vise versa. These vaues are then used as pixe weights in the actua deconvoution process, which is performed in the M step. As a resut, ony iniers with arge weights are used for deconvoution in the M step, whie outiers with ow weights are excuded. In Sec. 5, we provide more anaysis on this outier handing scheme. Agorithm Agorithm 1 shows the overa process for our EM-based deconvoution agorithm. For the initia estimate o, we use the resut of the deconvoution using a Gaussian prior, by setting a weights to one. Then, we perform the EM iterations. In the E step, we update weights w m, w h and w v given the currenty estimated. In the M step, we minimize Eq. (18) using the updated weights. We typicay used N iters 15 and σ 5/255, and adjusted λ according to the amount of noise in the input image. We consistenty used P in 0.9 for generating a resuts in the paper. To minimize Eq. (18), we run the CG method with 25 iterations. Fig. 2 shows an exampe of appying our method to a night city image, and its intermediate resuts for the atent image with the associated weights w m. 5. Anaysis on Outier Handing In this section we provide more insightfu anaysis on how our method can recover outier pixes. We aso demonstrate why we need to mode cipped pixes expicity in the cassification prior p(m k, ) Recovering missing information In a burred image b, due to the scattering nature of bur, a pixe x contains partia information of the origina intensities of neighboring pixes. If x is an inier, (a) (b) (c) (d) (e) (f) Figure 3. Recovering outier pixes. (a) ground truth image. (b, c, d) burred images with outiers of different sizes and intensities. (e) sparse deconvoution resut of (b). (f, g, h) our deconvoution resuts of (b, c, d), respectivey. (a) (b) (c) (d) (e) (g) (h) (a) (b) (c) Figure 4. Recovering high intensity pixes. (a) ground truth image with pixes of different high intensities. From eft to right, the origina intensities of the white dots are 5, 10, 20, 40, and 80. (b) burred image of (a), where high intensities are cipped. (c) sparse deconvoution resut of (b). (d) our deconvoution resut with ony cipped pixe handing. (e) our resut with both cipped pixe handing and outier handing. Right: pixe intensities of each image aong a scan ine crossing the eft three cipped pixes. wx m E[m x ]/2σ 2 is non-zero. Then, the observed intensity b x at x contributes to recovering pixe vaues around x by minimizing Eq. (18), whie the origina intensity x at x (d) (e)

6 is recovered from its own and neighboring pixe information. If x is an outier, x can sti be reconstructed using the information in the neighboring pixes around x, athough b x does not contribute to recovering pixe vaues because wx m is cose to zero. As a resut, in our method outiers are not just smoothed out, their origina intensities may be recovered actuay if the neighboring pixes maintain enough amount of information about them. To verify this observation, we conducted an experiment shown in Fig. 3. We syntheticay burred the origina image with a known kerne and added outier pixes of different intensities and sizes. The deconvoution resuts show that, athough the burred image is contaminated by outiers, underying textures can be recovered we for sma outiers. For arge outiers, neighboring pixes have ess information about the true intensities of the outier pixes. These pixes cannot be accuratey recovered in our method, and can ony be smoothed out. Cipped pixes can aso be recovered in the same way. In Fig. 4, we syntheticay added high intensity pixes with different intensity vaues into the origina image. We then burred the image with a known kerne, and cipped high intensity pixes. To study the effect of cipped pixe handing separatey from non-gaussian noise handing, we turn off the non-gaussian noise handing by setting P in 1. The deconvoution resut in Fig. 4d shows that added pixes can be better recovered when their origina intensities are ower. As the origina intensity goes higher, more neighboring pixes around the added pixe are saturated as we. After intensity cipping, the scattered portion of the origina high intensity in the neighboring pixes is aso ost, which makes recovering the origina intensity to be impossibe. Ringing artifacts are introduced in this case. Can the deconvoution agorithm generate reasonabe resuts even if origina high intensity pixes cannot be recovered from their neighbors? The answer is yes. Since the pixes around the high intensity pixes aso vioate the inear bur mode, they wi be treated as outiers. By deaing with them propery in the EM agorithm, we can reduce the ringing artifacts, even though we cannot recover the true intensities. Fig. 4e shows that our fu deconvoution method can successfuy recover a good atent image from the input image Effects of modeing cipped pixes Since cipped pixes aso break the inear bur mode, one may wonder if we can drop the dynamic range check in Eq. (7), and et the cipped pixes be handed together with other outiers. This eads to another EM procedure which is simiar to the one derived in Sec. 4, where the M step remains the same, and in the E step, E[m x ] is computed as E[m x ] N (b x f o x, σ)p in N (b x f o x, σ)p in + CP out. (20) Deconvoution resut without cipped pixe handing (a) (b) Error at each EM iteration (a) (b) (a) (b) Figure 5. Deconvoution resut of the burred image in Fig. 2 obtained without modeing cipped pixes. Bottom row: magnified patches of (a) the deconvoution resut without modeing cipped pixes and (b) the resut in Fig. 2. Upper right: pots of root mean squared errors (RMSE) from the ground truth at EM iterations for the methods without and with modeing cipped pixes. In Eq. (15), non-zero E[m x ] is obtained ony when f o x DR. In contrast, in Eq. (20), E[m x ] is aways non-zero regardess of f o x, even though it can be sma for a cipped pixe x. The difference ooks sma, but it has a significant impact on the performance of the agorithm. By competey excuding the cipped pixes from the optimization process, our method can avoid being distracted by the corrupted information from them, and converge to a better soution quicky. Fig. 5 shows an exampe where we appy Eq. (20) in the E step. Due to the proposed outier handing mechanism, the deconvoution resut does not have significant artifacts overa, except some moderate ringing artifacts around saturated regions (Fig. 5a). However, using Eq. (15) in the E step sti eads to a better resut (Fig. 5b). This is consistent with the pots of error vaues in Fig. 5, which shows that the agorithm converges to a more accurate resut just in a few iterations using Eq. (15). This eads to the concusion that for images with saturated pixes, which is a common scenario in night photographing, expicity modeing cipped pixes can hep the proposed agorithm achieve better resuts more efficienty. 6. Resuts We impemented our method in Matab. Our testing environment is a PC running MS Windows 7 64bit version with Inte Core i7 CPU and 12GB RAM, but we did not use the muti-core faciity. The average computation time is about six minutes for images of one mega-pixes. Our

7 Figure 6. Deconvoution resuts of a syntheticay burred image. From eft to right: burred input with uniform noise and saturated pixes, sparse deconvoution, L 1 -norm based deconvoution, our method, and the ground truth image. Figure 8. Comparison between Yuan et a. s method [18] and ours. From eft to right: Syntheticay burred inputs with uniform noise and saturated pixes, Yuan et a. s method, and our method. Our resuts show ess ringing artifacts than Yuan et a. s resuts. Figure 7. Deconvoution resuts of (b) and (c) of Fig. 3 using L 1 -norm based deconvoution. project webpage 1 provides high-resoution versions of the images shown in this section and additiona exampes, as we as Matab source code of our method. Fig. 6 shows a synthetic exampe. The input image was syntheticay burred and high intensity pixes were cipped. Uniform noise was aso added. We deconvoved the burred image using a sparse prior [11], L 1 -norm based deconvoution [2], and our method. The L 1 -norm based deconvoution method uses an L 1 -norm data fideity term, which is known to be robust to outiers, with a sparse image prior. For comparison, we impemented L 1 -norm based deconvoution using an IRLS method. In Fig. 6 we can see the resut of sparse deconvoution contains artifacts caused by both uniform noise and cipped pixes. The resut of L 1 -norm based deconvoution shows no artifacts from uniform noise, but sti has artifacts around saturated pixes. In contrast, our resut contains no artifacts for both uniform noise and saturated pixes. We aso measured peak-signa-to-noise ratios (PSNR) of the resuts. From the burred input to our resut in Fig. 6 (eft to right), PSNR vaues are 17.15, 16.46, 18.56, and Our resut achieves a much higher PSNR than the others. Whie we added outiers with the size of one pixe to the burred image in Fig. 6, outiers can be arger in practice. In Fig. 7, we appied L 1 -norm based deconvoution to the burred images in Fig. 3, to compare its performance on outier handing with our method. The resuts show that our method suppresses arge outiers better than L 1 -norm based deconvoution. This is because our bur mode better describes outiers than a Lapacian prior, which is the basis of the L 1 -norm data fideity term. Fig. 8 shows a comparison between the deconvoution method of Yuan et a. [18] and ours. As our mode expicity handes outiers, our resuts have no visibe artifacts around 1 outiers Figure 9. Comparison between Harmeing et a. s method [9] and ours. From eft to right: rea burred input with saturated pixes, Harmeing et a. s method, and our method. Saturated pixes are better recovered in our resut. saturated pixes and other outiers. In contrast, the resuts of Yuan et a. s method present noticeabe artifacts, and ess detais are recovered in their resuts. Fig. 9 shows a comparison between the deconvoution method of Harmeing et a. [9] and ours. Harmeing et a. s method detects saturated pixes by threshoding input burry images, which can be erroneous as there is no guidance on the optima threshod vaue. In contrast, our method detects saturation on the atent image without invoving a threshod, and better recovers saturated pixes. Fig. 10 shows deconvoution resuts of rea burred images. To obtain bur kernes for rea images, we used the bind deconvoution method of Cho and Lee [4]. Since outiers may degrade the quaity of kerne estimation, we cropped rectanguar regions without obvious outiers (e.g., saturated pixes) from input images, and estimated bur kernes using them. We then appied deconvoution methods to the input images using the estimated kernes. The resuts show that our method handes outiers most effectivey. In summary, our experimenta resuts show that our deconvoution method eads to significanty ess artifacts, such as ringing around saturated pixes or high-frequency noise, than previous approaches. 7. Concusion and Future Work In this paper, we anayzed various types of outiers, which cause severe ringing artifacts in previous deconvoution methods. Based on the outier anaysis, we proposed a robust, EM-based non-bind deconvoution method, which expicity detects outiers and propery handes them in the

8 Figure 10. Deconvoution resuts of rea burred images. From eft to right: burred inputs, sparse deconvoution, L 1 -norm based deconvoution, Yuan et a. s method [18], and our method. Our resuts show ess ringing artifacts than the others. deconvoution process. We further provided detaied anaysis on outier pixe recovery and cipped pixe handing of our approach. We evauated our method with both synthetic and rea-word bur images, and compared the resuts with other state-of-the-art methods. Previous studies have pointed out that rea burs are often non-uniform and spatiay varying [12]. Due to the imited space, our method was derived under the assumption that the bur is spatiay invariant. However, our method can be easiy extended to hande non-uniform burs. This can be achieved by repacing the convoution operation in our formuation with a non-uniform burring operation, e.g., the burring operation using a projective motion chain [17]. Despite the recent progress, bind deconvoution remains as a chaenging probem, especiay when the burred image contains outiers. Our approach handes outiers ony in the non-bind deconvoution step, not in the kerne estimation step. However, we have discovered that outiers have a significant impact on kerne estimation too. As future work we pan to incorporate outier modeing to the kerne estimation step, in order to deveop a bind deconvoution method that is robust enough against various outiers. Acknowedgements We thank Lu Yuan for generating the deconvoution resuts of his method. We aso thank Fickr user OiMax (Figs. 2 & 6) and Pusan Nationa University (the bottom row of Fig. 10) for sharing their images. This work was supported by the Brain Korea 21 Project, the Industria Strategic Technoogy Deveopment Program of MKE/MCST/KEIT (KI001820, Deveopment of Computationa Photography Technoogies for Image and Video Contents), and the Basic Science Research Program of MEST/NRF ( ). References [1] M. R. Banham and A. K. Katsaggeos. Digita image restoration. Signa Processing Magazine, IEEE, 14(2):24 41, Mar [2] L. Bar, N. Sochen, and N. Kiryati. Image deburring in the presence of impusive noise. Internationa Journa of Computer Vision, 70(3): , December , 7 [3] C. M. Bishop. Pattern Recognition and Machine Learning. Springer, [4] S. Cho and S. Lee. Fast motion deburring. ACM Trans. Graphics, 28(5):145:1 145:8, , 7 [5] S. Cho, Y. Matsushita, and S. Lee. Removing non-uniform motion bur from images. In Proc. ICCV 2007, pages 1 8, [6] M. A. Covington. Digita SLR Astrophotography. Cambridge University Press, [7] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman. Removing camera shake from a singe photograph. ACM Trans. Graphics, 25(3): , [8] M. Grossberg and S. Nayar. Modeing the Space of Camera Response Functions. IEEE Trans. Pattern Anaysis Machine Inteigence, 26(10): , Oct [9] S. Harmeing, S. Sra, M. Hirsch, and B. Schökopf. Mutiframe bind deconvoution, super-resoution, and saturation correction via incrementa EM. In Proc. ICIP 2010, pages , , 7 [10] N. Joshi, S. B. Kang, C. L. Zitnick, and R. Szeiski. Image deburring using inertia measurement sensors. ACM Trans. Graphics, 29(3):30:1 30:9, [11] A. Levin, R. Fergus, F. Durand, and W. T. Freeman. Image and depth from a conventiona camera with a coded aperture. ACM Trans. Graphics, 26(3):70:1 70:9, , 3, 5, 7 [12] A. Levin, Y. Weiss, F. Durand, and W. Freeman. Understanding and evauating bind deconvoution agorithms. In Proc. CVPR 2009, pages 1 8, , 8 [13] L. Lucy. An iterative technique for the rectification of observed distributions. Astronomica Journa, 79(6): , [14] W. Richardson. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am., 62(1), [15] L. I. Rudin, S. Osher, and E. Fatemi. Noninear tota variation based noise remova agorithms. Physica. D, 60: , Nov [16] Q. Shan, J. Jia, and A. Agarwaa. High-quaity motion deburring from a singe image. ACM Trans. Graphics, 27(3):73:1 73:10, , 2 [17] Y.-W. Tai, P. Tan, and M. S. Brown. Richardson-ucy deburring for scenes under projective motion path. IEEE Trans. Pattern Anaysis Machine Inteigence, Accepted. 8 [18] L. Yuan, J. Sun, L. Quan, and H.-Y. Shum. Progressive interscae and intra-scae non-bind image deconvoution. ACM Trans. Graphics, 27(3):74:1 74:10, , 7, 8