Robust Airlight Estimation for Haze Removal from a Single Image

Transcription

1 Robust Airlight Estimation for Haze Removal from a Single Image Matteo Pedone and Janne Heikkilä Machine Vision Group, University of Oulu, Finland matped,jth@ee.oulu.fi Abstract Present methods for haze removal from a single image require the estimation of two physical quantities which, according to the commonly used atmospheric scattering model, are transmission and airlight. The visual quality of images de-hazed with such methods is highly dependent on the accuracy of estimation of the aforementioned quantities. In this paper we propose a new method for reliable airlight color estimation that could be used in digital cameras to automatically de-haze images by removing unrealistic color artifacts. The main idea of our method is based on novel statistics gathered from natural images regarding frequently occurring airlight colors. The statistics are used to introduce a minimization cost functional which has a closed form solution, and is easy to compute. We compare our approach with current methods present in literature, and show its superior robustness with both images with artificially added haze, and real hazy photos. 1. Introduction When capturing images of real-world environments, there are often many causes which can contribute to an impairment of visibility. In this paper we are mainly interested in problems related to degradations caused by unfavorable atmospheric conditions. The presence in the air of aerosols and water droplets decreases the visibility range due to multiple scattering of light [7]. Depending on the type and concentration of the particles, one can observe mist, haze or fog. One of the visible consequences of this phenomenon is that far objects are less discernible, and lack contrast because they appear to fade gradually to an approximately homogeneous color, which is frequently called airlight in the literature. A detailed description and mathematical models of this process are presented in [8, 9]. De-hazing methods try to recover the original (spectral) radiance for each pixel of an image. This is an ill-posed problem, and authors have proposed different techniques to estimate the required unknown variables. The variables involved are the transmission, which is a function of the relative distances of the Figure 1. Example of haze removal with different estimated airlight colors. (top) original image; (center) de-hazing result with airlight color obtained with Fattal s method (three-dimensional search) ; (bottom) result obtained with airlight color estimated with our method. Resulting images were tone-mapped to enhance visibility. scene points from the observer, and the airlight; both contribute to the quality of the final result. Most of the present methods are mainly targeted to improve the quality of the estimated transmission, while often computing rough estimates of the airlight color. A wrong airlight color can cause 90

2 a de-hazed image to look like as if illuminated by an unrealistic light source (see Figure 1). The color artifacts present in the processed image give an unrealistic appearance to the objects in the scene that are far from the observer. Often such distortions cannot be considered tolerable in those applications where a device should produce a visually appealing de-hazed image to the end-user. Most methods start to manifest this drawback, as soon as the assumptions they make are violated. Such assumptions can be often too restrictive, depending on the application. In [8] Nayar et al. estimate the airlight color relying on the information conveyed by two images of the same scene, taken under different atmospheric conditions, which is not very practical. In [11] color constancy algorithms are used in order to circumvent the problem, however this idea implicitly assumes the airlight and the main illuminant coincide, which is a good approximation for scenes with a completely overcast sky, but is not true in general [7]. In [4] He et al. assume that an area with near zero transmission is present in the image, which is very restricting, as this assumption is frequently broken in aerial photos. In [2] Fattal makes the less restrictive assumption that there exist at least two small patches in the image whose pixels have approximately equal albedo, but contain areas where light is reflected differently; he then proposes two optimization schemes, one consists in finding a RGB triplet that minimizes the squared correlation between the de-hazed image and the transmission, and is essentially a three-dimensional search with gradient descent; the other one is a one-dimensional minimization and uses specific geometric constraints. However he does not discuss any reliable method to find valid patches, moreover the stability of its approach is sensitive to the goodness of the data contained in the patches. As a result, current methods satisfactorily recover the airlight color only with certain categories of images. The main contribution of this paper is the development of a method that works reliably within a broader range of images and with less strict assumptions on their content. We present a robust solution that only requires information retrieved from a single image, and introduce a cost function to be minimized which has a closed form solution, and is based on novel statistics from natural images. Finally, we show that our method outperforms the other approaches found in literature, with both real hazy images, and artificially degraded images. 2. Attenuation Model and Geometric Constraints In [8, 9] it has been proposed an applicable model for the attenuation of real pixel radiances due to multiple scattering in the atmosphere. Given a three component RGB vector R(x) representing the spectral radiance of a certain point x Figure 2. Attenuated radiances for two pixels with same albedo and same transmission. in the scene, another vector A representing the global environmental illumination (the airlight), the resulting attenuated radiance is given by: J(x) =tr(x)+(1 t)a (1) where J(x) is the observed color value, the parameter t [0, 1] is the transmission and is an exponential decay of the relative distance from x to the observer. The parameters R, t and A are not known and must be recovered. In [8, 2] it has been pointed out that (1) suggests some important geometrical constraints. In particular, given two points x and x with arbitrary corresponding transmissions t and t, such that R(x) = Bl 1 and R(x ) = Bl 2 for some real scalars{ l 1 } l 2 and vector B, one has that J(x), J(x ) Span B, Â. In this context B, l i,  are respectively the surface albedo, the amount of reflected light, and the airlight color, which is represented by the normalized airlight vector with l 2 norm. From these observations it immediately follows that different albedos B i form, together with the airlight-vector, two-dimensional subspaces S i R such that  S i. The subspaces are obviously planes (Figure 2), so considering only two pairs of linearly independent color vectors {J(x), J(x )}, {J(y), J(y )} with same albedos, and computing the respective unit-vectors ˆn x, ˆn y normal to each vector pair, one could theoretically retrieve the airlight color  as follows:  = ˆn x ˆn y (2) However there are practical problems that must be addressed and they are all discussed in the following sections. In the next section we describe one method to extract suitable image patches with same albedo, to be used for estimating the airlight color.. Extracting the Image Patches As it has been pointed out in the previous section, given two pair of pixels with the same albedo but different amount of reflected light, it is possible to obtain the airlight color. At this point, one must provide reliable criteria to identify constant albedo patches which manifest a non-constant amount 91

3 of reflected light (e.g. shadows or highlights). As illustrated in Figure 2, (1) dictates that one cannot directly deduce the albedo merely from the observed color values of J, because they are influenced by the airlight color. At this initial stage one is forced to provide an approximate value for Â; we use the RGB color vector w = 1 (1, 1, 1). This assumption allows one to easily find an invariant to attenuation which has a convenient form. In fact, given two orthogonal unit vectors î, ĵ such that î ĵ = Â, and considering p(x) =Projî ĵ (J(x)) the orthogonal projection of J(x) onto the plane î ĵ, it is easy to verify from (1) that ĵ p(x) =k R(x), î î + k R(x), ĵ () for some real scalar k. By normalizing p(x) one sees that hue angles of J do not depend on either t nor  = w, hence they are (approximately) invariant to attenuation. However, note that hue values are useful only when the pixels saturation is not close to zero. Since the luminance channel of the image can be directly used for finding shadowed and highlighted features, one would simply need to find patches with uniform hue direction but non-uniform luminance. In this context we define the hue direction as the complex quantity e i2θ, where θ is the hue angle. We randomly select a maximum of 2 patches Ω i with size 9x9 pixels which pass the following test: Mean Variance Ω hue2 - < 0.1 Ω sat > 0.01 < 0.01 Ω lum > 0.2 > < 0.9 < 0.01 Ω dark > 0.2 < These thresholds have been chosen manually and they did not turn out to be critical for the quality of the final results. Note that Ω hue2 is a complex quantity encoding the double hue angle as described above, and also recall that the dark channel of an image J(x) is given by [4] as: { } J dark (x) = min c {r,g,b} min {J c(z)} (4) z Ω(x) where the subscripts denote the respective color channel of the image, and that the dark channel can be directly used to estimate the transmission [4]. The last threshold needs further explanation. In fact, in natural images there are high chances that pixels with similar color represent points of the scene located at similar or equal depth. This assumption is often used by stereo matching algorithms [5]. When this is the case, the pixels inside a selected patch are associated with the same value of transmission, which is desirable. However, one needs to avoid to extract patches from areas Figure. Hue and saturation of the airlight color of 107 real images. The dashed line at represent the direction of the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the distribution. where the transmission is close to 0 or 1. The reason will become clear at the end of this section, and is related to the fact that computing the normal vector to the plane spanned by two vector that almost coincide, is unstable. Once the patches with same albedo are extracted, the pixel values of each of them are theoretically constrained to be coplanar, as discussed in Section 2, so an estimate of the associated normal vector n is given by: 2 arg min n, J(z) 2 + δ n,  est n z Ω(x) (5) such that n 2 =1 where Âest is an initial rough estimate of the airlight color and can be quickly obtained as described in [4] by selecting one pixel of highest intensity in the corresponding brightest area of the dark channel of the image, or alternatively by choosing the pixel with highest intensity in the image as in [8, 2]. The right-most term is used to penalize estimated normals which are not perpendicular enough to the initial airlight color estimate Âest. The quadratic functional in 5 can be easily minimized by standard methods based on singular value decomposition, for example. We will illustrate in Section 4.2 how the obtained values of ˆn for each patch can be used to compute the airlight color estimate according to (2). We always keep δ =. In the next section we show how the airlight color  can be estimated robustly, assuming N 2 patches are found. 4. Robust Airlight Color Estimation In this section, we discuss a novel strategy to obtain robust estimates of airlight colors. 92

4 11 7 Mean Angular Error Proposed Fattal() Fattal(1) He Gray World Angular Error Std. Dev β β Figure 4. Results for artificially added haze to the Middlebury database images. (left) Mean angular error for haze amounts corresponding to the scattering coefficient β; (right) Standard deviation of the angular errors collected Statistics for Airlight Colors We observed that extracting good patches and solving the minimization problems described in Section often yielded unrealistic results. This is due to the sensitivity of the method to the accuracy of the estimated normals in (5). We noticed such inaccuracies can result in an unrealistic estimated airlight color. We address this problem by enforcing the minimization with an additional constraint based on natural images. We collected 107 real photos from flickr.com portraying hazy scenes taken during both daylight and twilight, and manually extracted from each of them a 2x2 pixels patch of the sky; we then computed an average of the RGB values of all the pixels contained in it, and computed the resulting hue and saturation. Such values are potentially distorted by different white-balance settings in the cameras, on the other hand the standard illuminants theoretically suitable for those lighting conditions (e.g. D50, D55, D75,...) have chromaticity coordinates close to the srgb white-point [6], and approximately lie on the same hue line. A plot of the collected data is shown in Figure. We fitted a Gaussian distribution to the collected samples in the hue/saturation plane, obtaining a covariance matrix with the following eigenvalue/eigenvector pairs: (0.4982, ) and (0.8671, ). This gives support to the observation that the tonalities of the airlight colors in hazy conditions tend to scatter fairly closely around a hue angle of approximately , and that a high percentage of the samples falls into a narrow area with low saturation. Nonetheless, a further inspection of the plot in Figure suggests there is also a fair amount of highly saturated samples scattered far from the main direction; these occurrences are due to the rapid changes in hue and saturation manifested by the sky during a sunset [1] Closed Form Solution The statistics presented above suggest that, an inexpensive way of obtaining more robustness, is first to introduce a penalty function for estimates that are not reasonably close to the hue direction (Figure ), which in RGB space corresponds to a plane passing through the origin with normal ˆn sky =0.981r g b. On the other hand, it is reasonable to include an additional penalty for overly saturated estimates, though this makes sense mainly for daylight conditions. Since the airlight color is constrained to lie on the unit sphere of the RGB space, a convenient measure for saturation is the squared Euclidean distance between the estimate and the point w = 1 (1, 1, 1). We then propose the following objective function to be minimized: arg min  2 2 w c(n i ) Â, ni + λ Â, ˆnsky + γ  i such that  2 =1 (6) The parameters λ and γ control the multiple trade-off between closeness to the sky colors plane (represented by n sky ), closeness to the planes spanned by the pixel values in the N patches (the n i terms), and amount of saturation. Note that we also associate each estimated normal n with a certainty scalar given by: c(n) =exp ρ Ω(x) 1 n, J(z) 2 (7) z Ω(x) where ρ =10 4. The certainty scalars are useful to prevent the proposed thresholds to have a large impact in the final estimate. The objective function can be re-written in the 2 9

5 Figure 5. Example of failure with default parameters. (left) original image; (center) de-hazed with airlight color obtained with the default parameters; (right) result obtained by setting γ = form: min  T Q 2γw T   such that  T  =1, where Q = M T WM; the i-th row of M is the vector n T i, while the last one is nt sky, and W = diag(c(n 1 ),...,c(n N ),λ). This is a quadratic eigenvalue problem and it can be easily minimized by solving a sparse linear system as described by Gander et al. in []; first, the following polynomial eigenvalue problem P (Q, k)x =0 must be solved: (k 2 I 2k(QQ T )+(QQ T ) 2 4γ 2 ww T )x =0 (8) and once obtained the eigenvalue k, the airlight color is given by the solution of the linear system (QQ T ki)â =2γw (9) We use the default parameter values λ =, γ = Experiments We compared our proposed method with the approaches proposed by He et al. [4], the three-dimensional and the one-dimensional minimizations proposed by Fattal [2], and a color constancy method with gray-world assumption as used by Tan [11]. We tested our method with both artificially added haze, and real hazy photos. After airlight color estimation, the actual airlight radiance is obtained using the strategy discussed in [2], which consists in finding a radiance value that minimizes the squared correlation between the de-hazed image and the transmission in small uniform albedo patches; finally we remove the haze in all images using He s algorithm [4]. In the experiment with synthetic haze, we used 10 images from the Middlebury stereo images databases [10]; the images used in the experiment are identified in the Middlebury databases with the names Art, Books, Cones, Dolls, Laundry, Moebius, Midd1, Baby1, Monopoly, Reindeer, and were resized to pixels. The respective depthmaps provided in the database were used to produce hazy versions of the images according (1); the values for A were set with the airlight colors  gt we manually collected, as described in Section 4.1, while the transmission for a given pixel was set to t = e β(1 d) where d is the value of the depth-map, and β the scattering coefficient that controls the amount of visible haze. Some of the degraded images are shown in Figure 6. The accuracy of the recovered airlight color  was measured by the angular error cos Â, 1 Âgt. We performed 1070 comparisons, which were given by the 10 images degraded with all the 107 haze colors we collected. Results are reported in Figure 4. In the described experiment the proposed method outperforms the other ones in terms of angular error mean and standard deviation for most haze levels. For higher amount of haze (β >8) the simple gray world method yielded more accurate results. This is partly explained by the fact that when haze becomes very dense, the scene is expected to be dominated by the spectral radiance of the airlight. However all the images of the Middlebury database depict indoor scenes and most of them present a considerably large amount of pixels with neutral colors (mostly walls, papers, books, white cardboard etc.); we believe this fact could have possibly led to an overestimation of the performance of the gray-world algorithm, which we found often unsuitable for real hazy images. The results for real hazy photos are shown in Figure 7. Current methods in literature often yield an unrealistic color cast in the de-hazed areas which is not present with our proposed approach, see Figures 1 and 7. It is fairly evident in the left-most image of Figure 7, that a visible color cast is present in almost all the de-hazed results. Instead our method managed to recover an accurate airlight color and to produce an image which is free of unrealistic color artifacts. Also in the results for the second image of Figure 7 it is possible to observe color artifacts in the upper area, and we believe our methods yielded more neutral results. Moreover, although He s method was able to produce comparable results to our algorithm for the two right-most images, it failed in recovering a sufficiently accurate airlight color in the two left-most tests. As there is no general criterion to choose an appropriate method for a given arbitrary image, relying on a more stable method is often a preferred choice. The superior stability of our method is also confirmed by 94

6 the lower standard deviation of the angular error in the experiment with artificial haze. We noticed in few circumstances, when the haze color saturation was high, that the default values for the parameters in (6) did not yield good results, and we had to manually set the saturation penalty to γ = in order to obtain visually acceptable images (Figure 5). Nonetheless we found this situation occur mainly in photos taken during sunset, in which airlight colors are typically characterized by a relatively high level of saturation. All the experiments were run with a Matlab implementation of the proposed algorithm. The largest image size used in our experiments was pixel, and for images of this size, it took 1.44 seconds to get the airlight estimate on a 2.66Ghz processor with 1Gb of RAM. We also tried to downsample the large images to a size of pixels, achieving a computational speed of 0.65 seconds, with no loss of quality in the de-hazed image. Once obtained the airlight estimate, de-hazing a image with [4] took approximately 5.5 seconds; However 4.95 seconds of running time were due to the soft-matting algorithm to refine the estimated transmission map. Real-time soft-matting algorithms using the GPU are beginning to appear in the literature. 6. Conclusion In this paper we considered the problem of estimation of the airlight in haze removal algorithms. We first showed how wrong values for the airlight can cause an unrealistic color cast in a de-hazed image, and then discussed the limitations of present methods found in literature. Novel statistics for occurring airlight colors in hazy conditions were extracted from real images. Such statistics were then used to design a new robust solution for computing the color hue of the airlight. We performed experiments, with both artificially added haze, where the ground-truth haze colors were available, and then tested the proposed algorithm with real images. In both cases, we showed our method outperforms the other approaches present in literature, and produce satisfactory results also in difficult scenarios. While other methods might still be able to produce equally satisfactory results for some images, ours proved to be behave consistently with a greater number of different photos. References [1] E. Bruneton and F. Neyret. Precomputed atmospheric scattering. Proceedings of the 19th Eurographics Symposium on Rendering, [2] R. Fattal. Single image dehazing. ACM Transactions on Graphics. Proc. ACM SIGGRAPH, , 92, 94 [] W. Gander, G. H. Golub, and U. von Matt. A constrained eigenvalue problem. Linear Algebra and its Applications, :815 89, Figure 6. Some images from the Middlebury set degraded with artificial haze and scattering factor β ranging from 1 to 9. [4] K. He, J. Sun, and X. Tang. Single image haze removal using dark channel prior. IEEE Conference on Computer Vision and Pattern Recognition CVPR, , 92, 94, 95 [5] H. Hirschmller and D. Scharstein. Evaluation of cost functions for stereo matching. IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, [6] HunterLab. Equivalent white light sources, and CIE illuminants. an05_05.pdf. 9 [7] M. Minnaert. The Nature of Light and Color in the Open Air. Dover: New York, , 91 [8] S. G. Narasimhan and S. K. Nayar. Vision and the atmosphere. IJCV, 12(1):24 778, , 91, 92 [9] S. K. Nayar and S. G. Narasimhan. Vision in bad weather. The Proceedings of the Seventh IEEE International Conference on Computer Vision, pages , , 91 [10] D. Scharstein and R. Szeliski. High-accuracy stereo depth maps using structured light. IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, [11] R. Tan. Visibility in bad weather from a single image. proceeding of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR, , 94 95

7 Figure 7. Results with real images. (first row) original hazy image; (second row) de-hazed with airlight color obtained by a color constancy algorithm with gray-world assumption; (third row) with Fattal s three-dimensional minimization; (fourth row) with Fattal s onedimensional minimization; (fifth row) with He s method; (last row) with proposed method. 96