NOISE is inevitable in any image-capturing process. Even

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 7, JULY Differene-Based Image Noise Modeling Using Skellam Distribution Youngbae Hwang, Member, IEEE, Jun-Sik Kim, Member, IEEE, and In So Kweon, Member, IEEE Abstrat By the laws of quantum physis, pixel intensity does not have a true value, but should be a random variable. Contrary to the onventional assumptions, the distribution of intensity may not be an additive Gaussian. We propose to diretly model the intensity differene and show its validity by an experimental omparison to the onventional additive model. As a model of the intensity differene, we present a Skellam distribution derived from the Poisson photon noise model. This modeling indues a linear relationship between intensity and Skellam parameters, while onventional variane omputation methods do not yield any signifiant relationship between these parameters under natural illumination. The intensity-skellam line is invariant to sene, illumination, and even most of amera parameters. We also propose pratial methods to obtain the line using a olor pattern and an arbitrary image under natural illumination. Beause the Skellam parameters that an be obtained from this linearity determine a noise distribution for eah intensity value, we an statistially determine whether any intensity differene is aused by an underlying signal differene or by noise. We demonstrate the effetiveness of this new noise model by applying it to pratial appliations of bakground subtration and edge detetion. Index Terms Differene-based noise modeling, Skellam distribution, edge detetion, bakground subtration. Ç 1 INTRODUCTION NOISE is inevitable in any image-apturing proess. Even though the statistial nature of quantum physis an primarily aount for the soure of the noise, most existing noise modeling methods in the omputer vision ommunity do not use its physial harateristis, but simply try to model noise in a probabilisti framework. 1.1 Motivation In various omputer vision tasks, suh as edge detetion, orner detetion, and bakground subtration, it is important to determine whether or not the intensities of two pixels ome from the same sene radiane. Intensities that ome from the same radiane may vary for a number of reasons, and this variation is alled image noise. Conventionally, pixel intensity has been simply modeled as a sum of true intensity and additive noise. For example, the previous approahes [1], [2], [3] use the additive noise model to represent the observed intensity, I, ontaminated by image noise as I ¼ I 0 þnð; 2 Þ;. Y. Hwang is with the Multimedia IP Center, Korea Eletronis Tehnology Institute (KETI), #68 Yatap-dong, Bundang-gu, Seongnam-si, Gyeonggido , Korea. ybhwang@keti.re.kr.. J.-S. Kim is with the Robotis Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA kimjs@s.mu.edu.. I.S. Kweon is with the Department of Eletrial Engineering, KAIST, #3215 Eletrial Engineering Building (E3), Guseong-dong, Yuseong-gu, Daejeon, Korea. iskweon@ee.kaist.a.kr. Manusript reeived 6 May 2010; revised 14 Jan. 2011; aepted 10 Ot. 2011; published online 30 Nov Reommended for aeptane by S. Belongie. For information on obtaining reprints of this artile, please send to: tpami@omputer.org, and referene IEEECS Log Number TPAMI Digital Objet Identifier no /TPAMI ð1þ where I 0 is the true intensity and Nð; 2 Þ is the distribution of noise with the mean, and variane 2. For the ase of a Gaussian noise model, the mean is set to 0 and only the variane is used to estimate the noise distribution. For the ase of a Poisson model, the noise N follows a zero mean Poisson noise assuming that the other soures are muh smaller than shot noise [4]. Similarly to the Gaussian noise model, the mean is 0 and the variane is used to estimate intensity variation by image noise. Fig. 1 shows the intensity distributions and their parameterized estimation using the onventional additive noise models. We aptured a pattern image, as shown in Fig. 1a, and estimated the noise distribution by alulating the mean and variane of pixel intensities within a single path. Fig. 1b shows the fitting results of a olor hannel for a path using the Gaussian distribution and Poisson distribution. The measured intensity distributions are not even symmetri, and the estimation urves using the additive noise model do not fit to the real intensity distributions. When we use parameters from the erroneous modeling, the further proessing based on them may fail to obtain reliable outputs. The harateristi of the additive noise model that is vulnerable to illumination effets auses this erroneous ase. Beause pixel intensity is governed by quantum physis, it annot have a true value, but an only be onsidered to be a random variable. Bak to the original problem of determining whether or not two pixel intensity values orrespond to the same sene radiane, it is required to ompare two random variables of pixel intensity. Among the diverse approahes for omparing two random variables, we hoose to diretly model the distribution of intensity differene as a single random variable. The distribution of the intensity differenes shown in Fig. 1 is more symmetri and easy to fit to a parameterized /12/$31.00 ß 2012 IEEE Published by the IEEE Computer Soiety

2 1330 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 7, JULY 2012 Fig. 1. Histograms of intensity and intensity differene for a path (15th path). Conventional additive models do not explain the distribution well, while intensity differene modeling an explain the distribution better. model. This example shows that the differene of intensity values follows a parameterized distribution better than the distributions of the intensity itself, whih is usually assumed to follow an additive Gaussian or a Poisson distribution. The modeling of intensity differene makes it possible to statistially determine whether the differene is aused by an image noise or by a sene radiane hange. In addition, we demonstrate that a noise model based on intensity differene an eliminate the effet of illumination, ontrary to the onventional additive noise model. 1.2 Related Work There have been studies that have endeavored to estimate image noise by analyzing the amera imaging proess. Healey and Kondepudy [4] pointed out that there are five major noise soures, inluding shot noise, fixed pattern noise, and dark urrent noise. They desribed a alibration proedure that orrets for spatial nonuniformity aused by fixed pattern noise and by variation in dark urrent noise. They further insisted that the variane of the noise is linearly proportional to the intensity statistially. Their method requires hundreds of images for estimating sensor noise. Alter et al. [1] developed a noise model based on Poisson noise and quantization noise in low-light onditions. In their noise model, the mean and variane exhibit a relationship of exponentially deaying osillation under low-light onditions whih approahes linearity as intensity inreases. They also derived an intensity similarity measure based on their noise model and showed improved mathing auray. Liu et al. [2], [3] reently proposed another noise estimation method from a single image. They defined a noise level funtion as a variation of the standard deviation of noise with respet to image intensity. After learning the noise level funtion, whih desribes how the noise level hanges with brightness, they used Bayesian MAP estimation to infer the noise level funtion from a single image. They applied their noise model to bilateral filtering and Canny edge detetion [3]. Beause of their assumption that an image is pieewisely smooth, their method provides only the upper bounds of noise levels rather than the expeted distribution of the noise. Hene, their noise estimation is inappropriate for measuring the intensity differene between two pixels. For Canny edge detetion, their method only determines the higher threshold with Gaussian smoothing in a fixed sale. Therefore, their edge detetor annot take into aount the per-pixel effet of image noise, whih depends on brightness. Foi et al. [5] reently presented a signal-dependent noise model whih is omposed of a Poissonian part for photon-ounting proess and a Gaussian part for the remaining stationary disturbanes. They not only showed the derivation statistially, but also explained the relationship to elementary aspets of the digital sensor s hardware. In the main algorithm of noise modeling, they exploited the usual Gaussian approximation of the Poisson distribution, and thus their noise modeling an be regarded as an intensity-dependent Gaussian model. The authors mainly foused on estimating standard deviations aording to intensity under the pieewise smoothness assumption as well. They did not apply the estimated noise model to further appliations. 1.3 Contributions We present three ontributions in terms of noise modeling and its appliations. First, we show that modeling of intensity differene is more aurate and useful than the onventional additive noise model using variane, as we briefly introdued before. In Setion 2, a more detailed omparison between the two models will be provided, and it will show that the intensity differene modeling offers a more signifiant linear relation between the intensity and noise parameters than the onventional methods [1], [2], [3]. The proposed differene modeling is able to handle the illumination effets under natural lights without any loal transformation based on olor segmentation [2], [3]. Seond, we propose to use a parameterized distribution model Skellam distribution as an intensity differene model. In Setion 3, we derive it from the Poisson distribution of photons olleted at a single pixel onsidering shot and dark urrent noise. Noise modeling based on a Skellam distribution has been used before in the ontext of PET imaging [6], but to the best of our knowledge it has not been applied to natural images. We also make omparisons between the exat Skellam modeling and its Gaussian approximation for intensity differenes. The relation between the Skellam parameter and intensity is investigated in Setion 4 in various setups of a amera, suh as a CCD/CMOS sensor, gain, and exposure time, and we propose a simple method to obtain the parameter in a single natural image.

3 HWANG ET AL.: DIFFERENCE-BASED IMAGE NOISE MODELING USING SKELLAM DISTRIBUTION 1331 Fig. 2. Relationship between noise parameters and intensity in the temporal and spatial domains under natural illumination onditions. The statisti of intensity differene has a stronger relationship with intensity than the intensity variane, whih is onventionally used. Finally, we show that the proposed intensity differene model is useful for pratial appliations in Setions 5 and 6 in both temporal and spatial domains. Bakground subtration and edge detetion are transformed into the original problem of determining whether or not two pixel intensity values orrespond to the same sene radiane, and the diret modeling of an intensity differene an handle it statistially. The preliminary version of this paper appeared in [7]. In this version, we fous on the importane of the modeling of intensity differene whihever the underlying distribution is hosen by providing experimental results. In addition, we make a omparison between the proposed Skellam and Gaussian approximation, verify the proposed noise model for various CCD/CMOS ameras, and investigate the effet of amera settings, suh as gain and exposure time. A new appliation in the temporal domain, bakground subtration, is provided with quantitative evaluation. 2 VALIDITY OF MODELING INTENSITY DIFFERENCES In Fig. 1, we show that the intensity distributions are not explained well by the onventional additive noise model, while the intensity differene looks easier to model in a parameterized form. In this setion, we make a omparison between the onventional model and a parameterized model whih we will propose later. The additive noise model has been used to estimate the relationship between intensity and noise parameters in previous works [2], [3], [4], [5]. These works start from alulating varianes of pixels from the same sene radiane. We first look into the meaning of the variane in the onventional noise modeling. If one assumes that image noise e is additive and follows a Poisson distribution, intensity I is modeled as I ¼ I 0 þ e ; where e I e ; I e f P ði e j Þ ¼ ðieþ e ; ði e Þ! where I 0 is the true intensity and I e follows a Poisson distribution with variane used to model the noise. The Poisson distribution is shifted by so that image noise e has zero mean. For a homogeneous path P in whih the sample mean of the pixel intensities is I, we an derive the parameter by the variane of the intensities as Pði;jÞ2P ði Iði; jþþ2 ¼ : ð3þ n 1 The onventional noise variane estimation used in the previous methods [1], [2], [3], [5] atually estimates the parameter of an additive Poisson noise model. Note that if the variane is large enough, the Poisson noise model an be approximated by a Gaussian distribution. For the homogeneous pathes in the ColorCheker image shown in Fig. 2a aptured by a Point Grey Flea amera, we estimated the noise parameter of the Poisson distribution using (3), whih is atually the variane of intensities in pathes. As shown in Fig. 2b, it is diffiult to find the relationship between variane and intensity. On the other hand, the Skellam parameter ð1þ from the intensity differene, whih we will explain in the later setions, has a strong linear relationship with intensity, as shown in Fig. 2. The modeling of intensity differene shows a more signifiant relationship to intensity than onventional variane omputation, or equivalently, the additive Poisson modeling does. This is learer when we see the statistis in the temporal domain. We aptured 2,000 images under natural illumination onditions, as shown in Fig. 2a. As time went by, the intensity level dereased, as shown in Fig. 2d. In this ase, ð2þ

4 1332 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 7, JULY 2012 the intensity varianes, onventionally used for noise modeling, are greatly sattered in the temporal domain, as shown in Fig. 2e, and it is hard to see any parametri relation between intensity and its variane. As shown in Fig. 2f, the relationship between the intensity and the parameter from the intensity differene is almost linear and the relationship in both the temporal and the spatial domains is preserved even under the variation in illumination onditions. These results show that the modeling of the intensity differene provides a more valid and onsistent relationship than the onventional intensity variane between intensity and the noise parameters in both temporal and spatial domains. 3 INTENSITY DIFFERENCE MODEL Beause the pixel intensity inluding image noise should be onsidered as a random variable, two random variables should be ompared to determine whether or not two pixel intensity values orrespond to the same sene radiane. In order to ompare them, we model intensity differene diretly as a single random variable. 3.1 Skellam Distribution of Intensity Differenes Throughout this paper, we assume a linear response amera whose intensity is linearly proportional to the number of photons olleted at an imaging element. The number of photons reeived at a olletion site (usually a partiular pixel) is governed by the laws of quantum physis; thus it annot have a true value, but should be desribed as a random variable. This variation is alled photon noise. The probability distribution for n e photons at a site during an observation of time interval T seonds is known to have a Poisson distribution [1], [4], [8]: f P ðn e j ; TÞ ¼ ðtþn e e T ; ð4þ n e! where is the rate parameter, measured in photons per seond. The mean and the variane 2 of the distribution over an interval T in (4) are given by ¼ 2 ¼ T: It is obvious that the number of photons in a brighter pixel is greater than that in a darker pixel, and the expeted number of photons will inrease as brightness (or intensity) inreases. We will show this in Setion 4.1. Three traditional assumptions about the relationship between signal and noise generally do not hold for dominant photon-dependent noise [8]: Image noise is 1) not independent of the signal, 2) nor Gaussian, 3) nor additive. Beause the dominant photon-dependent noise is not additive, and the noise and true intensity of a pixel annot be separated, we diretly use the distribution of an intensity differene, rather than making an erroneous assumption of additive noise. Here, we generalize our previous model [7] to onsider all photon-dependent noise soures, inluding dark urrent noise. Dark urrent noise inreases the number of photons independently of an input signal. Beause the number of photons n t generated by dark urrent noise also follows a Poisson distribution [8], we an write this distribution as ð5þ f P ðn t j t Þ¼ ð tþ nt e t ; ð6þ n t! where t is the average number of photons resulting from dark urrent noise. Note that the sum of two Poisson random variables also follows a Poisson distribution whose parameter beomes the sum of the parameters of the two Poisson random variables. Finally, the distribution of noise with respet to both photon noise and dark urrent noise an be expressed as f P ðn e j p þ t Þ¼ ð p þ t Þ ne e ð pþ t Þ ; ð7þ n e! where p represents the number of photons produed by photon noise (or shot noise). The differene between the two Poisson random variables follows a Skellam distribution [9]. The probability mass funtion (pmf) of a Skellam distribution is a funtion of k ¼ n 1 n 2, the differene between two Poisson random variables n 1 and n 2. The differene of the two Poisson random variables in (7), whih is the distribution of the whole photon-dependent noise, an be written as ð1þ f S k; pd ;ð2þ pd ¼ e ðð1þ pd þð2þ pd Þ ð1þ! k=2 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð8þ pd I ð2þ jkj 2 ð1þ pd ð2þ pd ; pd where ð1þ pd ð1þ p þ ð1þ t, ð2þ pd ð2þ p þ ð2þ t, and I k ðzþ denotes the modified Bessel funtion of the first kind. In the speial ase that ð1þ ¼ ð2þ, a Gaussian distribution approximates a Skellam distribution for suffiiently large and k [10]. However, a Gaussian distribution annot approximate the Skellam distribution for low-intensity pixels beause neither nor k is suffiiently large. If we an estimate ð1þ pd and ð2þ pd from a given set of intensity differenes k, our noise model an onsider the noise soures together in this single formulation. In the remainder of this paper, the subsript pd is omitted for simpliity. 3.2 Skellam Parameter Estimation Under the assumption of a linear response amera, we an estimate the Skellam parameters from the statistis of intensity differenes. The mean S and variane 2 S of a Skellam distribution are given by S ¼ ð1þ ð2þ ; 2 S ¼ ð1þ þ ð2þ : From (9), we alulate the parameters ð1þ and ð2þ diretly as ð1þ ¼ S þ 2 S 2 ð9þ ; ð2þ ¼ S þ 2 S : ð10þ 2 The statistis S and 2 S are obtained as follows from eah pixel in an image sequene of a stati sene: P t S ¼ ði tði; jþ I tþ1 ði; jþþ ; ð11þ n P 2 S ¼ t ð S ði t ði; jþ I tþ1 ði; jþþ 2 ; ð12þ n 1

5 HWANG ET AL.: DIFFERENCE-BASED IMAGE NOISE MODELING USING SKELLAM DISTRIBUTION 1333 Fig. 3. Skellam distribution estimated with 10,000 stati images. The intensity differene follows a Skellam distribution well. (Larger images with more experimental results are available in the supplemental material, whih an be found in the Computer Soiety Digital Library at where I t ði; jþ denotes the intensity at the ði; jþ position at frame t, and n denotes the total number of images. To investigate the Skellam parameters of various olors, we aptured 10,000 stati images of a GretagMabeth ColorCheker using a Point Grey Sorpion amera (image resolution ¼ 1;600 1;200 pixels, exposure time ¼ 1=15 s). Throughout this paper, we use raw amera data that are not demosaiked as we are dealing with the statistis of the raw intensity that is linearly proportional to the number of photons. Fig. 3 shows that the intensity differene follows the Skellam distribution losely, whih onfirms that our assumption of the dominant photon-dependent noise is appropriate. Parameters of the Skellam distribution differ aording to a pixel olor. As we expeted, a blak pixel has smaller Skellam parameters than a gray pixel, beause ð1þ and ð2þ are the numbers of photons in the CCD ells during the apturing period. As shown in Fig. 3, we applied our Skellam modeling sheme to a large number of stati images. This modeling is possible using a single image of a olor pattern by assuming that all pixels in the spatial domain are mutually independent. In other words, the noise distribution in the spatial domain is the same as that in the temporal domain. To verify this assumption, we ompared noise parameters estimated in the spatial domain with those estimated in the temporal domain. We obtained the results in the temporal domain using (11) and (12). To estimate the noise in the spatial domain, we first seleted image pathes of the homogeneous olor regions in the pattern image shown in Fig. 1a. Using these image pathes, we applied our modeling strategy to the spatial domain using the following equations: P ði;jþ2p S ¼ ði tði; jþ I t ði þ d x ;jþ d y ÞÞ ; ð13þ n P 2 S ¼ ði;jþ2p ð S ði t ði; jþ I t ði þ d x ;jþ d y ÞÞ 2 ; ð14þ n 1 where ði; jþ 2P denotes all the points in the path, d x and d y represent disparities in the horizontal and vertial diretions, respetively, and n denotes the total number of pixels in the path. In this paper, d x and d y are set to one. Fig. 4 shows the omparison between the modeling results in the spatial domain and those in the temporal domain. For this experiment, we used a HITACHI HV-F22 amera (3CCD, image resolution ¼ 1; pixels, exposure time ¼ 1=7:5 s). The results shown in Fig. 4 verify that Skellam parameters estimated in the temporal and those Fig. 4. Comparison of the Skellam parameters estimated in the spatial and in the temporal domains. Sine the parameters from both domains are almost the same, intensity differene is an ergodi proess. estimated in the spatial domains are almost same. Thus, differene of intensity is an ergodi proess; noise parameters estimated in temporal domain an be used in spatial analysis and vie versa. There are slight differenes between the results in the temporal and spatial domains beause we hose only one pixel in a path in estimating the distribution in the temporal domain. Depending on whih pixel is seleted in a path, its Skellam parameters vary slightly. 3.3 Skellam versus Gaussian The distributions shown in Fig. 3 look similar to the zeromean Gaussian distribution, and in this setion we evaluate the two distributions, i.e., the proposed Skellam and the Gaussian quantitatively. To determine whih distribution fits better to the atual data, we use a 2 test for goodnessof-fit (GOF), whih is used to determine if a random sample is governed by a speifi distribution [11]. In the 2 test for goodness-of-fit, we ompute a measure for the given observed frequenies N i and the expeted frequenies e i as W ¼ Xr i¼1 ðn i e i Þ 2 e i ; ð15þ where r is the number of samples. Sine the measure W follows a 2 distribution, the ritial region C with a level of signifiane in the hypothesis test is alulated by 2 statistis as C ¼fw : w 2 ;r 1 g: ð16þ Therefore, if W< 2 ;r 1, the hypothesis that the expeted distribution by the assumed model follows the atual observed distribution is aepted. We set ¼ 0:05, the same as in [11]. We measured the goodness-of-fit for images aptured by a Canon EOS-1D Mark III amera with low sensitivity ISO100. We aptured a bright image and a dark image of the same sene as shown in Figs. 5a and 5. For eah path, we estimate the distribution of the intensity differene by the Skellam and the Gaussian distributions. Figs. 5b and 5d show the test statistis for GOF from the modeling with the ritial values 2 ;r 1, where r is the number of intensity levels in the histogram of eah path. Most of the modeling results using the Skellam distribution are less than the ritial values as shown in Figs. 5b and 5d. It means that the hypothesis that noise distributions for olor pathes follow the Skellam distribution is aeptable. However, some

6 1334 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 7, JULY 2012 Fig. 5. Comparison between the Skellam and Gaussian distributions by measuring goodness-of-fit in R hannel of bright and dark images. The Skellam distribution explains an intensity differene better than the Gaussian distribution. (More results for other hannels are available in the supplemental material, available online.) modeling results by the Gaussian distribution exeed the ritial values, and this shows that the intensity differene distribution is fitted better by the Skellam distribution than by the Gaussian distribution. Fig. 6 shows the omparison between Skellam and Gaussian distributions in two pathes in Fig. 5a for intensity differene. As the GOF indiates in Fig. 5, a Skellam distribution explains the measured intensity differene better. Though a Skellam distribution fits the measured intensity differenes better than the Gaussian distribution, as we have shown above, it is possible to approximate the distribution as a Gaussian to utilize its advantages, suh as its quadrati form, ontinuity, and differentiability. Note that, however, this does not mean that the intensity follows the additive Gaussian distribution, as we have shown in Fig. 1. Still, the diret modeling of the intensity differenes is more aurate than the additive modeling of intensity, whihever parameterized distribution is hosen in explaining the intensity differenes. 4 NOISE STATISTICS ESTIMATION USING SKELLAM PARAMETERS We have shown that Skellam distribution an explain intensity differene well, both in temporal and spatial domains. In this setion, we will show that there is a useful relationship between pixel intensity and Skellam parameters, and we will make use of this relationship to easily estimate noise parameters in a single natural image. 4.1 Linearity between Intensity and Skellam Parameters Under the assumption of a linear response amera, a linear relationship exists between pixel intensity and Skellam parameters ð1þ, estimated from intensity differenes. Fig. 7 shows the satter plot of an estimated Skellam parameter with respet to a sample mean of intensity measured in eah path for various ameras. As expeted, the sample means and Skellam parameters are linearly related, and we all this line an intensity-skellam line (ISL). We are able to estimate the Skellam parameters (that is, noise distribution) for eah intensity value I from a given ISL, representing the ISL as ð1þ ¼ f ISL ðiþ ¼R ISL I þ D ISL ; ð17þ where R ISL is the ratio between intensity and the Skellam parameter and D ISL is the Skellam parameter when expeted intensity is zero, whih refers to the dark urrent noise. Theoretially, the line should hange only aording to amera gain, not sene radiane or illumination, beause the amera gain dominantly determines the mapping from the number of photons to intensity. In other words, one an ISL of a amera has been determined, the line an be used for any image aptured by the amera as long as the amera gain is maintained. To validate this property, we aptured multiple images using a Sorpion amera under various hanges in amera aperture, illumination, and sene, while the amera gain is maintained, as shown in the left images of Fig. 8. We then manually seleted homogeneous olor pathes of the Color- Cheker in the images, and alulated Skellam parameters for eah path. The figure on the right in Fig. 8 is the satter plot of all pairs of intensity and Skellam parameters. All the sample points using all the pathes under various hanges are on a single line, whih means that the ISLs of all the images are the same. This result onfirms that an ISL is independent of illumination, amera aperture, and sene. Fig. 6. Comparison of fitting results by the Skellam distribution and the Gaussian distribution in R hannel. As verified in Fig. 5, the fitting result by the Skellam distribution has a smaller disrepany between the observed distribution and expeted distribution. Fig. 7. Linearity between intensity and Skellam parameters for HITACHI HV-F22 (3-CCD amera), Canon EOS 1D (CMOS Consumer DSLR amera), and Logiteh C600 (CMOS webam). (Larger images with more results are available in the supplemental material, available online.)

7 HWANG ET AL.: DIFFERENCE-BASED IMAGE NOISE MODELING USING SKELLAM DISTRIBUTION 1335 Fig. 8. Skellam parameters with respet to intensity under various illumination onditions and amera parameters, but fixed gain. Points in the same olor are obtained in the same image. The ISL remains fixed when the amera gain is maintained. We investigated the effet of amera gain to an ISL. Fig. 9a shows hanges in the slope R ISL and y-interept D ISL of an ISL with respet to amera gain. As expeted, a slope R ISL of an ISL is linearly proportional to a amera gain parameter. The y-interept D ISL, the amount of dark urrent noise, also inreases as amera gain inreases. We an represent the relationship between D ISL and the amera gain using a simple parametri form, suh as a seondorder polynomial. Another fator involved in taking a piture is exposure time. To see the effet of exposure time, we aptured a series of images with inreasing exposure time while keeping the amera gain onstant. The figure on the left in Fig. 9b shows the relationship between the ISL slope and exposure time. As exposure time inreases, the slope of the ISL remains nearly onstant; its variane is within This is expeted beause the relationship between intensity and the number of photons does not hange as exposure time varies. The figure on the right in Fig. 9b shows that the magnitude of dark urrent noise D ISL inreases as exposure time inreases beause longer integration time inreases the amount of dark urrent noise. Note that the hange of D ISL with respet to exposure time is muh smaller than that with respet to amera gain. In pratie, this hange of D ISL with respet to exposure time is negligible. If neessary, espeially in the ase of extremely long exposure time, the relationship between exposure time and dark urrent noise an also be represented in a parametri form, and the ISL parameters for a given exposure time an be determined from these parametri representation. To sum up the results of these experiments, the ISL is invariant under hanges of environment and amera parameters, as shown in Fig. 8, exept under hanges of amera gain and exposure time. However, we an determine the ISL parameters even under these hanges, one we have parametri representations of the ISL hanges with respet to the gain and exposure time. The parametri representation an be easily obtained using measurements like those shown in Fig. 9. In many pratial ases, it is suffiient to onsider only the effet of gain, and not that of exposure time. This linearity between intensity and a Skellam parameter is useful in estimating noise distributions. Ideally, only two pairs of intensity-skellam parameters are enough to define a line, although we generally use more 10 or 20 for robustness. One the ISL is determined, noise distribution Fig. 9. Relationship of ISL parameters to amera gain and exposure time. The ISL parameters under varying amera gain and exposure time an be represented in a parametri form. an be obtained for any intensity value in any image aptured by the amera unless its gain hanges. 4.2 Estimation of Intensity-Skellam Lines Using a Single Natural Image Estimation of an ISL requires homogeneous olor pathes, as illustrated by the line fitting shown in Fig. 7. However, finding a homogeneous path in a natural image is not trivial. Conventional approahes use olor segmentation to loate homogeneous regions [2], [3], but the segmentation is not trivial either. Instead, we use the properties of Skellam parameters to selet appropriate pathes. The Skellam mean is alulated as the mean of the differene between intensity of neighboring pixels in (13). Transition of olors between neighboring pixels auses a big shift in the Skellam mean in the olor transition. When the path ontains olor transition, the Skellam mean inreases sharply. Consequently, we an ollet homogeneous pathes to estimate the ISL by heking if their Skellam means are around zero. When an image is aptured under diretional illumination, the Skellam mean of a homogeneous path may be shifted slightly. To aommodate this, we first onstrut a histogram of Skellam means. By loalizing a peak in the Fig. 10. Histogram of the Skellam mean ð1þ ð2þ. Skellam means of homogeneous pathes are lose to zero, and a natural image tends to have a strong peak in the histogram of Skellam means.

8 1336 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 7, JULY 2012 Fig. 11. ISL estimation using a single natural image. The proposed automati path seletion method provides ISL parameters similar to those estimated by manually piked pathes in the olor pattern image. histogram of Skellam means, the mean shift effet an be onsidered. Fig. 10b shows the histogram of Skellam means for the input image of Fig. 10a. Although illumination was loated above the sene, the mean of estimated Skellam parameters shifted only slightly in our experiments. We an easily loalize the peak in the Skellam parameter differene and then use pathes that find a Skellam mean lose to the peak. However, pairs of intensity and Skellam parameters for the pathes are not exatly on a single line beause of Skellam parameter variation. In addition, some outliers may not be removed by the simple filtering using the peak of the histogram. To alleviate these problems, we apply a simple RANSAC algorithm to determine a line. Fig. 11 shows the estimation result using 1, pathes. We randomly seleted the pathes to redue time omplexity. For omparison, we show the line obtained from manually segmented olor patterns. The ISLs automatially estimated using a natural image are lose to those estimated using manually seleted pathes on the pattern image. 4.3 Determining an Intensity Aeptane Range Now we have noise distributions based on intensity differenes, and we an use the data to statistially test if an intensity differene omes from sensor noise or from different signals. Our strategy for determining this intensity aeptane range is to test a hypothesis of differene on the distribution (or to find onfidene intervals) given a onfidene level. To test a hypothesis, we use a umulative distribution funtion (df). Beause the pmf of a Skellam distribution is defined only at integer values, the df is alulated as F ðk; ð1þ ; ð2þ Þ¼ X K e ðð1þ þ ð2þ Þ ð1þ k=2 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi I jkjð2 ð1þ ð2þ Þ: ð2þ k¼ 1 The aeptane region for a ritial value I is ð18þ AðIÞ ¼fvjv <Ig¼F I; ð1þ ; ð2þ F I; ð1þ ; ð2þ : ð19þ We determine the intensity aeptane range I A, i.e., the ritial value, as I A ¼ arg max AðIÞ s:t: AðIÞ 1 ; ð20þ I where is the size of the type I error, whih is a true rejetion rate with onfidene level ð1 Þ100% [11]. One of the advantages of the proposed modeling is that the noise statistis are only dependent on amera gain. Thus, unless this gain hanges, it is possible to determine in advane a range of intensity variations aused by sensor noise. By alulating the intensity aeptane range for eah intensity value, we an build a look-up table of ranges. One we have a look-up table of preomputed onfidene intervals for a amera setting, we do not need to test a hypothesis for eah pixel every time. 5 BACKGROUND SUBTRACTION IN TEMPORAL DOMAIN In Setion 4, we have an aurate noise distribution for eah pixel in the image. In the temporal domain, bakground subtration, whih ompares a urrent image to a bakground image, is a good example for using the proposed differene-based noise modeling. For appliations using temporal domain analysis of two or more images, a probabilisti framework is usually adopted to model the unertainty of temporal hanges inluding image noise. For foreground extration, variane of intensity for eah pixel an be desribed using a multimodal distribution. Stauffer and Grimson [12] used a mixture of Gaussians as an intensity distribution, and Elgammal et al. [13] proposed a method using a nonparametri kernel estimation by building a histogram of intensity values. However, their probabilisti modeling may fail to estimate the distribution by image noise aurately without the diret onsideration of noise harateristis. 5.1 Bakground Subtration Strategy For eah pixel, the noise distribution statistially tells us whether the olor differene between the bakground and a urrent pixel omes from a foreground hange or from sensor noise. Classifiation is then performed by a statistial hypothesis test [11], or by simply looking up the intensity aeptane range table in Setion 4.3. This algorithm is extremely simple, involving a set of per-pixel operations: alulating the differene between two intensity values and omparing the differene with the intensity aeptane range value in a look-up table. Unlike other methods [13], we do not need any further proessing beause our noise modeling is so preise that subtration results have few false alarms, as shown in Fig. 12. In addition, the bakground update algorithm beomes simpler than those for other methods sine the foreground estimation result is very aurate. We update the bakground only when there is no interframe hange in onseutive T frames. This approah is feasible thanks to the highly aurate hange detetion between eah onseutive frame. 5.2 Bakground Subtration Results We applied our algorithm to outdoor sequenes aptured in bright and dark environments. We used a Point Grey Flea amera with resolution. Fig. 12a shows the results in a bright environment; the leftmost and subsequent images are the urrent and bakground images, respetively. Foreground detetion results were so aurate that our method was able to detet nearly true boundaries for moving people and ars. Compared with the results obtained by the mixture of Gaussians [12], results obtained

9 HWANG ET AL.: DIFFERENCE-BASED IMAGE NOISE MODELING USING SKELLAM DISTRIBUTION 1337 Fig. 12. Bakground subtration results. The proposed method works robustly even in the omplex bakground and a dark environment. (Additional larger results are available in the supplemental material, available online.) with our differene-based noise modeling provided muh smaller misses in spite of omplex bakgrounds. This is beause the mixture of Gaussians annot guarantee an exat distribution of pixel olor differenes, but instead provides merely rough approximation of Gaussian distributions for bakground estimation. In Fig. 12b, we present the foreground detetion results in a dark environment. The sequene was aptured outdoors at 9 p.m. Sine the images are very dark, detetion of moving objets is not easy, even by human vision. Intensity variation of the hanged regions in this dark environment was muh lower than that in the bright environment. Therefore, exat noise distribution for lassifying foreground pixels was essential in this situation. The proposed method was able to detet moving objets well beause eah pixel had an exat distribution aording to intensity. The method that used the mixture of Gaussians failed to detet most of moving objets due to overfitting of the bakground model. The overfit model inludes not only temporal olor hanges of the bakground, but also olor hanges of the foreground, whih makes it hard to disriminate foreground pixels from it. In Table 1, we make a quantitative omparison using manually seleted ground truth. Beause the proposed method is sensitive to the signal hange, it generates fewer false negatives than the mixture of Gaussians. Though the proposed method seems to generate more false positives than the mixture of Gaussians, almost all the false positives are near the shadow region, whih shows the sensitivity of the proposed model to intensity hanges. Note that the performane of the proposed model is not signifiantly degraded for the dark images without any parameter hange. All the ground-truths and experimental results are provided in the supplemental material, available online. 6 EDGE DETECTION IN SPATIAL DOMAIN We apply our noise modeling to edge detetion as a spatial domain appliation. Most edge detetors smooth an image using Gaussian kernels to suppress the image noise. However, the edge detetor based on our noise model does not smooth images beause the proposed algorithm presents the exat ranges of intensity variations aused by sensor noise. 6.1 Noise Handling for Feature Detetion For appliations based on a single image, most approahes do not model image noise expliitly in the spatial domain, but suppress image noise as a preproessing step. The TABLE 1 Performane Comparison Using Ground-Truths The proposed method generated fewer false negatives than the mixture of Gaussians while maintaining low false positive rates.

10 1338 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 7, JULY 2012 Fig. 13. Log-normalized edge measures under various illumination onditions: From the left, input image, Canny edge response, RGB gradient magnitude, quasi-invariants [14], generalized ompass operator [15], and the proposed edge measure. The proposed edge measure suppresses the noise while preserving edges even in the very dark image without hanging any parameters. (Larger images are available in the supplemental material, available online.) 8 d >< e yði; jþ I A yði; jþ ¼ ; if d I y ði; jþ IA ; >: A 0; if d y ði; jþ ð24þ <IA ; onventional strategy for extrating features suh as a orner and an edge is to preproess the images using Gaussian kernels to suppress the image noise via smoothing [14], [17]. A linear filtering method, suh as Gaussian smoothing, basially uts off high-frequeny omponents of signals as a low-pass filter, and it suppresses the important abrupt hanges of the signals as well as noise. Consequently, features usually defined by the high frequeny are affeted by the filtering: rounded orners, disonneted T-juntions, and displaements of features near intensity gradients [18]. To minimize this effet, filtering adaptive to the loal struture an be used [19], but it loses one of the key advantages of the omputational effiieny of the Gaussian smoothing by its linear separability. Although the parameters of Gaussian smoothing must be determined properly, the effets of the parameters have been onsidered in only a few studies [19]. 6.2 Edge Detetion Strategy Our strategy for edge detetion is similar to that for bakground subtration. When the differene between two neighboring pixels is within the intensity aeptane range, there is no edge; otherwise, there is an atual hange of sene olors, whih implies an edge. Magnitudes of derivatives in horizontal and vertial diretions at ði; jþ are defined as d x ði; jþ ¼ j I ði 1;jÞ I ði þ 1;jÞj; ð21þ d y ði; jþ ¼ j I ði; j 1Þ I ði; j þ 1Þj; ð22þ where I is an intensity value in the olor hannel, whih is one of r, g, and b in RGB olor spae. Then, edge measures in horizontal and vertial diretions are defined as e xði; jþ ¼ 8 d < x ði; jþ IA ; if d I x ði; jþ IA ; : A 0; if d x ði; jþ <IA ; ð23þ where I A denotes the intensity aeptane range, as determined in Setion 4.3 of the intensity I ði; jþ. This edge measure alulates the normalized distane of the intensity I ði; jþ from the intensity aeptane range I A. If the magnitude of a derivative is within the range I A, the edge measure is zero. The total edge measure of a pixel is obtained by summing all the horizontal and vertial edge measures as eði; jþ ¼ X e x ði; jþþx e yði; jþ: ð25þ Only when all the horizontal and vertial edge measures of a pixel are zero is the pixel regarded as a nonedge pixel. This method preemptively suppresses an edge response aused by sensor noise. One edge measures for all pixels are alulated, we apply nonmaximum suppression. In [17], two values are required for a hysteresis threshold to link edges after nonmaximum suppression. However, we regard the pixels that have nonzero edge measures as being edge pixels beause the intensity differene exeeds the intensity aeptane range determined from the noise distribution. Thus, we do not disard nonzero edge pixels when generating a final edge image. 6.3 Edge Detetion Results In the first experiment, we show the independent property of the proposed method to the brightness and illumination. We do not hange the parameters of eah detetor for images aptured by the same amera. Fig. 13 shows lognormalized edge measures under various illumination onditions in order to observe weak responses partiipating in the edge linking. We ompared the results of the proposed method with a 99 perent onfidene level to those of other edge operators. Our differene-based method suppresses the noise properly and detets edges even in very dark environments without hanging any parameters.

11 HWANG ET AL.: DIFFERENCE-BASED IMAGE NOISE MODELING USING SKELLAM DISTRIBUTION 1339 Fig. 14. Edge detetion results. The proposed method provides good noise suppression without image smoothing. (More results are available in the supplemental material, available online.) Using the parameter for the first image, the Canny operator does not detet the details under dark illumination (the seond image) and does not suppress the false edges aused by image noise (the third image), whih means that the parameter should be arefully hosen for eah image. We also ompared the proposed differene-based edge detetor to other existing olor edge detetors, i.e., RGB gradient magnitude, shadow-shading-speular quasi-invariants [14], and generalized ompass operator [15], using MATLAB odes provided by the authors. Color gradient and quasiinvariant methods provided large responses by image noise, whih an result in many false alarms, even under normal illumination ondition. Results by the quasiinvariant method also show poor onnetivity of edge measures, resulting in many short edges after edge linking. The generalized ompass operator worked well under the normal illumination, but did not generate responses in the dark image and made some responses based on noise. Fig. 14 shows the final results of the proposed edge detetion after nonmaximum suppression and binarization, ompared with other edge detetors. Conventional edge detetors [17] require user-defined tuning parameters for image smoothing and edge linking, whih are not easy to determine. Liu s method [3] statistially hooses the thresholds of the Canny method, but a user still needs to determine a proper smoothing filter. The olor tensor based edge detetors [16] and the ompass operator [15] requires a proper smoothing sale as well. Unlike the onventional edge detetors, the proposed edge detetor does not smooth images, but suessfully suppresses the image noise while preserving the fine details, as shown in Fig. 14. We have also ompared the Skellam and signal dependent Gaussian distributions for an intensity differene in the edge detetion task, as shown in Fig. 15. The input image was aptured by a Point Grey Flea amera with a low noise level and long exposure time. The only differene between Figs. 15b and 15 is the distribution for intensity differene. The Skellam distribution for intensity differene suppresses the image noise suessfully, while the Gaussian distribution does not, by underestimating the thresholds for eah intensity. This shows that the Skellam distribution an explain the intensity differene better than the Gaussian distribution for edge detetion. Fig. 15. Results of the edge detetion with 99.9 perent onfidene intervals using the proposed Skellam distribution and its signal-dependent Gaussian approximation for intensity differene. The Gaussian distribution underestimates the threshold values and does not suppress the image noise fully.

12 1340 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 7, JULY CONCLUSION In this paper, we propose to diretly model intensity differene for determining whether two intensity values ome from the same radiane or not. The pixel intensity should be expressed as a random variable, and the onventional additive model may not explain the intensity distribution due to the laws of quantum physis and illumination effets. We have presented some examples of the wrong assumptions of the onventional models, and verify that the diret modeling of intensity differene an be a better way to model the image noise. We also present a Skellam distribution for a linear response amera derived from the dominant photon noise assumption. The validity has been heked thoroughly with various ameras under various amera settings and illumination onditions. We have found that the level of intensity differene is the same in the spatial and in the temporal domains, and this property enables us to use a single image for the modeling. In addition, there exists a strong linear relationship between the intensity and the Skellam parameters measured in images even under the natural illumination. This differene-based model is only dependent on the amera gain, not on the sene, illumination, and aperture. One it is obtained, the noise level of the other intensity values an be determined simply. These properties make the sensor noise proess muh easier than the previous methods. The differene of two intensity values an be tested on every pixel by statistial tests based on the obtained distribution of intensity differene. To show the validity of the proposed intensity differene model, we applied it to bakground subtration in the temporal domain and to edge detetion in the spatial domain. In bakground subtration, foreground pixels an be aurately deteted using per-pixel operations based on the intensity aeptane range. Beause per-pixel detetion is aurate, further proess to refine the result is not required. In the spatial domain, we design a simple olor edge detetor in the same way of bakground subtration. Without any smoothing or filtering, the edge detetor based on the intensity differene model suppresses image noise, while it detets the atual radiane hanges well in both dark and bright images. As we presented in this paper, the proposed noise model an be used to detet meaningful signal hanges robustly to the image noise. Further works inlude development of image denoising, feature detetion, and optial flow estimation robust to the image noise. ACKNOWLEDGMENTS This researh is supported by the National Researh Laboratory (NRL) program (No. M ) of the Ministry of Siene and Tehnology (MOST) and the National Researh Foundation of Korea (NRF) grant funded by the Korea government (MEST) (No ). [2] C. Liu, R. Szeliski, S.B. Kang, C.L. Zitnik, and W.T. Freeman, Automati Estimation and Removal of Noise from a Single Image, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 30, no. 2, pp , Feb [3] C. Liu, W.T. Freeman, R. Szeliski, and S.B. Kang, Noise Estimation from a Single Image, Pro. IEEE Conf. Computer Vision and Pattern Reognition, pp , [4] G.E. Healey and R. Kondepudy, Radiometri CCD Camera Calibration and Noise Estimation, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 16, no. 3, pp , Mar [5] A. Foi, M. Trimehe, V. Katkovnik, and K. Egiazarian, Pratial Poissonian-Gaussian Noise Modeling and Fitting for Single- Image Raw-Data, IEEE Trans. Image Proessing, vol. 17, pp , Ot [6] M. Yavuz and J.A. Fessler, Maximum Likelihood Emission Image Reonstrution for Randoms-Preorreted PET Sans, Pro. IEEE Nulear Siene Symp. Conf. Reord, pp. 15/229-15/233, [7] Y. Hwang, J.-S. Kim, and I.-S. Kweon, Sensor Noise Modeling Using the Skellam Distribution: Appliation to the Color Edge Detetion, Pro. IEEE Conf. Computer Vision and Pattern Reognition, [8] I. Young, J. Gerbrands, and L. van Vliet, Image Proessing Fundamentals, The Digital Signal Proessing Handbook, V.K. Madisetti, D.B. Williams, eds., pp. XI-1-XI-81, CRC Press, [9] J.G. Skellam, The Frequeny Distribution of the Differene between Two Poisson Variates Belonging to Different Populations, J. Royal Statistial So.: Series A, vol. 109, no. 3, p. 296, [10] M. Abramowitz and I.A. Stegun, Handbook of Mathematis Funtions with Formulas, Graphs, and Mathematial Tables. Dover, [11] E.R. Dougherty, Probability and Statistis for the Engineering, Computing and Physial Sienes. Prentie Hall, [12] C. Stauffer and W. Grimson, Learning Patterns of Ativity Using Real-Time Traking, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 22, no. 8, pp , Aug [13] A. Elgammal, D. Harwood, and L. Davis, Non-Parameteri Model for Bakground Subtration, Pro. European Conf. Computer Vision, pp , [14] J. Weijer, T. Gevers, and J. Geusebroek, Edge and Corner Detetion by Photometri Quasi-Invariants, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 27, no. 4, pp , Apr [15] M.A. Ruzon and C. Tomasi, Edge, Juntion, and Corner Detetion Using Color Distributions, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 23, no. 11, pp , Nov [16] J.V.D. Weijer, T. Gevers, and A.W.M. Smeulders, Robust Photometri Invariant Features from the Color Tensor, IEEE Trans. Image Proessing, vol. 15, no. 1, pp , Jan [17] J. Canny, A Computational Approah to Edge Detetion, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 8, no. 6, pp , Nov [18] A. Blake and A. Zisserman, Visual Reonstrution. The MIT Press, [19] P. Saint Mar, J. Chen, and G. Medioni, Adaptive Smoothing: A General Tool for Early Vision, IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 13, no. 6, pp , June Youngbae Hwang reeived the BS, MS, and PhD degrees in eletrial engineering and omputer siene from KAIST in 2001, 2003, and 2009, respetively. He is urrently a senior researher at Korea Eletronis Tehnology Institute (KETI). After six months as a postdotoral researher for KAIST, he worked for the Robot Business Group at Samsung Tehwin as a senior researh engineer for one and a half years. His researh interests inlude image noise modeling, low-level proessing, video surveillane, 3D reonstrution, and robot loalization. He is a member of the IEEE. REFERENCES [1] F. Alter, Y. Matsushita, and X. Tang, An Intensity Similarity Measure in Low-Light Conditions, Pro. European Conf. Computer Vision, pp , 2006.

13 HWANG ET AL.: DIFFERENCE-BASED IMAGE NOISE MODELING USING SKELLAM DISTRIBUTION 1341 Jun-Sik Kim reeived the BS degree in eletroni engineering from Yonsei University in 1999, and the MS and PhD degrees in eletrial engineering and omputer siene from KAIST in 2001 and 2006, respetively. He is urrently a projet sientist at Carnegie Mellon University. After about one year as a postdotoral researher for KAIST, he worked for the Robotis Institute at Carnegie Mellon University as a postdotoral fellow for two years. His researh interests inlude amera alibration, 3D reonstrution, amera motion traking, sensor fusion, image sensor noise modeling, and aeleration using parallelized hardwares. He is a member of the IEEE. In So Kweon reeived the BS and MS degrees in mehanial design and prodution engineering from Seoul National University, Seoul, Korea, in 1981 and 1983, respetively, and the MS and PhD degrees in robotis from the Robotis Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, in 1986 and 1990, respetively. He worked for Toshiba R&D Center, Japan, and joined the Department of Automation and Design Engineering, KAIST, Seoul, Korea, in 1992, where he is now a professor with the Department of Eletrial Engineering. His researh interests are in omputer vision, robotis, pattern reognition, and automation. Speifi researh topis inlude invariant-based visions for reognition and assembly, 3D sensors and range data analysis, olor modeling and analysis, robust edge detetion, and moving objet segmentation and traking. He is a member of ICASE and the ACM. He is a member of the IEEE.. For more information on this or any other omputing topi, please visit our Digital Library at