THE human visual system perceives the color of an

Transcription

1 3612 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 8, AUGUST 2012 Chromaticity Space for Illuminant Invariant Recognition Sivalogeswaran Ratnasingam, Member, IEEE, and T. Martin McGinnity, Member, IEEE Abstract In this paper, an algorithm is proposed to extract two illuminant invariant chromaticity features from three image sensor responses. The algorithm extracts these chromaticity features at pixel level and therefore can perform well in scenes illuminated with nonuniform illuminant. An approach is proposed to use the algorithm with cameras of unknown sensitivity. The algorithm was tested for separability of perceptually similar colors under the International Commission on Illumination standard illuminants and obtained a good performance. It was also tested for color-based object recognition by illuminating objects with typical indoor illuminants and obtained a better performance compared to other existing algorithms investigated in this paper. Finally, the algorithm was tested for skin detection invariant to illuminant, ethnic background and imaging device. In this investigation, daylight scenes under different weather conditions and scenes illuminated by typical indoor illuminants were used. The proposed algorithm gives a better skin detection performance compared to widely used standard color spaces. Based on the results presented, the proposed illuminant invariant chromaticity space can be used for machine vision applications including illuminant invariant color-based object recognition and skin detection. Index Terms Chromaticity constancy, color constancy, color-based object recognition, illuminant invariant color space. I. INTRODUCTION THE human visual system perceives the color of an object largely independent of the viewing environment, including illuminant and scene geometry. However, an imaging device records the color of an object depending on the illuminant, scene geometry and the spectral characteristic of the imaging device. The dependency of the image sensor responses on the viewing environment (viewing illuminant and scene geometry) makes it difficult to use the recorded color as a reliable cue in machine vision applications. The Manuscript received June 21, 2011; revised January 5, 2012; accepted March 7, Date of publication April 3, 2012; date of current version July 18, This work was supported by the Centre of Excellence in Intelligent Systems (CoEIS) Project, funded by Northern Ireland Integrated Development Fund and InvestNI. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. David S. Taubman. S. Ratnasingam was with the Intelligent Systems Research Centre, School of Computing and Intelligent Systems, Magee Campus, University of Ulster, Londonderry BT48 7JL, U.K. He is currently with the NICTA, Canberra Research Laboratory, Canberra ACT 2601, Australia, and also the College of Engineering and Computer Science, Canberra 0200, Australia ( sivaloges@yahoo.co.in). T. M. McGinnity is with the Intelligent Systems Research Centre, School of Computing and Intelligent Systems, Magee Campus, University of Ulster, Londonderry BT48 7JL, U.K. ( tm.mcginnity@ulster.ac.uk). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TIP /$ IEEE challenge for recognizing the spectral reflectance of a surface of an object is to eliminate the effects of the light source that illuminates the object. This is often referred to as the problem of color constancy. In the past, researchers have proposed a number of different approaches for solving the color constancy problem, by making assumptions about the scene or illuminant or the imaging device or a combination of these. In this paper, we propose an algorithm that extracts two illuminant invariant chromaticity features from three sensor responses. Our objective is to create a 2-D chromaticity space that is similar to the chromaticity spaces rg or xy, but unlike these spaces ensure the proposed space is invariant to changes in the viewing illuminant. The performance of the proposed algorithm is significantly better than those compared in this paper, in that the proposed chromaticity space gives the same values for surfaces with the same reflectance, regardless of the illuminant. This 2-D illuminant invariant space could be used for illuminant invariant recognition purposes in machine vision applications. The structure of this paper is as follows. In Section II, we briefly review the related work and illustrate how this contribution extends existing research. In Section III, we present the mathematical formulation of the proposed algorithm and show the 2-D illuminant invariant space formed by the algorithm. The performance of the algorithm in separating perceptually similar colors is investigated in Section IV. In Section V, the algorithm is investigated for color-based recognition and skin detection invariant to illuminant and compared with alternative methods. Finally, conclusions are drawn in Section VI. II. RELATED WORK Von Kries [1] first proposed a chromatic adaptation model for studying the visual adaptation of the human visual system, assuming that the spectral sensitivity of the sensors does not overlap each other. This adaptation model has been used by several researchers to either estimate the illuminant effect on the image or to extract the illuminant invariant features [2]. Based on the assumption of both the blackbody model of the illuminant power spectrum and infinitely narrow band image sensors, Marchant and Onyango [3] proposed an algorithm to extract shadow invariant features. These authors developed a transformation for color images that is invariant to shading and shadow in daylight scenes. In particular, they process three image sensor responses in the linear scale to extract a reflectance feature that is approximately invariant to shading and shadow. Land and McCann [4] proposed a computational model ( retinex algorithm) for estimating the illuminant

2 RATNASINGAM AND MCGINNITY: CHROMATICITY SPACE FOR ILLUMINANT INVARIANT RECOGNITION 3613 effect on the scene by processing the logarithm of image sensor responses. The advantage of using the logarithm of image sensor responses for solving color constancy is that this allows the reflectance components to be easily separated from the illuminant dependent components. Separation of the illuminant and the reflectance components in the logarithm scale can be easily understood by considering the following basic image equation: R x,e = a x n x I x S x (λ)e x (λ)f(λ)dλ (1) ω where R x,e is the response of an image sensor with spectral sensitivity F(λ) and the dot product between the unit vectors a x n x represents the geometric factor of a scene. The unit vectors a x and n x represent the direction of the light source and the direction of surface normal, respectively. The quantity S(λ) is the surface reflectance of the scene at point x, E(λ) is the spectral power distribution of the light source and I x is the intensity of the illuminant at point x on the scene. The integration is over the visible range nm wavelength. To simplify the basic image equation, assuming the spectral sensitivity of the image sensor is infinitely narrow and applying the sifting property of the Dirac delta function results in R x,e = a x n x I x S x (λ i )E x (λ i ). (2) Taking logarithm of both sides of (2) will result in separation of the different components [4] [6] log(r x,e ) = log{g x I x }+log{e x (λ i )}+log{s x (λ i )} (3) where G x (= a x n x ) is the geometry factor. In (3) the first term depends on the scene geometry and intensity of the illuminant. The second and third terms depend on the chromaticity of the illuminant and the surface reflectance of the object, respectively. Taking logarithm to both sides of (2) has transformed the product terms into simple summation. From (3) it can be seen that a linear combination of sensors may be used to remove the illuminant effect. This is relatively easy compared to working with the linear form as in (1). Based on the infinitely narrow band sensor assumption and the blackbody assumption, Finlayson and Hordley [5] proposed an algorithm to extract an illuminant invariant reflectance feature. The authors first calculate the logarithm difference of sensor responses from one of the sensor responses. This normalization results in a 2-D space and subsequently an eigenvector decomposition was applied to find the two eigenvectors. Then the 2-D space is projected in the direction of the larger eigenvector to obtain an illuminant invariant feature. The authors refer this illuminant invariant feature as 1-D color constancy. However, Ebner [2] showed that in the 1-D color constancy many of the perceptually discriminable reflectances are indiscriminable (i.e., confused). Finlayson and Drew [7] proposed an algorithm by extending the 1-D color constancy by applying it on four image sensor responses to obtain two illuminant invariant features. Recently, Ratnasingam and Collins [6] proposed a relatively simple algorithm that processes four image sensor responses. Ratnasingam et al. [8] showed that the four sensor algorithms proposed by Finlayson and Drew [7] and Ratnasingam and Collins [6] are comparable in performance. However, most consumer cameras have only three sensor responses. Berwick and Lee [9] proposed a 2-D illuminant invariant chromaticity space by processing three sensor responses. In deriving this space the authors used a diagonal transform to model the sensor response and a translation to model the effect of variations in the illuminant spectrum. The main disadvantage of this approach is that it works only in scenes illuminated by a single uniform illuminant. Apart from these algorithms there are color constancy algorithms based upon making assumptions about the scene. One of the well-known algorithms is the grey world algorithm proposed by Buchsbaum [10]. This algorithm assumes that any scene is achromatic and therefore the average value of the sensor response of a scene provides an estimate of the illuminant that illuminates the scene. This algorithm requires a large number of colors to be present in the scene. A weakness of the algorithm is that if the scene has a dominant color (example large region of an image occupied by blue sky or grass land or tree) the algorithm results in an incorrect estimate for the illuminant. Similarly, the white patch algorithm is a simplified version of the Land and McCann s [4] Retinex algorithm [11], [12]. This algorithm relies on the assumption that there is a bright patch that reflects the entire light fell on it and the image sensor responses of the bright patch is used as an estimate of the illuminant. A drawback of this algorithm is a single bright pixel could lead to a bad estimate of the illuminant. Another well-known color constancy algorithm is the gamut mapping algorithm proposed by Forsyth [13]. As this algorithm imposes constraints on the illuminant and reflectance of the scene; this is also referred to as the constraint-based approach. The gamut mapping algorithm has two steps; in the first step two gamuts are obtained, namely the image gamut and the canonical gamut. The canonical gamut is created by taking all the surfaces present in the scene under canonical illuminant. In the second step, under von Kries [1] chromatic adaptation model, the scene illuminant is estimated using the image gamut and the canonical gamut. The main drawbacks of this algorithm are that it is difficult to implement, that it sometimes gives an estimate that is physically unrealizable or in worst cases gives no estimate at all. This algorithm also assumes many surfaces with different colors present in the scene. All three algorithms (white patch, grey world, and gamut mapping) assume that the scene is illuminated uniformly by a single illuminant. This assumption does not often hold in real world scenes. Color constancy algorithms based on Bayesian approaches have also been proposed by modelling the variability of the reflectance and illuminant as random variables. The illuminant is estimated from the posterior distribution conditioned on the given image [14], [15]. Despite the different approaches for extracting illuminant invariant chromaticity for solving color constancy [5], [6], [9], [16], [17], Hordley [18] and Funt et al. [19] concluded that the performance of the existing color constancy algorithms are not good enough for machine vision applications. More recently, Foster [20] investigated the color constancy problem

3 3614 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 8, AUGUST 2012 extensively and concluded that this is a challenging problem to solve and more research work is required. In the next section, we present the mathematical formulation of the proposed algorithm and show the 2-D illuminant invariant space formed by the algorithm III. ALGORITHM We propose an empirical method for extracting illuminant invariant chromaticity features based on the approach proposed by Marchant and Onyango [3]. In particular, Marchant and Onyango [3] processed in the linear scale and extracted a single feature that is invariant to shadow and shading. However, in our proposed method we use the logarithm of sensor responses to easily separate the illuminant component from the reflectance components. In particular, we extract two illuminant invariant features from three sensor responses as follows: F 1 = log(r 2 ) {αlog(r 1 ) + β log(r 3 )} (4) F 2 = log(r 3 ) {γ log(r 1 ) + δ log(r 2 )} (5) where R 1, R 2,andR 3 are image sensor responses and the sensors are numbered starting from the shortest wavelength. The quantities α, β, γ,andδ are the channel coefficients. The first illuminant invariant feature (F 1 ) is formed by estimating the illuminant effect on the sensor response 2 using the responses 1 and 3. Similarly, the second illuminant invariant feature (F 2 ) is formed by estimating the illuminant effect on the sensor response 3 using the responses 1 and 2. It is also possible to form a third feature that will be a combination of the first two features, but we do not exploit this in the current approach. The unwanted variations in the sensor responses caused by the scene geometry, illuminant intensity and power spectrum can be removed if the channel coefficients (α, β, γ, and δ) satisfy the following equation: 1 = α + β (6) λ 2 λ 1 λ 3 1 = γ + δ (7) λ 3 λ 1 λ 2 where λ 1, λ 2,andλ 3 are the wavelengths at which the three image sensors have effective peak sensitivity. To estimate the channel coefficients we use an optimization setting based on the steepest descent algorithm in such a way that the separability of the perceptually similar colors is maximized in the 2-D illuminant invariant chromaticity space formed by the two extracted features. Here, we empirically find the four channel coefficients that provide the best separation of perceptually similar reflectances under different illuminants. To generalize the algorithm for standard illuminants and to obtain a better color constancy performance with real world illuminants we optimize the channel coefficients with the International Commission on Illumination standard illuminants instead of blackbody illuminants. As we are interested in recognizing perceptually similar colors under different illuminants we have chosen 100 pairs of normalized Munsell reflectances with pair wise member separation of three CIELab units as used by Ratnasingam et al. [8]. Ratnasingam et al. [8] used a steepest F 2 F F (a) F 1 (b) Fig. 1. Illuminant invariant chromaticity feature space formed with normalized sensor responses of DXC930 camera and the parameters listed in Table I. In this space illuminant invariant chromaticity features extracted from 202 Munsell reflectances when illuminated by (a) CIE standard illuminant with correlated color temperature (CCT) 6500 K and (b) ten spectra of CIE standard illuminants are shown. The color of the points show the pseudo color of the Munsell test surfaces. descent algorithm to optimize the sensitivity functions and we use the same optimization method to optimize the channel coefficients. In this optimization, the Mahalanobis distance metric was used to determine the number of correctly identifiable color pairs and the channel coefficients were chosen in such a way that the number of correctly identified pairs is maximized [8]. In this optimization, Sony DXC930 camera responses were used as the image sensors and 20 spectra of CIE standard daylight with CCT between 4000 and K were used [8]. Fig. 1 shows the illuminant invariant chromaticity feature space formed when using the sensitivity functions of the Sony DXC930 camera. Each point on this space shows the two features extracted from one of the 202 Munsell samples with pseudo color when illuminated by (a) the CIE standard daylight illuminant with CCT 6500 K and (b) ten spectra of CIE standard illuminants. From this figure it can be seen that the perceptually similar reflectances are nonoverlapping and therefore the space has the basic ability to identify

4 RATNASINGAM AND MCGINNITY: CHROMATICITY SPACE FOR ILLUMINANT INVARIANT RECOGNITION F 2 F 1 points within boundary Percentage of Proposed algorithm, 6 CIELab units 20 Proposed algorithm, 10 CIELab units Finlayson and Drew, 6 CIELab units Finlayson and Drew, 10 CIELab units Full width at half maximum (nm) Fig. 2. Mahalanobis distance boundaries for a pair of Munsell test reflectance samples from 6 CIELab unit pair when illuminated by 20 spectra of CIE standard illuminants. Noise was modelled as Gaussian noise of 100 samples that represent Gaussian noise of 40 db. The cluster points denoted by + and o show the location of the features extracted from the pair of Munsell samples respectively. TABLE I OPTIMUM CHANNEL COEFFICIENTS FOR DXC930 CAMERA Parameter Optimum value Alpha Beta Gamma Delta perceptually similar reflectances. In particular, when changing the illuminant, the extracted reflectance features for surfaces are not overlapping and therefore should not be confused with one another, being discriminable in the proposed feature space. However, using one of the features F 1 or F 2 leads to confusion of some of the perceptually different colors. For example projecting the 2-D space shown in Fig. 1 in the direction of the x-axis is equivalent to only using F 2 and it can be seen that this projection would not be able to separate blues from greens. Similarly, projecting the space in the direction of the y-axis is equivalent to only using F 1, but in this case the same blues are confused with orange-reds. It can be seen that the confusing colors when using either F 1 or F 2 are different and therefore the two proposed features capture different illuminant invariant information about a scene. However, using both features (F 1 and F 2 ) together eliminates any confusion of separating perceptually different colors. In fact, the conclusion that a 2-D space is needed to separate quite different colors has been reached previously by Ebner [2]. Section V investigates the advantages of using two features instead of one for illuminant invariant color-based recognition. IV. SEPARABILITY OF PERCEPTUALLY SIMILAR COLORS The performance of the proposed algorithm was investigated for the ability to recognize perceptually similar colours under the CIE standard illuminants and compared with the four sensor algorithms proposed by Finlayson and Drew [7]. Finlayson and Drew s [7] algorithm starts with four sensor responses and in the first step the sensor responses were normalized Fig. 3. Separability results of the proposed algorithm and Finlayson and Drew s [7] algorithm when tested with perceptually similar Munsell reflectances and the CIE standard illuminants. The responses were multiplied with 40 db Gaussian noise and the resultant responses were quantized to 10 bits. with one of the responses to remove the intensity component and scene geometry component from the responses. The resultant-normalized responses were projected in the direction of illuminant-induced variation on the sensor responses. This projection results in a 2-D space. As there are no consumer cameras that have four sensors, in this investigation Gaussian sensitivity functions were used to model the image sensors. The proposed algorithm and the Finlayson and Drew [7] algorithm are investigated using three and four evenly spread Gaussian sensors in the visible region (400 to 700 nm), respectively. As described by Ratnasingam et al. [8], two sets of 100 pairs of normalized Munsell samples with member separation of six and ten CIELab units were chosen. In the CIELab color space six and ten unit distances are described as good and acceptable match of colors, respectively, in color reproduction [21]. This means in color reproduction, if the distance between the actual and reproduced color of a surface in the CIELab space is six units or below is considered as colorimetrically good reproduction of color, and if the distance is between six and ten CIELab units is considered as colorimetrically acceptable reproduction. Two sets of 20 spectra of CIE standard daylight were chosen, one for fitting the Mahalanobis distance boundary and the other for assessing the correct recognition of pairs as described by Ratnasingam and Collins [8]. For defining the cluster boundary occupied by a pair of Munsell sample, this Munsell pair was illuminated by the first illuminant set and the extracted features were projected onto the 2-D space. These two Munsell samples form two clusters when illuminated by 20 illuminants and multiplied by 100 samples of Gaussian noise [8]. Effectively each Munsell sample results in a cluster of 2000 points on the feature space (see Fig. 2). In the first step the mean of the two clusters were determined and a Mahalanobis distance boundary is drawn with equal distance from the two cluster centers in such a way that the two respective boundaries formed by the pair of samples touch each other (see Fig. 2). Once the cluster boundaries have been determined, the same Munsell pair was

5 3616 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 8, AUGUST 2012 TABLE II OPTIMUM CHANNEL COEFFICIENTS FOR 80-nm FWHM GAUSSIAN SENSORS Parameter Optimum value Alpha Beta Gamma Delta TABLE III PERFORMANCE OF THE PROPOSED ALGORITHM WHEN TESTED WITH MUNSELL AND MEASURED FLORAL REFLECTANCES BY ILLUMINATING WITH CIE STANDARD DAYLIGHT ILLUMINANTS.IN THIS TEST DXC930 CAMERA RESPONSES WERE APPLIED AS INPUT TO THE PROPOSED ALGORITHM Munsell, CIE daylight Floral, CIE daylight Pair wise distance in CIELab space Percentage separability (%) illuminated by the second set of illuminant and projected onto the feature space. The number of feature points fell within the correct Mahalanobis distance boundary was counted and the same procedure was repeated for all the pairs in a test reflectance set and the percentage of correct classification was calculated. The derivation of the proposed algorithm and Finlayson and Drew s [7] algorithm assume infinitely narrow band sensors. To study the impact of breaking this assumption on the performance of the algorithms, the full width at half maximum (FWHM) of the Gaussian sensors were varied between 20 and 200 nm. As described in Section III, the channel coefficients were optimized for interested sensor spectral widths using Gaussian sensitivity functions. The optimum coefficients for 80-nm FWHM are listed in Table II. To perform a realistic investigation of the algorithms, the sensor noise was modelled in such a way that the signal to noise ratio of the responses is 40 db and the resultant responses were quantized to 10 bits [8]. From the results presented in Fig. 3 it can be seen that Finlayson and Drew s [7] four sensor algorithms give slightly better performance. However, real world consumer cameras, including the DXC930, have sensor width (FWHM) in the range of nm. Comparing the performance of the algorithms, the difference in performance in this range of sensor width is small. Particularly, the proposed algorithm gives a better performance with sensor width between 60 and 80 nm compared to the performance obtained with sensor width between 20 and 40 nm. This is because in the proposed approach we have optimized the four channel coefficients with the actual sensitivity function of the image sensors used. Treating these four channel coefficients as independent variables resulted in a better performance even when the initial assumptions do not hold. However, Finlayson and Drew s [7] points within boundary Percentage of Max, 6 CIELab units 20 Max, 10 CIELab units Min, 6 CIELab units Min, 10 CIELab units Full width at half maximum (nm) Fig. 4. Performance of the proposed algorithm with Munsell test reflectance spectra (six units and 10 units pairs). In this test the surfaces were illuminated using the 20 spectra of CIE standard illuminants with the lowest intensity and the highest intensity of six decade range of illuminance variation. The sensors sensitivity functions were modelled using Gaussian functions of different FWHM. The responses were multiplied to 100 samples of sensor noise equivalent to 40 db and the resultant responses were quantized to 10 bits. four sensor algorithms impose an additional constraint that the sum of the channel coefficients is equal to one. This constraint results in degradation in performance of Finlayson and Drew s [7] algorithm when the sensor width increases. Table III lists the performance of the proposed algorithm using DXC930 camera responses in recognizing perceptually similar colors under CIE standard illuminants. From these results we can see that the proposed algorithm gives a good performance in identifying perceptually similar colors with three image sensors of realizable spectral width. To investigate the performance of the algorithm with real world high dynamic range (HDR) variation of illuminance we have tested our approach with 20 spectra of CIE standard illuminants, with illuminance varying over six decades. In particular, we have chosen six decade illuminance variation based on the fact that the shadow, skylight, and direct sunlight cause illuminance variation in the natural scene of the order of six decades [22]. To investigate the recognition of perceptually similar reflectance surfaces in HDR scenes, the 20 spectra of CIE standard test illuminants were scaled in such a way that the illuminance varies over six decades. The performance of the algorithm was investigated by illuminating the Munsell reflectance sets (six- and ten-unit pairs) by illuminants with the lowest and the highest values of intensity in the six decade range. The responses were multiplied with 40-dB Gaussian noise and the resultant responses were quantized to 10 bits. The performance of the algorithm was evaluated as described above by fitting the Mahalanobis distance boundaries around the clusters formed by each of the Munsell sample. Test results are shown in Fig. 4. From these results it can be seen that the proposed algorithm gives almost identical performance with the lowest and highest intensity illuminants of the six decade illuminance range. These results suggest that the algorithm can be used with the HDR scenes to extract useful illuminant invariant chromaticity features that can be used for recognition of perceptually similar colors.

6 RATNASINGAM AND MCGINNITY: CHROMATICITY SPACE FOR ILLUMINANT INVARIANT RECOGNITION x Relative power Wavelength (nm) Fig. 6. scenes. Spectral power distribution of the illuminants used to illuminate the TABLE IV PERCENTAGE OF CORRECT RECOGNITION OF OBJECTS FOR THE PROPOSED AND FINLAYSON AND HORDLEY S [5] ALGORITHM Fig. 5. Typical objects in the group of minimal specularities [19]. V. ILLUMINANT INVARIANT RECOGNITION A. Illuminant Invariant Object Recognition In this section, the performance of the proposed algorithm is investigated for illuminant invariant object recognition using the features F 1 and F 2. The advantage of using two features instead of one for color-based recognition is also investigated. In this investigation four different scenarios of light reflection on a surface are considered. The data used in this test are classified into four groups: nonnegligible dielectric specularities (nine scenes), minimal specularities (15 scenes), metallic specularities (ten scenes), and at least one fluorescent surface in the scene (five scenes) [19]. Some of the typical objects used in the minimal specularity group are shown in Fig. 5 [19]. In this experiment each of the scenes was captured using a Sony DXC930 camera by illuminating with 11 different light sources. The spectra of the light sources used in the experiment are shown in Fig. 6. These light sources include typical indoor illuminants including fluorescent tube and daylight simulators. The data set we used has 39 different objects (see typical objects in Fig. 5) in total and each of the objects was taken under 11 different light sources. Accordingly, 11 different images of the same object and 429 images in total were used in the test. As described by Funt et al. [19], color histograms were used for testing the algorithm for color-based object recognition. Color/chromaticity histogram has been applied by several researchers including Swain and Ballard [23], Funt and Finlayson [24] and Funt et al. [19]. In the first recognition step, a data base of illuminant invariant histograms was formed using the extracted illuminant invariant features for the images in each group. In the test phase the image obtained from the test object was applied to the proposed algorithm and No of objects Proposed algorithm (%) Finlayson and Hordley [5] algorithm (%) White patch algorithm [11] Grey world algorithm [10] Minimal specularities Dielectric specularities Metallic Specularites Fluorescent surfaces two illuminant invariant features were obtained. Using these two features a histogram was formed and this histogram was intersected with the data base of histograms and the object that results in the highest correlation with the query histogram is declared as the best matching object. As Funt et al. [19] suggests that having a small bin size gives a good performance, a histogram was used in this test. To investigate the ability of the proposed algorithm in recognizing objects based on color the number of objects used in each attempt was increased in each step by five. We compare the performance of the proposed algorithm with well-known color constancy algorithms such as white patch [11], [12], gray world [10], and the algorithm proposed by Finlayson and Hordley [5]. Object recognition results of the proposed algorithm and Finlayson and Hordley s [5] algorithm, white patch algorithm and grey world algorithm are given in Table IV. It can be seen that a significantly better performance was obtained for the proposed algorithm compared to all the other algorithms in all cases. The reason for the degradation in performance of Finlayson

7 3618 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 8, AUGUST 2012 Relative sensitivity Wavelength (nm) Fig. 7. CIE standard RGB color matching function (10 ). TABLE V OPTIMUM CHANNEL COEFFICIENTS FOR ANY CONSUMER CAMERA CALCULATED USING THE CIE STANDARD RGB COLOR MATCHING FUNCTIONS Parameter Optimum value Alpha Beta Gamma Delta and Hordley s [5] algorithm is that this algorithm uses only a single feature. As discussed in Section III, some of the perceptually different colors are difficult to separate with only one feature. It can also be noticed that the proposed algorithm also performs well under both matt and specular reflection of light. B. Illuminant Invariant Skin Detection In this section, the 2-D space formed by the proposed algorithm is tested for illuminant invariant skin detection under daylight and indoor illuminant. In particular, we test the space for detection of skin of people from different ethnic backgrounds. The detection of different ethnic skin types is possible as typically the skin reflectance function for different races differs only by a scale factor. The performance of the 2-D space is compared with other color spaces widely used for skin detection including RGB, HSV, and YCbCr. As it has been said that the chroma component of a color does not vary significantly with illuminant variation [25], chroma components of these color spaces were used (CbCr, HS, and rg) for skin detection. In the YCbCr color space Cb and Cr are used to form a 2-D space for skin detection. The RGB space was normalized using the sum of all the three components (R+G+B) to remove any dependency on the intensity and scene geometry and then two of the normalized (r and g) components were used to form a 2-D space and this space was used for skin detection. The HSV color space uses the properties of a color to represent it and this space is quite similar to the way in which the human visual system perceives color. In this color space, the HS sub-space was used for skin detection because the coordinate axis V varies TABLE VI PERFORMANCE OF THE PROPOSED ALGORITHM WHEN TESTED WITH MUNSELL AND MEASURED FLORAL REFLECTANCES BY ILLUMINATING WITH CIE STANDARD ILLUMINANTS.IN THIS TEST THE PROPOSED ALGORITHM WAS TESTED USING DXC930 CAMERA RESPONSES WITH THE CHANNEL COEFFICIENTS LISTED IN TABLES I AND V Munsell, CIE daylight Floral, CIE daylight Test results when using the channel coefficients listed in Table V Pair wise distance in CIELab space Percentage separability (%) Test results when using the channel coefficients listed in Table I Pair wise distance in CIELab space Percentage separability (%) substantially compared to H and S components when changing the illuminant [25]. In the discussion so far, the DXC930 camera with known sensitivity functions has been used. However, it is difficult to measure the sensitivity functions of a camera. Therefore it would be interesting if we can adapt the proposed algorithm in such a way as to be able to use it with sensor responses generated by a consumer camera of unknown sensitivity functions. To find the channel coefficients of a camera of unknown sensitivity functions a new approach is proposed. Here, instead of using the sensitivity functions of a camera, the CIE standard RGB color matching functions (shown in Fig. 7) were used. This is because in imaging devices the camera dependent sensor responses are transformed to CIE RGB color space. These RGB responses are effectively the responses of the CIE standard RGB color matching functions. Therefore, the CIE standard RGB color matching functions were applied as the imaging sensors in finding the optimum channel coefficients to form the two illuminant independent features. The optimum channel coefficients were obtained as described in Section III by using the RGB color matching functions instead of the camera sensitivity functions. The optimum channel coefficients are listed in Table V. We investigated the variation in the performance of the proposed algorithm when using the generic channel coefficients (listed in Table V) instead of the camera specific coefficients (listed in Table I). Here we tested the algorithm with the generic channel coefficients using DXC930 camera sensor responses for separability of perceptually similar reflectances. The results of the algorithm when using the channel coefficients listed in Tables I and V are listed in Table VI. From these results one can see that, when tested with Munsell reflectance data the algorithm gives better performance with the coefficients listed in Table I. However, when tested using six-units floral reflectance data with channel coefficients listed in Table V. The reason could be that we have optimized the channel coefficients using Munsell reflectance data set and Munsell data set does not have saturated colors. However, the measured floral data set has saturated colors [26].

8 RATNASINGAM AND MCGINNITY: CHROMATICITY SPACE FOR ILLUMINANT INVARIANT RECOGNITION 3619 (a) (b) (c) Fig. 8. Skin recognition test results for the proposed algorithm with three consumer cameras (Sony DSC-T10, Samsung P1200, Canon 450D) under shodow, skylight, and direct sunlight. In this test, we have chosen the subjects in such a way that they are from different ethnic backgrounds. (a) Sony DSC-T10 camera. (i) Shadow. (ii) Skylight. (iii) Direct sunlight. (b) Samsung P1200 camera. (i) Shadow. (ii) Skylight. (iii) Direct sunlight. (c) Canon 450D camera. (i) Shadow. (ii) Skylight. (iii) Direct sunlight. TABLE VII SKIN RECOGNITION OF THE PROPOSED ALGORITHM WHEN APPLYING THE GENERIC CHANNEL COEFFICIENTS DERIVED USING CIE STANDARD COLOR MATCHING FUNCTIONS (LISTED IN TABLE V). TEST RESULTS FOR THE THREE CAMERAS USED IN THIS TEST ARE LISTED Camera True positive False positive Sony DSC-T10 Camera Samsung P1200 camera Canon 450D camera The important conclusion from the results listed in Table VI is that the performance of the algorithm does not vary significantly. This means that the proposed algorithm can be used with any consumer camera that has an approximately linear transform of the CIE standard RGB color matching functions. To investigate the proposed algorithm for skin detection under daylight using images captured with unknown camera sensitivity functions, three different consumer cameras (Samsung P1200, Sony DSC-T10, Canon 450D) were used to capture the scene. As the daylight spectrum varies with weather conditions, images were taken under different weather conditions. In this investigation, to test the ability of the proposed algorithm in detecting skin tone regardless of the ethnic background, subjects of different ethnic background were present in the scene. To define the skin boundary the images captured with the Samsung P1200 camera was used. From these images, skin regions were manually extracted and applied as input to the algorithm to obtain illuminant invariant features. These features were projected onto the 2-D space and the Mahalanobis distance boundary was drawn in such a way that approximately 90% of the skin pixels are enclosed by the boundary. In defining the skin region, 90% of the training skin pixels were taken to account for any noise in the manually extracted skin regions (training skin patches). Once the skin region was defined, test images were applied as input to the algorithm and the extracted features were projected onto the 2-D space; pixels which fell inside the defined boundary were colored in red. Similarly, for other color spaces, the training skin regions were used to define the skin region in

9 3620 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 8, AUGUST 2012 (a) (b) (c) (d) (e) Fig. 9. Skin detection results for outdoor lighting under different weather conditions. The images were captured using (i) Samsung P1200 camera, (ii) Sony DSC-T10 Camera, and (iii) Canon 450D camera, respectively. (a) Original images of people with different ethnic background. (i) Shadow (Samsung P1200). (ii) Skylight (Sony DSC-T10). (iii) Direct sunlight (Canon 450D). (b) Skin detection results for the proposed 2-D space. (i) Shadow. (ii) Skylight. (iii) Direct sunlight. (c) Skin detection results for the HS space. (i) Shadow. (ii) Skylight. (iii) Direct sunlight. (d) Skin detection results for the CbCr space. (i) Shadow. (ii) Skylight. (iii) Direct sunlight. (e) Skin detection results for the normalized rg space. (i) Shadow. (ii) Skylight. (iii) Direct sunlight. the 2-D spaces. For a fair comparison a Mahalanobis distance boundary was fitted enclosing approximately 90% of the skin pixels in all the three color spaces. In the test phase any pixels that fell inside the defined boundary were made red.

10 RATNASINGAM AND MCGINNITY: CHROMATICITY SPACE FOR ILLUMINANT INVARIANT RECOGNITION 3621 (a) - (b) (c) (d) Fig. 10. Typical skin detection results for different ethnic people under indoor illuminants. (a) Test results for the proposed 2-D space. (b) Test results for the HS space. (c) Test results for the CbCr space. (d) Test results for the rg space. Experimental results for the proposed 2-D space with the images captured by three different cameras are given in Fig. 8. In this test, we used the generic channel coefficients (listed in Table V) calculated using the CIE standard color matching functions. In particular, we have shown typical results for the three cameras under shadow, skylight, and direct sunlight. For quantitative comparisons we also have listed the true positive and false positive for each camera in Table VII. From these results it can be seen that Sony DSC-T10 Camera gives the best true positive and the Canon 450D camera gives the least false positive. Here, we should note that the performance difference might be from the differences in the sensitivity functions of the image sensors used in these cameras; also other factors including gain, white balancing or other post-processing applied in each of these cameras. Experimental results for the proposed 2-D feature space and the other color spaces (HS, CbCr, and rg) are shown in Fig. 9. A quantitative comparison of the performance of these spaces is given in Table VIII. From the results shown in Fig. 9 and Table VIII, it can be seen that the normalized rg space gives the worst performance. Comparing the spaces CbCr and HS, the HS space performs slightly better than the CbCr space. The proposed 2-D space gives the highest true positive and least false positive for skin detection under different illumination conditions. To investigate the performance of the proposed 2-D space in detecting skin under indoor illuminants the space was tested with images captured under indoor illuminants [27]. These images were acquired over three years under different lighting conditions using unknown consumer cameras. In the

11 3622 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 8, AUGUST 2012 TABLE VIII TEST RESULTS FOR SKIN DETECTION UNDER DIFFERENT OUTDOOR ILLUMINATION CONDITIONS.IN THIS TEST OUR PROPOSED FEATURE SPACE IS COMPARED WITH CbCr, rg, AND HS SPACES USING GENERIC CHANNEL COEFFICIENTS Camera True positive False positive Proposed space CbCr space rg space HS space TABLE IX TEST RESULTS FOR SKIN DETECTION UNDER DIFFERENT INDOOR ILLUMINATION CONDITIONS.IN THIS TEST OUR PROPOSED FEATURE SPACE IS COMPARED WITH CbCr, rg, AND HS SPACES USING GENERIC CHANNEL COEFFICIENTS Camera True positive False positive Proposed space CbCr space rg space HS space skin detection test we have used the generic parameters listed in Table V. As described above a Mahalanobis distance boundary was drawn for each space and all pixels that fell inside the defined boundary were made red. Test results of the propsed 2-D illuminant invariant feature space and the other color spaces with scenes illuminated with indoor illuminants are shown in Fig. 10. The results show the detection of people from different ethnic backgrounds including Asian, Caucasian, African, and Chinese. Comparing these results the CbCr space gives the least performance; while the rg space gives better performance compared to that of the HS space (see Table IX). However, the proposed 2-D space gives the best performance among all the spaces. Based on the results presented in this section and the previous sections, the proposed algorithm gives better performance in terms of illuminant invariant color-based object recognition and skin detection. This suggests that the proposed 2-D feature space can be used in machine vision applications including color-based recognition of objects that do not have a rigid geometry. VI. CONCLUSION An algorithm has been proposed for extracting two illuminant invariant features from three sensor responses. This algorithm extracts the chromaticity features at pixel level. An approach has been proposed to use the algorithm with cameras of unknown sensitivity functions. The algorithm was tested for separability of perceptually similar colors under CIE standard illuminants and obtained a good performance. The algorithm was also tested for color-based object recognition and compared with the existing algorithms. The test results showed that the proposed algorithm gives a better performance compared to the well-known color constancy algorithms in recognizing objects based on illuminant invariant reflectance features. Finally, the proposed algorithm was tested for skin detection invariant to illuminant, ethnic background and imaging device. In this experiment the scenes were taken when illuminated by daylight under different weather conditions and typical indoor lights. The results presented show that the proposed 2-D illuminant invariant chromaticity space can be used for machine vision applications including illuminant invariant color-based object recognition and skin detection. REFERENCES [1] J. von Kries, Beitrag zur physiologie der gesichtsempfinding, Arch. Anat. Physiol., vol. 2, pp , [2] M. Ebner, Colour Constancy (Imaging Science and Technology). New York: Wiley, [3] J. A. Marchant and C. M. Onyango, Shadow-invariant classification for scenes illuminated by daylight, J. Opt. Soc. Amer. A, vol. 17, no. 11, pp , [4] E. H. Land and J. J. McCann, Lightness and retinex theory, J. Opt. Soc. Amer., vol. 61, no. 1, pp. 1 11, [5] G. D. Finlayson and S. D. Hordley, Colour constancy at a pixel, J. Opt. Soc. Amer. A, vol. 18, no. 2, pp , [6] S. Ratnasingam and S. Collins, Study of the photodetector characteristics of a camera for colour constancy in natural scene, J. Opt. Soc. Amer. A, vol. 27, no. 2, pp , [7] G. D. Finlayson and M. S. Drew, 4-sensor camera calibration for image representation invariant to shading, shadows, lighting, and specularities, in Proc. Int. Conf. Comput. Vis., 2001, pp [8] S. Ratnasingam, S. Collins, and J. Hernández-András, Optimum sensors for color constancy in scenes illuminated by daylight, J. Opt. Soc. Amer. A, vol. 27, no. 10, pp , [9] D. Berwick and S. W. Lee, A chromaticity space for specularity, illumination color-and illumination pose-invariant 3-D object recognition, in Proc. Int. Conf. Comput. Vis., 1998, pp [10] G. Buchsbaum, A spatial processor model for object colour perception, J. Franklin Inst., vol. 310, no. 1, pp , [11] B. V. Funt, V. Cardei, and K. Barnard, Learning color constancy, in Proc. IST/SID 4th Color Imag. Conf., Scottsdale, AZ, 1996, pp [12] V. C. Cardei and B. V. Funt, Committee-based color constancy, in Proc. IST/SID 7th Color Imag. Conf.: Color Sci., Syst. Appl., Scottsdale, AZ, 1999, pp [13] G. D. Forsyth, A novel algorithm for color constancy, Int. J. Comput. Vis., vol. 5, no. 1, pp. 5 36, [14] D. Brainard and W. Freeman, Bayesian color constancy, J. Opt. Soc. Amer. A, vol. 14, no. 7, pp , [15] H. Trussell and M. Vrhel, Estimation of illumination for color correction, in Proc. Int. Conf. Acoust., Speech, Signal Process., 1991, pp [16] S. Ratnasingam and S. Collins, An algorithm to determine the chromaticity under non-uniform illuminant, in Proc. Int. Conf. Imag. Signal Process., vol Jul. 2008, pp [17] G. D. Finlayson, S. D. Hordley, and P. M. Hubel, Colour by correlation: A simple unifying theory of colour constancy, in Proc. IEEE Int. Conf. Comput. Vis., Sep. 1999, pp [18] S. Hordley, Scene illuminant estimation: Past, present, and future, Color Res. Appl., vol. 31, no. 4, pp , [19] B. V. Funt, K. Barnard, and L. Martin, Is colour constancy good enough? in Proc. 5th Eur. Conf. Comput. Vis., 1998, pp [20] D. H. Foster, Color constancy, Vis. Res., vol. 51, pp , [21] A. Abrardo, V. Cappellini, M. Cappellini, and A. Mecocci, Art-works colour calibration using the VASARI scanner, in Proc. IST SID s 4th Color Imag. Conf.: Color Sci., Syst. Appl., Scottsdale, AZ, 1996, pp [22] G. C. Holst and T. S. Lomheim, CMOS/CCD Sensors and Camera Systems. Bellingham, WA: SPIE, [23] M. Swain and D. Ballard, Colour indexing, Int. J. Comput. Vis., vol. 7, no. 1, pp , Nov [24] B. V. Funt and G. D. Finlayson, Color constant color indexing, IEEE Trans. Pattern Anal. Mach. Intell., vol. 17, no. 5, pp , May [25] J. Yang, W. Lu, and A. Waibel, Skin-color modeling and adaptation, in Proc. 3rd Asian Conf. Comput. Vis., 1998, pp [26] S. E. J. Arnold, V. Savolainen, and L. Chittka, FReD: The floral reflectance spectra database, Nature Preced., 2008, doi: /npre [27] Vision Group of Essex University Face Database [Online]. Available:

12 RATNASINGAM AND MCGINNITY: CHROMATICITY SPACE FOR ILLUMINANT INVARIANT RECOGNITION 3623 Sivalogeswaran Ratnasingam (M 12) received the B.Sc. Eng. degree in electrical and electronic engineering from the University of Peradeniya, Peradeniya, Sri Lanka, in 2003, the M.Sc. degree (distinction) in mobile and satellite communications from the University of Surrey, Guildford, U.K., in 2006, and the D.Phil. degree from the University of Oxford, Oxford, U.K., in He was with the Intelligent Systems Research Centre, University of Ulster, Londonderry, U.K., from 2010 to 2011, where he worked on topics including haptics, bio-inspired models, and robotics. Since 2011, he has been with National ICT Australia, Canberra, Australia, where he has been working on multi spectral image processing, illuminant estimation, and scene understanding. Dr. Ratnasingam was a recipient of the Nokia prize for best overall performance in M.Sc. in T. Martin McGinnity (S 09 M 83) received the Degree in physics (First Class Hons.) in 1975 and the Ph.D. degree from the University of Durham, Durham, U.K., in He is a Professor of intelligent systems engineering with the Faculty of Computing and Engineering, University of Ulster, Londonderry, U.K. He is currently the Director of the Intelligent Systems Research Centre, Londonderry, which encompasses the research activities of approximately 100 researchers. He was an Associate Dean with the Faculty and the Director of the university s technology transfer company, Innovation Ulster, Londonderry, and a spin-off company, Flex Language Services. He is the author or co-author of more than 260 research papers. His current research interests include computational intelligence, and in particular on computational systems, which explore and model biological signal processing, particularly in relation to cognitive robotics and computation neuroscience. Dr. McGinnity was a recipient of the Senior Distinguished Research Fellowship and the Distinguished Learning Support Fellowship in recognition of his contribution to teaching and research. He is a fellow of the Institution of Engineering and Technology and a Chartered Engineer.