Exploiting Spatial and Spectral Image Regularities for Color Constancy

Transcription

1 Explotng Spatal and Spectral Image Regulartes for Color Constancy Barun Sngh and Wllam T. Freeman MIT Computer Scence and Artfcal Intellgence Laboratory July 2, 23 Abstract We study the problem of color constancy nferng from an mage the spectrum of the llumnaton and the reflectance spectra of all the depcted surfaces. Ths estmaton problem s underdetermned: many surface and llumnaton spectra can be descrbed as a lnear combnaton of 3 bass functons, gvng 3 unnowns per pxel, plus 3 parameters for a global llumnaton. A trchromatc vsual system maes fewer measurements than there are unnowns. We address ths problem by wrtng the reflectance spectra of small groups of pxels as a lnear combnaton of spato-spectral bass functons. These aggregated surface reflectance spectra requre fewer parameters to descrbe than the sum of the spectral parameters for the ndvdual surface pxels, gvng us more measurements than unnown parmaeters. We explore ths problem n a Bayesan context, showng when the problem s over or underdetermned based on analyzng the local curvature characterstcs of the log-lelhood functon. We show that usng the spato-spectral bass functons gves mproved reflectance and llumnaton spectral estmates when appled to real mage data. 1 Introducton Color s mportant n our understandng of the vsual world and provdes an effectve cue for object detecton and recognton. In general, however, the observed color of an object dffers from ts true color due to factors such as lghtng and orentaton. Color constancy refers to the ablty to perceve the color of an object as approxmately constant regardless of the color of the lght fallng upon t [7]. We are able to relably use color for a varety of vson tass largely because of our ablty to perform color constancy well. Inherently, the color constancy problem s underdetermned. The mage formaton process conssts of an llumnaton reflectng off of a surface, then beng passed through a set of sensors. The nput to the sensors s the product of the llumnaton spectrum and the surface s reflectance 1

2 spectrum. Dfferent combnatons of llumnaton and reflectance spectra can produce the same spectrum mpngng on the sensor, and varous sensor nputs can result n the same sensor responses. Due to these ambgutes, t may be mpossble to unquely separate the effect of the llumnaton and the surface reflectances n an mage. The goal of computatonal color constancy [18, 14, 4, 15, 3, 17, 1, 11, 19, 2, 6], therefore, s to determne an optmal separaton of reflectance and llumnaton under some metrc. Gven the sensor responses, we see an estmate of the reflectance and llumnaton spectra. Ths would appear to nvolve estmatng very many numbers. If we specfy a 3 nm spectrum at 1 nm ntervals, even assumng a sngle llumnaton spectrum over the whole mage, we would need 3N + 1 numbers to specfy the reflectance spectra for N pxels plus one llumnant. In order to smplfy the color constancy problem, low-dmensonal lnear models are used to descrbe llumnaton and reflectance spectra [13, 5, 16, 12]. A suffcently large porton of the energy n these spectra typcally 99% - can be descrbed usng as low as three-dmensonal models for both surfaces and llumnants. An addtonal constrant used by some approaches s that surfaces and llumnants must be physcally realzable, e.g. ther spectra cannot be negatve and surface reflectances must be less than 1 [1, 2, 6]. Even after these smplfcatons, the color constancy problem remans underdetermned and exstng approaches must mae further assumptons n order to obtan a soluton. Buchsbaum s Gray World algorthm assumes that the mean reflectance of all mages s the same, and the llumnaton s estmated usng ths mean [4]. Maloney and Wandell s Subspace method requres that a two-dmensonal model descrbe surface reflectances for the case of trchromatc sensors [17]. Gamut mappng methods [9, 8] explot the observed color gamut to estmate the llumnant spectrum, and can be combned wth realzablty constrants. Snce a Bayesan approach [2] can ncorporate the varous assumptons above nto ts pror probabltes, Bayesan decson theory [1] provdes a prncpled framewor whch we wll use for studyng the color constancy problem. A parameter countng argument reveals the underlyng problem we address n ths wor: f we use a 3-dmensonal model for the surface reflectance at each pxel, and a 3-dmensonal model for the llumnaton spectrum, for the case of trchromatc color sensors, we have more unnowns than observatons. We measure 3 numbers at each poston, yet have to estmate 3 numbers at each poston, plus the 3 numbers descrbng the llumnant. The problem s always underdetermned, requrng, n a Bayesan soluton, more relance on the pror probabltes or loss functon than on the lelhood functon to estmate the best answer. One would prefer the stuaton where the data lelhood functon domnated the other terms. In ths paper, we mae the problem overdetermned by ntroducng lnear bass functons that descrbe both spectral and spatal varatons n the surface reflectances. These spato-spectral bass functons explot mage regulartes whch allow us to specfy surfaces reflectances usng fewer numbers, yeldng more mage measurements than unnown parame- 2

3 ters. In the next sectons, we ntroduce the notaton and lnear bass functon analyss, then explore the Bayesan approach. We show wth smple examples that the structure of the posteror ndcates condtons under whch color constancy s stll undetermned by the data. We then apply our approach to real mages, verfyng usng hyperspectral mage data that our surface reflectance estmates mprove through usng the spato-spectral bass functons. 2 Lnear models We assume there s one global llumnaton n each scene and that the lght spectrum leavng each surface s the term-by-term product of the llumnaton and local reflectance spectrum. For a surface reflectance Sλ and llumnaton Eλ, the response at poston x of a photoreceptor wth a spectral response of R λ s: y x = λ R λeλs x λ, 1 The llumnant and surface spectra can be wrtten as lnear combnatons of the llumnaton bass functons E λ and reflectance bass functons S j λ, wth coeffcents e and s x j, respectvely, at poston x: Eλ = S x λ = L E λe, 2 =1 L S j λs x j, 3 j=1 where L s defned as the number of elements n the λ vector.e., the number of wavelength samples. Dong so allows us to wrte the renderng equaton as y x = λ L R λ E λe L =1 j=1 S j λs x j. 4 Summng over λ, we get a blnear form, L y x = L e G j, s x j, 5 =1 j=1 where G j, = λ R λe λs j λ. In general, we wsh to approxmate the true llumnant Eλ and surface spectra S x λ 3

4 usng lower dmensonal lnear models. Thus, we can defne Ẽλ = S x λ = d E =1 d S j=1 E λe, 6 S j λs x j, 7 where d E s the dmensonalty of the llumnant approxmaton Ẽλ and d S s the dmensonalty of the surface approxmaton S x λ. In addton, we can decompose 5 as: y x = d E = d S =1 j=1 d E d S =1 j=1 e G j, s x j + d E e G j, s x j + w x, L =1 j=d S +1 e G j, s x j + L d S =d E +1 j=1 e G j, s x j + L L =d E +1 j=d S +1 e G j, s x j where w x s a nose term representng the error due to the use of lower dmensonal models. A summary of the notaton used n ths paper s provded n Table 1. Equaton 8 s perhaps best understood by vewng color constancy as an estmaton problem. The goal s to estmate S x λ and Ẽλ gven yx. Thus, we vew the true llumnants as havng d E degrees of freedom as defned by the frst d E llumnant bass functons, wth remanng bass functons contrbutng n the form of nose. Analogously for surface reflectance, the frst d S bass functons defne the space of true reflectances and the remanng bass functons add nose. Note that whle the llumnaton nose and reflectance nose are orthogonal to the true llumnaton and reflectance, the projectons of the true data and the nose nto the sensor response space are not orthogonal. 8 3 Bayesan Approach to Color Constancy The Bayesan approach to color constancy utlzes a probablstc framewor to examne the problem. Ths framewor s based on three fundamental probablty dstrbutons - the pror, the lelhood, and the posteror. The pror descrbes the probablty that a certan set of parameters s the correct estmate wthout nowng anythng about the data. Ths dstrbuton s wrtten as P e, s, where e and s are the global llumnaton weghts and the surface weghts for all postons n an mage. The lelhood, gven by P y e, s, governs the relatonshp between the parameters beng estmated e and s and the observed data n ths case the set of all sensor responses n an mage, y. The posteror descrbes what s nown about the parameters beng 4

5 Symbol Symbol Defnton Symbol Symbol Defnton N number of pxels n an mage m number of pxels along one dmenson of a square bloc beng consdered wthn an mage λ wavelength γ ndex nto spato-spectral bass functons L number of wavelength samples length of λ vector K number of sensor responses avalable at any pxel Eλ spectral radance of llumnant E {} γ spato-spectral radance of llumnaton n m m bloc E λ th bass functon of llumnaton Ê γ th spato-spectral llumnaton bass functon e scalar weght of th llumnaton bass functon e row vector of all e e {} row vector of all e {} e {} d E scalar weght of th spato-spectral llumnaton bass functon d E dmensonalty of llumnaton representaton dmensonalty of spato-spectral llumnaton representaton Ẽλ approxmaton to llumnant from d E dmensonal Ẽ {}γ approxmaton to spato-spectral llum- lnear model nant from d E dmensonal lnear model S x λ surface reflectance at pont x S {x} γ spato-surface reflectance for pxel bloc {x} S j λ jth bass functon of surface reflectance Ŝ j γ jth spato-spectral surface reflectance bass functon s x j scalar weght of jth surface reflectance bass functon at poston x s {x} j weght of jth spato-spectral surface reflectance bass functon for pxel bloc {x} s x column vector of s x j for all j s{x} column vector of s {x} j for all j s matrx whose columns are gven by s x s {} matrx whose columns are gven by s {x} d S dmensonalty of surface reflectance representaton dmensonalty of spato-spectral surface reflectance representaton Sλ approxmaton to surface reflectance S {}γ approxmaton to spato-spectral surface from d S dmensonal lnear model reflectance from d S dmensonal lnear model R λ spectral senstvty of th sensor R γ spato-spectral senstvty of th sensor y x scalar response of th sensor at x y {x} d S scalar response of th sensor for pxel bloc x y x vector of y x for all y{x} vector of y {x} for all y matrx of y x vectors for all x y {} matrx of y {x} vectors for all x G j, 3-d tensor relatng llumnaton and reflectance weghts to sensor responses Ĝ j, Table 1: Symbol Notaton 3-d tensor relatng spato-spectral llumnaton and reflectance weghts to spatospectral sensor responses 5

6 estmated after havng observed the data and can be calculated usng Bayes rule as P e, s y = 1 P y e, sp e, s, 9 Z where the normalzaton constant 1 Z s ndependent of the parameters to be estmated. In addton to these probabltes, the Bayesan formulaton requres that we specfy a cost functon that defnes the penalty assocated wth a parameter estmate of ẽ, s when the true parameter values are e, s. For estmatng the relatve surface and llumnaton spectra, ndependent of any overall scale factor, we fnd that the loss functon has lttle effect [2], so we use MAP estmaton and maxmze the posteror probablty. The underdetermned nature of the general color constancy problem mples that there should be an nfnte set of solutons whch maxmze the posteror.to see that ths s the case, we must examne the propertes of equaton 9. Assumng ndependent and dentcally dstrbuted Gaussan observaton nose wth varance σ2 n the photoreceptor responses at each patch, the posteror probablty s: P e, s y = 1 e yx,j e G j, s x j 2/2σ2 P e, s. 1 Z x, If the posteror s flat along any dmenson n ths regon, there s a famly of possble solutons to the mnmzaton problem, whereas f the posteror s not flat along any dmensons, a fnte number of local mnma must exst. Furthermore, f we assume that the pror probablty P e, s vares slowly over the regon beng consdered, the structure of the posteror can be seen by examnng the local structure of the log lelhood functon, Ly e, s = x, y x,j e G j, s x j Local Curvature Analyss We explore how the structure of the posteror probablty reveals whether or not the color constancy problem s solvable. The local structure we are nterested n s the curvature of the log lelhood functon, whch tells us how movng n the parameter space affects our ablty to explan the observed data. We wll analyze how ths structure changes when we ntroduce low-level spatal nformaton nto the model, and use ths analyss to motvate the use of such nformaton for color constancy. The curvature of the log lelhood can be obtaned from an egen decomposton of ts Hessan matrx. The Hessan s a matrx of all second order dervatves and thus, for an N- pxel mage, s a d E + Nd S d E + Nd S matrx. The egenvectors of the Hessan matrx gve prncpal drectons of curvature and the correspondng egenvalues gve the extent of curvature n each of these drectons. 6

7 At a maxmum of the log lelhood, all egenvalues of the Hessan must be non-negatve meanng that error cannot decrease as we move away from the local maxmum. If there are any egenvalues equal to zero, ths mples that there are drectons of zero curvature - e.g., there are ways we can vary the estmated parameters wthout affectng the lelhood of the data. For a unque maxmum, all egenvalues of Hessan must be postve at the maxmum. Furthermore, the larger these egenvalues are, the more robust s ths soluton to observaton nose or unmodelled mage varatons. To help us understand the structure of the color constancy problem for real-world mages, we frst examne representatve specal cases n toy examples. Appendx B gves a full dervaton of the Hessan for the log lelhood functon gven by Underdetermned Case We constructed a toy problem n whch the sensor responses for all ponts n an mage are generated usng equaton 8 wth the nose term set to zero. To perform smulatons, we used a hyperspectral data set of 28 natural scenes collected at Brstol Unversty by Brelstaff, et. al. as descrbed n []. Each of these mages s pxels n sze and contans sensor responses to 31 spectral bands, rangng from 4 nm to 7 nm n 1 nm ntervals. Each scene also contans a Koda greycard at a nown locaton wth a constant reflectance spectrum of nown ntensty. The scene llumnant s approxmated as the spectrum recorded at the locaton of the greycard dvded by ts constant reflectance. Our dataset therefore conssts of 256x256x28 reflectance spectra and 28 llumnaton spectra. The frst three llumnaton and reflectance bass functons obtaned by applyng prncpal components analyss PCA to ths data are plotted n Fgure 1a and b, respectvely. We assume, wthout loss of generalty, a Gaussan model for sensor responses centered at 65 red, 55 green, and 45 nm blue as shown n Fgure 1c. Generatng 1 ponts usng random llumnaton and reflectance weghts and evaluatng the Hessan at the correct locaton n parameter space results n the egenvalue spectrum shown n Fgure 2. We can see that, as expected, there are 3 drectons of zero curvature. The three assocated egenvectors gve the drectons of zero curvature n parameter space. Any lnear combnaton of these egenvectors s also a drecton of zero curvature. It s nterestng to note that there are 3 llumnaton weghts and also three egenvectors wth correspondng zero egenvalues. Each of these egenvectors s of dmenson d E + Nd S, wth three of those dmensons correspondng to the llumnant weghts. The three egenvectors span the space of the 3-dmensonal llumnant space, meanng that for any set of llumnaton coeffcents, there exsts some set of reflectance weghts such that the lelhood assocated wth that set of parameters s equal to the lelhood assocated wth the correct set of parameters. In other words, ths means that we can move n any drecton n the 3-dmensonal space of the llumnant and stll fnd surface reflectance values that render to the observed sensor val- 7

8 ntensty arbtrary scale ntensty arbtrary scale wavelength, λ nm a Illumnaton Bass Functons wavelength, λ nm b Reflectance Bass Functons 5 sensor response wavelength, λnm c Sensor Responses Fgure 1: Bass functons and sensor responses used n toy examples. Bass functons are obtaned from a hyperspectral dataset. 8

9 ues. One example of ths s a multplcatve scalng up of the llumnaton and a correspondng scalng down of the surface reflectances. log of egenvalues curves wth slope = a Egenvalue spectrum b Graphc depcton Fgure 2: Egenvalue spectrum and correspondng curvature of the log lelhood for a toy example of the color constancy problem. The drectons of zero curvature and egenvalues at zero show that the problem s underdetermned Suffcent Sensors If there are 4 ndependent photoreceptor responses avalable nstead of 3, we would expect for a better behaved log-lelhood functon. However, there should stll be one drecton of zero curvature n the Hessan, correspondng to a smple scalng of the llumnaton and surfaces. The bass functons and sensor responses gven n Fgure 1 are used for ths example as well as a fourth sensor response whch s chosen to be random and thus ndependent of the other three. We agan generate 1 random ponts and fnd that the Hessan behaves as expected, as shown n Fgure 3. Not only does the Hessan have only one egenvalue equal to zero, the other egenvalues have a much larger value than the egenvalues of the Hessan n the underdetermned case. log of egenvalues curve wth slope = a Egenvalue spectrum b Graphc depcton Fgure 3: Egenvalue spectrum and correspondng curvature of the log lelhood for a case when the number of sensors exceeds the dmensonalty of the surface model. There s now only one drecton of zero curvature, correspondng to an unnown scale factor. 9

10 3.2 Bayesan Color Constancy n the Underdetermned Case Although there may be multple degrees of ambguty n the lelhood functon, n general, the Bayesan approach s stll able to produce a unque parameter estmate, because the pror or the loss functon wll favor one soluton over the others [2]. In the deal case, however, we would le for our parameter estmate to be domnated by the effect of the lelhood functon. That s, we would le for the estmate to be based on the data tself, not on pror assumptons about the data or how much we value partcular parts of the data. 4 Spato-spectral Bass Functons We now ntroduce an alternatve formulaton of the fnte-dmensonal models presented n Secton 2 that model both spatal and spectral propertes of groups of pxels n natural mages. The hypothess s that by tang characterstc spatal varatons of natural mages nto account, the number of parameters necessary to descrbe the reflectance of an mage wll be reduced so that the problem s no longer underdetermned. Certan physcal phenomena, such as nterreflectons, may generate characterstc spatal and spectral sgnatures whch allow estmaton of the desred spectra from mage data. Natural mages, le folage, may exhbt characterstc spatal changes n color. The spato-spectral bass functons allow us to explot these, or other, regulartes n the vsual world n order to solve the color constancy problem. We expect ths approach wll wor best n rchly textured mages, and worst n flat, color Mondran-type mages [18]. 4.1 Modfed Lnear Models Instead of usng lnear models to descrbe the reflectance of ndvdual pxels, we wll now use these models to descrbe groups of pxels. Wthout loss of generalty, we can group pxels nto m m blocs and use the same basc formulaton as was developed n Secton 2. In order to do so, t s necessary to convert blocs of pxels nto vector format. We do ths by rasterzng the pxels wthn the bloc. The reflectance of a bloc of pxels s defned as a vector of length m 2 L consstng of the reflectance of each pxel wthn the bloc staced on top of each other n raster order. The same process s used to descrbe the sensor response of the bloc as a vector of length m 2 K and the llumnaton as a vector of length m 2 L. The bass functons used to model the reflectance of blocs of pxels are now referred to as spato-spectral reflectance bass functons, snce they descrbe both the spectral and spatal characterstcs of a bloc of pxels. We shall denote a group of pxels by {x}, so that the llumnaton and reflectance of a bloc 1

11 of pxels s gven by: E {} γ = S {x} γ = m 2 L =1 m 2 L j=1 Ê γe {}, 12 Ŝ j γs {x} j, 13 where Êγ and Ŝjγ are the spato-spectral llumnaton and reflectance bass functons, e {} and s {x} j are the weghts assocated wth these bass functons, E {} γ s the llumnaton of all blocs n the mage, and S {x} γ s the reflectance of the bloc of pxels {x}. Note that the elements of the scene are now wrtten as a functon of γ rather than λ. Ths s due to the fact that the spato-spectral representaton contans nformaton about both the frequency and spatal characterstcs of the scene. Approxmatng these models wth fewer dmensons, we can defne Ẽ {} γ = S {x} γ = d E =1 d S j=1 Ê γe {}, 14 Ŝ j γs {x} j, 15 where Ẽ{} γ s the approxmate llumnaton of all blocs n an mage, constructed usng a d E dmensonal model, and S {x} γ s the approxmated reflectance for the bloc {x}, constructed usng a d S dmensonal model. We defne an m m bloc of pxels as havng m 2 K sensor outputs, wth K sensor outputs per pxel. Thus, we defne the sensor responses of the group of pxels as the bloc dagonal matrx wth m2 blocs along the dagonal, and Rλ... Rλ... Rγ = Rλ 16 R = [R 1 γ R 2 γ... R γ]. We let R γ refer to the th column of the matrx Rγ. Followng a dervaton analogous to that presented n Secton 2, we can wrte the sensor 11

12 output n a blnear form wth nose as: where y {x} y {x} = d E d S =1 j=1 e {} Ĝ j, s {x} j + w {x}, 17 s the sensor response of the bloc of pxels {x}, Ĝj, s defned as Ĝ j, = γ R γêγŝjγ, 18 and w {x} s the nose ntroduced from usng a lower-dmensonal lnear model. 4.2 The Advantage of Usng Spatal Informaton The advantage of ncorporatng spatal nformaton nto the lnear models used to descrbe scenes s that t provdes more descrptve power usng a fewer number of parameters. It s straghtforward to show that n even n the worst case,.e. when ncorporatng spatal nformaton provdes no beneft and ndvdual pxels are ndependent, the spato-spectral method cannot do worse than the tradtonal method. The amount of mprovement n descrptve power provded by the new method can be tested by fndng a lnear reflectance bass usng real mages for blocs of varous szes ncludng m = 1, whch corresponds to the tradtonal case of examnng each pxel ndependently and calculatng how much of the varance of a set of test mages s descrbed n each case. To do ths, we use the dataset of hyperspectral mages descrbed earler n Secton The amount of varance descrbed by lnear models of varyng dmensonalty when consderng bloc szes of m =1, 2, 3, and 4 s shown n Fgure 4. The fgure plots the number of bass functons per pxel the number of bass functons used to descrbe the m m bloc dvded by m2 aganst the amount of varance descrbed when usng that many bass functons. The lnear model was obtaned usng approxmately half of the data n the 28 scenes selected randomly and tested on the entre data set. It can be seen that usng a bloc sze of m = 2 dramatcally mproves performance. For example, the data shows that the error rate obtaned when descrbng each pxel wth 3 bass functons can be acheved by usng only 1.5 bass functons per pxel when descrbng an mage wth 2 2 blocs.e., 6 bass functons for the bloc. Increasng the bloc sze beyond m = 2 does not show much of an mprovement n reconstructon error. Ths s because the majorty of the advantage of usng blocs s to tae advantage of local flatness n an mage, whch s already done when 2 2 blocs are used. Fgure 5 plots the frst 12 spato-spectral reflectance bass functons for m = 2. These bass functons were extracted from the hyperspectral data usng the calbraton gray card n each mage of the dataset, n the same manner as descrbed n Secton It can be seen that the frst three bass functons are essentally the same as 12

13 square error of reconstructon pxel bloc 2 2 pxel bloc 3 3, 4 4 pxel blocs number of coeffcents per pxel n bloc Fgure 4: The squared reconstructon error usng lnear models of varyng dmensons for bloc szes of m =1, 2, 3, and 4. Usng bloc szes of m = 2 or larger greatly reduces the number of coeffcents needed to descrbe each pxel. the standard sngle pxel bases repeated at all 4 pxels. To see the advantage of usng spatal nformaton n the context of Bayesan color constancy, we must once agan analyze the local curvature of the posteror probablty. Followng the dervatons n Secton 3, we once agan mae the assumpton of a locally flat pror dstrbuton, allowng us to smplfy the problem and examne the log lelhood functon, now wrtten as Ly {} e {}, s {} = {x}, y {x},j 5 Experments wth Natural Images e {} 2. Ĝ j, s {x} j 19 We loo at groups of 4 pxels, grouped nto 1 sets of 2x2 pxel blocs. The 1 sets were selected from randomzed postons n the mage. In order to get the prors, we project our dataset of reflectances and llumnatons onto the bass functons gven n Fgures 1a for llumnaton and 5 for spato-spectral reflectance. We ft truncated Gaussans to the data and use these as our prors. Our goal s now to fnd the set of llumnaton and reflectance coeffcents that mnmze the posteror probablty. In order to do ths, we need to search over the space of all possble llumnaton and surface reflectance coeffcents. Snce the number of surface coeffcents scales wth the number of pxels n the mage, ths qucly becomes an ntractble problem. Followng [2], n order to mae the problem more feasble, we lmt our search to loo over the space of llumnaton coeffcents and solve for the surfaces at each teraton usng a determnstc relatonshp. In the underdetermned case, where the number of sensor responses at each pxel s equal to the number of surface coeffcents at each pxel, we solve for sur- 13

14 .6 poston 1.6 poston 2.6 poston 1.6 poston poston 3.6 poston 4.4 poston 3.6 poston a bass functons 1, 2, and 3 b bass functons 4, 5, and 6.4 poston 1.6 poston 2.4 poston 1.4 poston poston 3.6 poston 4.4 poston 3.4 poston c bass functons 7, 8, and 9 d bass functons 1, 11, and 12 Fgure 5: The frst 12 spato-spectral bass functons for 2x2 pxel blocs. Each bass functon s plotted for wavelength values from 4 to 7 nm on the x-axs. The frst 3 spato-spectral bass functons, a, show only spectral varatons and no spatal varaton. The next 3 bass functons, b, correspond to spatal dervatves. The fnal 6 ndcate a more complex relatonshp of spatal and spectral varatons, c and d. 14

15 faces that produce the sensor responses exactly gven an llumnaton ths s done by solvng equaton 8 assumng no sensor nose through a smple matrx nverson. Ths s equvalent to constranng our search to locatons of maxmum probablty n the lelhood. In the case where we have more sensor responses than surface coeffcents, we tae the pseudo-nverse to fnd the surface coeffcents correspondng to a gven llumnaton. Unfortunately, applyng ths technque of constranng the search space to real mages gves numercal problems. In these mages, the nose term n equaton 8, whch corresponds to the surface and llumnaton components not ncluded n a low-dmensonal model, causes the surface coeffcents that maxmze the lelhood term gven the true llumnant to be very dfferent from the true surfaces coeffcents. Ths maes t hard to properly analyze the posteror probablty correspondng to a gven lllumnant. To avod the numercal ssues mentoned above, we test our algorthm on mages constructed by frst projectng the true surface reflectances down to a number of dmensons equal to the number of sensor responses 3 for 1 pxel at a tme, 12 for 2x2 blocs, multplyng the reflectance mage by the true llumnant, and passng the result through the Gaussan sensors shown n Fgure 1. Fgure? shows that there s very lttle dfference between the mage obtaned usng the true reflectance and the mage obtaned after projectng the reflectance onto the lower-dmensonal subspace. We need to be clear: these are pre-processed mages, not real mages, although they are tantalzngly close to real mages. We show the benefts of usng spato-spectral bass functons n solvng color constancy usng the pre-processed mages, and expect that extensons to real mages wll follow n wor by us or by others. a Sample mage from dataset b Image after projecton Fgure 6: The effect of projectng reflectance to lower dmensons s very small. Fgure 7 shows results for our approach usng spato-spectral bass functons n a Bayesan approach wth the standard Bayesan approach and the Gray World approach. The partcular llumnant we wsh to estmate s not ft well by the pror, as seen n the fgure. Each algorthm s run on 15 random groups of 4 pxels chosen from the mage. Despte the low pror probablty of the llumnant, the Bayesan approach usng spato-spectral bass functons allows 15

16 us to locate the correct llumnant for each draw. Ths llustrates that the lelhood term does domnates our estmate, as we would expect. The standard Bayesan approach, examnng one pxel at a tme, and the Gray World approach both result n ncorrect estmates..3.3 ntensty arbtrary scale ntensty arbtrary scale wavelength λ a usng spato-spectral bass functons 4 7 wavelength λ b standard Bayesan approach.35.3 ntensty arbtrary scale wavelength λ c Gray World algorthm Fgure 7: Results of color constancy appled to a natural mage. Each algorthm was run 15 tmes, wth results shown n dotted lnes. The mean llmnaton from the pror s gven n the dashed lne as reference, and the true llumnant s mared by crcles. In our experments, we also found that the postvty constrant restrcted the set of possble llumnants more when usng 2x2 blocs than when consderng 1 pxel at a tme, so that the results for 2x2 blocs usng 12 coeffcents underdetermned were better than the results for the standard underdetermned case. 6 Conclusons We have studed the effect of usng spato-spectral surface reflectance bass functons to fnd a low-dmensonal estmate of surface reflectance characterstcs. These bass functons can convert the color constancy problem from an under-determned problem to an over-determned one. We can see ths n the curvature characterstcs of the 16

17 log-llhood functon at parameter local maxma. For natural mages, pre-processed to restrct the surface reflectance varatons to be explaned by an average of 3 bass functons per pxel, we fnd that usng spato-spectral bass functons allows for very accurate estmates of the global llumnaton spectrum and therefore of surface reflectance spectra, too. For the same mages, usng purely spectral bass functons results n an underdetermned problem, wth much hgher estmaton error for the optmal Bayesan estmate. Acnowledgments We than Davd Branard for many helpful conversatons. APPENDIX: The Hessan of the Log Lelhood Functons wth Blnear Forms H el e m = 2 L e l e m L = y x e l e e G j, s x j 2 l,j = x, x, = 2 x, = 2 x, e l y x,j y x,j y x,j e G j, s x j 2 e G j, s x j e l e G j, s x j j,j G lj, s x j e G j, s x j A-1 A-2 A-3 A-4 2L e l e m = 2 e m = 2 x, = 2 x, x, y x,j y x e m,j j G lj, s x j j e G j, s x j j e G j, s x j j G mj, s x j G lj, s x j G lj, s x j A-5 A-6 A-7 17

18 H s n l s p m = 2 L s n l sp m L s n l = s n l = s n l = s n l = 2 = 2 y x e G j, s x j 2 x,,j y x e G j, s x j 2 + y x,j x=n,j x n y n e G j, s n j 2,j y n e G j, s n j s n,j l y n,j e G j, s n j,j e G l, e G j, s n j A-8 e G j, s x j 2 A-9 A-1 A-11 A-12 2 L s n l sp m 2 L s n l sn m = A-13 = s n m = 2 = 2 2 y n,j s n m,j e G m, e G j, s n j e G j, s n j e G l, e G l, e G l, A-14 A-15 A-16 H el s p m = 2 L e l s p m 2L e l s p m = 2 y p e e G j, s p j e G m, A-17 l,j = 2 y p e e G j, s p j e G m, + l,j y p e G j, s p j e G m, A-18 e,j l = 2 G lj, s p j e G m, y p e G j, s p j G lm, A-19 j,j References [1] J. O. Berger. Statstcal decson theory and Bayesan analyss. Sprnger, [2] D. H. Branard and W. T. Freeman. Bayesan color constancy. Journal of the Optcal Socety of Amerca A, 147: , [3] D H Branard and B A Wandell. Analyss of the retnex theory of color vson. Journal of the Optcal Socety of Amerca A, 31: ,

19 [4] G. Buchsbaum. A spatal processor model for object colour percepton. J. Franln Inst., 31:1 26, 198. [5] CIE. Colormetry. Bureau Central de la CIE, [6] M. D Zmura, G. Iverson, and B. Snger. Probablstc color constancy. In R. D. Luce et. al., edtor, Geometrc representatons of perceptual phenomena: papers n honor of Tarow Indow s 7th brthday. Lawrence Erlbaum, Hllsdale, NJ. n press. [7] R. M. Evans. The percepton of color. Wley, New Yor, [8] G. Fnlayson and S. Hordley. Improvng gamut mappng color constancy. Image and Vson Computng, 9: , 2. [9] D. Forsyth. A novel approach to color constancy. Internatonal Journal of Computer Vson, 5:5 36, 199. [1] D. A. Forsyth. A novel approach to colour constancy. In Proc. 1st Intl. Conf. Computer Vson, pages IEEE, [11] R. Gershon and A. D. Jepson. The computaton of color constant descrptors n chromatc mages. Color Res. Appl., 14: , [12] T. Jaaselanen, J. Parnen, and S. Toyooa. A vector-subspace model for color representaton. Journal of the Optcal Socety of Amerca A, 7:725 73, 199. [13] D. B. Judd, D. L. MacAdam, and G. W. Wyszec. Spectral dstrbuton of typcal daylght as a functon of correlated color temperature. J. Opt. Soc. Am., page 131, [14] E H Land. The retnex theory of color vson. Scentfc Amercan, 2376:18 128, [15] E. H. Land. Recent advances n retnex theory and some mplcatons for cortcal computatons: color vson and the natural mage. Proc. Nat. Acad. Sc. USA, 8: , [16] L. T. Maloney. Evaluaton of lnear models of surface spectral reflectance wth small numbers of parameters. Journal of the Optcal Socety of Amerca A, 31: , [17] L T Maloney and B A Wandell. Color constancy: a method for recoverng surface spectral reflectance. Journal of the Optcal Socety of Amerca A, 31:29 33, [18] J. J. McCann, S. P. McKee, and T. H. Taylor. Quanttatve studes n retnex theory: a comparson between theoretcal predctons and observer responses to the color mondran experments. Vson Research, 16: , [19] H. J. Trussell and M. J. Vrhel. Estmaton of llumnaton for color correcton. In Proc. ICASSP, pages ,