Bootstrapping Color Constancy

Transcription

1 Bootstrappng Color Constancy Bran Funt and Vlad C. Carde * Smon Fraser Unversty Vancouver, Canada ABSTRACT Bootstrappng provdes a novel approach to tranng a neural network to estmate the chromatcty of the llumnant n a scene gven mage data alone. For ntal tranng, the network requres feedback about the accuracy of the network s current results. In the case of a network for color constancy, ths feedback s the chromatcty of the ncdent scene llumnaton. In the past 1, perfect feedback has been used, but n the bootstrappng method feedback wth a consderable degree of random error can be used to tran the network nstead. In partcular, the grayworld algorthm 2, whch only provdes modest color constancy performance, s used to tran a neural network whch n the end performs better than the grayworld algorthm used to tran t. Keywords: color constancy, neural networks, color correcton 1. ADDRESSING THE COLOR CONSTANCY PROBLEM 1.1. Why do we need color constancy? The colors of surfaces n an mage s determned n part by the color of the lght source under whch that mage was taken. Thus, varatons of the color of the llumnaton n a scene produce color shfts of the surfaces n that scene. Wthout color stablty, most problems where color s taken nto account (e.g. color based object recognton 3 systems and dgtal photography) wll be adversely affected even by small changes n the scene s llumnaton. For a human observer, however, the perceved color shfts due to changes n llumnaton are relatvely small; n other words, humans exhbt a relatvely hgh degree of color constancy A computatonal approach From a computatonal perspectve, we would lke to compensate for the effect that varatons n the color of the ncdent llumnaton have on the colors n an mage and create an mage wth the same colors as would be obtaned for the same scene under a standard, canoncal llumnant. In ths paper, we wll assume that the chromatcty of the scene llumnaton s constant spatally, although ts brghtness mght vary across the mage. The goal of a machne color constancy system wll be taken to be the accurate estmaton of the chromatcty of the scene llumnaton from a 3-band dgtal color mage of the scene. After estmatng the llumnant s chromatcty, the scene can then be color corrected 5 based on a dagonal, or coeffcent-rule transformaton. In order to acheve color constancy, we proposed a mult-layer neural network 1. Based only on the chromatcty hstogram of the nput mage, the neural network computes an estmate of the scene s llumnaton. The network was traned on scenes that were syntheszed from databases of surface reflectances and llumnant spectral power dstrbutons. In addton to ths data, the camera sensor senstvty functons were also known. For each syntheszed scene n the tranng set, the network was traned by feedng the bnarzed hstogram of the scene s chromatctes to ts nput layer. The network outputs an estmate of the chromatcty of the scene llumnaton whch s compared to the actual llumnaton used to generate that scene and the network s nternal weghts are then adjusted usng the backpropagaton method 6. The network can also be traned usng data from real mages nstead of syntheszed ones. Both methods have dsadvantages. To synthesze scenes, the spectral senstvty functons of the camera must be known and other artfacts 7 (e.g. nose, speculartes, camera non-lnearty, flare, etc.) that often appear n real mages must also be * Correspondence: Bran Funt. Other author nformaton: Emal: {funt, vcarde}@cs.sfu.ca; School of Computng Scence, Smon Fraser Unversty, 8888 Unversty Drve, Burnaby, B.C., Canada, V5A 1S6.

2 taken nto account. On the other hand, usng real data requres a large set of mages for whch the actual llumnant has been measured. The bootstrappng approach that wll be presented n ths paper helps address both problems. 2. BOOTSTRAPPING THE NEURAL NETWORK TRAINING ALGORITHM 2.1. The bootstrappng algorthm Consder a set of mages taken under unknown llumnants wth an uncalbrated camera (.e. a camera wth unknown sensor senstvtes). The only assumpton we make, and whch s easy to confrm gven the actual data, s that the mages have a relatvely large number of colors. The llumnaton chromatcty s estmated usng the grayworld algorthm (GW) descrbed below, whch assumes that the average color of the scene s gray and that any departure from ths average n the mage s caused by the color of the llumnant. We generate a large tranng data set by sub-samplng the gven set of mages, whch s then used for tranng a neural network. Instead of the actual llumnant, the network receves the grayworld algorthm s estmate of the llumnaton chromatcty. The same grayworld estmate s used for all sub-samplngs of an mage. Thus, we are able to tran on a large data set derved from real mages wthout knowng the actual llumnant of the mages. We tested ths algorthm both on a very large number of artfcal mages generated from a database of 100 llumnants and 260 surface reflectances and on real mages taken wth a dgtal camera. Although traned wth nexact target values, the neural network performs better than the GW algorthm that was ntally used to tran t, especally for scenes wth a small number of surfaces The gray world algorthm To llustrate the bootstrappng algorthm, we used a verson of the gray world algorthm. The chromatcty of the llumnant s determned from a weghted average of all the pxels n an mage. Ths average s computed relatve to the gray world average of the colors n the database. Ths helps compensate for the fact that the dstrbuton of surface colors s not necessarly precsely gray. For example, R llum, the red component of the llumnant s computed as: R µ R llum = R µ R DB sensor where µ R s the average red of the mage, µ R DB s the average red of the database and R sensor s the camera sensor response for whte under the canoncal llumnant. In our experments, we took the canoncal llumnant to be the one for whch the camera s calbrated (.e. a perfect whte patch wll generate equal responses for all three camera sensors). Consequently we have: R sensor =G sensor =B sensor =255 (2) For artfcal mages, the database average s computed from all the surface reflectances used to generate the data set, whle for real mages, the database average s computed as the average over a large number of mages. The absolute RGB values of the llumnant are not mportant, snce we are nterested only n ts chromatcty and not n ts ntensty. Whle there are a number of chromatcty spaces that could be used for our experment, we used only the one descrbed n the next secton, to be consstent wth prevous neural network experments The neural network archtecture The neural network we used s a Perceptron 6 wth two hdden layers (H-1 and H-2), depcted n Fg. 1. The nput layer (In) conssts of a large number of bnary nputs representng the chromatcty of the RGBs n the scene. Each mage RGB from a scene s transformed nto the rg-chromatcty space: r=r/(r+g+b) and g=g/(r+g+b) (3) Ths space s unformly sampled wth a step S, so that all chromatctes wthn the same samplng square of sze S are taken as equvalent. Each samplng square maps to a dstnct network nput neuron. Ths chromatcty representaton dsregards any spatal nformaton n the orgnal mages and takes nto consderaton only the chromatctes present n the scene. The nput neurons are set ether to 0 ndcatng that an RGB of chromatcty rg s not present n the scene, or 1 ndcatng that t s present. Fg. 2 shows the representaton of an mage taken under two dfferent llumnants (a tungsten bulb and a (1)

3 blush fluorescent tube). The dots n the fgure represent chromatctes that are present the mages and they correspond to the actve nput nodes of the neural network for these two mages. Ths s the only nformaton that the neural network receves as nput. Adaptve Layer Out In H-1 H-2 Fgure 1. The Neural Network Archtecture Fluorescent Lght Tungsten Lght g 0.4 g r r Fgure 2. The chromatcty map of the same mage taken under two dfferent llumnants We have made experments wth dfferent szes for the nput layer and obtaned comparable color constancy results n all cases. The networks used n ths paper have 3600 nput nodes. The frst hdden layer (H-1) contans 400 neurons and the second layer (H-2), 40 neurons. The output layer (Out) conssts of only two neurons, correspondng to the chromatcty values of the llumnant. Prevous experments 1,7 showed that the sze of the hdden layers can vary wthout affectng the performance of the network. All neurons have a sgmod actvaton functon of the form:

4 1 y = ( A θ 1+ e ) where A s the actvaton (the weghted sum of the nputs) and θ s the threshold. (2) 2.4. Neural network tranng The neural network was traned usng the backpropagaton algorthm 6 a gradent descent algorthm that mnmzes the network s output errors. The tranng data s composed of a large set of nput vectors (derved from syntheszed or real mages) and, for each nput vector, a target output vector. Ths data set s presented a number of tmes (epochs) to the network durng whch tme the network adjusts ts weghts and thresholds to mnmze the output error (defned as the RMS error between the actual output values of the network and the target values). Intal tests performed wth a standard neural network archtecture descrbed above showed that t took a large number of epochs to tran the neural network. To overcome ths problem, a seres of mprovements has been developed and mplemented 7, 8 : The gamut of the chromatctes encountered durng tranng and testng s much smaller than the whole, theoretcal, chromatcty space. Thus, we modfed the neural network s archtecture, such that the frst hdden layer (H-1) receves nput only from the actve nodes (the nput nodes that were actvated at least once durng tranng). The nactve nodes (those nodes that were not actvated at any tme) are purged from the neural network, together wth ther lnks to the frst hdden layer. The network s archtecture (see Fg. 1) s actually modfed only durng the frst tranng epoch. The lnks from the frst hdden layer are redrected only towards the neurons n the nput layer that are actve,.e. those that correspond to exstng chromatctes whle lnks to nactve nodes are elmnated. Due to the fact that the szes of the layers are so dfferent, we used dfferent learnng rates for each layer, proportonal to the fan-n of the neurons n that layer 9. These optmzatons shortened the tranng tme by a factor of more than 10, to about 5-6 epochs. The error functon that we used for tranng and testng the network s the Eucldean dstance n the rg-chromatcty space between the target and the estmated llumnant. 3. EXPERIMENTS DONE ON SYNTHESIZED IMAGES 3.1. Generatng the synthetc mages Expermentng wth synthetc mages has the advantage that the whole envronment s carefully controlled. Moreover, we can generate an arbtrarly large number of mages, such that we obtan stable results when testng the algorthms. For syntheszed mages, the user can set the number of the patches comprsng a scene. A patch s generated for each chromatcty present n the mage. The RGB color of a patch s computed from ts randomly selected surface reflectance S j, the spectral dstrbuton of the llumnant E k (selected at random, but the same for all patches n a scene) and by the spectral senstvtes of camera sensors ρ accordng to: k j R k j R = E S ρ, G = E S G ρ and B E k j = S B ρ (3) The ndex s over the wavelength doman, correspondng to wavelengths n the range of 380nm to 780nm. The sensor senstvtes are those of a SONY DXC-930 CCD vdeo camera, wth gamma correcton turned off and calbrated for tungsten lght. The database of surface reflectances contans 260 measured reflectances. The database of llumnant power spectra s composed of 100 llumnants. All llumnants were ether measured drectly or generated as a lnear combnaton of two measured llumnants, n order to provde a more unform chromatc dstrbuton. The color of the llumnants ranges from fluorescent lghts wth blue flters to reddsh ncandescent lghts, coverng a wde range of naturally occurrng llumnants. Fg. 3 shows the dstrbuton n the rg-chromatcty space of the llumnants and surfaces. The chromatcty of the surfaces s shown as vewed under a perfect whte llumnant. The data used for tranng the neural network was generated as follows. Frst we generated synthetc mages (100 mages per llumnant) composed of 35 dfferent surfaces each, and n a later experment, we used scenes wth 50 surfaces nstead of 35. Then nstead of provdng the neural network wth the exact chromatctes of the llumnants used for

5 generatng these scenes, we used the llumnant estmates gven by the GW algorthm. It must be noted that snce the GW algorthm s calbrated for the surface database that we used, t yelds very accurate estmates of the llumnant f all (or most) of the surfaces from the database are present n a scene. 260 Surfaces 100 Illumnants g g r r Fgure 3. Database llumnants and surfaces 3.2. Tranng the neural network on synthetc mages The nput data to the neural network s a tranng set composed of 10,000 mages and the estmates of the correspondng llumnants provded by the GW algorthm. The network sub-samples these mages nto a larger set of 100,000 scenes. Each scene s generated by choosng a random number of surfaces from one of the nput mages. The mnmum number of surfaces per scene was set to 10, whle the maxmum was gven by the number of surfaces n the nput mages. Thus, f the number of surfaces n the synthetc mages s equal to 35, the sub-sampled scenes have from 10 to 35 surfaces. In the case of a scene composed of 35 surfaces, the whole mage s passed to the network. In a second experment, where we generated synthetc mages composed of 50 surfaces per scene, the network was traned on scenes composed of 10 to 50 surfaces per scene. The llumnant of each sub-sampled mage s nherted from the synthetc mage from whch t was generated. Thus the grayworld algorthm bases ts estmate on the full mage, whle only the sub-sampled mage s passed to the network. As a result, the grayworld estmate s more accurate and more stable than t would be f t were computed on only the sub-sampled data. The networks are traned for ten epochs on a set of scenes for whch the llumnant s not known exactly. Even so, the average error rates drop fast to around 0.015, whch s a satsfactory value. In order to compare the bootstrappng algorthm to prevous experments 1, 7, 8, where the network was traned wth exact llumnant values, we also generated a separate tranng set of 100,000 scenes, composed of 10 to 50 surfaces each, for whch the exact llumnant values were provded to the network. In all other respects, the network was traned n the same way as before Results of experments All networks were tested on the same data. The test set s composed of 50,000 scenes generated from the databases descrbed above. Each scene contans from 3 to 60 randomly selected surfaces. We compare the estmates of the neural networks (two traned usng nexact llumnant estmates and one traned usng exact values) aganst two other algorthms: the gray world algorthm (GW), descrbed above, and the whte patch algorthm (WP). The WP algorthm s a verson of the retnex algorthm 10 that ndependently scales each channel of the mage (R,G,B) by the maxmum pxel value found n each channel. Ths s equvalent to estmatng the color of the llumnant as beng the color gven by the maxmum pxel value on the R,G and B channels. Snce n syntheszed scenes there are no clpped pxels (.e. pxels for whch the sensor response on a channel s saturated), the WP algorthm performs much better then when tested on real world mages where clppng usually

6 occurs. The results are shown n Fg. 4. The error s computed as the Eucldean dstance n the rg-chromatcty space between the actual llumnant and the estmate gven by an algorthm. The errors of the neural networks and of the two other algorthms (GW and WP) are plotted aganst the number of patches n the scene Error NN - 35 NN - 50 NN - exact GW WP Number of surfaces n a scene Fgure 4. Results of the estmates of dfferent color constancy algorthms on synthetc data Even when traned wth nexact llumnant chromatctes n the tranng set, the neural network s able to make qute good estmates of the llumnaton s chromatcty. Even more nterestng, t exhbts a 'bootstrappng' capablty, yeldng better results than the grayworld algorthm that traned t, especally on mages wth few colors. For 35 or fewer surfaces n a scene, all neural networks yeld better results than the GW and WP algorthms. The neural network that was traned on exact data (NN-exact shown n Fg. 4) performs consstently better than GW and WP even on larger numbers of surfaces per scene. The neural network (NN-35) that was traned on scenes composed of maxmum 35 surfaces and on llumnant estmates provded by GW s surpassed by the GW and WP algorthms for scenes wth more than 35 surfaces. Ths happens for two reasons. One, GW and WP converge to almost zero estmaton error as the number of surfaces n the scene approaches the number of surfaces n the database. Two, the neural network performs worse on scenes contanng more surfaces than t ever encountered durng tranng (35 n ths case). Smlar results, although not as dstnct, occur n the case of the neural network (NN-50) traned on scenes wth a maxmum 50 surfaces and on llumnant estmates provded by GW. Ths network s also surpassed by both GW and WP algorthms for scenes wth more than 50 surfaces. 4. EXPERIMENTS DONE ON REAL IMAGES 4.1. Usng real mages for tranng and testng Workng wth real mages mposes some constrants on the whole expermental setup. Frst, the number of real mages avalable s lmted. Second, the number of llumnants s also lmted. However, f both tranng and testng are done on data derved from real mages, the camera senstvty functons need not to be known. Moreover, artfacts that can not be avoded n real mages (e.g. nose and flare) are mplctly modeled nto the tranng set and thus the neural network can yeld good results when tested on mages that contan the same type of artfacts. For the experments done on real mages we used mages of natural scenes taken wth a Kodak DCS460 dgtal camera. The orgnal resoluton of 2,000-by-3,000 pxels was reduced to around 1,000-by-600 for all mages. We dvded the mages

7 nto two sets, one to be used for tranng and the other for testng. The mages were lnearzed to compensate for the bult-n gamma correcton, but otherwse we dd not make any other assumptons regardng the data. To remove part of the nose and to reduce the resoluton, the mages were also resampled. The chromatctes of the llumnants were measured by takng mages of a standard whte patch when takng mages of the scenes. Ths whte patch s not present n the man mages, however, snce t would otherwse bas the WP results, whch partally rely on the presence of a whte patch n the mage. The database average used to compensate the GW algorthm was computed by averagng all surface RGBs n the mages. Although t does not provde a perfect compensaton, as t does wth the synthetc data where the surface dstrbuton s known n advance, t does mprove the GW llumnaton estmates. The results of experments done on real mages show the postve effect of the database compensaton Tranng the neural networks on data derved from real mages We used 47 mages for tranng. Each mage was sub-sampled nto a number of scenes contanng from 10 to 300 randomly selected pxels from the orgnal mages. The llumnant correspondng to these scenes was nherted from the estmaton provded by the GW algorthm whch computed the llumnant based on the entre mage. A second neural network was traned on exact llumnant feedback. A total of 47,000 scenes were generated (1,000 scenes from each mage) for the tranng set. Both networks were traned separately for 10 epochs Testng on real mages In a frst experment on real mages, we tested both neural networks on 39 mages. We compared the bootstrapped neural network (.e. traned on llumnant estmates provded by the GW algorthm) to a neural network traned on exact feedback. Comparsons are also made to the WP and GW algorthms and the Illumnaton Chromatcty Varaton (ICV). ICV s not a color constancy algorthm per se, rather t s smply a measure of the average shft n the rg-chromatcty space between a chosen canoncal llumnant and the correct llumnants. Ths can be consdered as a worse case estmaton, where we smply pck some llumnant and take t as the llumnant estmate for all nput mages. In our experments, the canoncal llumnant was selected to be the one for whch the CCD camera was color balanced; n other words, the llumnant for whch the mage of a standard whte patch records dentcal values on all three color channels. The results are shown n Table 1. The mean error represents the average estmaton error over all mages. The standard devaton s also shown. Both neural networks perform much better than the other color constancy algorthms. The GW algorthm wth database compensaton has a small average error, too. However, n the general case, where the statstcs of the surfaces are not known a pror (see GW wthout database compensaton n Table 1), the results of the GW algorthm are worse, comparable to those of the WP algorthm. Table 1. Results of color constancy algorthms on real mages Color Constancy Algorthm Mean Error Std. Dev. Illumnaton Chromatcty Varaton (ICV): Grey World ( GW wthout database compensaton): Grey World (GW wth database compensaton): Whte Patch (WP): Neural Network traned on GW llumnant estmates: Neural Network traned on exact llumnant data Testng on data derved from real mages A second experment was done on a large number of scenes, sub-sampled from the 39 test mages. A total of 78,000 scenes were generated (2,000 from each mage), each scene contanng 5 to 80 dstnct surfaces. The estmate of the algorthms was compared aganst the llumnant correspondng to these scenes. The llumnants were nherted from the actual llumnants of the test mages. The results of the test are shown n Fg. 5. The bootstrapped neural

8 network (NN-80) has better results than the GW algorthm; however, the network traned on exact data (NN-exact) has the best results Error NN - 80 NN - exact GW Number of surfaces n a scene Fgure 5. Results of the estmates of dfferent color constancy algorthms on data derved from real mages 5. CONCLUSION A neural network for color constancy can be traned to estmate the chromatcty of the ncdent scene llumnaton wthout havng exact knowledge of the llumnaton chromatcty n the tranng set. The network learns to make a better estmate than the smple grayworld algorthm used n ntally tranng t. Ths substantally smplfes the effort requred to obtan or synthesze the tranng set. Ths approach works even f the camera sensors are unknown, thus provdng an easy way for color correctng mages taken wth an uncalbrated camera. 6. REFERENCES 1. B. Funt, V. Carde, and K. Barnard, Learnng Color Constancy, Proc IS&T/SID Fourth Color Imagng Conf., pp , Scottsdale, Arzona, November G. Buchsbaum, A Spatal Processor Model for Object Colour Percepton, J. Frankln Insttute, 310 (1), pp. 1-26, M. Swan and D. Ballard, Color Indexng, Int. J. of Computer Vson, 7:1, pp , D.H. Branard and W.T. Freeman, Bayesan Color Constancy, J. Opt. Soc. Am. A, 14(7), , G. Fnlayson, M. Drew, and B. Funt, Color Constancy: Generalzed Dagonal Transforms Suffce, J. Opt. Soc. Am. A, 11(11), pp , D.E. Rumelhart, G.E. Hnton, and R.J. Wllams, Learnng Internal Representatons by Error Propagaton, n Parallel Dstrbuted Processng: Exploratons n the Mcrostructure of Cognton. Volume I: Foundatons, D.E. Rumelhart, J.L. McClelland and the PDP Research Group, eds., pp , MIT Press, Cambrdge, MA, B. Funt, V. Carde, K. Barnard, Neural Network Color Constancy and Specular Reflectng Surfaces, AIC Color 97, Kyoto, Japan, May 25-30, V. Carde, B. Funt, and K. Barnard, Adaptve Illumnant Estmaton Usng Neural Networks, Proc. of the 8 th Int. Conf. on Artfcal Neural Networks, pp , Skövde, Sweden, D. Plaut, S. Nowlan, and G. Hnton, Experments on Learnng by Back Propagaton, Techncal report, CMU-CS , Carnege-Mellon Unversty, Pttsburgh, USA, Land E.H. The Retnex Theory of Color Vson. Scentfc Amercan, pp , 1977.