IEEE Int. Conf. on Rootics nd Automtion, ICRA'7 Rom, Itli, April 27 Control Cmer nd Light Source Positions using Imge Grdient Informtion Eric Mrchnd Astrct In this pper, we propose n originl pproch to control cmer position nd/or lighting conditions in n environment using imge grdient informtion. Our gol is to ensure good viewing condition nd good illumintion of n oject to perform vision-sed tsk (recognition, trcking, etc.). Within the visul servoing frmework, we propose solutions to two different issues: mximizing the rightness of the scene nd mximizing the contrst in the imge. Solutions re proposed to consider either sttic light nd moving cmer, eitheror moving light nd sttic/moving cmer. The proposed method is independent of the structure, color nd spect of the ojects. Experimentl results on oth synthetic nd rel imges re finlly presented. I. OVERVIEW In this pper we investigte the prolem of reltive plcement etween n oject, cmer nd light source. Ensuring n optiml plcement of the cmer or of light source is n essentil step in the development of industril vision systems. Indeed good lighting conditions ensure good imge qulity nd thus enle to simplify or improve reliility of vision lgorithms. Most of the reserch regrding illumintion re focused on shpe from shding (eg, [24], light source position estimtion (eg, [9]), trcking (eg, [14], [7]). Some of these works ssume the conservtion of the point luminnce over the imge sequence [12] ut most of them ssume more complex illumintion models such s the Phong model [19] or the Torrnce-Sprrow model [22]. Nevertheless, few works hve considered lighting conditions, nd especilly illumintion control or cmer control wrt. illumintion conditions, within rootics tsks or ctive vision. Skne nd Sto [2] present n utomtic plnning method of light source nd cmer plcement to minimize shdow cused y the surrounding environment. Cown et l. [2][4] extend the CAD-sed system presented in [3] in order to mintin the rightness of the oject surfce within the dynmic rnge of the cmer [2] (the surfce must not e either too right or too drk). Furthermore light plcement hs to e optimized for edge detection [4]. The method presented in [3] is used to synthesize 3-D regions of cceptle cmer loctions for the specified tsk. Ech criterion (sptil resolution, field of view, visiility, edge contrst, cmer dynmic rnge, etc.) llows to define 3-D regions which provide the spce of possile viewpoints when they intersect. The system ICE presented in [23] determines the est cmer view nd light source loction to optimlly Author is with INRIA, IRISA, Lgdic project, F-3 Rennes, Frnce; emil: Eric.Mrchnd@iris.fr oserve given edge nd to mximize the ccurcy of its position. The cmer nd light positions re chosen such tht mesurements dt cn e otined with minimum uncertinty. Minly contrst on the edge is considered nd the system is sed on the illumintion model descried in [22]. Murse nd Nyr [17] used n eigenspce-sed method to determine the illumintion for which the ojects re most distinguishle for recognition purpose. More recently Eltoft et l. [] proposed system thn cn optimlly enhnce imge fetures such s edges or points y ctive scene illumintion. More complex illumintion models re considered [11], [21]. Let us not tht in most of these systems good knowledge of the oject or of the environment hs to e known in order to evlute off-line the vrious criteri relted to the specified tsk nd to determine the est lightsource nd cmer loction. In the different context of 2D trcking, Hger et l.[8] derive the interction mtrix tht link the time vrition of imge intensity to the 2D motion of n oject. In this pper, we lso consider models used in motion nlysis nd determine the vrition of imge intensity due to cmer or light source motion. Oviously the underlying model, sed on the derivtion of the opticl flow constrint eqution (OFCE) is, pprently, very restrictive. Nevertheless, experimentl results show tht it remins usle in mny cses. This pper presents method to control cmer position with respect to light source. Our gol is to ensure good illumintion of n oject or good cmer loction to e le to perform efficiently vision-sed tsks. Within the visul servoing frmework, we propose solutions to two different issues: mximizing the rightness of the scene nd mximizing the contrst or grdient in the imge. Solutions re proposed to consider either sttic light nd moving cmer, or moving light nd sttic/moving cmer. Thnks to the simplicity of the illumintion model sed on the OFCE, the proposed method is independent of the structure, color nd spect of the ojects. Two different goodness functions my e proposed to chieve this gol: one is directly sed on the intensity within the imge while the second is sed on the intensity grdient. To outline the issue, our primry gol will e to move the cmer while the lighting remins sttic (see Figure 1.). Then, we will propose to move the lighting while the cmer remins sttic (see Figure 1.). In the reminder of this pper we first recll the opticl flow constrint eqution nd show how it cn e used to control moving cmer in Section II. Goodness functions sed on rightness re shown in Section III nd their integrtion http://www.iris.fr/lgdic
sttic light light R moving cmer sttic cmer Fig. 1. Controlling lighting conditions. () sttic light/moving cmer () moving light/sttic cmer within visul servoing control lw presented in Section IV. Finlly, experimentl results showing the vlidity of our pproch re presented. II. TEMPORAL VARIATION OF THE LIGHTING INFORMATION ) Opticl flow constrint eqution: The sic hypothesis ssumes the temporl constncy of the rightness for physicl point etween two imges. This hypothesis leds to the so-clled opticl flow constrint eqution (OFCE) tht links the temporl vrition of the luminnce to the imge point motion. More precisely, ssuming tht the point hs displcement (dx, dy) T in the time intervl dt, the previous hypothesis leds to: I(x + dx, y + dy, t + dt) = I(x, y, t). (1) A first order Tylor expnsion of this eqution gives: I x dx + I y I dy + dt =. (2) t Denoting dx dt = ẋ nd dy dt = ẏ the motion of the point in the imge nd I x = I x nd I y = I y the sptil grdient of the intensity nd I t = I the temporl vrition of the luminnce, we finlly otin the opticl flow constrint eqution given y: I = I x ẋ I y ẏ (3) ) Interction mtrix ssocited to the luminnce: Our gol is to link the temporl vrition of the luminnce to the time vrition of the cmer pose (or kinemtic screw v = (v, ω) where v is the instntneous liner velocity nd ω is the instntneous ngulr cmer velocity). This is in fct strightforwrd knowing the interction mtrix ssocited to the point. We indeed hve: ẋ = ( 1/Z x/z xy (1 + x 2 ) y ) v (4) tht we cn rewrite ẋ = L x v nd ẏ = ( 1/Z y/z 1 + y 2 xy x ) v () tht we rewrite ẏ = L y v. Using these equtions nd the OFCE we hve I t = I dr dt or: I = ( I x L x + I y L y ) = L I(x,y) v (6) L I(x,y) is the interction mtrix ssocited to the luminnce of point in the cse of moving point nd sttic cmer. III. CONTROLLING LIGHTING CONDITIONS As lredy stted, our gol is to control the illumintion of n oject. We will then consider two informtions relted to the lighting condition: the intensity in the imge. For such tsk, our gol will e to mximize the perceived luminnce of the oject in the imge. the contrst. Mximizing the luminnce is not lwys significnt. Indeed, for some oject, too much light my suppress some informtion (due, for exmple to speculrities). Therefore, in second time we will try to mximize the vlue of the intensity grdients in the imge (which is relted to contrst informtion). With respect to these specifictions of good lighting condition, we cn propose two cost functions tht reflect these ehviors. A. Mximizing the luminnce Our gol is to position the cmer wrt. the enlightened spect of the oject. We therefore wnt to mximize the quntity of light (re)emitted y the oject of interest nd perceived y the cmer to ensure good lighting condition. Applying the proposed methodology, we wnt to mximize the following cost function: h s = I(x, y) (7) where I(x, y) is the intensity of the 2D point (x, y). The vrition of the cost function h s due to cmer motion, tht will e used to control cmer or light source motion (see Section IV), is then given y = x y I(x, y) (8)
where r denote the cmer pose (trnsltion nd rottion). I(x,y) is nothing ut the interction mtrix L I(x,y) s defined in (6). Considering eqution (6) we got: = ( I x L x + I y L y ). (9) B. Mximizing the contrste. If our gol is to mximize the contrst within the imge, good criterion is to mximize the sum of the components of the sptil intensity grdient within the imge. The corresponding cost function is given y: h s = [ I 2 x + I 2 ] y. (1) As in Section III-A We therefore need to compute the grdient hs tht is in fct given y: = ( hs x L x + h ) s y L y (11) with nd ( 2 ) x = 2 I I x 2 x + 2 I I x y y ( 2 ) y = 2 I I x y x + 2 I I y 2 y After some rewriting, we finlly get: = 2 [( 2 ) I x 2 I x + 2 I y x I y L x ( 2 ] I + x y I x + 2 I y 2 I y )L y (12) IV. INTRODUCING ILLUMINATION CONSTRAINTS IN VISUAL SERVOING In this section we study how to use the constrints presented in Section III to control the cmer or the light source position. In oth cses the method relies on the well known visul servoing pproch nd tkes dvntge of the redundncy frmework. A. Positionning Cmer wrt. Visul Trgets The imge-sed visul servoing consists in specifying tsk s the regultion in the imge of set of visul fetures[6][1]. A good review nd introduction to visul servoing cn e found in [13]. Let us denote s the set of selected visul fetures used in the visul servoing tsk. To ensure the convergence of s to its desired vlue s, we need to know the interction mtrix L s tht links the motion of the oject in the imge to the cmer motion. It is defined y the clssicl eqution [6]: ṡ = L s (s, Z) v (13) where ṡ is the time vrition of s (the motion of s in the imge) due to the cmer motion v. The prmeters Z involved in L s represent the depth informtion of the considered ojects expressed in the cmer frme. A vision-sed tsk e 1 is defined y: e 1 = C(s s ) (14) where s is the desired vlue of the selected visul fetures, s is their current vlue (mesured from the imge t ech itertion of the control lw), nd C, clled comintion mtrix, hs to e chosen such tht CL s is full rnk. It cn e defined s C = L + s (s,p). For mking e 1 exponentilly decreses nd then ehves like first order decoupled system, the cmer velocity given s input to the root controller is given y: v = λe 1 (1) where λ is the proportionl coefficient involved in the exponentil convergence of e. B. Introducing constrints within the positioning tsk If the vision-sed tsk does not constrin ll the n root degrees of freedom, secondry tsk cn e performed nd we otin the following tsk function: where e = W + We 1 + (I 6 W + W)e 2 (16) e 2 is secondry tsk. Usully e 2 is defined s the grdient of cost function h s to e minimized (e 2 = hs ). This cost function is minimized under the constrint tht e 1 is relized. W + nd I 6 W + W re two projection opertors which gurntee tht the cmer motion due to the secondry tsk is comptile with the regultion of s to s. Indeed, thnks to the choice of mtrix W, I 6 W + W elongs to Ker L s, which mens tht the reliztion of the secondry tsk will hve no effect on the vision-sed tsk. The control is now given y: v = λe (I 6 W + W) e 2 t (17) Considering redudncy in visul servoing hs een lredy considered [1], [18] ut usuly relted to root mnipulility. Informtion directly extrcted from the imges hve een lso considered (eg, in [1] for occlusion voidnce). C. Eye-in-hnd versus Eye-to-light control To control the cmer/light source reltive position, we will consider two cses. In the former one, the cmer is controlled nd focused on the oject while the light remins sttic. This experimentl context is not lwys the most interesting one. Indeed, if the cmer is moving the spect of the oject will chnge with time. It is often more interesting to control the light position nd orienttion while the cmer remins sttic. This is the second cse tht is considered. Deling with the former cse, the cmer is focused to the oject of interest using clssicl visul servoing tsk. If s = (x, y) defined the oject center of grvity, s is defined
s s = (, ) nd the tsk function tht enforces the focusing tsk nd ensures good lighting of the oject is given y: e = W + WL + s (s s ) + (I W + W) (18) where hs is given y either eqution (9) or (12). Considering the second cse, oject is sttic in the imge (cquired y cmer C 1 ) nd we wnt to mximize rightness or contrst y moving the light-source. Here gin we consider the visul servoing frmework to point the light towrd the oject of interest nd to chieve good conditions. We first dd to the light second cmer C 2 whose opticl xis is ligned with the light direction. The min tsk is specified s simple focusing tsk tht constrins the virtul cmer/light system (two dof re constrined). We then consider the redundncy to control the cmer/light system to impose correct illumintion of the oject within the imge cquired y the other cmer. The tsk function is then defined s: e = W + W L + s (s s ) }{{} min focusing tsk ( +(I W + R R[ R W) T t] ) hs R }{{} secondry tsk defined wrt. to the other cmer (19) with R nd t denotes the rottionl nd trnsltionl mpping of the fixed cmer frme R C1 onto the moving cmer/light frme R C2. Let us note here tht if the cmer C 1 is now moving, the prolem remins exctly the sme s long s we know the trnsformtion R nd t etween the cmer nd the light (see Result in prgrph V-B). V. EXPERIMENTAL RESULTS Results otined in this section hs een otined either in simultion using Open GL simultion tools or rel rootics cell t IRISA. The system hs een implemented using the ViSP softwre [16]. A. Eye-in-hnd coordintion 1) Simultion: The gol of this first simultion is to vlidte our pproch on simple scene. The gol is to perform positioning tsk wrt. sphere nd to control the cmer in order to see this sphere under good lighting condition (criterion (7) is considered). In this experiment the light-source is sttic nd the cmer is moving s descried y Figure 1. Control lw presented in eqution (18) is considered. The dvntge of the sphere is tht its spect remins the sme whtever the cmer position. Only the sphere luminnce will e modified. In this experiment we considered positionl light source. Results of this positioning tsk re presented on Figure 2.. It is worth noting tht the verge intensity increses very smoothly (see Figure 2.). We lso plot the distnce etween the cmer nd the oject-light xis (see Figure 2.c). We cn note tht this distnce tends towrds zero, i.e. t the end of the positioning tsk, the cmer is locted etween the sphere nd the light s cn e expected (see Figure 2.d). z 26 24 22 2 18 16 14 12 1 cost 2 4 6 8 1 12 optimizing lighting conditions x 16 14 12 1 8 6 4 2 distnce 2 4 6 8 1 12 cmer loction trget light 1 y c Fig. 2. [Simultion] Positioning wrt. sphere under good lighting conditions: () scene oserved y the cmer (illumintion increses) () verge intensity in the imge (c) distnce to sphere-light xis (d) cmer/sphere/light position over time 2) Rel Experiments: ) Mximizing luminnce on sphere: The sme experiment ws crried out on our experimentl setup. A white ll is lighted y spot. As in the previous section the cmer mounted on the root end-effector is focused on the ll nd controlled using eqution (18) in order to mximize the ll luminnce. As expected, the luminnce increses (see Figure 3--c nd plot 4.c) until the cmer/root moves etween the ll nd the light-source creting lighting occlusion (see Figure 3.d nd the lst itertion of plot 4.c). As expected, the ehvior of the system is very similr to simultion results presented in the previous prgrph. Similr results for this oject is otined when the contrst goodness function is considered. ) Mximizing luminnce on complex oject: Sme experiment cn e done with more complex oject (see Figures. nd.). Although the shpe of the oject is modified during the experiment, the verge luminnce increses s specified in the tsk (see Figure 6 tht is relted to imges in Figure.). c) Mximizing contrst on complex oject: In Figure 7 we consider the goodness function sed on the contrst informtion (tht is mximize the norm of the grdient in the imge). As cn e seen on Figure 7, the d
c d grdient in the imge increses which is due to oth the light nd the modifiction in the oject spect due to the cmer motion. It is cler tht the lst imge of the oject is etter suitle, due the presence of importnt grdient, for tsk such s recognition or trcking. Fig. 3. [Rel experiment] Positioning wrt. sphere () first imge (c) luminnce increses (d) the cmer is now etween the sphere nd the light (tht is the ctul expected position ut tht in prctise crete light occlusion ) 16 1 cost function hs hs.dt 14 13 12 8 7 h_s 11 6 1 99 98 4 97 96 2 3 4 6 7 8 9 1 11 Fig. 4. [Rel experiment] Positioning wrt. sphere :cost function h s tht reflects ll luminnce 3 2 1 1 1 2 2 3 3 4 4 Fig. 7. Scotch experiment: mximizing grdient/contrst () imges of the sequence () evolution of the goodness function h s In this prgrph we considered moving cmer nd sttic light-source. The consequence of such configurtion is tht it implies modifictions in the spect of the scene which is not lwys suitle. In the next experiment we consider sttic cmer nd moving light source. B. Eye-in-hnd/Eye-to-light coordintion Fig.. 1 98 96 94 Mximizing luminnce on more complex oject cost function hs hs.dt As regrds this issue, we first perform positioning experiment involving complex oject. We consider, in simultion, model of the Venus of Milo. In this experiment we first consider sttic cmer nd moving light s depicted in Figure 1. In second time, when minimum of the cost function is reched, we impose n ritrry motion to the cmer. The light must then move in order to mintin correct lighted of the sttue. The results presented (see Figure 8) show the vlidity of our pproch for oth goodness function (luminnce on Figure 8 nd contrst on Figure 8). One cn see tht the light trjectories round the sttue on Figure V-B. 92 9 88 86 2 3 3 4 4 6 Fig. 6. Mximizing luminnce (correspond to the experiments presented on [Figure ]) : goodness function h s VI. CONCLUSION We presented method to ensure correct viewing or illumintion of n oject using visul servoing scheme nd only luminnce or grdient informtion. The illumintion model considered in this is indeed very corse nd is in mny cses flse. Nevertheless, it llows to servo the cmer or the light source in order to chieve good illumintion
Fig. 8. Illuminting the Venus of Milo () mximizing the venus luminnce () Mximizing the contrst. In the three first columns the cmer remin fixed then n ritrry motion is given to the cmer. The light source moves to ensure the specified tsk. z cmer loction light position (intensity) light position (contrste) trget initil 12 light loction initil cmer loction 1 8 6 4 2 2 4 Fig. 9. 2 2 4 x 6 8 1 optimizing lighting condition 1 Illuminting the Venus of Milo : Cmer nd light trjectory of the scene (t lest wrt. the considered criteri). Experimentl results in simultion or on rel scenes show the vlidity of the pproch. Nevertheless, it is well known tht imge luminnce of scene depend of the ojects (ledo, reflectnce,...), of reltive surfce cmer orienttion, nd of the cmer/oject/light source position. Future work will e devoted to study more complex illumintion models. This my require either more informtion out the scene (3D model nd surfce informtion), or the estimtion of the unknown prmeters (such light source position). REFERENCES [1] F. Chumette, E. Mrchnd. A redundncy-sed itertive scheme for voiding joint limits: Appliction to visul servoing. IEEE Trns. on Rootics nd Automtion, 17():719 73, Octoer 21. [2] C.K. Cown, A. Bergmn. Determining the cmer nd light source loction for visul tsk. IEEE Int. Conf. on Rootics nd Automtion, ICRA 89, pp. 9 14, Scottsdle, My 1989. [3] C.K. Cown, P.D. Kovesi. Automtic sensor plcement from vision tsk requirements. IEEE Trns. on Pttern Anlysis nd Mchine intelligence, 1(3):47 416, My 1988. [4] C.K. Cown, B. Modyur. Edge-sed plcement of cmer nd light source for oject recognition nd loction. IEEE Int. Conf. on Rootics nd Automtion, ICRA 93, volume 2, pp. 86 92, 1993. y 1 1 2 2 [] T. Eltoft, R.J.P. de Figueiredo. Illumintion control s mens of enhncing imge fetures in ctive vision systems. IEEE Trns. on Imge Processing, 4(11):12 13, Novemer 199. [6] B. Espiu, F. Chumette, P. Rives. A new pproch to visul servoing in rootics. IEEE T. on Rootics nd Automtion, 8(3):313 326, 1992. [7] M. Gouiffès, C. Collewet, C. Fernndez-Mloigne, A. Trémeu. Feture point trcking : roustness to speculr highlights nd lighting vritions. ECCV 26, pp. 92 93, Grz, My 26. [8] G. Hger, P. Belhumeur. Efficient region trcking with prmetric models of geometry nd illumintion. IEEE Trns. on Pttern Anlysis nd Mchine Intelligence, 2(1):12 139, Octoer 1998. [9] K. Hr, K. Nishino, K. Ikeuchi. Light source position nd reflectnce estimtion from single view without the distnt illumintion ssumption. IEEE Trns. on Pttern Anlysis nd Mchine Intelligence, 27(4):493, April 2. [1] K. Hshimoto, editor. Visul Servoing : Rel Time Control of Root Mnipultors Bsed on Visul Sensory Feedck. World Scientific Series in Rootics nd Automted Systems, Vol 7, World Scientific Press, Singpor, 1993. [11] B. Horn. Root Vision. MIT Press, Cmridge, 1987. [12] B.K.P. Horn, B.G. Schunck. Determining opticl flow. Artificil Intelligence, 17(1-3):18 23, August 1981. [13] S. Hutchinson, G. Hger, P. Corke. A tutoril on visul servo control. IEEE T. on Rootics nd Automtion, 12():61 67, 1996. [14] H. Jin, P. Fvro, S. Sotto. Rel-Time feture trcking nd outlier rejection with chnges in illumintion. ICCV, pp. 684 689, July 21. [1] E. Mrchnd, G.-D. Hger. Dynmic sensor plnning in visul servoing. IEEE Int. Conf. on Rootics nd Automtion, volume 3, pp. 1988 1993, Leuven, Belgium, My 1998. [16] E. Mrchnd, F. Spindler, F. Chumette. ViSP for visul servoing: generic softwre pltform with wide clss of root control skills. IEEE Rootics nd Automtion Mgzine, 12(4):4 2, Dec 2. [17] H. Murse, S.K. Nyr. Illumintion plnning for oject recognition using prmetric eigenspces. IEEE Trns. on Pttern Anlysis nd Mchine Intelligence, 16(12):1219 1227, 1994. [18] B. Nelson, P.K. Khosl. tegrting sensor plcement nd visul trcking strtegies. IEEE ICRA 94, pp. 131 136, Sn Diego, My 1994. [19] B.T. Phong. Illumintion for computer generted pictures. Communiction of the ACM, 18(6):311 317, June 197. [2] S. Skne, T. Sto. Automtic plnning of light source nd cmer plcement for n ctive photometric stereo system. IEEE ICRA 91, pp. 18 187, Scrmento, April 1991. [21] H.D. Tgre, R.J.P. DeFigueiredo. A frmework for the construction of reflectnce mps for mchine vision. Computer Vision, Grphics, nd Imge Processing, 7(3):26 282, 1993. [22] K.E. Torrnce, E.M. Sprrow. Theory for off-speculr reflection from roughened surfcesw. J. of the Opticl Society of Americ, 7:11 1114, 1967. [23] S.K. Yi, R.M. Hrlick, L.G. Shpiro. Optiml sensor nd light-source positioning for mchine vision. CVIU, 61(1):122 137, Jnury 199. [24] R. Zhng, P.S. Tsi, J.E. Cryer, M. Shh. Shpe-from-shding: survey. IEEE PAMI, 21(8):69 76, August 1999.