Colour 3D system characterization

Transcription

1 Colour 3D system chrcteriztion Alexndr Lthuilière Frnck Mrzni Yvon Voisin LE2I, UMR CNRS 5158 UFR Sciences et Techniques Antenne d Auxerre BP AUXERRE CEDEX FRANCE.lthuiliere@iutlecreusot.u-bourgogne.fr f.mrzni@u-bourgogne.fr y.voisin@u-bourgogne.fr Abstrct This work is prt of lrge project. We work on stereoscopic system for 3D colour surfce reconstruction. This system is bsed on structured light principle. An LCD-projector sends colour coded structured light, nd 3CCD cmer snps few colour imges. This lst one llows the clcultion of both 3D nd colour informtion of object surfce in the scene with tringultion lgorithm. The system needs importnt clibrtion steps: geometricl one to clculte 3D coordintes nd colorimetric one to mesure the colour of the object in sme time to decode colour coded structured light. This rticle shows chrcteriztion protocol for colour 3D system. The model used is described nd the method is pplied to the two cses: 3CCD cmer nd LCD projector. Technicl Trck TPC5 Signl nd Imge Processing Min Keywords D7. Computer vision D5. Imge processing I. INTRODUCTION The min im of this work is the reconstruction of both 3D nd colour of n object surfce. The 3D cquisition in rtificil vision is mde with cmers. This cquisition needs two points of view or more. This principle is nmed stereovision. The cquisition cn be mde with one cmer in motion or with two or more fixed cmers in different plces. The cquisition cn be pssive or ctive. In pssive system, only informtion in cmer imges is used. The min step is the stereo mtching. The cquisition process gives two imges of the sme object tken from two points of view. The epipolr geometry [1] llows stereo mtching between the two views. The lst step is the clcultion of the 3D coordintes of the object by tringultion. In such pssive system, the biggest problem is the dense stereo mtching [2]. An ctive system needs n externl ctive light. In such system, the ctive light gives n dditionl texture onto the object surfce. The stereo mtching is generlly esier thn in pssive system. The light cn be sent by Lser diode or n LCD projector. With Lser diode, detection of ptterns into imges is esier becuse of the monochromticity of the Lser bem. The cmer cn use filter centred on Lser wvelength for optiml detection. Two techniques re vilble for cquiring n entire object. The bem cn move for sweeping the entire object with cylindricl lens nd moving mirrors. The bem cn be diffrcted with diffrction pttern for projecting more tht one line or point onto the object. But the most often in this cse, the mtching between projected bems nd detected ptterns is not very esy. All ptterns present the sme shpe nd colour. If prt of the pttern is occluded, the 3D reconstruction cn be mde with errors. With n LCD projector, we cn project more elborte ptterns without sweeping [3]. We cn project coded pttern in one shot or long the time. The coding cn be in grey level or in colour. The interested reder cn refer to the following pper [4] for clssifiction of coded techniques. For the mjor prt of ll precedent methods, the reconstruction is esier if the object is white or with uniform soft colour. For objects with strong colour, lot of methods bsed on the use of colour coded structured light give wrong results. There re others kinds of methods for the 3D nd colour reconstruction. They must be chosen ccording to the ppliction. For visuliztion, the esier nd low cost solution is the texture mpping [5]. If only visuliztion of the scene is interesting, the scene modelling [6] is good. This method is very useful for reconstruction of lrge scene nd uses lot of cmers. The reconstruction is bsed on voxel crving. The Ntionl Reserch Council Cnd is developing new kind of 3D colour cmer, mde with three Lser bems t three different wvelengths [7]. For mesuring the colour of the scene surfce, system clled spectrophotometer exists. It provides n ccurte mesure of spectrum with very low sptil resolution. Another pproch is bsed on the use of spectrl reflectnce of the object surfce with good sptil resolution, with multispectrl cmer. See [8] for detils bout this kind of system with results independent of the illumintion. For mesuring pproximtely the colour of the scene surfce, method like in [9] cn be used. In this rticle, the im is not relly the mesure of the colour of the scene surfce but the knowledge of it. So, the uthors choose the colour of the projected coded structured light. In the work described in this pper, colour coded structured light is used for n esy mtching nd, in sme time, the mesure of the colour scene surfce. For this, colour 3D model is needed. Section 2 exposes first bsic model. This model is not stisfctory for our ppliction. Tht is why we modified it nd two sets of corrections for the imges re used: flt-field imge nd sensor nonlinerity. These opertions must be done within cmer imges nd projected imges. Section 3 exposes both clibrtion nd cquisition protocol. Section 4 shows results nd discusses them before concluding this work in Section 5. II. MODEL

2 We decided to bse our work on Cspi s model. Indeed, other models exist, like those described in [1], [11], [12], [13], but they give solution like Cspi pproch. A. Cspi s model The whole system cn be described with one eqution which llows clcultion of the object colour. In this pper [Cspi], uthors use system composed of n LCD-projector nd 3CCD cmer. The system is modelled by this eqution: () I M M = AKP +, (1) R = G B RR GR BR RG GG BG RB GB BB k R k G r R + P g G k b B B.(2) M=[R,G,B] T is given pixel in cmer imge. I=[r,g,b] T is the projection instruction given to the LCD-projector. The vector M =[R,G,B ] T is extrcted from the imge of the object cquired under the mbient light t pixel loction. k R, k G nd k B re reflectnce of the object in colour system of primries. We put these coefficients in the K mtrix llowing complete mtrix nottion of the model. This mtrix is loction-dependnt nd unknown for scene to reconstruct. The opertor P is the trnsformtion from the projection instruction to ctul illumintion. Cspi nd l. ssume tht the three colour chnnels re decoupled; hence P models three sclr trnsformtions. P nd P -1 re represented by three look-up tbles. A is 3x3 mtrix. This mtrix is the projector-cmer coupling mtrix, clled cross-correltion mtrix. In this mtrix, projector chnnel gives response in ll cmer chnnels. Tht is why A is nerly digonl nd the sme for ll the pixels. For determining this mtrix, we project lot of uniform colour plns onto uniform white screen. We chose the white ptch of Mcbeth Color Checker. In this cse, K is the identity mtrix. With colours sent by the projector nd colours red in corresponding cmer imges, A cn be clculted with lest squre minimiztion. B. Improvements In the model we use, P cn not be determined. It is replced with unitry vector. But for tking into ccount rdiometric chrcteriztions of the cmer nd the LCDprojector, flt-field nd nonlinerity of the two sets re mesured. This ide comes from the chrcteriztion of multispectrl cmer described in this pper [14]. For ech pixel, proportionlity coefficient, nmed FF for flt-field, is clculted: M '= FF. c M. (3) where M is the imge cquired by the cmer nd M' the fltfield corrected one. For the LCD-projector, this eqution becomes I'= FFp. I. (3b) where I is the instruction projected imge nd I' the flt-field corrected one. Nonlinerities cn be expressed by three Look-Up-Tbles for ech set, nmed LUT R, LUT G nd LUT B. For shorter writing, these three components cn be written in one LUT. LUT R pplies itself on the red component of M' for the cmer (I' for the projector) nd it is the sme thing for other chnnels. Let M'' is the cmer imge (resp. I'' the instruction projector imge) corrected with nonlinerities. ( ) M" = LUT M ', (4) c ( ) I" = LUT I'. (4b) p All these pre-processes re pplied on both cmer nd projected imges. Then, the model becomes: M = AKI +, (5) M Finlly, the model cn be written in the following eqution: LUT ( FF. M ) A. K. LUT ( FF. I) LUT ( FF. M ) =. (6) c c p p + III. CHARACTERIZATION PROTOCOL Determintion of ll prmeters of this model is necessry. The cquisition process begins by cquiring imges for LCD-projector flt-field. For these imges, the projector needs to be ner the screen. These imges need to be cquired only one time before 3D cquisition. If the lighting conditions not chnge, these imges remin vlid. Then, the projector nd the cmer re positioned in front of the work re. The system should be in the sme condition thn for further 3D cquisition. Sets form stereoscopic system. The ngle between the two elements mesures bout 35 for best 3D recovery [15]. All others imges cn be cquired for the next steps of chrcteriztion. A. Cmer cse The cmer needs preliminry djustments: zoom, focusing, uto-white blnce off, gin off nd fix exposure time. White blnce is djusted mnully. We snp n imge of white screen nd set R, G nd B gin vlues for hving white vlue ( ) in ll pixels in this imge. 1) Flt-field: Flt-field correction is necessry in our cse for tking into ccount the light derivtion onto the screen. An imge of white uniform screen under mbient light is snpped. The lbortory hs clibrted mbient illuminnt D65. For voiding tking into ccount dust nd flw in the screen or on the cmer lens, more thn one imge is snpped with shifting the screen. The screen is lrge white pln. During experiments, six imges t six positions of the screen re snpped. In ech pixel, the men of the six imges is clculted. All imges re colour ones. All clcultions must be mde chnnel by chnnel. The men of the centre of the result imge is clculted. In perfect cse, this vlue should hve been the sme for ll the pixels of the entire c c

3 imge. The lst opertion is the clcultion of the coefficient in ech pixel, for reching the vlue of the men of the centre. The coefficients imge is nmed FF for Flt-Field. 2) Sensor nonlinerity: The sensor response is not uniform in response t stimulus coming from blck to white. For checking this response, n imge of the ptches coming from blck to white of Mcbeth Color Checker is snpped. This imge is snpped under D65 illuminnt. For hving theoreticl vlue, ech ptch is cquired with spectrophotometer. A polynomil form is clculted between these two sets of vlues: theoreticl vlues come from spectrophotometer in X xis nd vlues come from cmer imge on Y xis. The polynomil form is fourth degree form becuse it is the best to model this curve. imge of flt-field. Six imges re cquired for voiding dust on the screen, nd tken into ccount only the men of six extended imges. On this stge, the sme step of the process for the cmer cse is done: the men of the centre of the imge is clculted. Then it is pplied to ll imges for clcultion of the FF coefficient pixel by pixel. = = = = =.9999 = Fig. 2. Exmple of cquisitions for projector flt-field clcultion. 1/5s b. 1/125s Fig. 1. Grphic representtion of cmer LUT in two cses: exposure time 1/5s nd 1/125s The LUT from theoreticl vlues to experimentl cmer vlues is extrcted from the polynomil form. We must reverse this polynomil form for extrcting the LUT from experimentl cmer vlues to theoreticl ones. This LUT is then pplied to ll imges corrected with the FF imge. B. Projector cse Cmer flt-field nd nonlinerity need to be done before chrcterizing the projector. For the projector, we determine corrections chnnel by chnnel for voiding projector cross-tlk. 1) Flt-field: In the LCD-projector cse, flt-field is mesure of the illumintion shifting sent by the projector. Flt-field is informtion linked to ech projected pixel. All projected pixels must be seen by the cmer. For this, the cmer snps n imge of the projection rectngle. The projector must be plced close to the screen. The cmer does not chnge its position. It is importnt becuse the point of view nd lighting direction re the sme in ll cquisitions. The projection rectngle is extended t the size of the imge with clculting n homogrphy. For voiding projector cross-tlk, three imges re sent successively: one in red, one in green nd one in blue. The colour level is chosen in function of the level of sturtion in cmer imge. The combintion of the three plns with extension gives the 2) Sensor nonlinerity: This step needs to plce the projector nd the cmer in the stereoscopic conditions. In this cse, sensor nonlinerity cn not be mde with only one imge snpped. For clculting the three LUT, imges re projected with colour where one chnnel increse one by one. We project (5 ), (1 ), (15 ), (2 ) nd (255 ). The sme process is done for others chnnels with the sme scheme. Now, we cn put ll points in curve nd clculte the corresponding four degrees polynomil form. The LUT is obtined by inversion of this polynomil form. = =.9998 = /5s b. 1/125s Fig. 3. Grphic representtion of projector LUT C. Mtrix A determintion = =.9997 = Before cquiring imges for clculting A mtrix, flt-field nd nonlinerity of cmer nd projector must be djusted. Then, the cross-tlk between cmer chnnels nd projector ones cn be estimted. For this step lot of imges must be tken. We chose to snp the white ptch of Mcbeth under set of projected

4 colour one by one. In this cse, K is the unity mtrix. With corrected imges by previous pre-processing, we cn clculte A with lest squres minimiztion. IV. RESULTS AND ANALYSES We use 3-CCD colour cmer SONY DXC-91P nd presenttion projector Cnon LV The results for flt-field nd for nonlinerity, both for the cmer nd the projector, re shown. A. Cmer cse In this cse, the projector does not send ny imge. 1) Results for flt-field: the cquisitions re mde with exposure time 1/5s nd 1/125s. The cse with 1/125s gives sme FF coefficients. Direction of mbient light Exposure time: 1/5s. Fig. 4. FF coefficients for the cmer: representtion chnnel by chnnel in 3D surfce These surfces show verticl edges brightest thn the centre. This phenomenon comes from the position of the mbient light: t the top of the screen. FF1/5s FF1/125s C R C G C B 1 45 coeffmx coeffmin.79.8 men stndrd devition <%< Tble. 1. Different prmeters of the imges of cmer flt-field in two cses: exposure time 1/5s nd 1/125s (for the blue chnnel) In Tble. 1, FF sttisticl results re shown. The first three lines re the RGB vlues of the men of the FF imge. The men of coefficients is ner 1. In the cse of 1/125s exposure time, the mximum coefficient is high nd loctes on the edge. This coefficient enlightens n edge problem. For vlidting this model for the cmer, severl screens of different colour hve been cquired. We work gin without projection. We use coloured ppers. We snp successively these new screens. These imges re corrected with the cmer FF coefficients. The men (m) nd stndrd devition (sd) chnnel by chnnel (R, G nd B) of primry (prim) imges nd imge corrected with FF of the cmer (corff) re shown. In Tble 2, for the screen of colour light brown, the reder cn notice decresing of the stndrd devition in ech chnnel. The decresing is not on ll chnnels. We cn see wek decresing on the chnnel less importnt for the colour: red nd green for blue screen nd green for purple screen. light brown blue purple prim corff prim corff prim corff mr mg mb sdr sdg sdb Tble. 2. Vlidtion of our method of determintion of the cmer flt-field 2) Results for nonlinerity: In Fig. 1., we show theoreticl vlues vs. reel vlues nd four degrees polynomil form used for modeling these curves, in two cses: exposure times 1/5 nd 1/125. The correltion coefficient in ll cses is ner.99. The four degrees polynomil form is good form for modelling this dt. B. Projector cse 1) Results for flt-field: Fig. 5. Three exmples of cquisition for projector flt-field clcultion (FF1, FF2, FF3)

5 The projector flt-field cn not be determined without tking n imge with the cmer. The projector FF cn be determined with n imge of the projection rectngle nd the clculus of n homogrphy to hve FF coefficients on ech projected pixel. This sttement is vlidted with the cquisition of three different rectngles of projection: FF1, FF2 nd FF3, like in Fig. 5. Comprison between the three cses gives the influence of the homogrphy clculus. In ech three cses, one imge in red, in green nd in blue re projected. All imges re snpped successively. FF2 Fig. 6. FF coefficients for the projector: representtion chnnel by chnnel in 3D surfce FF1 FF2 FF3 C R C G C B coeffmx coeffmin men stndrd devition <%< Tble. 3. Different prmeters of the imges of projector flt-field in three cses, with exposure time 1/5s: FF1, FF2 nd FF3 FF2-FF1 FF2-FF3 Mx min men stndrd devition Tble. 4. Different prmeters of the difference between imges of projector flt-field with exposure time 1/5s: FF1, FF2 nd FF3 The surfces in Fig. 6 show tht the imge is brighter in the centre thn on edges. This phenomenon is clled igniting. Another phenomenon cn be noticed: the mjor prt of projector FF coefficients is up to the coefficient 1. The correction is mde like n enlightenment of the imge. In Tble. 4, clcultion of the difference between the coefficient imges FF1 nd FF3 compred to FF2 shows wek differences. Homogrphy hs not importnt influence. For the following clcultions, we chose the FF2. For vlidtion, one imge with one instruction, in this exmple: (1 1 1), nd one imge with the instruction (1 1 1) corrected by the inverse of the projector fltfield re projected. We wnt hve reel projection with the instruction (1 1 1) in ech pixel in out of the projector. For this, the projection instruction is clculted with the inverse of the projector flt-field. In Tble. 5. results re presented for the imge corrected by flt-field nd nonlinerities cmer. 111 corffp mr mg mb sdr sdg sdb Tble. 5. This tble shows the vlidtion of the projector flt-field with exposure time: 1/5s This Tble shows nother phenomenon: the blue vlue is higher thn the instruction. All cquisitions re mde under D65 illuminnt. This illuminnt simultes the dy light nd comprises lot of blue wvelengths. 2) Results for nonlinerity: We notice in Fig. 3. step from to 1 in theoreticl vlues. This step is chrcteristic of the projector. For presenttion projector, grey levels fewer thn 1 re not interested for humn eye. The vlue of the gmm correction of the projector modifies the length of the step nd the slope of the rest of the curve. Curves show tht the step is longer when the gmm correction increses. Remrk: the Fig. 3. is obtined with medi gmm corrections.. with gmm corrections b. without gmm corrections Fig. 7. Experiment with nd without gmm corrections: projector nonlinerity (exposure time: 1/5s)

6 V. CONCLUSION This rticle presents chrcteriztion protocol for 3D nd colour cquisition system. The protocol both for cquiring imges nd for processing chrcteriztion prmeters is described. The system, 3CCD cmer nd LCD projector, needs n ccurte knowledge of the colour response of ech set. With this colour model, stereo mtching of the colour coded structured light is esier. So, the 3D colour reconstruction of strong coloured object becomes esier. For future work, we will use methods like those used in mtching under uncontrolled mbient light [16]. Such methods void tking into ccount the mbient light. Becuse our stereoscopic system should llow determining both the colour nd 3D surfce of objects, we will test mtching lgorithms between projected ptterns nd cquired one. This mtching will llow tringultion step for 3D reconstruction nd finlly to ttch colour on ech reconstructed 3D point. A comprison of the results between rw imges nd pre-processed imges ccording to our chrcteriztion protocol should llow concluding bout the interest of the chrcteriztion described in this pper. VI. ACKNOWLEDGMENT This work is supported by the CNRS nd Burgundy region. VII. REFERENCES [1] R. Horud, O. Mong, "Vision pr ordinteur : outils fondmentux", Editions Hermès, 2 nde édition, 1995 [2] S. Chmbon, "Correltion-bsed mtching of color imges with occlusions", originl title: "Mise en correspondnce stéréoscopique d'imges couleur en présence d'occulttions", PhD Thesis, University of Toulouse III, Décembre 25 [3] L. Zhng, B. Curless, nd S. M. Seitz, "Rpid Shpe Acquisition Using Color Structured Light nd Multipss Dynmic Progrmming", In Proceedings of the 1st IEEE Interntionl Symposium on 3D Dt Processing, Visuliztion, nd Trnsmission, pp.24-36, Pdov, Itly, June 22 [4] J. Bttle, M. Mouddib, J. Slvi, "Recent Progress in Coded Structured Light s Technique to Solve the Correspondence Problem: Survey", Pttern recognition, vol.31, no.7, pp , 1998 [5] P.E. Debevec, C.J. Tylor, J. Mlik, "Modeling nd Rendering Architecture from Photogrphs: A hybrid geometry- nd imge bsed pproch", SIGGRAPH 6, pp.11-2, August 1996 [6] G.G. Slbugh, W.B. Culbertson, T. Mlzbender, M.R. Stevens, R.W. Schfer, "Methods for Volumetric Reconstruction of Visul Scenes", Interntionl Journl of Computer Vision, vol.57, no.3, pp , My 24 [7] R. Bribeu, M. Rioux, nd G. Godin, "Color reflectnce modelling using polychromtic lser rnge sensor", IEEE PAMI, vol.14, no.2, pp , Februry 1992 [8] A. Mnsouri, F. Mrzni, J. Hrdeberg, P. Gouton, "Opticl clibrtion of multispectrl imging system bsed on interference filters", Journl of Opticl Engineering, vol.44, no.2, pp , Februry 25 [9] D. Cspi, N. Kiryti nd J. Shmir, "Rnge imging with dptive color structured light", IEEE PAMI, vol.2, no.5, My 1998 [1] K. Brnrd, B. Funt, "Cmer Chrcteriztion for Color Reserch", Color reserch nd ppliction, vol.27, no.3, pp , June 22 [11] T. Mitsung, S.K. Nyr, "Rdiometric Self Clibrtion", IEEE, Proc. Of Computer Vision nd Pttern Recognition, vol.1, p.1374, June 1999 [12] K. Nyr, H. Peri, M. Grossberg, P.N. Belhumeur, "A Projection System with Rdiometric Compenstion for Screen Imperfections", Interntionl Workshop on Projector Cmer Systems (PROCAMS), Nice, Frnce, October 23 [13] E. Mrszlec, M. Pietikinen, "On-Line Color Cmer Clibrtion", Proc. 12th Interntionl Conference on Pttern Recognition, vol.1, pp , October 9-13, Jeruslem, Isrel, 1994 [14] A. Mnsouri, F.S. Mrzni, P. Gouton, "Systemtic noise chrcteriztion of CCD cmer: ppliction to multispectrl imging system", Complex Systems Intelligence nd Modern Technologicl Applictions (CSIMTA), Specil Session on Color Imge Processing nd Anlysis for Mchine Vision, pp , Cherbourg, Frnce September 24 [15] S. Woo, A. Dipnd, F. Mrzni, Y. Voisin, "Determintion of n optiml configurtion for direct correspondnce in n ctive stereovision system", Proc. on 2nd IASTED INT. Conf. VIIP, pp , Mlg, Spin, September 22 [16] D. Muselet, "Object recognition by nlysis of color imges cquired under uncontrolled illumintions", originl title: "Reconnissnce utomtique d'objets sous éclirge non contrôlé pr nlyse d imges couleur", PhD Thesis, University of Lille I, Juillet 25