A high-resolution and high-speed 3D imaging system and its application on ATR

Transcription

1 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR homas. Lu and en-hsn Chao Jet Propulson Laborator/Calforna Insttute of ehnolog, 48 Oak Grove Dr. M/S 33-3, Pasadena CA 99 ABSRAC he paper presents an advaned 3D magng sstem based on a ombnaton of stereo vson and lght projeton methods. A sngle dgtal amera s used to take onl one shot of the objet and reonstrut the 3D model of an objet. he stereo vson s aheved b emplong a prsm and mrror setup to splt the vews and ombne them sde b sde n the amera. he advantage of ths setup s ts smple sstem arhteture, eas snhronzaton, fast 3D magng speed and hgh aura. he 3D magng algorthms and potental applatons are dsussed. For AR applatons, t s rtall mportant to etrat mamum nformaton for the potental targets and to separate the targets from the bakground and lutter nose. he added dmenson of a 3D model provdes addtonal features of surfae profle, range nformaton of the target. It s apable of removng the false shadow from amouflage and reveal the 3D profle of the objet. It also provdes arbtrar vewng angles and dstanes for tranng the flter bank for nvarant AR. he sstem arhteture an be saled to take large objets and to perform area 3D modelng onboard a UAV. Kewords: Stereo vson, olor gratng projeton, 3D magng, modelng, AR.. INRODUCION We lve n a three dmensonal (3D) world. But most data we reord and dspla are stll n D or D forms. he 3 rd dmenson, the depth nformaton s often lost. In man ases, the depth data s rtal to the measurement of the objet surfae shape and profle, the perepton of the objet dstane, and t s mportant to ad the human judgments. In reent ears the advanement of omputer tehnolog has made tremendous progress n measurng, modelng, and vsualzng 3D objets. hs evoluton auses an mportant demand for more omple and realst 3D models. he problem s that sntheszng realst models s dffult and tme-onsumng, and thus epensve. Researhers have been nvestgatng methods to aqure 3D nformaton from objets and senes for man ears -. here are several tehnques for 3D measurements.. Laser sannng tehnques he laser sannng tehnque uses a narrow laser beam or pont to san the objet surfae. A vdeo amera aptures the deformed laser lnes or dots refleted from the objet surfae. he omputer alulates the dsplaement of the laser lne from the objet surfae aganst a straght lne. hen the surfae shape an be plotted 3. he laser sannng method s a robust and aurate 3D modelng tehnque. It s wdel used n ndustral nspeton and rapd prototpng. However, t s generall slow and not sutable for lve or fast movng objets.. Lght projeton tehnques he phase shft, Moré, olor or varable gratng projeton methods, et. belong to ths ategor 4, 5. he methods have grd patterns projeted on the objet, etrat the grd or phase deformaton from the refleted mage and dedue the surfae profle of the objet. he are generall faster than the laser sannng method, but the aura suffers when the surfae ondton s omple (varable teture olor, llumnaton, or surfae dsontnut, et.).3 Stereo vson hs method s smlar to ts bologal ounterpart - human vson. wo ameras fous on the objet from two vewng angles 6-8. he dsplaement of the objet features n the two mages represents the depth hanges of the objet. e-mal: homas..lu@jpl.nasa.gov, el: (88) , Fa: (88) Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6

2 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR herefore, the objet surfae an be modeled aordng to the dspart of the two mages. Man stereo vson tehnques have been developed that snhronzes multple ameras to aqure three-dmensonal models of objets 9,. One of the stereo vson methods, the photogrammetr has been dealng wth the etraton of hgh aura measurements from mages. hese tehnques mostl requre prese albraton and there s almost no automaton. he detaled aquston of models s therefore tme-onsumng. he stereo vson tehnques have been used wdel n the robot vson applatons. One an buld a 3D area surfae model from a sequene of D mages taken b dgtal and vdeo ameras. he user aqures the mages from the hgh resoluton ameras. Nether the amera moton nor the amera settngs have to be known. he obtaned 3D model s an aurate model of the ground surfae, and the surfae albedo s obtaned from the mage sequene as well. hs approah has been eplored over the last few ears. However, there were stll requrements of omple albraton, user nteraton, and etended omputatons. he man problems assoated wth these methods are hgh omputatonal requrement and low resoluton and aura due to the lak of relable features on the objet surfae..4 me of flght methods Usng short pulse laser to measure the tme of flght from the laser soure to the objet has beome popular n remote sensng applatons. In the remote sensng ommunt, radar and laser rangng have been used to measure the 3D geometr of the earth. It s stll epensve to operate. he paper desrbes a 3D magng tehnque that performs fast and aurate 3D magng usng a sngle dgtal amera to perform stereo vson wth an optonal olor-gratng projeton.. SYSEM ARCHIECURE We present a 3D magng sstem that uses a prsm optal setup to ombne two stereo vew pars nto one mage n a dgtal amera. he 3D magng tehnolog ombnes the strutured lght projeton and stereo vson tehnques n a unque wa: t splts a sngle amera vew nto stereo vson. It has an opton to projet a olor-oded struture lght onto the objet usng a snhronzed flash lght soure 3. It aheves 3D magng n a sngle flash (< / th seond), thus robust to objet moton and hangng envronment. It s also aurate, down to. mm n depth resoluton. We also present advaned 3D modelng algorthms for fast and aurate 3D model generaton. he sstem nludes followng omponents, as llustrated n Fgure.. A ommeral off-the-shelf dgtal amera aptures D olor mages of objet;. A mrror and prsm based stereo deve ombnes two stereo vews nto a sngle mage; 3. An optonal flash projetor projets a olor-oded pattern onto the objet to mprove the 3D resoluton and aura. he flash projetor s snhronzed wth the amera shuttle. 4. A albraton board to albrate the nternal and eternal parameters of the 3D sstem. 5. Software reates the 3D model for a stereo mage par. Mrror Dgtal Camera Optal Prsm Vrtual Camera Mrror D Z Mrror Objet Dgtal Camera Optal Prsm H Pattern Projetor Mrror Vrtual Camera (a) (b) Fgure : A sngle dgtal amera based 3D magng sstem for hgh-resoluton and hgh-speed 3D magng: (a) Illustraton of a optal mrror and prsm setup; (b) Shemat dagram of the equvalent vrtual ameras stereo vson. Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6

3 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR 3. 3D IMAGING ALGORIHM 3. 3D representatons An ee at pont C vews a pont M n the world oordnate. he pont M projets a pont m on the retnal plane R. For a world pont M= (X, Y, Z) and the orrespondng mage pont m= (, ). Usng the homogeneous representaton of the ponts a lnear projeton equaton s obtaned: ~ X Y Z hs projeton s llustrated n Fgure. he optal as passes through the enter of projeton and s orthogonal to the retnal plane R. Its nterseton wth the retnal plane s defned as the prnpal pont Camera model An atual amera has a foal length f, where f s defned as the dstane between the enter of projeton and the mage plane. Wth a CCD amera the relaton between the oordnates n the mage and the phsal oordnates depends on the sze and shape of the pels and of the poston of the CCD hp n the amera. M () R m C Fgure : An ee at poston C vews a pont M n the world. he pont M projets a pont m on the retnal plane R. he mage oordnates are obtaned through the followng equatons: f s f R R () where f and f are the foal length measured n wdth and heght of the pels, = [ ] s the prnpal pont, and s a fator aountng for the skew due to non-retangular pels. he above upper trangular matr s alled the albraton matr K of the amera. So, the followng equaton desrbes the transformaton from retnal oordnates to mage oordnates. m = Km R (3) Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6 3

4 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR 3.3 Projeton matr he followng epresson s obtaned for a amera wth spef ntrns albraton and wth a spef poston and orentaton: f ~ s f R X R t Y Z 3 (4) whh an be smplfed to m ~ PM (5) where R s a rotaton matr and t = [t t t z ] a translaton vetor. he 3 4 matr P s alled the amera projeton matr. [] 3.4 wo vew geometr Gven an mage pont n one mage t restrts the poston of the orrespondng mage pont n another mage, as shown n Fgure 3. hs relatonshp an be obtaned from the albraton or even from a set of pror pont orrespondenes. Although the eat poston of the sene pont M s not known, t s bound to be on the lne of sght of the orrespondng mage pont m. hs lne an be projeted n another mage and the orrespondng pont m s bound to be on ths projeted lne l. l and l are sad to be n eppolar orrespondene (.e. the orrespondng pont of ever pont on l s loated on l, and ve versa). R M? L C m l e e R m? l C Fgure 3: Correspondene between two vews. Even when the eat poston of the 3D pont M orrespondng to the mage pont m s not known, t has to be on the lne through C and C whh ntersets the mage plane n m. Sne ths lne projets to the lne l n the other mage, the orrespondng pont m should be loated on ths lne. More generall, all the ponts loated on the plane defned b C and C have ther projeton l on and l. Ever plane passng through both enters of projeton C and C results n suh a set of orrespondng eppolar lnes, as an be seen n Fgure 5. All these lnes pass through two spef ponts e and e. hese ponts are alled the eppoles, Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6 4

5 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR and the are the projeton of the enter of projeton n the opposte mage. hs eppolar geometr an also be epressed mathematall: ' m Fm (6) hs matr F s alled the fundamental matr. When the albraton s not known, equaton (6) an be used to ompute the fundamental matr F. Ever par of orrespondng ponts gves one onstrant on F. he rank F =, therefore 7 pont orrespondenes are suffent to ompute F through nonlnear algorthms Camera aberraton Pratall, the amera model has some error aused b the optal aberraton and msalgnment of the CCD hp to the lenses. he lens aberraton an be represented b ' ' ( ( ) k ) k ( ( ) k ) k ( ( ) ) (7) where and are the deal pont poston, and aberraton parameters. ' and ' the atual pont poston. k and k are the amera lens 3.6 Camera albraton he amera albraton proedure alulates the eternal parameters R, t, and the nternal parameters K. In order to alulate the parameters, we map the known ponts M,,... N n the 3D spae to the orrespondng ponts m,,... N n the D mages. We reeve the followng equaton: R tm,, N m K,... (8) If we fnd enough number of pont pars, then we an alulate the nternal and eternal parameters. In general, the ponts need to be n dfferent planes. hat requres buldng a speal albraton objet. We used a smplfed flat plate wth hess board pattern on t. We take 3 5 shots of the albraton board wth the 3D amera wth the board n dfferent planes, as shown n Fgure 4. he orner ponts of the squares on the hess board are deteted b the software. he geometral relatonshp among the ponts on the same board s known. hen, we an alulate the parameters from the equaton. he parameters are further optmzed b applng the Bundle Adjustment algorthm (a) (b) Fgure 4: Calbraton proedure: (a) a hessboard pattern; (b) 5 postons of the hessboard for the amera albraton. Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6 5

6 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR 3.7 Image retfaton After we have obtaned the amera nternal parameters, we an retf the mage. he retfaton proess an be arred out from Eq. 8. After retfaton, the 3D pont pars are on the same parallel lnes between two stereo mage pars, as shown n Fg. 5. herefore, t saves tme for the stereo mathng proess to shft n one dmenson onl. (a) Fgure 5: After retfaton, the stereo pont pars are on the same eppolar lnes; (b) a 3D model wth olor teture. (b) 4. EXPERIMENAL RESULS We have used the 3D magng sstem to take lve and stat objets. he model szes var from ~ m to meter. A dgtal amera (Olmpus dgtal amera) wth 5 Mega pels was used. he lens foal length was mm. he amera shuttle speed was /5s. Sngle shot was taken for eah mage vew. Software was mplemented to download the mages from the dgtal amera va a USB lnk, to albrate the amera parameters, to reonstrut the 3D models, to stth the models together, and to dspla the 3D models. Fgure 6 shows the 3D reonstruton of the fae of a model. A olor-oded grd pattern was flashed onto the surfae as the amera aptures the stereo mage par. he software reonstruted the 3D model wth detals of the faal features learl seen. Smoothng of the surfae gves a better vsual appearane, but sarfes the resoluton. (a) (b) () Fgure 6: 3D magng of the supture of a grl: (a) amera stereo vew pars under projeted lght; (b) reonstruted 3D model; () 3D model after surfae smoothng. Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6 6

7 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR Fgure 7 shows the 3D magng of a ruse shp model Queen Mar. he model s ~. meter long. Appromatel 8 shots were taken from 36 degree vews. he 3D models are stthed together to form a omplete 3D model. (a) (b) () Fgure 7: 3D magng of models: ruse shp Queen Mar model was maged n 36 degrees; (a) () multple vews were stthed together to form a omplete 3D model; Fgure 8 llustrates the 3D magng and modlng of several mltar models, a Patrot tank, and a hunderwng fghter jet. From Fgures 8(a) (), we an see, the false shadow of the amourage an be removed and the true surfae struture an be revealed b usng the 3D magng method. It provdes valuable 3D nformaton of an objet for AR applatons. he 3D model an be rotated and saled to arbtrar vewng angle and sze to math the target orantaton and sze n the mage. It s also an eellent model for buldng tranng database for preson target reognton 5-6. (a) (b) () Fgure 8: 3D magng of models: (a) a Patrot tanks sde vew (top) and ts underlng 3D struture (bottom); (b) the sde porton of a hunderwng fghter jet (top) and ts 3D model (bottom), and () the top 3D vew of the hunderwng wth olor teture map. 4. Varaton of amera feld-of-vew to aommodate objet szes and dstanes he above eperments were arred out n the laborator usng a standard dgtal amera wth a lens of mm foal length. It s albrated to mage an objet of ~ 3 3 m n a dstane ~ meter. he dstane between the vrtual amera par s ~ m. If an objet s larger than 3 m n sze, several partal mages ma be taken and then stthed together. he advantage of ths approah s the hgh resoluton of the reonstruted models. But the stthng of the models b the software ould ntrodue sstemat error n the model. If the objet s muh larger than the urrent one, then we need to pull the amera further awa from the objet, and separate the ameras further. he role of thumb of the vewng angle between two ameras s ~ 5 degrees. For eample, the ameras an be plaed meters awa from an objet. he amera separaton would be ~ 4 meters. he objet sze an be n the order of 6 6 m. In the UAV applaton of magng the sene and reonstrut the 3D world vew, we ma use onl one amera on-board the UAV to ontnuousl mage the sene as the UAV moves n ts ourse. he neghborng mages n the sequene form stereo pars. he amera nternal and eternal parameters an be estmated based on fed feature ponts on the Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6 7

8 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR ground. Eah stereo mage par an form a 3D model of the sene t sees. he omputaton s ver ntensve n ths applaton. Optmzaton and dgtal FPGA parallel proessng board ma be used to aheve real-tme proessng. 4. Aura test eperment hs eperment tests the error and nose effet of the 3D algorthm on a flat surfae. he flatness of the plane also represents the resoluton of the 3D objet. When we smooth the surfae, the flatness s nreased, but the resoluton s redued. he 3D amera takes a pture of a flat surfae. he 3D model s reonstruted. A Lease-mean-square (LMS) algorthm s used to onstrut a flat plan and alulate the dstane from the ponts of the 3D model to the flat plane. he square root error represents the flatness measure. Assume N ponts n spae; P,, z, (,,... N); For a pont on the 3D model P, the error In matr form: assume the are on a plane s Use LMS method, we an alulate the parameters a, b, and : z a b (9) z a b,,,... N () Z AX () A ( X E E Z X ) X Z ( X Z Z) A (3) he mean error n z dreton s he surfae area error s E E / N (4) /( a b ) (5) p We use samples n the enter, and the 4 orners to represent a large surfae. Aordng to the stereo magng theor, f the amera dstane s d, the dstane from the amera to the objet s z, the amera foal length s f, and then the 3D resoluton s dz du z f d (6) If the amera parameters are z = 5.mm, d = 97.9mm, f = 57, then the theoretal resoluton s.546 (mm/pel). A sub-pel mathng algorthm an be used n the 3D alulaton; the mnmum resoluton s nreased between /3 to / of a pel. herefore the resoluton s nreased to.546/3 =.55(mm), and the mean error s dereased to.55/3 =.7mm. Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6 8

9 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR he test result s shown n table I. estng ponts were sampled from fve areas of the plate: the upper left, upper rght, enter, lower left, and lower rght regons. he average error was alulated and t shows the error was as low as.4 mm n an area of 3 3 m plate. 3D Model Error (mm) Unsmoothed Flat Plane Smoothed Flat Plane Upper Left Upper Rght Center Lower Left Lower Rght Average able I: Eperment results of the aura of a 3D model of a flat plane. he average mean error s ~.4 mm. 4.3 Potental applatons 3D vson an be used to mprove the robustness of automat target reognton. he 3D magng tehnque enables real-tme 3D aquston of 3D features and profles that a regular amera an not aheve. A 3D model of a target gves the omplete nformaton, e.g., the phsal szes, surfae profle, llumnaton/shadng, the vewng angles and the dstane nformaton that an not be gven b the onventonal magng modalt. hese are rtal nformaton that an tremendousl help target deteton, reognton and dentfaton. It s also a valuable tool to dgtze a model for the onstruton of a flter bank of targets for target ID. It performs quanttatve measurement of target features and 3D vsualzaton of objets. Other applatons nlude 3D faal ID for seurt survellane. he 3D sstem an be adapted b the estng setup of a sngle dgtal amera to apture the 3D fae model n stead of a D mug shot. It an redue the error aused b the ambent lght, vewng angles and the make up for robust bometr ID. In medal feld, the 3D magng sstem an be used to ad the real-tme surgal gudane. It an be used to perform 3D modelng, smulaton and trakng of lve human bod. It an be used to regster the patent s bod wth ultrasound, MRI and C mages. It has potental applatons n plast and reonstrutve surger - 3D modelng, smulaton and vsualzaton and quanttatve measurement of bod parts; t has potental applaton n non-ontat burn patents faal mask generaton n burn enters. In ndustral applatons, t an be used n mahne vson, real-tme 3D magng and nspeton, rapd prototpng, and 3D magng and modelng of mahne parts. 5. CONCLUSIONS We have demonstrated a fast and aurate 3D magng sstem usng a sngle dgtal amera. he sstem ombnes the stereo vson and the olor-oded gratng projeton methods to aheve fast and robust 3D magng. he aura of the 3D magng sstem an be. mm n an area of 3 3 m area. he sstem an be saled to mage large objets. he sstem algorthm an also be ustomzed to proess mage sequenes from a vdeo amera onboard a UAV. he sstem has potental applatons n a broad feld. 6. ACKNOWLEDGEMENS he researh desrbed n ths paper was arred out b the Jet Propulson Laborator, Calforna Insttute of ehnolog, under a ontrat wth the Natonal Aeronauts and Spae Admnstraton. he authors wsh to thank Frank Wu and John J. Zhang for valuable nput and software mplementaton. Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6 9

10 A hgh-resoluton and hgh-speed 3D magng sstem and ts applaton on AR 7. REFERENCES. J. H. Brunng, D. R. Herrott, J. E. Gallagher, D. P. Rosenfeld, A. D.Whte, and D. J. Brangao, Dgtal wavefront measurng nterferometer for testng optal surfae and lenses. Appl. Opt., 3: , B. Horn, Robot Vson, MI Press, M. Levo, J. Gnsberg, J. Shade, D. Fulk, K. Pull, B. Curless, S. Rusnkewz, D. Koller, L. Perera, M. Gnzton, S Anderson, J. Davs, "he Dgtal Mhelangelo Projet: 3D Sannng of Large Statues," Proeedngs of the 7th annual onferene on Computer graphs and nteratve tehnques, pp.3-44,. 4. G. J. Agn and P.. Hghnam, "Movable Lght-Strpe Sensor for Obtanng hree-dmensonal Coordnate Measurements," SPIE Pro. Vol. 36, p. 36, K. Hardng, "More Interferometr for Industral Inspeton," Lasers and Applatons, p. 73, Nov O. Faugeras and G. osan, Camera Calbraton for 3D Computer Vson, Internatonal Workshop on Mahne Vson and Mahne Intellgene, pp. 4-47, oko, R. Hartle, Estmaton of relatve amera postons for unalbrated ameras, Computer Vson -ECCV 9, Leture Notes n Computer Sene, Vol. 588, Sprnger-Verlag, pp , O. Faugeras, hree-dmensonal Computer Vson: a Geometr Vewpont, MI press, P. Beardsle, P. orr and A. Zsserman 3D Model Aquston from Etended Image Sequenes, Computer Vson - ECCV 96, Leture Notes n Computer Sene, Vol. 65, Sprnger-Verlag, pp , R. Koh, M. Pollefes and L. Van Gool, Automat 3D Model Aquston from Unalbrated Image Sequenes, Proeedngs Computer Graphs Internatonal, pp , Hannover, C. Slama, Manual of Photogrammetr, Ameran Soet of Photogrammetr, Falls Churh, VA, USA, 4th edton, 98.. N. Haala, and C. Brenner, Generaton of 3D t models from arbrone laser sannng data, Proeedngs of EARSEL workshop on LIDAR remotesensng on land adn sea, alln, Estona, pp. 5-, Lu, J. Zhang, 3 Dmensonal Imagng Sstem, U.S. patent No. 6,5,63,. 4. Z. Zhang, R. Derhe, O. Faugeras and Q.-. Luong, A robust tehnque for mathng two unalbrated mages through the reover of the unknown eppolar geometr, Artfal IntellgeneJournal, Vol.78, pp.87-9, Otober J. C. Hanan,.-H. Chao, C. Assad, C. L. Hughlett, H. Zhou,. Lu, Closed-Loop Automat arget Reognton and Montorng Sstem, p. 44-5, Optal Pattern Reognton XVI; SPIE 586, Lu, C. L. Hughlett, H. Zhou, -H. Chao, J. C. Hanan, Neural network post-proessng of grasale optal orrelator, Pro. SPIE 598, Optal Informaton Proessng III, 5. Pro. SPIE 645, SPIE Smposum on Defense and Seurt, Optal Pattern Reognton XVII, 6