Real time depth mapping performed on an autonomous stereo vision module

Transcription

1 1 Real tme depth mappng performed on an autonomous stereo vson module Jeroen Smt 1, Rchard Klehorst 2, Anteneh Abbo 2, Jan Meuleman 1 and Gerard van Wllgenburg 1 1 Wagenngen Unversty, Bornsesteeg 59, 6708 PD Wagenngen, The Netherlands 2 Phlps Research Laboratores, Prof. Holstlaan 4, 5656 AA Endhoven, The Netherlands E-mal: rchard.klehorst@phlps.com Abstract Keywords: stereo vson, real-tme, depth estmaton At the moment many applcatons such as robot navgaton, obstacle avodance, 3D user nterfaces are lmted n ther capabltes due to lmted avalablty of 3D measurement systems. Therefore, research n stereo magng s a hot topc, because t offers a multlateral and robust way to reconstruct lost depth nformaton. The few avalable state of the art stereo vson solutons have dsadvantages such as cost, sze, nflexblty, hgh power consumpton and are often ncapable of makng depth maps n real tme. In ths publcaton a new low cost, autonomous, stereo vson module whch computes a real tme (30 fps) dense depth map n VGA format ( ), s presented. The module houses two uncalbrated off-the-shelf CMOS sensors, and a massve parallel programmable processor (Xetal). Due to the parallelsm the processng performance of the actual stereo matchng algorthm s hghly effcent. I. INTRODUCTION The research on obtanng depth nformaton of a real world scene s very actve [1] [2] [3] [4] [5] [6], because applcatons where depth or dstance clues could be appled are ncreasng. Depth estmaton s one of the most vtal vsual tasks whch humans can do almost effortlessly whle for computers t s a dffcult and challengng task. Nowadays most depth estmaton systems don t perform depth maps n real tme. However, these systems are n hgh demand, especally lookng at the latest developments of consumer robotcs, moble telecommuncatons and 3D user nterfaces applcatons. Some avalable systems are able to process depth estmaton n real tme, but these are expensve, power consumng and large, whch s undesrable for the last mentoned applcatons. Therefore n ths publcaton we want to show that t s possble to acheve real tme depth mappng on a cheap, flexble, small and low power module. Depth estmaton by vson can be performed n several ways such as moton sensng, tme-of-flght of (nfrared lght) and stereo vson: 1. Depth sensng by moton has an advantage that only one camera s requred to obtan depth nformaton from mage sequences [4]. On the other hand, the system s rather lmted because exact movement of the camera wth respect to the scene or dmensons of the objects must be known. Therefore, ths method s hghly senstve for an ncorrect depth estmaton. Also only the relatve depth s measured and objects or camera have to move. 2. Snce (nfrared) lght travels essentally at a constant speed, f one knows the elapsed travel tme, the dstance to a certan feature can be computed [7] [8]. In other words, t s possble to develop a 3D map of the surfaces n the scene. A bg dsadvantage of these systems s that they are senstve for dfferences n object reflectances. 3. Stereo vson provdes the most multlateral and robust way to reconstruct lost depth nformaton. It reles on one fundamental fndng: f two shots of a gven scene are captured from two dfferent vewponts, then the resultng mages wll dffer slghtly due to the effect of a perspectve projecton. The correspondences of the stereo par can be used effectvely to reconstruct the three-dmensons of the scene, va a procedure known as stereo matchng. The dstance that the coordnates of an object n one mage are shfted wth respect to the same object n the other mage, relatve to ts local coordnate system, s expressed as a dsparty, and ths s the fundamental measure requred to reconstruct a scene. In ths research we have chosen stereo vson, because of above mentoned advantages. Besdes that, a proper sensor setup and a hardware confguraton s requred to mnmze the effect of several potental sources of errors, that makes locatng correct mage pars dffcult: 1. Occluson problems can occur because projecton takes place from dfferent vewponts. A symmetrcal arrangement of more than two cameras, helps to reduce the effects of occluson substantally [9]. However, these are more expensve and dmensons of these systems are nevtably greater. 2. Symmetres presented n a stereo par gves multple 306

2 potental correspondents for a gven pxel and leads to ambguous matches. Also, objects wth a monochrome surface are only detected correctly at the edges. Projected magery from a lght source, such as a Dgtal Lght Projector (DLP), can be used to artfcally create a structure onto the objects [10]. On the other hand these artfcal features are subject to change wth varatons of objects and envronmental condtons. 3. Changes of ntensty of the same 3D pont n a stereo par may occur due to the dfferent vewng postons. 4. Accuracy s lmted to the resoluton and depth of nput data. Moreover, above mentoned dffcultes are all hghly dependent on lens dstortons and sensor chps calbraton. When these sensor systems are uncalbrated, they can dstort 2D nput data sgnfcantly [11]. Snce, horzontal or vertcal dsplacement, yaw, ptch, roll, radal and tangental lensdstorton n uncalbrated systems could occur, stereo matchng objects wth structure could become mpossble and ncorrect depth estmatons and dsplacement errors occur wth monochrome objects. Therefore many researchers calbrate ther systems and can make use of the eppolarty constrant [9] [12] [3]. Because, accurate calbratng of sensor systems s a tme consumng and elaborate assgnment and therefore less sutable for consumer applcatons. We choose to challenge uncalbrated sensor systems n ths paper. Intally we detect depth at the vertcal edges of objects, regardng there s an eppolar lne for vertcal structures and subsequently fll n the objects n a second phase. Besdes the nevtable dffcultes regardng the sensor setup and confguraton, especally the processng requred for real-tme stereo matchng [1] [2] [3] [12], has lmted the desgn of small real-tme stereo vson modules. In ths paper we deal wth these challenges and present a new low cost, autonomous, stereo vson module whch computes a real-tme (30 fps) dense depth map n VGA format ( ). The power of ths module les n the fact that we make use of a Sngle Instructon Multple Data (SIMD) processor and specally desgned and optmzed algorthms. The contents of ths paper are reflected n the tasks: In Secton II the choce of hardware s motvated. The used stereo vson algorthms and ts performance are descrbed n Secton III and Secton IV respectvely. The conclusons and future work are dscussed n Secton V. II. HARDWARE ARCHITECTURE As mentoned before, stereo matchng s very cumbersome for general purpose computers and processors. Not only because of the computatonal effort, but also because of the data rates and electrcal power nvolved. Therefore we used the Xetal chp [13] n our stereo vson module. Besdes the Xetal chp, the stereo vson module houses two uncalbrated off-the-shelf VGA ( pxels) 10-bts RGB CMOS mage sensors, whch results n a powerful stereo-sensor-dsp combnaton. The Xetal chp (Fgure I) s desgned especally for hgh-performance pxel processng n magng applcatons. Xetal s a programmable dgtal vdeo IC wth a massve parallel processor. Fgure I: Xetal Desgn Ths chp conssts of a Lnear Processor Array (LPA) wth 320 Processng Elements (PE). Each PE houses an Arthmc Logc Unt (ALU) and a Multply Accumulator (MAC). The nput data for the PE s receved through a 10 bts bus. Each PE s assgned to two mage columns wth the possblty to addtonally access two left and two rght neghborng pxels. Ths allows for rght and processng element pxel wrte access read access Fgure II: Processng Element Layout left drect addressng of sx pxels (Fgure II). Snce all processor elements share the same decodng logc t s possble to smultaneously execute one LPA nstructon on all 320 elements. The LPA performs all the DSP operatons on the data stored n the lne memores. The Global Controller (GC) performs tasks such as condtonal executon, teraton and synchronzaton. The statstcal computatons are performed by the seral processor and are used to control the sensor parameters or to update flter coeffcents. The mage or vdeo nput data s a VGA sze frame (matrx) wth up to 10-bt dgtzed sgnals at a max- 307

3 mum rate of 30 frames/second. Xetal can not receve two dfferent 10 bts sgnals at the same tme. Therefore hardware mxes the two sensor nput sgnals of 10 bts nto one sgnal of 10 bts. Xetal has 16 lne memores for temporary storage and 4 sequental lne memores for nput-output purposes. Each lne memory holds 640- pxels at 10-bt resoluton. The nterface to the vdeo nput and output s acheved va a sngle-channel-nput and a three channel-output port. The I2C nterface s used to download program code to the chp. Because of the fully parallel archtecture, hgh performances and data-rates can be acheved for a modest power consumpton. The power consumpton manly depends on the mage parameters,.e., the number of rows per frame, the frame rate, and on the program that runs on the parallel processor. Ths can go down to 30 mw for smple applcatons such as a dgtal camera for vdeo conferencng. For more sophstcated applcatons, such as stereo depth mappng t can go up to 200 mw. III. STEREO VISION ALGORITHM OUTLINE The outlne of the complete stereo vson algorthm s shown n Fgure III. A. Separatng Snce the hardware mxed the two 10 bts RG B nput sgnals nto one 10 bts RG B nput sgnal, the sgnal must be separated to obtan the left and rght mage. The upper 5 bts of the mxed sgnal belong to the left sensor and the lower 5 bts to the rght sensor. B. Convert RG B nto Y It s not desrable to perform a matchng procedure n all three color spaces [14]. Therefore the RG B sgnal s converted nto the lumnance (Y ). In ths research we perform the stereo matchng algorthm only on the ntensty (Y ) of the nput sgnals. C. Low-pass flterng A low-pass flter s used to reduce the effects of nose and to cancel out large varatons from pxel to pxel, because we are not relyng on the eppolar lne prncple. Ths routne performs a 5 1 low-pass flter wth an equal wegth of 0.2 and not for example a 5 5. D. Stereo matchng The fundamental problem n stereo vson s that of locatng correspondng or matchng ponts n the two mages. We used a 6 pxels wde area based matchng algorthm, whch s characterzed by the fact that t compares RGB left RGB2Y Y left low pass mxed sensor sgnal RGB separatng sgnals matchng mn flter fllng dense depth map RGB rght RGB2Y Y rght low pass sparse depth map Fgure III: Algorthm outlne wndows of pxel values n the two mages, n order to fnd the best match. We compute the Sum of Absolute Dfferences (S AD) (1) for each wndow between the left and rght mage by shftng one mage lne a preset number of pxels. Because we look for domnant vertcal features, t s not necessary that one has to search for a certan pxel from 308

4 the left mage n the complete rght mage. A pont to be matched essentally, becomes the centre of the wndow of pxels, whch s compared wth smlarly szed wndows n the other mage. Matchng measures were used to provde a numercal measure of the smlarty between a wndow of pxels n one mage and a wndow n another mage, and hence are used to determne the optmmum match. There are two smple matchng measures whch are sutable for hardware mplementaton, the S AD (1) and Zero mean Sum of Squared Dfferences (SS D). Prevous research showed that the ntroduced complexty n the S S D matchng measure does not gve any sgnfcant mprovement n matchng qualty [15]. where Y r S AD = N =1 Y r Y l, (1) and Y l are respectvely the ntensty (Y ) of the rght and left pxel. Snce the processor s parallel, each computaton s performed at the same tme for each pxel on the current mage lne. When the actual S AD s smaller then the prevous stored smallest S AD at the specfc locaton, the value wll be overwrtten by the actual S AD and correspondng number of pxel shfts. After all, the number of pxel shfts at the lowest S AD per pxel, represents the depth of an object. Due to ths drect lnear relaton, absolute depth nformaton can be easly derved by trangulaton methods. Table I: Relatve number of nstructons for the stereo matchng routnes per lne tme (n %) task load separatng sensor sgnals 8% RGB to Y converson 5% low-pass flterng 3% stereo matchng 64% mnmum flter 3% object fllng 10% control 7% be correctly matched. We also experenced that a depth resoluton of only 5 bts s lmtng the stereo matchng performance of objects. Ths s due to the fact that mnor real world color dfferences have been canceled out by the 10 to 5 bts converson, whch causes naccurate ntensty levels. Our dsparty search range s 19 pxels, whch results n our specfc setup that we can detect objects as close as 1 m to the stereo sensors, ndependent on the sze of the object. The dsparty range s lmted because of the amount of nstructons per lne tme. The actual stereo matchng routne requres more than half of the avalable nstructons per lne tme (Table I). E. Mnmmum flter A mnmum (gray value eroson) flter s used to reduce the effects of small pxel regons whch contan false depth nformaton. The mnmum flter replaces the current depth value of a pxel by the mnmum depth value wthn a wndow of 3 3 pxels. F. Object fllng Optmal stereo matches are only found at vertcal edges of or n the objects. The found sparse depth map has to be flled n. Fllng of objects n a scene s a dffcult task for a PC or processor. Some complex methods exst for object fllng [16]. IV. MEASUREMENTS AND PERFORMANCE A. Stereo matchng performance We observed that the uncalbrated CMOS sensors especally had a sgnfcant horzontal offset (of several lnes). Snce we are makng use of a vrtual eppolarty constrant only vertcal edges of monochrome objects can Fgure IV: Setup overvew: The larger fgure shows the setup whle the screenshot shows the real tme depth map. From the ntensty of the pxels the dstance to the camera can be obtaned. For nstance, the larger (and further) s slghtly darker than the smaller object, whch s closer. The two objects at the wall are even further away, therefore even darker. 309

5 V. CONCLUSIONS AND FUTURE WORK Depth estmaton s becomng an mportant applcaton n several consumer applcatons. However, only few avalable solutons exst and these have dsadvantages such as cost, sze, nflexblty, hgh power consumpton and are often ncapable of makng depth maps n real-tme. Tll now, especally the processng requred for complete real tme stereo matchng prohbted ntegraton of the whole applcaton nto a flexble, small szed, cheap, autonomous, low-power module. Ths publcaton shows that ths ntegraton can be acheved by: Makng an adequate choce for the processng archtecture, Desgnng and optmzng algorthms specfc for the selected archtecture and purpose. As a result we present a low cost, autonomous, stereo vson module whch computes a real-tme (30 fps) dense depth map n VGA format ( ). Despte some hardware lmtatons such as lmted number of ntensty levels and uncalbrated sensor systems, very promsng results have been shown. Applcatons such as object avodance, navgaton, 3D user nterfaces and detectng objects of nterest can take the benefts from ths new module. Future research wll focus on further development of the algorthms, e.g. mprovng the stereo matchng and object fllng algorthms. However, adjustments on the algorthms are dependent on the qualty of the nput data from the mage sensors and especally the avalable processng archtecture. Acton wll be taken to obtan more bts from the sensors, ncrease the dsparty range and to see the effects of low cost calbraton. mages, Pattern Recognton Letters, vol. 25, no. 4, pp , [7] Canesta, Canesta nfrared sensor chps [8] A. B. Robert Lange, Peter Setz and S. Lauxterman, Demodulaton pxels n ccd and cmos technologes for tme-of-flght rangng, n IEEE Vrtual Realty ProceedngsIST/SPIE Internatonal Symposum on Electronc Imagng, January [9] T. Kanade, A. Yoshda, K. Oda, H. Kano, and M. Tanaka, A stereo-machne for vdeo-rate dense depth mappng and ts new applcatons, n IEEE Computer Vson and Pattern Recognton Conference, vol. 24, June [10] W. B. Seales, G. Welch, and C. O. Jaynes, Real-tme depth warpng for 3-d scene reconstructon, n IEEE Aerospace Conference Proceedngs, vol. 3, March [11] J. Smt, L. G. van Wllgenburg, and J. Meuleman, Accurate calbraton and 3d data recovery based on physcal camera models ncludng lens dstortons, Master s thess, Wagenngen Unversty, The Netherlands, [12] S. B. Goldberg, M. W. Mamone, and L. Matthes, Stereo vson and rover navgaton software for planetary exploraton, n IEEE Aerospace Conference Proceedngs, vol. 5, March [13] A. Abbo and R. Klehorst, Xetal Software Framework Programmng Gudelnes. Phlps Research Laboratores, NatLab, [14] J. Nnot, Real tme depth estmaton from bnocular cameras usng the xetal processor, Techncal Report PR-TN- 2003/00797, Phlps Research, November [15] J. Banks, M. Bennamoun, and P. Corke, Non-parametrc technques for fast and robust stereo matchng, n IEEE Speech and Image Technologes for Computng and Telecommuncatons, TENCON, December [16] M. Martn, M. Martn, C. Alberola-López, and J. Ruz-Alzola, A topology based fllng algorthm, Computers & Graphcs, vol. 25, no. 3, pp , REFERENCES [1] Y. Ruchek, A herarchcal neural stereo matchng approach for real-tme obstacle detecton, n IEEE Intellgent Transportaton Systems Proceedngs, October [2] M. Hart, Y. Ruchek, and A. Koukam, A votng stereo matchng method for real-tme obstacle detecton, n IEEE Robotcs and Automaton Proceedngs, ICRA, vol. 2, September [3] S. Kagam, K. Okada, M. Inaba, and H. Inoue, Desgn and mplementaton of onbody real-tme depthmap generaton system, n IEEE Robotcs and Automaton Proceedngs, ICRA, vol. 2, Aprl [4] Y. L. Murphey, J. Chen, J. Crossman, J. Zhang, P. Rchardson, and L. Seh, Depthfnder, a real-tme depth detecton system for aded drvng, n IEEE Intellgent Vehcles Symposum Proceedngs, October [5] S. K. Park and I. S. Kweon, Robust and drect estmaton of 3-d moton and scene depth from stereo mage sequences, Pattern Recognton, vol. 32, no. 9, pp , [6] M. I. Fanany and I. Kumazawa, A neural network for recoverng 3d shape from erroneous and few depth maps of shaded 310