Real-Time Stereo Vision Techniques

Transcription

1 Rel-Time Stereo Vision Techniques Christos Georgouls nd Ionnis Andredis Lortory of Electronics, Deprtment of Electricl nd Computer Engineering Democritus University of Thrce Xnthi 6700, Greece Astrct This PhD ims t the design nd implementtion of rel-time hrdre-sed stereo vision systems. Stereo vision dels ith imges cquired y stereo cmer setup, here the disprity eteen the stereo imges llos depth estimtion ithin scene. In the first prt of my reserch ne hrdreefficient rel time disprity mp computtion module s developed hich involved fully prllel-pipelined design, for the overll module, relized on single FPGA device. The design technique s extended to include the hrdre implementtion of fuzzy inference system (FIS), to del more efficiently ith the detection of conjugte pirs in stereo imges, hich is chllenging reserch prolem knon s the correspondence prolem, i.e. to find for ech point in the left imge, the corresponding point in the right one. The performnce of the hrdre implemented design methodologies s evluted ginst the stte of the rt methods to prove their efficiency. I. INTRODUCTION Distnce clcultion of scene points ithin n cquired imge reltive to the position of cmer is one of the importnt tsks of computer vision system. This llos depth estimtion nd environment reconstruction []. The most common method for depth extrction is the use of stereo cmer setup consisting of to cmers displced y knon distnce ith co-plnr opticl xes. Point to point mtching to the intensity imges cquired y the stereo setup, derives the depth imges, or the so clled disprity mps [2]. If the stereo imge pirs re ccurtely rectified, the mtching procedure cn e performed in one dimension, since horizontl scn lines reside on the sme epipolr line. This cn e seen in Fig.. A point P in one imge plne my hve risen from ny of points in the line C P, nd my pper in the lternte imge plin t ny point on the so-clled epipolr line E 2 []. Thus the serch for corresponding imge points eteen the stereo imge pir is reduced ithin the sme scn line. The horizontl distnce in pixel coordintes eteen possile corresponding points is the disprity. A disprity mp consists of ll the disprity vlues mong the stereo imge pir. The extrction of the disprity mp efficiently dels ith prolems such s 3D reconstruction, moile root nvigtion nd ostcle voidnce, positioning, etc [3,4]. A chllenging reserch prolem knon s the correspondence prolem comprises the detection of conjugte pirs in stereo imge pirs, i.e. to find the for ech point in the left imge, its corresponding in the right imge [5]. To perform point mtching, the points should e distinctly different from their surrounding pixels. Thus suitle feture extrction should precede stereo mtching. The to mjor ctegories of feture extrction lgorithms re: re-sed [6,7] nd re sed [8,9]. Are-sed lgorithms produce dense disprity informtion using locl pixel intensity mesures. Feture-sed lgorithms use specific point, depending on the extrcted fetures. Although they produce more ccurte result, the resulting disprity informtion is sprse nd they re significntly sloer compred to re-sed lgorithms. Thus in pplictions ere speed demnd is dominnt, re-sed methods hve een preferred. Only in the recent yers rel-time methods hve een reported, due to the increse of CPU nd custom hrdre structure speeds [0]. Dense stereo lgorithms provide dense disprity informtion, ut hve een proved to ignore possile uncertinties during the mtching stge. To overcome this issue, vrious methods hve een employed, in order to minimize the uncertinty nd efficiently perform n elimintion of possile flse mtches during the mtching process [2,6]. Most of these methods rely on the use of sum of solute differences (SAD) or correltion lgorithms, hich re not highly cple to minimize the flse mtches. In the first prt of this PhD, novel three-stge hrdre implemented module s proposed, hich is cple of ddressing the stereo vision correspondence prolem, producing disprity mps in rel-time speeds, reching up to 275 frmes per second for 640 x 480 pixel resolution stereo pir ith 80 levels of disprity. Its novelty compred to previous lgorithms is its high processing speed. Remining flse correspondences re eliminted using cellulr utomt (CA) filter, producing semi-dense disprity mps ith miniml noise []. The current reserch of this PhD includes the hrdre implementtion of to-stge module, hich is le to produce dense disprity mps in rel-time speeds, reching up to 439 frmes per second for 640 x 480 pixel resolution stereo pir ith 80 levels of disprity. The novelty compred to previous reported lgorithms is the comintion of highly ccurte dense disprity informtion extrction, long ith rel-time processing speed. A fuzzy inference system (FIS) is implemented to del ith the elimintion of flse mtches, improving even more the disprity mp ccurcy nd coverge. The proposed module cn implement n efficient module suitle for rel-time stereo vision pplictions [2]. In Section II the proposed lgorithms re descried nd the proposed hrdre structure detils re given in Section III. In Section IV experimentl results re shon nd finlly, conclusions re drn in Section V.

2 SAD( j, l µ - v - I ( i + µ, j + ν) I ( i + µ, j d + ν ), (2) here I l nd I r denote the left nd right imge pixel gryscle vlues, d is the disprity rnge, is the indo size nd j re the coordintes (ros, columns) of the center pixel of the indo for hich the SAD is computed. To extrct the pproprite disprity vlue for ech pixel, serch in the SAD for ll disprity vlues (d min up to d mx ) is performed. At the disprity rnge here the computed SAD vlue for given pixel is minimum, this vlue is ssigned s the disprity mp vlue for the given pixel. r D( rg min SAD( j, (3) d [ dmin, dmx ] Figure. Geometry of Epipolr Lines Figure 2. Block Digrm of the proposed 3-stge system II. PROPOSED ALGORITHMS A. First Method An overvie of the lgorithm is presented in Fig. 2. The locl vrition of the stereo pir imges is initilly computed for 2 x 2 nd 5 x 5 pixel indo sizes. The disprity mp is then computed using n SAD indo sed technique, here the indo size is determined y the locl vrition results. Finlly, CA method is used to filter the noise due to flse mtched points. All three steps re fully implemented in hrdre on n FPGA device. ) Locl Vrition estimtion: The mesure of locl vrition, in terms of pixel gryscle vlue, over the imge through vrile sized indos, cn provide ith more efficient disprity mp evlution. The mesure of locl vrition is provided y (). N N LV ( p) I ( µ () i j The locl vrition for given indo ith centrl pixel p is clculted ccording to the neighoring pixel gryscle vlues, here N is the selected squre indo size nd µ is the verge gryscle vlue of the given indo. For the cse of the 2 x 2 pixel indo the locl vrition is clculted for the upper left pixel. 2) Adptive indo serch: To perform the disprity mp genertion n SAD indo sed lgorithm s used to find the corresponding points in the stereo imge pir, hich is given elo. 3) Cellulr utomt filtering: The resulting disprity mp is usully corrupted due to the disprity mp ssignment stge descried in (3). In order to enhnce the disprity mp, ithout the loss of 3D informtion, simple CA pproch is employed. Filtering ith CA pproch s introduced, s it presents more efficient noise removl compred to stndrd filtering techniques [3], hile preserving imge detils. The folloing 2D trnsition rules re tken into ccount for the disprity mp filtering; the first rule sttes tht ) if C ( i 0 nd j 0) then C 0, ) if j 0 j j - j - C j 7 ( i 0 nd j 0) then C j The second Rule of CA sttes tht, c) if if C j ( i 0 nd j 0) nd C j j - 3., then C, 0 ( i {,} nd j {,}) then C, 0. i j j -3 For given disprity mp ith, i.e. 6 disprity levels, 6 inry imges ere creted y decomposition, here C imge hs logic ones on every pixel tht hs vlue in the disprity mp, nd logic zeros elsehere, nd so on. The CA rules ere pplied seprtely to ech C d inry imge nd the resulting disprity mp s then recomposed y the folloing formul: D Cd ( d, d [ d min, d ] (4) i j ( mx B. Second Method An overvie of this lgorithm is presented in Fig. 3. The disprity mp is clculted using n SAD indo sed technique, using color intensity imges. The three color components intensity vlues, (R, G nd B), re tken into ccount for the clcultion of the SAD vlue. The squre indo size selected for the implementtion hs fixed size of 7 x 7 pixels. The resulting disprity mp is then filtered using 3-input -output Mmdni type fuzzy inference system (FIS), to minimize incorrect mtches during the previous step.

3 TABLE I. MAXIMUM ERRORS (ME) FOR THE MEMBERSHIP FUNCTIONS USED Figure 3. Block Digrm of the proposed 2-stge system Both steps re fully implemented in hrdre relized on n FPGA device. ) Color SAD indo sed technique: Color intensity imges ere used, here R, G nd B color components from the stereo pir imges ere fed to the module. The sum of solute differences (SAD) lock-mtching method used to extrct the corresponding points in the color stereo imge pir is presented elo. SAD( j, 3 µ - v - k I ( i + µ, j + ν, k ) I l r ( i + µ, j d + ν, k), (5) here I l nd I r denote the left nd right imge pixel vlues, d is the disprity rnge, is the indo size, k is the corresponding RGB spce color component, ( for Red, 2 for Green nd 3 for Blue), nd j re the coordintes (ros, columns) of the center pixel of the indo for hich the SAD is computed. Hving constructed the SAD, the disprity vlue for ech pixel is clculted ccording to (3). 2) Fuzzy inference system filtering: As me mentioned previously, numerous flse mtches re introduced during the disprity vlue ssignment stge (3). This type of rndom noise is not tolerle, if dense disprity mp extrction is required. Additionlly if high processing speed is demnded, n lgorithm to fullfill oth these requirements in efficient y needs to e relized. A 3-input -output fuzzy inference system (FIS) hich holds 27 rules s implemented. The clculted disprity vlues from the previous step ere fed into the FIS module. A set of three disprity vlues, [(, (j-3), (j+3)], here j re the coordintes (ros, columns) of the D msk center pixel, ere used s the three FIS inputs. ording to the selected rules the corresponding msk center pixel s ressigned ne disprity vlue provided y the FIS output. Tests proved tht the resulting disprity mp exhiits n increse of more thn 0% in totl ccurry. The folloing formul descries the FIS opertion. Numer of its, N ME hrdre complexity increses, complicting the hrdre implementtion of the system. The mximum errors (ME) evluted for the memership functions used cn e seen in Tle I. N represents the numer of its per memership degree nd the cses for 4 up to 8 ere considered. The vlues of the memership functions used in the present design re represented y 6-it inry ords. III. HARDWARE STRUCTURES DESCRIPTION A. First Method The module s implemented in prllel pipelined hrdre rchitecture, relized on single FPGA device of the Strtix II fmily of Alter Devices. The typicl operting clock frequency s found to e 256 MHz. The hrdre design rchitecture is shon in Fig. 4. Detils for the hrdre description cn e found in []. ) Speed Issues: The hrdre module presented in Fig. 4, s designed to clculte disprity mps for stereo pir tht present rnge of disprity up to 80 pixels. The reltionship eteen the numer of frmes processed per second nd the processed imge idth, ssuming squre imges, is pproximted y (7). frmes sec [( imge idth in ) ] pixels 2) FPGA device specifictions: The rchitecture presented in Fig. 4, s implemented using Alter Qurtus II schemtic editor. It hs een simulted to prove functionllity, nd once tested, finlly mpped on n FPGA device. The nlyticl specifictions of the trget device re given in Tle II. (7) D( FIS[ D(, D( j 3), D( j + 3)] (6) The numer of its used for the digitiztion of the memership functions determines the ccurcy of the representtion nd ffects the results of fuzzy rules. One sic type of error due to the finiteness of the ord length in digitl implementtions of fuzzy inference systems re the memership function errors. These errors result from the digitiztion of the memership function vlues [4]. As the resolution increses, ccurcy improves, ut t the sme time, Figure 4. FPGA Design of the proposed 3-stge module

4 B. Second Method The proposed module s implemented in hrdre on single FPGA device of the Strtix III fmily of Alter Devices. Prllel pipelined rchitecture s folloed considering speed issues. The typicl operting clock frequency s found to e 38 MHz. The overll hrdre design rchitecture is presented in Fig. 5. For more hrdre rchitecture detils see [2]. ) Speed issues: The hrdre presented in Fig. 5, s designed to provide ith rel time disprity mp extrction, for stereo imges ith up to 80 levels of disprity. The chieved speed performnce, concerning the frmes per second reltive to imge pixel resolution, ssuming squre imges, is pproximted y (8). frmes 0 8 (imge idth in pixels ) 2 sec [ ] Figure 6. Resulting disprity mps for () corridor nd () cones stereo pir respectively, long ith originl imge pirs (8) 2) FPGA device specifictions: The overll rchitecture s relized on single FPGA device using Alter Qurtus II schemtic editor. The designed s finlly mpped to device, fter succefull testing nd simultion. The specifictions of the trget device re given in Tle III. IV. EXPERIMENTAL RESULTS A. First Method The proposed module is using n dptive technique here the support indo size is selected utomticlly depending on the locl vrition of the support neighorhood. This minimizes the percentge of flse correspondences during the mtching stge. The CA post processing filter, enles stisfctory elimintion of remining flse correspondences, preserving detils in the resulting disprity mp, in contrst to existing filtering methods tht noticely lter disprity vlues in order to remove unnted noise. The generted disprity mps re shon in Fig. 6. The hrdre implemented module presents higher processing rtes compred to the existing methods in terms of speed, enling the method pproprite in rel-time demnding pplictions. Tle IV presents the processing speed of the relized module, for n operting frequency of 256 MHz nd for imge pirs ith disprity rnge of 80 levels. Quntittive results of the proposed module under vrious configurtions cn e seen in Tle V, nd compred to previous lgorithms in Tle VI respectively. In Tle VII the proposed module is compred to previous lgorithms in terms of speed. The chieved ccurcy long ith the speed performnce of the proposed module, enle n efficient module tht cn perform dequtely enough in rel stereo vision pplictions, ith disprity rnge usully eteen 60 nd 20 levels, regrding nerly 0% decrese in ccurcy nd more thn 2,000% increse in speed compred to previous lgorithms [58], Tles VI nd VII. TABLE II. Device Registers 84, Alter EP2S 80F02 0C3 TABLE III. ALUTs LABs pins 59 (84,307/ 43,520) 83 (7484/ 8970) 3 (25/ 743) SPECIFICATIONS OF TARGET DEVICE (SECOND METHOD) Device lock memory its Tot. Registers ALUTs Alter EP3SL 340H 52C3 < (6,/ 6,662,528) 5, (208,940/ 270,400) TABLE IV. Figure 5. FPGA Design of the proposed 2-stge module SPECIFICATIONS OF TARGET DEVICE (FIRST METHOD) Com. functions Disprity Levels Imge size Frmes/s LABs 96 (2,94 / 3,520) pins 8 (56/ 744) FRAME RATE OUTPUT OF THE REALIZED MODULE (FIRST METHOD) x x x x x024 65

5 TABLE V. QUANTITATIVE RESULTS OF THE PROPOSED MODULE UNDER VARIOUS CONFIGURATIONS (FIRST METHOD) Corridor Cones Teddy Without CA filtering,. With CA filtering. TABLE VI. QUANTITATIVE RESULTS OF THE PROPOSED METHOD COMPARED TO PREVIOUS ALGORITHMS (FIRST METHOD) (disp. levels6) Venus (disp. levels20) Proposed method [5] [6] [7] [8] TABLE VII. Proposed method [6] [5] [8] [7] SPEED COMPARISON TO PREVIOUS ALGORITHMS (FISRT METHOD) Time increse (384 x 2) (disp. levels 6) Venus (434 x 383) (Disp. levels20) , B. Second Method The proposed module uses n SAD indo sed technique to find the corresponding points in the stereo pir. It uses RGB intensity imges, to minimize flse correspondences due to more efficient discrimintion eteen the possile cndidte pixels, since RGB component vlues provide etter intensity informtion vrition. Moreover, the FIS filtering post processing technique minimizes even more flse disprity estimtions, improving the resulting disprity mp ccurcy nd coverge, hile mintining imge detils. The resulting disprity mps re depicted in Fig. 7. Tle VIII presents the processing speed of the relized module, for n operting frequency of 38 MHz nd for imge pirs ith disprity rnge of 80 levels. Quntittive results of the proposed module under vrious configurtions cn e seen in Tle IX, nd compred to previous lgorithms in Tle X respectively. In Tle XI the proposed module is compred to previous lgorithms in terms of speed. V. CONCLUSSIONS A three-stge [], nd to-stge [2], modules ddressing the stereo vision mtching prolem, hve een proposed. The modules ere implemented in hrdre, iming t rel-time pplictions. Reference [], employs n dptive indo constrint serch s ell s CA pproch to flse reconstruction removl, hile [2] employs color SAD fixed indo size technique, long ith FIS module for remining flse mtch elimintion. Both from qulittive nd quntittive terms, concerning the qulity of the produced disprity mps reltive to the frme rte outputs, highly efficient methods deling ith the stereo correspondence prolem hve een proposed. Efficient performnce under vrious levels of disprity, mintining the frme rte output lmost constnt, hile the disprity rnge increses from smll up to lrge vlues. Chnges of less thn frme per second for [] nd [2], hve een mesured for the modules corresponding output rtes, hile the disprity rnges from 6 up to 80 levels for 640 x 480 pixel resolution imge pir. Rel-time speeds rted up to 275 nd 439 frmes per second, for 640 x 480 imge resolution pir ith 80 levels of disprity, re chieved y [] nd [2] respectively. This mkes the proposed modules suitle for rel stereo vision pplictions. Additionlly ech of the proposed modules, fitted on single Alter FPGA device. As result, they could e pplied to enle efficient systems including high-speed trcking nd moile roots, oject recognition nd nvigtion, iometrics, vision-guided rootics in the utomotive industry, three-dimensionl modeling nd mny TABLE VIII. Disprity Levels Imge size Frmes/s TABLE IX x x x x x QUANTITATIVE RESULTS OF THE PROPOSED MODULE UNDER VARIOUS CONFIGURATIONS (SECOND METHOD) FRAME RATE OUTPUT OF THE REALIZED MODULE (SECOND METHOD) Venus Cones Teddy Without FIS filtering,. With FIS filtering. TABLE X. QUANTITATIVE RESULTS OF THE PROPOSED METHOD COMPARED TO PREVIOUS ALGORITHMS (SECOND METHOD) Figure 7. Resulting disprity mps for () tsuku nd () teddy stereo pir respectively, long ith originl imge pirs Proposed method [5] [6] [7] [8] (disp. levels6) Venus (disp. levels20)

6 TABLE XI. Proposed method SPEED COMPARISON TO PREVIOUS ALGORITHMS (SECOND METHOD) (384 x 2) (disp. levels 6) Venus (434 x 383) (Disp. levels20) Time increse [6] [5] [8] [7] , more. Results confirm tht semi-dense, for [], nd dense, for [2], disprity mps cn e efficiently clculted ithout the expense of speed reduction. The ccurcy of the clculted disprity mps is reduced y nerly 0% compred to previous methods. The min novelty for oth implementtions compred to previous lgorithms is the high processing speed. The trde-off eteen disprity mp qulity nd performnce speed ill lys e present in this type of implementtions. Despite the reduced resulted ccurcy compred to [5-8], the proposed modules cn effectively e pplied to rel-time stereo vision pplictions due to their speed performnce. REFERENCES [] R. Jin, R. Kstur B.G. Schunck, McGr-Hill, Ne York, 5. [2] O. Fugers, Three-dimensionl Computer Vision: A geometric viepoint, MIT Press, Cmridge, MA, 3. [3] D. Murry, C. Jennings, Stereo vision sed mpping for moile root, in Proc. of the IEEE Interntionl Conference on Roottics nd Automtion (ICRA 7), 7, pp [4] D. Murry nd J. Little, Using rel-time stereo vision for moile root nvigtion, Auton. Roots, vol. 8, no. 2, pp. 6 7, [5] S.T. Brnrd, W.B. Thompson, Disprity nlysis of imges, IEEE Trnsctions on Pttern Anlysis Mchine Intelligence, vol. 2, pp , 980. [6] L. Di Stefno, M. Mrchionn S. Mttocci, A fst re-sed stereo mtching lgorithm, Imge nd Vision Computing, vol. 22, no. 2, pp , [7] K. Muhlmnn, D. Mier, J. Hesser, R. Mnner, Clculting dense disprity mps from color stereo imges, n efficient implementtion, Interntionl Journl of Computer Vision, vol 47, no. 3, pp. 79-, [8] J.R. Jordn, A.C. Bovik, Using chromtic informtion in edge-sed stereo correspondence, Computer Vision Grphics, Imge Processing: Imge Understnd, vol. 54, no., pp. 98-8,. [9] A. Bumerg, Relile feture mtching cross idely seprted vies, in Proc. of the IEEE Conference on Computer Vision nd Pttern Recognition (CVPR 2000), 2000, pp [0] J.L. Croley, J. Piter, Introduction to the specil issue: interntionl conference on vision systems, Mchine Vision Applictions, vol. 6, no., pp. 4-5, [] C. Georgouls, L. Kotouls, G. Ch. Sirkoulis, I. Andredis, A. Gstertos, Rel-time disprity mp computtion module, Microprocessors & Microsystems, vol. 32, no. 3, pp.59-70, [2] C. Georgouls, I. Andredis, A rel-time fuzzy hrdre structure for disprity mp computtion, In preprtion. [3] V. Murino, U. Cstelln A. Fusiello, Disprity Mp Restortion y Integrtion of Confidence in Mrkov Rndom Fields Models, in Proc. of the Interntionl Conference on Imge Processing (ICIP 200), 200, no. 2, pp [4] I. Del Cmpo nd J. M. Trel, Consequences of the digitiztion on the performnce of fuzzy logic controller, IEEE Trnsctions on Fuzzy Systems, vol. 7, pp. 85, 9. [5] M. Gong, Y.H. Yng, Fst Unmiguous Stereo Mtching Using Reliility-Bsed Dynmic Progrmming, IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence, vol. 27, no. 6, pp , [6] M. Gong, Y.H. Yng, Ner rel-time relile stereo mtching using progrmmle grphics hrdre, in: Proc. of the IEEE Conference on Computer Vision nd Pttern Recognition (CVPR 2005), 2005, pp [7] O. Veksler, Dense Fetures for Semi-Dense Stereo Correspondence, Interntionl Journl of Computer Vision, vol. 47, no. -3, pp , [8] O. Veksler, Extrcting Dense Fetures for Visul Correspondence ith Grph Cuts, in: Proc. of IEEE Conference on Computer Vision nd Pttern Recognition (CVRP 2003), 2003, pp