A Parallel Reconfigurable Architecture for Real-Time Stereo Vision

Transcription

1 2009 Iteratioal Cofereces o Embedded Software ad Systems A Parallel Recofigurable Architecture for Real-Time Stereo Visio Lei Che Yude Jia Beijig Laboratory of Itelliget Iformatio Techology, School of Computer Sciece, Beijig Istitute of Techology, Beijig , PRC {chelei,jiayude}@bit.edu.c Abstract I this paper, a parallel recofigurable architecture is proposed for real-time stereo visio computatio. The architecture is divided ito four compoets: iput port, output port, memory ad processor. We use task partitio methods to achieve the maximum parallel ad full pipelie processig of the algorithm implemetatio. We also adopt memory maagemet to decrease the latecy of memory access time ad accelerate the processig speed. Data badwidth cotrol is employed to reduce the hardware resource cosumptio while maitaiig precisio demad of computatio. Based o the proposed architecture ad desig method, we have developed a miiature stereo visio machie (MSVM33) to geerate high-resolutio dese disparity maps at the video rate for real-time applicatios. 1. Itroductio Stereo visio, which is ispired by huma visual process, computes the disparity betwee correspodece poits i images captured by multiple cameras for distace measuremet. Real-time stereo visio has bee widely used i itelliget robot avigatio, smart huma-computer iteractio, itelliget surveillace, etc. Stereo visio algorithms ivolve a large umber of regular, repetitive ad computatioally itesive operatios o large sets of structured data. However, traditioal software implemetatio of these algorithms, which rus o CPU-based platforms usig sequetial istructio executio mode ad fixed cotrol mechaism, caot satisfy the complete video-rate processig demad. I order to sustai high computatioal load ad real-time throughput of these stereo visio tasks, several hardware-based computig solutios have bee reported [1-10]. The sum of absolute differeces (SAD) or sum of square differeces (SSD) [1, 3 6, 9, 10], cesus matchig [2, 8] ad local weighted phase correlatio (LWPC) [7] stereo algorithms have bee implemeted i these systems, which maily make use of hardware such as digital sigal processors (DSPs) [1, 3], field programmable gate arrays (FPGAs) [2, 4-7], applicatio specific itegrated circuits (ASICs) [8, 9] ad graphic processor uits (GPUs) [10]. All systems [1-10] are advaced for a age, ad have bee widely used i practical applicatios. The architecture of these systems ca be maily divided ito two categories: PC-based geeral purpose architecture [10] ad customized special purpose architecture [1-9]. Systems based o geeral purpose architecture are usually physically large, high power cosumptio, ad these disadvatages limit the utilizatio of this architecture i dyamic, variable, practical applicatios. I cotrast, customized special purpose architecture provides a alterative solutio to system implemetatio. ASIC is desiged ad optimized for specific algorithms ad operatios, so systems usig ASICs ca achieve superior performace for a limited set of applicatios. However, ASICs eed log desig cycle ad restrict the flexibility of the system ad exclude ay post-desig optimizatios ad upgrades i features ad algorithms [11]. Recofigurable systems with great flexibility reduce the time, cost ad expertise requiremets i hardware-based algorithm implemetatio, ad ca be reprogrammed to facilitate improvemet ad modificatio i desig at ru-time without sacrificig system performace. I this paper, we preset a parallel recofigurable architecture for stereo visio computatio at the video rate. We use task partitio methods to achieve the maximum parallel ad full pipelie processig of the algorithm implemetatio. Memory maagemet is adopted to decrease the latecy of memory access time ad accelerate the processig speed. Data badwidth cotrol is employed to reduce the hardware resource cosumptio while maitaiig precisio demad of computatio. We have developed a FPGA-based real-time miiature stereo visio machie (MSVM33) based o this architecture to prove the effectiveess of the architecture /09 $ IEEE DOI /ICESS

2 The remaider of this paper is orgaized as follows. Sectio 2 describes the architecture ad desig methods. Sectio 3 itroduces the implemetatio of our real-time miiature stereo visio machie. The performace ad experimetal results of the machie are discussed i Sectio Architecture Computer visio algorithms perform a variety of data maipulatio, sigal processig, mathematical ad graph theoretic computatios. Typically, these tasks are performed repeatedly i some sequece, ad thus are characterized by sigificat amout of temporal ad spatial parallelism [12]. Recofigurable computig utilizes hardware which ca be adapted at ru-time to facilitate greater flexibility without compromisig performace [11]. Combiig the advatages of parallel ad recofigurable mechaism, we preset a parallel recofigurable architecture for real-time stereo visio implemetatio Architecture Overview This architecture is iteded to cope with image data acquisitio, stereo visio computatio ad data trasmissio i real-time, ad desiged to be adapted for a variety of stereo visio algorithms. The architecture cotais four compoets: iput port, output port, memory, ad processor as illustrated i Fig. 1. Figure 1. Architecture of the real-time stereo visio computig Processor ad memory are fudametal compoets of the architecture ad critical for overall performace. The mai desig methods are: (1) Operatios o a image are implemeted i sub-images, i colum ad row directio respectively ad i serial pipelie stages to achieve the maximum parallel ad full pipelie processig i the processor. (2) Pig-pog mode maagemet decreases the latecy time i memory access. (3) Asychroous FIFO desig accelerates the processig speed. (4) Data badwidth cotrol maitais precisio demad of computatio ad reduces the hardware resource cosumptio. Processor servers as the computer s CPU i this architecture. It deals with data trasmissio, parallel ad pipeliig processig, memory access cotrol ad commuicatio betwee each compoet i the architecture. Memory icludes large, low speed, exteral memory ad small, high speed, iteral memory. Exteral memory is used for iput/output data stream bufferig ad itermediate computig results storage. Iteral memory provides storage resource for FIFO, look-up-table, IP cores, etc, which are commoly used i code program executio of hardware descriptio laguages (HDLs). The iput port maily cosists of multiple CMOS or CCD optical sesors. The fuctio of this compoet is to cotrol these sesors to capture image data sychroously for stereo matchig ad trasfer acquired data to the memory. The role of output port is to deliver the computig results, which are stored i the memory, out of the whole architecture to the host for high level practical applicatios usig commo trasmissio iterfaces such as PCI, USB, IEEE 1394, etc Parallel ad Pipeliig Processig As low level of visio, may stereo visio algorithm computatios o image pixels are repetitive ad oly related to the values of local eighborhood pixels. May computig operatios i these algorithms ca be precisely decomposed ito image row ad image colum directio respectively or withi a acceptable RMS error rage. Meawhile, the operatio itself ca also be divided ito some idepedet stages. Due to these characteristics, it makes the parallel ad full pipelie implemetatio of these algorithms reasoable. By traslatig algorithm ito proper parallelism level ad optimal log pipelie stages, the architecture ca achieve outstadig computig performace [13]. As the most timig-critical ad hardware resource cosumptio part, we adopt two kids of parallelism i the implemetatio of stereo visio algorithms: temporal ad spatial parallelism. (1) Spatial parallelism (processig uit duplicatio): computatios o oe image are costructed by copies of the same processig module, ad the image is partitioed ito sub-images, the these processig modules are executed o sub-images i parallel. As stereo visio deals with multiple images, operatios ca be computed amog multiple images i parallel. 33

3 (2) Temporal parallelism (processig time overlappig): computatios i the algorithms are traslated ito serial separate ad idepedet pipelie stages. Fig.2 illustrates the parallel ad pipelie desigs adopted i our work, ad these desigs ca be spatial mapped oto the fuctioal uits of processor ad memory resource i the architecture. (a) (b) (c) Figure 2. Parallel ad pipelie processig. (a) The whole image is partitioed ito sub-images (for example, if =4, each sub-image is a quarter of the whole image), ad processig module which have the same fuctio ca execute the computatio o the sub-images i parallel. (b) Operatio is computed i parallel amog multiple images, ad ca be decomposed ito colum ad row directio respectively i oe image. (c) Operatio o oe pixel ca be implemeted as -stages serial pipelie mode. Pixels move through the pipelie stage by stage so that whe the pipelie is full, separate ad idepedet computatio are beig executed cocurretly, each i a differet stage. Furthermore, a fial computatio result emerges from the pipelie every clock cycle. 34

4 2.3. Memory Access ad Data Badwidth Cotrol The data size is large whe multiple sesors are used together i oe system. Cosider, for istace, whe we oly use grayscale image data (8 bits per pixel respectively) from three sesors, the throughput is almost 211Mbits per secod for fps applicatios. Data also eeds to be stored ito ad loaded from memory durig processig, so the memory access cost is usually oe of the critical bottleecks of the executio time. The mai methods for memory access ad data badwidth cotrol are as follows: (1) Memory access maagemet i pig-pog mode. The data storage i memory is based o pig-pog mode operatio to decrease waitig latecy time for data storage or load. For example, usig two chips of memory i tur, oe chip is used for bufferig a frame of sychroously captured image data from iput ports while the processor loads the previous frame of image data from the other chip for stereo visio computig at the same time. (2) Asychroous FIFO desig. Large amout of data, such as origial image data stream ad itermediate computig results, should be stored i big low speed memory without cosumig valuable small high speed memory. I order to achieve maximum performace, we desig a asychroous FIFO for efficiet data exchage betwee two kids of memory i the processig. The iput of the FIFO is 16 bits data loaded from exteral memory every two clock cycle at 60MHz clock rate, ad FIFO outputs 8 bit data each clock cycle at 120MHz clock rate to the iteral memory. Processig speed is multiplied with this method ad the serial output data is suitable for high speed full pipelie processig which is metioed before. (3) Data badwidth cotrol. I differetial computig processig, there are a large degree of variatio i data badwidth ad the required precisio of the operads ad operatio is ot the same. Aalyzig the data flow ad executio processig, we precisely choose the badwidth for each data. Eve the badwidth for the same data i differet processig stages is exactly chose respectively. This desig ot oly satisfies the precisio demad of computig, but also reduces the hardware resource requiremet i algorithm implemetatio. The three methods are sigificat i desig, ad they make our architecture properly balace the hardware resource cosumptio, circuit size, peripheral chips versus performace trade-off. 3. Implemetatio Based o the architecture described i the previous sectio, we have desiged a triocular miiature stereo visio machie. This system is composed of three parts: stereo head, processig uit ad trasmissio iterface as show i Fig. 3. Three triagular cofiguratio COMS optical sesors o the stereo head ca sychroously capture pixels grayscale or color images at 30fps. Triocular stereo visio system, with little icreasig of hardware cost, ca achieve better results tha a typical biocular stereo visio system by providig a secod pair of image sesors for reducig the ambiguity greatly. Three image sesors are placed at right agle to reduce distortios i triocular epipolar rectificatio. The processig uit maily icludes a high performace FPGA ad tightly coupled local SRAMs. FPGA serves as the processig core of the whole system ad its pricipal fuctios ca be summarized as follows: (a) supplyig exteral sychroous sigal for image sesors o the stereo head because images for stereo matchig must be captured simultaeously; (b) executig stereo matchig computatio to calculate disparities; (c) maagig data access cotrol of SRAMs; (d) outputtig grayscale images ad correspodig dese disparity maps simultaeously. For high computatioal requiremets ad real-time throughput of the system, the fuctio ad orgaizatio of te chips of SRAM are as follows: two chips for bufferig iitial grayscale images captured by CMOS optical sesors o the stereo head i tur, six chips for storig the itermediate results of stereo matchig computatio ad two chips for bufferig output grayscale ad disparity images i tur. The trasmissio iterface maily icludes a FPGA, a DSP ad IEEE1394 chipsets. FPGA cotrols the data trasmissio. DSP ad IEEE1394 chipsets implemet the IEEE 1394a high serial bus protocol for trasferrig data to the host computer at a speed of 400Mbps. Correspodig to the architecture metioed i Sectio2, i this system, the stereo head ad trasmissio iterface serves as the iput port ad output port respectively. The processig uit, which is the primary part i the system, works as processor ad memory. The block diagram of the miiature stereo visio machie is show i Fig

5 (a) Stereo head (b) Processig uit (c) Trasmissio iterface Figure 3. Miiature stereo visio machie (MSVM33) overview Figure 4. Block diagram of the system There are five mai modules i the FPGA o the processig uit: image data capture module, image data pre-processig module, triocular epipolar rectificatio module, stereo visio computig module ad disparity extractio ad post-processig module. Image data capture module cotrols the multiple optical sesors to capture images simultaeously ad stores the data i the SRAM chips. Image data pre-processig module maily performs image filterig algorithms to reduce image oise i iitial captured images. Triocular epipolar rectificatio module is used to simplify the stereo match computatio. The correspodece search is doe oly alog image colum or row lies i rectified images [14]. The disparity extractio ad post-processig module calculates the right value from cadidate disparities ad combies the disparity value with itesity value of each pixel for output trasmissio. For the valid disparities, a simple parabolic iterpolatio method is adopted for sub-pixel value disparity estimatio from cadidate disparities. The core module i the system is the stereo visio computig module, which computes the cadidate disparities for each pixel i the referece image with the adaptive aggregatio based cooperative stereo visio algorithm [15]. The mai procedure of the cooperative algorithm is described i Fig. 5. Let C1, C2 ad C3 deote the optical ceters of three image sesors respectively. The correspodig images are I 1, I 2 ad I 3, ad I 1 is the referece image. The sum of the absolute differece (SAD) of photometric similarity is used for the disparity space iitializatio. We defie oe disparity level as oe pixel i row lie of rectified image, ad the SAD value of two rectified stereo pairs is give by SADc ( r d) = I2( c d r) I1( c r) +, (1) I c r C C C C d I c r 3 ( ) 1( ) where I(c, r) represets itesity value of pixel at locatio (c, r) i the rectified image ad d is the disparity value. The iitial values of disparity space D 0 (c r d) is computed usig ormalized SAD D 1 SAD ( c r )/ 512 =. (2) 0 d The process of disparity space iteratio is give below: Calculate the sum of support area S cetered o (c, r, d) i disparity space S c r d = D x y z ( ) ( ) = ( x y z ) φ z φ y φ x φ Compute the sum of ihibitio area I D ( x y z ). (3) 36

6 I = S ( x y z) = ( x y z ) ϕ Max S ( c r i) S i= 1, (4) where Max is the disparity search rage. Compute the ihibitio gai usig equatio (3) ad (4) G = S I α, (5) where αis the coefficiet of ihibitio magificatio. Iteratively update the disparity space usig equatio (2) ad (5) D ( c r d) = D ( c r d) G ( c r d) (6) image data pre-processig module ad disparity extractio ad post-processig module with differet algorithms or processig methods. Users ca also modify parameters i the adaptive aggregatio based cooperative stereo visio algorithm such as the size of disparity map, the search rage of disparities, the size of support area, etc. This recofigurable feature allows the system to be applied to differet specific tasks. 4. Performace The adaptive aggregatio based cooperative stereo visio algorithm [15] is implemeted etirely with a sigle chip of Virtex4 Xilix FPGA o the processig uit. All modules iside FPGA are writte i hardware descriptio laguage VHDL. The itegrated developmet eviromet of Xilix ISE is used for sythesis, place ad route. The sythesis report for the algorithm is listed i Table 1. Table 1. Sythesis report for adaptive aggregatio based cooperative stereo visio algorithm Figure 5. Cooperative algorithm procedure. represets the umber of iteratios. Repetitious disparity space iteratio is implemeted by the same iteratio module cascaded. Differet iteratios are performed simultaeously ad multiple iteratios are executed i parallel with this method. The values of disparity space will become coverget withi limited times of iteratio, ad stereo visio computig module will output cadidate disparities for disparity extractio ad post-processig module. To cofigure our stereo visio machie for a particular practical applicatio, the user ca replace the The global clock frequecy is 60MHz. The stereo visio computig module with adaptive aggregatio based cooperative stereo visio algorithm rus at 120MHz clock frequecy, ad up to 187MHz sythesized by Xilix XST. The stereo visio machie ca achieve a throughput of more tha 30fps with pixels dese disparity maps i 32-pixel disparity searchig rage ad 5-bit disparity precisio. By compariso, the same cooperative stereo visio algorithm ruig with software programmed i C laguage takes approximately 3s whe rus o a Dell Petium D 3.40GHz PC with 1.0GB memory. The sigificat speedup performace relative to software implemetatio is maily due to the parallelism of the architecture, full pipelie programmig implemetatio ad optimized data flow path. Fig. 6 shows examples of the system fial output images i pair which are sychroously trasferred to the host computer. 37

7 (a) (b) (c) (d) (e) (f) Figure 6. Examples of system output images. (a)-(c) are captured grayscale images ad (d)-(f) are correspodig disparity maps of (a)-(c) respectively. I disparity map, the closer the objects to the camera, the lighter the itesity level is, ad black areas represet the ucofidet values which are rejected after computig. The resolutio of disparity maps, the disparity search rage, the frame rate, the clock frequecy of the stereo matchig computatio ad power cosumptio of the whole system are closely related to the stereo visio system overall performace. The performace of our miiature stereo visio machie is characterized i Table 2. Table 2. Miiature Stereo Visio Machie (MSVM33) Specificatios Camera Number 3 Iput Image Size (max) Disparity Search Rage (max) 32 Disparity Map Size (max) Frame Rate (max) 30fps Algorithm Cooperative Processor FPGA IO Iterface IEEE 1394 System Size 9cm 7cm Clock Frequecy 120MHz Power Cosumptio <7W 5. Coclusio We have preseted a parallel recofigurable architecture for real-time stereo visio. This architecture is show to be flexible ad effective for differet algorithms implemetatio, improvemet ad modificatio i fuctioality desig. We have also proposed some critical desig methods closely related to the overall performace of the architecture. Based o the proposed architecture ad methods, a miiature stereo visio machie (MSVM33) has bee implemeted for real-time dese high-resolutio disparity mappig. MSVM33 is portable, low cost, low power cosumptio, high-speed computatio ad flexible for differet specific tasks. It proves that our architecture ad desig methods are suitable for real-time stereo visio practical applicatios. Ackowledgemets This work was partially supported by the Natural Sciece Foudatio of Chia ( ), the Chiese High-Tech Program (2006AA01Z120), ad Beijig key disciplie program. Refereces [1] T. Kaade, A. Yoshida, K. Oda, H. Kao, ad M. Taaka. A Stereo Machie for Video-Rate Dese Depth Map-pig ad Its New Applicatios. I IEEE Computer Society Coferece o Computer Visio ad Patter Recogitio, pp , [2] J. Woodfill ad B. Vo Herze. Real-time Stereo Visio o the PARTS Recofigurable Computer. I IEEE 38

8 Symposium o FPGAs for Custom Computig Machies, pp , [3] K. Koolige. Small Visio Systems: Hardware ad Implemetatio. I Iteratioal Symposium o Robotics Research, pp , [4] S. Kimura, T. Shibo, H. Yamaguchi, E. Kawamura, ad K. Nakao. A Covolver-Based Real-Time Stereo Machie (SAZAN). I IEEE Computer Society Coferece o Computer Visio ad Patter Recogitio, pp , [5] Y. Jia, Y. Xu, W. Liu, C. Yag, Y. Zhu, X. Zhag, ad L. A. A Miiature Stereo Visio Machie for Real-Time Dese Depth Mappig. I Proceedigs of the Iteratioal Coferece o Computer Visio Systems, pp , [6] Y. Jia, X. Zhag, M. Li, ad L. A. A Miiature Stereo Visio Machie (MSVM-III) for Dese Disparity Mappig. I Proceedigs of the Iteratioal Coferece o Patter Recogitio, pp , 2004 [7] A. Darabiha, W. MacLea, ad J. Rose. Recofigurable Hardware Implemetatio of a Phase-Correlatio Stereo Algorithm. Machie Visio ad Applicatios, 17(2): , [8] J. Woodfill, G. Gordo, D. Jurasek, T. Brow, ad R. Buck. The Tyzx DeepSea G2 Visio System, A Taskable, Embedded Stereo Camera. I Proceedigs of the Coferece o Computer Visio ad Patter Recogitio Workshop, pp , [9] M. Basal, A. Jai, T. Camus, ad A. Das. Towards a Practical Stereo Visio Sesor. I IEEE Computer Society Coferece o Computer Visio ad Patter Recogitio, pp.63-69, [10] R. Yag ad M. Pollefeys. Multi-Resolutio Real-Time Stereo o Commodity Graphics Hardware. I IEEE Computer Society Coferece o Computer Visio ad Patter Recogitio, pp , [11] K. Bodalapati ad V. Prasaa. Recofigurable Computig Systems. I Proceedigs of the IEEE, 90(7): , [12] A. Choudhary, S. Das, N. Ahuja, ad J. Patel. A Recofigurable ad Hierarchical Parallel Processig Architecture: Performace Results for Stereo Visio. I Proceedigs of the Iteratioal Coferece o Patter Recogitio, pp , [13] J. Diaz, E. Ros, A. Prieto, ad F. Pelayo. Fie Grai Pipelie Systems for Real-Time Motio ad Stereo-Visio Computatio. Iteratioal Joural of High Performace Systems Architecture, 1(1):60-68, [14] L. A, Y. Jia, J. Wag, X. Zhag, ad M. Li. A Efficiet Rectificatio Method for Triocular Stereovisio. I Proceedigs of the Iteratioal Coferece o Patter Recogitio, pp.56-59, [15] M. Li ad Y. Jia. Adaptive Aggregatio Based Cooperative Stereo Visio. Joural of Software, 19(7): ,