Real-Time Stereo Vision on a Reconfigurable System

Transcription

1 Real-Time Stereo Vision on a Reconfigurable System SungHwan Lee, Jongsu Yi, and JunSeong Kim School of Electrical and Electronics Engineering, Chung-Ang University, 221 HeukSeok-Dong DongJak-Gu, Seoul, Korea {lshcau, xmxm2718}@wm.cau.ac.kr, junkim@cau.ac.kr Abstract. Real-time three-dimensional vision would support various applications including a passive system for collision avoidance. It is a good alternative of active systems, which are subject to interference in noisy environments. In this paper, we investigate the optimization of real-time stereo vision with respect to resource usage. Correlation techniques using a simple sum of absolute differences(sad) is popular having good performance. However, processing even a small image takes seconds. In order to provide depth maps at frame rate around 30fps, which typical cameras can provide, hardware accelerations are necessary. Regular structures, linear data flow and abundant parallelism make the correlation algorithm a good candidate for reconfigurable hardware. We implemented versions of SAD algorithms in VHDL and synthesized them to determine resource requirements and performance. By decomposing a SAD correlator into column SAD calculator and row SAD calculator with buffers in between we showed around 50% savings in resource usage. By altering the shape of correlation windows we found that a short and wide rectangular window reduced storage requirements without sacrificing quality compared to a square one. 1 Introduction A collision avoidance system for a device with mobility requires the ability to build a three-dimensional map of its environment. Traditionally, this has been accomplished by active sensors, which send a pulse - either electromagnetic or sonar - and detect the reflected return[1, 2]. Such active systems work well in the environments with small number of moving devices and thus the probability that active sensors will interfere is low. However, when the density of moving objects becomes high, active systems are easily left in noisy environments. Lots of moving objects with a wide range of speeds and directions create that many reflections at various strengths. A sensor could easily be confused by extremely weak reflections of its own and strong pulses from other objects. Passive systems, on the other hand, are much less sensitive to environmental interference. Stereo vision is one of the representative passive systems. Typical cameras can provide 30 or more images per second and each pair of images can provide a complete three-dimensional map of the environment. However, processing even small low resolution images takes more than a second in software. This is well below the frame rates obtainable with commodity cameras and may be far too slow to enable even relatively slow moving objects to avoid colliding each other. Thus, hardware accelerators are required in order to obtain real-time 3D environment maps. Software simulations T.D. Hämäläinen et al. (Eds.): SAMOS 2005, LNCS 3553, pp , c Springer-Verlag Berlin Heidelberg 2005

2 300 S. Lee, J. Yi, and J. Kim have determined that correlation techniques using a simple sum of absolute differences (SAD) algorithm perform well [3, 4]. In our previous work [5], we showed that accurate real-time three-dimensional maps are feasible with modern FPGA technology. A SAD correlator with its associated accuracy and speed requirements could be fitted onto a single commercially available FPGA. In this paper, we present versions of optimization of the SAD correlator with respect to resource usage. By decomposing the original SAD correlator into column SAD calculator and row SAD calculator with buffers in between we reduced the number of adders from the original SAD correlator. Also, by utilizing rectangular windows instead of traditional square windows we saved more resources without sacrificing accuracy. In the remainder of the paper, Section 2 briefly surveys the stereo image matching techniques, Section 3 provides the concept of the SAD algorithm and Section 4 introduces our SAD correlators. Section 5 then provides the results of our experiments. Finally, Section 6 summarizes our results and conclusions. 2 Stereo Image Matching Stereo vision refers the problem of extracting 3-dimensional structure from two(or more) images taken from different viewpoints[6]. Image matching is an important part in stereo vision system involving two main problems: correspondence and reconstruction. The correspondence problem consists of determining, given a pair of stereo images, which parts in the left(right) image correspond to which parts in the right(left) image. Since there are parts of a scene projected on a single image only it must be able to tell the parts in each image that should not be matched. The reconstruction problem consists of determining, given a set of corresponding parts of a pair of stereo images, 3-dimensional location and structure of the observed objects. Ideally, we want to find all matching pixels of a pair of stereo images. However, the value of brightness of a single pixel is too low to determine its correspondence. Instead, sets of pixels are used for real stereo matching algorithms. Correlation-based methods and feature-based methods are the two representation stereo matching classification [6, 7]. In correlation-based methods, image windows, arrays of neighboring pixels, of fixed size are used. Given a pair of stereo images, one window is fixed in the left(right) image and the other window is moving in the right(left) image. By comparing the windows from the pair of images correlation is measured and the window, that maximizes the similarity criterion, is determined. Normalized Cross-Correlation(NCC), Sum of Squared Differences(SSD), Sum of Absolute Differences(SAD), Census, and Rank algorithms are popular matching metrics[7, 8]. Feature-based methods use a sparse set of features instead of image windows. These include occlusion edges, vertices of linear structures, prominent surface markings, zero crossings and patches by the Moravec operator[9]. Corresponding elements from a pair of stereo images are given by the most similar feature pair, the one associated to the minimum distance. Feature-based methods cannot detect small changes in stereo images and is not suitable when images have no boundary.

3 Real-Time Stereo Vision on a Reconfigurable System SAD Algorithm Area-based correlation algorithms attempt to find the best match between a window of pixels in one image and a window in the other image[6]. The matching process is illustrated in Figure 1. The window centerd on pixel P in the left image is moved through the disparity range until the best match is found with a windwos centerd at P in the right image. aligning the two cameras to meet the epipdar constraint ensures that P must lie on the same scan line in each image [6, 7]. Fig. 1. Correlation based matching In the SAD algorithm, the criterion for the best match is minimization of the sum of the absolute differences of corresponding pixels in a window. The correlation algorithm has a regular structure and simple data flow making it good for implementation in reconfigurable hardware. The SAD function is defined to be C(x,y,δ)= wh 1 y=0 ww 1 I R (x,y) I L (x + δ,y) (1) x=0 The SAD function C(x,y,δ) is evaluated for all possible values of the disparity, δ, and the minimum is chosen. In the equation, I R () and I L () mean right image and left image respectively. The x, y represent coordinates in pixel in a single image, ww and wh represent window width and height, and δ represents disparity number. For parallel camera axes, δ ranges from 0 for objects at infinity to for objects at the closest possible to the camera. The correlation algorithm has regular structures having abundant parallelism - C(x,y,δ) can be evaluated in parallel for each δ [0, ]. The SAD function requires only adders and comparators for which modern FPGAs provide good supports. However, accurate depth maps require large disparity ranges and high resolution images - both of which provide challenges to fitting a full correlator on a single FPGA. 4 SAD Correlator We have implemented versions of the SAD correlation algorithm in VHDL and synthesized them to determine their resource requirements and performance. Each version completely accomplishes the SAD correlation algorithm and has the following features:

4 302 S. Lee, J. Yi, and J. Kim Fig. 2. Block diagram for the SAD correlator v1.2 Table 1. SAD Correlator Resource requirements SAD correlator Object Count Size shift register register 2 sl (wh 1)+ + 1 buffer maker subtractor + 1 wh adder + 1 wh 1 buffer buffer 1 ww ( + 1) disparity calculator adder + 1 ww 1 minimum detector comparator 1 SAD correlator v1.0[5] fully implements the SAD correlation algorithm without any optimization SAD correlator v1.1 decomposes SAD correlator into column SAD calculator (buffer maker) and row SAD calculator (disparity calculator) placing buffers in between reducing the number of adders from the SAD correlator v1.0 SAD correlator v1.2 does a certain approximation in SAD calculation by ignoring the least significant bit (reducing the number of bits in adders): at the cost of accuracy fo further save space from the SAD correlator v1.1 A block diagram of the SAD correlator v1.2 is shown in Figure 2. Pixels stream in from both cameras into the long left and right shift registers, which store sufficient

5 Real-Time Stereo Vision on a Reconfigurable System 303 pixels so that all the pixels in a correlation window are available to the buffer maker at the same time. The key parameters determining the size and performance of an SAD correlator are 1 the scan line length, sl, 2 the window width, ww, 3 the window height, wh, and 4 the maximum disparity,. Basic resource requirements are indicated in Table 1. To a first approximation, the resource requirements for an SAD correlator are given by: cost SAD 2 (sl (wh 1)+ + 1) c reg (shi ft register) +( + 1) (wh c AD +(wh 1) c sum (bu f f er maker) +ww ( + 1) c reg (bu f f er) +( + 1) (ww 1) c sum (disparity calculator) + c comp (comparator) +c overheads (control, etc.) Where c AD is the cost of absolute difference circuit, c sum is the cost of an adder, c comp is the cost of a comparator, c reg is the cost of a pixel register, c overheads is the cost of control and steering logic. This relation should be a good predictor for low values of all the application parameters, where all overheads can be lumped effectively into the single overheads term. Key contributors to the delay of the correlator are from the (wh 1) adders in buffer makers and the (ww 1) adders in disparity calculators. A simple VHDL model which performs the additions in a loop adds a delay of O(wh + ww 2) to the circuit. However, for better performance we use a tree adder, which costs delay of O(log(wh + ww 2)). Note that the synthesizer is able to produce a compact circuit with the tree adder using the + operator, despite the triangular shape of the tree. (2) 5 Experimental Results For this experiment we use Xilinx Virtex-II XC2V8000 FPGAs[10] with scan line length, sl = 320 and the maximum disparity, = 32 (accuracy depends on the disparity value, so we ran trials to determine the value of ). Figure 3 shows the hardware Fig. 3. Resource usage vs. window size (ww wh) for versions of the SAD correlator

6 304 S. Lee, J. Yi, and J. Kim resource usage for the SAD correlators with various window sizes. The X-axis represents the square window sizes used in the experiment and Y-axis is the number of slices occupied. (a) original left-image (b) original right-image Fig. 4. A pair of Tsukuba images for test inputs Figure 5 shows samples of resulting depth maps of versions of SAD correlator with windows size of 8 8(ww wh). A pair of the input images in figure 4 - Tsukuba image [3, 11] - consisting of pixels was used as a test inputs. From the figure 5, we can see that there is little difference in depth map among the versions of SAD correlator. (a) SAD correlator v1.0 (b) SAD correlator v1.1 (c) SAD correlator v1.2 Fig. 5. Sample depth maps of versions of the SAD correlator with 8 8 window Figure 6 shows the simulation waveform using the Tsukuba image. The process time of one frame image is less than 120,000 c.c. The table 2 summarizes the performance of the SAD correlator v1.0 with various test images including the Tsukuba images, when it works in 10 MHz. It is unnecessarily common to use square matching windows in correlation-style stereo algorithms. If you carefully look at the algorithm you can see that matching

7 Real-Time Stereo Vision on a Reconfigurable System 305 Fig. 6. The simulation waveform using the Tsukuba images Table 2. The performance of the SAD correlator v1.0 in various environments Image size maximum disparity window size rate (pixel) ( ) (ww wh) (frame/sec) process only uses a small part of each scan line at any time - specifically, ww from the left image and ww+ +1 from the right image. The remaining pixels are stored in shift registers for use in subsequent cycles. Pixels in surrounding scan lines are only used to support matching by reducing noise effects. Figure 7 shows samples of depth maps of SAD correlator v1.2 for the same Tsukuba input images with various rectangular windows. We can find that a short and wide (wh < ww) window produces similar matching quality to the square one. However, We can see that a considerable amount of space can be saved in an FPGA by using rectangular (wh < ww) windows. Figure 8 shows a FPGA resource usage for various rectangular window sizes as well as square ones. We can conclude that a rectangular window in SAD correlators is worth utilizing, especially when ww is sufficiently large, since it saves lots of space (nearly 50% profit) without sacrificing quality. (a) 16 4 window (b) 16 8 window (c) window Fig. 7. Sample depth maps of the SAD correlator v1.2 with various window sizes (ww wh)

8 306 S. Lee, J. Yi, and J. Kim 6 Conclusion Fig. 8. Resource usage vs. window size (ww wh) for various disparities Accurate real-time 3D depth maps are feasible with modern FPGA technology. While the feasibility of a proposed application can be testified, in principle, by simply counting circuit elements needed to implement a module and using those counts in equation 2, place and route tools have to work from high level models and may have problems allocating and laying out circuits that a human engineer may not. FPGA implementations are also constrained by availability of routing resources and this factor is much harder to estimate than logic cell needs, thus practical trials of the type we carried out here are counts to determine real cost factors. Decomposing a SAD correlator into column SAD calculator and row SAD calculator with buffers in between reduces number of adders: easily saving around 50% in resource usage. Simulation results show that altering the shape of the correlation window can further reduce the number of cells needed for inactive parts of scan lines. The SAD correlator v1.2 described in this paper will provide real-time performance at pixel clock rates up to 10 MHz. Acknowledgements This research was partially supported by the MIC(Ministry of Information and Communication), Korea, under the Chung-Ang University HNRC-ITRC(Home Network Research Center) support program supervised by the IITA(Institute of Information Technology Assessment). References 1. Olson, C.F.: Maximum-likelihood image matching. IEEE Trans. Pattern Anal. Mach. Intell. 24 (2002) Sebe, N., Lew, M.S.: Maximum likelihood stereo matching. In: ICPR 00: Proceedings of the International Conference on Pattern Recognition (ICPR 00)-Volume 1, Washington, DC, USA, IEEE Computer Society (2000) 1900

9 Real-Time Stereo Vision on a Reconfigurable System Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision 47 (2002) Leclercq, P., Morris, J.: Assessing stereo algorithm accuracy. In: IVCNZ 02: Proceedings of Image and Vision Computing 02, University of Auckland, Auckland, New Zealand (2002) Yi, J., Kim, J., Li, L., Morris, J., Lee, G., Leclercq, P.: Real-time three dimensional vision. In Yew, P.C., Xue, J., eds.: Asia-Pacific Computer Systems Architecture Conference. Volume 3189 of Lecture Notes in Computer Science., Springer (2004) Barnard, S.T., Fischler, M.A.: Computational stereo. ACM Comput. Surv. 14 (1982) Brown, M.Z., Burschka, D., Hager, G.D.: Advances in computational stereo. IEEE Trans. Pattern Anal. Mach. Intell. 25 (2003) Leclercq, P., Morris, J.: Robustness to noise of stereo matching. In: ICIAP 03: Proceedings of the 12th International Conference on Image Analysis and Processing, Washington, DC, USA, IEEE Computer Society (2003) Grimson, E.L.: Computatoinal experiments with a feature based stereo algorithm. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-7 (1985) Virtex- platform fpgas : Complete data sheet (2002) Middlebury stereo vision page (2005)