A comparison of stereo image processing hardware platforms

Size: px

Start display at page:

Download "A comparison of stereo image processing hardware platforms"

Angela Tyler
5 years ago
Views:

1 A comparison of stereo image processing hardware platforms Matthew Linder Valde Systems, Inc. Abstract This white paper compares two different hardware platforms for stereo image processing systems. It discusses their advantages and disadvantages along with other factors for consideration. It then looks at how these approaches are implemented in actual products. 1. Introduction Image processing systems have evolved significantly since their introduction. Major advances have occurred in both the performance of computer hardware and the algorithms that run on them. One basic approach used today for standard single camera systems is to install a camera interface card in a personal computer (PC). The card controls the interfaces to the camera and digitizes the incoming analog signal if necessary. Another alternative is cameras with digital interfaces such as USB or Firewire can be plugged directly into the PCs standard ports without the need for an interface card. These approaches, while not perfect, are very prevalent due to their wide understanding and available product base. However, when stereo systems using this approach are designed, the shortcomings of these approaches are magnified, and the limitations become apparent. Stereo image processing systems have some unique requirements that are not well addressed by these architectures. The alternatives to the PC architecture are few, and may involve custom designed hardware, as in [2]. While affective, the current state of the technology led to solutions that were bulky, cumbersome, and difficult to program. Recently, however, the increasing performance of the latest generation of Digital Signal Processors (DSPs) has increased to a point where they can achieve performance comparable to a 2.5GHz Pentium, while running at only 500MHz. Also, the latest high-speed Field Programmable Gate Arrays (FPGAs) are also contributing significantly to the performance of stereo image processing systems. These FPGAs can achieve performance levels better than a PC by as much as 100 times. These trends can reduce the size, cost, and power consumption of stereo image processing systems while simultaneously increasing the performance. 2. Requirements Image processing system requirements vary greatly for each application since they all have different priorities. Some applications value real time processing above all else, others value accuracy most. Still others require small size or low power consumption. The ease of application development is another parameter that must be considered. Significant delays can occur if the development of the application requires a large learning curve or custom development tools. In addition to these general requirements, stereo image processing systems have other requirements as well. Foremost among these is the need for performance. Not only is there twice as much data as a single camera system, the need to process it in real-time increases. This is due to the fact that the additional data places strain on the system not just from a computational standpoint, but also for memory and bandwidth resources. With more data to operate on, most systems are forced to sacrifice either accuracy, or speed. 3. PC Approach For stereo applications on a PC platform, the most common approach is to simply duplicate the camera input path and add an additional camera. This has advantages and disadvantages. The main advantage the PC architecture has going for it is familiarity, and to a lesser degree, price. From an architectural

2 point of view, the two main disadvantages are the data bottleneck in the system and the performance of the operating system. The bottleneck occurs when data from the camera is transferred to the CPU. This bus is most often the PCI bus, but may be different depending on how the camera is interfaced. Regardless of the interface method, this bus is a shared resource that must be allocated for many different system uses. The raw theoretical throughput of the bus is often sufficient, but in practice, due to competing resources it is often significantly less. Even if this was not an issue with single camera systems, the two cameras in a stereo configuration only compound this problem with twice the amount of data. The performance of the operating system can also be far from being ideal for realtime applications. Although there are many choices for PC operating systems, some version of Microsoft Windows is most often chosen, due to the software packages and image processing routines available for it. Alternative operating systems, while better, are still not optimal. The decades of legacy support that Windows requires and the myriad of devices that need to be supported conspire to make it much larger and more complex than required. Additionally, the software is always in a state of flux, with constant patches and upgrades. This necessitates a regression test of software applications for compatibility every time a new version of Windows is released. Finally, there is the cost, as this software is not inexpensive, and if the developed system is to be deployed in large numbers, royalty fees can become significant. 4. DSP/FPGA Approach As an alternative to a standard PC-based system, there are many image processing products that use custom hardware with a variety of microprocessors. These standard products work very well for the applications for which they were designed, which is principally industrial automation and inspection. Since these vision sensors come with their own software packages, there is no level of customization allowed. If the application requires a custom function not supported by the library, there are few options. Also, often these products have just the level of performance required for the application, since adding unnecessary levels of performance only drives up the cost and power consumption. DSPs can be programmed with standard C/C++ software, and while all software is not directly portable, a complete rewrite is often not required. DSPs offer the performance of a desktop PC in a much smaller, lower power package. Due to the proliferation of consumer video products such as digital cable TV and digital satellites, there has been an explosion in video-oriented DSPs. These DSPs offer video ports directly into the processor, negating the need for DMA accesses of video information via an external memory bus, freeing it up for use by the processing algorithms. FPGAs have been used for stereo image processing before in [1][2][3][4][5][6]. These implementations demonstrate the level of performance achievable by pure hardware based solutions. However impressive the performance levels are, these implementations still suffer from the traditional problems of being difficult to program as well as being large and cumbersome. The latest generation of FPGAs has improved significantly. New FPGA architectures lend themselves to signal processing applications, further enhancing their usefulness. Also the tools to program them have improved, with many offering Graphical User Interfaces (GUIs). In addition, there are many pre tested and compiled libraries available. A system that combines a DSP and an FPGA, offers flexibility and familiarity of software development with the DSP and the highest performance with the FPGA. 5. Comparisons In this section, comparisons are made between traditional PC-based approaches and DSP / FPGA combinations. These comparisons are based on the following parameters: size, cost, power, flexibility, and performance. In each case, the merits of each will be discussed. 5.1 Size While size may be an important consideration when selecting the most appropriate solution, it is often one of

3 the first parameters to be sacrificed. The answer frequently is just to make the implementation bigger. Unfortunately, size is also difficult to quantify, since there are few industryaccepted standards, and because as integration continues, sizes continue to shrink. However, these process improvements benefit both approaches equally, and, therefore, a snapshot in time of currently available approaches is used. A survey of available compact PCs reveals an average size of approximately 250 cubic inches. This varies greatly but is limited to a degree by the IO required. The space required for all the connectors can be significant, and if the unit allows plug-in cards or PCMCIA slots, the size goes up considerably. By comparison, a solution with a DSP and an FPGA occupies only approximately 60 cubic inches. This seems to be due to the fact that the internal electronics and IO requirements are not bound by requirements for legacy devices and software. 5.2 Cost Depending upon the application, cost may or may not be of concern. Often, cost can be used to offset other parameters such as performance. It is possible to purchase additional performance, smaller size, or lower power, but this can lead to an imbalance in the price requirement. Here, the compact PCs seem at first to benefit from the market pressures at work in the consumer world. Manufacturers constant drive to reduce cost has benefits here. However, the low initial cost often does not include aspects such as any additional external devices or interfaces required. Costs for industrial temperature grade components are often required, as are additional mechanical mounting considerations for various environments. Once all these costs are added, the price can rise quickly, and a compact PC that started out with an initial price of around $999 can easily reach the $2k to $3k range. Commercially available DSP/FPGA solutions have similar $2k to $3k price ranges since they often do not benefit from the economies of scale enjoyed by the compact PCs. 5.3 Power Power consumption is often the second most critical design consideration after performance. The reason is simple. Since some systems are battery powered, the length of time they can remain operational is determined by the amount of energy stored onboard. Higher power consumption units translate directly into reduced operational time. In this area, the compact PC solution falls significantly short. Even with the advent of energy star compliant PCs and advanced power management techniques, the power consumed by a typical compact PC is approximately 50 watts. Large amounts of power consumed also translate into large amounts of heat that must be removed. This alone can often be an issue since systems are often enclosed in environmentally sealed enclosures. Excess heat must be dissipated into the mechanics or vented outside. To cope with these elevated power levels, compact PCs often incorporate forced convection cooling. This can also be a liability since failure of the cooling fan will lead to overheating and system shutdown. Often these fans are also equipped with an air filter to prevent debris from entering the unit. This means preventative maintenance is required for periodic filter cleaning. By comparison, a DSP/FPGA solution consumes approximately 10 watts maximum. These power levels allow extended operational times and remove the requirement for an external fan, which allows a sealed enclosure for more environmentally extreme environments. 5.4 Flexibility The requirement of flexibility is extremely important since requirements and algorithms are constantly changing. Traditionally, there have been two extremes when it comes to flexibility; the most flexible being software based and the most inflexible being hardware based. The problem is that these extremes are on opposite ends of the performance spectrum, with pure software the slowest, and pure hardware the fastest. Compact PCs are the embodiment of software flexibility. With a digital interface camera, there is no dedicated

4 hardware processing specifically for image processing. However, with the very well understood and supported platform, software changes are very inexpensive and quick. The problem with FPGAs is the relatively difficulty of programming them, as typically code is written in a high level VHDL or Verilog language, with specific hardware knowledge required for efficient implementations. Some solutions exist for easing this requirement using a more graphical approach, but the development of the higher level overall imaging processing function must still be done. A method to program the FPGA and DSP is needed that does not require the developer to be an expert on the hardware, or write all the high and low-level software functions. Recently Matlab from the MathWorks has integrated both DSP and FPGA targets for its software. This graphical development environment allows the developer to start with an existing library of high-level functions. The application can be built graphically, then the DSP compiler and FPGA software are called to generate the necessary code. This code is then easily be downloaded to the hardware. 5.5 Performance All the above parameters are meaningless if the system is not able to perform the entire image processing functions in the time necessary. The topic of performance is difficult to quantify since it depends heavily upon the software architecture. DSPs are specialized processors targeted at signal processing applications. Their architecture allows them to achieve performance levels comparable to common desktop PCs, even while running at only half the clock speed. Although DSPs can be programmed in well understood C/C++, to achieve performance levels promised, the code must often be hand optimized. Even leveraging existing library functions, the higher level image processing application must still be written. FPGAs can provide ten times greater efficiency than DSP-based implementations, especially for data flow algorithms with minimal control processing. The reason for this efficiency is the inherent parallelism available in an FPGA as compared to the Fetch->Compute->Store serial processing of a DSP. This means that FPGA coprocessors can finish computations for multiple functions in a single clock cycle. A pure DSP-based implementation would take many more clock cycles to execute multiple functions. For this reason, an FPGA coprocessors performance can be orders of magnitude higher than a DSP. The compact PC is a much better defined situation. It suffers from the bottleneck and operating system constraints mentioned earlier. Even if these constraints can be eliminated or reduced, the CPU itself often suffers from other demands placed upon it by other devices, such as disk drives and display cards. This can slow the CPU down and lead to wildly varying execution times. Some image processing software makes use of a limited number of multi-media instructions (MMX). Although an improvement, it is often not significant. Just how does a DSP stack up to a typical desktop PC? This is a difficult comparison to make, since the execution times vary depending on the efficiency of the software code, and what else the processor is doing. To make a meaningful comparison, results must be measured using multiple tests. The functions below consist of common signal processing routines Time 10 (ms) P4 (2.53 GHz, 266 MHz DDR RAM) TMS320C64x (500 MHz, 133 MHz SDRAM) Function While all applications are different, and since the mix of functions changes with each application, it is only possible to say the results are comparable. It is important to remember these comparisons are based on a DSP only vs. a CPU only. When a DSP is combined with an FPGA, the results would be vastly different. The comparison of these two situations has many permutations and variables and an in-depth analysis is outside the scope of this paper.

6. Solutions Until now, the data cited in this paper has come from a comparison of high level concepts of the merits of different hardware approaches.

Their products vary in size and mounting configurations, but they all embody the repackaged desktop PC concept. One of their products is shown in the photo below.

5 6. Solutions Until now, the data cited in this paper has come from a comparison of high level concepts of the merits of different hardware approaches. Here, actual products will be discussed to get a better feel for how these concepts are implemented. For the compact PC, systems from a company called Small PC ( are shown here. Their products vary in size and mounting configurations, but they all embody the repackaged desktop PC concept. One of their products is shown in the photo below. Representing the DSP/FPGA approach is a new product from Valde Systems, ( The VS1501 has the processing power of a Pentium, while only consuming 6 watts (max) of power. It contains the latest generation of DSPs, with up to 5760 MIPS available. It has two IEEE-1394 (FireWire) digital camera interfaces for stereo imaging applications. Other features include an industry standard 10/100 Ethernet port, discrete IO for interaction with external devices and a small rugged 6x5x2 inch package. Future products such as the VS1502 will contain high performance FPGAs as well. The VS1501 product is shown in the photo below. 7. Conclusion At low levels, comparing a desktop PC to a DSP is an apples-to-oranges comparison. The PC is much more flexible and can, of course, be used in many more applications. This flexibility is its greatest strength and weakness. Specialized purpose-built hardware will always be better suited for a specific application than a general-purpose solution. The problem with the custom hardware is the amount of time and effort required to realize the benefit from it. Although that level of effort was high in the past, better software development environments for DSP and FPGA code development mean this is no longer as much of a factor. 8. References [1] H. Harlin, C. Zheng. Stereo Video processing for Depth Map. University of Washington. [2] J. Woodfill and Von Herzen, Real-time stereo vision on the PARTS reconfigurable computer, The 5th Symposium on FCCM Proceedings, Pages: , [3] O. Faugeras, et al., Real time correlation based stereo: algorithm, implementations and applications, Research Report 2013, INRIA Sophia-Antipolis, [4] A. Darabiha, Video-Rate Stereo Vision on Reconfigurable Hardware, Master s thesis, University of Toronto, [5] A. Darabiha, J. Rose, W. J. MacLean. Video-Rate Stereo Depth Measurement on Programmable Hardware. University of Toronto [6] B. Draper, J. R. Beveridge, A.P. W. Böhm, C. Ross, M. Chawathe, J. Hammes. Accelerated Image Processing on FPGAs. Colorado State University

Stereo Video Processing for Depth Map

Stereo Video Processing for Depth Map Harlan Hile and Colin Zheng University of Washington Abstract This paper describes the implementation of a stereo depth measurement algorithm in hardware on Field-Programmable