Consumer graphics cards for fast image processing based on the Pixel Shader 3.0 standard

Transcription

1 Consumer graphics cards for fast image processing based on the Pixel Shader 3.0 standard G. Monti, C. Lindner, F. Puente León, A. W. Koch Technische Universität München, Institute for Measurement Systems and Sensor Technology, Theresienstraße 90 / N5, Munich, Germany gianni.monti@tum.de Summary Optical inspection and metrology for industrial applications often require fast image processing. Today this is mostly reached by special developed hardware, which represents a relevant cost factor in image processing systems. Furthermore, such special solutions often cannot be easily adapted to new requirements. Otherwise, CPU based systems feature a good programmability but often not enough processing power for real time purposes. The required computational performance and flexibility can be provided by the latest issue of consumer graphics cards for personal computers equipped with a fourth generation Graphics Processing Unit (GPU) supporting the Pixel Shader 3.0 standard. The GPU contains freely programmable units for geometrical and mathematical processing of textured objects by shader programs. The objective of this paper is to show that the 3.0 standard shaders are now ready to be integrated in industrial environments for real time imaging applications. To this end, we focus on common image processing algorithms, such as linear and nonlinear filters. A benchmark has been implemented to compare the speed of GPU shader algorithms to those running on CPU systems. Keywords: image processing, pixel shader, benchmark, real time 1 Introduction Due to the persistent growing requisites of PC games, such as realistic scenes or fast animations, graphics cards for personal computers have developed from simple output devices to powerful parallel graphics computers. The core part is the GPU, whose present generation belongs with 220 million transistors to the most complex processors on the market. Its superscalar architecture allows fast processing of broad data streams with an instruction set of middle complexity in SIMD (Single Instruction Multiple Data) style. For this purpose the GPU can be connected through a 256 bit wide data bus to high performance GDDR3 RAMs (with sizes of up to 256 MB), in order to perform data rates of 35 Gbyte/s. The evolution of GPUs through different generations shows that the former functionalities given by hardwired components has been discarded for most parts in favor of a freely programmable rendering pipeline. Figure. 1 describes the typical structure of a modern rendering pipeline.

2 Figure 1: Programmable rendering pipeline. A defined number of incoming vertices will be connected to a polygon surface and positioned in a coordinate system. After a rasterization, the surface can be colored by interpolating the color values of the vertices. In the next step each single raster cell is filled with the corresponding result value of a processed input image or by a predefined value. The processing algorithms destinated to the programmable vertex and pixel system are implemented by shaders, which are specific programs that runs on GPU. Two classes of Shaders will be distinguished: vertex shaders and pixel (or fragment) shaders. While vertex shaders provide geometrical transformations, pixel shaders perform a pixel color manipulation of the textured target object. A simplified implementation of both shader types can be achieved by the standardized high level languages: Cg, HLSL and GLSL. In comparison to the last shader standard 2.0, new features permit the implementation of an extensive range of image processing algorithms. The shader standard 3.0 enables the processing of 32 bpp floating point images with a maximum of four channels and a precision of 128 bpp. Thus, high dynamic range input data (featuring more than 8 bpp) from modern capture devices can now be processed. With a Multiple Render Target extension, real and complex data can be rendered in one step, which increases the speed of complex data algorithms. While pixel manipulation of former standards was accomplished in a predifined way for all image pixels, the new Dynamic Branching method permits a dynamic handling of each pixel. Flow control of image and parametric data as well as initialization, compiling, rendering and general hardware control will be achieved by a common Application Processing Interface (API) like OpenGL or DirectX. All shader development languages and APIs are freely available. 2 Integration of graphics cards in imaging systems Utilization of consumer graphics cards for industrial image processing purposes demands an efficient system integration. In considering that graphics cards are subsystems of CPU based personal computers, the following integration methods are reasonable: Direct Operation, Parallel Mode, Combined Mode.

3 Figure 2: Direct operation of a graphics card. Figure 3: Mode. System integration in Parallel In a Direct Operation solution, the I/O image data streams flow directly through the device interfaces of the graphics card; see Fig. 2. The unidirectional incoming control dataflow over the PC interface is quiet small, so that the CPU side components must not achieve a high performance if, required interface standards are given. Video input interfaces can exclusively be connected with S-Video compatible analog capture devices. Image processing results are dumped through a DVI interface. Due to the low maximal resolution and frame rate, as far as the presence of video inputs is limited to graphics cards models of medium performance, the described integration is an inexpensive solution for simple imaging tasks. If a fast image acquisition from high resolution cameras is required, the graphics card should be integrated in the Parallel Mode; see Fig. 3. Any camera interface standard that may be connected to the personal computer with a DMA access to the computer s RAM will be supported. Examples are IEEE 1394, USB, and Camera Link. In Parallel Mode, the graphics card operates like a parallel processing subsystem. After image and parametric data are transferred from RAM to GRAM, a CPU indipendent data processing will be performed. After rendering, data results will be transferred back to RAM. The data transfer rate of the PC bus is decisive for the execution of real time applications. In comparison with the AGP8x port, the data rates have been doubled since the recent introduction of the PCI-Express bus (PCI-E), which confirms the Parallel Mode concept. Additionally, the PCI-E based nvidia SLI technology makes GPU power scalable. The adoption of the first generation SLI Motherboards have shown that GPU processing Power can be increased by a factor of 1.84 simply by insertion of two coupled PCI-E graphics cards. Because Parallel Mode integration graphics cards support a wide range of capture devices, and the adoption of high speed PC interfaces assure fast processing tasks, this is the adequate solution for GPU based real time image processing. The Combined Mode is similar to the Parallel

4 Mode, except that the that the rendering result is output by the DVI interface and not transferred to the PC RAM. This is particularly interesting for real time visual inspection systems for medical prevention and diagnostics, materials examination and similar purposes. Whereas a Direct Operation of graphics cards has already investigated within previous projects at our Institute [4], this paper focuses on a Parallel Mode integration. 3 The image processing framework Our framework is based on the nvidia GeForce 6800 chip set series, wich is part of high end consumer graphics cards with 3.0 shader support. These are integrated in personal computers with Intel and AMD central processing units. Due to the Parallel Mode concept, image processing tasks will be entirely performed inside the graphics card, which means that no CPU processing power is required during the computation on the GPU side. The CPU is mostly disposable and can be used for machine control or human-machine communication tasks. The framework structure is characterized by three interdependent programming layers (Fig. 4). A C++ program running on the CPU constitutes the first layer which represents the basic structure of the framework. Beside of the graphics card hardware initialization, image and parametric data settings and flow control, also initiation of the rendering process is accomplished here. Image processing algorithms are implemented through Cg shaders in the third layer. Cg (C for graphics) [6] is a multiple API (DirectX and OpenGL) high level language for shader programming. Its instruction set is adapted dynamically to the hardware, and deliver a good basis for implementation of scientific algorithms. In our framework, shaders will be compiled at runtime, which facilitates a rapid modification. The programming code of the second layer connects the first with the third layer through OpenGL API commands, which encapsulate the functionality of the graphics card hardware. Contrary to the Windows based DirectX, the platform independent OpenGL enables our framework to run on different operating systems: Windows, Linux and MacOS. Figure 4: Multilayer framework. Appropriate OpenGL extensions enable a fast image processing with the graphics card. The GLUT library is used to obtain platform independent handles to the graphics card. To avoid the limitations of a conventional to Frame Buffer rendering, we use an off-line rendering technique [5] with multiple pixel buffers, which are special areas of the GRAM used as render target. In this manner, Frame Buffer resources can be used by the operating system for user interactions, and the performance of rendering tasks is indipendent of the OS-Interrupts. For implementation of modular algorithms Multiple Rendering Passes and Render to Texture [7] assure fast operation. Since the transfer

5 Filter Ram GPU GRam CPU Type to GRam Processing to RAM Processing 7 7 Edge Detection Moving Average Table 1: Convolution benchmark results in milliseconds. and mapping of textures is more convenient than arrays or matrices, large parametric data will be stored as images. 4 Benchmarks and results In this section, the suitability of our GPU based image processing for real time purposes will be compared to equivalent CPU algorithms implemented by utilization of the last Intel Performance Primitives (IPP) library running on a 3 GHz Intel Itanium processor. The results of the shader framework were obtained by using a graphics card with the nvidia GeForce 6800 Ultra chip-set and a PCI-E 16x interface. All benchmark results were obtained by averaging of 50 values. Shader processing times are subdivided in image upload (to GRAM), rendering and image download (to RAM). For capability determination of the afore described shader system we have realized benchmarks for the following filter classes: convolution filters, morphological filters. 4.1 Convolution benchmark The wide application range of Convolution filters is mostly used for image enhancement, edge detection and many other processing tasks. Convolution filtering of an input image g(i, j) is given by: r(i, j) = g(i, j) h(i, j) := L 1 2 l= L 1 2 M 1 2 m= M 1 2 g(l, m) h(i l, j m), (1) where h(i, j) denotes the filter kernel function with odd dimensions L and M. To compare processing times between GPU and CPU driven Convolution tasks we have processed a RGB image of 1600x1200 pixels with a dynamic range of 12 bit through two rendering passes. At each rendering step the image is processed by an onedimensional Kernel, in direction of i and j respectively. The processing time comparison of Tab. 1 is obtained by applying a 31x31 kernel for moving average and a 7x7 kernel for edge detection.

6 Size of RAM to GPU Processing GRAM CPU Processing s(i, j) GRAM to RAM Binary Gray scale RGB Table 2: Morphological benchmark results ( : erosion; : dilation; : opening; : closing) in milliseconds. 4.2 Morphological benchmark Morphological filters [1,2] are nonlinear filters which typically used for edge detection, noise removal, image enhancement, image segmentation and binary image analysis. The variety of different morphological filters is based on the combination of two basic operations: dilation D{ } and erosion E{ } which are related to set theory. They can be described respectively by the following equations: D{g(i, j)} = g(i, j) s(i, j) = E{g(i, j)} = g(i, j) s(i, j) = max {g(i l, j m) + s(l, m)}, (2) (l,m) D s min (l,m) D s {g(i + l, j + m) s(l, m)}, (3) where g(i, j) represents the input image, s(i, j) the structuring element, and D s := supp{s(i, j)} the support set of s(i, j). In our benchmark, s(i, j) is set to zero. Table 2 compares the processing times for dilation, erosion, opening (Eq. 4) and closing (Eq. 5) with a 3 3 and a 9 9 square structuring element applied to a 8 bit binary, a gray valued and an RGB image of pixels. g(i, j) s(i, j) = (g(i, j) s(i, j)) s(i, j) (4) g(i, j) s(i, j) = (g(i, j) s(i, j)) s(i, j) (5) 4.3 Discussion In consideration of the predominant data transfer times through the PC bus, the analysis of the applied benchmarks shows that image computing with graphics cards has advantages for processing intensive algorithms. Contrary to the image size dependent data transfer times, the GPU rendering times remain approximately constant which means that the required processing power of the used algorithms is much less than the GPU can achieve. Another important fact is that RGB images are processed faster

7 than gray value images. In order to triplicate the computation speed of gray value image streams, they can be assembled to virtual RGB images. Overall, Graphics Processing Units can be used for an efficient processing of large sized images with complex algorithms. 5 Conclusions The paper has shown that real time industrial image processing with Graphics Processing Units of modern consumer graphics cards can be a cost efficient alternative to dedicated image processing hardware. For continually changing exigencies of a growing image processing market, the shader based implementation of image processing algorithms represents a flexible and efficient instrument. Future improvement, enhancement and availability of Graphics Processing Units is guaranteed by the persistent increasing demands of the PC games industries. With different projects, our Institute will continue to investigate and develop applications for industrial inspection and measurement tasks with such devices. References [1] J. Serra, Imaging Analysis and Mathematical Morphology, Vol. II: Theoretical Advances, Academic Press, London, [2] R.M. Haralick, S.R. Sternberg, R.M. Zhuang, Imaging Analysis and Mathematical Morphology, IEEE Trans. Pattern Analysis Mach. Int. 9(4), , [3] R.C. Gonzalez, R.E. Woods, Digital Image Processing, Prentice Hall, [4] A. Purde, A. Meixner, H. Schweizer, T. Zeh, A. Koch, Pixel shader based real-time image processing for surface metrology, IEEE Instrum. and Meas. Techn. Conference, [5] C. Wynn, Using P-Buffers for Off-Screen Rendering, Game Developer Conference, [6] W.R. Mark, R.S. Glanville, K. Akeley, M.J. Kilgard, Cg: A system for programming graphics hardware in a C-like language, nvidia Corporation, [7] C. Wynn, OpenGL Render-to-Texture, [8] M. Segal, K. Akeley, The OpenGL Graphics System: A Specification (Version1.5),