GPU-Based Real-Time Beating Heart Volume Rendering Using Dynamic 3D Texture Binding

Transcription

1 GPU-Based Real-Time Beating Heart Volume Rendering Using Dynamic 3D Texture Binding Qi Zhang a,b, Roy Eagleson a,b,c and Terry M. Peters a,b a Imaging Research Laboratories, Robarts Research Institute, London, ON CANADA; b Biomedical Engineering, University of Western Ontario, London, ON CANADA; c Department of Electrical and Computer Engineering, University of Western Ontario, London, ON CANADA ABSTRACT Simulation and real-time display of dynamic three-dimensional (3D) cardiac images has important applications in minimally invasive image-guided cardiac surgery and therapy. However, in practice, the high computational cost usually prohibits its application in a real-time medical environment, or else the low image quality does not satisfy the clinical requirements. Surface based models or orthogonal planes are often employed instead, but in the process important intra cardiac data are lost, and intuitive spatial anatomical relationships are eliminated. In this paper, we first take advantage of the programmability, parallelism and increased computational precision of modern graphics processing units (GPUs) to build a real-time 3D rendering engine based on ray-casting, directly running on the graphics vertex and fragment processors, in which the OpenGL Shading Language (GLSL) is utilized as a GPU programming API. This approach provides enhanced image quality similar to software-based implementations, but its rendering speed is competitive with the traditional but inferior quality slice based volume rendering approaches. Then, we propose a new dynamic volume texture binding scheme, and embedded it into our 3D rendering engine to permit real-time visualize the dynamic beating heart. Keywords: Dynamic binding, real-time volume rendering, ray-casting, GPU, OpenGL Shading Language (GLSL), 4D cardiac visualization 1. INTRODUCTION Cardiovascular diseases are the single most frequent cause of death in North America for both men and women among all racial and ethnic groups. Their early diagnosis and treatment is crucial in order to reduce mortality and to improve patients quality of life. Traditional open heart surgery involved a sternotomy and the use of a heartlung machine. Newer techniques have utilized minimally-invasive approaches, often supported by Robotics. 1, 2 However, a new technique that performs surgery on targets inside the beating heart by introducing tools into the chambers through the heart wall was introduced recently by Guiraudon et al.. 3 Under these conditions high quality visualization is required, and can be provided by integrating real-time intra-cardiac ultrasound with a registered pre-operatively acquired dynamic MR or CT volume. 4 High quality visualization of this dynamic environment is critical to the success of these new surgical procedures. Even with the high speed CPU processors available in today s computers, to achieve interactive visualization of a given medical data set, it is necessary to make use of additional graphics hardware support. 5 Often a surface-based approach is not an optimal visual solution for imaging of volumetric dynamic data since the intra anatomical structure cannot be visualized and many important details are lost. Volume rendering, which preserves such details, therefore is a preferred approach to visualize such data. 6, 7 However, volume visualization is computationally expensive, even with graphics hardware support, the high quality real-time dynamic cardiac visualization is still a challenge for current popular 3D rendering algorithms and computational environments. Further author information: (Send correspondence to Q. Z.) Q. Z.: qzhang@imaging.robarts.ca, Telephone: ext R. E.: eagleson@uwo.ca, Telephone: ext T. M. P.: tpeters@imaging.robarts.ca, Telephone: ext

2 In this paper, we describes our solution to these challenges. New generation graphics hardware provide us programmability, on which we can directly implement complicated volume rendering algorithms. We have developed a 3D rendering engine, which is based on an optimized ray-casting algorithm running directly on the vertex and fragment processors. The GPU contains multiple graphics processors working in parallel, and floating point precision computation is supported by the graphics vertex and fragment processors. The parallelism and increased computational precision allow our algorithm to render the cardiac data set with high image quality in real-time. In addition, we describe a new dynamic volume texture binding technique, which allows us to dynamically bind the MR cardiac volumes to our 3D rendering engine during dynamic cardiac visualization without degrading performance. 2. METHODS We first introduce the GPU programming pipeline, followed by a detailed description of our GPU-based raycasting. Then we present the approaches of our 4D MR data acquisition. Finally, we illustrate a new dynamic texture binding technique and its application in our real-time beating heart visualization Programmable GPUs Modern GPUs are designed to operate on large continuous streams of vertex and fragment data. Vertices are points in 3D space that define graphics primitives like triangles, polygons, rectangles, etc, which are used to build up the geometry of the scene to be displayed. Textures are images that can be mapped onto any of the graphics primitives to add details to a scene. The term fragment is used to describe the data element before they are output as pixels to the screen. Texture mapping follows the vertex stage in the graphics pipeline, during which the texture image is interrogated for the correct color to be added to the fragment before it is converted to a pixel or pixels. In older graphics hardware, the geometry sent to the GPUs was static, and the operations that could be performed on fragments were limited. The GPU functions were therefore restricted to certain fixed pipelines such as texturing and lighting. The new generation of graphics cards has evolved to a situation where the vertex and texture units are now user programmable. These elements are commonly referred to as vertex and fragment processors, which can be programmed using high-level programming languages such as the OpenGL Shading Language (GLSL), created by the OpenGL ARB (Architecture Review Board) for programming graphics processing units directly, permitting control of the graphics pipeline at the lowest level. Vertex shaders are designed to allow more control over vertex transformation on the GPU itself, and fragment shaders are introduced to allow more control of how textures are applied to the fragments. A further improvement in the new GPUs is the increase in pixel depth from 32 bits per pixel to 128 bits per pixel, which means that each red, green, blue, and alpha component can now have 32-bit floating point precision throughout the graphics pipeline. The increased data precision, combined with the enhanced programmability of the current GPUs, means that a complicated 3D rendering algorithm can now be implemented directly on GPUs with much greater efficiency and visual realism effects than ever before Ray-Casting Pipeline Ray-casting is based on a model that describes real world physical phenomena. The behavior of light interacting with translucent volume, which is illustrated by the optical model evaluation equation (1), is a popular technique for rendering scalar volumes. I(a, b) = b a g(s)e s a τ(x)dx ds (1) where I(a, b) is the intensity of one pixel, ds is the direction of the ray; g(s) is the source term describing the illumination model, and τ(x) is the extinction coefficient. The ray-casting procedure extends a ray from a specified camera position through each pixel on the screen viewport, and into the volume. These rays are then sampled at uniform or variable intervals, and for each sample point the volume is interpolated to acquire a scalar value in that point. Transfer functions are used to map the scalars to colors and opacities. For each ray, these

3 MR scanner ECG signal Data collection Data storage Volume texture loading Real-time dynamic 3D cardiac visualization system (CPU) ID: 1 ID: 19 ID: 20 Dynamically texture binding Dyn_ID = i ID = Dyn_ID Real-time volume Fragment ray casting } processors Pixel color Graphics Hardware (GPU) One cardiac cycle 20 volumetric MR images Texture ID Sending i (i = 1,,20) Dynamic 3D cardiac real-time visualization Figure 1. The process of cardiac dynamic real-time visualization. 20 MR images are acquired from a 1.5T GE CVi scanner with time information derived from ECG signal. Twenty MR volumes are loaded into graphics cards as a set of volume textures, and a dynamic texture binding technique is employed to real-timely bind different volume texture ID with our ray-casting based 3D rendering engine, which directly runs on the vertex and fragment processors. This 4D MR cardiac visualization process renders each of the volumes as a new frame in real-time. values are composed according to the ray-casting integral equation (1) and the intersected screen pixel is set to the resulting color. This process is repeated for each screen pixel and results in an image of the volume. Ray-casting has the advantage of flexibility and high rendered image quality. Unfortunately, the software based implementations are too slow to be suitable for real-time medical applications. However, modern GPUs provide us programmability and parallelism, which can be thought of as highly parallel Single Instruction Multiple Data (SIMD) type processors. Our ray-casting algorithm is implemented by GLSL vertex and fragment shaders, which runs directly on the graphics vertex and fragment processors. The floating point precision support on these graphics processors ensures high image quality. The rendering pipeline is described by the following sequence. 1. Load the volumetric raw cardiac data and color table to the graphics memory as two multi-textures, one 3D and the other 2D: uniform sampler3d volumetexture and uniform sampler2d colortable. Their texture coordinates are loaded to the vertex shader, and are transformed to two built-in varying variables, i.e., vertex coordinates. These coordinates are interpolated, then input to the fragment shader. 2. In the situation when the volumetric raw cardiac dataset is not a cube, we scale a normalized cube along each axis by using a set of scaling factors, which are derived by using each of the three side lengths (according to x, y and z directions) to divide the maximum side length of the volume dataset. 3. In the fragment shader, the interpolated coordinates are first scaled by the scaling factors derived from the step 2, and is then used as the point of entry for the casting ray. The algorithm then obtains the camera position from the model view matrix. The casting ray entry point and the camera are utilized to compute the direction of the casting ray, and then is scaled by the scaling factors, which are the same as the ones used to scale the interpolated coordinates. 4. The volume is sampled at each sampling point pos along the casting ray: texture3d(volumetexture, vec3(pos)), acquiring voxel values from the R, G, B and Alpha channels. The value in the alpha channel, in which the scale value of the volume dataset is stored, and the value from the alpha channel of the previous sampling step, are employed to sample the 2D texture based color table: texture2d(colortable, vec2(scalar)). 5. Alpha blending is utilized to assign the derived color value to the destination color. 6. If the opacity has exceed a preset opacity threshold, exit the loop; otherwise, go to the next step in the loop. If the casting ray has left the volume, exit the loop; otherwise, go to the next step in the loop.

4 Figure 2. GPU-based ray-casting results of volumetric cardiac images with different colors and volume classifications. 7. Alpha blend the derived destination color with the background color, and set the result as the output fragment color gl FragColor D MR Data Set of the Human Heart The dynamic 3D MR images of a volunteer were acquired in the coronal plane using a 1.5T GE CVi scanner (GE medical systems, Milwaukee). This in-vivo human heart was scanned in a sequence to obtain a series of 20 ECG-gated, 4D MR images, each depicting a snapshot of the heart anatomy at different time-point in the cardiac cycle. This 3D MR cardiac image sequence (CIS) represents the motion of the human heart throughout a cardiac cycle, and is utilized as the 4D data set of human heart in our dynamic cardiac real-time visualization system Cardiac Dynamic Visualization Fig. 1 describes the cardiac dynamic simulation process. The twenty MR volumetric images are first loaded into graphics card memory as a set of s in our dynamic volume visualization system, during which a unique volume texture ID is set to each of these MR cardiac volumes when these s are generated. During the real-time dynamic rendering process, our ray-casting based real-time 3D rendering engine runs directly on the GPU s vertex and fragment processors with rays being cast from every screen pixel through the texture volume to perform the pixel color computation. The actual texture volume through which the rays traverse is dynamically determined by the texture ID, which is sent by our C++ based dynamic cardiac visualization system using volume texture binding technique. This dynamic volume texture binding technique enables our GPU-based real-time 3D rendering engine to visualize different cardiac volume at different time point, without the necessity of physically reloading the MR cardiac volumes during the rendering process. In our system, we send the texture ID of the 20 MR volumetric images to our real-time ray-casting based 3D rendering engine in a loop sequence, so we can dynamically visualize the beating heart with high image quality. 3. EXPERIMENT RESULTS We have implemented our real-time 4D cardiac visualization on a PC equipped with an Nvidia GeForce 6800 CS GPU with 256MB graphics memory, a Pentium IV 3.2GHz CPU processor and 3GB DDR2 533 main memory. The application system is developed with C++, OpenGL and OpenGL Shading Language (GLSL). The running environment is Windows XP, and Nvidia driver version is Each of the 20 MR cardiac image volumes has a dimension of with a voxel size of mm Static Heart In our static visualization experiment, we select four volumetric cardiac images from our series of 4D MR cardiac image dataset, serial numbers 1, 6, 18 and 20 respectively. The viewport window size is For the perspective projection matrix used in our rendering system, the field of view angle is 60 0, the aspect ratio of

5 the viewport window s width to its height is 4/3, the near clipping plane is 0.1, and the far clipping plane is 4.0. Under these conditions we test the performance of our rendering engine measured by frame rate (frames per second). We find the rendering speed is almost uniform, does not have notisable variations with time. To demonstrate the rendering efficiency, we select continuous 10 seconds from our testing experiment and list the results in Table 1, from which we can see the uniformity of the rendering performance, and the approximate frames rates: minimum is 36, average is 37 and maximum is 42. Fig. 2 demonstrates the image quality of four rendered cardiac volumes using our 3D rendering engine, in which a transfer function is utilized to assign different colors, and to classify different regions of the heart. Based on the judgement of physicians, the image quality is superior and good enough for real-world clinical applications. Table 1. Performance of static heart volume rendering, measured by FPS (Frames Per Second). Time (second) Minimum FPS Average FPS Maximum FPS Beating Heart Our dynamic visualization testing environment is the same as for the above static cardiac dataset experiment. Table 2 illustrates our visualization system performance for continuous 10 seconds during our rendering efficiency test, from which we can see that the performance uniformity and frame rates are almost the same as the static heart rendering experiment. So we can conclude that with our dynamic volume texture binding technique, the volume change during the rendering process does not affect our our system s rendering speed. Fig. 3 shows four examples of the cardiac images during the dynamic visualization procedure, for images with the texture ID 1, 6, 18 and 20 respectively. The cardiac volumes are clipped to demonstrate the internal cardiac structure deformations. Table 2. Performance of beating heart volume rendering, measured by FPS (Frames Per Second). Time (second) Minimum FPS Average FPS Maximum FPS CONCLUSIONS AND FUTURE WORK Although the analysis of static volumetric heart images can yield information such as chamber volumes and morphology, the dynamic 3D image analysis and display provides more intuitive information on cardiac behavior and physiologic parameters. 8 Surface rendering cannot demonstrate the intuitive anatomical structure, and general texture mapping based volume rendering techniques usually cannot satisfy the image quality requirement for surgical visualization applications. Standard ray-casting algorithms create superior image quality, but in general the high computational cost makes them too slow to be applied in a real-time medical environment. In this paper, we have addressed these difficulties. Taking advantage of the programmability, increased computation precision and parallelism of modern GPUs, we have developed a volume rendering algorithm based on ray-casting that directly running on graphics vertex and fragment processors, in which an early ray termination acceleration technique is embedded. Our 3D engine has been utilized to demonstrate the real-time display of high quality MR cardiac images. In addition, we have implemented a new dynamic volume texture binding

6 Figure 3. Three MR images are snapped from a dynamic cardiac cycle, which are clipped to emphasize the heart deformations, and the serial numbers 1, 6, 18 and 20. technique, which has been embedded into our visualization system to render a series of 4D MR cardiac images in real-time, enabling us to dynamically change the texture volume without degrading the rendering efficiency. The next step will be to use our dynamic volume visualization system in a real-time multi-modality display of 4D cardiac images, that will incorporate the real-time cardiac registration results, in which 2D and 3D ultrasound will be dynamically visualized in the same rendering window as the dynamic 3D MR cardiac images. ACKNOWLEDGMENTS The authors acknowledge the financial support from the Canadian Institute of Health Research (CIHR), the Ontario Research & Development Challenge Fund (ORDCF), the Canada Foundation for Innovation (CFI), and the Ontario Innovation Trust (OIT). REFERENCES 1. G. Lehmann, D. Habets, D. W. Holdsworth, T. M. Peters, and M. Drangova, Simulation of intra-operative 3D coronary angiography for enhanced minimally invasive robotic cardiac intervention, in MICCAI (2), T. Dohi and R. Kikinis, eds., 2489, pp , T. Mäkelä, P. Clarysse, O. Sipilä, N. Pauna, Q. Pham, T. Katila, and I. E. Magnin, A review of cardiac image registration methods, IEEE Trans. Med. Imaging 21(9), pp , G. M. Guiraudon, D. L. Jones, A. C. Skanes, D. Bainbridge, C. M. Guiraudon, S. M. Jensen, X. Yuan, M. Drangova, and T. M. Peters, En bloc exclusion of the pulmonary vein region in the pig using off pump, beating, intra-cardiac surgery: A pilot study of minimally invasive surgery for atrial fibrillation, Ann Thorac Surg. 80, pp , X. Huang, N. A. Hill, J. Ren, G. Guiraudon, D. R. Boughner, and T. M. Peters, Dynamic 3D ultrasound and mr image registration of the beating heart, in MICCAI (2), T. Dohi and R. Kikinis, eds., 3750, pp , R. Westermann and T. Ertl, Efficiently using graphics hardware in volume rendering applications, in SIGGRAPH 98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pp , ACM Press, (New York, NY, USA), S. Roettger, M. Kraus, and T. Ertl, Hardware-accelerated volume and isosurface rendering based on cellprojection, C. Rezk-Salama, K. Engel, M. Bauer, G. Greiner, and T. Ertl, Interactive volume on standard pc graphics hardware using multi-textures and multi-stage rasterization, in HWWS 00: Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on Graphics hardware, pp , ACM Press, (New York, NY, USA), E. A. Zerhouni, D. M. Parish, W. J. Rogers, A. Yang, and E. P. Shapiro, Human heart: Tagging with mr imaging - a method for noninvasive assessment of myocardial motion, Radiology 169(1), pp ,