Real-Time Photo-Realistic Rendering for Surgical Simulations with Graphics Hardware

Transcription

1 Real-Time Photo-Realistic Rendering for Surgical Simulations with Graphics Hardware Mohamed. ElHelw, Benny P. Lo, ra Darzi, Guang-Zhong Yang Royal Society/Wolfson Medical Image Computing Laboratory, Imperial College London, London, United Kingdom bstract: Computer-based surgical simulations are being increasingly used for training and skills assessment. They provide an efficient and cost effective alternative to traditional training methods. To allow for both basic and advanced skills assessment, the required perceptual fidelity is essential to capturing the natural behavior of the operator. The level of realism in terms of object and scene appearance determines the faithfulness and hence the degree of immersion experienced by the trainee in the virtual world. This paper presents a novel photo-realistic rendering approach based on real-time per-pixel effects by using the graphics hardware. Improved realism is achieved by a combined use of specular reflectance and refractance maps to model the effect of surface details and mucous layer on the overall visual appearance of the tissue. The key steps involved in the proposed technique are described, and quantitative performance assessment results demonstrate the practical advantages of the proposed technique. 1 Introduction In minimal invasive surgery (MIS), virtual and augmented reality based systems are rapidly becoming an integral part of surgical training. Current high-fidelity simulators offer the opportunity for safe, repeated practice and objective measurement of performance. They provide an economical and time saving solution for acquiring, as well as assessing basic surgical skills [1]. In particular, surgical simulators are found to be valuable for training MIS procedures where the complexity of instrument controls, restricted vision and mobility, difficult hand-eye co-ordination and the lack of tactile perception require a high degree of operator dexterity []. lthough these simulators can accelerate the development of hand-eye skills, there are serious shortcomings with the current technology, particularly in the photo-realism they provide. Hitherto, a significant amount of research has been carried out in photorealistic rendering of simulated surgical scenes [3] and it remains one of the major technical challenges due to the complexity and diversity of internal tissue structures and surfaces properties [4]. With the recent advances in computer graphics architecture, it is possible to provide high fidelity rendering at interactive rates. Highly programmable graphics processor units (GPUs), including floating-point vertex and fragment processors, can offload complex vertex and pixel operations from the central processing unit to the G.-Z. Yang and T. Jiang (Eds.): MIR 4, LNCS 315, pp , 4. Springer-Verlag Berlin Heidelberg 4

2 Real-Time Photo-Realistic Rendering for Surgical Simulations 347 GPU, allowing greater control over the graphics pipeline for real-time per-pixel shading and other procedural effects [5]. Moreover, shading calculations can be performed at the pixel level [6] as opposed to the vertex level in case of fixed functionality pipeline, hence reducing the aliasing of specular highlights and improving visual realism. The purpose of this paper is to present a novel rendering technique based on the programmable graphics pipeline for laparoscopic simulation. Specular reflections are modulated by using a set of reflectance maps, i.e. maps encoding the surface normal distribution, which define the surface light interaction properties. Results are further enhanced with the improved visual appearance of the semi-transparent mucous layer on the top of tissue surface. We describe the key steps involved in the proposed technique and demonstrate its advantages over conventional approaches. Method Specular highlights constitute a vital clue in MIS procedures where 3-dimensional perception is diminished due to the use of -dimensional screens. Surgeons usually rely on specular highlights as a reference for depth, orientation and deformation. Consequently, it is important for surgical simulators to reproduce these highlights as realistic as possible. In the existing literature, a number of approaches for simulating specular highlights have been introduced. Standard graphics PIs such as OpenGL [7] and DirectX simulate this effect by using the Phong lighting model [8] computed at the vertices. Other techniques use environment mapping to map an image of the specular source onto the surface. The results obtained with these methods, however, generally lack visual realism due to the fact that tissue surfaces are not perfectly smooth. more physically accurate model should consider a rough surface augmented with microstructure details such as that proposed by Torrance and Sparrow [9]. major drawback of this approach is that physically-based reflection models can be computationally prohibitive. Figure 1. sample colour texture image (left) and its associated reflectance map (right) For real-time applications, a reflectance map can be used to describe a perturbed normal value for each image pixel (texel). This map can be either derived empirically or generated by using conventional noise functions, e.g Perlin noise [1]. Since the type of noise affects the shape of specular regions, different functions can be used for varying tissue types. Figure 1 illustrates a sample colour texture image and its

3 348 M.. ElHelw et al. associated reflectance map. In this case, the reflectance map is obtained by using a noise image where every texel is considered as a height field, i.e. each texel encodes a single height value at that texel. The normal of the surface at each texel is then found by computing the cross product of the pair of vectors formed by that texel and its neighbors. The calculation is repeated for the other neighboring texels and the average of the normals is stored. larger texel area can be considered in cases when smoother normals are to be computed. During runtime, texture mapping is used for each triangle in the geometric model to extract the per-pixel reflectance map normals used for calculating the specular highlights. However, the normals in the reflectance map are defined in their own coordinate system, therefore they have to be transformed into a coordinate system that is local to the triangle being processed. Such coordinate system, known as the object local surface or texture-space coordinate system, can be defined by using three vectors which constitute its basis: the surface tangent (T), the bi-tangent (B), and the normal (N) as shown in Figure (). Figure. (Left) n example of per-triangle TBN-based coordinate systems. (Right) Per-vertex TBN bases (GBR respectively) used for the tissue model rendered in the results section. Based on this definition, the first two vectors can be computed from the partial derivatives of the object-space coordinates of the triangle in terms of its texture coordinates [11], and, x y T =, u u, x y B =, v v, z B = u z C = v o o B1 1 C 1 1 B C (, B, C ) = [( x, u, v ) ( x, u, v )] [ ( x, u, v ) ( x, u v )] o o 1 1 1, D = (, B, C ) ( x, u, v ) where denotes the dot product, (x, y, z ), (x 1, y 1, z 1 ), (x, y, z ) and (u, v ), (u 1, v 1 ), (u, v ) represent the triangular object- and texture-space coordinates respectively.

4 Real-Time Photo-Realistic Rendering for Surgical Simulations 349 Subsequently, (N) can be computed from the cross product of (T) and (B) or the normal supplied by the original model can be used alternatively. By computing the basis vectors in texture-space, the GPU can be used to efficiently transform the object-space vectors required for specular calculations into the texture-space by using a rotation matrix (R) ( R) T = B N x x x T B N y y y T B N z z z Since (R) is defined for each triangle in the geometric model, a per-vertex rotation matrix is needed to ensure consistent highlights across the triangular mesh. This is obtained by averaging the rotation matrices of the triangles sharing the vertex. In practice, the rotation matrix is calculated for each vertex in pre-processing with its value at the pixel level being interpolated during the rasterisation step of the graphics pipeline, as schematically illustrated in Figure 3. Figure 3. simplified block diagram of the programmable graphics pipeline. To further enhance the visual realism, the effect of surface mucous, which is a wet refractive transparent or semi-transparent layer found on top of the tissue, is combined with the above model. In laparoscopic views, the mucous layer significantly influences the surface appearance by reflecting and refracting incoming light rays. Replicating the mucous effects is a challenging problem and several factors have to be considered including the thickness of the layer, its light interaction properties, and the density and distribution of solid particles within the layer. In this study, mucous is simulated by using a set of refractance maps generated by methods similar to reflectance maps. However, vectors extracted from a refractance map are used to linearly blend between original surface colour and mucous layer colour, which accounts for surface colour variations. 3 Results The proposed technique has been applied to endoscopic surgical simulations. fragment shader was implemented for NVIDI FX graphics hardware, coded in Cg [1]. Figure (4) depicts the results obtained by using the described method compared

5 35 M.. ElHelw et al. to the conventional OpenGL multi-texturing approach. It is evident that the method effectively avoids the problem of plastic-like surface and provides realistic specular highlights. Furthermore, by varying the colour of the mucous layer and using different noise types, tissue appearance can be modified. Figure (5) demonstrates the effect of different noise functions on the visual appearance of the rendered surface. Figure 4. Different views the surface rendered by using the proposed method (left) versus OpenGL multi-texturing approach (right). Notice the plastic-like surface and the hexagonal shape of the specular highlights with the multi-texturing method. To assess the overall computational burden of the proposed algorithm, a detailed performance analysis was carried out. The effect of using different viewport resolutions and polygon counts on performance is demonstrated in Figure 6. It is shown that the viewport resolution is inversely proportional to the achieved frame rate, which is due to the fact that the fragment program is executed for each rendered pixel. This problem can be alleviated by future graphics hardware with more fragment pipelines. Increasing the scene polygonal count, on the other hand, has a gradual impact on performance unless extensive vertex programs are used. In fact, the performance in the case of programmable graphics hardware is dependant on several factors such as the length and complexity of vertex and fragment programs and the amount of data transferred between the CPU and GPU each cycle.

6 Real-Time Photo-Realistic Rendering for Surgical Simulations 351 Figure 5. The effect of different noise functions on the overall visual appearance of the rendering results. Shown above are four types of noise with decreasing frequency (clockwise from top-left) Figure 6. Performance assessment of real-time per-pixel shading with graphics hardware for different viewport resolutions and polygon counts.

7 35 M.. ElHelw et al. 4 Discussions and Conclusion In this paper a novel photo-realistic rendering method suitable for surgical simulation is described. It is based on using combined reflectance and refractance maps to model the effect of surface details and mucous layer on the overall visual appearance of the rendering results. The programmable graphics hardware is used to allow for per-pixel control and to carry out most of the required computations. In addition to the high fidelity rendering results achieved, the computational performance achieved makes it suited for interactive MIS simulation. With the use of general-purpose capabilities of the GPU, it is also possible to migrate simulation tasks such as collision detection and deformation computation to the GPU [13], allowing the entire simulation system to be seamlessly integrated. References 1. Shah, J. and Darzi,.: Simulation and Skills ssessment. International Workshop on Medical Imaging and ugmented Reality (MIR '1), Hong Kong (1) 5-9. Bro-Nielsen, M.: Simulation Techniques for Minimally Invasive Surgery. Journal of Minimally Invasive Therapy & llied Technologies (1997) Neyret, F., Heiss, R. and Senegas F.: Realistic Rendering of an Organ Surface in Real- Time for Laparoscopic Surgery Simulation. The Visual Computer. Vol. 18 No. 3 () Stylopoulos, N., e.t al: CELTS: Clinically-Based Computer Enhanced Laparoscopic Training System. Medicine Meets Virtual Reality (3) 5. Peercy, M.S., Olano, M., irey, J., and Ungar, J.: Interactive Multi-Pass Programmable Shading. Proceedings of SIGGRPH () Eveirtt, C.: Mathematics of Per-Pixel Lighting. /object/ mathematicsofperpixellighting.html 7. Neider J., Davis, T. and Woo, M.: OpenGL Programming Guide. nd edn. ddison Wesley (1997) 8. Phong, B., T.: Illumination for Computer Generated Pictures. Communications of the CM, Vol. 18 No. 6 (1975) Torrance, K.E. and Sparrow, E.M.: Theory for Off-Specular Refelction from Roughned Surfaces. Optical Society of merica, Vol. 57 No. 9 (1976) Ebert, D.S. (ed.): Texturing and Modelling: Procedural pproach. P Professional (1994) 11. Fernando, R. and Kilgard, M.J.: The Cg Tutorial. ddison Wesley (3) 1. Mark, W.R., Glanville, R.S., keley K. and Kilgard, M.J.: Cg: System for Programming Graphics Hardware in a C-Like Language. Proceedings of SIGGRPH (3) Fernando, R. (ed.): GPU Gems. ddison Wesley (4)