User-friendly Environment for Virtual Endoscopy

Transcription

1 User-friendly Environment for Virtual Endoscopy Jeong-Jin Lee 1, Yun-Mo Koo 2 and Yeong Gil Shin 1 1 School of Computer Science and Engineering, Seoul National University San 56-1, Shilim-Dong, Kwanak-Gu, Seoul , Korea 2 3Dmed co.,ltd Research Park, SNU, San 4-8, Bongcheon-Dong, Kwanak-Gu, Seoul , Korea ABSTRACT Virtual endoscopy has recently arisen as a powerful diagnostic tool for examining the interior structures of bronchus, colon, and etc. In contrast to optical endoscopy that is invasive, virtual endoscopy is non-invasive and thus it is comfortable to patients. In addition, while the optical endoscopy enables observers to examine the hollow structures in the direction that the camera moves, the virtual endoscopy can visualize the structures in any direction. We present an interactive virtual endoscopy system that is user-friendly and useful for the diagnosis. The navigation path is generated in the real time without any preprocessing such as segmentation procedures. The endoscopy system provides a 3D volume rendered image as a navigation map, and has the MPR image along the path and a 2D axial image as a reference image. These reference images are good guides to diagnose tissues and their neighbors. The endoscopy images are generated using the volume rendering and thus it has a good quality comparing with surface rendering. The experimental results show the comfortable user interface of our endoscopy system. Keywords: Virtual Endoscopy, User Interface, Camera Control, Path Planning 1. INTRODUCTION Optical endoscopy is used for an inspection of lower gastrointestinal tract including a colon. This procedure is lengthy and painful even by the hands of an experienced operator [1]. Furthermore, the navigation and the camera control are not easy[2]. Alternatively, virtual endoscopy is a procedure to navigate without inserting an optical probe [5]. Comparing with the optical endoscopy, it has many attractive features [3] [4]. First, its navigation is noninvasive. It also allows easy controls of the virtual camera and various navigations without pains. While the optical endoscopy enables observers to navigate in the direction that the camera moves, users can navigate in any direction using the virtual endoscopy. Many virtual endoscopy systems have the difficulty in generating the navigation path for the observation automatically.

2 The automatic path generation requires the segmentation in many cases [8] [11]. As the become more complex, it s difficult to get the segmentations fully-automatically. For example, the leakage can be induced from the entrance of the lung in the virtual bronchoscopy [7]. In these cases, users must do segmentations manually. The path generation also requires large computational time to get 3D skeletons, minimum distance fields [6] [9] [11]. In this work, we eliminate the segmentation process of the path generation. The path is generated and modified interactively in the observation time. Our system also provides the 3D volume rendered image as the navigation map that shows the external view of the lesion to be observed. The region of interest is interactively rendered as the given opacity threshold value. Tissues and structural obstructions that block the view of the endoscopic region of interest can be removed. The 3D volume rendered images help users observe neighbor structures of endoscopic regions. Users can generate the path and locate the virtual camera on any position watching the 3D overview image. Many virtual endoscopy systems use three reference 2D views including the sagittal, coronal, transverse view [3] [4] [8]. These views are not easy for users to navigate inward. Some previous approaches give the outline view by the surface rendering [8]. The position of the current camera is indicated at this view. The surface rendering has less image quality than the volume rendering. And the extraction of the surface cannot be maid interactively by changing opacities. Our endoscopy system displays reformatted image along the path. The path MPR image can diminish diagnostic time. The paper is organized as follows. In section 2, we present an overview of our endoscopy system briefly. In section 3, we explain the method of the path generation and modification. In section 4, we show how the MPR image along the path is produced. In section 5, we explain the interactive volume rendering method briefly. Finally, we present experimental results in section 6, and conclusions in section OVERVIEW OF THE ENDOSCOPY SYSTEM

3 3D Volume Rendering Image Set the camera Endoscopy Image Set the path Set the camera Set the camera Axial Image Set the path path MPR Image Figure 1. Virtual Endoscopy System Organization The virtual endoscopy system is composed of four views (Fig. 1). The 3D volume-rendered image, the navigation path, and position of the camera are displayed on the 3D volume rendering view. The transverse image at the given slice is showed on the axial view. The navigation image is displayed on the endoscopy view. The virtual endoscopy system uses the volume-rendered image as the reference image without preprocessing 3D volume data. Using this reference image, users can make, and modify the path very comfortably. If the curvilinear path is made, the automatic navigation along the path can be done in the direction of the path, or opposite. As described early, our navigation system is composed of the automated part and manually controlled part. This composed system is known as very comfortable for users [10]. The MPR image is also generated along the given path. This image helps users look neighborhoods of the. When users specify the path or camera on the 3D volume rendering view, the endoscopy image will be generated. The 2D axial images are generated by the position of the camera. The users can change the position and orientation of the camera on the 3D volume-rendered view or 2D axial view. They can move the position of the camera along the path, or arbitrarily. The path MPR image will be made if the users make the path on the 3D volume-rendered view or 2D view. The path MPR image can be rotated arbitrarily. 3. PATH GENERATION By clicking the mouse only a few times where users want to navigate through, they get the path in the real time. Users can give the points on 2D axial view or on 3D volume-rendered view. If users aren t satisfied with the obtained path, they can edit the path interactively by moving the control point of the path.

4 3. 1 Generating the path Generating the path is mainly composed of two parts, getting the 3D control points from the user input and making the smooth path with the 3D control points. First, we show how we get the 3D coordinate of the clicked point given in the image. Because we know the z- coordinate on the 2D axial view, calculating the 3D coordinate of the clicked point on the 2D axial view is trivial. But, it is not easy to get the 3D coordinate when users give the control point on the 3D volume rendered view. We assume that the endoscopic region is revealed using sculptings. Then, we find the center point of the organism, which we are interested in. According to 3D viewing pipeline, the coordinate, p is related to the image coordinate, p image by the following equation. p = M p (1) image viewing Using the homogeneous coordinate, p image = ( xi, yi, zi,1), p = ( xo, yo, zo,1), we know x i, yi by the input of the mouse. If we know the depth of the image, z i, we can determine the position of the, p using the following equation. p = M p (2) 1 viewing image As we know the volume size, we can find the minimum and maximum possible depth of the image by calculating the intersection of the boundary volume and the viewing ray. We get the depth of the image by increasing it from the minimum to the maximum. There are two kinds of the interesting region for the navigation, filled cylinder shape or hollow cylinder shape (Fig. 2). Our algorithm can find the center point of the either case correctly. Increasing the depth and calculating the coordinate by the given depth, we look up the density of the voxel matching the coordinate. In the first case, we describe the procedure in case of the hollow cylinder shape. Comparing this density value with the threshold, we can determine whether that voxel belongs to the boundary or not. If this density value is between the lower threshold and the upper threshold, that voxel belongs to the boundary. If we find two points on the boundary through the ray, the average of two points can be the center point of the hollow cylinder shape. Then, this point can be where users want to navigate through. In the second case, we describe the procedure in case of the filled cylinder shape. In this case, the method is different a little from the above case. Traversing the ray by increasing the depth, we indicate the point where the ray meets the for the first time. Due to its shape, the ray that is proceeded meets the. Then, we count the number of continuing voxels by increasing the depth. Doing that, we indicate the second point when the ray meets the space for the first time. As the same in the first case, the average of two indicated points can be the center point of the filled cylinder shape.

5 Then, this point can be where the user wants to navigate through. (a) hollow cylinder shape (b) filled cylinder shape Figure 2. Types of Object Shapes In the third case, there may be no intersection between the ray and the. In this case, we calculate the intersection of the ray and the volume. Then, the average of two intersection points can be the user -interested point. If users give two or more points by the mouse input, the 3D curvilinear path is generated by interpolating these given user-interesting points. So, user-input points become the control points of the 3D curve. For the automatic navigation, we extract the set of points from the curve. Because the position of the camera must be changed steadily for the automatic navigation, the distance of neighbor points can be assigned 1. Those points become the camera position during the navigation. And, the camera direction is pre-calculated at the navigation point for the smooth navigation. If not doing so, we feel uncomfortable during the navigation. The direction of the camera is determined interpolating the tangent vector of neighbor control points Modifying the path If users aren t satisfied with the generated path, they can interactively change the path seeing what the changed path is. This procedure can be done on the 2D view or the 3D view. On each view, the way in which the path is changed is different a little. On the 3D volume-rendered view, users can see the overall path. If users go into the edit mode, the control point of the path is indicated. So, users only change the control point, not all the points. If users change the position of the control point, we get the image coordinate of the moved control point. So, using the same routine of the path generation, section 3.1, the depth of the new control point can be guessed. Then, both of the image coordinate and the depth make the coordinate of the new control point using the eq. (2). As in the 2D case, we can re-generate the path by using the modified control point. Because the user can see what they modify on the real time, the edit procedure of the path is very easy and effective. On the 2D axial view, users can see only the part of the path on the given slice that has the same z coordinate. If they change the position of the point on the path, the changed position is always located on the same slice. If they move the point, the control point is also changed. Then, we re-generate the path by using the changed control point. But, the movement of the point is restricted on the given slice and users can see only the part of the path. The edit procedure is more difficult than

6 on the 3D rendered view. 4. MPR IMAGES ALONG THE PATH 2D image 3D path Figure 3. Generating path MPR image We generate the MPR image along the path by projecting the curvilinear path on the 2D plane (Fig. 3). The procedure of making the MPR image is described as follows. The reconstruction of multiplanar reformatting image from the 3D image is mainly composed of two steps. First, sample points must be extracted. They will be the basis of the image produce. Second, the image is reconstructed with the sample points. In the first step, the curve is expressed by parameters, and equidistance points can be obtained by assigning the parameter appropriate values. The distance of these points is determined by the trade-offs between complexities and smoothes of the curve. Then, we calculate the unit direction vector between two control points and extract sample points that have the equidistance, 1 by using this vector. In the second step, the viewing direction vector of the 3D image is (0, 0, -1) in the image coordinate. We get the viewing direction vector in the coordinate by multiplying the inverse viewing matrix. This vector and 3D sample points make the reconstruction plane. Pixel values are accumulated at each sample point going until the boundary of the volume in the viewing direction. This calculation for one point makes one scanline of the reconstruction image. So, all sample points make the reconstruction image by using these procedures. 5. INTERACTIVE VOLUME RENDERING We make the endoscopy image by using volume rendering. A ray casting and min-max structure are used. To skip empty space efficiently, we use the hierarchy structure of two levels of min-max structures [12]. The volume rendering is made progressively. We provide four levels of the image quality. Therefore, the user increases the image quality more, and then the navigation speed is diminished by that amount. Experimental results prove our algorithm is tolerable for the high quality image. And our navigation system is very plausible for the low image quality.

7 6. EXPERIMENTAL RESULTS Figure 4. Virtual Endoscopy System Views (a) 3D volume-rendered overview (upper left) (b) Navigation view (upper right) (c) 2D axial view (lower left) (d) MPR image view along the path (lower right) First, we prepare 3D volume rendered image for the endoscopy. Users load files of a region of interest and open them in the 3D volume rendering mode. They apply the fine tuning to adjust threshold and opacity or select given threshold preset. And they perform sculpts to remove obstructing tissues and structures blocking endoscopic regions of interest. Then, the image is prepared for the navigation like Fig. 4 (a). There are four view screens of the endoscopy system, Fig. 4. The 3D volume-rendered overview, Fig. 4 (a), displays the 3D volume-rendered image and the navigation path and position of the camera. Our system uses the 3D volume image as the navigation map. Users can create navigation paths on the 3D image and navigate automatically along the pre-defined pathways. The transverse screen, Fig. 4 (c) displays a slice by slice transverse images at the point of view of endoscopy. The user can scroll through transverse images with the mouse wheel or arrow keys. Location of the slice is indicated on the 3D volume rendering screen as a horizontal intersecting line. The MPR path screen, Fig. 4 (d) displays a 2D reformatted image of the specified path. The user can view virtual endoscopy on the endoscopy screen, Fig. 4 (b). Due to its one-dimensional nature of the path, users aren t always satisfied with the automatic navigation. In our system,

8 users can control the camera manually and see exactly what they want to see. Users can modify the position and orientation of the camera. This procedure can be done on 2D or 3D. If the user change the position of the camera by moving or clicking the mouse on the 2D axial view, we know the image coordinate of the position. On the 2D view, the z-coordinate of the scene is already known. So, the coordinate of the position is easily calculated. The direction of the camera can also be changed. The rotation of the camera has two modes, red mode, and blue mode. In 'red mode', the camera rotates on the view plane. So, the axis of the rotation is z-axis. In 'blue mode', the camera rotates on the perpendicular plane to the view plane. The axis is on the view plane and perpendicular to the current camera direction. This rotation about the axis is implemented using the quaternion. Using the same procedure of the path generation, section 3.1, the moved position of the camera on the 3D volumerendered view can be calculated. So, users can move the camera where they want to see. Like the 2D case, there are two rotation modes, red mode, and blue mode. In 'red mode', the camera rotates on the view plane. This implies that the changed direction of the camera has the same image depth with the previous direction. Using this fact, we calculate the coordinate of the camera direction. In 'blue mode', the camera rotates on the perpendicular plane to the view plane. The user can rotate the orientation of the camera about the axis that is perpendicular to the current camera direction and on the view plane. This is also implemented using the quaternion. 7. CONCLUSIONS The virtual endoscopy system is non-invasive to patients, and physicians can diagnose the inside of colons without pains to patients. Using the real endoscopy physicians navigate only in the direction of moving where the camera goes. However, the virtual endoscopy enables physicians to navigate in the opposite direction of the camera movement. This system is very useful for the diagnosis. Previous systems have difficulty in user interfaces, and this is the most important reason of inactive use of the virtual system. In this paper, we described useful interfaces of the path generation and camera control. The 3D volume-rendered view was given as the navigation map. Users could generate the path where they wanted to navigate on the 3D volume-rendered view in the real time. Also, if they didn t satisfy the generated path, they could modify the path interactively. Using interpolated tangent vectors, the smooth and user-comfortable navigation was also guaranteed. For more detail navigations, users could control the position and orientation of the camera easily using various methods. So, this allowed the virtual camera to fly through the user specified area. In addition to interior endoscopy views of areas of interest our system generated external views without segmentations. Besides, reference MPR images along the path added more effective method to diagnose. This interface can be applied to virtual bronchoscopy, colonoscopy, and angioscopy effectively. 8. ACKNOWLEDGMENTS The method described on this paper obtained the patent, for the invention in Korea.

9 REFERENCES [1] S. Dogramadzi, C. R. Allen, and G. D. Bell, Computer Controlled Colonoscopy, IEEE Instrumentation and measurement technology conference, pp. 210 쳌213, May, [2] S. J. Phee, W. S. Ng, I. M. Chen, F. Seow-Choen, and B. L. Davies, Automation of Colonoscopy Part II: Visual-Control Aspects, IEEE Engineering in Medicine and Biology Magazine, Vol. 17, Issue 3, pp. 81 쳌88, May 쳌June, [3] Tong-Yee Lee, Ping-Hsien Lin, Chao-Hung Lin, Yung-Nien Sun, and Xi-Zhang Lin, Interactive 3-D Virtual Colonoscopy System, IEEE Transactions on information technology in biomedicine, Vol. 3, No. 2, pp. 139 쳌150, June, [4] L. Hong, A. Kaufman, Y. Wei, A. Viswambharan, M. Wax, and Z. Liang, 3D Virtual Colonoscopy, IEEE Symposium on Frontier in Biomedical Visualization, pp , October, [5] A.K. Hara, C.D. Johnson, J.E. Reed, R.L. Ehman, and D.M. Ilstrup, Colorectal Polyp Detection with CT Colography: Two-versus Three-dimensional Techniques, Radiology, Vol. 200, pp. 49 쳌54, [6] Rui C. H. Chiou, Arie E. Kaufman, Zhengrong Liang, Lichan Hong, and Miranda Achniotou, Interactive Path Planning for Virtual Endoscopy, IEEE Nuclear Science Symposium 1998, Vol. 3, pp 쳌2072, [7] Mie Sato, Sarang Lakare, Ming Wan, Arie Kaufman, Zhengrong Liang, and Mark Wax, An Automatic Colon Segmentation for 3D Virtual Colonoscopy, IEICE Transactions on information and system, Vol. E84-D, No. 1, pp. 201 쳌 208, January, [8] Lichan Hong, Shigeru Muraki, Arie Kaufman, Dirk Bartz, and Taosong He, Virtual Voyage: Interactive Navigation in the Human Colon, Proceedings of ACM SIGGRAPH 1997, pp , August, [9] Taosong He, and Lichan Hong, Reliable Navigation for Virtual Endoscopy, IEEE Nuclear Science Symposium 1999, Vol. 3, pp 쳌1343, [10] Tinsley A. Galyean, Guided Navigation of Virtual Environments, ACM Syposium on interactive 3D graphics 1995, pp. 103 쳌104, [11] Yong Zhou, Arie Kaufman, Arthur W. Toga, Three-dimensional Skeleton and Centerline Generation Based on an Approximate Minimum Distance Field, The Visual Computer, Vo. 14, pp. 303 쳌314, [12] Jae Jeong Choi and Yeong Gil Shin, Efficient Multidimensional Volume Rendering, Medical Imaging 99, San Diego, USA, February, 1999.