Virtual Endoscopy: Modeling the Navigation in 3D Brain Volumes

Transcription

1 ACBME-137 Virtual Endoscopy: Modeling the Navigation in 3D Brain Volumes Aly A. Farag and Charles B. Sites Computer Vision and Image Processing Laboratory University of Louisville, KY Stephen Hushek and Thomas Moriarty Department of Neurological Surgery University of Louisville, KY Abstract Minimally invasive neuroendoscopy involves matching the two-dimensional images seen by the endoscopes to the three-dimensional reality of a patient that is summarized pre-operatively in CT and/or MRI data sets. To remedy the inaccuracies and difficulties associated with classical endoscopy, a robust real time computer-assisted navigation capability is required for neuroendoscopy. In this article, we propose a computational geometry-based approach for virtual endoscopy. The approach is also being implemented on hardware. I. Introduction The ultimate goal of image-guided minimally invasive endoscopic technology is to enable the neurosurgeon to navigate through the brain, i.e., allocate and visualize where the surgical tool is in the brain at any time during the surgical procedure. Below, we describe two approaches, one for virtual endoscopy based on computational geometry and shape description, the other based on practical implementation of the above concepts. The overall idea of simulating an endoscope is as follows: Given a stack of MRI data (or MRA), a 3D model is generated. Using the scanner parameters, the 3D volume generated by the segmentation process can be calibrated with respect to the actual dimensions of the human head. Endoscopy tries to map the projections (2D images) that the surgeon sees from the endoscope, during the surgical procedure, into the 3D volume generated from the MRI or MRA stack (i.e., a 2D to 3D mapping). Virtual endoscopy will do the reverse operation; i.e., generate 3D mouse that can display portions of the 3D data in the computer. The projection of the 3D data in the field of view of the 3D mouse into a particular plane will correspond to the 2D image that the surgeon sees using a real endoscope. Now, the problem becomes the following: given the 3D volume seen by the 3D mouse, and the stack of MRI (or MRA) data that generated that volume, can we use the data in the stack to estimate the projections of the volume seen by the mouse? Or, conversely, can we use the projections of the volume seen by the 3D mouse, the location of the optical center (and focal point) of the virtual endoscope (the 3D mouse) to estimate 2D images (segments of the slices forming the MRI stack) that were involved in generating that 3D volume?

2 ACBME-138 If the above is possible, then real time endoscopic navigation will be simulated as follows: 1. Generate a calibrated 3D volume. 2. Generate the projections of a 3D segment seen by a 3D mouse that navigates through the 3D volume. 3. Identify the set of slices of the MRI data that were used to generate the segment seen by the 3D mouse. 4. Using the projection of the 3D segments, the location of the 3D mouse (i.e., its optical center), identify the portions of the MRI slices that closely correspond to the projections. 5. The software system should simulate real time navigation through a 3D volume as follows: show 3D segments, projections, and corresponding 2D images from the raw MRI (or MRA) data. Continual changing of the location of 3D mouse and displaying the portions of the 2D raw data that generated the volume in the field of view of the mouse, will provide the navigational component that corresponds to mapping the 2D image seen by an endoscope into the 3D volume of the brain. The mathematics behind the above concepts is being formulated using shape analysis and computational geometry. Below, we show a proof of concept based on simulations. II. Virtual Endoscopy Simulations We are using the '3D mouse' simulator for input of the endoscope's position (X,Y,Z) and orientation (Rho, Theta, and Phi) as well as extension (length). Several panels are displayed simultaneously. We are currently using an MRA stack of images 256x256x116. The main panel Show s the orthogonal projections at the tip of the endoscope. An optional 'reslice' image (the plane normal to the endoscope's vector) can be displayed. A second window panel shows the orthogonal sections individually with the current endoscope tip position located with cross hairs. The third panel shows the 'reslice' image plane that would be normal to the endoscope vector. The GUI panel also shows a 3D volume rendering of the MRA stack and the Endoscope position in reference to it. The last image is the Endoscope view. This is created by moving the volume rendering camera to the location of the endoscope tip, and modifying it's view angle to match that of the endoscope. Currently that is 90 degrees modeled after the endoscope we have in the lab. We currently cannot model the depth of field of the endoscope simulation; so we have a depth of field from 0 to infinity. (This is a computational speed issue). But for the most part, we should be able to see similarities we need. Fig 1.A was the first step. It shows the 3Dmouse panel and an arbitrary 'reslice' plane normal to the mouse vector. Fig 1.B simulates endoscope in operation. The simulated mouse controls are on the right. Then endoscope view is the third window bottom pane. Fig. 1.C shows the main window panel with a Y-axis plane and the reslice plane is shown. Fig. 1.D shows the orthogonal views with the cross hairs showing the position of then endoscope (3D mouse) tip.

3 ACBME-139 Fi.g 1.A: The 3D-mouse panel and an arbritary 'reslice' plane normal to the mouse vector.. Fig 1.B: An endoscope simulation in operation.

4 ACBME-140 Fig. 1.C: The main window panel with a Y-axis plane and the reslice plane is shown. Fig. 1.D: Orthogonal views with the cross hairs showing the position of then endoscope (3D mouse) tip. Fig. 1.E: The Reslice Projection window. Fig. 1.F: The Volume Render display with the mouse..

5 ACBME-141 The above results are only the first step towards the proof of concept. A number of following steps are in progress to study the software accuracy issue, speed, and validation. III. Rigid Endoscopy Endoscopes generally exist in two types: rigid and non-rigid. In this paper, we focus on modeling and calibration of rigid endoscopes. The images captured by the endoscope camera can be used to map the imaged region of the patient's brain to its 3D model. Making such mapping requires modeling of the relationship between the 2D images and the 3D world. Camera calibration is a process, which models this relationship. Basically, there are two aspects associated with camera calibration: calibration of the internal parameters of a camera (intrinsic parameters) and pose estimation of a camera system relative to a 3D world reference system (extrinsic parameters). As the endoscope moves, the extrinsic model parameters are changed. Therefore, these parameters need re-calibration. On the other hand, adjusting the zoom and focus of the endoscope camera modifies the intrinsic calibration parameters of the camera and the parameters of the camera positioning system. In addition, the use of variable focal lengths (especially small ones) introduces geometrical distortion for objects, which are not close to the optical axis. It is therefore mandatory to develop methods to automatically recalibrate the visual sensor. In the following we describe a model of the endoscope camera and then provide techniques for its calibration in case of a moving camera and in case of variable focus and zoom. A. Camera Model The result of camera calibration is an explicit transformation that maps a 3D world point M=(X,Y,Z,1) T into the 2D pixel m=(u,v,1) T. In the pinhole camera model, the relationship between M and m is given by s m = A[ R t], (1) α u c u0 A = 0 α v v where s is an arbitrary scale factor; (R t), called the extrinsic parameters, are respectively the rotation (in terms of three angles: R x, R y and R z ) and translation (t=(t x t y t z ) T ) components of the camera transformation; A is called the camera intrinsic matrix, and (u 0,v 0 ) are the coordinates of the principal point, α u and α v the scale factors in image u and v axes, and c the parameter describing the skewness of the two image axes. The camera mapping can be represented by a 3 x 4 projection matrix, P, that encompasses all these parameters. This camera model ignores lens distortion which is often accounted for in the camera model by adding some distortion parameters [6]. However, these parameters can be estimated in the captured images by a precalibration process [10]. Then all images can be undistorted before calibration proceeds. The decoupling between distortion parameters from the others will allow us to maintain the simple relation in (1) thus making the use of the model easier.

6 ACBME-142 Given a sufficient number, N, of reference world points, M i =(X i Y i Z i 1) T, as well as their corresponding pixel positions, m i =(u i v i 1) T, the camera calibration problem is to estimate the 11 camera parameters, in other words, the projection matrix P, that minimize N E = P M i m i= 1 i (2) Generally, the 2D image pixels m i are extracted from a captured image of a calibration pattern. B. Lens Distortion Calibration The endoscope camera typically has small focal length, which results in considerable lens distortion The assumed pinhole camera model described above, does not consider lens distortion which is often accounted for in the camera model by adding some distortion coefficients [6]. However, we believe that it is better to estimate the distortion coefficients in the captured images then correct for the distortion effects by an independent process performed before calibration proceeds. By separating the estimation of lens distortion coefficients from the calibration process, the effect of the correlation between lens distortion coefficients and other camera model parameters is minimized [8]. The method that we propose to use to correct for lens distortion is based on the idea that lens distortion causes straight lines in the scene to appear as curves in the image [11]. The algorithm tries to find the distortion parameters that map the images curves to straight lines. Once the distortion parameters are calibrated, the captured images can be undistorted before processing for camera calibration and other vision tasks. C. Camera Model Calibration The existing techniques to solve this problem can be broadly classified into two main categories: Linear Techniques: In this category, camera parameters are computed directly through a non-iterative algorithm based on a closed-form solution (e.g., [15]). Having advantages of speed and simplicity, these techniques provide less accurate results. Nonlinear Minimization: With this type of scheme, an iterative minimization algorithm is employed to solve for the camera parameters (e.g., [6]). This approach may achieve high accuracy, and allow easy adaptation of any complex model of imaging. However, since the algorithm is iterative, the procedure may end up with a farfrom-optimal solution unless a good enough initial guess is available. To find such initial solutions, the linear techniques are often employed. Other methods incorporate the direct closed-form solution for most of the calibration parameters and some iterative solution for the others. Among these methods, Tsai's method [7] may be the most popular one. We have recently proposed a new solution of the camera calibration problem based on using a multilayer feedforward network (MLFN) [9], which can be classified into the second category of nonlinear minimization techniques. This approach relaxes the requirement to start the

7 ACBME-143 non-linear calibration procedure with a good initial guess, which is required by other nonlinear camera calibration techniques. This property is very useful when no such initial solution is available, or when this starting point is rather far away from the optimal solution. Our neural network-based approach (called neurocalibration) is able to calibrate the camera implicitly by providing the camera projection matrix and explicitly by specifying the camera intrinsic and extrinsic parameters[10]. D. Calibration of a Moving Endoscope If the camera is stationary, we do not have to re-calibrate again. Yet in all endoscope applications, the camera will be moving; This changes the camera extrinsic parameters while the intrinsic parameters can be safely assumed constant. This implies the recalculation of the perspective projection matrix. Being mounted rigidly on the digitizer arm, the camera location in the 3D space can be measured. The arm can provide the transformation (4 x 4 matrix), denoted by H, that relates the new position and orientation of the camera with respect to the position and orientation at which the camera was initially calibrated. extrinsic parameters. Calibrating cameras with changing zoom and focus raises several challenges [12]. The calibration problem becomes characterizing how the parameters of the fixed camera model vary with lens zoom and focus settings. The calibration approach generally, involves first calibrating a conventional static camera model at a number of lens settings spanning the lens' control space. To model how the terms of the static camera model vary with lens setting, partial lookup tables and interpolations or fitting multi-variable polynomials [12] can be used. One way to look to the task of zoomlens camera calibration is as a combination of fixed parameter camera calibration and function interpolation over a large collection of data. The features of our neurocalibration approach enable us to present an all-neural framework for zoom-lens calibration [10]. This framework consists of a number of MLFNs learning concurrently, independently and cooperatively, to capture the variations of model parameters across optical lens settings. Our approach has the following key features, as opposed to other techniques (e.g., [14], [13], [12]): This transformation is used to update the camera extrinsic parameters by postmultiplying the extrinsic matrix [R t] by H. As such, the new camera perspective projection matrix is found. E. Calibration of Zoom-adjustable Endoscope Camera Adjusting the endoscope camera's zoom and/or focus help bring some imaged parts in more details and in more focus. However, it complicates the camera models since it changes the camera intrinsic and It is general; it can consider, any number/combination of lens control parameters, e.g., zoom, focus and/or aperture. It can capture complex variations in the model parameters across control space. All of the parameters are fitted to the calibration data at the same time, while in other approaches, one parameter is fitted at a time and the final level of error generally depends on the order in which the models are fit to the data.

8 ACBME-144 IV Experimental Results The calibration approach has been tested with real data. An image of a calibration pattern ( see Fig. 2) whose points are know in 3D space is used. The 3D points and their corresponding 2D points extracted from the image are used for calibration. To test the quality of the calibration, an image of a model of a patient's head is acquired by the endoscope camera with the endoscope rigidly mounted on a digitizer arm. Then another image is acquired after the camera has been moved. The motion matrix provided by the arm is used to compute the new model of the camera as shown in Fig. 3. Since the camera is calibrated in the two position, the epipolar geometry between the two acquired images can be recovered. Corresponding points between the two images should lie on corresponding "epipolar" lines. How far a point is from its corresponding epipolar line measures the calibration accuracy. The root mean square error of the distances of the marked points from the epipolar lines is 0.2, which shows the high calibration accuracy of the endoscope camera. Fig. 3: The recovered epipolar geometry shows the accuracy of the calibrated camera. Acknowledgments This project has been partially funded by the Whitaker Foundation Research Grant No and the Norton Healthcare Organization Grants Fig. 2: Calibration pattern

9 ACBME-145 References 1. Eldeib, A. Farag, Paul Larson, and T. Moriarty, A Stereovision Technique for Image Guided Neurosurgery, Proceedings of the International Conference on Computer-Assisted Radiology (CAR-2000), Los Angeles CA, Jan 31, A. Eldeib, S. Yamany, A. Farag, and T. Moriarty, Volume Registration by Surface Point Signature and Mutual Information Maximization with Applications in Intra-Operative MRI Surgeries, Proc. IEEE International Conference on Image Processing (ICIP'2000), Vancouver, BC, Canada, September C. Sites, A. Farag, S. Hushek, T. Moriarty, A Fast Automatic Method for 3D Volume Segmentation of the Human Cerebrovascular Journal of Computer-Assisted Surgery (In preparation). 4. M. N. Ahmed and A. A. Farag, Two-stage Neural Network for Volume Segmentation of Medical Images, Pattern Recognition Letters, Vol. 18, No 11-13, pp , November M. N. Ahmed, S. M. Yamany, N. A. Mohamed, and A. A. Farag, A Modified Fuzzy C-Means Algorithm for MRI Bias-Field Estimation and Adaptive Segmentation, International Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI 99), Cambridge, England, pp , September J. Weng, Paul Cohen and M. Herniou, Camera calibration with distortion models and accuracy evaluation, IEEE Trans. Patt. Anal. Machine Intell, Vol. 14, No. 10, Oct Roger Tsai, A versatile camera calibration technique for highaccuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses," IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, Aug S. Shih, Y. Hung and W. Lin, Accuracy analysis on the estimation of camera parameters for active vision systems," Proc. Int. Conf. Pattern Recognition (Vienna, Austria)}, Vol. 1, Aug Moumen Ahmed, Elsayed Hemayed and Aly Farag, ''A Neural Network That Can Tell Camera Calibration Parameters", Proc. IEEE International Conference on Computer Vision, Greece, June Moumen Ahmed and Aly Farag, ''A Neural Optimization Framework for Zoom-lens Camera", Proc. IEEE International Conference on Computer Vision and Pattern Recognition, SC, June Moumen Ahmed and Aly Farag, ''Nonmetric calibration of Camera Lens Distortion", Proc. IEEE International Conference on Image Processing, Greece, Oct (To appear).

10 ACBME R. G. Wilson, Modeling and calibration of automated zoom lenses", PhD dissertation, Dept. Elect. Comp. Eng., Carnegie Mellon Univ., A. Wiley and K. Wong, Geometric calibration of zoom lenses for computer vision metrology," Photogrammetric Eng. Remote Sensing, Vol. 61, No. 1, Jan K. Tarabanis, R. Tsai and D. Goodman, Calibration of a computer controlled robotic vision sensor with a zoom lens," CVGIP: Image Understanding, Vol. 59, No. 2, Jan Olivier Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, 1993.