A New Method for CT to Fluoroscope Registration Based on Unscented Kalman Filter

A New Method for CT to Fluoroscope Registration Based on Unscented Kalman Filter Ren Hui Gong, A. James Stewart, and Purang Abolmaesumi School of Computing, Queen s University, Kingston, ON K7L 3N6, Canada purang@cs.queensu.ca Abstract. We propose a new method for CT to fluoroscope registration which is very robust and has a wide capture range. The method relies on the Unscented Kalman Filter to search for an optimal registration solution and on modern commodity graphics cards for fast generation of digitally reconstructed radiographs. We extensively test our method using three different anatomical data sets and compare it with an implementation of the commonly used simplex-based method. The experimental results firmly support that, under the same testing conditions, our proposed technique outperforms the simplex-based method in capture range while providing comparable accuracy and computation time. 1 Introduction Registration of CT to fluoroscope images is a fundamental task in Computer- Assisted Orthopaedic Surgery (CAOS) and Radiotherapy (CART). In the case of CAOS, registration of preoperative CT to a set of intraoperative fluoroscope images can be used to map the virtual patient (i.e. the preoperative CT) displayed on a screen to the physical patient in the operating room (OR). The registration precisely links the points in CT with the corresponding points in OR so that the CT can be used to guide the intervention. In the case of CART, registration of CT to portal images allows precise configuration of treatment beams so that the radiations are focused on tumors/lesions thus the damage to healthy tissues remains minimal. The CT to fluoroscope registration problem can be briefly described as finding a geometric transform that brings CT, denoted moving data, to the patient s coordinate frame, in which fluoroscope images, denoted fixed data, are captured on the fly during the surgery or treatment. It involves determining the type of the transform as well as an initial guess of its parameters, defining a similarity measure between CT and fluoroscope images, and selecting an optimization method that updates the parameters iteratively until an optimal similarity value is achieved. A clinically usable CAOS or CART system requires accurate, fast and robust registration between the fixed and the moving data. A number of methods have been proposed to address the CT to fluoroscope registration problem in the past decade. Based on what type of information is used in registration, these methods can be roughly classified into three categories [1]: R. Larsen, M. Nielsen, and J. Sporring (Eds.): MICCAI 26, LNCS 419, pp. 891 898, 26. c Springer-Verlag Berlin Heidelberg 26

892 R.H. Gong, A.J. Stewart, and P. Abolmaesumi feature-based, intensity-based and hybrid methods. Feature-based methods extract geometric features, such as landmarks, curves and surfaces, from both CT and fluoroscope images and use them in the subsequent registration. This class of methods is often more efficient because reduced amount of data are involved. However, the errors in feature extraction propagate to errors in registration. Intensitybased methods directly use all or partial intensity values in the original images and no feature extraction is required. This group of methods tends to be more accurate and robust, but at the same time, their computation costs are usually more expensive. The hybrid methods have both advantages and aim to provide accurate and fast registrations. CT to fluoroscope registration is a 2D-3D registration problem, in which one key challenge is to find an appropriate way to compare 3D CT with 2D fluoroscope. One solution is to generate intermediate simulated fluoroscope images, called digitally reconstructed radiographs (DRRs), from CT and compare them with the corresponding real fluoroscope images. Alternatively, one can reconstruct a 3D volume from the fluoroscope images and compare it with the 3D CT [1]. Majority of current methods fall into the first category, and have focused on fast DRR generation methods [2, 3, 4, ] or robust similarity measures [6, 7, 8, 4] between DRR and fluoroscope, and have relied on simplex or gradient descent [8] to search for an optimal registration. In this paper, we propose to use the Unscented Kalman Filter (UKF) as the optimization method. UKF has some features that could potentially benefit the CT to fluoroscope registration problem, e.g. no derivatives calculation is required, dealing with multiple observations simultaneously, estimating the variance along with the state, etc. To validate [9] our approach, we extensively test our method on various phantom data sets and compare the results with those of a simplex-based approach. The remaining of this paper is organized as follows: We explain our method in details in Section 2; Section 3 describes the testing scenarios and presents the experimental results; and Section 4 concludes the paper. 2 Methods In this section we discuss the major constituents of our method, i.e. the spatial transform, the similarity measure, the UKF-based optimization algorithm, and a hardware-based DRR generation technique. 2.1 Transform For CT to fluoroscope registration, a rigid transform is a natural choice in order to bring CT points into the patient s coordinate frame. The transform has six parameters: three Euler angles in degrees for rotation and three scalars in millimeters for translation. This representation was chosen because 1) it has fewer parameters compared with the cases that rotation is represented using quarternion or angle-axis, and 2) the rotation and translation components have same or similar dynamic ranges. The number of parameters and the scales

A New Method for CT to Fluoroscope Registration 893 between parameters are important factors that may have significant impact on the performance and robustness of a registration algorithm. An initial guess of the transform parameters is required to start the registration. This is done by manually selecting a few landmarks from CT, e.g. 3-4 points on the bone surface, and identifying their correspondences in the fluoroscope images, then the initial value is computed by solving an absolute orientation problem using singular value decomposition [1]. The chosen landmarks should be visible in both CT and fluoroscope images. If they are carefully selected and identified, the computed initial transform can yield an initial mean target registration error (mtre) within 3cm, which is often enough to start our registration method. 2.2 Similarity Measure Similarity measure is a metric that judges how similar the two compared data sets are. It is a function of the transform parameters and is computed from either the intensities or the extracted features of the two data sets being registered. The function can be linear or nonliner, and can have a single value or multiple values. For some similarity measures, the derivatives w.r.t. transform parameters can be computed at low cost, while for some other similarity measures, direct calculation of derivatives is not possible. A variety of similarity measures [6, 7, 8, 4] have been suggested for comparing DRR with fluoroscope. The commonly used ones are normalized correlation (NC), variance-weighted correlation (VWC), mutual information (MI), pattern intensity (PI), gradient difference (GD), and gradient correlation (GC). These are highly nonlinear functions and, except for MI [8], explicit calculation of derivatives is not possible. Each function is suitable for a different situation and has different computation cost. Thus our method was designed in a way that all current measures can be used. Due to the loss of depth information during fluoroscope acquisition, one single fluoroscope is often not enough to reach an accurate registration, thus multiple fluoroscope images from different viewpoints are used in our approach, which means our similarity measure is multiple valued and each value corresponds to a DRR, fluoroscope pair. Most current methods for CT to fluoroscope registration combine the values during the optimization process, in which information is lost. 2.3 Optimization As most current similarity measures for CT to fluoroscope registration are nonlinear, no direct derivatives and multi-valued, we adopted UKF [11] as the optimization method in our approach. UKF is a sequential least squares optimization technique employed for solving nonlinear systems. It estimates both the state and the state variance, and deals with multiple observations simultaneously during the estimation process. It requires no derivatives, Jacobian or Hessian calculations. Instead, UKF uses the Unscented Transform (UT) [11] to carefully select a minimal set of sample points to learn about the behaviors of a nonlinear system.

894 R.H. Gong, A.J. Stewart, and P. Abolmaesumi The sample points completely capture the true mean and variance of the state and, when propagating through the true nonlinear system, capture the posterior mean and variance accurately up to at least the second order Taylor series approximation. We use UKF to estimate the transform parameters in a chosen similarity measure. The system models (state and observation) have the following forms: x i = x i 1 + N(,s 2 x ) (1) y i = SM(x i )+N(,s 2 y) where x is the transform parameters to be estimated, SM is the nonlinear similarity measure between the DRRs and the corresponding fluoroscope images, σ x and σ y are the variances of the process and measurement noises intrinsic to the system. As multiple fluoroscope images are used in our method, y is a multidimensional measurement vector. The UKF algorithm recursively estimates the state and its variance. Figure 1 outlines the algorithm. Input: Initial state and its variance A sequence of observations (constant in our problem) UKF parameters Initialize UKF with the provided UKF parameters Until a satisfying SM value is achieved, do Generate sample points from the current state and variance using UT Propagate the points through the state and observation models sequentially Compute the gain using the results from the previous step Update the state and variance using the computed gain and current observations Fig. 1. The Unscented Kalman Filter (UKF) algorithm The UKF parameters have great impact on the convergence speed, and were determined empirically by try and fail. The observations are the target value of the chosen similarity measure, which are constant and known in our problem, e.g. 1. for normalized correlation. 2.4 DRR Generation We use an improved hardware-based technique, i.e. the Adaptive Slice Geometry Volume Rendering Algorithm [2], to speed up the DRR generation process. The algorithm improves the common view-aligned 3D texture-mapping based method by adaptively slicing the volume based on image content. Firstly, the CT is partitioned into a set of axis-aligned bounding boxes (AABBs) based on a user-defined transfer function that removes the empty voxels. Then the AABBs are sliced perpendicular to the viewing direction (i.e. the focal axis of the C-arm) in order back to front. Finally, the powerful OpenGL features of consumer-grade graphics cards, including 3D texture mapping, multi-texturing, and fragment programs, are employed to render and blend the slices into a final

A New Method for CT to Fluoroscope Registration 89 DRR. We tested our implementation on an ATI Radeon X8 card with 26MB video memory by rendering CT volumes of size 12x12x26 into DRRs of size 473x473, and have achieved the speed of 2- frames per second, depending on the image content in CT. 3 Experiments and Results We evaluate our method using images of three different phantoms: a small scaphoid bone, a large pelvis, and a long and thin femur, all with embedded fiducial markers for gold-standard validations. A general simplex-based method was implemented along with the UKF approach for the purpose of comparison. For all the experiments, normalized correlation was used as the similarity measure, and the same set of observation and measurement noise assumptions were made for all UKF experiments. We first ran experiments using synthetic fluoroscope images, i.e. DRRs, in which case the gold-standard is known. For each bone, a set of DRRs from three orthogonal views were generated with the CT positioned at the origin of patient s coordinate frame. To find whether viewing angle of fluoroscope affect the method or not, a second set of DRRs was also generated for the scaphoid bone by rotating the CT about axis (1,1,1) for 4 degrees. Next we did experiments using real fluoroscope images and used the embedded fiducials for validation. Four metal markers were attached to the femur surface and visible on both CT and fluoroscope images. We reconstructed 3D markers from the fluoroscope images by back-projection, which were then registered to the CT markers using a fiducial registration [1]. The registration result was used as the gold-standard. For each experiment, the corresponding method ran 1 times with the initial transforms generated by adding random rotations (±12 ) and translations (±12mm) to the gold-standard. We then recorded the initial and s. The mtre was computed as the average difference between the positions of a point set mapped by the evaluated transform and the gold-standard transform. 2 randomly selected points from the region bounding the bone were used when calculating the mtres. Figure 2 shows the initial and s for each experiment. Obviously, the UKF-based method consistently worked well for the full range (-2mm) of initial errors, while the simplex-based method worked well only for the range (-mm). A registration is said failed if the is larger than 2mm for synthetic fluoroscope experiments, or 3mm for real fluoroscope experiments. We define accuracy as the mean of the s of all successful registrations, and capture range as the that at least 9% of registrations succeed if they are within the range. Figure 3 shows the distributions of the s for selected experiments. It is obvious that the two methods achieved similar accuracies. Table 1 shows the capture range for each experiment. Again the UKF-based method obtained larger capture ranges for all experiments. Because the time spent in DRR generation is dominant in both methods, we use the average number of DRR generations as the measure to evaluate the

896 R.H. Gong, A.J. Stewart, and P. Abolmaesumi Scaphoid CT to DRRs pose 1 UKF-based Method 2 1 1 2 Simplex-based Method 2 1 1 2 Scaphoid CT to DRRs pose 2 2 1 1 2 2 1 1 2 2 2 Pelvis CT to DRRs 1 1 2 1 1 2 2 2 Femur CT to DRRs 1 1 2 1 1 2 2 2 Femur CT to fluoroscopes 1 1 2 1 1 2 Fig. 2. Initial and s for each experiment. Left column: results of UKFbased method. Right column: results of simplex-based method. The first four rows show results for the CT to DRRs registration experiments, and the last row shows the results for the CT to fluoroscope registration experiments. All units are millimeters. Final mtres above 2mm are removed from the graphs (approximately 3% of the total points for the UKF-based experiments of scaphoid and pelvis). computation time. We have observed that, if initialized properly, the UKF-based method required slightly more DRR generations than the simplex-based method to reach a successful registration. But the differences were very small (within

A New Method for CT to Fluoroscope Registration 897 Scaphoid CT to DRRs pose 1 Pelvis CT to DRRs Femur CT to fluoroscopes 1 8 6 4 2 1 2 1 8 6 4 2 1 2 1 8 6 4 2 1 2 2 16 12 8 4 1 2 2 16 12 8 4 1 2 1 8 6 4 2 1 2 Fig. 3. Distributions of s for selected experiments. The first row shows the results for the UKF-based method, and the second row shows the results for the simplex-based method. The units of s are millimeters. Table 1. Capture ranges for all experiments Scaphoid pose 1 Scaphoid pose 2 Pelvis Femur CT to DRRs Femur CT to fluoros UKF 12mm 12mm 2mm 2mm 11mm Simplex.8mm.mm 9.6mm 9.2mm 8mm 8 8 6 rotx roty rotz 6 transx transy transz variance 4 variance 4 2 2 1 6 11 16 21 iteration number 1 6 11 16 21 iteration number Fig. 4. The variance curves of one experiment (CT to DRRs registration, femur data). Left: the variance curves for rotation parameters. Right: the variance curves for translation parameters. DRR generations, which takes less than seconds for a graphics card producing DRRs at rate 3fps). We also observed that, if the initial position was far away from the gold-standard, at some point the UKF-based method required less DRR generations than the simplex-based method, which indicates that the UKF-based method has faster convergence speed. The variance provided along with the state by UKF gives additional information about the confidence on transform parameters. If the variance of a parameter is small enough, we say that the estimated parameter value is accurate. The variance curves will tell whether the registration converges or diverges. Figure 4 shows

898 R.H. Gong, A.J. Stewart, and P. Abolmaesumi in one experiment how the variance of each parameter changes as the registration converges to the gold-standard. 4 Conclusions We presented a new method for CT to fluoroscope registration that uses UKF to search for an optimal solution and a hardware-based volume rendering technique for fast DRR generation. The experimental results showed that our method outperformed the simplex-based method by having a larger capture range while providing comparable accuracy and computation time. In addition, the new method provides the variance of the resulting parameters as a by-product, which can be used as an additional metric to judge the confidence about the registration. We can conclude that, in the situations that simplex-based method could not perform consistently well, UKF is a preferable alternative choice for the 2D-3D registration problems. Future work will include the extension of the method to register multi-fragment bone fractures simultaneously to a set of intra-operative fluoroscope images. Finding the UKF parameters in a systematic way is also an area deserving further research. References 1. Tomazevic, D., Likar, B., Permus, F.: 3-d/2-d registration by integrating 2-d information in 3-d. IEEE Trans. Med. Imag. 2(1) (26) 17 27 2. Bethune, C., Stewart, J.: Adaptive slice geometry for hardware-assisted volume rendering. Journal of Graphics Tools 1(1) (2) 7 3. Russakoff, D., Rohlfing, T., Rueckert, D., Shahidi, R., Kim, D., Maurer, C.: Fast calculation of digitally reconstructed radiographs using light fields. In: Medical Imaging. (23) 4. J.Bayouth, D.L., Kanade, T.: Transgraph: Interactive intensity-based 2-d/3-d registration of x-ray and ct data. SPIE Medical Imaging 3979 (2) 38 396. Lacroute, P., Levoy, M.: Fast volume rendering using a shear-warp factorization of the viewing transformation. Computer Graphics 28 (1994) 41 48 6. Penney, G., Weese, J., Little, J., Desmedt, P., Hill, D., Hawkes, D.: A comparison of similarity measures for use in 2-d/3-d medical image registration. IEEE Trans. Med. Imag. 17(4) (1998) 86 9 7. Livyatan, H., Yaniv, Z., Joskowicz, L.: Gradient-based 2-d/3-d rigid registration of fluoroscopic x-ray to ct. IEEE Trans. Med. Imag. 22(11) (23) 139 146 8. Zollei, L., Grimson, E., Norbash, A., Wells, W.: 2d-3d rigid registration of x- ray fluoroscopy and ct images using mutual information and sparsely sampled histogram estimators. CVPR 2 (21) 696 73 9. van de Kraats, E., Penney, G., Tomazevic, D., van Walsum, T., Niessen, W.: Standardized evaluation of 2d-3d registration. In: MICCAI. (24) 74 81 1. Tang, T., Ellis, R., Fichtinger, G.: Fiducial registration from a single x-ray image: A new technique for fluoroscopic guidance and radiotherapy. In: MICCAI. (2) 2 11 11. Wan, E., van der Merwe, R.: The unscented kalman filter for nonlinear estimation. In: IEEE Symposium 2 (AS-SPCC). (2)