Oriented Gaussian mixture models for non-rigid 2D/3D coronary artery registration

Transcription

1 Oriented Gaussian mixture models for non-rigid 2D/3D coronary artery registration N. Baka, C. T. Metz, C. J. Schultz, R.-J. van Geuns, W. J. Niessen, T. van Walsum Abstract 2D/3D registration of patient vasculature from preinterventional CTA to interventional X-ray angiography is of interest to improve guidance in percutaneous coronary interventions. In this paper we present a novel feature based 2D/3D registration framework, that is based on probabilistic point correspondences, and show its usefulness on aligning 3D coronary artery centerlines derived from CTA images with their 2D projection derived from interventional X-ray angiography. The registration framework is an extension of the Gaussian mixture model (GMM) based point-set registration to the 2D/3D setting, with a modified distance metric. We also propose a way to incorporate orientation in the registration, and show its added value for artery registration on patient datasets as well as in simulation experiments. The oriented GMM registration achieved a median accuracy of.6 mm, with a convergence rate of 8% for non-rigid vessel centerline registration on 2 patient datasets, using a statistical shape model (SSM). The method thereby outperformed the iterative closest point (ICP) algorithm, the GMM registration without orientation, and two recently published methods on 2D/3D coronary artery registration. Index Terms GMM, point-set, SSM, PCI I. INTRODUCTION Fusing pre-interventional and interventional image information during image guided vascular interventions is expected to improve guidance in interventions such as percutaneous coronary intervention (PCI), or interventional neuroradiology. The fusion is generally carried out via 2D/3D registration of vessel trees extracted from pre-operative CT or MR angiography to their 2D projections in intra-operative X-ray fluoroscopy [], [2], [3], [4]. In this work we propose a novel registration algorithm for the alignment of mono-plane 2D and 3D vessel trees, and show its performance for rigid as well as non-rigid registration. Various methods have been proposed for 2D/3D registration, see [5] for a recent review. The transformation model used in 2D/3D vascular registration can be a rigid model, such as for the cerebral vasculature [6], [7], [8] and coronary arteries if matching is performed in CT an X-ray images acquired at the same cardiac phase [9], []. Also nonrigid transformations can be used, e.g. for coronary artery registration [], [], [4], [2], [3], and liver vasculature registration [3]. Asterisk indicates corresponding author. N. Baka, T. van Walsum and W. J. Niessen are with the Departments of Medical Informatics and Radiology of the Erasmus MC - University Medical Center Rotterdam, P.O. Box 24, 3 CA Rotterdam, the Netherlands ( n.baka@erasmusmc.nl). C. T. Metz was with the Departments of Medical Informatics and Radiology, Erasmus MC, Rotterdam, The Netherlands C. J. Schultz was at the Department of Cardiology, Erasmus MC, Rotterdam, The Netherlands R.-J. van Geuns is at the Department of Radiology and Cardiology, Erasmus MC, Rotterdam, The Netherlands Regarding the similarity metric used, vascular 2D/3D registration methods can be categorized into image based (dense) or feature based (sparse) methods. Here we focus on feature based methods due to their favorable computational properties. In feature based vascular 2D/3D registration, 2D and 3D centerlines are generally considered as sets of points without known correspondence. The misalignment between 2D and 3D centerlines is most often calculated in the 2D plane after projection of the 3D centerline [], [2], [3], [4], [8], [9], [6], [2], [4]. The point correspondence calculation between 2D and 3D centerlines required for quantifying their distance may be implicit, using distance maps [4], [6], [8], [9], [4], [5] and KD-trees [2], or explicit by point correspondence optimization [], [2], [6]. Implicit methods tend to be fast, but as their point correspondence mainly relies on the Euclidean distance, their optimization may be vulnerable to local minima caused by wrong correspondences. Explicit methods can formulate point correspondence assignment in a more sophisticated manner, e.g. by enforcing a one-to-one relation between 3D and 2D centerlines, such as proposed by [2]. In this case, however, a large amount of vessel points and false centerline detections make the matching slow. Other information commonly included in the correspondence building is vessel orientation [], [2], [2] and vessel likelihood [], though further descriptors such as shape context [7] may also be used to ensure a realistic point correspondence. Despite the growing number of included features, the calculated correspondences may be wrong causing local minima in the optimization. The 2D/3D point-set registration methods cited above were inspired and generalized from 2D/2D and 3D/3D point-set registration methods. The most common 3D/3D point-set matching algorithm is iterative closest points (ICP) by Besl et al. [8]. In ICP a point correspondence assignment by minimal Euclidean distance, and a consequent best transformation minimizing the sum of squared distances is alternately calculated. Many variants of this algorithm have been proposed since the 99-s, to make point correspondence more efficient [9], [2], more educated [2], or less crisp [22], [23], [24]. As discussed in the previous paragraph, 2D/3D registration evolved in terms of efficiency, and using additional features besides the Euclidean distance. However, till now, little to no work has been done to assess the effect of crisp correspondence assignment. In 3D point registration problems, it was shown that local minima can be avoided by assigning softcorrespondences from one point to several points rather than assigning one crisp correspondence [22]. Soft-correspondences were assigned using a Gaussian function and the Euclidean distance, with decreasing Gaussian width. In coherent point drift proposed by Myronenko et al. [24], one of the point sets

2 2 was regarded as a Gaussian mixture model, being the generator of the other point set. Alignment thus optimized the likelihood that one point-set was generated by the other. To account for outliers in the registered dataset, a small uniform distribution can be added to the Gaussian mixture. The method was shown to outperform ICP, and was robust to outliers if the uniform distribution was weighted properly. Jian and Vemuri [23] proposed a more symmetric probabilistic alignment, regarding both point-sets as Gaussian mixture models, and optimize the overlap L2 distance of the two mixtures. This method was shown to outperform coherent point drift and LM-ICP [25] in robustness, and achieved a high accuracy and good convergence in non-rigid registration of real datasets. In this paper, we propose a probabilistic 2D/3D registration method for vessel centerline alignment. This method is an extension of the Gaussian mixture model (GMM) based 3D/3D registration method [23]. The contributions of this work are as follows: We extend the GMM registration framework to the 2D/3D case, and propose a novel distance metric to be optimized. We propose a way to incorporate orientation information in the registration for registering structured point-sets, such as vessel trees or surface points. We apply non-rigid 2D/3D GMM registration with the use of statistical shape models (SSM). Another advantage of GMM based registration over other numerical optimization based algorithms is the analytical calculation of the Jacobian matrix needed for gradient descent based optimizers. This is expected to speed up registration by eliminating the Jacobian estimation using finite differences, especially when large number of parameters have to be optimized, such as during non-rigid registration. In this work nonrigid registration of the 3D centerlines and the 2D centerlines of a fluoroscopic frame is performed using a patient and cardiac phase specific SSM of the coronary arteries. We evaluate the method in simulation experiments as well as on 2 patient datasets. In the simulation experiments we assess the effect of missing centerline segments and false positive centerline detections on the registration performance, as those issues are likely to occur in X-ray images due to low vessel contrast, instruments, and surrounding anatomical structures. We compare the results to three crisp correspondence based methods: the 2D/3D iterative closest point algorithm (ICP), a recent 2D/3D vascular registration algorithm developed in our lab [], and a thresholded distance transform based 2D/3D registration method similar to [4], [9], [5]. II. METHODS In this section we first shortly summarize the 3D/3D GMM as proposed by Jian and Vemuri [23]. We then introduce the novel GMM based 2D/3D registration, and describe the oriented GMM registration framework developed for directed point clouds. In the last subsection statistical shape models are proposed for parameterizing non-rigid registrations. A detailed description of incorporating SSM based deformation parameterization in the GMM registration framework is given. A. 3D/3D GMM In the Gaussian mixture model registration presented by [23] both the moving point-set M and scene point-set S are regarded as continuous density functions. The probability density function of a Gaussian mixture is defined as p(x) = k i= ϕ(x µ i, Σ i ), where ϕ(x µ i, Σ i ) is a Gaussian distribution with mean µ i and variance Σ i. In the GMM registration k is the number of points in the point-set, µ i -s denote the spatial positions of the points, and all variances are equal and isotropic. The actual size of the variance is a free parameter that can be set by the user. Due to its computational efficiency, the L2 distance metric is chosen as the similarity metric between Gaussian mixtures: d L2 (S, M, θ) = (gmm(s) gmm(m θ )) 2 dx, () where gmm(q) refers to the Gaussian mixture density constructed from point-set Q, and M θ = T (M, θ) represents the moving point-set transformed with a transformation parametrized by vector θ. With substitutions f = gmm(t (M, θ)) and g = gmm(s), the cost function can be written as d L2 = f 2 dx 2 fg dx+ g 2 dx. Since g remains unchanged, the optimization only concerns the first two terms. In case of rigid registration the first term is also constant, and can be disregarded. The metric can be efficiently implemented as fg dx = i,j fi g j dx, where i and j run over all points in M and S, and f i g j dx is related to Gaussian convolution, and has a closed-form solution: ϕ(x µ, Σ )ϕ(x µ 2, Σ 2 )dx = ϕ( µ µ 2, Σ + Σ 2 ). (2) For numerical optimization of the similarity metric with respect to the transformation model parameterized by θ, the gradient can be analytically calculated by applying the chain rule: d L2 θ = d L2 M θ T (M, θ) = G, (3) M θ θ θ where G = d L2 M θ can be calculated as a byproduct of the Gaussian convolution while calculating d L2. The transformation derivative T (M,θ) θ has an analytic solution for rigid transformation models as well as for linear non-rigid deformation models such as B-splines, thin plate splines [23], and SSM based deformation parameterizations (see Section II-D). B. 2D/3D GMM The main difference between the 2D/3D and the 3D/3D GMM based registration framework is the cost function minimized. To calculate the cost function of the 2D/3D method, we propose to project the 3D moving points to the scene plane prior to constructing the mixture density. The L2 distance in plane is thereby: d 2D/3D L2 (S, M, θ) = (gmm(s) gmm(p (M θ ))) 2 dx, (4) where P ( ) represents the projection transformation, S the 2D point-set, and M the 3D moving point-set. For numeric optimization of this metric with respect to θ (the parameterization

3 3 of the transformation model) the gradient can be calculated analytically as d L2 θ = d L2 ) P (M θ) M θ P (M θ M θ θ = G P (M θ) (M, θ) T. M θ θ (5) In the case of parallel projection the term P (M θ) M θ will simply be equal the projection matrix. In the case of perspective projection, the effective projection matrix depends on the projected point s distance from the plane. The general perspective projection gradient is thereby P (M θ ) = P M θ M θ = (6) M θ M θ (d S /d Mθ )P M θ = P Mθ d S M θ d 2 P M θ n T, (7) M θ where n is the plane normal, and P Mθ = d S /d Mθ P represents the effective projection matrix at any point in M θ, that is derived from the parallel projection matrix P and a scaling calculated by the source-plane d S and source-point d Mθ distances. A derivation of Equation 7 is given in the appendix. The L2 metric d L2 f 2 dx 2 fg dx consists of two terms. The second term is the most important, as it is maximal when the Gaussian mixtures f and g overlap most, dragging the two point-sets toward each other. The first term is the inner product of f with itself. This term is minimal if the points in the moving point set are so far that for any Gaussian f i and f j i in the mixture f i f j dx. The value of this term increases with decreasing point distance, it thus counter-acts shrinkage or self-intersection in the moving pointset. While this is justified in 3D, penalizing self intersection in the 2D/3D context will penalize intersection of centerlines due to projection, resulting in projection directions that are less preferred than others. This may create false optima in the registration. The overlap measure alone on the other hand may have an optimum when the moving point-set projects to a single point on the plain. An intermediate solution may suit better the 2D/3D optimization task. We therefore introduced a weighting α, such that d L2 α f 2 dx 2 fg dx, α [, ], (8) where α = results in the original L2 metric, and α = results in a pure overlap metric. C. Oriented GMM (OGMM) We aim to match coronary segments with the same orientation. Including orientation in the GMM based point matching may therefore provide extra robustness in the registration. In the remainder of this section we describe the novel cost function used for 2D/3D GMM based registration of oriented points. Similar to the 2D/3D GMM introduced in Section II-B, the 3D moving points are projected on the plane before calculating the cost function. Unlike the previous section, however, we An extension of the 3D/3D GMM registration to oriented points can be derived similarly. now represent each point p in S and P (M θ ) by a four element vector containing its 2D coordinates (p x, p y ) and a unit orientation vector (v x, v y ). p = p x p y v x v y, where v2 x + v 2 y =. (9) With this representation a 4D GMM registration can be applied, minimizing the same distance function as defined in Equation 4 and 8. Care should be taken though when setting the variance of the multi-dimensional Gaussian of each point. Generally, the covariance matrix Σ will be block-diagonal as point coordinates and directions are independent: ( ) Σp Σ =, () Σ v where Σ p containing point coordinate variances can be set similar to the standard GMM, enabling multi-scale optimization. The second block, containing the orientation variance Σ v, relates to the angular tolerance and not to the scalespace property of the method. It is therefore set differently. We propose a diagonal covariance matrix with elements σv 2 calculated such, that any points i M and j S which have a direction difference larger than a threshold γ, should have zero overlap, thus F v (i, j) = f i g j dx. In that case this point-pair does not contribute to the gradient, and to the registration, independently from the two points Euclidean distance. The orientation scale σ v is thus set such that ϕ(x v i, Σ v )ϕ(x v j, Σ v ) dx = ϕ( v i v j, Σ v + Σ v ), if () angle(v i, v j ) γ, thus v i v j 2 sin(γ/2). (2) This is accomplished by setting three standard deviations of the resulting Gaussian equal to the length of the difference vector corresponding to γ: 3 2σ v = 2 sin(γ/2). (3) Figure shows an example for γ = 9. Left, the two vectors v i and v j are depicted with their difference drawn as a dashed vector. The upper right graph shows the overlap between two Gaussian distributions against their orientation difference in mm, and the lower graph shows the same in the angle domain. In many situations, an orientation and its opposite cannot be distinguished. This is e.g. the case for 2D/3D coronary artery registration where the 2D orientation is extracted from the local image structure of the X-ray image. The above framework can be used in this situation by taking every point twice in the Gaussian mixture model, once with a positive, and once with a negative orientation. This will not affect the distance function value, if the orientation variance is set such that the effect of any pair of points with an angle larger than 9 is suppressed, as illustrated in Figure 2. The overlap value generated by a vector v i, and the vectors v j and v j becomes: F v (ij) = ϕ(v i v j, 2Σ v ) + ϕ(v i + v j, 2Σ v ). (4)

4 4 F v (i,j).5 F v (i,j).5 2 v i v j angle(i,j) in degrees Fig.. Diagrams showing the angular overlap F v (i, j) for point i and j when threshold γ = 9. The upper right graph shows the overlap depending on the vector difference, while the lower graph shows the same overlap measure plotted in the angle domain. Actually, setting the orientation scale larger than for 9 would also not deteriorate the optimization outcome, as optimization is more concerned about the gradient than the actual function value. Figure 3 shows overlap values of two vectors of varying angles with different orientation scales. The orientation scales were calculated for a large range of angle thresholds (γ = 4 to 8 ). The largest scale is beyond the range of scales that can be represented with an angle threshold. All curves have in common that the gradient at an angle difference of 9 is zero, and there is a slope toward smaller angle differences, guiding the optimization in the correct direction. F v (i,j) angle(i,j) in degrees Fig. 2. Illustration of the case when orientation v j and its opposite v j cannot be distinguished. Left: unit circle depicting that one of the angles between ±v j and v i is always 9, and the other is 9. Right: Diagram showing the Gaussian overlap when threshold γ = 9 is used to set the orientation covariance. The blue curve shows the point j with the original orientation, and the magenta curve with its opposite. (i,j) F v.5 γ=4 γ=6 γ=9 γ=2 γ=8 σ v = angle(i,j) in degrees Fig. 3. Orientation overlap versus the angle of two vectors, plotted for different orientation scales (γ thresholds). D. Non-rigid registration using a SSM In this work we use statistical shape models for parameterizing non-rigid registration. Such deformation parameterization is more restrictive than B-spline or thin-plate-spline (TPS) based registrations, which is advantageous in the ill-defined problem of mono-plane 2D/3D registration. We first describe the construction of the SSM, and subsequently describe its use for non-rigid registration. Statistical shape models (SSM) can model shape variation from a population [26]. This is achieved by representing every training shape s by anatomically corresponding points. After eliminating the pose differences from the shapes, each shape can be regarded as a point in a space with dimensionality equal to the number of corresponding points times the point dimensionality. The set of training shapes forms a point cloud in this space, on which principal component analysis (PCA) can be applied to derive a linear model of shape variations: m = m + Θb + ϵ, (5) where m is the mean shape, Θ is a matrix with the main modes of variation in its columns, b is the parameter vector, and ϵ is a remaining error when applying lossy PCA. The model can be used to generate unseen shapes by varying the parameter vector b. The probability of a shape to occur in nature depends on the distance of parameter b from the Gaussian distribution derived from the training shapes. This distance is commonly quantified by the Mahalanobis distance d Mahal = b T C b, with C being the covariance matrix of the training shapes in parameter space (a diagonal matrix with the variance of the modes as elements). To use statistical shape models for non-rigid registration, the moving point set M is generated as M(b) = m + Θb, (6) where m and Θ are pre-computed from a training set, and b parametrizes the non-linear shape changes. We ensure that SSM parameters stay within a plausible range by adding the Mahalanobis distance penalty d Mahal to the distance function, such that d SSM L2 (S, m, b, θ) = d L2 + βd Mahal, (7) where β is a weighting constant. For numerical optimization, the gradient of this cost function with respect to the deformation parameters b is calculated as d SSM L2 b = d L2 b + β d Mahal b, (8) with d L2 / b calculated similarly to Equation 5, and d Mahal / b = 2C b. A. Data III. EXPERIMENTAL SETUP The test datasets were collected from PCI patients for which both pre-interventional 4D CTA and interventional X-ray angiography (XA) were available. The XA data from [] was used in this paper. The data was acquired on a Siemens Axiom Artis biplane system. As no calibration data was available,

5 5 only sequences of the primary C-arm were included. For evaluation, 2 XA datasets were used, coming from patients (two patients underwent two interventions). We selected one heart cycle from each sequence for registration and evaluation. Sequences contained five right and seven left trees. Pixel size of the XA data was.22x.22 mm2. One sequence was acquired at a frame rate of 3 frames per second (fps), the others at 5 fps. The average number of frames in a cardiac cycle was 3. The statistical shape model for the coronary arteries at a given cardiac phase was built from 4D CTA datasets. This training set consisted of 5 patients scanned between 26 and 2 in our hospital, including the patients from the test dataset. For all experiments the current test patient was excluded from the training set, resulting in an actual training set size of 5. The images include a large variety of anatomies and pathologies, and were retrospectively obtained for the model building. CTA images were reconstructed at every 5% of the cardiac cycle. The voxel size of the 3D+t CTA sequence was approximately.7x.7x.9 mm3. The same datasets were used in [], [27]. B. 2D and 3D vessel extraction The goal of this work was to register 2D and 3D vascular centerlines. The 3D centerlines were extracted semiautomatically form pre-interventional end-diastolic CTA images, as proposed in [28]. The aortic ostium and vessel end points were manually selected. Subsequently, the centerline connecting the ostium and the vessel end point was extracted with a minimum cost path approach on a combined vesselness filtered and intensity image [29]. Three scales were used for the vesselness filter (.8,.26, and 2 mm) with parameters set to: αv =.5, βv =.4, γv = 23. The centerlines were then refined by segmenting the vessel [3] and recalculating the center. Finally, the vessel segments were combined to form a 3D vessel tree. The 2D centerlines were automatically extracted from interventional X-ray angiography (XA) images as follows. First, static objects were eliminated by subtracting the median of the first frames from each frame in the sequence. Then, the vessels were enhanced using the Frangi vesselness filter [29] using three scale levels (.6,, and 2 mm), with parameters αf and βf set to.5, as proposed in the original paper, and γf set to.67 times the maximum Hessian eigenvalue, but never larger than. The three scales were combined by taking the maximum response at each pixel. Subsequently, a nonmaximum suppression was used to retain the ridges in the image. These ridges were then blurred and thresholded using a hysteresis threshold with upper bound.7, and lower bound.. Finally, centerlines shorter than 8 pixels were removed. An example frame is shown in Figure 4. C. Patient and phase specific SSM To be able to use the end-diastolic CTA derived 3D centerline for matching to an X-ray frame of a different cardiac phase f, the possible shapes of the 3D centerline at that phase were modeled. This model was built similar to the approach Fig. 4. Example of the automatic centerline extraction. From left to right: original frame, frame after background removal, and extracted centerlines. presented in []. We hypothesized that coronary arteries move together with the cardiac surface, thus their motion can be interpolated from the motion of the near-by cardiac surface. We then segmented the 4 chambers and the aorta from all 4D cardiac CT images with point correspondence, constituting the training set. For each test patient the end-diastolic CTA was segmented in the same way resulting in 4 chambers and the aorta with point correspondence to the training set. In this end-diastolic time-point we defined the landmark points that were near-by the 3D coronary centerline. Finally, for a given phase f, we looked up the motion of the relevant landmark points from end-diastole in all training volumes, and used this motion to interpolate the 3D coronary artery shape at that phase. This results in 5 estimates of the shape, from which we built a statistical shape model using principal component analysis (PCA) [26]. We performed this model construction for every fluoroscopic frame. The cardiac phase f at each frame was calculated by linear interpolation from the percentage RR interval in the ECG. D. Evaluation metric The registrations were evaluated in 2D by calculating the average distance from the ground truth 2D centerline points to the projected 3D centerline points. In the experiments with real image data, manual centerlines served as ground truth. Manual centerline annotations on the fluoroscopic frames were only performed for those vessel segments, that were extracted from CTA. In the simulation experiments the projection of the 3D vessel tree served as ground truth. Evaluation in the missing centerline simulation experiments was performed only on the visible centerline segments. Success was defined as an average distance smaller than 2 mm, in accordance with previous studies [], [4]. E. Experiments We performed three sets of experiments. In the first experiments we assessed the registration performance of the 2D/3D GMM and OGMM methods on simulated data with missing segments, false detections, and non-linear deformations. In the second set of experiments we evaluated the performance on 2 patient datasets, and in the last experiment we analyzed the effect of parameter settings (α regularization weight, γ angle threshold). In the results we denote the regularization weight after the GMM and OGMM abbreviations, thus GMM. denotes GMM registration with α =..

6 6 We compared the performance of the 2D/3D GMM and OGMM in each experiment with three competing methods: 2D/3D ICP;, a numerical registration algorithm recently proposed by our lab []; and, a thresholded distance transform based registration inspired by [4], [9], [5]. For all methods, the non-rigid registration transformation was parametrized by a SSM, as shown in Equation 6. The same SSM, constructed per patient and cardiac phase as described in Section III-C, was used in all methods. For non-rigid registration, a Mahalanobis distance penalty was added to the cost functions of all methods to constrain the parameter values. The 2D/3D ICP was implemented in three steps: ) selecting the closest scene points to all projected model points in 2D, 2) back-projecting the scene points to 3D such that the distance to the corresponding model point is smallest, and 3) updating the model points using rigid and non-rigid transformations to minimize the Euclidean distance between the corresponding 3D point pairs. These three steps were alternated until a maximum number of alternations was reached, or the parameters changed less than a threshold. The metric was implemented as proposed in []. In short, we defined the distance Θ ij between projected centerline point i from M and 2D centerline point j from S as: Θ ij = α G ( F j ) + ( cosγ ij e D 2 ij σ G 2 ), (9) where D ij denotes the Euclidean distance between points i and j, γ ij is the angle between centerline direction at point i and j on the plane, F j is the vesselness value of the 2D centerline at point j, and α G =. and σg 2 [, 5, 3] are user-defined parameters. Correspondence between moving and scene point sets was created by assigning for every point i in the moving set, the point j from the scene point set that minimized Θ ij. The final cost function was the weighted average of the corresponding point distances Θ ij, where the weight was zero if the point i was outside the FOV, and it was the average distance of the point to its neighbors in the projection otherwise. For the distance transform () based registration the distance transform of the scene point set was calculated once before the registration. The cost function value was then attained by projecting the moving centerline points to 2D, and averaging the distances read out from the. For robustness with respect to false centerlines and missing centerline segments, the distance transform was thresholded with decreasing threshold values. Similar cost functions were used in [4], [5], [9]. ) Simulation experiments: In the first set of experiments we assessed the registration performance of GMM and OGMM with regard to missing data, false positive centerline detections, and non-rigid deformations in simulation studies. The simulated 2D projection data was created by projecting the 3D end diastolic vessel centerlines, extracted from CTA, to the actual X-ray detector plane (with projection matrix derived from the DICOM header). Thereby we kept the realistic 3D centerlines and projection angle. The centerlines were downsampled by a factor of 3, from 69 points to 23 points on average. All patient anatomies were used in the simulations. Every experiment was performed 9 times with different starting poses for the registration. The starting poses were drawn randomly, 3 from a uniform distribution with bounds ±3 mm and degree, 3 with ±6 mm and degree, and 3 with ± 9 mm and degree. We performed experiments with %, 25%, and 5% false positive or missing centerlines. An example case for these simulated centerlines is shown in Figure 5. When deleting centerline segments, a random centerline point was selected, around which all points within a random radius were erased. This process was repeated until the number of erased points reached the desired threshold. Similarly, false positive centerline segments of random lengths were added starting from random points, going in random directions, until the desired number of centerline pixels was reached. In the non-rigid deformation a coronary tree generated by the SSM was projected to simulate the 2D centerlines. The model parameters were drawn randomly from the normal distribution of a systolic SSM, and placed randomly away from the starting position. The starting shape for registration was the mean shape of the SSM. The SSM was built retaining 99% of the variance, resulting in approximately 3 shape parameters. An example non-rigid deformed projection is shown in Figure 5. Fig. 5. Example cases for the simulations. First row: complete projected centerline tree, 25% missing centerlines, 5% missing centerline segments. Second row: 25% false centerline segments, 5% false centerline segments, and non-rigidly deformed shape. 2) 2D/3D registration of patient data: In the second set of experiments, the performance of the proposed methods was assessed for non-rigid 2D/3D registration on patient data. The centerlines were extracted as described in Section III-B, and were down-sampled by a factor of 6, resulting on average in 49 points, with a median neighboring point distance of.3 mm. The SSM used for registration in each frame was built as described in Section III-C. Similar to the simulation experiments, we retained 99% of the variance during SSM construction, and down-sampled the SSM by a factor of 3, resulting in 23 points for a vessel tree on average, with a median neighboring point distance of.5 mm. Every experiment was performed 9 times, with different starting poses. Starting poses were provided by 3 observers (3 poses each) on the first frames of each cardiac cycle. This pose included only the in-plane translation; rotation and outof-plane translation initialization was taken from the DICOM header. The initial poses were propagated to all other frames in

7 7 the sequence by using the translation of the coronary arteries obtained from the average heart cycle (using the 4D CTA training set). 3) Parameter influences: In this experiment we investigated the effect of the proposed regularization parameter α, and the angle threshold γ on the 2D/3D registration outcome, based on frames selected from one heart cycle of 2 patient datasets. Registrations were performed with all 9 starting poses described in Section III-E2. The total number of registrations were 9 (starting poses) x 2 (sequences) x 3 (average number of frames per sequence) = 44. When looking at the effect of α, γ was fixed to 6. When investigating the effect of γ, α was fixed to.3. F. Implementation details The tested registration methods have several parameters that have to be set prior to the experiments. An overview of these parameters is given in Table I. The parameters were set based on pilot experiments on two datasets. We left these two datasets in the test set, as their results were comparable to the results of unseen patients. The Mahalanobis weighting β for all methods was set such, that it rarely produced a parameter value larger than three standard deviations from the mean. We used a single scale and increased transformation complexity (translation, rigid, combined rigid and non-rigid) for the GMM method, while decreasing scales with the highest transform complexity (combined rigid and non-rigid) for the OGMM method. These strategies were optimal for the two methods in the pilot experiments. The angle threshold of OGMM was set to γ = 6. To handle small field-of-views (FOV), d L2 was normalized in the implementation with the number of model points used to calculate it, equaling the number of model points in the FOV. For the OGMM experiments we simplified the gradient calculation by only calculating the gradient for spatial motion, not for orientation change. The main motive for this was to reduce the complexity of the computations. When projecting a unit 3D vector, the resulting 2D vector needs to be renormalized to unit length. This non-linear operation makes it difficult to compute the orientation gradient analytically in the 2D/3D case. Without analytical gradient calculation, however, the computation time needed for optimization becomes orders of magnitude higher. Orientation still has an effect on the optimization, as points with a large orientation difference will be further away in the spatial-orientation space than points with similar orientation. Orientation thereby has a weighting effect on the spatial gradient. All methods were implemented in Matlab (R2b, Math- Works), using the optimization toolbox (version 6.). The GMM methods were optimized with a quasi-newton optimizer, as the cost function can be negative. The other methods were optimized with a least squares minimizer (Levenberg- Marquardt). IV. RESULTS We visualize the results as cumulative histograms of the errors up to the convergence limit of 2 mm. Because of outliers, the reported significance values were calculated by paired twosided Wilcoxon rank sum tests at the 5% significance level. A. Simulation Experiments Cumulative histogram of errors SSM simulation 8 6 ICP 4 GMM 2d/3d.5 2 GMM 2d/3d.3 OGMM 2d/3d.5 OGMM 2d/3d Fig. 6. Results of the simulated non-rigid alignment experiments, with no additional or erased centerline segments. Figure 7 shows the results obtained on simulated images with missing centerline segments. The first graph shows that with complete data all methods, except ICP, perform well, achieving median accuracies of.22 mm,.23 mm,.28 mm and.3 mm with the, OGMM.5, and GMM.5 methods respectively. ICP had a median accuracy of.77 mm. All methods except the GMM 2D/3D had convergence rates of 97% or higher. GMM.5 converged in 9% of the cases. When erasing parts of the 2D centerline points, the OGMM method proved most stable. OGMM.5 achieved a median accuracy of.22 mm (9% convergence) with 5% missing segments, while and deteriorated to.66 mm and.7 mm accuracy and 92% and 87% convergence respectively. GMM and ICP deteriorated most in terms of convergence, achieving only 72% and 76% success rates. OGMM thereby significantly outperformed the other methods with half the centerlines missing (p <.9). No real difference between OGMM.5 and OGMM.3 could be noticed. Figure 8 shows the results obtained with added false centerline segments. The % added centerlines represents the same settings as the % erased centerlines, but with different random displacements used for initialization. Accordingly, OGMM outperformed, GMM and ICP (p < 4 ), but was not significantly different than. The median accuracy of in this experiment was though lower than before,.52 mm. By adding false centerlines the median performance of all methods stayed relatively constant. The algorithm deteriorated the most from median error.52 mm to.6 mm, however, it still outperformed ICP having an accuracy of.75 mm (p < 3 ). Figure 6 shows the SSM based simulation experiment results. The best median results were achieved by OGMM.5, and, being.48 mm,.5 mm and.52 mm respectively. OGMM.5 and were not significantly different. ICP was significantly worse than all others (all p <.3). B. 2D/3D registration of patient data Figure shows the results on all frames of all patients for the SSM matching on real data. Unlike the simulation ex-

8 8 TABLE I PARAMETER SETTINGS USED IN ALL RIGID AND NON-RIGID EXPERIMENTS. THE SCALE PARAMETER FOR GMM WAS SET OPTIMAL ACCORDING TO THE PROPOSED FORMULA IN [23], AND RESULTED IN SCALES BETWEEN 7-9 PIXELS. THE REST OF THE PARAMETERS WERE SET IN PRIOR EXPERIMENTS ON TWO DATASETS. Parameters GMM OGMM ICP stopping criteria max func. Eval 4 min. func. change min. param change gradient weight translation (mm) - rotation (degree) - SSM - alpha (see Equation 8) several several beta (Mahalanobis distance weighting) scales (pixel) opt., opt., opt. 2,,5 -,5,3 6,3,5 transformation pyramid transl., rigid, all all, all, all rigid, all transl., rigid, all rigid, all, all optimizer quasi-newton quasi-newton Levenb-Marqu. Levenb.-Marqu. Levenb.-Marqu. finite differences min change Cumulative histogram of errors % erased 8 6 ICP 4 GMM 2d/3d.5 2 GMM 2d/3d.3 OGMM 2d/3d.5 OGMM 2d/3d Cumulative histogram of errors 25% erased Cumulative histogram of errors 5% erased Fig. 7. Results of the simulated rigid alignment experiments with from left to right: %, 25%, and 5% erased centerline segments. Cumulative histogram of errors % added 8 6 ICP 4 GMM 2d/3d.5 2 GMM 2d/3d.3 OGMM 2d/3d.5 OGMM 2d/3d Cumulative histogram of errors 25% added Cumulative histogram of errors 5% added Fig. 8. Results of the simulated rigid alignment experiments with from left to right: %, 25%, and 5% added false centerline segments. periments, on real data OGMM.5 and OGMM.3 performed significantly different (p < 9 ). OGMM.3 performed best with 8% convergence, and median accuracy of.6 mm. It thereby outperformed all other methods (p < 5 ). OGMM.5,,, GMM.5 and ICP performed similar to each other, with median accuracies of.25 mm,.36 mm,.43 mm,.48 mm and.5 mm respectively. Per patient results are heterogeneous in the sense that on 8 patients OGMM.3 outperformed, while on 3 patients performed better. On the dataset of Patient 2 all GMM methods fail. This patient was imaged for the right coronary tree, with many small vessels visible in the X-ray angiography that were not part of the 3D centerlines. Example frames of subject 3, 5, 8 and 2 with projected registered 3D centerlines in blue are shown in Figure 3. Figure 9 shows three stages of the GMM, OGMM, and methods when aligning to a patient frame with additional and missing centerline segments. The median running times for, GMM.5, OGMM.5, ICP and were 2 sec, 5 sec, 6 sec, 4 sec and 3 sec in comparable Matlab based implementations on a single Linux cluster core with 2. GHz speed and 5 GB ram. A partial C implementation of the OGMM method reduced running times to 6 sec. C. Parameter influences Figure shows the effect of changing α on the patient datasets. The graphs show the performance of the ICP, and methods as reference. Both GMM and OGMM methods benefit from lowering α, with an optimal value around.3. OGMM outperforms GMM for α <.7, and performs best for α =., achieving a median 2D centerline distance of.5 mm. GMM outperformed ICP in terms of median error when alpha is set between.5 and.3.

9 9 σ = 2 σ = σ = 5 Transl. Rigid All OGMM.3 GMM.3 Transl. Rigid All Thresh = 6 Thresh = 3 Thresh = 5 Fig. 9. Example fits for three stages of OGMM, GMM, and, for an early systolic frame of a patient. The blue centerline points were automatically extracted from a fluoroscopic frame, and the red crosses show the projected 3D centerline. The final 2D centerline distances for OGMM, GMM, and are:.7 mm,.85 mm,.98 mm, and.79 mm. Fig.. Percentage frames [%] Cumulative histogram of errors Real data ICP GMM 2d/3d.5 GMM 2d/3d.3 OGMM 2d/3d.5 OGMM 2d/3d Results of the non-rigid 2D/3D alignment of patient data. values between 4 and 8 produced similarly good results. A too small angle threshold produced worse convergence as well as higher errors, most probably as only a small fraction of the points could contribute to the gradient. Median error [mm] Conv Median error [mm] 2.5 ICP γ threshold (degree) Fig. 2. Influence of the γ threshold used to set the orientation scale, evaluated on the patient data. The upper graph shows the median 2D centerline distance, the lower the convergence rate. The solid lines show the performance of the and methods for reference. Conv α parameter ICP Fig.. Influence of the α parameter on the registration with GMM (blue diamonds) and OGMM (magenta x-es). For reference the ICP and performance is also marked. The bottom graph shows the convergence rate, the top graph shows the median accuracy. Figure 2 shows the effect of changing the orientation scale according to angle threshold γ (see Section II-C). The method proved robust for setting the orientation scale. In all tested settings, with angle threshold from 2 to 8, the OGMM outperformed GMM. The optimal threshold was γ = 6, but V. DISCUSSION AND CONCLUSIONS We proposed a novel probabilistic 2D/3D vessel centerline registration framework, by extending the GMM registration to 2D/3D, and integrating orientation information in the framework. We evaluated the performance of the 2D/3D GMM and OGMM against three crisp correspondence based methods: ICP; a thresholded distance transform based registration (); and, a method that also takes vessel orientation into account. We hypothesized that the original L2 distance metric for GMM registration would not be appropriate for 2D/3D registration, as it penalizes over-projection of the vessel centerlines on the X-ray plane. However, in clinical practice imaging is also performed in directions which produce overlapping

10 GMM.3, acc = 2.42 mm OGMM.3, acc =.2 mm, acc =.2 mm, acc =.75 mm GMM.3, acc =.78 mm OGMM.3, acc =.62 mm, acc =.99 mm, acc =.38 mm GMM.3, acc =.47 mm OGMM.3, acc =.9 mm, acc =.66 mm, acc =.74 mm GMM.3, acc = 3.36 mm OGMM.3, acc = 3.9 mm, acc = 2.2 mm, acc = 2.22 mm Fig. 3. Example non-rigid 2D/3D fitting results on the real data. Projected 3D centerlines are shown in blue, and the automatically extracted centerline points are shown in red. Rows show one frame from subject 3, 5, 8, and 2. Columns from left to right show fitting results with GMM, OGMM, and. vessels. To solve this problem, we proposed to down-weight the self-intersection penalty term in the L2 function. Our hypothesis was confirmed in the parameter influence experiments (Figure ), where GMM and OGMM with α =.5 and.3 outperformed the variants with the L2 metric.the weight parameter α had an optimum at.3 for GMM and. for OGMM, but a relatively wide range of parameter values provided satisfactory results. A very low self-intersection penalty is not desirable, as it may have the global optimum at a shrunk projected model, translated far from the X-ray source. Incorporation of the orientation feature in the 2D/3D GMM registration framework enhanced the accuracy of the registration, justifying the increased complexity. OGMM.3 outperformed the GMM method in simulation as well as patient experiments, independently of the γ threshold parameter used to set the orientation scale for the Gaussians (Figure 2). Though the best results were achieved with a threshold γ = 6, OGMM proved to be robust against setting the scale parameter

11 (Figure 2). The combination of probabilistic correspondence and orientation feature (OGMM.3) performed the best in this study. The method, employing a robust Euclidean distance measure, and the method also employing an orientation feature, was outperformed by OGMM.3 in the simulation experiments (Figure 7), as well as on patient data. This was the case for several parameter settings of the OGMM (Figure, 2). We reported median running times on the patient datasets (Section IV-B). The method was the fastest with 2 sec. The required time increases linearly with the number of parameters, and stays relatively constant for increased number of points. GMM was the second fastest with 5 sec. The computation of the distance function of GMM is quadratically dependent on the number of points in the naive implementation. The effect of the number of parameters is optimizer dependent, the time needed for Jacobian calculation increases marginally. Evaluation of the OGMM cost function is about four times slower than the GMM due to the doubled point dimensionality and the doubled point size in the scene point set (to include opposite orientations). The original 6 sec computation time with a pure matlab implementation could though be decreased to 6 sec with implementing the cost function in C. The computational cost of was 3 sec, and increases quadratically with the number of points in the naive implementation. In this work the Jacobian for optimization was calculated using finite differences, making computational time linearly dependent on the number of parameters. ICP registration took 4 sec, and is linearly dependent on the number of parameters. The median accuracy of.6 mm achieved with OGMM.3 favorably compares with other 2D/3D coronary artery registration publications. In our previous work, [], we achieved about.4 mm accuracy with a slightly different data preprocessing, and modified error metric. Metz et al. [4] report a median accuracy of.47 mm based on 26 clinical datasets. Serradell et al. [2] achieved a median accuracy of.9 mm on 7 patient datasets. Turgeon et al. [] reported average 3D accuracy of 2.9 mm from realistic, but simulated biplane data reconstruction. Rivest-Henault et al. [5] reported a mean 2D accuracy of.9 mm for 5 patients with bi-plane fluoroscopic data. Care should be taken though with direct comparisons, as all methods use slightly different error metrics, data, and pre-processing. All evaluations presented in this paper were based on projection errors in 2D. The error in the out-of-plane direction is expected to be larger due to the mono-plane setup. In this work we focused on mono-plane 2D/3D registration, as mono-plane fluoroscopic setup is commonly employed for PCI, delivers lower dose to the patient than a bi-plane system, and is the only option if a navigation system is used. The proposed method is easily extended to biplane registration though, if projection geometry is known for each plane. In this case the cost function value of each plane could be added to derive the overall cost, that will be minimized. The code of the 2D/3D GMM and OGMM methods is made publicly available, and can be downloaded under the GPLv3 license from our website: In conclusion, in this work we successfully extended the GMM registration framework to the 2D/3D setting with an adjusted distance metric. Also, we proposed a way to incorporate orientation in the registration, and showed its added value for artery registration on patient datasets as well as in simulation experiments. The oriented GMM registration outperformed GMM without orientation, ICP, and both tested state of the art vessel registration methods. VI. ACKNOWLEDGEMENTS This work was financially supported by ITEA project 939, Mediate. VII. APPENDIX In this appendix we derive Equation 7, for the 2D/3D GMM gradient calculation with perspective projection. The effective perspective projection matrix for any point can be derived from the parallel projection matrix, and a scaling factor. Figure 4 shows the principle. Let Q be any point in the Fig. 4. Perspective projection of a point Q to the plane with normal n, from projection source S. point-set M to be projected, and S be the projection source, whose projection S is the origin on the plane. Let [ ] P P P = 2 P 3 (2) P 2 P 22 P 23 be the parallel projection matrix, such that P Q = Q. In this case Q = P persp Q = ap Q = aq, (2) where P persp is the perspective projection matrix, and a is the scaling factor. Based on the triangle similarity of SQR and SQ S, the scaling can be determined as a = S Q S Q = d S d S = d Q n( Q S), (22) where n is the projection plane normal. We now want to derive the change in projected point location due to the change in 3D point location P persp Q Q = ap Q Q = [ ap Q, ap Q Q y, ap Q Q z ]. (23) As Q = [Q x, Q y, Q z ] is a 3 dimensional vector, it is insightful to write out Q = P persp Q for all coordinates: [ ] P P P persp Q = 2 P 3 a Q x Q P 2 P 22 P y = (24) 23 Q z [ ] P Q = x a + P 2 Q y a + P 3 Q z a, (25) P 2 Q x a + P 22 Q y a + P 23 Q z a were a = d S (Q x n x + Q y n y + Q z n z S x n x S y n y S z n z ). (26)